# Human Experience Filter — Design Spec

**Date:** 2026-06-04
**Status:** Approved design (pre-implementation)
**Repo:** `human-experience-filter-art`

> **Revision — 2026-06-04 (display architecture):** the display moved from *four
> projectors (one per wall) + a procedural side-wall renderer* to a **single
> panoramic projector spanning the three walls the viewer faces, showing the real
> selected video** (launch content focus: **nature video**). This supersedes the
> original §6 (four projectors) and **removes §7 (procedural side walls)**, and
> reshapes the roadmap's sub-project 3 (player drives one pano output) and
> sub-project 5 (procedural side walls — **dropped**). Affected sections below are
> updated in place and marked. The coordinate model (§2), selection (§3), and
> catalog (§4) are unchanged.

---

## 1. What this is

A single-viewer immersive art installation. One person sits in a chair in a small
room; a **single panoramic projector wraps the real selected video across the
three walls they face**. A DJ-style control panel lets them "tune" their
experience along a small set of felt axes; the system finds the public-domain
media nearest that tuning and plays it (launch content focus: **nature video**).
The piece is an *experience filter*: the viewer dials in how they want to feel and
think, and the room answers with found human artifacts that match.

Design constraints that shaped everything below:

- **One viewer, one room, all local.** No networking, no streaming, no
  multi-user, no session recording.
- **Made of real found media.** The library is curated public-domain / openly
  licensed work, not algorithmically generated filler. Curation is the artwork.
- **Built to grow.** Launch lean; the data model and tooling are designed so the
  library can expand over time without rework.

---

## 2. Coordinate model

The control panel defines a point in a 5-control experience-space.

| Control       | Type           | Range                     | Meaning                                   |
|---------------|----------------|---------------------------|-------------------------------------------|
| Content mode  | 4-way selector | None / Audio / Video / A+V | Which media channels are active           |
| Left          | knob           | 0–4                       | How analytical / verbal / structured      |
| Right         | knob           | 0–4                       | How artistic / emotional / abstract       |
| Dark          | knob           | 0–4                       | How somber / heavy / ominous              |
| Light         | knob           | 0–4                       | How uplifting / serene / joyful           |

Two **independent planes**, each formed by two knobs that are *not* opposite ends
of one slider:

- **Brain plane** = Left × Right. Both high → "whole brain." Both low → "quiet
  mind." One-sided → pure analytical or pure artistic.
- **Mood plane** = Dark × Light. Both high → "bittersweet." Both low → "neutral."
  One-sided → pure somber or pure uplifting.

Each plane is a 5×5 grid (25 cells). Combined space is 25 × 25 = 625 knob
configurations per content mode.

**Axis semantics (the tagging rubric):**

- **Left (analytical/verbal/structured):** narration, language, logic, sequence,
  instruction, explanation, documentary, spoken word.
- **Right (artistic/emotional/abstract):** music, abstract visuals, dance,
  nature, awe, non-verbal, dreamlike, intuitive.
- **Dark (somber/heavy):** ominous, melancholic, night, decay, minor-key,
  storms, noir.
- **Light (uplifting/serene):** joyful, hopeful, bright, sunrise, gardens,
  major-key, calm.

**None** is the void / rest state: walls fall to black, audio goes silent.

---

## 3. Selection algorithm

Every catalog record carries a coordinate `(left, right, dark, light)` and a
`mode`. Selection is **nearest-match**, not exact-cell lookup, so a sparse
library still resolves meaningfully at every knob position.

On a knob change:

1. **Filter by mode.** Restrict to records whose `mode` matches the selector.
   In `A+V` mode the pool may also fall back to `audio` and `video` records when
   `av` records near the point are thin.
2. **Distance.** Compute weighted Euclidean distance from the knob coordinate to
   each candidate's coordinate. Weights let the brain plane and mood plane be
   tuned independently later (default: equal weight).
3. **Pool + pick.** Take the N nearest (default N = 3–5) and choose one,
   shuffling within the pool so the same corner doesn't always yield the
   identical piece.
4. **Play.** Crossfade from the current piece. The chosen 5–15 min segment loops
   until the knobs move again.

`None` short-circuits the algorithm: fade to black + silence.

---

## 4. Content catalog (the spine of the repo)

A version-controlled catalog with **one record per piece**. The media files
themselves live on the player's drive (too large for git); the repo holds the
**metadata + pointers**. Format: **JSON Lines** (append-friendly, diff-friendly).

Record schema:

```
id              stable unique id
title           human title
source_url      where it came from
source_archive  e.g. internet_archive | musopen | librivox | nasa | freesound
license         public_domain | cc0 | cc_by | cc_by_nc
attribution     required text when license demands it
mode            audio | video | av
left            0–4   (curatorial)
right           0–4   (curatorial)
dark            0–4   (curatorial)
light           0–4   (curatorial)
duration_s      segment length in seconds
file_path       path on the player drive
resolution      e.g. 1920x1080 (video only)
dominant_color  hex, computed from the media (video only)
review_status   proposed | approved
rationale       one-line justification for the proposed coordinates
reviewed_at     timestamp when the human approved (null while proposed)
notes           free text
```

This catalog is what grows from lean → dense, and what *both* the player and the
sourcing/review tools read. "How far from done" is simply the count of records
still in `review_status: proposed`.

---

## 5. Division of labor — tagging

Tagging decomposes into mechanical fields and one curatorial field group.

- **Mechanical / automatic** (tooling sets these): `mode` (from whether the file
  has meaningful audio/video), `license` + `attribution` + `source_*` (from
  origin), `dominant_color` (computed), `duration_s`, `resolution`.
- **Curatorial** (`left`, `right`, `dark`, `light`): the genuine artistic act.

**Workflow — "I draft everything, you review every piece":**

1. **Discover & ingest** (tooling + assistant): pull candidates from the
   public-domain pools, populate all mechanical fields, write records as
   `review_status: proposed` with **assistant-proposed coordinates** and a
   one-line `rationale` each.
2. **Review** (human, every piece): a review tool walks unreviewed records one
   at a time — showing proposed coordinates, the rationale, and a preview frame —
   and the human accepts or types a correction. Acceptance flips the record to
   `approved` and stamps `reviewed_at`.

Every coordinate is human-blessed before it can be selected in the room
(optionally the player can be restricted to `approved` records only).

---

## 6. Hardware

The microcontroller and the media player are **separate roles** — an Arduino
cannot decode HD video or drive projectors.

- **Arduino — control panel only.** Reads the four knobs + the 4-way selector and
  sends the five values to the player over USB serial. This is what Arduinos are
  good at.
- **Raspberry Pi 5 (or small mini-PC) — brain + player.** Holds the catalog,
  reads the Arduino's serial stream, runs the selection algorithm, and plays
  media. **The hard drive plugs into the Pi**, not the Arduino.
- **Projector (single, panoramic).** *Revised 2026-06-04.* The Pi drives **one
  panoramic projector that spans the three walls the viewer faces**, showing the
  real selected video across the whole field. There is no longer a primary/side
  split and no procedural side-wall renderer (the former §7). This both simplifies
  the hardware (one output, one projector) and removes the need to *compute*
  `dominant_color` for ambient walls.

Parts delta vs. the original sketch: add a ~$80 Pi and use a single panoramic
(ultra-wide / short-throw) projector instead of four; everything else (Arduino,
knobs, hard drive) stays.

---

## 7. ~~Procedural side walls~~ — REMOVED (2026-06-04)

**Superseded by the §6 single-panoramic-projector revision.** The three walls the
viewer faces now show the **real** selected video (the pano projector spans them),
so there is no separate procedural ambient renderer. The mood axis is felt through
the chosen content itself rather than a synthetic side-wall wash.

Consequences:

- **Sub-project 5 (procedural side walls) is dropped** from the roadmap.
- `dominant_color` loses its only consumer; computing it becomes **optional /
  opt-in** in the ingest tooling (sub-project 2 spec §5.2) rather than a required
  mechanical tag. The field stays in the catalog schema (unused-but-harmless,
  default `""`) for a possible future ambient/lighting use.

> *Original intent (for the record): the three non-primary walls rendered a slow
> color wash — hue from the playing piece's `dominant_color`, brightness from the
> Dark/Light knobs — to make the mood axis felt in peripheral vision.*

---

## 8. Sourcing strategy (lean now, dense later)

Pools, prioritized by license cleanliness, mapped to the axes:

- **Left-brain (verbal/analytical):** LibriVox (readings), Prelinger / Internet
  Archive educational & industrial films, public-domain lectures, instructional
  animation.
- **Right-brain (abstract/artistic):** Musopen (public-domain classical), NASA
  imagery & space footage (public domain), early/silent cinema dream sequences,
  abstract animation, nature.
- **Dark mood:** storms, night, noir, minor-key works, decay, somber readings.
- **Light mood:** sunrise/gardens, major-key works, bright nature, uplifting
  verse.
- **Audio pools:** Musopen, Free Music Archive (CC), Internet Archive audio,
  Freesound (CC0 field recordings).

**License stance:** prefer Public Domain / CC0; accept CC-BY (attribution
recorded in the catalog); CC-BY-NC acceptable only while the installation stays
non-commercial. Every record stores its license + source so each piece is
defensible.

---

## 9. Sizing

- **Pieces:** lean launch ~**120–200** tagged pieces; design supports growth to
  ~**800**.
- **Per piece:** medium segment ≈ 10 min. Primary-wall video at 1080p ≈
  0.6–0.8 GB (~3 GB at 4K). Audio ≈ 20–100 MB.
- **Drive:** **1 TB** SSD for a 1080p library, **2 TB** for 4K, both with
  comfortable headroom. The drive is not the constraint; curation labor is.

---

## 10. Repo layout

```
hef/          shared library: catalog model + selection (sub-project 1, built)
catalog/      the tagged content catalog (JSONL)
player/       media player driving the single panoramic projector (runs on the Pi)
firmware/     Arduino sketch (reads knobs/selector → USB serial)
tools/        sourcing/ingest, mechanical tagging, review tool (sub-project 2)
docs/         this spec + operator/build guide
```

*(Revised 2026-06-04: `sidewalls/` removed with §7; the selection algorithm
shipped in the shared `hef/` package, not under `player/`, since the tools import
it too — see the sub-project-1 plan.)*

---

## 11. Explicitly out of scope (YAGNI)

- Networking / streaming — everything runs locally on the Pi.
- Multi-user / multi-viewer.
- Session recording or analytics.
- Automatic ML coordinate tagging — coordinates are a human curatorial act by
  design.

---

## 12. Open implementation questions (for the plan, not blockers)

- Player stack on the Pi (e.g. mpv/ffmpeg-based vs. a custom renderer), how
  crossfades are handled, and how a single wide/panoramic output is driven (one
  ultra-wide surface vs. spanned displays).
- Exact serial protocol/framing between Arduino and Pi.
- Whether the player hard-restricts to `approved` records.
- ~~Whether side walls run on the primary Pi's extra outputs or a second Pi~~ —
  resolved: no procedural side walls (§7 removed).
- ~~Ingest tooling language and where the per-archive downloaders live~~ —
  resolved by the sub-project-2 spec: Python under `tools/`, per-archive fetchers
  in `tools/ingest/`.
