# Human Experience Filter — Roadmap

The installation is built as **five independent sub-projects**, each producing
working, testable software on its own. Each gets its own spec → plan → implement
cycle. This file is the index and dependency map; it is the source of truth for
"what's left."

Design reference: [`specs/2026-06-04-human-experience-filter-design.md`](./superpowers/specs/2026-06-04-human-experience-filter-design.md).

---

## Status at a glance

| # | Sub-project                     | Status        | Unblocks |
|---|---------------------------------|---------------|----------|
| 1 | Catalog & Selection Core        | ✅ Done        | everything |
| 2 | Ingest & Tagging / Review tools | ✅ Done        | a real library |
| 3 | Player Runtime (Pi)             | ⏳ In progress | the room runs |
| 4 | Arduino Firmware (control panel)| ◻ Not started | real knobs |
| 5 | ~~Procedural Side Walls~~       | ❌ Dropped     | — (superseded) |

**Dependency order:** 1 → 2, and 1 → 3. Sub-projects 3 and 4 share a serial
protocol contract and can then proceed in parallel. Sub-project 5 was **dropped**
(2026-06-04): the move to a single panoramic projector showing real video across
the three walls removes the procedural side walls — see the design spec §6/§7.

```
        ┌─────────────────────────┐
        │ 1. Catalog & Selection  │ ✅
        └───────────┬─────────────┘
                    │
        ┌───────────┴───────────┐
        ▼                       ▼
┌───────────────┐      ┌─────────────────────┐
│ 2. Ingest /   │      │ 3. Player Runtime   │
│    Review     │─────▶│    (Pi) → 1 pano     │
└───────────────┘ feeds│    projector         │
   (fills catalog)      └──────────┬──────────┘
                          serial   ▼
                       ┌────────────────┐
                       │ 4. Firmware    │
                       └────────────────┘

(5. Procedural Side Walls — dropped 2026-06-04, single pano projector)
```

---

## 1. Catalog & Selection Core ✅

**Done — merged to `main`, 37 tests.** The dependency root every other
sub-project imports.

- `hef/catalog.py` — `Record` model, validation, JSONL load/save/append.
- `hef/selection.py` — weighted-Euclidean distance, mode filtering with A+V
  fallback, nearest-match `select()`.

Plan: [`2026-06-04-catalog-and-selection-core.md`](./superpowers/plans/2026-06-04-catalog-and-selection-core.md).

---

## 2. Ingest & Tagging / Review tools ✅

**Done — merged to `main` via PR #1 (session 0002, 2026-06-04); 111 tests pass
(2 opt-in real-ffprobe tests skip when the binaries are absent).** The `tools/`
package turns the manual catalog into the assisted *draft-then-review* flow from
the spec, and can populate the library from public-domain sources.

**Delivers (`tools/`):**

- **Ingest** — per-archive fetchers (Internet Archive / Prelinger, Musopen,
  LibriVox, NASA, Free Music Archive, Freesound) that download a candidate and
  write a catalog record.
- **Mechanical tagging** — auto-fill `mode` (probe streams with `ffprobe`),
  `license` / `attribution` / `source_*` (from origin), `duration_s`,
  `resolution`, and `dominant_color` (computed from the video).
- **Coordinate drafting** — propose `left/right/dark/light` with a one-line
  `rationale`, written as `review_status: proposed`.
- **Review CLI** — walk unreviewed records one at a time (show proposed
  coordinates + rationale + a preview frame), accept or correct, flip to
  `approved` and stamp `reviewed_at`.

Spec: [`2026-06-04-ingest-tagging-review-tools.md`](./superpowers/specs/2026-06-04-ingest-tagging-review-tools.md).

**Depends on:** sub-project 1 (`hef.catalog`).
**Done when:** ✅ you can run ingest against a source, get `proposed` records, review
them to `approved`, and `load_catalog`/`validate_catalog` validate the result —
covered by tests on the mechanical-tagging and review state transitions.
**External deps:** `ffmpeg`/`ffprobe` only (no image lib — `dominant_color`, when
requested, is ffmpeg-only and opt-in), per-archive download access.
**Decisions settled:** first-ship archives = **LibriVox, NASA, Internet Archive**
(Musopen/FMA/Freesound deferred stubs); media under a `--media-root` with
`file_path` stored relative to it (spec §6.3); only additive `hef.catalog` change
was `validate_catalog` + `index_by_id`.

---

## 3. Player Runtime (Pi) ⏳ (in progress)

**Goal:** the thing that makes the room run — read the controls, **alter** the
neutral base footage toward the knob state, and play it across the single
panoramic projector. Per the machine-altered-perception revision the knobs no
longer *select* a pre-tagged clip; they drive an **alteration engine** (design
§4/§5) over a small neutral base library.

**Slice 1 — alteration engine + player core (pure logic) ✅ Done.** Merged to
`main` (session 0006); 56 tests. The new `player/` package, with all I/O
(video/audio/serial) behind injected interfaces:

- `player/controls.py` — `Controls`, the full panel state (7-way content dial +
  4 experience knobs + volume + brightness); the serial-contract data shape
  shared with sub-project 4.
- `player/content.py` — `resolve_content`, the §6 7-way dial → (audio source,
  video on/off) table.
- `player/alteration.py` — `plan_alteration`, knob vector → `RenderPlan`
  (mood `ColorGrade` with center=identity §5; Left `AnalyticalOverlay`; Right
  `Restyle`), encoding the §4.2 layering rule (substrate vs overlay; Left & Right
  stack).
- `player/state.py` — `Player`, the state machine that diffs successive controls
  into transitions (`FADE_TO_BLACK`/`FADE_FROM_BLACK` on video on/off,
  `CROSSFADE` on clip/restyle-variant swap, `LIVE_UPDATE` for the continuous
  grade/overlay/level ops per §4.3, `NONE` when unchanged).

Plan: [`2026-06-05-player-alteration-core.md`](./superpowers/plans/2026-06-05-player-alteration-core.md).

**Slice 2 — simulator-first alteration preview ✅ Done.** Merged to `main`
(session 0009). Wires the alteration engine into the web simulator so the look is
tunable by eye before any hardware, and **reconciles** the unmerged session-0007
design with the merged session-0008 design (the Left-HUD conflict): Left is a
**runtime overlay** (authored annotation track + per-language string tables),
Right is a **discrete pre-baked** flow-stabilized variant, Dark/Light a **live**
grade — superseding 0007's baked-HUD 5×5 grid. Engine: a parameterized
`Calibration` (tuned by eye, baked into `DEFAULT_CALIBRATION`), `Restyle.variant`
(discrete) replacing the continuous blend, and `AnalyticalOverlay.level`. Sim:
`/api/alteration` + `/api/clips` over `simulator/clips.py`, an alteration-preview
UI, and one neutral clip with a real flow-stabilized Right variant from the POC.
Design:
[`2026-06-07-reconciled-simulator-alteration-slice-design.md`](./superpowers/specs/2026-06-07-reconciled-simulator-alteration-slice-design.md);
plan:
[`2026-06-07-reconciled-simulator-alteration-slice.md`](./superpowers/plans/2026-06-07-reconciled-simulator-alteration-slice.md).

**Slice 3 — scale-ring navigation (simulator) ✅ Done.** Merged to `main`
(session 0011). Adds the **endless rotary-encoder** control — *relative* (vs the
absolute 0–4 experience knobs) — that walks a **closed ring** of neutral
"scales of nature" clips, with placeholder AI zoom/warp **transitions** between
adjacent scales and a small→large **wrap** (the infinite-zoom payoff, scales
design §3). The navigation math is canonical Python (`player/ring.py`:
`ScaleRing` + `advance_ring`, one transition clip per edge, played forward
inward / reversed outward); the sim exposes `GET /api/ring` + `POST
/api/ring/advance` and the browser only plays the returned transition(s) then
settles on the target scale, keeping the live knob alteration on top. Two cheap
**true-PD** scales were added so the ring is demonstrable — `cosmos` (NASA/Hubble)
and `abyss` (NOAA Ocean Exploration), provenance recorded in the manifest, bytes
generated as labelled placeholders by `simulator/setup_scales_media.py`. The
expensive real multi-strength flow-stabilized Right re-bake is **deferred** (new
scales carry a raw base only) until the ring is liked. Plan:
[`2026-06-07-scale-ring-navigation.md`](./superpowers/plans/2026-06-07-scale-ring-navigation.md).
Session 0012 added the design §3 **fast-spin** behavior on top: past an opt-in
speed threshold (`DEFAULT_FAST_SPIN_THRESHOLD = 3`, `abs(delta)` as the spin-speed
proxy) a quick spin collapses to a single **blended pass** instead of chaining
every transition, so fast navigation stays responsive on placeholder media.

**Remaining slices (not started):**

- **Runtime renderer** — drive the single panoramic projector via mpv/ffmpeg;
  GPU shaders for the luma-keyed mood grade + analytical-overlay compositing;
  realize crossfade/fade timing and the 5–15 min loop. (Open decision: player
  stack — mpv via IPC vs. ffmpeg vs. custom.)
- **Serial input** — real USB-serial reader on the Pi + the **3⇄4 framing
  contract** with the firmware (the keyboard/serial stand-in already works via a
  `Controls` stream).
- **White-noise generation** — runtime white/pink noise for that content position.
- **Content pipeline + label track** — source strict-PD neutral bases, run them
  through the offline pipeline, and author the annotation tracks that drive the
  Left-HUD overlay. The Right dream is now **real-time** (Kuwahara WebGL shader,
  session 0013 — no per-clip ML bake), so the pipeline is
  **source → crossfade-loop → master+proxy transcode → hand-authored /
  motion-tracked labels → manifest**, not ML baking. Historical note: the
  flow-stabilized SD-turbo v2v bake (session 0012, `simulator/bake_right_variants.py`)
  was superseded when the operator found the baked-SD Right "too fluid" and the
  design pivoted to the deterministic real-time dream; `bake_right_variants.py`
  is now parked. Spec:
  [`2026-06-24-content-pipeline-design.md`](./superpowers/specs/2026-06-24-content-pipeline-design.md).

  - **Increment 1 — deterministic tooling + 5-scale ring (session 0014) ✅ Done.**
    `tools/pipeline/` (frame/loop/transcode/manifest, pure + unit-tested + an
    opt-in integration test on real footage); the backward-compatible keyframed
    annotation-track schema with client interpolation; and the **full 5-scale ring**
    (cosmos → orbit → forest → reef → abyss) with orbit + reef as placeholder
    scales alongside the existing cosmos/abyss/forest. Plan:
    [`2026-06-24-content-pipeline-increment-1.md`](./superpowers/plans/2026-06-24-content-pipeline-increment-1.md).

  - **Increment 2 — hybrid motion-track pass + author mode (pending).** The
    `tools/pipeline/track.py` classical OpenCV tracker + optional ML detect+track,
    plus a simulator **author mode** for reviewing/editing annotation tracks in the
    browser. Real strict-PD bases for all five scales sourced per
    [`docs/content-sourcing.md`](./content-sourcing.md).

  - **Audio + separated Visual/Audio controls (session 0020) ✅ Done.** The
    simulator now has **sound**: per-altitude soundtracks that follow the Altitude
    dial + a synthesized pink **white-noise** bed, behind two orthogonal controls —
    **Visual** (on/off) × **Audio** (off / soundtrack / white_noise) — replacing
    the bundled 7-way Content dial. `tools/pipeline/audio_ops` + `audio_run` +
    `simulator/build_audio_media.py` (ffmpeg loop/loudnorm/noise, gitignored
    assets), `Scale.audio` in the manifest, a pure `player/audio.py` resolver
    (retiring `player/content.py`), server-resolved `render.audio.url`, and a client
    A/B `<audio>` crossfade layer. This is the "Content step" of the future control
    surface (it reframes the proposed networked-control-surface spec §3/§5). Spec:
    [`2026-06-26-audio-video-separated-controls-design.md`](./superpowers/specs/2026-06-26-audio-video-separated-controls-design.md);
    plan:
    [`2026-06-26-audio-video-separated-controls.md`](./superpowers/plans/2026-06-26-audio-video-separated-controls.md).

  - **Still deferred:** real **i2v ring transitions** (need a generative-video
    model + adjacent real bases); Pi renderer + serial/firmware; the **music**
    audio layer (reserved dial position, no assets yet).
- **Catalog model changes** — audio *source* + "neutral base" vs "altered
  variant" flag (sub-project 2 territory, design §13).

**Depends on:** sub-project 1 (`Coordinate`); a neutral base library from
sub-project 2 to be meaningful, but developable against a hand-authored library
and a **keyboard/serial stand-in** before firmware exists.
**Done when:** given a base library and a stream of control values, it plays the
correct altered segment, loops, crossfades, and goes dark on `Off` — the
decision logic is covered (slice 1, serial mocked); the renderer realizes it.
**Hardware:** Raspberry Pi 5 (holds the drive + base library + variants).
**Open decisions:** player stack (mpv via IPC vs. ffmpeg vs. custom); whether the
player hard-restricts to `approved` records (the `approved_only` flag is plumbed
but inert until backed by a real catalog); knob→strength calibration (the §3 vs
§4.2/§5 reconciliation — see the slice-1 plan / session 0006 transcript).

---

## 4. Arduino Firmware (control panel) ◻

**Goal:** the physical panel — turn knob/selector positions into serial messages.

**Delivers (`firmware/`):**

- Read four knobs (`left`, `right`, `dark`, `light`) and the 4-way content-mode
  selector.
- Quantize/debounce analog inputs to the `0..4` integer scale.
- Send the five values to the Pi over USB serial in a defined framing.

**Depends on:** only the **serial protocol contract** shared with sub-project 3 —
agree that first, then 3 and 4 proceed in parallel.
**Done when:** turning a knob produces the expected serial message (verifiable on
a serial monitor) and the Pi player reacts.
**Hardware:** Arduino + pots/encoders + a 4-position selector.
**Open decisions:** pots vs. detented encoders; exact serial framing.

---

## 5. ~~Procedural Side Walls~~ ❌ Dropped (2026-06-04)

**Superseded.** The move to a single panoramic projector showing the real selected
video across the three walls (design spec §6/§7) removes the procedural ambient
renderer entirely — the mood axis is felt through the chosen content itself. With
it goes the only consumer of `dominant_color`, which becomes optional/opt-in in
the ingest tooling (sub-project 2 spec §5.2).

> *Original goal (for the record): render a slow color wash on the three
> non-primary walls — hue from the playing piece's `dominant_color`, brightness
> from the dark/light knobs — to make the mood axis felt in peripheral vision.*

---

## Cross-cutting decisions to settle as we go

Carried from the design spec's open-questions list — none block sub-project 2:

- **Serial protocol framing** between Arduino and Pi (needed before 3 ⇄ 4).
- **Player stack** on the Pi (mpv / ffmpeg / custom) and how the single panoramic
  output is driven.
- **`approved`-only enforcement** in the player.
- ~~**Side walls** on the primary Pi vs. a second Pi~~ — resolved: side walls dropped.
- ~~**Media storage layout** on the drive~~ — settled by the sub-project-2 spec
  §6.3: media under a `media-root`, `file_path` stored relative to it; the player
  joins it with its drive mount.
