# Human Experience Filter — Roadmap The installation is built as **five independent sub-projects**, each producing working, testable software on its own. Each gets its own spec → plan → implement cycle. This file is the index and dependency map; it is the source of truth for "what's left." Design reference: [`specs/2026-06-04-human-experience-filter-design.md`](./superpowers/specs/2026-06-04-human-experience-filter-design.md). --- ## Status at a glance | # | Sub-project | Status | Unblocks | |---|---------------------------------|---------------|----------| | 1 | Catalog & Selection Core | ✅ Done | everything | | 2 | Ingest & Tagging / Review tools | ✅ Done | a real library | | 3 | Player Runtime (Pi) | ⏳ In progress | the room runs | | 4 | Arduino Firmware (control panel)| ◻ Not started | real knobs | | 5 | ~~Procedural Side Walls~~ | ❌ Dropped | — (superseded) | **Dependency order:** 1 → 2, and 1 → 3. Sub-projects 3 and 4 share a serial protocol contract and can then proceed in parallel. Sub-project 5 was **dropped** (2026-06-04): the move to a single panoramic projector showing real video across the three walls removes the procedural side walls — see the design spec §6/§7. ``` ┌─────────────────────────┐ │ 1. Catalog & Selection │ ✅ └───────────┬─────────────┘ │ ┌───────────┴───────────┐ ▼ ▼ ┌───────────────┐ ┌─────────────────────┐ │ 2. Ingest / │ │ 3. Player Runtime │ │ Review │─────▶│ (Pi) → 1 pano │ └───────────────┘ feeds│ projector │ (fills catalog) └──────────┬──────────┘ serial ▼ ┌────────────────┐ │ 4. Firmware │ └────────────────┘ (5. Procedural Side Walls — dropped 2026-06-04, single pano projector) ``` --- ## 1. Catalog & Selection Core ✅ **Done — merged to `main`, 37 tests.** The dependency root every other sub-project imports. - `hef/catalog.py` — `Record` model, validation, JSONL load/save/append. - `hef/selection.py` — weighted-Euclidean distance, mode filtering with A+V fallback, nearest-match `select()`. Plan: [`2026-06-04-catalog-and-selection-core.md`](./superpowers/plans/2026-06-04-catalog-and-selection-core.md). --- ## 2. Ingest & Tagging / Review tools ✅ **Done — merged to `main` via PR #1 (session 0002, 2026-06-04); 111 tests pass (2 opt-in real-ffprobe tests skip when the binaries are absent).** The `tools/` package turns the manual catalog into the assisted *draft-then-review* flow from the spec, and can populate the library from public-domain sources. **Delivers (`tools/`):** - **Ingest** — per-archive fetchers (Internet Archive / Prelinger, Musopen, LibriVox, NASA, Free Music Archive, Freesound) that download a candidate and write a catalog record. - **Mechanical tagging** — auto-fill `mode` (probe streams with `ffprobe`), `license` / `attribution` / `source_*` (from origin), `duration_s`, `resolution`, and `dominant_color` (computed from the video). - **Coordinate drafting** — propose `left/right/dark/light` with a one-line `rationale`, written as `review_status: proposed`. - **Review CLI** — walk unreviewed records one at a time (show proposed coordinates + rationale + a preview frame), accept or correct, flip to `approved` and stamp `reviewed_at`. Spec: [`2026-06-04-ingest-tagging-review-tools.md`](./superpowers/specs/2026-06-04-ingest-tagging-review-tools.md). **Depends on:** sub-project 1 (`hef.catalog`). **Done when:** ✅ you can run ingest against a source, get `proposed` records, review them to `approved`, and `load_catalog`/`validate_catalog` validate the result — covered by tests on the mechanical-tagging and review state transitions. **External deps:** `ffmpeg`/`ffprobe` only (no image lib — `dominant_color`, when requested, is ffmpeg-only and opt-in), per-archive download access. **Decisions settled:** first-ship archives = **LibriVox, NASA, Internet Archive** (Musopen/FMA/Freesound deferred stubs); media under a `--media-root` with `file_path` stored relative to it (spec §6.3); only additive `hef.catalog` change was `validate_catalog` + `index_by_id`. --- ## 3. Player Runtime (Pi) ⏳ (in progress) **Goal:** the thing that makes the room run — read the controls, **alter** the neutral base footage toward the knob state, and play it across the single panoramic projector. Per the machine-altered-perception revision the knobs no longer *select* a pre-tagged clip; they drive an **alteration engine** (design §4/§5) over a small neutral base library. **Slice 1 — alteration engine + player core (pure logic) ✅ Done.** Merged to `main` (session 0006); 56 tests. The new `player/` package, with all I/O (video/audio/serial) behind injected interfaces: - `player/controls.py` — `Controls`, the full panel state (7-way content dial + 4 experience knobs + volume + brightness); the serial-contract data shape shared with sub-project 4. - `player/content.py` — `resolve_content`, the §6 7-way dial → (audio source, video on/off) table. - `player/alteration.py` — `plan_alteration`, knob vector → `RenderPlan` (mood `ColorGrade` with center=identity §5; Left `AnalyticalOverlay`; Right `Restyle`), encoding the §4.2 layering rule (substrate vs overlay; Left & Right stack). - `player/state.py` — `Player`, the state machine that diffs successive controls into transitions (`FADE_TO_BLACK`/`FADE_FROM_BLACK` on video on/off, `CROSSFADE` on clip/restyle-variant swap, `LIVE_UPDATE` for the continuous grade/overlay/level ops per §4.3, `NONE` when unchanged). Plan: [`2026-06-05-player-alteration-core.md`](./superpowers/plans/2026-06-05-player-alteration-core.md). **Slice 2 — simulator-first alteration preview ✅ Done.** Merged to `main` (session 0009). Wires the alteration engine into the web simulator so the look is tunable by eye before any hardware, and **reconciles** the unmerged session-0007 design with the merged session-0008 design (the Left-HUD conflict): Left is a **runtime overlay** (authored annotation track + per-language string tables), Right is a **discrete pre-baked** flow-stabilized variant, Dark/Light a **live** grade — superseding 0007's baked-HUD 5×5 grid. Engine: a parameterized `Calibration` (tuned by eye, baked into `DEFAULT_CALIBRATION`), `Restyle.variant` (discrete) replacing the continuous blend, and `AnalyticalOverlay.level`. Sim: `/api/alteration` + `/api/clips` over `simulator/clips.py`, an alteration-preview UI, and one neutral clip with a real flow-stabilized Right variant from the POC. Design: [`2026-06-07-reconciled-simulator-alteration-slice-design.md`](./superpowers/specs/2026-06-07-reconciled-simulator-alteration-slice-design.md); plan: [`2026-06-07-reconciled-simulator-alteration-slice.md`](./superpowers/plans/2026-06-07-reconciled-simulator-alteration-slice.md). **Slice 3 — scale-ring navigation (simulator) ✅ Done.** Merged to `main` (session 0011). Adds the **endless rotary-encoder** control — *relative* (vs the absolute 0–4 experience knobs) — that walks a **closed ring** of neutral "scales of nature" clips, with placeholder AI zoom/warp **transitions** between adjacent scales and a small→large **wrap** (the infinite-zoom payoff, scales design §3). The navigation math is canonical Python (`player/ring.py`: `ScaleRing` + `advance_ring`, one transition clip per edge, played forward inward / reversed outward); the sim exposes `GET /api/ring` + `POST /api/ring/advance` and the browser only plays the returned transition(s) then settles on the target scale, keeping the live knob alteration on top. Two cheap **true-PD** scales were added so the ring is demonstrable — `cosmos` (NASA/Hubble) and `abyss` (NOAA Ocean Exploration), provenance recorded in the manifest, bytes generated as labelled placeholders by `simulator/setup_scales_media.py`. The expensive real multi-strength flow-stabilized Right re-bake is **deferred** (new scales carry a raw base only) until the ring is liked. Plan: [`2026-06-07-scale-ring-navigation.md`](./superpowers/plans/2026-06-07-scale-ring-navigation.md). Session 0012 added the design §3 **fast-spin** behavior on top: past an opt-in speed threshold (`DEFAULT_FAST_SPIN_THRESHOLD = 3`, `abs(delta)` as the spin-speed proxy) a quick spin collapses to a single **blended pass** instead of chaining every transition, so fast navigation stays responsive on placeholder media. **Remaining slices (not started):** - **Runtime renderer** — drive the single panoramic projector via mpv/ffmpeg; GPU shaders for the luma-keyed mood grade + analytical-overlay compositing; realize crossfade/fade timing and the 5–15 min loop. (Open decision: player stack — mpv via IPC vs. ffmpeg vs. custom.) - **Serial input** — real USB-serial reader on the Pi + the **3⇄4 framing contract** with the firmware (the keyboard/serial stand-in already works via a `Controls` stream). - **White-noise generation** — runtime white/pink noise for that content position. - **Offline v2v variant pipeline** — author the pre-baked Right restyle variants via the local flow-stabilized SD pipeline (now ~free, not a paid API — see the scales-library/right-axis design §1/§4); a real multi-strength flow-stabilized re-bake per base clip + the multilingual label/string tables + TTS. The **scale-ring transitions** (forward/reverse per-edge morphs) are part of this offline pipeline too. - **Vertical slice done (session 0012):** the Right-variant half is productionized and run for the **forest** scale — `simulator/bake_right_variants.py` (the POC flow-restyle algorithm, repo-resident) bakes real strengths 1–4 on a by-eye keyframe-strength ramp; forest now carries real Right variants in the sim, so the dreamlike-axis look is judgeable on real footage. **Still deferred:** real Right variants for cosmos/abyss (need their real strict-PD bases sourced first) and the **real i2v ring transitions** (need a 2nd real base + a generative-video model not in the POC). Plan: [`2026-06-08-v2v-vertical-slice-forest-right-variants.md`](./superpowers/plans/2026-06-08-v2v-vertical-slice-forest-right-variants.md). - **Catalog model changes** — audio *source* + "neutral base" vs "altered variant" flag (sub-project 2 territory, design §13). **Depends on:** sub-project 1 (`Coordinate`); a neutral base library from sub-project 2 to be meaningful, but developable against a hand-authored library and a **keyboard/serial stand-in** before firmware exists. **Done when:** given a base library and a stream of control values, it plays the correct altered segment, loops, crossfades, and goes dark on `Off` — the decision logic is covered (slice 1, serial mocked); the renderer realizes it. **Hardware:** Raspberry Pi 5 (holds the drive + base library + variants). **Open decisions:** player stack (mpv via IPC vs. ffmpeg vs. custom); whether the player hard-restricts to `approved` records (the `approved_only` flag is plumbed but inert until backed by a real catalog); knob→strength calibration (the §3 vs §4.2/§5 reconciliation — see the slice-1 plan / session 0006 transcript). --- ## 4. Arduino Firmware (control panel) ◻ **Goal:** the physical panel — turn knob/selector positions into serial messages. **Delivers (`firmware/`):** - Read four knobs (`left`, `right`, `dark`, `light`) and the 4-way content-mode selector. - Quantize/debounce analog inputs to the `0..4` integer scale. - Send the five values to the Pi over USB serial in a defined framing. **Depends on:** only the **serial protocol contract** shared with sub-project 3 — agree that first, then 3 and 4 proceed in parallel. **Done when:** turning a knob produces the expected serial message (verifiable on a serial monitor) and the Pi player reacts. **Hardware:** Arduino + pots/encoders + a 4-position selector. **Open decisions:** pots vs. detented encoders; exact serial framing. --- ## 5. ~~Procedural Side Walls~~ ❌ Dropped (2026-06-04) **Superseded.** The move to a single panoramic projector showing the real selected video across the three walls (design spec §6/§7) removes the procedural ambient renderer entirely — the mood axis is felt through the chosen content itself. With it goes the only consumer of `dominant_color`, which becomes optional/opt-in in the ingest tooling (sub-project 2 spec §5.2). > *Original goal (for the record): render a slow color wash on the three > non-primary walls — hue from the playing piece's `dominant_color`, brightness > from the dark/light knobs — to make the mood axis felt in peripheral vision.* --- ## Cross-cutting decisions to settle as we go Carried from the design spec's open-questions list — none block sub-project 2: - **Serial protocol framing** between Arduino and Pi (needed before 3 ⇄ 4). - **Player stack** on the Pi (mpv / ffmpeg / custom) and how the single panoramic output is driven. - **`approved`-only enforcement** in the player. - ~~**Side walls** on the primary Pi vs. a second Pi~~ — resolved: side walls dropped. - ~~**Media storage layout** on the drive~~ — settled by the sub-project-2 spec §6.3: media under a `media-root`, `file_path` stored relative to it; the player joins it with its drive mount.