human-experience-filter-art/docs/ROADMAP.md

Human Experience Filter — Roadmap

The installation is built as five independent sub-projects, each producing working, testable software on its own. Each gets its own spec → plan → implement cycle. This file is the index and dependency map; it is the source of truth for "what's left."

Design reference: specs/2026-06-04-human-experience-filter-design.md.

---

Status at a glance

#	Sub-project	Status	Unblocks
1	Catalog & Selection Core	✅ Done	everything
2	Ingest & Tagging / Review tools	✅ Done	a real library
3	Player Runtime (Pi)	⏳ In progress	the room runs
4	Arduino Firmware (control panel)	◻ Not started	real knobs
5	Procedural Side Walls	❌ Dropped	— (superseded)

Dependency order: 1 → 2, and 1 → 3. Sub-projects 3 and 4 share a serial protocol contract and can then proceed in parallel. Sub-project 5 was dropped (2026-06-04): the move to a single panoramic projector showing real video across the three walls removes the procedural side walls — see the design spec §6/§7.

        ┌─────────────────────────┐
        │ 1. Catalog & Selection  │ ✅
        └───────────┬─────────────┘
                    │
        ┌───────────┴───────────┐
        ▼                       ▼
┌───────────────┐      ┌─────────────────────┐
│ 2. Ingest /   │      │ 3. Player Runtime   │
│    Review     │─────▶│    (Pi) → 1 pano     │
└───────────────┘ feeds│    projector         │
   (fills catalog)      └──────────┬──────────┘
                          serial   ▼
                       ┌────────────────┐
                       │ 4. Firmware    │
                       └────────────────┘

(5. Procedural Side Walls — dropped 2026-06-04, single pano projector)

---

1. Catalog & Selection Core ✅

Done — merged to main, 37 tests. The dependency root every other sub-project imports.

• hef/catalog.py — Record model, validation, JSONL load/save/append.
• hef/selection.py — weighted-Euclidean distance, mode filtering with A+V fallback, nearest-match select().

Plan: 2026-06-04-catalog-and-selection-core.md.

---

2. Ingest & Tagging / Review tools ✅

Done — merged to main via PR #1 (session 0002, 2026-06-04); 111 tests pass (2 opt-in real-ffprobe tests skip when the binaries are absent). The tools/ package turns the manual catalog into the assisted draft-then-review flow from the spec, and can populate the library from public-domain sources.

Delivers (tools/):

• Ingest — per-archive fetchers (Internet Archive / Prelinger, Musopen, LibriVox, NASA, Free Music Archive, Freesound) that download a candidate and write a catalog record.
• Mechanical tagging — auto-fill mode (probe streams with ffprobe), license / attribution / source_* (from origin), duration_s, resolution, and dominant_color (computed from the video).
• Coordinate drafting — propose left/right/dark/light with a one-line rationale, written as review_status: proposed.
• Review CLI — walk unreviewed records one at a time (show proposed coordinates + rationale + a preview frame), accept or correct, flip to approved and stamp reviewed_at.

Spec: 2026-06-04-ingest-tagging-review-tools.md.

Depends on: sub-project 1 (hef.catalog). Done when: ✅ you can run ingest against a source, get proposed records, review them to approved, and load_catalog/validate_catalog validate the result — covered by tests on the mechanical-tagging and review state transitions. External deps: ffmpeg/ffprobe only (no image lib — dominant_color, when requested, is ffmpeg-only and opt-in), per-archive download access. Decisions settled: first-ship archives = LibriVox, NASA, Internet Archive (Musopen/FMA/Freesound deferred stubs); media under a --media-root with file_path stored relative to it (spec §6.3); only additive hef.catalog change was validate_catalog + index_by_id.

---

3. Player Runtime (Pi) ⏳ (in progress)

Goal: the thing that makes the room run — read the controls, alter the neutral base footage toward the knob state, and play it across the single panoramic projector. Per the machine-altered-perception revision the knobs no longer select a pre-tagged clip; they drive an alteration engine (design §4/§5) over a small neutral base library.

Slice 1 — alteration engine + player core (pure logic) ✅ Done. Merged to main (session 0006); 56 tests. The new player/ package, with all I/O (video/audio/serial) behind injected interfaces:

• player/controls.py — Controls, the full panel state (7-way content dial + 4 experience knobs + volume + brightness); the serial-contract data shape shared with sub-project 4.
• player/content.py — resolve_content, the §6 7-way dial → (audio source, video on/off) table.
• player/alteration.py — plan_alteration, knob vector → RenderPlan (mood ColorGrade with center=identity §5; Left AnalyticalOverlay; Right Restyle), encoding the §4.2 layering rule (substrate vs overlay; Left & Right stack).
• player/state.py — Player, the state machine that diffs successive controls into transitions (FADE_TO_BLACK/FADE_FROM_BLACK on video on/off, CROSSFADE on clip/restyle-variant swap, LIVE_UPDATE for the continuous grade/overlay/level ops per §4.3, NONE when unchanged).

Plan: 2026-06-05-player-alteration-core.md.

Slice 2 — simulator-first alteration preview ✅ Done. Merged to main (session 0009). Wires the alteration engine into the web simulator so the look is tunable by eye before any hardware, and reconciles the unmerged session-0007 design with the merged session-0008 design (the Left-HUD conflict): Left is a runtime overlay (authored annotation track + per-language string tables), Right is a discrete pre-baked flow-stabilized variant, Dark/Light a live grade — superseding 0007's baked-HUD 5×5 grid. Engine: a parameterized Calibration (tuned by eye, baked into DEFAULT_CALIBRATION), Restyle.variant (discrete) replacing the continuous blend, and AnalyticalOverlay.level. Sim: /api/alteration + /api/clips over simulator/clips.py, an alteration-preview UI, and one neutral clip with a real flow-stabilized Right variant from the POC. Design: 2026-06-07-reconciled-simulator-alteration-slice-design.md; plan: 2026-06-07-reconciled-simulator-alteration-slice.md.

Slice 3 — scale-ring navigation (simulator) ✅ Done. Merged to main (session 0011). Adds the endless rotary-encoder control — relative (vs the absolute 0–4 experience knobs) — that walks a closed ring of neutral "scales of nature" clips, with placeholder AI zoom/warp transitions between adjacent scales and a small→large wrap (the infinite-zoom payoff, scales design §3). The navigation math is canonical Python (player/ring.py: ScaleRing + advance_ring, one transition clip per edge, played forward inward / reversed outward); the sim exposes GET /api/ring + POST /api/ring/advance and the browser only plays the returned transition(s) then settles on the target scale, keeping the live knob alteration on top. Two cheap true-PD scales were added so the ring is demonstrable — cosmos (NASA/Hubble) and abyss (NOAA Ocean Exploration), provenance recorded in the manifest, bytes generated as labelled placeholders by simulator/setup_scales_media.py. The expensive real multi-strength flow-stabilized Right re-bake is deferred (new scales carry a raw base only) until the ring is liked. Plan: 2026-06-07-scale-ring-navigation.md. Session 0012 added the design §3 fast-spin behavior on top: past an opt-in speed threshold (DEFAULT_FAST_SPIN_THRESHOLD = 3, abs(delta) as the spin-speed proxy) a quick spin collapses to a single blended pass instead of chaining every transition, so fast navigation stays responsive on placeholder media.

Remaining slices (not started):

• Runtime renderer — drive the single panoramic projector via mpv/ffmpeg; GPU shaders for the luma-keyed mood grade + analytical-overlay compositing; realize crossfade/fade timing and the 5–15 min loop. (Open decision: player stack — mpv via IPC vs. ffmpeg vs. custom.)
• Serial input — real USB-serial reader on the Pi + the 3⇄4 framing contract with the firmware (the keyboard/serial stand-in already works via a Controls stream).
• White-noise generation — runtime white/pink noise for that content position.
• Offline v2v variant pipeline — author the pre-baked Right restyle variants via the local flow-stabilized SD pipeline (now ~free, not a paid API — see the scales-library/right-axis design §1/§4); a real multi-strength flow-stabilized re-bake per base clip + the multilingual label/string tables + TTS. The scale-ring transitions (forward/reverse per-edge morphs) are part of this offline pipeline too.
  • Vertical slice done (session 0012): the Right-variant half is productionized and run for the forest scale — simulator/bake_right_variants.py (the POC flow-restyle algorithm, repo-resident) bakes real strengths 1–4 on a by-eye keyframe-strength ramp; forest now carries real Right variants in the sim, so the dreamlike-axis look is judgeable on real footage. Still deferred: real Right variants for cosmos/abyss (need their real strict-PD bases sourced first) and the real i2v ring transitions (need a 2nd real base + a generative-video model not in the POC). Plan: 2026-06-08-v2v-vertical-slice-forest-right-variants.md.
• Catalog model changes — audio source + "neutral base" vs "altered variant" flag (sub-project 2 territory, design §13).

Depends on: sub-project 1 (Coordinate); a neutral base library from sub-project 2 to be meaningful, but developable against a hand-authored library and a keyboard/serial stand-in before firmware exists. Done when: given a base library and a stream of control values, it plays the correct altered segment, loops, crossfades, and goes dark on Off — the decision logic is covered (slice 1, serial mocked); the renderer realizes it. Hardware: Raspberry Pi 5 (holds the drive + base library + variants). Open decisions: player stack (mpv via IPC vs. ffmpeg vs. custom); whether the player hard-restricts to approved records (the approved_only flag is plumbed but inert until backed by a real catalog); knob→strength calibration (the §3 vs §4.2/§5 reconciliation — see the slice-1 plan / session 0006 transcript).

---

4. Arduino Firmware (control panel) ◻

Goal: the physical panel — turn knob/selector positions into serial messages.

Delivers (firmware/):

• Read four knobs (left, right, dark, light) and the 4-way content-mode selector.
• Quantize/debounce analog inputs to the 0..4 integer scale.
• Send the five values to the Pi over USB serial in a defined framing.

Depends on: only the serial protocol contract shared with sub-project 3 — agree that first, then 3 and 4 proceed in parallel. Done when: turning a knob produces the expected serial message (verifiable on a serial monitor) and the Pi player reacts. Hardware: Arduino + pots/encoders + a 4-position selector. Open decisions: pots vs. detented encoders; exact serial framing.

---

5. Procedural Side Walls ❌ Dropped (2026-06-04)

Superseded. The move to a single panoramic projector showing the real selected video across the three walls (design spec §6/§7) removes the procedural ambient renderer entirely — the mood axis is felt through the chosen content itself. With it goes the only consumer of dominant_color, which becomes optional/opt-in in the ingest tooling (sub-project 2 spec §5.2).

Original goal (for the record): render a slow color wash on the three non-primary walls — hue from the playing piece's dominant_color, brightness from the dark/light knobs — to make the mood axis felt in peripheral vision.

---

Cross-cutting decisions to settle as we go

Carried from the design spec's open-questions list — none block sub-project 2:

• Serial protocol framing between Arduino and Pi (needed before 3 ⇄ 4).
• Player stack on the Pi (mpv / ffmpeg / custom) and how the single panoramic output is driven.
• approved-only enforcement in the player.
• Side walls on the primary Pi vs. a second Pi — resolved: side walls dropped.
• Media storage layout on the drive — settled by the sub-project-2 spec §6.3: media under a media-root, file_path stored relative to it; the player joins it with its drive mount.