diff --git a/docs/superpowers/specs/2026-06-24-content-pipeline-design.md b/docs/superpowers/specs/2026-06-24-content-pipeline-design.md new file mode 100644 index 0000000..a5b2107 --- /dev/null +++ b/docs/superpowers/specs/2026-06-24-content-pipeline-design.md @@ -0,0 +1,248 @@ +# HEF — Content Pipeline (source → process → label → manifest) + +**Date:** 2026-06-24 +**Status:** Proposed design (pre-implementation) — session 0014 +**Repo:** `human-experience-filter-art` +**Anchor:** `docs/ROADMAP.md` sub-project 3 (Player Runtime) content needs; the +spiritual successor to sub-project 2 (Ingest & Tagging tools), retargeted from the +retired coordinate-catalog model to the manifest content model. +**Refines:** [`2026-06-07-scales-library-and-right-axis-pipeline-design.md`](./2026-06-07-scales-library-and-right-axis-pipeline-design.md) +§2 (scales library), §2.1 (strict-PD sourcing), §6 (open sourcing questions). +**Builds on:** [`2026-06-22-right-axis-deterministic-dream-design.md`](./2026-06-22-right-axis-deterministic-dream-design.md) +(the real-time dream — **no per-clip ML bake**) and +[`2026-06-22-affect-channel-hud-design.md`](./2026-06-22-affect-channel-hud-design.md) +(scene-anchored affect words). + +--- + +## 1. Why this exists + +The installation has an alteration engine, a scale ring, a Left analytical HUD, an +affect channel, and a real-time Right "dream" — but it has **no real content**. The +three ring scales today are: `cosmos` and `abyss` carrying *placeholder bytes* (PD +provenance recorded, but the media is synthetic), and `forest` carrying a *POC +sample* (a trimmed Yosemite clip licensed "look-tuning only; not shipped content"). +There is no repeatable way to turn a public-domain source clip into a shippable, +seamless-looping, labelled neutral base. + +This design defines that pipeline: **source → frame → seamless-loop → transcode → +label/author → manifest entry.** One big anchor fact reshapes it versus the old +plan — the Right dream is now a **real-time deterministic shader** (session 0013), +so there is **no per-clip ML restyle bake** anymore. The pipeline's offline cost is +therefore **transcode + human authoring**, not generative-video baking. The one +deliberate exception is the motion-tracking label pass chosen this session (§3), +which re-introduces a bounded offline compute step — by choice, for realism. + +## 2. Decisions settled this session + +| # | Decision | Choice | Consequence | +|---|----------|--------|-------------| +| 1 | Scale set | **Lean 5** — cosmos · orbit · forest/land · reef · abyss (microscopic dropped) | Ring wraps **abyss → cosmos**; replaces the POC forest + both placeholder bases with real strict-PD footage | +| 2 | Seamless loop | **Crossfade loop** (tail→head overlap) | Any source clip becomes an indefinite seamless dwell loop; a transcode-time step | +| 3 | Label placement | **Motion-tracked, hybrid** | ML proposes tracks where it can; hand-seed + classical tracker elsewhere. Re-introduces an offline pass + an authoring step. Semantics always hand-authored | +| 4 | Format | **Master + 1080p proxy** | High-quality master retained per clip; a 1080p H.264 proxy drives the sim; Pi chooses later | + +Microscopic was dropped because PD microscopy *video* (vs stills) is the thinnest +pool and hardest to source well; it can be added later by re-running the pipeline. + +## 3. The pipeline — six stages + +Each stage is a deterministic unit with one job (handbook §4.2: deterministic tools, +not prose an agent re-derives). Stages 1–4 and 6 are mechanical; stage 5 is the one +human-in-the-loop step. + +``` + (1) source ──▶ (2) frame ──▶ (3) loop ──▶ (4) transcode ──▶ (6) manifest + provenance trim/crop crossfade master + proxy entry + & scale seam │ ▲ + └──▶ (5) label/author (hybrid track) +``` + +### 3.1 Stage 1 — source & provenance +Acquire the strict-PD source clip and record license + source URL + capture notes. +Reuse sub-project-2 fetchers where they map (a NASA fetcher already exists); add +NOAA Ocean Exploration / NPS / USGS fetchers (or accept a manual download + a +provenance record). The "no explicit license → assume PD, verify" trap from the +scales design §2.1 applies: **"free stock" (Pexels/Pixabay) is NOT public domain.** +Output: the raw source file + a provenance record (license, source, url, fetched_at). + +### 3.2 Stage 2 — frame (trim / crop / scale) +Pick the in/out points; crop to the target aspect; scale toward the format target +(§6). The panoramic projector aspect is still unmeasured (open question), so v1 +frames to a standard wide 16:9 for the sim; the master retains maximum available +width so a later pano crop/extension has pixels to work with. Output: a framed, +silent (audio stripped — bases are video-only; audio is the separate content-dial +channel) intermediate. + +### 3.3 Stage 3 — seamless crossfade loop +Make the clip loop indefinitely with no visible seam by overlapping the tail into +the head with a short crossfade (~0.5–1.0 s). The final looped clip is shorter than +the source by the overlap; during the overlap region motion is a blend (acceptable +for calm ambient nature footage). This is the chosen loop strategy (clean-cut and +palindrome were rejected — seam risk and reversed motion respectively). Output: a +loop-clean clip whose first and last frames join seamlessly. + +### 3.4 Stage 4 — transcode (master + proxy) +Emit two encodes per clip (§6 for exact params): a **master** (high quality, up to +4K, retained as the archival/Pi-candidate source) and a **1080p H.264 proxy** (what +the sim plays today, matching current `base.mp4` behaviour). Both video-only, +`yuv420p`, even dimensions, `+faststart`. + +### 3.5 Stage 5 — label & author (the hybrid motion-track pass) +The only human-in-the-loop stage, and the only one that adds offline ML compute. + +- **Semantics are always hand-authored.** A generic detector cannot produce the + installation's semantic keys — "spiral galaxy", "siphonophore", `z ≈ 0.04`, + `~2.1 m³/s` are out-of-distribution. The author owns the label **keys**, + **strings**, and `min_level`; tracking only propagates **box geometry**. +- **Hybrid geometry (decision 3).** Where a model can lock onto a subject (a fish + on the reef, an animal in the forest), an ML detect+track pass (YOLO+ByteTrack / + SAM2-class, lazy-imported like the parked baker imported torch) proposes a track + the author maps to a key. Where it can't (cosmos, abyss, microscopic-type + content), the author **hand-seeds** a box on a keyframe and a **classical tracker** + (OpenCV CSRT / optical-flow) propagates it; the author fixes drift. Both paths + emit the same artifact. +- **Channel rules.** `detected.*` object labels **track**; `measure.*` readouts and + `feel.*` affect words stay **static / scene-anchored** (a depth readout or "awe" + is not pinned to a moving pixel). This matches the affect-channel design (affect + is scene-level) and keeps measurement chips legible. +- Output: an **annotation track** per object label (§4) plus the static measurement + and affect placements, all keyed and stringed. + +### 3.6 Stage 6 — manifest assembly +Write the clip entry (base/proxy/master paths, license, source, annotations with +tracks, affect, strings) into `simulator/sample_media/manifest.json`, and register +the scale in the `ring` section with its transition edges (§5). Idempotent: stage 6 +owns the manifest entry so re-running earlier stages never clobbers authored labels +(mirrors how `setup_sample_media.py` was made to never clobber a real bake). + +## 4. Data model — annotation tracks + +Today an annotation is a single static box: `{"key", "box":[x,y,w,h], "min_level"}`. +Add an **optional keyframed track**; static labels are unchanged (backward +compatible): + +```json +{ + "key": "detected.fish", + "min_level": 1, + "track": [ + {"t": 0.0, "box": [0.30, 0.40, 0.10, 0.08]}, + {"t": 0.5, "box": [0.52, 0.38, 0.10, 0.08]}, + {"t": 1.0, "box": [0.30, 0.40, 0.10, 0.08]} + ] +} +``` + +- `t` is **normalized 0→1 over the looped clip** — resolution- and fps-independent, + so the same track is correct for both master and proxy and survives a re-transcode. + (A track that loops cleanly has matching boxes at `t=0` and `t=1`.) +- **Runtime:** if `track` is present, the client interpolates the box at the current + loop-normalized playback time (linear between keyframes); otherwise it uses the + static `box`. This lives **client-side**, consistent with the existing split + (alteration math in Python; label selection/placement in the client). `Clip` gains + the track only as data inside `annotations`; `player/alteration.py` is **unchanged** + (labels are not part of `RenderPlan`). +- `measure.*` keep a single static `box`; `feel.*` keep a single static `at:[x,y]`. + +## 5. The 5 scales, sources, and ring + +| Scale | Strict-PD source | Notes | +|-------|------------------|-------| +| `cosmos` | NASA / Hubble / JWST | 🟢 abundant true PD (17 U.S.C. §105); replaces placeholder | +| `orbit` | NASA / ISS (Earth from orbit) | 🟢 true PD; **new** scale | +| `forest` | NPS / USGS (US land/wildlife) | 🟢 true PD (US locations); replaces the non-shippable POC sample | +| `reef` | NOAA Ocean Exploration (shallow) | 🟢 true PD worldwide; **new** scale | +| `abyss` | NOAA Ocean Exploration (deep sea) | 🟢 true PD worldwide; replaces placeholder | + +**Ring order (large → small, zooms inward; wraps small → large):** +`cosmos → orbit → forest → reef → abyss → (wrap) cosmos`. Five scales → five ring +edges. The wrap seam is now **abyss → cosmos** (deep sea back out to the cosmos) — +the infinite-zoom payoff, minus the microscopic step. + +**Transitions** between the new adjacent edges remain the **deferred i2v generative +work** (scales design §3 / §6) — they need a generative-video model and two real +adjacent bases, heavier than anything in this pipeline. Until then new edges carry +**placeholder transitions** (the `setup_scales_media.py` ffmpeg `xfade=zoomin` +generator already produces these), exactly as cosmos/forest/abyss do today. This +pipeline's job is the **bases + labels**; transitions are tracked but out of scope +here (§8). + +## 6. Format target + +| | Master | Proxy (sim) | +|---|--------|-------------| +| Resolution | source, capped ~3840 wide | 1920×1080 (fit wide) | +| Codec | H.264 High | H.264 High | +| Quality | CRF ~18 | CRF ~20 | +| FPS | preserve source (cap 30) | 30 | +| Pixel fmt | `yuv420p`, even dims | `yuv420p`, even dims | +| Audio | none (stripped) | none (stripped) | +| Flags | `+faststart` | `+faststart` | + +The sim plays the **proxy** as today's `base_file`. The **master** is retained per +clip for the eventual Pi/pano encode once the panoramic resolution is measured. +Browser HEVC support is too spotty to make HEVC the sim path, which is why the proxy +stays H.264 (the master can be re-encoded to HEVC for the Pi later without +re-sourcing). + +## 7. Tooling & where it lives + +- **New `tools/pipeline/` package** — deterministic ffmpeg-backed verbs for stages + 1–4 and 6 (`fetch`/provenance, `frame`, `loop`, `transcode`, `manifest`), reusing + sub-project-2 fetchers where valid. This **supersedes** the coordinate-era + ingest *drafting* (the `left/right/dark/light` Record model retired with the + machine-altered-perception pivot); the pipeline targets the **manifest**, not + catalog `Record`s. +- **Stage 5 tracking** — `tools/pipeline/track.py`: classical OpenCV tracker + (always available) + an optional lazy-imported ML detect+track path. Emits + candidate keyframed tracks for the author to accept/correct. +- **Authoring surface** — extend the **simulator** with an **author mode** (it + already renders the exact HUD overlay and is the by-eye judge): scrub the clip, + draw/seed/adjust boxes, run the tracker, assign keys/`min_level`/strings, place + affect anchors, write the manifest entry. Reusing the preview as the editor keeps + "what you author is what plays." **This is the largest new piece** — see the + delivery order (§8), which lets v1 prove the schema before the full editor exists. + +## 8. Delivery / rollout order + +Build order within this Feature (Solution-Design rollout strategy — not a task +list): + +1. **Increment 1 — real footage, no ML.** Stages 1–4 + 6 + the §4 track schema + + client interpolation. Source the 5 strict-PD bases, crossfade-loop them, emit + master+proxy, wire them into the ring. Author a *couple* of tracks **by hand as + JSON** to prove the track schema and the runtime interpolation. Outcome: the + whole ring is **real, seamless-looping footage** judgeable by eye — the + highest-value, lowest-risk step, and it stands alone. +2. **Increment 2 — the hybrid track pass + author mode.** `track.py` (classical + + optional ML) and the simulator author mode; label all five scales properly. + +**Deferred (tracked, not built here):** the real **i2v ring transitions** for the +new edges (need a generative-video model + adjacent real bases); the Pi renderer + +serial/firmware (per the simulator-first directive); the panoramic-resolution +master encode (waits on a measured pano spec). + +> Scope note: if Increment 2's author mode grows large enough to be its own +> increment in practice, split it at execution time. The design is one coherent +> pipeline; the increments above are its sensible build order, not separate +> Features. + +## 9. Open questions (for the plan, not blockers) + +- **Panoramic resolution/aspect** — unmeasured (scales design §6). Drives the master + crop/extension strategy; v1 frames 16:9 for the sim and retains max width. +- **ML model choice** for the hybrid track path (YOLO+ByteTrack vs SAM2-class) and + whether it earns its place given how few scales it can usefully fire on. +- **Classical tracker drift** on long/fast clips — keyframe interval, re-seed cadence. +- **Source fetchers** — which of NOAA / NPS / USGS get real fetchers vs manual + download + provenance record for v1. +- **Crossfade overlap length** per scale (slow cosmos vs busier reef) — by eye. + +## 10. Out of scope (YAGNI) + +- Generative i2v **ring transitions** for the new edges (deferred; placeholders hold). +- **Audio** sourcing/alteration (separate content-dial channel; already deferred). +- The **Pi/pano** encode + renderer and serial/firmware (simulator-first). +- Reviving the retired coordinate-catalog `Record` drafting. +- Per-frame **dense** tracks (sparse keyframed tracks interpolated at runtime instead).