benstull/human-experience-filter-art
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(spec): content pipeline design — source → process → label → manifest (session 0014)

Browse Source 
Brainstormed + scoped the content pipeline that turns strict-PD source clips into
shippable, seamless-looping, labelled neutral bases for the scale ring.

Decisions settled this session:
- Scale set: lean 5 (cosmos · orbit · forest/land · reef · abyss; microscopic dropped)
- Loop: crossfade tail→head seam
- Labels: motion-tracked, hybrid (ML proposes where it can, hand-seed + classical
  tracker elsewhere); label semantics always hand-authored
- Format: master + 1080p H.264 proxy

Anchor fact: the Right dream is real-time (no per-clip ML bake, session 0013), so
the pipeline is transcode + authoring — the one offline ML step is the chosen
motion-track pass. Six stages + an optional keyframed annotation-track schema
(backward compatible) + a 2-increment rollout (real footage no-ML first, then the
hybrid track pass + simulator author mode). i2v ring transitions + Pi/pano deferred.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ben Stull
2026-06-24 07:12:52 -07:00
parent cfd8317965
commit e210de6f84
1 changed files with 248 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/specs/2026-06-24-content-pipeline-design.md
+248
View File
@@ -0,0 +1,248 @@
# HEF — Content Pipeline (source → process → label → manifest)
**Date:** 2026-06-24
**Status:** Proposed design (pre-implementation) — session 0014
**Repo:** `human-experience-filter-art`
**Anchor:** `docs/ROADMAP.md` sub-project 3 (Player Runtime) content needs; the
spiritual successor to sub-project 2 (Ingest & Tagging tools), retargeted from the
retired coordinate-catalog model to the manifest content model.
**Refines:** [`2026-06-07-scales-library-and-right-axis-pipeline-design.md`](./2026-06-07-scales-library-and-right-axis-pipeline-design.md)
§2 (scales library), §2.1 (strict-PD sourcing), §6 (open sourcing questions).
**Builds on:** [`2026-06-22-right-axis-deterministic-dream-design.md`](./2026-06-22-right-axis-deterministic-dream-design.md)
(the real-time dream — **no per-clip ML bake**) and
[`2026-06-22-affect-channel-hud-design.md`](./2026-06-22-affect-channel-hud-design.md)
(scene-anchored affect words).
---
## 1. Why this exists
The installation has an alteration engine, a scale ring, a Left analytical HUD, an
affect channel, and a real-time Right "dream" — but it has **no real content**. The
three ring scales today are: `cosmos` and `abyss` carrying *placeholder bytes* (PD
provenance recorded, but the media is synthetic), and `forest` carrying a *POC
sample* (a trimmed Yosemite clip licensed "look-tuning only; not shipped content").
There is no repeatable way to turn a public-domain source clip into a shippable,
seamless-looping, labelled neutral base.
This design defines that pipeline: **source → frame → seamless-loop → transcode →
label/author → manifest entry.** One big anchor fact reshapes it versus the old
plan — the Right dream is now a **real-time deterministic shader** (session 0013),
so there is **no per-clip ML restyle bake** anymore. The pipeline's offline cost is
therefore **transcode + human authoring**, not generative-video baking. The one
deliberate exception is the motion-tracking label pass chosen this session (§3),
which re-introduces a bounded offline compute step — by choice, for realism.
## 2. Decisions settled this session
| # | Decision | Choice | Consequence |
|---|----------|--------|-------------|
| 1 | Scale set | **Lean 5** — cosmos · orbit · forest/land · reef · abyss (microscopic dropped) | Ring wraps **abyss → cosmos**; replaces the POC forest + both placeholder bases with real strict-PD footage |
| 2 | Seamless loop | **Crossfade loop** (tail→head overlap) | Any source clip becomes an indefinite seamless dwell loop; a transcode-time step |
| 3 | Label placement | **Motion-tracked, hybrid** | ML proposes tracks where it can; hand-seed + classical tracker elsewhere. Re-introduces an offline pass + an authoring step. Semantics always hand-authored |
| 4 | Format | **Master + 1080p proxy** | High-quality master retained per clip; a 1080p H.264 proxy drives the sim; Pi chooses later |
Microscopic was dropped because PD microscopy *video* (vs stills) is the thinnest
pool and hardest to source well; it can be added later by re-running the pipeline.
## 3. The pipeline — six stages
Each stage is a deterministic unit with one job (handbook §4.2: deterministic tools,
not prose an agent re-derives). Stages 1–4 and 6 are mechanical; stage 5 is the one
human-in-the-loop step.
```
(1) source ──▶ (2) frame ──▶ (3) loop ──▶ (4) transcode ──▶ (6) manifest
provenance trim/crop crossfade master + proxy entry
& scale seam │ ▲
└──▶ (5) label/author (hybrid track)
```
### 3.1 Stage 1 — source & provenance
Acquire the strict-PD source clip and record license + source URL + capture notes.
Reuse sub-project-2 fetchers where they map (a NASA fetcher already exists); add
NOAA Ocean Exploration / NPS / USGS fetchers (or accept a manual download + a
provenance record). The "no explicit license → assume PD, verify" trap from the
scales design §2.1 applies: **"free stock" (Pexels/Pixabay) is NOT public domain.**
Output: the raw source file + a provenance record (license, source, url, fetched_at).
### 3.2 Stage 2 — frame (trim / crop / scale)
Pick the in/out points; crop to the target aspect; scale toward the format target
(§6). The panoramic projector aspect is still unmeasured (open question), so v1
frames to a standard wide 16:9 for the sim; the master retains maximum available
width so a later pano crop/extension has pixels to work with. Output: a framed,
silent (audio stripped — bases are video-only; audio is the separate content-dial
channel) intermediate.
### 3.3 Stage 3 — seamless crossfade loop
Make the clip loop indefinitely with no visible seam by overlapping the tail into
the head with a short crossfade (~0.5–1.0 s). The final looped clip is shorter than
the source by the overlap; during the overlap region motion is a blend (acceptable
for calm ambient nature footage). This is the chosen loop strategy (clean-cut and
palindrome were rejected — seam risk and reversed motion respectively). Output: a
loop-clean clip whose first and last frames join seamlessly.
### 3.4 Stage 4 — transcode (master + proxy)
Emit two encodes per clip (§6 for exact params): a **master** (high quality, up to
4K, retained as the archival/Pi-candidate source) and a **1080p H.264 proxy** (what
the sim plays today, matching current `base.mp4` behaviour). Both video-only,
`yuv420p`, even dimensions, `+faststart`.
### 3.5 Stage 5 — label & author (the hybrid motion-track pass)
The only human-in-the-loop stage, and the only one that adds offline ML compute.
- **Semantics are always hand-authored.** A generic detector cannot produce the
installation's semantic keys — "spiral galaxy", "siphonophore", `z ≈ 0.04`,
`~2.1 m³/s` are out-of-distribution. The author owns the label **keys**,
**strings**, and `min_level`; tracking only propagates **box geometry**.
- **Hybrid geometry (decision 3).** Where a model can lock onto a subject (a fish
on the reef, an animal in the forest), an ML detect+track pass (YOLO+ByteTrack /
SAM2-class, lazy-imported like the parked baker imported torch) proposes a track
the author maps to a key. Where it can't (cosmos, abyss, microscopic-type
content), the author **hand-seeds** a box on a keyframe and a **classical tracker**
(OpenCV CSRT / optical-flow) propagates it; the author fixes drift. Both paths
emit the same artifact.
- **Channel rules.** `detected.*` object labels **track**; `measure.*` readouts and
`feel.*` affect words stay **static / scene-anchored** (a depth readout or "awe"
is not pinned to a moving pixel). This matches the affect-channel design (affect
is scene-level) and keeps measurement chips legible.
- Output: an **annotation track** per object label (§4) plus the static measurement
and affect placements, all keyed and stringed.
### 3.6 Stage 6 — manifest assembly
Write the clip entry (base/proxy/master paths, license, source, annotations with
tracks, affect, strings) into `simulator/sample_media/manifest.json`, and register
the scale in the `ring` section with its transition edges (§5). Idempotent: stage 6
owns the manifest entry so re-running earlier stages never clobbers authored labels
(mirrors how `setup_sample_media.py` was made to never clobber a real bake).
## 4. Data model — annotation tracks
Today an annotation is a single static box: `{"key", "box":[x,y,w,h], "min_level"}`.
Add an **optional keyframed track**; static labels are unchanged (backward
compatible):
```json
{
"key": "detected.fish",
"min_level": 1,
"track": [
{"t": 0.0, "box": [0.30, 0.40, 0.10, 0.08]},
{"t": 0.5, "box": [0.52, 0.38, 0.10, 0.08]},
{"t": 1.0, "box": [0.30, 0.40, 0.10, 0.08]}
]
}
```
- `t` is **normalized 0→1 over the looped clip** — resolution- and fps-independent,
so the same track is correct for both master and proxy and survives a re-transcode.
(A track that loops cleanly has matching boxes at `t=0` and `t=1`.)
- **Runtime:** if `track` is present, the client interpolates the box at the current
loop-normalized playback time (linear between keyframes); otherwise it uses the
static `box`. This lives **client-side**, consistent with the existing split
(alteration math in Python; label selection/placement in the client). `Clip` gains
the track only as data inside `annotations`; `player/alteration.py` is **unchanged**
(labels are not part of `RenderPlan`).
- `measure.*` keep a single static `box`; `feel.*` keep a single static `at:[x,y]`.
## 5. The 5 scales, sources, and ring
| Scale | Strict-PD source | Notes |
|-------|------------------|-------|
| `cosmos` | NASA / Hubble / JWST | 🟢 abundant true PD (17 U.S.C. §105); replaces placeholder |
| `orbit` | NASA / ISS (Earth from orbit) | 🟢 true PD; **new** scale |
| `forest` | NPS / USGS (US land/wildlife) | 🟢 true PD (US locations); replaces the non-shippable POC sample |
| `reef` | NOAA Ocean Exploration (shallow) | 🟢 true PD worldwide; **new** scale |
| `abyss` | NOAA Ocean Exploration (deep sea) | 🟢 true PD worldwide; replaces placeholder |
**Ring order (large → small, zooms inward; wraps small → large):**
`cosmos → orbit → forest → reef → abyss → (wrap) cosmos`. Five scales → five ring
edges. The wrap seam is now **abyss → cosmos** (deep sea back out to the cosmos) —
the infinite-zoom payoff, minus the microscopic step.
**Transitions** between the new adjacent edges remain the **deferred i2v generative
work** (scales design §3 / §6) — they need a generative-video model and two real
adjacent bases, heavier than anything in this pipeline. Until then new edges carry
**placeholder transitions** (the `setup_scales_media.py` ffmpeg `xfade=zoomin`
generator already produces these), exactly as cosmos/forest/abyss do today. This
pipeline's job is the **bases + labels**; transitions are tracked but out of scope
here (§8).
## 6. Format target
| | Master | Proxy (sim) |
|---|--------|-------------|
| Resolution | source, capped ~3840 wide | 1920×1080 (fit wide) |
| Codec | H.264 High | H.264 High |
| Quality | CRF ~18 | CRF ~20 |
| FPS | preserve source (cap 30) | 30 |
| Pixel fmt | `yuv420p`, even dims | `yuv420p`, even dims |
| Audio | none (stripped) | none (stripped) |
| Flags | `+faststart` | `+faststart` |
The sim plays the **proxy** as today's `base_file`. The **master** is retained per
clip for the eventual Pi/pano encode once the panoramic resolution is measured.
Browser HEVC support is too spotty to make HEVC the sim path, which is why the proxy
stays H.264 (the master can be re-encoded to HEVC for the Pi later without
re-sourcing).
## 7. Tooling & where it lives
- **New `tools/pipeline/` package** — deterministic ffmpeg-backed verbs for stages
1–4 and 6 (`fetch`/provenance, `frame`, `loop`, `transcode`, `manifest`), reusing
sub-project-2 fetchers where valid. This **supersedes** the coordinate-era
ingest *drafting* (the `left/right/dark/light` Record model retired with the
machine-altered-perception pivot); the pipeline targets the **manifest**, not
catalog `Record`s.
- **Stage 5 tracking** — `tools/pipeline/track.py`: classical OpenCV tracker
(always available) + an optional lazy-imported ML detect+track path. Emits
candidate keyframed tracks for the author to accept/correct.
- **Authoring surface** — extend the **simulator** with an **author mode** (it
already renders the exact HUD overlay and is the by-eye judge): scrub the clip,
draw/seed/adjust boxes, run the tracker, assign keys/`min_level`/strings, place
affect anchors, write the manifest entry. Reusing the preview as the editor keeps
"what you author is what plays." **This is the largest new piece** — see the
delivery order (§8), which lets v1 prove the schema before the full editor exists.
## 8. Delivery / rollout order
Build order within this Feature (Solution-Design rollout strategy — not a task
list):
1. **Increment 1 — real footage, no ML.** Stages 1–4 + 6 + the §4 track schema +
client interpolation. Source the 5 strict-PD bases, crossfade-loop them, emit
master+proxy, wire them into the ring. Author a *couple* of tracks **by hand as
JSON** to prove the track schema and the runtime interpolation. Outcome: the
whole ring is **real, seamless-looping footage** judgeable by eye — the
highest-value, lowest-risk step, and it stands alone.
2. **Increment 2 — the hybrid track pass + author mode.** `track.py` (classical +
optional ML) and the simulator author mode; label all five scales properly.
**Deferred (tracked, not built here):** the real **i2v ring transitions** for the
new edges (need a generative-video model + adjacent real bases); the Pi renderer +
serial/firmware (per the simulator-first directive); the panoramic-resolution
master encode (waits on a measured pano spec).
> Scope note: if Increment 2's author mode grows large enough to be its own
> increment in practice, split it at execution time. The design is one coherent
> pipeline; the increments above are its sensible build order, not separate
> Features.
## 9. Open questions (for the plan, not blockers)
- **Panoramic resolution/aspect** — unmeasured (scales design §6). Drives the master
crop/extension strategy; v1 frames 16:9 for the sim and retains max width.
- **ML model choice** for the hybrid track path (YOLO+ByteTrack vs SAM2-class) and
whether it earns its place given how few scales it can usefully fire on.
- **Classical tracker drift** on long/fast clips — keyframe interval, re-seed cadence.
- **Source fetchers** — which of NOAA / NPS / USGS get real fetchers vs manual
download + provenance record for v1.
- **Crossfade overlap length** per scale (slow cosmos vs busier reef) — by eye.
## 10. Out of scope (YAGNI)
- Generative i2v **ring transitions** for the new edges (deferred; placeholders hold).
- **Audio** sourcing/alteration (separate content-dial channel; already deferred).
- The **Pi/pano** encode + renderer and serial/firmware (simulator-first).
- Reviving the retired coordinate-catalog `Record` drafting.
- Per-frame **dense** tracks (sparse keyframed tracks interpolated at runtime instead).