# HEF — Content Pipeline (source → process → label → manifest)

**Date:** 2026-06-24
**Status:** Graduated — operator-approved (session 0014); **Increment 1 built & merged** (PR #13); real PD sourcing + rotating pool curated (PR #14/#15). **Increment 2 BUILT** (session 0016 — pool model + tiers + windows in PR #16; `track.py` + author mode in PR B; see §11).
**Repo:** `human-experience-filter-art`
**Anchor:** `docs/ROADMAP.md` sub-project 3 (Player Runtime) content needs; the
spiritual successor to sub-project 2 (Ingest & Tagging tools), retargeted from the
retired coordinate-catalog model to the manifest content model.
**Refines:** [`2026-06-07-scales-library-and-right-axis-pipeline-design.md`](./2026-06-07-scales-library-and-right-axis-pipeline-design.md)
§2 (scales library), §2.1 (strict-PD sourcing), §6 (open sourcing questions).
**Builds on:** [`2026-06-22-right-axis-deterministic-dream-design.md`](./2026-06-22-right-axis-deterministic-dream-design.md)
(the real-time dream — **no per-clip ML bake**) and
[`2026-06-22-affect-channel-hud-design.md`](./2026-06-22-affect-channel-hud-design.md)
(scene-anchored affect words).

---

## 1. Why this exists

The installation has an alteration engine, a scale ring, a Left analytical HUD, an
affect channel, and a real-time Right "dream" — but it has **no real content**. The
three ring scales today are: `cosmos` and `abyss` carrying *placeholder bytes* (PD
provenance recorded, but the media is synthetic), and `forest` carrying a *POC
sample* (a trimmed Yosemite clip licensed "look-tuning only; not shipped content").
There is no repeatable way to turn a public-domain source clip into a shippable,
seamless-looping, labelled neutral base.

This design defines that pipeline: **source → frame → seamless-loop → transcode →
label/author → manifest entry.** One big anchor fact reshapes it versus the old
plan — the Right dream is now a **real-time deterministic shader** (session 0013),
so there is **no per-clip ML restyle bake** anymore. The pipeline's offline cost is
therefore **transcode + human authoring**, not generative-video baking. The one
deliberate exception is the motion-tracking label pass chosen this session (§3),
which re-introduces a bounded offline compute step — by choice, for realism.

## 2. Decisions settled this session

| # | Decision | Choice | Consequence |
|---|----------|--------|-------------|
| 1 | Scale set | **Lean 5** — cosmos · orbit · forest/land · reef · abyss (microscopic dropped) | Ring wraps **abyss → cosmos**; replaces the POC forest + both placeholder bases with real strict-PD footage |
| 2 | Seamless loop | **Crossfade loop** (tail→head overlap) | Any source clip becomes an indefinite seamless dwell loop; a transcode-time step |
| 3 | Label placement | **Motion-tracked, hybrid** | ML proposes tracks where it can; hand-seed + classical tracker elsewhere. Re-introduces an offline pass + an authoring step. Semantics always hand-authored |
| 4 | Format | **Master + 1080p proxy** | High-quality master retained per clip; a 1080p H.264 proxy drives the sim; Pi chooses later |

Microscopic was dropped because PD microscopy *video* (vs stills) is the thinnest
pool and hardest to source well; it can be added later by re-running the pipeline.

## 3. The pipeline — six stages

Each stage is a deterministic unit with one job (handbook §4.2: deterministic tools,
not prose an agent re-derives). Stages 1–4 and 6 are mechanical; stage 5 is the one
human-in-the-loop step.

```
 (1) source ──▶ (2) frame ──▶ (3) loop ──▶ (4) transcode ──▶ (6) manifest
 provenance     trim/crop     crossfade     master + proxy        entry
                & scale        seam              │                  ▲
                                                 └──▶ (5) label/author (hybrid track)
```

### 3.1 Stage 1 — source & provenance
Acquire the strict-PD source clip and record license + source URL + capture notes.
Reuse sub-project-2 fetchers where they map (a NASA fetcher already exists); add
NOAA Ocean Exploration / NPS / USGS fetchers (or accept a manual download + a
provenance record). The "no explicit license → assume PD, verify" trap from the
scales design §2.1 applies: **"free stock" (Pexels/Pixabay) is NOT public domain.**
Output: the raw source file + a provenance record (license, source, url, fetched_at).

### 3.2 Stage 2 — frame (trim / crop / scale)
Pick the in/out points; crop to the target aspect; scale toward the format target
(§6). The panoramic projector aspect is still unmeasured (open question), so v1
frames to a standard wide 16:9 for the sim; the master retains maximum available
width so a later pano crop/extension has pixels to work with. Output: a framed,
silent (audio stripped — bases are video-only; audio is the separate content-dial
channel) intermediate.

### 3.3 Stage 3 — seamless crossfade loop
Make the clip loop indefinitely with no visible seam by overlapping the tail into
the head with a short crossfade (~0.5–1.0 s). The final looped clip is shorter than
the source by the overlap; during the overlap region motion is a blend (acceptable
for calm ambient nature footage). This is the chosen loop strategy (clean-cut and
palindrome were rejected — seam risk and reversed motion respectively). Output: a
loop-clean clip whose first and last frames join seamlessly.

### 3.4 Stage 4 — transcode (master + proxy)
Emit two encodes per clip (§6 for exact params): a **master** (high quality, up to
4K, retained as the archival/Pi-candidate source) and a **1080p H.264 proxy** (what
the sim plays today, matching current `base.mp4` behaviour). Both video-only,
`yuv420p`, even dimensions, `+faststart`.

### 3.5 Stage 5 — label & author (the hybrid motion-track pass)
The only human-in-the-loop stage, and the only one that adds offline ML compute.

- **Semantics are always hand-authored.** A generic detector cannot produce the
  installation's semantic keys — "spiral galaxy", "siphonophore", `z ≈ 0.04`,
  `~2.1 m³/s` are out-of-distribution. The author owns the label **keys**,
  **strings**, and `min_level`; tracking only propagates **box geometry**.
- **Hybrid geometry (decision 3).** Where a model can lock onto a subject (a fish
  on the reef, an animal in the forest), an ML detect+track pass (YOLO+ByteTrack /
  SAM2-class, lazy-imported like the parked baker imported torch) proposes a track
  the author maps to a key. Where it can't (cosmos, abyss, microscopic-type
  content), the author **hand-seeds** a box on a keyframe and a **classical tracker**
  (OpenCV CSRT / optical-flow) propagates it; the author fixes drift. Both paths
  emit the same artifact.
- **Channel rules.** `detected.*` object labels **track**; `measure.*` readouts and
  `feel.*` affect words stay **static / scene-anchored** (a depth readout or "awe"
  is not pinned to a moving pixel). This matches the affect-channel design (affect
  is scene-level) and keeps measurement chips legible.
- Output: an **annotation track** per object label (§4) plus the static measurement
  and affect placements, all keyed and stringed.

### 3.6 Stage 6 — manifest assembly
Write the clip entry (base/proxy/master paths, license, source, annotations with
tracks, affect, strings) into `simulator/sample_media/manifest.json`, and register
the scale in the `ring` section with its transition edges (§5). Idempotent: stage 6
owns the manifest entry so re-running earlier stages never clobbers authored labels
(mirrors how `setup_sample_media.py` was made to never clobber a real bake).

## 4. Data model — annotation tracks

Today an annotation is a single static box: `{"key", "box":[x,y,w,h], "min_level"}`.
Add an **optional keyframed track**; static labels are unchanged (backward
compatible):

```json
{
  "key": "detected.fish",
  "min_level": 1,
  "track": [
    {"t": 0.0, "box": [0.30, 0.40, 0.10, 0.08]},
    {"t": 0.5, "box": [0.52, 0.38, 0.10, 0.08]},
    {"t": 1.0, "box": [0.30, 0.40, 0.10, 0.08]}
  ]
}
```

- `t` is **normalized 0→1 over the looped clip** — resolution- and fps-independent,
  so the same track is correct for both master and proxy and survives a re-transcode.
  (A track that loops cleanly has matching boxes at `t=0` and `t=1`.)
- **Runtime:** if `track` is present, the client interpolates the box at the current
  loop-normalized playback time (linear between keyframes); otherwise it uses the
  static `box`. This lives **client-side**, consistent with the existing split
  (alteration math in Python; label selection/placement in the client). `Clip` gains
  the track only as data inside `annotations`; `player/alteration.py` is **unchanged**
  (labels are not part of `RenderPlan`).
- `measure.*` keep a single static `box`; `feel.*` keep a single static `at:[x,y]`.

## 5. The 5 scales, sources, and ring

| Scale | Strict-PD source | Notes |
|-------|------------------|-------|
| `cosmos` | NASA / Hubble / JWST | 🟢 abundant true PD (17 U.S.C. §105); replaces placeholder |
| `orbit` | NASA / ISS (Earth from orbit) | 🟢 true PD; **new** scale |
| `forest` | NPS / USGS (US land/wildlife) | 🟢 true PD (US locations); replaces the non-shippable POC sample |
| `reef` | NOAA Ocean Exploration (shallow) | 🟢 true PD worldwide; **new** scale |
| `abyss` | NOAA Ocean Exploration (deep sea) | 🟢 true PD worldwide; replaces placeholder |

**Ring order (large → small, zooms inward; wraps small → large):**
`cosmos → orbit → forest → reef → abyss → (wrap) cosmos`. Five scales → five ring
edges. The wrap seam is now **abyss → cosmos** (deep sea back out to the cosmos) —
the infinite-zoom payoff, minus the microscopic step.

**Transitions** between the new adjacent edges remain the **deferred i2v generative
work** (scales design §3 / §6) — they need a generative-video model and two real
adjacent bases, heavier than anything in this pipeline. Until then new edges carry
**placeholder transitions** (the `setup_scales_media.py` ffmpeg `xfade=zoomin`
generator already produces these), exactly as cosmos/forest/abyss do today. This
pipeline's job is the **bases + labels**; transitions are tracked but out of scope
here (§8).

## 6. Format target

| | Master | Proxy (sim) |
|---|--------|-------------|
| Resolution | source, capped ~3840 wide | 1920×1080 (fit wide) |
| Codec | H.264 High | H.264 High |
| Quality | CRF ~18 | CRF ~20 |
| FPS | preserve source (cap 30) | 30 |
| Pixel fmt | `yuv420p`, even dims | `yuv420p`, even dims |
| Audio | none (stripped) | none (stripped) |
| Flags | `+faststart` | `+faststart` |

The sim plays the **proxy** as today's `base_file`. The **master** is retained per
clip for the eventual Pi/pano encode once the panoramic resolution is measured.
Browser HEVC support is too spotty to make HEVC the sim path, which is why the proxy
stays H.264 (the master can be re-encoded to HEVC for the Pi later without
re-sourcing).

## 7. Tooling & where it lives

- **New `tools/pipeline/` package** — deterministic ffmpeg-backed verbs for stages
  1–4 and 6 (`fetch`/provenance, `frame`, `loop`, `transcode`, `manifest`), reusing
  sub-project-2 fetchers where valid. This **supersedes** the coordinate-era
  ingest *drafting* (the `left/right/dark/light` Record model retired with the
  machine-altered-perception pivot); the pipeline targets the **manifest**, not
  catalog `Record`s.
- **Stage 5 tracking** — `tools/pipeline/track.py`: classical OpenCV tracker
  (always available) + an optional lazy-imported ML detect+track path. Emits
  candidate keyframed tracks for the author to accept/correct.
- **Authoring surface** — extend the **simulator** with an **author mode** (it
  already renders the exact HUD overlay and is the by-eye judge): scrub the clip,
  draw/seed/adjust boxes, run the tracker, assign keys/`min_level`/strings, place
  affect anchors, write the manifest entry. Reusing the preview as the editor keeps
  "what you author is what plays." **This is the largest new piece** — see the
  delivery order (§8), which lets v1 prove the schema before the full editor exists.

## 8. Delivery / rollout order

Build order within this Feature (Solution-Design rollout strategy — not a task
list):

1. **Increment 1 — real footage, no ML.** Stages 1–4 + 6 + the §4 track schema +
   client interpolation. Source the 5 strict-PD bases, crossfade-loop them, emit
   master+proxy, wire them into the ring. Author a *couple* of tracks **by hand as
   JSON** to prove the track schema and the runtime interpolation. Outcome: the
   whole ring is **real, seamless-looping footage** judgeable by eye — the
   highest-value, lowest-risk step, and it stands alone.
2. **Increment 2 — the hybrid track pass + author mode.** `track.py` (classical +
   optional ML) and the simulator author mode; label all five scales properly.

**Deferred (tracked, not built here):** the real **i2v ring transitions** for the
new edges (need a generative-video model + adjacent real bases); the Pi renderer +
serial/firmware (per the simulator-first directive); the panoramic-resolution
master encode (waits on a measured pano spec).

> Scope note: if Increment 2's author mode grows large enough to be its own
> increment in practice, split it at execution time. The design is one coherent
> pipeline; the increments above are its sensible build order, not separate
> Features.

## 9. Open questions (for the plan, not blockers)

- **Panoramic resolution/aspect** — unmeasured (scales design §6). Drives the master
  crop/extension strategy; v1 frames 16:9 for the sim and retains max width.
- **ML model choice** for the hybrid track path (YOLO+ByteTrack vs SAM2-class) and
  whether it earns its place given how few scales it can usefully fire on.
- **Classical tracker drift** on long/fast clips — keyframe interval, re-seed cadence.
- **Source fetchers** — which of NOAA / NPS / USGS get real fetchers vs manual
  download + provenance record for v1.
- **Crossfade overlap length** per scale (slow cosmos vs busier reef) — by eye.

## 11. Increment 2 — schema decisions (session 0016)

Increment 2 is the second build order from §8. It is mostly **schema work**, so
this section pins the concrete data shapes before the build (the operator asked for
a short design pass first). Each subsection is a new, **backward-compatible**
extension — Increment 1 manifests keep rendering unchanged.

### 11.1 Rotating clip pool (supersedes one-clip-per-scale)

Each ring scale references a **pool** of vetted clips
(`docs/content-candidate-pool.md`); the player picks **one member at random when
the viewer lands on that scale** (zooming the ring in/out). This supersedes
decision §5's one-base-per-scale.

- **Manifest:** `ring.scales[i]` gains `"pool": ["clip_id", …]` (≥1). The legacy
  `"clip_id"` stays as the **primary** member for back-compat (= `pool[0]` when a
  pool is written); a scale with only `clip_id` and no `pool` behaves exactly as
  Increment 1 (pool of one).
- **Canonical logic (`player/ring.py`):** `Scale` gains `pool: tuple[str, …]`;
  `members` = `pool or (clip_id,)`. A **pure** `pick_clip_id(scale, r)` takes an
  injected `r ∈ [0,1)` and returns `members[int(r·len)]` — randomness stays
  **injected** (DI), so the function is pure/testable and the Pi reuses it.
- **Where the pick happens:** at the **API boundary** (`simulator/app.py`), which
  may be impure: the advance endpoint calls `pick_clip_id(landed_scale,
  random.random())` and returns it as `target_clip_id`. A `delta=0` advance is the
  **initial / re-roll** pick (no-op move, fresh random member). The client loads
  `move.target_clip_id` directly instead of deriving the clip from the index. One
  selection implementation, Python-canonical (browser still "only renders").
- **Rename + cleanup:** ring scale `forest` → **`coast`** (new land↔sea scale
  between `orbit` and `reef`). Drop the leftovers `cosmos_traverse`,
  `orbit_westcoast`, old `forest`, old `reef` (per the pool doc's cleanup list);
  the cosmos pool's primary stays `cosmos` (Orion). New ring order:
  `cosmos → orbit → coast → reef → abyss → (wrap) cosmos`; edges regenerated
  (`orbit-coast`, `coast-reef` placeholders via `setup_scales_media.py`).

### 11.2 Time-windowed tracked labels

Objects drift **into and out of frame**, so a label appears only **while its
object is on screen** and its box **tracks** the object (the §4 keyframed `track`).

- **Manifest:** an annotation gains optional `"appear"` and `"disappear"`
  (loop-normalized `t ∈ [0,1]`, same clock as `track`). The label renders only
  while `appear ≤ loopT ≤ disappear`. To let an object straddle the loop seam, a
  **`disappear < appear` window wraps** (visible when `loopT ≥ appear` **or**
  `loopT ≤ disappear`). Absent both fields → always-present (Increment-1 behavior).
- **Runtime:** client-side in `renderOverlay` — a window test gates the annotation
  before the existing `boxAt(loopT)` placement. Engine unchanged (labels are not in
  `RenderPlan`). The **abyss + reef** pool clips (creatures entering/leaving) are
  the authored test material.

### 11.3 Progressive LEFT-brain detail tiers (replaces flat `min_level` gating)

The Left knob now drives **how many** labels show **and how analytical** each is —
no fading. Low Left = **fewer, more general** labels; high Left = **more** labels +
**more analytical** detail.

- **Per-object `salience` (1–4):** how prominent the object is. It sets the Left
  level at which the label first appears: `first_level = 5 − salience` (salience 4 →
  appears at Left 1; salience 1 → only at Left 4). So raising Left brings in
  lower-salience objects.
- **Detail tiers (escalate with Left):** the displayed string escalates
  `tier 1 general ("eel") → 2 specific ("moray eel") → 3 scientific
  (*Gymnothorax*) → 4 scientific + a fact (lifespan)`. The tier shown =
  `clamp(left − first_level + 1, 1, n_tiers)` — an object shows tier 1 the moment
  it appears and climbs one tier per further Left notch until its tiers run out.
- **Strings carry the tiers:** `strings.en[key]` may be a **list** (indexed by
  `tier−1`) instead of a string; a plain string stays static (back-compat, tier
  ignored). Keeps one key per object and the i18n table as the single source.
- **Back-compat:** an annotation with `min_level` and no `salience` keeps flat
  gating (`first_level = min_level`, single-tier string). `measure.*` readouts stay
  single-tier, gated by their level (a measurement is not a tiered object).
- **No engine change:** tier selection is client-side against the existing
  `overlay.level` (the Left knob) already in the `RenderPlan`.

### 11.4 Progressive RIGHT-brain emotion tiers

The affect **vocabulary** gets more sophisticated as Right rises — basic words at
Right 1, nuanced/compound feelings at high Right — mirroring §11.3 for affect.

- **Appearance rule unchanged:** affect surfaces only when **both** knobs are up;
  an anchor's `min_level` is tested against `strength = min(left, right)` as today.
- **Emotion tiers:** `strings.en[feel.key]` may be a **list**; the word shown =
  `clamp(right, 1, n_tiers)` (the Right knob, read from `dream.strength` already in
  the `RenderPlan`). A plain string stays static (back-compat).
- **No engine change:** `AffectOverlay` already carries `strength`; the Right level
  comes from `dream.strength`. Tier selection is client-side.

### 11.5 `tools/pipeline/track.py` (hybrid) + simulator author mode

The stage-5 tooling from §3.5 / §7, delivered as a **second PR** after the
experience PR (§8's split note).

- **`track.py`:** a **classical** path (OpenCV CSRT / optical-flow) seeded by a
  hand-drawn box on a keyframe, propagated and **sampled to a sparse keyframed
  `track`** (normalized `{t, box}` over the loop, §4); plus an **optional,
  lazy-imported ML** detect+track path (YOLO+ByteTrack-class) that proposes tracks
  the author maps to keys. Both emit the same `track` artifact + an `appear`/
  `disappear` window. Pure helpers (normalization, keyframe sampling,
  track→manifest) are unit-tested; an **opt-in** integration test runs the real
  tracker on a pool clip (mirrors the Increment-1 real-footage test).
- **Author mode (simulator):** reuse the preview UI as the editor — scrub a pool
  clip, draw/seed/correct boxes, run the tracker, assign `key` / `salience` /
  tiered strings / `appear`-`disappear` / affect anchors, and **write the manifest
  entry** (an endpoint upserts via `tools/pipeline/manifest.upsert_clip` and
  persists). Semantics (keys, strings, scientific names, facts) are **always
  hand-authored** — the tracker only propagates **box geometry** (§3.5).

## 12. Out of scope (YAGNI)

- Generative i2v **ring transitions** for the new edges (deferred; placeholders hold).
- **Audio** sourcing/alteration (separate content-dial channel; already deferred).
- The **Pi/pano** encode + renderer and serial/firmware (simulator-first).
- Reviving the retired coordinate-catalog `Record` drafting.
- Per-frame **dense** tracks (sparse keyframed tracks interpolated at runtime instead).
