tools(pipeline): hybrid track.py + simulator label author mode

Content-pipeline Increment 2, part 2 (the tooling). Implements design §11.5
(docs/superpowers/specs/2026-06-24-content-pipeline-design.md):

- tools/pipeline/track.py — the stage-5 geometry pass. Classical path: a
  hand-seeded normalized box propagated by OpenCV Lucas-Kanade optical flow
  (CSRT used instead when a contrib build provides it), sampled to a SPARSE
  loop-normalized keyframed track + an appear/disappear window. Optional ML
  detect+track path lazy-imports ultralytics (clear ImportError if absent;
  base install needs only cv2). Pure helpers (normalize/denormalize, loop_t,
  infer_window, sample_track, track_to_annotation) are unit-tested; the cv2
  propagation is an opt-in integration test on real abyss_wow footage.
  Semantics are never produced here — geometry only.

- Author mode — /author.html + author.js/.css reuse the preview stage: pick a
  pool clip, scrub, drag a seed box, Run tracker (or Add as static box), author
  the LEFT detail tiers (general -> scientific+fact) + salience, shift-click to
  place affect anchors with RIGHT emotion tiers, and Save to manifest. Backend:
  POST /api/author/track (runs track_seed on the clip's base) + POST
  /api/author/clip (idempotent upsert via tools.pipeline.manifest — keeps media
  + provenance, replaces only authored content, reloads in place). The tracker
  propagates box geometry; all strings/scientific names/facts are hand-typed.

- Repointed the pipeline integration test off the retired forest base to the
  cosmos pool primary. USER_GUIDE simulator section brought current (pools,
  coast, 3-knob Mood, real-time dream, progressive tiers, author mode).

267 passed / 2 skipped (+ track pure + opt-in real-footage + author endpoint
tests). Author UI by-eye review deferred to the operator (no Chrome on this box).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
BenStullsBets
2026-06-24 18:14:32 -07:00
parent 12408e505e
commit 70fd367c70
10 changed files with 1023 additions and 48 deletions
+48 -37
View File
@@ -296,24 +296,17 @@ exists. (The earlier selection-era "curator's X-ray" view was retired when the
piece moved from *selecting* clips to *altering* them.)
**One-time setup — populate the sample footage** (look-tuning only; not shipped
content):
content). The ring uses real strict-PD footage in a **rotating pool** per scale
(`docs/content-candidate-pool.md`); regenerate the manifest + new transition
placeholders with:
python simulator/setup_sample_media.py # stage the forest neutral base
python -m simulator.bake_right_variants --scale forest # real Right variants (SD on MPS, ~11 min)
python simulator/setup_scales_media.py # the cosmos + abyss scales + ring transitions
python simulator/build_pool_manifest.py --media # write the pool manifest + coast-edge transitions
`setup_sample_media.py` stages the forest neutral base from the session-0008 POC
artifacts (`~/hef-poc/out/neutral.mp4`). `bake_right_variants.py` then produces
the **real** flow-stabilized painterly Right variants (strengths 14) for the
forest scale — SD img2img keyframes + Farneback optical-flow tweens on Apple MPS,
the temporally-calm POC algorithm, productionized (scales design §1/§4). The
keyframe-strength ramp per Right level is by-eye tunable in
`simulator/bake_right_variants.py`. `setup_scales_media.py` makes the scale
**ring** demonstrable: cheap synthetic placeholder bases for the two true-PD
scales (`cosmos` = NASA/Hubble, `abyss` = NOAA Ocean Exploration) plus the
per-edge zoom/warp transition clips — these scales carry a raw base only (no real
Right variants yet; the baker can extend to them once their real footage is
sourced). The `.mp4` binaries are gitignored.
`build_pool_manifest.py` holds the hand-authored label/affect content and emits
`simulator/sample_media/manifest.json` (the 19-clip pool baseline). The pool clips
themselves are sourced via the content pipeline (`tools/pipeline/run.py
process_clip`); the `.mp4` binaries are gitignored. (`setup_scales_media.py` is the
older one-base-per-scale generator, now superseded.)
**Run it (Docker):**
@@ -331,27 +324,45 @@ then open http://localhost:8000.
- **Content dial** — picks audio/video channel; "off" and audio-only positions go
to black walls.
- **Scale ring (endless encoder)** — `⊖ out` / `in ⊕` (or scroll the stage) walk a
*closed ring* of neutral "scales of nature" clips — cosmos → forest → abyss and
back around (diving past the smallest **wraps** to the largest). It is *relative*
(an endless encoder), distinct from the absolute 04 knobs: each step plays a
short placeholder zoom/warp **transition**, then settles on the next scale, with
the current knob alteration still applied. The current scale is named beside the
buttons (`name (i/N)`). A **fast spin** (scroll several detents at once, ≥3)
collapses to a single quick **blended pass** straight to the destination scale
instead of grinding through every transition (scales design §3); slow single
steps still chain one full transition per scale crossed.
- **Four experience knobs (04):**
- **Dark / Light** — a live runtime color grade (cool/dark ↔ warm/bright; equal
or zero = the raw footage).
- **Right (dreamlike)** — selects a discrete pre-baked, flow-stabilized restyle
variant and crossfades to it (strength 0 = raw base).
- **Left (analytical)** — a live overlay: labelled boxes from the clip's authored
annotation track, with more annotations appearing at higher levels. Text is
shaped live (the simulator analogue of the Pi's Pango/HarfBuzz path).
- **Calibration sliders** — adjust the grade/overlay gain curves live; once a look
is liked, bake the values into `DEFAULT_CALIBRATION` in `player/alteration.py`.
*closed ring* of neutral "scales of nature" — **cosmos → orbit → coast → reef →
abyss** and back around (diving past the smallest **wraps** to the largest). Each
scale is a **rotating pool** of vetted clips: landing on a scale plays a random
pool member, so the ring feels fresh each pass. It is *relative* (an endless
encoder), distinct from the absolute 04 knobs: each step plays a short
placeholder zoom/warp **transition**, then settles, with the current alteration
still applied. The current scale + chosen member is named beside the buttons
(`scale · member (i/N · pool N)`). A **fast spin** (≥3 detents at once) collapses
to one quick **blended pass** instead of grinding through every transition.
- **Three experience knobs:**
- **Mood (4 dark .. 0 .. +4 light)** — a live runtime color grade (cool/dark ↔
warm/bright; 0 = the raw footage).
- **Right (dreamlike, 04)** — a **deterministic real-time painterly dream** (a
WebGL Kuwahara restyle of the live frames; holds still across the loop, goes
trippy at max). It also drives **progressive emotion tiers**: when both knobs
are up, the affect words escalate from basic to compound as Right rises.
- **Left (analytical, 04)** — a live HUD of labelled boxes from the clip's
authored annotation track. Labels use **progressive detail tiers**: low Left
shows fewer, more general labels (high-salience objects only); raising Left
brings in more objects *and* escalates each label general → specific →
scientific → +fact. Time-windowed tracked labels appear only while their
subject is on screen and follow it.
- **RenderPlan readout** — always shows the exact numbers the engine produced (the
project's honesty "X-ray," now over the alteration model).
project's honesty "X-ray," over the alteration model).
The base clips, Right variants, Left annotation track, and string tables come from
The base clips, the Left annotation tracks (with tiers + appear/disappear
windows), affect anchors, and per-tier string tables all come from
`simulator/sample_media/manifest.json`.
### Authoring labels — the author mode
Open **http://localhost:8000/author.html** to author a clip's labels by eye
(content-pipeline §11.5). Pick a pool clip, **scrub** to where a subject is on
screen, **drag a box** over it, set the label key + salience + the four detail
tiers, then **Run tracker** to propagate the box into a keyframed track (classical
OpenCV optical flow) with an appear/disappear window — or **Add as static box**
for a fixed label. Shift-click the stage to place an **affect anchor** and type its
emotion tiers. **Save to manifest** writes the entry back via the pipeline's
idempotent upsert (keeping the clip's media + provenance, replacing only the
authored content). The tracker only propagates **box geometry** — every label
string, scientific name, and fact is **hand-authored**, because a generic detector
can't produce "*Gymnothorax*" or a lifespan.