diff --git a/docs/USER_GUIDE.md b/docs/USER_GUIDE.md
index 59f737d..b77a611 100644
--- a/docs/USER_GUIDE.md
+++ b/docs/USER_GUIDE.md
@@ -296,24 +296,17 @@ exists. (The earlier selection-era "curator's X-ray" view was retired when the
piece moved from *selecting* clips to *altering* them.)
**One-time setup — populate the sample footage** (look-tuning only; not shipped
-content):
+content). The ring uses real strict-PD footage in a **rotating pool** per scale
+(`docs/content-candidate-pool.md`); regenerate the manifest + new transition
+placeholders with:
- python simulator/setup_sample_media.py # stage the forest neutral base
- python -m simulator.bake_right_variants --scale forest # real Right variants (SD on MPS, ~11 min)
- python simulator/setup_scales_media.py # the cosmos + abyss scales + ring transitions
+ python simulator/build_pool_manifest.py --media # write the pool manifest + coast-edge transitions
-`setup_sample_media.py` stages the forest neutral base from the session-0008 POC
-artifacts (`~/hef-poc/out/neutral.mp4`). `bake_right_variants.py` then produces
-the **real** flow-stabilized painterly Right variants (strengths 1–4) for the
-forest scale — SD img2img keyframes + Farneback optical-flow tweens on Apple MPS,
-the temporally-calm POC algorithm, productionized (scales design §1/§4). The
-keyframe-strength ramp per Right level is by-eye tunable in
-`simulator/bake_right_variants.py`. `setup_scales_media.py` makes the scale
-**ring** demonstrable: cheap synthetic placeholder bases for the two true-PD
-scales (`cosmos` = NASA/Hubble, `abyss` = NOAA Ocean Exploration) plus the
-per-edge zoom/warp transition clips — these scales carry a raw base only (no real
-Right variants yet; the baker can extend to them once their real footage is
-sourced). The `.mp4` binaries are gitignored.
+`build_pool_manifest.py` holds the hand-authored label/affect content and emits
+`simulator/sample_media/manifest.json` (the 19-clip pool baseline). The pool clips
+themselves are sourced via the content pipeline (`tools/pipeline/run.py
+process_clip`); the `.mp4` binaries are gitignored. (`setup_scales_media.py` is the
+older one-base-per-scale generator, now superseded.)
**Run it (Docker):**
@@ -331,27 +324,45 @@ then open http://localhost:8000.
- **Content dial** — picks audio/video channel; "off" and audio-only positions go
to black walls.
- **Scale ring (endless encoder)** — `⊖ out` / `in ⊕` (or scroll the stage) walk a
- *closed ring* of neutral "scales of nature" clips — cosmos → forest → abyss and
- back around (diving past the smallest **wraps** to the largest). It is *relative*
- (an endless encoder), distinct from the absolute 0–4 knobs: each step plays a
- short placeholder zoom/warp **transition**, then settles on the next scale, with
- the current knob alteration still applied. The current scale is named beside the
- buttons (`name (i/N)`). A **fast spin** (scroll several detents at once, ≥3)
- collapses to a single quick **blended pass** straight to the destination scale
- instead of grinding through every transition (scales design §3); slow single
- steps still chain one full transition per scale crossed.
-- **Four experience knobs (0–4):**
- - **Dark / Light** — a live runtime color grade (cool/dark ↔ warm/bright; equal
- or zero = the raw footage).
- - **Right (dreamlike)** — selects a discrete pre-baked, flow-stabilized restyle
- variant and crossfades to it (strength 0 = raw base).
- - **Left (analytical)** — a live overlay: labelled boxes from the clip's authored
- annotation track, with more annotations appearing at higher levels. Text is
- shaped live (the simulator analogue of the Pi's Pango/HarfBuzz path).
-- **Calibration sliders** — adjust the grade/overlay gain curves live; once a look
- is liked, bake the values into `DEFAULT_CALIBRATION` in `player/alteration.py`.
+ *closed ring* of neutral "scales of nature" — **cosmos → orbit → coast → reef →
+ abyss** and back around (diving past the smallest **wraps** to the largest). Each
+ scale is a **rotating pool** of vetted clips: landing on a scale plays a random
+ pool member, so the ring feels fresh each pass. It is *relative* (an endless
+ encoder), distinct from the absolute 0–4 knobs: each step plays a short
+ placeholder zoom/warp **transition**, then settles, with the current alteration
+ still applied. The current scale + chosen member is named beside the buttons
+ (`scale · member (i/N · pool N)`). A **fast spin** (≥3 detents at once) collapses
+ to one quick **blended pass** instead of grinding through every transition.
+- **Three experience knobs:**
+ - **Mood (−4 dark .. 0 .. +4 light)** — a live runtime color grade (cool/dark ↔
+ warm/bright; 0 = the raw footage).
+ - **Right (dreamlike, 0–4)** — a **deterministic real-time painterly dream** (a
+ WebGL Kuwahara restyle of the live frames; holds still across the loop, goes
+ trippy at max). It also drives **progressive emotion tiers**: when both knobs
+ are up, the affect words escalate from basic to compound as Right rises.
+ - **Left (analytical, 0–4)** — a live HUD of labelled boxes from the clip's
+ authored annotation track. Labels use **progressive detail tiers**: low Left
+ shows fewer, more general labels (high-salience objects only); raising Left
+ brings in more objects *and* escalates each label general → specific →
+ scientific → +fact. Time-windowed tracked labels appear only while their
+ subject is on screen and follow it.
- **RenderPlan readout** — always shows the exact numbers the engine produced (the
- project's honesty "X-ray," now over the alteration model).
+ project's honesty "X-ray," over the alteration model).
-The base clips, Right variants, Left annotation track, and string tables come from
+The base clips, the Left annotation tracks (with tiers + appear/disappear
+windows), affect anchors, and per-tier string tables all come from
`simulator/sample_media/manifest.json`.
+
+### Authoring labels — the author mode
+
+Open **http://localhost:8000/author.html** to author a clip's labels by eye
+(content-pipeline §11.5). Pick a pool clip, **scrub** to where a subject is on
+screen, **drag a box** over it, set the label key + salience + the four detail
+tiers, then **Run tracker** to propagate the box into a keyframed track (classical
+OpenCV optical flow) with an appear/disappear window — or **Add as static box**
+for a fixed label. Shift-click the stage to place an **affect anchor** and type its
+emotion tiers. **Save to manifest** writes the entry back via the pipeline's
+idempotent upsert (keeping the clip's media + provenance, replacing only the
+authored content). The tracker only propagates **box geometry** — every label
+string, scientific name, and fact is **hand-authored**, because a generic detector
+can't produce "*Gymnothorax*" or a lifespan.
diff --git a/docs/superpowers/specs/2026-06-24-content-pipeline-design.md b/docs/superpowers/specs/2026-06-24-content-pipeline-design.md
index f121d65..08a276f 100644
--- a/docs/superpowers/specs/2026-06-24-content-pipeline-design.md
+++ b/docs/superpowers/specs/2026-06-24-content-pipeline-design.md
@@ -1,7 +1,7 @@
# HEF — Content Pipeline (source → process → label → manifest)
**Date:** 2026-06-24
-**Status:** Graduated — operator-approved (session 0014); **Increment 1 built & merged** (PR #13); real PD sourcing + rotating pool curated (PR #14/#15). **Increment 2 schema pinned + building** (session 0016 — see §11).
+**Status:** Graduated — operator-approved (session 0014); **Increment 1 built & merged** (PR #13); real PD sourcing + rotating pool curated (PR #14/#15). **Increment 2 BUILT** (session 0016 — pool model + tiers + windows in PR #16; `track.py` + author mode in PR B; see §11).
**Repo:** `human-experience-filter-art`
**Anchor:** `docs/ROADMAP.md` sub-project 3 (Player Runtime) content needs; the
spiritual successor to sub-project 2 (Ingest & Tagging tools), retargeted from the
diff --git a/simulator/app.py b/simulator/app.py
index 658bedf..df0f24a 100644
--- a/simulator/app.py
+++ b/simulator/app.py
@@ -8,6 +8,7 @@ endpoints (/api/select, /api/catalog/meta) and the X-ray are retired.
from __future__ import annotations
import hashlib
+import json
import os
import random
import time
@@ -84,6 +85,29 @@ class RingAdvanceRequest(BaseModel):
delta: int
+class AuthorTrackRequest(BaseModel):
+ """Author mode: propagate a hand-seeded box on a clip into a keyframed track
+ (content-pipeline §11.5). `seed_box` + `seed_t` are normalized."""
+
+ clip_id: str
+ key: str
+ seed_box: list[float] = Field(min_length=4, max_length=4)
+ seed_t: float = Field(default=0.0, ge=0.0, le=1.0)
+ salience: int = Field(default=4, ge=1, le=4)
+ n_keyframes: int = Field(default=5, ge=2, le=20)
+ max_frames: Optional[int] = None
+
+
+class AuthorClipRequest(BaseModel):
+ """Author mode: persist the authored labels/affect/strings for an existing
+ pool clip into the manifest (geometry from the tracker, semantics by hand)."""
+
+ clip_id: str
+ annotations: list = []
+ affect: list = []
+ strings: dict = {}
+
+
def _load_clips(manifest_path: Optional[Path]):
path = Path(manifest_path) if manifest_path else DEFAULT_MANIFEST
if path.exists():
@@ -100,6 +124,7 @@ def _load_ring(manifest_path: Optional[Path]):
def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
app = FastAPI(title="HEF Alteration Simulator")
+ app.state.manifest_path = Path(manifest_path) if manifest_path else DEFAULT_MANIFEST
app.state.clips = _load_clips(manifest_path)
app.state.ring = _load_ring(manifest_path)
@@ -158,6 +183,56 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
chosen = pick_clip_id(landed, random.random())
return ring_move_to_dict(move, app.state.ring, chosen)
+ @app.post("/api/author/track")
+ def api_author_track(req: AuthorTrackRequest):
+ # Author mode (§11.5): run the classical optical-flow tracker on a clip's
+ # base footage, returning the keyframed track geometry for the author to
+ # accept/correct. Semantics (key/strings/tiers) stay the author's.
+ clip = next((c for c in app.state.clips if c.id == req.clip_id), None)
+ if clip is None:
+ raise HTTPException(status_code=404, detail=f"unknown clip {req.clip_id!r}")
+ base = MEDIA_DIR / clip.base_file
+ if not base.exists():
+ raise HTTPException(status_code=404, detail=f"media absent: {clip.base_file}")
+ try:
+ import cv2
+ except ImportError:
+ raise HTTPException(status_code=503, detail="opencv-python not installed")
+ from tools.pipeline.track import track_seed
+
+ cap = cv2.VideoCapture(str(base))
+ total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
+ cap.release()
+ seed_frame = max(0, min(total - 1, round(req.seed_t * total))) if total else 0
+ return track_seed(
+ base, req.key, tuple(req.seed_box), seed_frame=seed_frame,
+ salience=req.salience, n_keyframes=req.n_keyframes, max_frames=req.max_frames,
+ )
+
+ @app.post("/api/author/clip")
+ def api_author_clip(req: AuthorClipRequest):
+ # Author mode (§11.5 / stage 6): persist authored labels/affect/strings for
+ # an EXISTING pool clip — idempotent upsert that keeps the clip's media +
+ # provenance and replaces only the authored content. Reloads in place so
+ # the preview reflects the edit without a restart.
+ from tools.pipeline.manifest import upsert_clip
+
+ path = app.state.manifest_path
+ data = json.loads(path.read_text())
+ existing = next((c for c in data.get("clips", []) if c["id"] == req.clip_id), None)
+ if existing is None:
+ raise HTTPException(status_code=404, detail=f"unknown clip {req.clip_id!r}")
+ merged = dict(existing)
+ merged["annotations"] = req.annotations
+ merged["affect"] = req.affect
+ if req.strings:
+ merged["strings"] = req.strings
+ data = upsert_clip(data, merged)
+ path.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n")
+ app.state.clips = load_manifest(path)
+ app.state.ring = load_ring(path)
+ return {"ok": True, "clip": merged}
+
@app.middleware("http")
async def _no_cache(request, call_next):
# Dev preview server: never let a browser serve a stale app.js/style.css.
diff --git a/simulator/static/author.css b/simulator/static/author.css
new file mode 100644
index 0000000..8229fca
--- /dev/null
+++ b/simulator/static/author.css
@@ -0,0 +1,64 @@
+/* Author-mode-only styling (content-pipeline §11.5). Reuses style.css for the
+ shared stage/panel chrome; this adds the box-draw layer + editor widgets. */
+
+.author #draw {
+ position: absolute;
+ inset: 0;
+ width: 100%;
+ height: 100%;
+ cursor: crosshair;
+}
+.author #draw .seed-box {
+ fill: rgba(80, 200, 255, 0.12);
+ stroke: #50c8ff;
+ stroke-width: 0.4;
+ vector-effect: non-scaling-stroke;
+}
+.author #draw .track-box {
+ fill: rgba(120, 255, 160, 0.10);
+ stroke: #78ffa0;
+ stroke-width: 0.4;
+ stroke-dasharray: 1.5 1;
+ vector-effect: non-scaling-stroke;
+}
+.author #draw .affect-dot {
+ fill: rgba(190, 150, 255, 0.85);
+ stroke: #fff;
+ stroke-width: 0.25;
+ vector-effect: non-scaling-stroke;
+}
+
+.author .scrub {
+ display: flex;
+ align-items: center;
+ gap: 8px;
+ margin-top: 8px;
+}
+.author .scrub input[type="range"] { flex: 1; }
+.author .mono { font-family: ui-monospace, monospace; font-size: 12px; opacity: 0.85; }
+
+.author .row { display: flex; gap: 8px; margin-top: 6px; flex-wrap: wrap; }
+.author fieldset label { display: block; margin: 4px 0; font-size: 13px; }
+.author fieldset input[type="text"],
+.author fieldset input[type="number"] { width: 100%; box-sizing: border-box; }
+.author fieldset input[type="number"] { width: 5em; }
+
+.author ul.authored { list-style: none; padding: 0; margin: 6px 0; }
+.author ul.authored li {
+ font-size: 12px;
+ padding: 3px 6px;
+ margin: 2px 0;
+ background: rgba(255, 255, 255, 0.05);
+ border-radius: 3px;
+ display: flex;
+ justify-content: space-between;
+ gap: 6px;
+}
+.author ul.authored .rm {
+ background: none;
+ border: none;
+ color: #f88;
+ cursor: pointer;
+ font-size: 12px;
+}
+.author header .hint { font-size: 12px; opacity: 0.8; max-width: 70ch; }
diff --git a/simulator/static/author.html b/simulator/static/author.html
new file mode 100644
index 0000000..a4f486f
--- /dev/null
+++ b/simulator/static/author.html
@@ -0,0 +1,79 @@
+
+
+
+
+
+ HEF — Label Author Mode
+
+
+
+
+
HEF — Label Author Mode
+
Scrub a pool clip · drag a box on a subject · run the tracker · author the tiers · save to the manifest. Geometry is propagated; semantics are hand-authored (content-pipeline §11.5). ← back to preview
+
+
+
+
+
+
+
+
+
+
+ 0.00
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/simulator/static/author.js b/simulator/static/author.js
new file mode 100644
index 0000000..7dbb7e5
--- /dev/null
+++ b/simulator/static/author.js
@@ -0,0 +1,254 @@
+// Label author mode (content-pipeline §11.5): scrub a pool clip, drag a seed box
+// on a subject, run the classical tracker to propagate a keyframed track, author
+// the LEFT detail tiers + RIGHT emotion tiers, and save the entry to the manifest.
+// Geometry comes from the tracker; SEMANTICS (keys, strings, tiers) are hand-typed.
+const $ = (id) => document.getElementById(id);
+const vid = $("vid"), draw = $("draw");
+const SVGNS = "http://www.w3.org/2000/svg";
+
+let clips = [];
+let currentClip = null;
+let seedBox = null; // [x,y,w,h] normalized, from a drag
+let pendingTrack = null; // {track, appear, disappear} from the tracker (or static)
+let affectPoint = null; // [x,y] normalized, from a shift-click
+let annotations = []; // authored annotation dicts (carry _tiers)
+let affect = []; // authored affect dicts (carry _tiers)
+
+function svg(tag, attrs, parent) {
+ const el = document.createElementNS(SVGNS, tag);
+ for (const k in attrs) el.setAttribute(k, attrs[k]);
+ if (parent) parent.appendChild(el);
+ return el;
+}
+const mono = (n) => Number(n).toFixed(3);
+
+async function loadClips() {
+ clips = (await (await fetch("/api/clips")).json()).clips || [];
+ const sel = $("clip");
+ sel.innerHTML = "";
+ for (const c of clips) {
+ const o = document.createElement("option");
+ o.value = c.id; o.textContent = c.id;
+ sel.appendChild(o);
+ }
+ if (clips.length) selectClip(clips[0].id);
+}
+
+function selectClip(id) {
+ currentClip = clips.find((c) => c.id === id) || null;
+ if (!currentClip) return;
+ $("clip-meta").textContent = `${currentClip.title} — ${currentClip.license}`;
+ vid.src = "/media/" + currentClip.base_file;
+ vid.currentTime = 0;
+ annotations = []; affect = []; seedBox = null; pendingTrack = null; affectPoint = null;
+ renderLists(); renderDrawLayer();
+ $("seed-readout").textContent = "no box — drag on the stage";
+ $("track-readout").textContent = "";
+ $("affect-readout").textContent = "no point — shift-click the stage";
+}
+
+// --- scrub ---
+function seekT() { return vid.duration ? vid.currentTime / vid.duration : 0; }
+$("play").addEventListener("click", () => { vid.paused ? vid.play() : vid.pause(); });
+$("seek").addEventListener("input", () => {
+ if (vid.duration) vid.currentTime = (+$("seek").value / 1000) * vid.duration;
+});
+vid.addEventListener("timeupdate", () => {
+ if (!vid.duration) return;
+ $("seek").value = String(Math.round(seekT() * 1000));
+ $("time").textContent = seekT().toFixed(3);
+});
+
+// --- draw a seed box / place an affect point on the stage ---
+function evtNorm(e) {
+ const r = draw.getBoundingClientRect();
+ return [(e.clientX - r.left) / r.width, (e.clientY - r.top) / r.height];
+}
+let dragStart = null;
+draw.addEventListener("mousedown", (e) => {
+ if (e.shiftKey) { // shift-click = affect anchor
+ affectPoint = evtNorm(e).map((v) => Math.min(Math.max(v, 0), 1));
+ $("affect-readout").textContent = `affect @ ${affectPoint.map(mono).join(", ")}`;
+ $("add-aff").disabled = false;
+ renderDrawLayer();
+ return;
+ }
+ dragStart = evtNorm(e);
+});
+draw.addEventListener("mousemove", (e) => {
+ if (!dragStart) return;
+ seedBox = boxFrom(dragStart, evtNorm(e));
+ renderDrawLayer();
+});
+window.addEventListener("mouseup", (e) => {
+ if (!dragStart) return;
+ seedBox = boxFrom(dragStart, evtNorm(e));
+ dragStart = null;
+ pendingTrack = null;
+ $("seed-readout").textContent = `seed [${seedBox.map(mono).join(", ")}] @ t=${seekT().toFixed(3)}`;
+ $("add-ann").disabled = false;
+ renderDrawLayer();
+});
+function boxFrom(a, b) {
+ const x = Math.min(a[0], b[0]), y = Math.min(a[1], b[1]);
+ const w = Math.abs(b[0] - a[0]), h = Math.abs(b[1] - a[1]);
+ const cl = (v) => Math.min(Math.max(v, 0), 1);
+ return [cl(x), cl(y), Math.min(w, 1 - cl(x)), Math.min(h, 1 - cl(y))];
+}
+
+function renderDrawLayer() {
+ draw.innerHTML = "";
+ if (seedBox) {
+ const [x, y, w, h] = seedBox.map((n) => n * 100);
+ svg("rect", { x, y, width: w, height: h, class: "seed-box" }, draw);
+ }
+ if (pendingTrack && pendingTrack.track && vid.duration) {
+ const b = boxAt(pendingTrack.track, seekT());
+ if (b) {
+ const [x, y, w, h] = b.map((n) => n * 100);
+ svg("rect", { x, y, width: w, height: h, class: "track-box" }, draw);
+ }
+ }
+ if (affectPoint) {
+ const [x, y] = affectPoint.map((n) => n * 100);
+ svg("circle", { cx: x, cy: y, r: 1.2, class: "affect-dot" }, draw);
+ }
+}
+// Interpolate a track box at loop-normalized t (mirrors the preview renderer).
+function boxAt(track, t) {
+ if (!track || !track.length) return null;
+ if (t <= track[0].t) return track[0].box;
+ if (t >= track[track.length - 1].t) return track[track.length - 1].box;
+ for (let i = 1; i < track.length; i++) {
+ if (t <= track[i].t) {
+ const a = track[i - 1], b = track[i], f = (t - a.t) / (b.t - a.t);
+ return a.box.map((v, j) => v + (b.box[j] - v) * f);
+ }
+ }
+ return track[track.length - 1].box;
+}
+// Animate the pending track preview while playing.
+function previewLoop() { renderDrawLayer(); requestAnimationFrame(previewLoop); }
+
+// --- run the tracker on the seed box ---
+$("run-track").addEventListener("click", async () => {
+ if (!seedBox || !currentClip) { $("track-readout").textContent = "draw a seed box first"; return; }
+ const key = $("ann-key").value.trim() || "detected.object";
+ $("track-readout").textContent = "tracking…";
+ const resp = await fetch("/api/author/track", {
+ method: "POST", headers: { "content-type": "application/json" },
+ body: JSON.stringify({
+ clip_id: currentClip.id, key, seed_box: seedBox, seed_t: seekT(),
+ salience: +$("ann-salience").value, n_keyframes: +$("ann-keyframes").value,
+ }),
+ });
+ if (!resp.ok) { $("track-readout").textContent = "tracker error " + resp.status; return; }
+ const ann = await resp.json();
+ pendingTrack = { track: ann.track, appear: ann.appear, disappear: ann.disappear };
+ $("track-readout").textContent =
+ `track: ${ann.track.length} keyframes · window ${mono(ann.appear)}–${mono(ann.disappear)}`;
+ $("add-ann").disabled = false;
+});
+
+// Add the current seed as a STATIC (untracked) box at the scrub position.
+$("add-static").addEventListener("click", () => {
+ if (!seedBox) return;
+ pendingTrack = { static: true, box: seedBox.slice() };
+ $("track-readout").textContent = "static box (no track) at the drawn position";
+ $("add-ann").disabled = false;
+});
+
+function tiers(ids) {
+ const vals = ids.map((id) => $(id).value.trim());
+ while (vals.length && !vals[vals.length - 1]) vals.pop(); // drop empty trailing tiers
+ return vals.length ? vals : null;
+}
+
+$("add-ann").addEventListener("click", () => {
+ const key = $("ann-key").value.trim();
+ if (!key) { $("track-readout").textContent = "a label key is required"; return; }
+ const t = tiers(["t1", "t2", "t3", "t4"]);
+ const ann = { key, salience: +$("ann-salience").value, _tiers: t || key };
+ if (pendingTrack && pendingTrack.track) {
+ ann.track = pendingTrack.track;
+ ann.appear = pendingTrack.appear; ann.disappear = pendingTrack.disappear;
+ } else if (pendingTrack && pendingTrack.static) {
+ ann.box = pendingTrack.box;
+ } else if (seedBox) {
+ ann.box = seedBox.slice();
+ } else { $("track-readout").textContent = "draw + (optionally) track a box first"; return; }
+ annotations.push(ann);
+ seedBox = null; pendingTrack = null; $("add-ann").disabled = true;
+ renderLists(); renderDrawLayer();
+});
+
+$("add-aff").addEventListener("click", () => {
+ const key = $("aff-key").value.trim();
+ if (!key || !affectPoint) return;
+ const t = tiers(["a1", "a2", "a3", "a4"]);
+ affect.push({ key, at: affectPoint.slice(), min_level: +$("aff-min").value, _tiers: t || key });
+ affectPoint = null; $("add-aff").disabled = true;
+ renderLists(); renderDrawLayer();
+});
+
+function renderLists() {
+ const al = $("ann-list"); al.innerHTML = "";
+ annotations.forEach((a, i) => {
+ const li = document.createElement("li");
+ const kind = a.track ? `track ${a.track.length}kf ${mono(a.appear)}–${mono(a.disappear)}` : "static";
+ const tip = Array.isArray(a._tiers) ? a._tiers.join(" → ") : a._tiers;
+ li.textContent = `${a.key} (s${a.salience}, ${kind}) — ${tip}`;
+ li.appendChild(rm(() => { annotations.splice(i, 1); renderLists(); }));
+ al.appendChild(li);
+ });
+ const fl = $("aff-list"); fl.innerHTML = "";
+ affect.forEach((f, i) => {
+ const li = document.createElement("li");
+ const tip = Array.isArray(f._tiers) ? f._tiers.join(" → ") : f._tiers;
+ li.textContent = `${f.key} (min ${f.min_level}) — ${tip}`;
+ li.appendChild(rm(() => { affect.splice(i, 1); renderLists(); }));
+ fl.appendChild(li);
+ });
+}
+function rm(fn) {
+ const b = document.createElement("button");
+ b.textContent = "✕"; b.className = "rm"; b.onclick = fn;
+ return b;
+}
+
+// Load whatever is already authored for this clip into the editor.
+$("load-existing").addEventListener("click", () => {
+ if (!currentClip) return;
+ const strings = (currentClip.strings && currentClip.strings.en) || {};
+ annotations = (currentClip.annotations || []).map((a) => ({ ...a, _tiers: strings[a.key] || a.key }));
+ affect = (currentClip.affect || []).map((f) => ({ ...f, _tiers: strings[f.key] || f.key }));
+ renderLists();
+});
+
+// Assemble strings.en from each entry's _tiers and POST to the manifest.
+$("save").addEventListener("click", async () => {
+ if (!currentClip) return;
+ const en = {};
+ const strip = (x) => { const { _tiers, ...rest } = x; return rest; };
+ const anns = annotations.map((a) => { en[a.key] = a._tiers; return strip(a); });
+ const affs = affect.map((f) => { en[f.key] = f._tiers; return strip(f); });
+ const resp = await fetch("/api/author/clip", {
+ method: "POST", headers: { "content-type": "application/json" },
+ body: JSON.stringify({ clip_id: currentClip.id, annotations: anns, affect: affs, strings: { en } }),
+ });
+ $("save-readout").textContent = resp.ok
+ ? `saved ${anns.length} labels + ${affs.length} emotions to the manifest ✓`
+ : "save error " + resp.status;
+ if (resp.ok) { // refresh the in-memory clip copy
+ clips = (await (await fetch("/api/clips")).json()).clips || [];
+ currentClip = clips.find((c) => c.id === currentClip.id) || currentClip;
+ }
+});
+
+$("clip").addEventListener("change", (e) => selectClip(e.target.value));
+
+async function main() {
+ await loadClips();
+ requestAnimationFrame(previewLoop);
+}
+main();
diff --git a/tests/test_pipeline_integration.py b/tests/test_pipeline_integration.py
index 6cd2b35..d8a8b75 100644
--- a/tests/test_pipeline_integration.py
+++ b/tests/test_pipeline_integration.py
@@ -4,7 +4,9 @@ import pytest
from tools.pipeline.run import probe_duration, process_clip, resolve_ffmpeg
-_FOREST = Path("simulator/sample_media/forest/base.mp4")
+# A real pool base (forest was retired with the rotating-pool rename, session 0016);
+# cosmos/base.mp4 is the cosmos pool primary and the most stable PD clip on disk.
+_BASE = Path("simulator/sample_media/cosmos/base.mp4")
def _ffmpeg_available() -> bool:
@@ -18,20 +20,18 @@ def _ffmpeg_available() -> bool:
_HAS_FF = _ffmpeg_available()
-@pytest.mark.skipif(not _FOREST.exists(),
- reason="forest POC base absent (run simulator/setup_sample_media.py)")
-def test_probe_duration_reads_forest_base():
- if not _FOREST.exists():
- pytest.skip("forest POC base absent")
- assert probe_duration(_FOREST) > 0
+@pytest.mark.skipif(not _BASE.exists(),
+ reason="cosmos base absent (run simulator/build_pool_manifest.py)")
+def test_probe_duration_reads_real_base():
+ assert probe_duration(_BASE) > 0
@pytest.mark.skipif(not _HAS_FF,
reason="neither system ffmpeg nor imageio-ffmpeg available")
-@pytest.mark.skipif(not _FOREST.exists(),
- reason="forest POC base absent (run simulator/setup_sample_media.py)")
+@pytest.mark.skipif(not _BASE.exists(),
+ reason="cosmos base absent (run simulator/build_pool_manifest.py)")
def test_process_clip_emits_proxy_and_master(tmp_path):
- out = process_clip(_FOREST, tmp_path, overlap=0.5, start=0.0, duration=4.0)
+ out = process_clip(_BASE, tmp_path, overlap=0.5, start=0.0, duration=4.0)
assert out["proxy"].exists() and out["master"].exists()
# Proxy is exactly 1920×1080 — verified with cv2, no ffprobe dependency.
import cv2
diff --git a/tests/test_pipeline_track.py b/tests/test_pipeline_track.py
new file mode 100644
index 0000000..6b09b0d
--- /dev/null
+++ b/tests/test_pipeline_track.py
@@ -0,0 +1,120 @@
+"""tools/pipeline/track.py — the hybrid motion-track pass (content-pipeline §11.5).
+
+Pure geometry helpers are unit-tested here; the cv2 optical-flow propagation is an
+opt-in integration test that runs on a real pool clip when present."""
+
+from pathlib import Path
+
+import pytest
+
+from tools.pipeline.track import (
+ clamp01_box,
+ denormalize_box,
+ infer_window,
+ loop_t,
+ normalize_box,
+ sample_track,
+ track_to_annotation,
+)
+
+
+# --- pure helpers -----------------------------------------------------------
+
+
+def test_clamp01_box_keeps_box_on_screen():
+ assert clamp01_box((-0.2, 0.1, 0.5, 0.5)) == [0.0, 0.1, 0.5, 0.5]
+ # size trimmed so the box never runs past the right/bottom edge
+ assert clamp01_box((0.8, 0.8, 0.5, 0.5)) == [0.8, 0.8, pytest.approx(0.2), pytest.approx(0.2)]
+
+
+def test_normalize_denormalize_round_trip():
+ px = (192, 108, 384, 216)
+ norm = normalize_box(px, 1920, 1080)
+ assert norm == [pytest.approx(0.1), pytest.approx(0.1), pytest.approx(0.2), pytest.approx(0.2)]
+ assert denormalize_box(norm, 1920, 1080) == (192, 108, 384, 216)
+
+
+def test_loop_t_is_frame_over_total():
+ assert loop_t(0, 100) == 0.0
+ assert loop_t(50, 100) == 0.5
+ assert loop_t(5, 0) == 0.0 # guard divide-by-zero
+
+
+def test_infer_window_spans_tracked_frames():
+ appear, disappear = infer_window([10, 11, 30, 31], 100)
+ assert appear == pytest.approx(0.1)
+ assert disappear == pytest.approx(0.31)
+ # no frames -> full clip
+ assert infer_window([], 100) == (0.0, 1.0)
+
+
+def test_sample_track_downsamples_to_sparse_keyframes():
+ # 21 dense frames -> at most n_keyframes, including the first and last
+ per_frame = {i: [i / 100, 0.2, 0.1, 0.1] for i in range(0, 21)}
+ track = sample_track(per_frame, total_frames=100, n_keyframes=5)
+ assert len(track) <= 5
+ ts = [k["t"] for k in track]
+ assert ts == sorted(ts) # sorted by t
+ assert ts[0] == pytest.approx(0.0) # first tracked frame
+ assert ts[-1] == pytest.approx(0.2) # last tracked frame (frame 20 / 100)
+ assert len(set(ts)) == len(ts) # de-duplicated on t
+
+
+def test_sample_track_keeps_all_when_already_sparse():
+ per_frame = {0: [0.1, 0.1, 0.1, 0.1], 60: [0.5, 0.3, 0.1, 0.1]}
+ track = sample_track(per_frame, total_frames=120, n_keyframes=5)
+ assert [k["t"] for k in track] == [pytest.approx(0.0), pytest.approx(0.5)]
+
+
+def test_sample_track_empty_is_empty():
+ assert sample_track({}, total_frames=100) == []
+
+
+def test_track_to_annotation_assembles_geometry_only():
+ track = [{"t": 0.0, "box": [0.1, 0.2, 0.1, 0.1]}, {"t": 0.5, "box": [0.4, 0.2, 0.1, 0.1]}]
+ ann = track_to_annotation("detected.jelly", track, salience=3, appear=0.0, disappear=0.55)
+ assert ann["key"] == "detected.jelly"
+ assert ann["salience"] == 3
+ assert ann["track"] == track
+ assert ann["appear"] == 0.0 and ann["disappear"] == 0.55
+ # no semantics leaked in — strings/tiers are the author's, added elsewhere
+ assert "strings" not in ann and "_tiers" not in ann
+
+
+def test_track_to_annotation_omits_window_when_absent():
+ ann = track_to_annotation("detected.x", [], salience=4)
+ assert "appear" not in ann and "disappear" not in ann
+
+
+# --- opt-in: classical optical-flow propagation on a real clip --------------
+
+_CLIP = Path("simulator/sample_media/abyss_wow/base.mp4")
+
+
+def _cv2_available() -> bool:
+ try:
+ import cv2 # noqa: F401
+ return True
+ except Exception:
+ return False
+
+
+@pytest.mark.skipif(not _CLIP.exists(),
+ reason="abyss_wow base absent (run simulator/build_pool_manifest.py)")
+@pytest.mark.skipif(not _cv2_available(), reason="opencv-python not available")
+def test_track_seed_produces_a_sparse_annotation_on_real_footage():
+ from tools.pipeline.track import track_box, track_seed
+
+ seed = (0.2, 0.3, 0.15, 0.18)
+ per_frame, total = track_box(_CLIP, seed, seed_frame=0, max_frames=40)
+ assert total > 0
+ assert per_frame[0] == pytest.approx(list(seed), abs=0.02)
+ assert len(per_frame) > 1 # propagated beyond the seed frame
+
+ ann = track_seed(_CLIP, "detected.jelly", seed, seed_frame=0, n_keyframes=5, max_frames=40)
+ assert ann["key"] == "detected.jelly"
+ assert 2 <= len(ann["track"]) <= 5 # sparse keyframes
+ for kf in ann["track"]:
+ assert 0.0 <= kf["t"] <= 1.0
+ assert all(0.0 <= v <= 1.0 for v in kf["box"])
+ assert 0.0 <= ann["appear"] <= ann["disappear"] <= 1.0
diff --git a/tests/test_simulator_api.py b/tests/test_simulator_api.py
index fcd227a..9acdeb8 100644
--- a/tests/test_simulator_api.py
+++ b/tests/test_simulator_api.py
@@ -250,3 +250,78 @@ def test_delta_zero_is_an_initial_pool_pick(pool_client):
assert data["to_index"] == 1
assert data["steps"] == []
assert data["target_clip_id"] in members
+
+
+# --- author mode (content-pipeline §11.5) ---
+
+
+@pytest.fixture
+def author_client(tmp_path):
+ p = tmp_path / "manifest.json"
+ p.write_text(json.dumps({
+ "clips": [{
+ "id": "abyss_wow", "title": "World of Water", "base_file": "abyss_wow/base.mp4",
+ "license": "PD", "source": "NOAA", "right_variants": {},
+ "annotations": [], "affect": [], "strings": {"en": {}},
+ }],
+ }))
+ return TestClient(create_app(manifest_path=p)), p
+
+
+def test_author_clip_persists_labels_and_keeps_provenance(author_client):
+ client, path = author_client
+ body = {
+ "clip_id": "abyss_wow",
+ "annotations": [{"key": "detected.jelly", "salience": 4,
+ "appear": 0.0, "disappear": 0.5,
+ "track": [{"t": 0.0, "box": [0.1, 0.2, 0.1, 0.1]}]}],
+ "affect": [{"key": "feel.unease", "at": [0.5, 0.5], "min_level": 1}],
+ "strings": {"en": {"detected.jelly": ["jelly", "comb jelly", "Ctenophora", "Ctenophora · cilia rows"],
+ "feel.unease": ["uh", "unease", "disquiet", "a creeping disquiet"]}},
+ }
+ resp = client.post("/api/author/clip", json=body)
+ assert resp.status_code == 200 and resp.json()["ok"] is True
+ # persisted to disk
+ saved = json.loads(path.read_text())["clips"][0]
+ assert saved["annotations"][0]["key"] == "detected.jelly"
+ assert saved["affect"][0]["key"] == "feel.unease"
+ assert saved["strings"]["en"]["detected.jelly"][2] == "Ctenophora"
+ # provenance preserved (only authored content replaced)
+ assert saved["base_file"] == "abyss_wow/base.mp4"
+ assert saved["license"] == "PD"
+ # reflected by /api/clips without a restart
+ served = client.get("/api/clips").json()["clips"][0]
+ assert served["annotations"][0]["key"] == "detected.jelly"
+
+
+def test_author_clip_unknown_clip_404(author_client):
+ client, _ = author_client
+ resp = client.post("/api/author/clip", json={"clip_id": "nope", "annotations": []})
+ assert resp.status_code == 404
+
+
+def test_author_track_unknown_clip_404(author_client):
+ client, _ = author_client
+ resp = client.post("/api/author/track", json={
+ "clip_id": "nope", "key": "detected.x", "seed_box": [0.1, 0.1, 0.1, 0.1]})
+ assert resp.status_code == 404
+
+
+_REAL_CLIP = __import__("pathlib").Path("simulator/sample_media/abyss_wow/base.mp4")
+
+
+@pytest.mark.skipif(not _REAL_CLIP.exists(),
+ reason="abyss_wow base absent (run simulator/build_pool_manifest.py)")
+def test_author_track_on_real_clip_returns_keyframes():
+ # Against the real default manifest + footage: the tracker yields a sparse track.
+ client = TestClient(create_app())
+ resp = client.post("/api/author/track", json={
+ "clip_id": "abyss_wow", "key": "detected.jelly",
+ "seed_box": [0.2, 0.3, 0.15, 0.18], "seed_t": 0.0,
+ "salience": 4, "n_keyframes": 5, "max_frames": 30,
+ })
+ assert resp.status_code == 200
+ ann = resp.json()
+ assert ann["key"] == "detected.jelly"
+ assert 2 <= len(ann["track"]) <= 5
+ assert 0.0 <= ann["appear"] <= ann["disappear"] <= 1.0
diff --git a/tools/pipeline/track.py b/tools/pipeline/track.py
new file mode 100644
index 0000000..a93af46
--- /dev/null
+++ b/tools/pipeline/track.py
@@ -0,0 +1,297 @@
+"""Stage 5 geometry — the hybrid motion-track pass (content-pipeline §3.5 / §11.5).
+
+Turns a hand-seeded box (or an ML detection) on one keyframe into the sparse,
+loop-normalized keyframed `track` the manifest stores (design §4) plus an
+appear/disappear window (§11.2). SEMANTICS are never produced here — the author
+owns the `key`, strings, `salience` and tiers; this module only propagates BOX
+GEOMETRY.
+
+Two geometry paths, same artifact:
+ - **classical** (always available): a hand-seeded box propagated by OpenCV
+ Lucas-Kanade optical flow (a CSRT tracker is used instead when a contrib
+ build provides one). Translation model — right for calm drifting subjects
+ (the abyss/reef creatures). Re-seeds features when too few survive.
+ - **ML detect+track** (optional, lazy-imported): a YOLO+tracker pass proposing
+ boxes the author maps to a key. Imported only on demand so the base install
+ needs no torch/ultralytics; a clear ImportError tells you what to `pip
+ install` if you ask for it without it present.
+
+The PURE helpers (normalization, keyframe sampling, window inference, annotation
+assembly) are unit-tested; the cv2/ML propagation is covered by the opt-in
+integration test on real footage.
+"""
+
+from __future__ import annotations
+
+from typing import Optional
+
+Box = tuple[float, float, float, float] # (x, y, w, h)
+
+
+# --- pure geometry helpers --------------------------------------------------
+
+def clamp01_box(box: Box) -> list[float]:
+ """Clamp a normalized box into the visible [0,1] frame, keeping it on-screen
+ (origin clamped, then size trimmed so x+w<=1, y+h<=1)."""
+ x, y, w, h = box
+ x = min(max(x, 0.0), 1.0)
+ y = min(max(y, 0.0), 1.0)
+ w = min(max(w, 0.0), 1.0 - x)
+ h = min(max(h, 0.0), 1.0 - y)
+ return [x, y, w, h]
+
+
+def normalize_box(box_px: Box, width: int, height: int) -> list[float]:
+ """Pixel box -> normalized [0,1] box (the manifest's resolution-independent
+ form). Clamped on-screen."""
+ x, y, w, h = box_px
+ return clamp01_box((x / width, y / height, w / width, h / height))
+
+
+def denormalize_box(box_norm: Box, width: int, height: int) -> tuple[int, int, int, int]:
+ """Normalized [0,1] box -> integer pixel box for the cv2 tracker."""
+ x, y, w, h = box_norm
+ return (round(x * width), round(y * height), round(w * width), round(h * height))
+
+
+def loop_t(frame_idx: int, total_frames: int) -> float:
+ """Loop-normalized time of a frame in [0,1), matching the client's
+ `currentTime / duration` clock (design §4)."""
+ if total_frames <= 0:
+ return 0.0
+ return frame_idx / total_frames
+
+
+def infer_window(tracked_frames: list[int], total_frames: int) -> tuple[float, float]:
+ """The appear/disappear window (§11.2) covering the frames where a box was
+ tracked, as loop-normalized t. The first/last tracked frame bound it."""
+ if not tracked_frames:
+ return (0.0, 1.0)
+ lo, hi = min(tracked_frames), max(tracked_frames)
+ return (loop_t(lo, total_frames), loop_t(hi, total_frames))
+
+
+def sample_track(
+ per_frame: dict[int, Box], total_frames: int, n_keyframes: int = 5
+) -> list[dict]:
+ """Down-sample a dense per-frame box map to a SPARSE keyframed track
+ (design §4: sparse keyframes interpolated at runtime, never dense per-frame).
+
+ Picks `n_keyframes` frames evenly across the tracked span (always including
+ the first and last tracked frame) and emits `[{t, box}, ...]` sorted by t,
+ de-duplicated on t. Pure — takes already-normalized boxes."""
+ if not per_frame:
+ return []
+ frames = sorted(per_frame)
+ n = max(2, n_keyframes)
+ if len(frames) <= n:
+ picks = frames
+ else:
+ lo, hi = frames[0], frames[-1]
+ step = (hi - lo) / (n - 1)
+ targets = [lo + round(step * i) for i in range(n)]
+ # snap each target to the nearest actually-tracked frame
+ picks = sorted({min(frames, key=lambda f: abs(f - t)) for t in targets})
+ out, seen = [], set()
+ for f in picks:
+ t = round(loop_t(f, total_frames), 4)
+ if t in seen:
+ continue
+ seen.add(t)
+ out.append({"t": t, "box": clamp01_box(per_frame[f])})
+ return out
+
+
+def track_to_annotation(
+ key: str,
+ track: list[dict],
+ *,
+ salience: int = 4,
+ appear: Optional[float] = None,
+ disappear: Optional[float] = None,
+) -> dict:
+ """Assemble a manifest annotation from a propagated track + the author's
+ semantics. Geometry (track + window) comes from this module; `key`,
+ `salience` and the tiered strings are the author's (added separately)."""
+ ann: dict = {"key": key, "salience": salience, "track": track}
+ if appear is not None:
+ ann["appear"] = round(appear, 4)
+ if disappear is not None:
+ ann["disappear"] = round(disappear, 4)
+ return ann
+
+
+# --- classical propagation (cv2; covered by the opt-in integration test) -----
+
+def _new_csrt():
+ """A CSRT tracker if this OpenCV build ships one (contrib), else None — then
+ the optical-flow path is used. Kept tiny so the import stays lazy."""
+ import cv2
+ for factory in ("TrackerCSRT_create",):
+ if hasattr(cv2, factory):
+ return getattr(cv2, factory)()
+ legacy = getattr(cv2, "legacy", None)
+ if legacy is not None and hasattr(legacy, "TrackerCSRT_create"):
+ return legacy.TrackerCSRT_create()
+ return None
+
+
+def _features_in_box(gray, box_px, max_corners: int = 60):
+ """Good-feature points inside a pixel box, in full-frame coords (for LK)."""
+ import cv2
+ import numpy as np
+
+ x, y, w, h = (int(v) for v in box_px)
+ h_img, w_img = gray.shape[:2]
+ x0, y0 = max(x, 0), max(y, 0)
+ x1, y1 = min(x + w, w_img), min(y + h, h_img)
+ if x1 - x0 < 4 or y1 - y0 < 4:
+ return None
+ roi = gray[y0:y1, x0:x1]
+ pts = cv2.goodFeaturesToTrack(roi, max_corners, 0.01, 5)
+ if pts is None:
+ return None
+ pts = pts.reshape(-1, 2) + np.array([x0, y0], dtype="float32")
+ return pts.reshape(-1, 1, 2)
+
+
+def track_box(
+ video_path,
+ seed_box_norm: Box,
+ *,
+ seed_frame: int = 0,
+ max_frames: Optional[int] = None,
+ min_features: int = 8,
+) -> tuple[dict[int, list[float]], int]:
+ """Propagate a hand-seeded normalized box FORWARD from `seed_frame` with
+ optical flow (or CSRT when available). Returns `(per_frame_norm_boxes,
+ total_frames)` — feed `per_frame` to `sample_track`. Translation model: the
+ box follows the median displacement of features inside it, re-seeding when too
+ few survive. Impure (reads the video) — opt-in integration test."""
+ import cv2
+ import numpy as np
+
+ cap = cv2.VideoCapture(str(video_path))
+ total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
+ if seed_frame:
+ cap.set(cv2.CAP_PROP_POS_FRAMES, seed_frame)
+ ok, frame = cap.read()
+ if not ok:
+ cap.release()
+ raise ValueError(f"could not read frame {seed_frame} of {video_path}")
+ h_img, w_img = frame.shape[:2]
+ box = list(denormalize_box(seed_box_norm, w_img, h_img)) # px [x,y,w,h]
+
+ csrt = _new_csrt()
+ if csrt is not None:
+ csrt.init(frame, tuple(int(v) for v in box))
+
+ prev_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
+ pts = None if csrt is not None else _features_in_box(prev_gray, box)
+ out: dict[int, list[float]] = {seed_frame: normalize_box(box, w_img, h_img)}
+
+ fidx = seed_frame
+ while True:
+ ok, frame = cap.read()
+ if not ok:
+ break
+ fidx += 1
+ if csrt is not None:
+ ok2, b = csrt.update(frame)
+ if ok2:
+ box = [b[0], b[1], b[2], b[3]]
+ else:
+ gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
+ if pts is None or len(pts) < min_features:
+ pts = _features_in_box(prev_gray, box)
+ if pts is not None and len(pts):
+ nxt, status, _ = cv2.calcOpticalFlowPyrLK(prev_gray, gray, pts, None)
+ if nxt is not None:
+ good_old = pts[status == 1]
+ good_new = nxt[status == 1]
+ if len(good_new) >= 3:
+ dx = float(np.median(good_new[:, 0] - good_old[:, 0]))
+ dy = float(np.median(good_new[:, 1] - good_old[:, 1]))
+ box[0] += dx
+ box[1] += dy
+ pts = good_new.reshape(-1, 1, 2)
+ else:
+ pts = None
+ prev_gray = gray
+ out[fidx] = normalize_box(box, w_img, h_img)
+ if max_frames is not None and fidx - seed_frame >= max_frames:
+ break
+ cap.release()
+ return out, total
+
+
+def track_seed(
+ video_path,
+ key: str,
+ seed_box_norm: Box,
+ *,
+ seed_frame: int = 0,
+ salience: int = 4,
+ n_keyframes: int = 5,
+ max_frames: Optional[int] = None,
+) -> dict:
+ """End-to-end classical pass: seed -> propagate -> sample -> annotation
+ (geometry only; the author adds strings/tiers). Impure (reads the video)."""
+ per_frame, total = track_box(
+ video_path, seed_box_norm, seed_frame=seed_frame, max_frames=max_frames
+ )
+ track = sample_track(per_frame, total, n_keyframes=n_keyframes)
+ appear, disappear = infer_window(list(per_frame), total)
+ return track_to_annotation(
+ key, track, salience=salience, appear=appear, disappear=disappear
+ )
+
+
+# --- optional ML detect+track (lazy-imported) --------------------------------
+
+def detect_and_track(video_path, *, model: str = "yolov8n.pt", classes=None) -> list[dict]:
+ """Optional ML path (content-pipeline §3.5): run a detect+track model and
+ return candidate tracks `[{track_id, label, salience, track:[{t,box}],
+ appear, disappear}]` for the author to map to keys. Heavy deps are
+ lazy-imported so the base install stays light.
+
+ Raises a clear ImportError naming the install if `ultralytics` is absent —
+ this path is opt-in by design; the classical `track_seed` needs only cv2."""
+ try:
+ from ultralytics import YOLO # noqa: F401
+ except ImportError as e: # pragma: no cover - exercised only without the dep
+ raise ImportError(
+ "the ML detect+track path needs ultralytics — "
+ "`pip install ultralytics` (optional; the classical track_seed path "
+ "uses only opencv-python)."
+ ) from e
+ import cv2 # noqa: F401 (pragma: no cover below — needs the real model)
+
+ yolo = YOLO(model) # pragma: no cover
+ cap = cv2.VideoCapture(str(video_path)) # pragma: no cover
+ total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0) # pragma: no cover
+ cap.release() # pragma: no cover
+ per_track: dict[int, dict[int, list[float]]] = {} # pragma: no cover
+ labels: dict[int, str] = {} # pragma: no cover
+ for fidx, res in enumerate( # pragma: no cover
+ yolo.track(source=str(video_path), persist=True, classes=classes, stream=True)
+ ):
+ if res.boxes is None or res.boxes.id is None: # pragma: no cover
+ continue
+ w_img, h_img = res.orig_shape[1], res.orig_shape[0] # pragma: no cover
+ for box, tid, cls in zip( # pragma: no cover
+ res.boxes.xywh.tolist(), res.boxes.id.tolist(), res.boxes.cls.tolist()
+ ):
+ cx, cy, bw, bh = box # pragma: no cover
+ nb = normalize_box((cx - bw / 2, cy - bh / 2, bw, bh), w_img, h_img)
+ per_track.setdefault(int(tid), {})[fidx] = nb # pragma: no cover
+ labels[int(tid)] = yolo.names.get(int(cls), str(cls))
+ out = [] # pragma: no cover
+ for tid, frames in per_track.items(): # pragma: no cover
+ track = sample_track(frames, total) # pragma: no cover
+ appear, disappear = infer_window(list(frames), total) # pragma: no cover
+ out.append({ # pragma: no cover
+ "track_id": tid, "label": labels.get(tid, ""), "salience": 4,
+ "track": track, "appear": round(appear, 4), "disappear": round(disappear, 4),
+ })
+ return out # pragma: no cover