Compare commits

...

27 Commits

Author SHA1 Message Date
BenStullsBets 2eb752b5bc fix(sim): content-hash media URLs + fade pool-landing clip swap
Two fixes to the simulator's media handling, both surfaced by the cosmos clip swap.

1. Permanent cache-bust via content hash. New `GET /api/media-versions` returns a
   short sha1 token per served file (clip bases + ring transitions + reverses),
   cached by (mtime, size) so a re-bake is picked up without a restart. The client
   fetches it at boot and appends `?v=<hash>` to every /media URL, so a clip
   re-baked under the same path (e.g. a re-sourced cosmos base) gets a fresh URL
   the browser can't serve stale — fixing the recurring stale-clip problem at the
   root (supersedes the `cache: "reload"` belt-and-suspenders, which stays).

2. Fade the pool-landing swap. The baked ring transition zooms into the scale
   PRIMARY, but the rotating pool picks a random member on landing — so a
   non-primary pick hard-cut from the primary to the chosen clip (e.g. coast
   birdrock -> surfgrass, the artifact the operator saw). On landing, when the pick
   differs from the primary, mask the swap behind a short fade through the existing
   #black overlay (CSS opacity transition); same-clip landings still do a plain
   (re)load. General across all pooled scales.

271 passed, 2 skipped (+2: media-version content-hash + absent-file omission).
app.js syntax-checked; endpoint verified live (29 files; token matches a manual
sha1 of the bytes).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 13:03:56 -07:00
benstull aa56dfe145 Merge pull request 'fix(sim): bust HTTP cache on media preload so re-baked clips show' (#22) from fix/preload-cache-bust into main 2026-06-25 19:40:50 +00:00
BenStullsBets ed267c554c fix(sim): bust HTTP cache on media preload so re-baked clips show
The preload fetched each clip with the default HTTP cache, so a clip re-baked
under the same path (e.g. the cosmos base re-sourced to the Webb Cosmic Cliffs
flythrough) kept showing stale footage after a reload — worst case forever, when
a pre-`no-cache` build had pinned it `immutable, max-age=1y` (a plain reload never
revalidates an immutable entry). Fetch the preload with `cache: "reload"`, which
bypasses the HTTP cache, pulls fresh bytes, and updates the cache entry; the
in-memory blob cache still gives instant in-session altitude swaps. One fetch per
(re)load, so no steady-state cost.

269 passed, 2 skipped. app.js syntax-checked.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 12:40:34 -07:00
benstull 7da7446740 Merge pull request 'feat(sim): real ring zoom transitions + Dev Mode + cosmos → Webb Cosmic Cliffs' (#21) from feature/ring-transitions-real into main 2026-06-25 19:31:22 +00:00
BenStullsBets d24f829276 feat(content): swap cosmos primary to the Webb "Cosmic Cliffs" flythrough
Operator found the Orion SVS 30957 flythrough merely good and asked for a more
dramatic nebula. Replaces the cosmos-scale primary with STScI's "Exploring the
Cosmic Cliffs in 3D" (Carina Nebula, Webb NIRCam — NASA SVS 31348), sourced from
its 4K master (Clifs-3d-STScI.mp4, 3840x2160) via the content pipeline:
trim 10-34s (clears the title card ~8s and end-credit card ~102s) + crossfade-loop
-> 4K master + supersampled 1080p base (5.5 Mbps, crisper than the old 2.9 Mbps
1080p-native Orion encode).

- build_pool_manifest META: cosmos -> SVS 31348 Cosmic Cliffs, new CCBY_WEBB
  credit (NASA, ESA, CSA, STScI). Labels retuned for the Carina framing
  (emission nebula / protostar; redshift -> distance ~7,500 ly).
- Regenerated manifest.json (also corrects stale JPL-Orion provenance left when
  129bb23 updated META but not the manifest) + all ring-edge transitions from the
  new base (cosmos-orbit / abyss-cosmos now zoom from the Cosmic Cliffs).
- docs/content-candidate-pool.md: record the swap + history.

269 passed, 2 skipped. New base verified text-free + full-frame by-eye across the
loop; final look to be confirmed by the operator (no Chrome on the box).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 10:48:44 -07:00
BenStullsBets de33cbe86d claim human-experience-filter-art session 0018 (placeholder) + sessions.json entry 2026-06-25 09:43:05 -07:00
BenStullsBets 129bb23cb9 feat(content): swap cosmos Orion to a text-free SVS flythrough
Replaces the JPL "Orion: Dust and Death" primary (captioned throughout — we'd
cropped it, losing framing) with STScI's "Flight Through the Orion Nebula"
(visible light, NASA SVS 30957) — a full-frame, text-free flythrough of the same
nebula. Sourced via the content pipeline (trim 10–34s + crossfade-loop → 1080p
master + base); cosmos-orbit / abyss-cosmos transitions regenerated from it.

- build_pool_manifest META: cosmos -> SVS 30957, license CC-BY-class (NASA/ESA/
  STScI), new trim window.
- Remove the now-obsolete decaption_cosmos() crop helper (the new source needs
  no caption removal).
- docs/content-candidate-pool.md: record the source swap; drop the crop note.

269 passed, 2 skipped. New clip verified text-free + full-frame by-eye;
final look to be confirmed by the operator.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 09:28:10 -07:00
BenStullsBets a2ef792f0f fix(sim): revalidate /media instead of pinning it immutable
The `immutable, max-age=1y` media header pinned stale footage in the browser for
a year, so a re-sourced or re-cropped clip (e.g. the de-captioned cosmos) never
showed without a manual cache clear. Switch /media to `no-cache` (store, but
revalidate every load): instant altitude swaps already come from the in-memory
blob preload, so the HTTP cache only matters on (re)load — where a conditional
request is a cheap 304 when unchanged but picks up edited clips immediately.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 09:20:14 -07:00
BenStullsBets 53cb89b4c6 fix(content): crop NASA burned-in captions off the cosmos Orion clip
The cosmos primary is NASA/JPL's "Orion: Dust and Death" — an annotated
visualization with lower-third captions ("DUST FILAMENT", "BLUE BUBBLE", …)
burned in throughout, so trimming can't avoid them (only ~4s of our window is
caption-free). The candidate-pool doc wrongly called it text-free.

- build_pool_manifest.decaption_cosmos(): crops the cosmos base/master to the
  top 66% of frame height (16:9 preserved, center-x, ~1.5x tighter framing),
  re-fit to 1920x1080 — the recorded, reproducible form of the fix (media is
  gitignored). Applied in place; re-ran generate_media() so the cosmos-orbit /
  abyss-cosmos transitions use the cropped base.
- docs/content-candidate-pool.md: correct the "text-free" note; document the crop.

269 passed, 2 skipped. De-caption verified by-eye on both caption moments;
final look to be confirmed by the operator.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 09:16:57 -07:00
BenStullsBets 0201690ce0 feat(sim): real zoom transitions on every ring edge, both directions
The cosmos→orbit, reef→abyss and abyss→cosmos transition clips were stale
leftovers from an earlier 5-scale ring, built against old/placeholder footage —
so those edges flashed placeholder frames mid-zoom. Only orbit-coast and
coast-reef had been rebuilt from the current real bases.

- build_pool_manifest.generate_media(): now rebuilds EVERY ring edge from the
  current real primary pool bases (the orbit-coast recipe applied uniformly),
  overwriting the stale clips, and bakes a reversed companion (<edge>.rev.mp4)
  per edge via an ffmpeg `reverse` pass.
- app.js: a `reversed` step (an ascending / zoom-OUT ring move) now plays the
  baked .rev clip instead of the forward zoom-in, so going up recedes from the
  current scale. Both directions are added to the in-memory preload set.

Media is gitignored (generated locally / on deploy); regenerated all 10 clips
locally. 269 passed, 2 skipped; node --check clean; clips serve 200.
Visual result to be verified by-eye by the operator.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 08:34:50 -07:00
BenStullsBets 0928988abc fix(sim): collapse Dev Mode panel when disabled
The generic .hidden { display:none } is defined earlier in the stylesheet than
.dev-panel { display:flex }; equal specificity meant source order won and the
panel stayed visible with the toggle off. A two-class .dev-panel.hidden rule
wins, so all dev controls/data truly hide when Dev Mode is disabled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 08:26:10 -07:00
BenStullsBets a1c9c5c760 feat(sim): Dev Mode — pool clip picker + live analysis under one toggle
Adds a single "Dev Mode" switch at the bottom of the control panel (off by
default, persisted in localStorage). When on, a panel appears below it with all
dev controls and data; when off the default view is just the experience + the
three control fieldsets.

Dev panel:
- Pool picker: a <select> of the current altitude's pool members — choose any
  clip to override the random landing pick; a "Re-roll random" button re-runs
  the server-canonical pick. Rebuilds on each landing.
- Active clip metadata (id, title, source, license, base_file).
- Annotations & affect: keys with salience/window/tier counts (read-only;
  editing stays in /author.html).
- RenderPlan JSON — relocated here from the always-visible panel.
- Cache & playback: preload status (N/total), memory-vs-network for the active
  clip, live resolution/duration/loop position.

Frontend only — no API changes; everything is read from data the renderer
already has. Factored pickRandomMember() out of landScale() so the initial
landing, ring advances, and the re-roll button share one pick.

Design: docs/superpowers/specs/2026-06-25-simulator-dev-mode-design.md
269 passed, 2 skipped; node --check + server smoke clean. Frontend behavior
to be verified by-eye by the operator (no headless browser in repo yet).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 08:19:39 -07:00
benstull ab2edf77bb Merge pull request 'perf(sim): preload all clips into memory for instant altitude swaps' (#19) from feature/preload-media-cache into main 2026-06-25 14:48:44 +00:00
BenStullsBets d19fe4e8fa perf(sim): preload all clips into memory for instant altitude swaps
Each altitude change reset the single <video> element's src and re-fetched
the scale's footage over the network, compounded by the dev no-store policy
applying to /media too — so every scale change paid a full fetch+decode.

- simulator/app.py: split the cache middleware — app shell/JS/CSS stay
  no-store (by-eye dev iteration), but /media is now cacheable
  (public, max-age, immutable) since the clips are static.
- simulator/static/app.js: preload every ring clip (19 base + 5 transition,
  ~350 MB) into in-memory blob object URLs at startup — non-blocking, the
  first scale plays immediately while the rest stream in, ordered current
  scale outward. mediaUrl() serves the cached blob when ready, else falls
  back to the network. A small progress chip self-removes when done.
- tests: assert app assets stay no-store and /media is cacheable.

269 passed, 2 skipped. Frontend verified by-eye by the operator (no headless
browser in repo yet).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 07:47:32 -07:00
BenStullsBets 9a83c0132d claim human-experience-filter-art session 0017 (placeholder) + sessions.json entry 2026-06-25 07:41:36 -07:00
BenStullsBets 7fd9aabd0e add sessions/0016/SESSION-0016.0-TRANSCRIPT-2026-06-24T17-48--2026-06-24T21-46.md + replace placeholder/variant SESSION-0016.0-TRANSCRIPT-2026-06-24T17-48--2026-06-24T18-18.md 2026-06-24 21:46:35 -07:00
benstull 7a32ff1acc Merge pull request 'feat(sim): turnable Altitude knob (replaces ring buttons)' (#18) from feature/altitude-knob into main 2026-06-25 04:45:09 +00:00
BenStullsBets 5fae4f9005 feat(sim): replace ring buttons with a turnable "Altitude" knob
The scale ring's ⊖ out / in ⊕ buttons are replaced by a circular **Altitude**
knob (the endless rotary encoder, drawn as a dial). cosmos (highest) sits at the
top; turning clockwise descends cosmos → orbit → coast → reef → abyss and past
the deepest **wraps back to the top** — the same endless ring the server already
models (advance + wrap unchanged).

- Built dynamically from `ring.scales` (labels + ticks per scale, a needle that
  points at the current scale, an ALTITUDE caption).
- Drag to turn: the rotation accumulates and commits whole detents on release, so
  a big spin becomes one fast blended pass (reuses the proven wheel/detent path);
  scroll the knob to step; tap a label to jump the shortest way around.
- Pure frontend — no engine/API change. JS syntax-checked; live server probed
  (assets serve, dial present, abyss→cosmos wrap intact). 23 sim-API tests green.
  By-eye review deferred to the operator (no Chrome on this box).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 21:44:51 -07:00
BenStullsBets 7628a5321f add sessions/0016/SESSION-0016.0-TRANSCRIPT-2026-06-24T17-48--2026-06-24T18-18.md + replace placeholder/variant SESSION-0016.0-TRANSCRIPT-2026-06-24T17-48--INPROGRESS.md 2026-06-24 18:18:13 -07:00
BenStullsBets bcd4d32b19 update sessions/0016/SESSION-0016.0-TRANSCRIPT-2026-06-24T17-48--INPROGRESS.md 2026-06-24 18:17:37 -07:00
benstull c4452396a1 Merge pull request 'tools(pipeline): hybrid track.py + simulator label author mode' (#17) from feature/content-pipeline-track-author into main 2026-06-25 01:14:59 +00:00
BenStullsBets 70fd367c70 tools(pipeline): hybrid track.py + simulator label author mode
Content-pipeline Increment 2, part 2 (the tooling). Implements design §11.5
(docs/superpowers/specs/2026-06-24-content-pipeline-design.md):

- tools/pipeline/track.py — the stage-5 geometry pass. Classical path: a
  hand-seeded normalized box propagated by OpenCV Lucas-Kanade optical flow
  (CSRT used instead when a contrib build provides it), sampled to a SPARSE
  loop-normalized keyframed track + an appear/disappear window. Optional ML
  detect+track path lazy-imports ultralytics (clear ImportError if absent;
  base install needs only cv2). Pure helpers (normalize/denormalize, loop_t,
  infer_window, sample_track, track_to_annotation) are unit-tested; the cv2
  propagation is an opt-in integration test on real abyss_wow footage.
  Semantics are never produced here — geometry only.

- Author mode — /author.html + author.js/.css reuse the preview stage: pick a
  pool clip, scrub, drag a seed box, Run tracker (or Add as static box), author
  the LEFT detail tiers (general -> scientific+fact) + salience, shift-click to
  place affect anchors with RIGHT emotion tiers, and Save to manifest. Backend:
  POST /api/author/track (runs track_seed on the clip's base) + POST
  /api/author/clip (idempotent upsert via tools.pipeline.manifest — keeps media
  + provenance, replaces only authored content, reloads in place). The tracker
  propagates box geometry; all strings/scientific names/facts are hand-typed.

- Repointed the pipeline integration test off the retired forest base to the
  cosmos pool primary. USER_GUIDE simulator section brought current (pools,
  coast, 3-knob Mood, real-time dream, progressive tiers, author mode).

267 passed / 2 skipped (+ track pure + opt-in real-footage + author endpoint
tests). Author UI by-eye review deferred to the operator (no Chrome on this box).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 18:14:32 -07:00
benstull 12408e505e Merge pull request 'content(sim): rotating clip pools + progressive tracked labels & emotions' (#16) from feature/content-pipeline-increment-2 into main 2026-06-25 01:05:58 +00:00
BenStullsBets edca8a9615 content(sim): rotating clip pools + progressive tracked labels & emotions
Content-pipeline Increment 2, part 1 (the experience). Implements design §11
(docs/superpowers/specs/2026-06-24-content-pipeline-design.md):

- Rotating pool (§11.1): each ring scale references a POOL of vetted clips
  (docs/content-candidate-pool.md); the player picks a random member on landing.
  player.ring.Scale gains `pool` + pure `pick_clip_id(scale, r)` (injected
  randomness); the pick happens at the API boundary (random.random()), a delta=0
  advance is the initial/re-roll pick. clips.py parses/exposes the pool; the
  client lands on move.target_clip_id. Back-compat: a scale with only clip_id is
  a pool of one.
- Rename ring scale forest -> coast (new land<->sea scale, orbit..reef); ring
  order cosmos -> orbit -> coast -> reef -> abyss -> wrap. Cleanup: dropped the
  leftover local media (cosmos_traverse, orbit_westcoast, old forest/reef, the
  Increment-1 orbit/abyss single bases); new coast-edge transition placeholders.
- Time-windowed tracked labels (§11.2): annotations gain appear/disappear
  (loop-normalized, wraps across the seam); a label shows only while on screen
  and its box tracks the subject. abyss + reef pools authored as test material.
- Progressive LEFT detail tiers (§11.3): per-object `salience` (first shows at
  Left 5-salience) + tiered strings (general -> specific -> scientific -> +fact)
  escalating with the Left knob; replaces flat min_level gating. Strings may be a
  tier list or a static string (back-compat). No engine change (client reads
  overlay.level).
- Progressive RIGHT emotion tiers (§11.4): affect vocabulary escalates basic ->
  compound with the Right knob (dream.strength); appearance still gated by
  min(left,right). No engine change.

New simulator/build_pool_manifest.py holds the hand-authored content and emits
the 19-clip pool manifest (the reproducible baseline). setup_scales_media.py
marked superseded. 251 passed / 4 skipped (+ ring pool + API pool/window tests).
By-eye visual review deferred to the operator (no Chrome on this box).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 18:05:13 -07:00
BenStullsBets 4270c265e3 claim human-experience-filter-art session 0016 (placeholder) + sessions.json entry 2026-06-24 17:49:11 -07:00
benstull b28f930e11 Merge pull request 'docs: operator-selected content candidate pool' (#15) from content-candidate-pool into main 2026-06-25 00:39:59 +00:00
BenStullsBets 7be4f18e77 docs: operator-selected content candidate pool (rotating-pool model)
Records the 5-scale rotating clip pools the operator selected + vetted from the
candidate gallery (19 clips: cosmos 4, orbit 3, coast 4, reef 5, abyss 3), each
strict-PD/CC-BY-class, word-free, human-free, with source URL + trim window so
they are re-sourceable (media stays gitignored). Captures: the rotating-pool model
(random clip per scale on ring landing), the forest→coast rename, the cosmos
#3-Orion vs #6-Traverse processing mix-up to fix, and the rejected/leftover clips
to clean up. Feeds the next session (Increment 2 expanded: pool wiring + tracked
progressive labels + progressive emotions).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 17:39:54 -07:00
26 changed files with 4778 additions and 291 deletions
+3 -1
View File
@@ -6,5 +6,7 @@ media/
.superpowers/
*.egg-info/
# Simulator sample media (look-tuning only; populate via setup_sample_media.py
# / setup_scales_media.py). All scale + transition binaries are gitignored.
# / build_pool_manifest.py). All scale + transition binaries are gitignored.
simulator/sample_media/**/*.mp4
# Local-only candidate review gallery (re-buildable; not shipped content).
simulator/static/review.html
+48 -37
View File
@@ -296,24 +296,17 @@ exists. (The earlier selection-era "curator's X-ray" view was retired when the
piece moved from *selecting* clips to *altering* them.)
**One-time setup — populate the sample footage** (look-tuning only; not shipped
content):
content). The ring uses real strict-PD footage in a **rotating pool** per scale
(`docs/content-candidate-pool.md`); regenerate the manifest + new transition
placeholders with:
python simulator/setup_sample_media.py # stage the forest neutral base
python -m simulator.bake_right_variants --scale forest # real Right variants (SD on MPS, ~11 min)
python simulator/setup_scales_media.py # the cosmos + abyss scales + ring transitions
python simulator/build_pool_manifest.py --media # write the pool manifest + coast-edge transitions
`setup_sample_media.py` stages the forest neutral base from the session-0008 POC
artifacts (`~/hef-poc/out/neutral.mp4`). `bake_right_variants.py` then produces
the **real** flow-stabilized painterly Right variants (strengths 14) for the
forest scale — SD img2img keyframes + Farneback optical-flow tweens on Apple MPS,
the temporally-calm POC algorithm, productionized (scales design §1/§4). The
keyframe-strength ramp per Right level is by-eye tunable in
`simulator/bake_right_variants.py`. `setup_scales_media.py` makes the scale
**ring** demonstrable: cheap synthetic placeholder bases for the two true-PD
scales (`cosmos` = NASA/Hubble, `abyss` = NOAA Ocean Exploration) plus the
per-edge zoom/warp transition clips — these scales carry a raw base only (no real
Right variants yet; the baker can extend to them once their real footage is
sourced). The `.mp4` binaries are gitignored.
`build_pool_manifest.py` holds the hand-authored label/affect content and emits
`simulator/sample_media/manifest.json` (the 19-clip pool baseline). The pool clips
themselves are sourced via the content pipeline (`tools/pipeline/run.py
process_clip`); the `.mp4` binaries are gitignored. (`setup_scales_media.py` is the
older one-base-per-scale generator, now superseded.)
**Run it (Docker):**
@@ -331,27 +324,45 @@ then open http://localhost:8000.
- **Content dial** — picks audio/video channel; "off" and audio-only positions go
to black walls.
- **Scale ring (endless encoder)** — `⊖ out` / `in ⊕` (or scroll the stage) walk a
*closed ring* of neutral "scales of nature" clips — cosmos → forest → abyss and
back around (diving past the smallest **wraps** to the largest). It is *relative*
(an endless encoder), distinct from the absolute 04 knobs: each step plays a
short placeholder zoom/warp **transition**, then settles on the next scale, with
the current knob alteration still applied. The current scale is named beside the
buttons (`name (i/N)`). A **fast spin** (scroll several detents at once, ≥3)
collapses to a single quick **blended pass** straight to the destination scale
instead of grinding through every transition (scales design §3); slow single
steps still chain one full transition per scale crossed.
- **Four experience knobs (04):**
- **Dark / Light** — a live runtime color grade (cool/dark ↔ warm/bright; equal
or zero = the raw footage).
- **Right (dreamlike)** — selects a discrete pre-baked, flow-stabilized restyle
variant and crossfades to it (strength 0 = raw base).
- **Left (analytical)** — a live overlay: labelled boxes from the clip's authored
annotation track, with more annotations appearing at higher levels. Text is
shaped live (the simulator analogue of the Pi's Pango/HarfBuzz path).
- **Calibration sliders** — adjust the grade/overlay gain curves live; once a look
is liked, bake the values into `DEFAULT_CALIBRATION` in `player/alteration.py`.
*closed ring* of neutral "scales of nature" — **cosmos → orbit → coast → reef →
abyss** and back around (diving past the smallest **wraps** to the largest). Each
scale is a **rotating pool** of vetted clips: landing on a scale plays a random
pool member, so the ring feels fresh each pass. It is *relative* (an endless
encoder), distinct from the absolute 04 knobs: each step plays a short
placeholder zoom/warp **transition**, then settles, with the current alteration
still applied. The current scale + chosen member is named beside the buttons
(`scale · member (i/N · pool N)`). A **fast spin** (≥3 detents at once) collapses
to one quick **blended pass** instead of grinding through every transition.
- **Three experience knobs:**
- **Mood (4 dark .. 0 .. +4 light)** — a live runtime color grade (cool/dark ↔
warm/bright; 0 = the raw footage).
- **Right (dreamlike, 04)** — a **deterministic real-time painterly dream** (a
WebGL Kuwahara restyle of the live frames; holds still across the loop, goes
trippy at max). It also drives **progressive emotion tiers**: when both knobs
are up, the affect words escalate from basic to compound as Right rises.
- **Left (analytical, 04)** — a live HUD of labelled boxes from the clip's
authored annotation track. Labels use **progressive detail tiers**: low Left
shows fewer, more general labels (high-salience objects only); raising Left
brings in more objects *and* escalates each label general → specific →
scientific → +fact. Time-windowed tracked labels appear only while their
subject is on screen and follow it.
- **RenderPlan readout** — always shows the exact numbers the engine produced (the
project's honesty "X-ray," now over the alteration model).
project's honesty "X-ray," over the alteration model).
The base clips, Right variants, Left annotation track, and string tables come from
The base clips, the Left annotation tracks (with tiers + appear/disappear
windows), affect anchors, and per-tier string tables all come from
`simulator/sample_media/manifest.json`.
### Authoring labels — the author mode
Open **http://localhost:8000/author.html** to author a clip's labels by eye
(content-pipeline §11.5). Pick a pool clip, **scrub** to where a subject is on
screen, **drag a box** over it, set the label key + salience + the four detail
tiers, then **Run tracker** to propagate the box into a keyframed track (classical
OpenCV optical flow) with an appear/disappear window — or **Add as static box**
for a fixed label. Shift-click the stage to place an **affect anchor** and type its
emotion tiers. **Save to manifest** writes the entry back via the pipeline's
idempotent upsert (keeping the clip's media + provenance, replacing only the
authored content). The tracker only propagates **box geometry** — every label
string, scientific name, and fact is **hand-authored**, because a generic detector
can't produce "*Gymnothorax*" or a lifespan.
+123
View File
@@ -0,0 +1,123 @@
# Content candidate pool — operator-selected, vetted (2026-06-24)
> **WIRED (session 0016).** The rotating pool below is now the live manifest:
> `simulator/build_pool_manifest.py` regenerates `simulator/sample_media/manifest.json`
> from this pool; the ring scale `forest`→`coast` rename + the cleanup list (bottom)
> are done; the player picks a random pool member on landing
> (`player.ring.pick_clip_id`). See content-pipeline design §11.
The 5 scales each get a **rotating pool** of clips. When the viewer lands on a
scale by zooming the ring in/out, the player **randomly picks one clip from that
scale's pool**. (This supersedes the one-clip-per-scale model — see
[`docs/superpowers/specs/2026-06-24-content-pipeline-design.md`](./superpowers/specs/2026-06-24-content-pipeline-design.md).)
Every clip below was operator-selected from the candidate gallery, is **strict
public-domain or CCBYclass with attribution**, and was vetted **wordfree /
humanfree** (windows trimmed past title cards, captions, telemetry burnin,
divers, construction, logos). All processed via `tools/pipeline/run.py
process_clip` (trim → crossfadeloop → 1080p proxy). **Media is gitignored**;
this file is the resourceable record. Sim ids are the `simulator/sample_media/<id>/` dirs.
License rule confirmed by operator: **CC0 / CCBY / CCBYSA are acceptable with
a credit line** (our overlays + dream are derivatives, which these allow);
**CCBYND is NOT usable** (no derivatives); **CCBYNC only if the installation
stays noncommercial**.
## 🌌 cosmos (pool of 4)
| id | clip | source | window (s) | license / credit |
|----|------|--------|-----------|------------------|
| `cosmos` | Cosmic Cliffs — Carina Nebula flythrough | NASA SVS 31348 — https://svs.gsfc.nasa.gov/31348/ (`Clifs-3d-STScI.mp4`, *Exploring the Cosmic Cliffs in 3D*, Webb NIRCam, **4K**) | 1034 | CCBYclass — credit **NASA, ESA, CSA, STScI** |
> ️ **Cosmos primary swapped to the Webb "Cosmic Cliffs" flythrough (2026-06-25, session 0018).**
> History: the original primary was NASA/JPL's *"Orion: Dust and Death"* (captions
> burned in → cropping lost framing); it was then swapped to STScI's text-free
> *Flight Through the Orion Nebula* (SVS 30957, visible light). The operator found
> the Orion clip merely good and asked for a more dramatic nebula, so the primary is
> now STScI's **Exploring the Cosmic Cliffs in 3D** (Carina Nebula, Webb NIRCam,
> **SVS 31348**) — a full-frame, text-free 3D flythrough sourced from its **4K**
> master (`Clifs-3d-STScI.mp4`, 3840×2160) → supersampled 1080p base + 4K master.
> Trim 1034s clears the opening title card (gone by ~8s) and the end-credit card
> (~102s). The earlier `decaption_cosmos` crop helper remains removed.
| `cosmos_galaxies` | Flying Through Galaxies | NASA SVS — https://svs.gsfc.nasa.gov/vis/a010000/a014900/a014950/14950_Galaxies_FlyThrough_4k.mp4 | 832 | PD |
| `cosmos_hudf` | Hubble Ultra Deep Field zoom | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030600/a030687/hudf-b-1920x1080p30.mov | 2044 | CCBYclass — credit **NASA, ESA, STScI** |
| `cosmos_xdf` | eXtreme Deep Field flythrough | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030600/a030681/hxdf_fly-b-1920x1080p30.mov | 226 | CCBYclass — credit **NASA, ESA, STScI** |
> ⚠️ **DISCREPANCY to fix next session.** Operator selected cosmos **#1, #3
> (Orion), #4, #5**. During processing, **#6 Galaxy Traverse was run by mistake
> instead of #3 Orion**. The cosmos primary now lives at
> `simulator/sample_media/cosmos/base.mp4` as the Webb Cosmic Cliffs flythrough
> (SVS 31348 — see the cosmos-primary note above). **`cosmos_traverse` is an UNSELECTED extra on disk — drop it** (or
> confirm with operator). The current ring scale id is `cosmos`; when the pool is
> wired, `cosmos` becomes one pool member.
## 🛰️ orbit (pool of 3)
| id | clip | source | window (s) | license |
|----|------|--------|-----------|---------|
| `orbit_planetearth` | ISS — View of Planet Earth | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030700/a030771/ISS_View_of_Planet_Earth_2160p.mp4 | 1034 | PD |
| `orbit_crewobs` | ISS — Crew Earth Observations | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030700/a030771/ISS_Crew_Earth_Observations_2160p.mp4 | 933 | PD |
| `orbit_bluemarble` | NPP "Blue Marble" rotating globe (CG) | NASA SVS — https://svs.gsfc.nasa.gov/vis/a000000/a004500/a004550/BlueMarble_starfield_4k_2160p30_v2.mp4 | 1034 | PD |
> Operator also selected aurora→Perth (#4) and westcoast lights (#5); both
> **rejected as UNUSABLE** — #4 has a date/time burned into every frame, #5 has the
> ISS module + solar panel in frame throughout. No clean window exists in either.
## 🏖️ coast (pool of 4) — NEW scale, replaces the granite/waterfall `forest`
| id | clip | source | window (s) | license |
|----|------|--------|-----------|---------|
| `coast_birdrock` | Bird rock + ocean (aerial) | NPS Channel Islands — https://www.nps.gov/nps-audiovideo/audiovideo/663a2f13-6183-4b3e-a4a9-6698a84619d01080p.mp4 | 327 | PD |
| `coast_surfgrass` | Surfgrass / coralline tidepool | NPS Cabrillo — https://www.nps.gov/nps-audiovideo/audiovideo/de7d1cf2-e422-45b5-aa66-4c69e29a4d811080p.mp4 | 226 | PD |
| `coast_elkbeach` | Elk resting in coastal grass/fog | NPS Redwood — https://www.nps.gov/nps-audiovideo/legacy/redw/4634070E-F68D-4F74-E43174448168AC8A/redw-elkbeach_1280x720.mp4 | 2038 | PD (720p source) |
| `coast_drakesbeach` | Elephant seals on dark sand | NPS Point Reyes — https://www.nps.gov/nps-audiovideo/audiovideo/1cfd8165-1061-4d75-af53-b46c2e3ed2701080p.mp4 | 2035 | PD |
> Operator also selected the Lifeboat Station "elephant seal timelapse" (moreoptions
> #3) — **FAILED**: the NPS page's video resolved to the wrong footage (an aerial of
> the bay with road/parking/boats, no seals). An alternate source UUID
> (`50a7a20a-9e52-4643-b186-c61ce1cef0f7`) appeared in the page HTML and could be
> retried next session if the seal timelapse is still wanted.
>
> **Scale rename:** the live ring scale is still `forest` (Yosemite). Next session
> renames it `coast` and points it at this pool. The old real Yosemite base
> (`simulator/sample_media/forest/`) becomes unused.
## 🐠 reef (pool of 5)
All NOAA Fisheries broll (PD, credit "NOAA Fisheries"); Brightcove download
endpoint `https://download.gallery.brightcove.com/api/gallery/account/659677166001/site/site-600408/video/<VIDEO_ID>/download`.
| id | clip | VIDEO_ID | window (s) | notes |
|----|------|----------|-----------|-------|
| `reef_spawning` | Snapper school + yellow fish over sunlit reef | 1553921798001 | 5078 | clean window from a 12min reel |
| `reef_hawkfish` | Humphead parrotfish over reef (Hawaiian atoll) | 6039896460001 | 250274 | |
| `reef_lionfish` | Lionfish hovering + blue snapper | 4088881464001 | 1236 | underwater clean; humans on land after ~190s |
| `reef_snapper` | Red snapper schooling (Gulf, greenish water) | 5305427942001 | 731 | |
| `reef_coralspacific` | Pacific coral macro, spotted fish | 5231464663001 | 3054 | diverfree window (divers at 625s) |
## 🦑 abyss (pool of 3)
All NOAA Ocean Exploration (PD, 17 U.S.C. §105). These have creatures drifting
**into and out of frame** — the prime material for timewindowed tracked labels.
| id | clip | source | window (s) | creatures |
|----|------|--------|-----------|-----------|
| `abyss_wow` | "World of Water" | https://oceanexplorer.noaa.gov/wp-content/uploads/2018/05/wow-1280x720-1.mp4 | 1034 | jelly + ctenophore + siphonophore enter/exit |
| `abyss_midwaterexp` | "Midwater Exploration" | https://oceanexplorer.noaa.gov/wp-content/uploads/2019/09/midwater-exploration-1920x1080-1.mp4 | 1640 | white medusa + red worm enter/exit |
| `abyss_hiding` | "Hiding in the Dark" | https://oceanexplorer.noaa.gov/wp-content/uploads/2018/05/dark-1280x720-1.mp4 | 2044 | crimson jelly + ctenophore enter/exit |
> Operator also selected Crossota (#4) and comb jelly (#5); both **rejected** —
> Crossota has only ~17s clean (too short for a 24s window), comb jelly has a
> species caption covering the whole clip.
## On disk but NOT in any pool — ✅ CLEANED UP (session 0016)
Removed from local media (all were gitignored, so repo-invisible):
- `cosmos_traverse` — unselected (the #6/#3 mixup above). **dropped.**
- `orbit_westcoast` — rejected (ISS hardware in frame). **dropped.**
- `forest` — the old Yosemite base, superseded by the `coast` pool. **dropped.**
- `reef` — the original placeholder/early reef base, superseded by `reef_*`. **dropped.**
- `orbit` / `abyss` — the Increment1 single real bases (NASA ISS Exp65 / ctenophore),
not in the new pools. **dropped** (the cosmos base — now the Webb Cosmic Cliffs flythrough, SVS 31348 — stays as the cosmos pool primary).
- `review.html` gallery — now `.gitignore`d (localonly, rebuildable).
@@ -1,7 +1,7 @@
# HEF — Content Pipeline (source → process → label → manifest)
**Date:** 2026-06-24
**Status:** Graduated — operator-approved (session 0014); **Increment 1 built & merged** (PR #13). Increment 2 (hybrid track pass + author mode) + real PD sourcing pending.
**Status:** Graduated — operator-approved (session 0014); **Increment 1 built & merged** (PR #13); real PD sourcing + rotating pool curated (PR #14/#15). **Increment 2 BUILT** (session 0016 — pool model + tiers + windows in PR #16; `track.py` + author mode in PR B; see §11).
**Repo:** `human-experience-filter-art`
**Anchor:** `docs/ROADMAP.md` sub-project 3 (Player Runtime) content needs; the
spiritual successor to sub-project 2 (Ingest & Tagging tools), retargeted from the
@@ -239,7 +239,114 @@ master encode (waits on a measured pano spec).
download + provenance record for v1.
- **Crossfade overlap length** per scale (slow cosmos vs busier reef) — by eye.
## 10. Out of scope (YAGNI)
## 11. Increment 2 — schema decisions (session 0016)
Increment 2 is the second build order from §8. It is mostly **schema work**, so
this section pins the concrete data shapes before the build (the operator asked for
a short design pass first). Each subsection is a new, **backward-compatible**
extension — Increment 1 manifests keep rendering unchanged.
### 11.1 Rotating clip pool (supersedes one-clip-per-scale)
Each ring scale references a **pool** of vetted clips
(`docs/content-candidate-pool.md`); the player picks **one member at random when
the viewer lands on that scale** (zooming the ring in/out). This supersedes
decision §5's one-base-per-scale.
- **Manifest:** `ring.scales[i]` gains `"pool": ["clip_id", …]` (≥1). The legacy
`"clip_id"` stays as the **primary** member for back-compat (= `pool[0]` when a
pool is written); a scale with only `clip_id` and no `pool` behaves exactly as
Increment 1 (pool of one).
- **Canonical logic (`player/ring.py`):** `Scale` gains `pool: tuple[str, …]`;
`members` = `pool or (clip_id,)`. A **pure** `pick_clip_id(scale, r)` takes an
injected `r ∈ [0,1)` and returns `members[int(r·len)]` — randomness stays
**injected** (DI), so the function is pure/testable and the Pi reuses it.
- **Where the pick happens:** at the **API boundary** (`simulator/app.py`), which
may be impure: the advance endpoint calls `pick_clip_id(landed_scale,
random.random())` and returns it as `target_clip_id`. A `delta=0` advance is the
**initial / re-roll** pick (no-op move, fresh random member). The client loads
`move.target_clip_id` directly instead of deriving the clip from the index. One
selection implementation, Python-canonical (browser still "only renders").
- **Rename + cleanup:** ring scale `forest`**`coast`** (new land↔sea scale
between `orbit` and `reef`). Drop the leftovers `cosmos_traverse`,
`orbit_westcoast`, old `forest`, old `reef` (per the pool doc's cleanup list);
the cosmos pool's primary stays `cosmos` (Orion). New ring order:
`cosmos → orbit → coast → reef → abyss → (wrap) cosmos`; edges regenerated
(`orbit-coast`, `coast-reef` placeholders via `setup_scales_media.py`).
### 11.2 Time-windowed tracked labels
Objects drift **into and out of frame**, so a label appears only **while its
object is on screen** and its box **tracks** the object (the §4 keyframed `track`).
- **Manifest:** an annotation gains optional `"appear"` and `"disappear"`
(loop-normalized `t ∈ [0,1]`, same clock as `track`). The label renders only
while `appear ≤ loopT ≤ disappear`. To let an object straddle the loop seam, a
**`disappear < appear` window wraps** (visible when `loopT ≥ appear` **or**
`loopT ≤ disappear`). Absent both fields → always-present (Increment-1 behavior).
- **Runtime:** client-side in `renderOverlay` — a window test gates the annotation
before the existing `boxAt(loopT)` placement. Engine unchanged (labels are not in
`RenderPlan`). The **abyss + reef** pool clips (creatures entering/leaving) are
the authored test material.
### 11.3 Progressive LEFT-brain detail tiers (replaces flat `min_level` gating)
The Left knob now drives **how many** labels show **and how analytical** each is —
no fading. Low Left = **fewer, more general** labels; high Left = **more** labels +
**more analytical** detail.
- **Per-object `salience` (14):** how prominent the object is. It sets the Left
level at which the label first appears: `first_level = 5 salience` (salience 4 →
appears at Left 1; salience 1 → only at Left 4). So raising Left brings in
lower-salience objects.
- **Detail tiers (escalate with Left):** the displayed string escalates
`tier 1 general ("eel") → 2 specific ("moray eel") → 3 scientific
(*Gymnothorax*) → 4 scientific + a fact (lifespan)`. The tier shown =
`clamp(left first_level + 1, 1, n_tiers)` — an object shows tier 1 the moment
it appears and climbs one tier per further Left notch until its tiers run out.
- **Strings carry the tiers:** `strings.en[key]` may be a **list** (indexed by
`tier1`) instead of a string; a plain string stays static (back-compat, tier
ignored). Keeps one key per object and the i18n table as the single source.
- **Back-compat:** an annotation with `min_level` and no `salience` keeps flat
gating (`first_level = min_level`, single-tier string). `measure.*` readouts stay
single-tier, gated by their level (a measurement is not a tiered object).
- **No engine change:** tier selection is client-side against the existing
`overlay.level` (the Left knob) already in the `RenderPlan`.
### 11.4 Progressive RIGHT-brain emotion tiers
The affect **vocabulary** gets more sophisticated as Right rises — basic words at
Right 1, nuanced/compound feelings at high Right — mirroring §11.3 for affect.
- **Appearance rule unchanged:** affect surfaces only when **both** knobs are up;
an anchor's `min_level` is tested against `strength = min(left, right)` as today.
- **Emotion tiers:** `strings.en[feel.key]` may be a **list**; the word shown =
`clamp(right, 1, n_tiers)` (the Right knob, read from `dream.strength` already in
the `RenderPlan`). A plain string stays static (back-compat).
- **No engine change:** `AffectOverlay` already carries `strength`; the Right level
comes from `dream.strength`. Tier selection is client-side.
### 11.5 `tools/pipeline/track.py` (hybrid) + simulator author mode
The stage-5 tooling from §3.5 / §7, delivered as a **second PR** after the
experience PR (§8's split note).
- **`track.py`:** a **classical** path (OpenCV CSRT / optical-flow) seeded by a
hand-drawn box on a keyframe, propagated and **sampled to a sparse keyframed
`track`** (normalized `{t, box}` over the loop, §4); plus an **optional,
lazy-imported ML** detect+track path (YOLO+ByteTrack-class) that proposes tracks
the author maps to keys. Both emit the same `track` artifact + an `appear`/
`disappear` window. Pure helpers (normalization, keyframe sampling,
track→manifest) are unit-tested; an **opt-in** integration test runs the real
tracker on a pool clip (mirrors the Increment-1 real-footage test).
- **Author mode (simulator):** reuse the preview UI as the editor — scrub a pool
clip, draw/seed/correct boxes, run the tracker, assign `key` / `salience` /
tiered strings / `appear`-`disappear` / affect anchors, and **write the manifest
entry** (an endpoint upserts via `tools/pipeline/manifest.upsert_clip` and
persists). Semantics (keys, strings, scientific names, facts) are **always
hand-authored** — the tracker only propagates **box geometry** (§3.5).
## 12. Out of scope (YAGNI)
- Generative i2v **ring transitions** for the new edges (deferred; placeholders hold).
- **Audio** sourcing/alteration (separate content-dial channel; already deferred).
@@ -0,0 +1,64 @@
# Simulator Dev Mode — design
> 2026-06-25 · session 0017 · planning-and-executing
> Anchor: leaf task (simulator dev-tooling — R2b, no upstream design needed)
## Problem
While previewing the experience, the operator can't choose *which* clip in an
altitude's rotating pool is shown — the server picks a random pool member on each
landing (content-pipeline §11.1). For by-eye review they want to step through every
clip in a pool on demand, and to see the engine/clip data behind what's on screen.
The RenderPlan JSON readout is also always visible, cluttering what should read as
an audience-facing preview.
## Goal
A single **Dev Mode** toggle at the bottom of the control panel. Off by default
(persisted in `localStorage`). When on, a panel appears directly below it with all
dev controls and data; when off, the normal view is just the experience + the three
control fieldsets.
## Scope
Frontend only — `simulator/static/{index.html,app.js,style.css}`. No API changes:
every datum Dev Mode shows is already available client-side (`/api/clips`,
`/api/ring`, the `/api/alteration` response, and the in-memory preload cache). The
separate `/author.html` remains the place to *edit* labels; Dev Mode is
inspect-and-select only.
## Dev panel contents (top → bottom)
1. **Pool picker** — a `<select>` of the current altitude's pool members
(`clip_id · title`), its value reflecting the clip currently playing. Choosing a
member sets `activeClipId` to it and reloads the base media, overriding the random
landing pick. A `🎲 Re-roll random` button re-runs the server-canonical random
pick for the current scale. The dropdown is rebuilt on every scale landing, so it
always lists the current pool; single-clip scales (cosmos/orbit) show one entry.
2. **Active clip**`id`, `title`, `source`, `license`, `base_file`.
3. **Annotations & affect** — each annotation key with salience (or legacy
`min_level`), time window (`appear``disappear`), and tier count; each affect key
with `min_level` and tier count. Read-only.
4. **RenderPlan** — the relocated `#readout` (the `/api/alteration` JSON).
5. **Cache & playback** — preload status (`N/total cached`), whether the active
clip is served **from memory** (a preloaded blob) vs the network, and live video
resolution / duration / loop position.
## Behavior
- **Pool override is per-landing.** Changing altitude re-rolls randomly as today;
the operator then picks again from the new scale's pool. The pick is not persisted
across landings (YAGNI — "select any video while on an altitude").
- **Refresh points.** Pool picker + clip metadata + annotations rebuild whenever the
scale lands (hooked into `renderScaleReadout()`) and after a manual pick/re-roll.
The RenderPlan readout updates on each `update()` (knob change) as today. The
cache/playback stats refresh on a light interval (~500 ms) while Dev Mode is on,
and the cache count also ticks up as the preloader fills.
- **`pickRandomMember()`** is factored out of `landScale()` so the initial landing,
ring advances, and the Re-roll button all share the one server-canonical pick.
## Testing
No JS unit harness in the repo and no API change, so verification is: `node --check`
on `app.js`, the Python suite stays green (unaffected), and by-eye review by the
operator (no headless browser yet — §9 E2E policy noted, machinery pending).
+30 -1
View File
@@ -50,10 +50,25 @@ class RingError(ValueError):
@dataclass(frozen=True)
class Scale:
"""A node on the ring: a scale of nature and the base clip it plays."""
"""A node on the ring: a scale of nature and the clip(s) it can play.
A scale carries a rotating POOL of vetted clips (content-pipeline design
§11.1): when the viewer lands on the scale, the player picks one member at
random (`pick_clip_id`). `clip_id` is the PRIMARY member (the deterministic
default / pool[0]); `pool` is the full set. A scale written with only
`clip_id` and no `pool` is a pool of one — exactly the Increment-1 behavior,
so old manifests are unchanged.
"""
id: str
clip_id: str
pool: tuple[str, ...] = ()
@property
def members(self) -> tuple[str, ...]:
"""The clip ids this scale can play — the pool, or just `clip_id` when
no pool was authored (a pool of one)."""
return self.pool if self.pool else (self.clip_id,)
@dataclass(frozen=True)
@@ -134,6 +149,20 @@ def scale_at(ring: ScaleRing, index: int) -> Scale:
return ring.scales[index % len(ring)]
def pick_clip_id(scale: Scale, r: float) -> str:
"""Pick one clip id from a scale's rotating pool, given an injected uniform
`r` in [0, 1) (content-pipeline design §11.1).
Randomness is INJECTED so this stays a pure function — testable and shared
with the Pi player; the impure draw (`random.random()`) happens once at the
API/runtime boundary. `r` is clamped into [0, 1) so an off-by-epsilon caller
never indexes out of range; a pool of one always returns its single member.
"""
members = scale.members
r = min(max(r, 0.0), 0.999999)
return members[int(r * len(members))]
def advance_ring(
ring: ScaleRing,
from_index: int,
@@ -0,0 +1,117 @@
# Session 0016.0 — Transcript
> App: human-experience-filter-art
> Start: 2026-06-24T17-48 (PST)
> Type: planning-and-executing
> Posture: yolo
> Claude-Session: 0fde1d36-7c64-414f-8f49-8669b5dd2263
> Status: **FINALIZED**
## Launch prompt
`/goal` Content pipeline Increment 2 (expanded) — rotating clip pools + time-windowed
TRACKED labels + progressive left-brain detail + progressive right-brain emotions.
Short design pass first (schema changes), then build.
> Anchor: design `docs/superpowers/specs/2026-06-24-content-pipeline-design.md`
> (graduated, R2a-eligible). Repo is roadmap-anchored, no Gitea issue tracker (R1
> deviation, as all prior sessions). Increment 2 of that design.
## Pre-state
`main` at `b28f930` (PR #15 — operator-curated rotating-pool doc merged). All 19
vetted pool clips on disk (gitignored) plus the leftovers to drop. The ring was the
Increment-1 one-base-per-scale model (cosmos/orbit/forest/reef/abyss), flat
`min_level` label gating, static or simple-track boxes, scene-level affect gated by
`min(left,right)`. `tools/pipeline/` had stages 14 + 6 (no `track.py`). 246 tests.
## Arc
1. **Session gate + init.** Classified the `/goal` as planning-and-executing,
claimed session **0016** (peeked an ENDED-UNFINALIZED 0015 — its PR #15 work was
already merged, not live — and proceeded; no worktree needed, solo session).
Verified clean `main`, read the pool doc + spec + sub-project-3 memory + the live
code (ring, clips, app, app.js, alteration, pipeline).
2. **Design pass (operator asked for it — schema changes).** Appended **§11** to the
content-pipeline spec pinning the concrete shapes: rotating pool, time-windowed
tracks, progressive Left detail tiers (salience + tiered strings), progressive
Right emotion tiers, and the `track.py` + author-mode tooling. Key call: tiers
need **no engine change** — the client reads `overlay.level` (Left) and
`dream.strength` (Right) already in the `RenderPlan`.
3. **PR #16 — the experience.** `Scale.pool` + pure `pick_clip_id` (injected
randomness); server-side random pick at the API boundary (delta=0 = initial pick).
`forest``coast` rename + ring reorder + orphan-media cleanup. Time-windowed
tracked labels (`appear`/`disappear`, seam-wrapping). Salience-gated +
tier-escalating Left labels; tier-escalating Right emotions; tiered-or-static
strings (back-compat). New `build_pool_manifest.py` emits the 19-clip pool
manifest (abyss+reef authored as tracked/windowed showcase). Merged via Gitea API.
4. **PR #17 — the tooling.** `tools/pipeline/track.py`: classical LK optical-flow box
tracker (this opencv build has no CSRT) → sparse keyframed track + window; optional
lazy ML path. Simulator AUTHOR MODE (`/author.html` + `/api/author/{track,clip}`):
scrub, drag a seed box, run tracker, author tiers/affect, save to manifest
(idempotent upsert, keeps provenance). Repointed the pipeline integration test off
the retired forest base; USER_GUIDE sim section brought current. Merged.
## Cut state
`main` at `c445239` (PR #17 merge). **267 passed / 2 skipped.** Clean tree, fully
pushed, no open PRs. Both feature branches deleted. All 5 goal items delivered.
Verification was API/logic-level + JS syntax + live-server probes (5 scales with
pools, rotating picks, new edges, 19 clips served, tracker returns keyframes on real
abyss_wow footage, committed manifest untouched by the read-only probe). **The
VISUAL look was not screenshot-verified — no Chrome on this box** — so operator
by-eye review is the next step (matches the established deferred-review pattern).
## Follow-up (post-finalize, same conversation) — PR #18
Operator asked to replace the scale-ring `⊖ out / in ⊕` buttons with a **circular
"Altitude" knob** that turns all the way around (including abyss→cosmos). Built a
turnable SVG dial in `simulator/static/app.js` (`buildDial` / `renderDial` / drag +
wheel handlers) + `style.css`: cosmos (highest) at top, clockwise descends
cosmos→orbit→coast→reef→abyss and **wraps** back up. Dragging accumulates rotation
and commits whole detents on release (a big spin → one fast blended pass, reusing
the proven wheel/detent path); scroll the knob to step; tap a label to jump the
shortest way around. **Pure frontend** — the endless ring + wrap already existed
server-side, so no engine/API change. JS syntax-checked; 23 sim-API tests green;
live probe confirmed the dial replaced the buttons and the abyss→cosmos wrap is
intact. Merged as **PR #18** (`main` at `7a32ff1`). Visual look still operator-by-eye
(no Chrome on this box).
## Deployment pipeline (§9)
No stage run: the HEF simulator is a **local dev tool** (simulator-first directive),
not a deployed app — it has no PPE/prod infra. Nothing shippable through the §9
pipeline this session; the session output is the merged code on `main`.
## Plan archival
No separate `superpowers:writing-plans` artifact was produced — the design pass was a
spec §11 edit (merged in PR #16) and execution was direct TDD tracked by the
transcript `## Plan` block + the task list. Nothing to `submit-plan.sh`; this app has
no content repo anyway (roadmap-anchored).
## Deferred decisions
- **Adopting/finalizing the ENDED-UNFINALIZED session 0015 placeholder** was skipped
— its PR #15 work is already on `main` and its outcomes are fully recorded in
memory; finalizing a prior dead session was judged an unnecessary detour. Low
confidence; flagging for the operator.
- **By-eye visual review deferred** to the operator (no Chrome here) — consistent with
the prior-session pattern, but the look of the tiers/windows/pool rotation AND the
new Altitude knob is unconfirmed visually.
- **Light-vs-rich label authoring:** only abyss + reef + the scale primaries are
richly authored; the other pool members carry lighter labels. Finishing them is the
author-mode follow-up (next goal).
## Next-session prompt
`/goal` Operator by-eye review of Increment 2 + the Altitude knob in the sim
(`make sim-local`, :8000): turn the Altitude knob (drag/scroll, wraps abyss→cosmos;
pools rotate per landing), sweep Left (progressive label tiers
general→scientific+fact), sweep Right with Left up (progressive emotions
basic→compound), watch abyss/reef time-windowed tracked labels enter/leave + track,
and exercise `/author.html`. Then author real labels for the remaining pool members
via author mode. After the content is liked, the next build frontier is the deferred
i2v ring transitions (generative-video model + adjacent real bases) and/or the Pi
renderer + serial/firmware (per [[simulator-first-before-hardware]]), per ROADMAP.md.
@@ -0,0 +1,22 @@
# Session 0017.0 — Transcript
> App: human-experience-filter-art
> Start: 2026-06-25T07-41 (PST)
> Type: planning-and-executing
> Posture: yolo
> Claude-Session: bcb069f2-0fba-4a66-ae8e-f40b7917a48c
> Status: **PLACEHOLDER — claimed at session start; finalized at session end.**
>
> This file reserves session ID 0017 for human-experience-filter-art. The driver replaces this
> body with the full transcript and renames the file to its final
> SESSION-0017.0-TRANSCRIPT-2026-06-25T07-41--<end>.md form at session end.
## Launch prompt
_(launch prompt not captured at claim time)_
## Deferred decisions
_Autonomous-mode low-confidence calls the driver made and would have
liked operator input on. Appended as the session runs; surfaced at
finalize. Empty if none._
@@ -0,0 +1,25 @@
# Session 0018.0 — Transcript
> App: human-experience-filter-art
> Start: 2026-06-25T09-42 (PST)
> Type: planning-and-executing
> Posture: yolo
> Claude-Session: b96041e5-c682-49f5-88ee-380ea7abb6af
> Status: **PLACEHOLDER — claimed at session start; finalized at session end.**
>
> This file reserves session ID 0018 for human-experience-filter-art. The driver replaces this
> body with the full transcript and renames the file to its final
> SESSION-0018.0-TRANSCRIPT-2026-06-25T09-42--<end>.md form at session end.
## Launch prompt
```
Find a better public-domain nebula flythrough video for the cosmos scale (the current text-free SVS Orion clip lost quality when cropped to remove its text overlay), or upscale the existing one if no better source exists.
```
## Deferred decisions
_Autonomous-mode low-confidence calls the driver made and would have
liked operator input on. Appended as the session runs; surfaced at
finalize. Empty if none._
+9
View File
@@ -43,5 +43,14 @@
},
"0015": {
"title": ""
},
"0016": {
"title": ""
},
"0017": {
"title": ""
},
"0018": {
"title": ""
}
}
+154 -7
View File
@@ -8,7 +8,9 @@ endpoints (/api/select, /api/catalog/meta) and the X-ray are retired.
from __future__ import annotations
import hashlib
import json
import os
import random
import time
from pathlib import Path
from typing import Optional
@@ -26,7 +28,12 @@ from player.alteration import (
)
from player.content import resolve_content
from player.controls import CONTENT_POSITIONS
from player.ring import DEFAULT_FAST_SPIN_THRESHOLD, advance_ring
from player.ring import (
DEFAULT_FAST_SPIN_THRESHOLD,
advance_ring,
pick_clip_id,
scale_at,
)
from simulator.clips import load_manifest, load_ring, ring_move_to_dict, ring_to_dict
STATIC_DIR = Path(__file__).parent / "static"
@@ -52,6 +59,41 @@ def _asset_version() -> str:
return hashlib.sha1("|".join(parts).encode()).hexdigest()[:16]
# Content-hash tokens for served media, so the client can append `?v=<hash>` to a
# /media URL. A clip re-baked under the SAME path (e.g. a re-sourced cosmos base)
# changes its hash → its URL → a fresh fetch, busting any cached prior bytes
# permanently (even an immutable-pinned entry a plain reload can't revalidate).
# Cached by (mtime_ns, size) so the full-file hash is recomputed only when the
# file actually changes — a re-bake is picked up without a server restart.
_media_hash_cache: dict[str, tuple[int, int, str]] = {}
def _media_version(rel: str) -> Optional[str]:
"""Short content hash of the media file at `rel` under MEDIA_DIR, or None if
it's absent. Cheap on repeat calls: re-hashes only when (mtime, size) change."""
path = MEDIA_DIR / rel
try:
st = path.stat()
except OSError:
return None
cached = _media_hash_cache.get(rel)
if cached and cached[0] == st.st_mtime_ns and cached[1] == st.st_size:
return cached[2]
h = hashlib.sha1()
with path.open("rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
token = h.hexdigest()[:12]
_media_hash_cache[rel] = (st.st_mtime_ns, st.st_size, token)
return token
def _rev_file(file: str) -> str:
"""The baked zoom-out companion path for a transition file (mirrors the
client's `reverseFile`): `<edge>.mp4` -> `<edge>.rev.mp4`."""
return file[:-4] + ".rev.mp4" if file.endswith(".mp4") else file
class ControlsModel(BaseModel):
content: str
left: int = Field(ge=0, le=4)
@@ -78,6 +120,29 @@ class RingAdvanceRequest(BaseModel):
delta: int
class AuthorTrackRequest(BaseModel):
"""Author mode: propagate a hand-seeded box on a clip into a keyframed track
(content-pipeline §11.5). `seed_box` + `seed_t` are normalized."""
clip_id: str
key: str
seed_box: list[float] = Field(min_length=4, max_length=4)
seed_t: float = Field(default=0.0, ge=0.0, le=1.0)
salience: int = Field(default=4, ge=1, le=4)
n_keyframes: int = Field(default=5, ge=2, le=20)
max_frames: Optional[int] = None
class AuthorClipRequest(BaseModel):
"""Author mode: persist the authored labels/affect/strings for an existing
pool clip into the manifest (geometry from the tracker, semantics by hand)."""
clip_id: str
annotations: list = []
affect: list = []
strings: dict = {}
def _load_clips(manifest_path: Optional[Path]):
path = Path(manifest_path) if manifest_path else DEFAULT_MANIFEST
if path.exists():
@@ -94,6 +159,7 @@ def _load_ring(manifest_path: Optional[Path]):
def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
app = FastAPI(title="HEF Alteration Simulator")
app.state.manifest_path = Path(manifest_path) if manifest_path else DEFAULT_MANIFEST
app.state.clips = _load_clips(manifest_path)
app.state.ring = _load_ring(manifest_path)
@@ -129,6 +195,24 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
def api_clips():
return {"clips": [c.to_dict() for c in app.state.clips]}
@app.get("/api/media-versions")
def api_media_versions():
"""Per-file content-hash tokens the client appends to /media URLs as
`?v=<hash>`. Covers every served file: each clip's base footage plus each
ring transition and its baked reverse. A re-baked clip's hash changes, so
its URL changes and the browser refetches — a permanent cache-bust."""
files = {c.base_file for c in app.state.clips}
if app.state.ring is not None:
for t in app.state.ring.transitions:
files.add(t.file)
files.add(_rev_file(t.file))
versions = {}
for f in sorted(files):
v = _media_version(f)
if v:
versions[f] = v
return {"versions": versions}
@app.get("/api/ring")
def api_ring():
if app.state.ring is None:
@@ -145,15 +229,78 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
req.delta,
fast_spin_threshold=DEFAULT_FAST_SPIN_THRESHOLD,
)
return ring_move_to_dict(move, app.state.ring)
# Rotating pool: pick a random member of the LANDED scale (content-pipeline
# §11.1). A delta=0 advance is the initial / re-roll pick (a no-op move that
# still yields a fresh random clip). The pure pick takes injected randomness.
landed = scale_at(app.state.ring, move.to_index)
chosen = pick_clip_id(landed, random.random())
return ring_move_to_dict(move, app.state.ring, chosen)
@app.post("/api/author/track")
def api_author_track(req: AuthorTrackRequest):
# Author mode (§11.5): run the classical optical-flow tracker on a clip's
# base footage, returning the keyframed track geometry for the author to
# accept/correct. Semantics (key/strings/tiers) stay the author's.
clip = next((c for c in app.state.clips if c.id == req.clip_id), None)
if clip is None:
raise HTTPException(status_code=404, detail=f"unknown clip {req.clip_id!r}")
base = MEDIA_DIR / clip.base_file
if not base.exists():
raise HTTPException(status_code=404, detail=f"media absent: {clip.base_file}")
try:
import cv2
except ImportError:
raise HTTPException(status_code=503, detail="opencv-python not installed")
from tools.pipeline.track import track_seed
cap = cv2.VideoCapture(str(base))
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
cap.release()
seed_frame = max(0, min(total - 1, round(req.seed_t * total))) if total else 0
return track_seed(
base, req.key, tuple(req.seed_box), seed_frame=seed_frame,
salience=req.salience, n_keyframes=req.n_keyframes, max_frames=req.max_frames,
)
@app.post("/api/author/clip")
def api_author_clip(req: AuthorClipRequest):
# Author mode (§11.5 / stage 6): persist authored labels/affect/strings for
# an EXISTING pool clip — idempotent upsert that keeps the clip's media +
# provenance and replaces only the authored content. Reloads in place so
# the preview reflects the edit without a restart.
from tools.pipeline.manifest import upsert_clip
path = app.state.manifest_path
data = json.loads(path.read_text())
existing = next((c for c in data.get("clips", []) if c["id"] == req.clip_id), None)
if existing is None:
raise HTTPException(status_code=404, detail=f"unknown clip {req.clip_id!r}")
merged = dict(existing)
merged["annotations"] = req.annotations
merged["affect"] = req.affect
if req.strings:
merged["strings"] = req.strings
data = upsert_clip(data, merged)
path.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n")
app.state.clips = load_manifest(path)
app.state.ring = load_ring(path)
return {"ok": True, "clip": merged}
@app.middleware("http")
async def _no_cache(request, call_next):
# Dev preview server: never let a browser serve a stale app.js/style.css.
# The whole point is by-eye iteration, so a plain refresh must always get
# the latest assets (the heuristic cache otherwise hides JS/CSS edits).
async def _cache_policy(request, call_next):
# Dev preview server: never let a browser serve a stale app.js/style.css
# by-eye iteration needs a plain refresh to always get the latest assets.
# Media uses `no-cache` (store, but REVALIDATE every load): instant altitude
# swaps already come from the in-memory blob preload (static/app.js), so the
# HTTP cache only matters on (re)load — where a conditional request returns a
# cheap 304 when unchanged, yet picks up a re-sourced/re-cropped clip
# immediately. `immutable` was wrong here: it pinned stale footage for a year
# so edited clips never showed without a manual cache clear.
response = await call_next(request)
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
if request.url.path.startswith("/media/"):
response.headers["Cache-Control"] = "no-cache"
else:
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
return response
if MEDIA_DIR.exists():
+423
View File
@@ -0,0 +1,423 @@
"""Build the rotating-pool manifest for the 5 ring scales (content-pipeline §11).
This is the reproducible, hand-authored BASELINE for Increment 2's manifest:
each ring scale carries a POOL of vetted clips (docs/content-candidate-pool.md);
the player picks one member at random on landing (player.ring.pick_clip_id). The
authored label/affect content lives here in readable Python and is emitted to
`simulator/sample_media/manifest.json` — far less error-prone than hand-editing 19
clips of JSON, and a clean place for the simulator author mode to regenerate from.
Authoring model (design §11.3 / §11.4):
- LEFT detail tiers: each label carries `salience` (1-4; first shows at Left
level `5 - salience`) and a tiered string list (general -> specific ->
scientific -> +fact) that escalates with the Left knob.
- RIGHT emotion tiers: affect words are a tiered list per feeling (basic ->
compound) escalating with the Right knob; still gated on min(Left, Right).
- TIME-WINDOWED + TRACKED labels: a label may carry `appear`/`disappear`
(loop-normalized) and a keyframed `track`; it shows only while on screen and
its box follows the subject. The abyss + reef pools are the test material.
Strings: a value may be a LIST (indexed by tier-1) or a plain string (static,
back-compat). Affect appears only when both knobs are up.
Usage:
python simulator/build_pool_manifest.py # write manifest.json
python simulator/build_pool_manifest.py --media # + generate new transition placeholders
"""
from __future__ import annotations
import json
import shutil
import subprocess
import sys
from pathlib import Path
MEDIA = Path(__file__).parent / "sample_media"
MANIFEST = MEDIA / "manifest.json"
# --- Pools: scale id -> ordered clip ids (primary first) (content-candidate-pool.md) ---
POOLS: dict[str, list[str]] = {
"cosmos": ["cosmos", "cosmos_galaxies", "cosmos_hudf", "cosmos_xdf"],
"orbit": ["orbit_planetearth", "orbit_crewobs", "orbit_bluemarble"],
"coast": ["coast_birdrock", "coast_surfgrass", "coast_elkbeach", "coast_drakesbeach"],
"reef": ["reef_lionfish", "reef_spawning", "reef_hawkfish", "reef_snapper", "reef_coralspacific"],
"abyss": ["abyss_wow", "abyss_midwaterexp", "abyss_hiding"],
}
# Ring order (large -> small, wraps): cosmos -> orbit -> coast -> reef -> abyss -> cosmos.
RING_ORDER = ["cosmos", "orbit", "coast", "reef", "abyss"]
PD = "public-domain (US Gov, 17 U.S.C. §105)"
PD_NPS = "public-domain (NPS, no © — 17 U.S.C. §105)"
CCBY_STSCI = "CC-BY-class — credit NASA, ESA, STScI"
CCBY_WEBB = "CC-BY-class — credit NASA, ESA, CSA, STScI"
# --- Per-clip provenance: id -> (title, license, source) ---
META: dict[str, tuple[str, str, str]] = {
# cosmos
"cosmos": ("Cosmic Cliffs — Carina Nebula flythrough (NASA/ESA/CSA/STScI)", CCBY_WEBB,
"NASA SVS 31348 Clifs-3d-STScI (Exploring the Cosmic Cliffs in 3D, Webb NIRCam); "
"4K source, trim 1034s, crossfade-loop. Text-free — replaces the Orion SVS 30957 flythrough."),
"cosmos_galaxies": ("Flying Through Galaxies (NASA SVS)", PD,
"NASA SVS a014950 14950_Galaxies_FlyThrough_4k; trim 832s, crossfade-loop"),
"cosmos_hudf": ("Hubble Ultra Deep Field zoom (NASA/ESA/STScI)", CCBY_STSCI,
"NASA SVS a030687 hudf-b-1920x1080p30; trim 2044s, crossfade-loop"),
"cosmos_xdf": ("eXtreme Deep Field flythrough (NASA/ESA/STScI)", CCBY_STSCI,
"NASA SVS a030681 hxdf_fly-b-1920x1080p30; trim 226s, crossfade-loop"),
# orbit
"orbit_planetearth": ("ISS — View of Planet Earth (NASA SVS)", PD,
"NASA SVS a030771 ISS_View_of_Planet_Earth_2160p; trim 1034s, crossfade-loop"),
"orbit_crewobs": ("ISS — Crew Earth Observations (NASA SVS)", PD,
"NASA SVS a030771 ISS_Crew_Earth_Observations_2160p; trim 933s, crossfade-loop"),
"orbit_bluemarble": ("NPP “Blue Marble” rotating globe (NASA SVS)", PD,
"NASA SVS a004550 BlueMarble_starfield_4k_2160p30_v2; trim 1034s, crossfade-loop"),
# coast
"coast_birdrock": ("Bird rock + ocean, aerial (NPS Channel Islands)", PD_NPS,
"NPS Channel Islands AV 663a2f13; trim 327s, crossfade-loop"),
"coast_surfgrass": ("Surfgrass / coralline tidepool (NPS Cabrillo)", PD_NPS,
"NPS Cabrillo AV de7d1cf2; trim 226s, crossfade-loop"),
"coast_elkbeach": ("Elk resting in coastal fog (NPS Redwood)", PD_NPS,
"NPS Redwood redw-elkbeach_1280x720; trim 2038s, crossfade-loop (720p source)"),
"coast_drakesbeach": ("Elephant seals on dark sand (NPS Point Reyes)", PD_NPS,
"NPS Point Reyes AV 1cfd8165; trim 2035s, crossfade-loop"),
# reef
"reef_lionfish": ("Lionfish hovering over reef (NOAA Fisheries)", PD,
"NOAA Fisheries b-roll VIDEO_ID 4088881464001; trim 1236s, crossfade-loop"),
"reef_spawning": ("Snapper school over sunlit reef (NOAA Fisheries)", PD,
"NOAA Fisheries b-roll VIDEO_ID 1553921798001; trim 5078s, crossfade-loop"),
"reef_hawkfish": ("Humphead parrotfish over reef (NOAA Fisheries)", PD,
"NOAA Fisheries b-roll VIDEO_ID 6039896460001; trim 250274s, crossfade-loop"),
"reef_snapper": ("Red snapper schooling (NOAA Fisheries)", PD,
"NOAA Fisheries b-roll VIDEO_ID 5305427942001; trim 731s, crossfade-loop"),
"reef_coralspacific": ("Pacific coral macro (NOAA Fisheries)", PD,
"NOAA Fisheries b-roll VIDEO_ID 5231464663001; trim 3054s, crossfade-loop"),
# abyss
"abyss_wow": ("“World of Water” (NOAA Ocean Exploration)", PD,
"NOAA Ocean Exploration wow-1280x720-1; trim 1034s, crossfade-loop"),
"abyss_midwaterexp": ("“Midwater Exploration” (NOAA Ocean Exploration)", PD,
"NOAA Ocean Exploration midwater-exploration-1920x1080-1; trim 1640s, crossfade-loop"),
"abyss_hiding": ("“Hiding in the Dark” (NOAA Ocean Exploration)", PD,
"NOAA Ocean Exploration dark-1280x720-1; trim 2044s, crossfade-loop"),
}
# --- Affect vocabulary per SCALE (shared by all the scale's pool members) -------
# Each: (key, [x, y], min_level, [tier1 basic .. tier4 compound]). The word shown
# escalates with the RIGHT knob; appears only when both knobs are up (§11.4).
AFFECT: dict[str, list[tuple]] = {
"cosmos": [
("feel.wonder", [0.50, 0.44], 1, ["wow", "wonder", "awe", "transcendent awe"]),
("feel.vastness", [0.20, 0.70], 2, ["big", "vastness", "immensity", "a dizzying immensity"]),
("feel.insignificance", [0.66, 0.60], 3, ["small", "smallness", "insignificance", "a humbling insignificance"]),
("feel.longing", [0.42, 0.84], 4, ["want", "longing", "yearning", "an aching yearning"]),
],
"orbit": [
("feel.serenity", [0.50, 0.44], 1, ["calm", "serenity", "peace", "a quiet, weightless peace"]),
("feel.fragility", [0.22, 0.68], 2, ["thin", "fragility", "vulnerability", "a tender fragility"]),
("feel.unity", [0.66, 0.58], 3, ["one", "unity", "belonging", "a borderless belonging"]),
("feel.distance", [0.42, 0.82], 4, ["far", "distance", "remoteness", "an exquisite remoteness"]),
],
"coast": [
("feel.ease", [0.50, 0.44], 1, ["nice", "ease", "calm", "an unhurried calm"]),
("feel.nostalgia", [0.22, 0.68], 2, ["miss", "nostalgia", "wistfulness", "a salt-air wistfulness"]),
("feel.belonging", [0.66, 0.58], 3, ["home", "belonging", "rootedness", "a tidal rootedness"]),
("feel.melancholy", [0.42, 0.82], 4, ["sad", "melancholy", "longing", "a soft seaward longing"]),
],
"reef": [
("feel.delight", [0.50, 0.44], 1, ["fun", "delight", "joy", "a darting, bright joy"]),
("feel.abundance", [0.22, 0.68], 2, ["full", "abundance", "richness", "a teeming richness"]),
("feel.curiosity", [0.66, 0.58], 3, ["look", "curiosity", "fascination", "an absorbed fascination"]),
("feel.immersion", [0.42, 0.82], 4, ["in", "immersion", "absorption", "a held, breathless absorption"]),
],
"abyss": [
("feel.unease", [0.50, 0.44], 1, ["uh", "unease", "disquiet", "a creeping disquiet"]),
("feel.fascination", [0.22, 0.68], 2, ["ooh", "fascination", "wonder", "a forbidden wonder"]),
("feel.isolation", [0.66, 0.58], 3, ["alone", "isolation", "solitude", "a crushing solitude"]),
("feel.dread", [0.42, 0.82], 4, ["fear", "dread", "foreboding", "a slow, cold foreboding"]),
],
}
def static_label(key, salience, tiers, box):
"""A fixed-box tiered label (always on-screen, salience-gated by Left)."""
return {"key": key, "salience": salience, "box": box, "_tiers": tiers}
def tracked_label(key, salience, tiers, appear, disappear, track):
"""A time-windowed tracked label: shows only in [appear, disappear] and its
box follows the subject (§11.2). `track` = [(t, [x,y,w,h]), ...]."""
return {
"key": key, "salience": salience, "appear": appear, "disappear": disappear,
"track": [{"t": t, "box": b} for t, b in track], "_tiers": tiers,
}
def measure(key, value, box, min_level):
"""A single-tier measurement readout (not a tiered object), gated by Left."""
return {"key": key, "box": box, "min_level": min_level, "_tiers": value}
# --- Labels per clip. `_tiers` is lifted into the strings table by build(). ------
# Showcase tracked/windowed labels live on the abyss + reef pools (the creatures
# drift through frame). Other clips get a few salience-gated static tiered labels.
LABELS: dict[str, list[dict]] = {
# ---------- cosmos ----------
"cosmos": [
static_label("detected.nebula", 4, ["cloud", "nebula", "emission nebula", "emission nebula · the Carina star-forming region"], [0.20, 0.42, 0.5, 0.42]),
static_label("detected.star", 2, ["star", "young star", "protostar", "protostar · a newborn star, <1 Myr old"], [0.54, 0.12, 0.12, 0.14]),
measure("measure.distance", "≈7,500 ly", [0.06, 0.06, 0.2, 0.08], 3),
],
"cosmos_galaxies": [
static_label("detected.galaxy", 4, ["galaxy", "spiral galaxy", "barred spiral", "barred spiral · ~10¹¹ stars"], [0.34, 0.30, 0.30, 0.34]),
measure("measure.distance", "~Mly", [0.06, 0.06, 0.18, 0.08], 3),
],
"cosmos_hudf": [
static_label("detected.galaxy", 4, ["smudge", "galaxy", "early galaxy", "early galaxy · z ≈ 16, light Gyr-old"], [0.40, 0.36, 0.20, 0.22]),
measure("measure.field", "deep field", [0.06, 0.06, 0.2, 0.08], 3),
],
"cosmos_xdf": [
static_label("detected.galaxy", 4, ["dot", "galaxy", "faint galaxy", "faint galaxy · among the earliest seen"], [0.42, 0.38, 0.18, 0.2]),
],
# ---------- orbit ----------
"orbit_planetearth": [
static_label("detected.cloud_band", 4, ["clouds", "cloud band", "cumulus field", "cumulus field · tropical convection"], [0.30, 0.30, 0.30, 0.20]),
static_label("detected.limb", 2, ["edge", "Earths limb", "atmospheric limb", "atmospheric limb · ~100 km of air"], [0.05, 0.70, 0.9, 0.12]),
measure("measure.altitude", "~408 km", [0.06, 0.06, 0.18, 0.08], 3),
],
"orbit_crewobs": [
static_label("detected.coastline", 4, ["land", "coastline", "continental margin", "continental margin · land meets sea"], [0.30, 0.34, 0.34, 0.26]),
measure("measure.altitude", "~408 km", [0.06, 0.06, 0.18, 0.08], 3),
],
"orbit_bluemarble": [
static_label("detected.globe", 4, ["Earth", "the globe", "terrestrial planet", "terrestrial planet · 12,742 km across"], [0.28, 0.18, 0.44, 0.6]),
],
# ---------- coast ----------
"coast_birdrock": [
static_label("detected.surf", 4, ["waves", "surf", "breaking swell", "breaking swell · wind-driven, ~10 s period"], [0.20, 0.55, 0.6, 0.3]),
static_label("detected.searock", 2, ["rock", "sea stack", "coastal sea stack", "sea stack · wave-cut residual rock"], [0.30, 0.25, 0.22, 0.3]),
],
"coast_surfgrass": [
static_label("detected.surfgrass", 4, ["grass", "surfgrass", "Phyllospadix", "Phyllospadix · a marine seagrass, not algae"], [0.18, 0.40, 0.5, 0.4]),
static_label("detected.coralline", 2, ["pink", "coralline algae", "crustose coralline", "crustose coralline · calcified red algae"], [0.6, 0.55, 0.2, 0.2]),
],
"coast_elkbeach": [
static_label("detected.elk", 4, ["elk", "Roosevelt elk", "Cervus canadensis roosevelti", "C. c. roosevelti · largest elk subspecies"], [0.34, 0.42, 0.3, 0.3]),
static_label("detected.fog", 2, ["fog", "coastal fog", "advection fog", "advection fog · warm air over cold upwelling"], [0.05, 0.05, 0.9, 0.25]),
],
"coast_drakesbeach": [
static_label("detected.seal", 4, ["seals", "elephant seals", "Mirounga angustirostris", "M. angustirostris · males to 2,000 kg"], [0.25, 0.55, 0.5, 0.3]),
static_label("detected.sand", 1, ["sand", "dark sand", "mineral-dark beach", "dark beach · eroded coastal sediment"], [0.05, 0.8, 0.9, 0.15]),
],
# ---------- reef (showcase: tracked + windowed) ----------
"reef_lionfish": [
tracked_label(
"detected.lionfish", 4,
["fish", "lionfish", "Pterois volitans", "Pterois volitans · venomous spines, invasive in the Atlantic"],
0.05, 0.95,
[(0.05, [0.15, 0.30, 0.16, 0.18]), (0.5, [0.50, 0.38, 0.16, 0.18]), (0.95, [0.72, 0.34, 0.16, 0.18])],
),
tracked_label(
"detected.snapper", 2,
["fish", "blue snapper", "Lutjanus", "Lutjanus · schools over reef structure"],
0.0, 0.6,
[(0.0, [0.05, 0.55, 0.12, 0.1]), (0.3, [0.22, 0.5, 0.12, 0.1]), (0.6, [0.40, 0.56, 0.12, 0.1])],
),
measure("measure.depth", "18 m", [0.06, 0.06, 0.16, 0.08], 3),
],
"reef_spawning": [
tracked_label(
"detected.school", 4,
["fish", "fish school", "snapper aggregation", "snapper aggregation · synchronized spawning run"],
0.0, 1.0,
[(0.0, [0.20, 0.30, 0.2, 0.18]), (0.5, [0.55, 0.34, 0.2, 0.18]), (1.0, [0.20, 0.30, 0.2, 0.18])],
),
static_label("detected.coral", 2, ["coral", "reef coral", "scleractinian", "scleractinian · reef-building stony coral"], [0.15, 0.6, 0.35, 0.3]),
measure("measure.depth", "22 m", [0.06, 0.06, 0.16, 0.08], 3),
],
"reef_hawkfish": [
tracked_label(
"detected.parrotfish", 4,
["fish", "parrotfish", "Bolbometopon muricatum", "B. muricatum · humphead, grazes reef into sand"],
0.1, 0.9,
[(0.1, [0.55, 0.25, 0.2, 0.22]), (0.5, [0.35, 0.34, 0.2, 0.22]), (0.9, [0.12, 0.40, 0.2, 0.22])],
),
measure("measure.depth", "12 m", [0.06, 0.06, 0.16, 0.08], 3),
],
"reef_snapper": [
tracked_label(
"detected.snapper", 4,
["fish", "red snapper", "Lutjanus campechanus", "L. campechanus · long-lived, can reach 50+ yr"],
0.0, 0.7,
[(0.0, [0.10, 0.3, 0.16, 0.14]), (0.35, [0.40, 0.36, 0.16, 0.14]), (0.7, [0.68, 0.30, 0.16, 0.14])],
),
],
"reef_coralspacific": [
static_label("detected.coral", 4, ["coral", "coral colony", "Pacific scleractinian", "Pacific scleractinian · a symbiosis with algae"], [0.2, 0.4, 0.4, 0.4]),
tracked_label(
"detected.spotfish", 2,
["fish", "spotted fish", "reef damselfish", "reef damselfish · territorial over its coral"],
0.2, 0.8,
[(0.2, [0.6, 0.25, 0.1, 0.09]), (0.5, [0.45, 0.3, 0.1, 0.09]), (0.8, [0.3, 0.26, 0.1, 0.09])],
),
],
# ---------- abyss (showcase: tracked + windowed; creatures drift through) ----------
"abyss_wow": [
tracked_label(
"detected.jelly", 4,
["jelly", "comb jelly", "Ctenophora", "Ctenophora · swims by beating rows of cilia"],
0.0, 0.55,
[(0.0, [0.05, 0.25, 0.14, 0.18]), (0.3, [0.30, 0.32, 0.14, 0.18]), (0.55, [0.52, 0.28, 0.14, 0.18])],
),
tracked_label(
"detected.siphonophore", 3,
["chain", "siphonophore", "Siphonophorae", "Siphonophorae · a colony of clones, can exceed 40 m"],
0.45, 1.0,
[(0.45, [0.80, 0.6, 0.1, 0.3]), (0.7, [0.6, 0.45, 0.1, 0.34]), (1.0, [0.45, 0.3, 0.1, 0.38])],
),
measure("measure.depth", "1,200 m", [0.06, 0.06, 0.16, 0.08], 2),
],
"abyss_midwaterexp": [
tracked_label(
"detected.medusa", 4,
["jelly", "white medusa", "Scyphozoa", "Scyphozoa · the swimming medusa stage of a jellyfish"],
0.0, 0.6,
[(0.0, [0.55, 0.20, 0.16, 0.2]), (0.3, [0.40, 0.34, 0.16, 0.2]), (0.6, [0.22, 0.30, 0.16, 0.2])],
),
tracked_label(
"detected.worm", 2,
["worm", "red worm", "polychaete", "polychaete · a free-swimming bristle worm"],
0.5, 1.0,
[(0.5, [0.10, 0.7, 0.12, 0.1]), (0.75, [0.3, 0.6, 0.12, 0.1]), (1.0, [0.5, 0.66, 0.12, 0.1])],
),
measure("measure.depth", "1,600 m", [0.06, 0.06, 0.16, 0.08], 2),
],
"abyss_hiding": [
tracked_label(
"detected.jelly", 4,
["jelly", "crimson jelly", "Scyphozoa", "Scyphozoa · red is invisible in the lightless deep"],
0.0, 0.7,
[(0.0, [0.15, 0.28, 0.16, 0.2]), (0.35, [0.42, 0.36, 0.16, 0.2]), (0.7, [0.66, 0.30, 0.16, 0.2])],
),
measure("measure.depth", "2,140 m", [0.06, 0.06, 0.16, 0.08], 2),
],
}
# Human-readable strings for affect/measurement keys are produced from the tiered
# data above; this maps measurement keys to nothing extra (their value IS the string).
def _affect_for_clip(scale: str) -> tuple[list, dict]:
"""Build the affect list + its strings for a scale's pool member."""
entries, strings = [], {}
for key, at, min_level, tiers in AFFECT[scale]:
entries.append({"key": key, "at": at, "min_level": min_level})
strings[key] = tiers
return entries, strings
def _labels_for_clip(clip_id: str) -> tuple[list, dict]:
"""Build the annotation list + its strings, lifting `_tiers` into the table."""
anns, strings = [], {}
for raw in LABELS.get(clip_id, []):
a = {k: v for k, v in raw.items() if k != "_tiers"}
anns.append(a)
strings[a["key"]] = raw["_tiers"]
return anns, strings
def _clip_entry(scale: str, clip_id: str) -> dict:
title, license_, source = META[clip_id]
anns, lab_strings = _labels_for_clip(clip_id)
affect, aff_strings = _affect_for_clip(scale)
return {
"id": clip_id,
"title": title,
"base_file": f"{clip_id}/base.mp4",
"license": license_,
"source": source,
"right_variants": {},
"annotations": anns,
"affect": affect,
"strings": {"en": {**lab_strings, **aff_strings}},
}
def build_manifest() -> dict:
"""Assemble the full rotating-pool manifest dict."""
clips = []
for scale in RING_ORDER:
for clip_id in POOLS[scale]:
clips.append(_clip_entry(scale, clip_id))
scales = [{"id": s, "clip_id": POOLS[s][0], "pool": POOLS[s]} for s in RING_ORDER]
transitions = []
n = len(RING_ORDER)
for i in range(n):
a, b = RING_ORDER[i], RING_ORDER[(i + 1) % n]
transitions.append({"file": f"transitions/{a}-{b}.mp4", "model": "placeholder-zoom"})
return {"clips": clips, "ring": {"scales": scales, "transitions": transitions}}
# --- Generate the per-edge transition clips (zoom morph + reversed companion) ---
def _ffmpeg() -> str:
if shutil.which("ffmpeg"):
return "ffmpeg"
import imageio_ffmpeg
return imageio_ffmpeg.get_ffmpeg_exe()
def _make_transition(ff: str, scale_a: str, scale_b: str) -> Path:
"""A zoom/warp morph between two scales, from their PRIMARY pool members'
bases (mirrors setup_scales_media.py but keyed by scale->primary). This is the
recipe behind the well-liked orbit-coast edge; generate_media() now applies it
uniformly to every edge so none carry stale footage from an earlier ring."""
a = MEDIA / POOLS[scale_a][0] / "base.mp4"
b = MEDIA / POOLS[scale_b][0] / "base.mp4"
out = MEDIA / "transitions" / f"{scale_a}-{scale_b}.mp4"
out.parent.mkdir(parents=True, exist_ok=True)
norm = "trim=0:3,setpts=PTS-STARTPTS,scale=1280:720,fps=25,setsar=1,format=yuv420p"
subprocess.run([
ff, "-y", "-i", str(a), "-i", str(b), "-filter_complex",
f"[0:v]{norm}[a];[1:v]{norm}[b];"
"[a][b]xfade=transition=zoomin:duration=1.5:offset=0.75,format=yuv420p[v]",
"-map", "[v]", "-an", str(out),
], check=True, capture_output=True)
return out
def _make_reverse(ff: str, forward: Path) -> Path:
"""The zoom-OUT companion of a forward (zoom-in) edge: the same clip played
backward, written alongside it as `<edge>.rev.mp4`. An ascending ring move
crosses its edge `reversed`; the renderer then plays this file, so zooming out
recedes from the current scale back to the higher one instead of zooming in."""
out = forward.with_suffix(".rev.mp4")
subprocess.run([
ff, "-y", "-i", str(forward), "-vf", "reverse", "-an", str(out),
], check=True, capture_output=True)
return out
def generate_media() -> None:
"""(Re)build EVERY ring-edge transition from the current real primary bases —
the orbit-coast recipe applied uniformly, overwriting any stale clips left from
an earlier ring — plus a reversed companion per edge for zoom-out moves."""
ff = _ffmpeg()
n = len(RING_ORDER)
for i in range(n):
a, b = RING_ORDER[i], RING_ORDER[(i + 1) % n]
fwd = _make_transition(ff, a, b)
_make_reverse(ff, fwd)
print(f"generated transitions/{a}-{b}.mp4 (+ .rev) from real bases")
def main(argv: list[str]) -> None:
manifest = build_manifest()
MANIFEST.write_text(json.dumps(manifest, indent=2, ensure_ascii=False) + "\n")
print(f"wrote {MANIFEST}{len(manifest['clips'])} clips, "
f"{len(manifest['ring']['scales'])} ring scales")
if "--media" in argv:
generate_media()
if __name__ == "__main__":
main(sys.argv[1:])
+37 -8
View File
@@ -73,6 +73,18 @@ def load_manifest(path: str | Path) -> list[Clip]:
return [_clip_from_dict(c) for c in data["clips"]]
def _scale_from_dict(s: dict[str, Any]) -> Scale:
"""A ring scale node, with its rotating clip pool (content-pipeline §11.1).
Accepts a `pool` list (the rotating model) and/or a legacy single `clip_id`.
The primary `clip_id` is the explicit one if given, else the first pool
member; `pool` defaults to `(clip_id,)` for a pre-pool manifest.
"""
pool = tuple(s.get("pool", []))
clip_id = s.get("clip_id") or (pool[0] if pool else "")
return Scale(id=s["id"], clip_id=clip_id, pool=pool)
def load_ring(path: str | Path) -> ScaleRing | None:
"""Load the optional `ring` section as a `ScaleRing`, or None if absent.
@@ -85,9 +97,7 @@ def load_ring(path: str | Path) -> ScaleRing | None:
ring = data.get("ring")
if not ring:
return None
scales = tuple(
Scale(id=s["id"], clip_id=s["clip_id"]) for s in ring.get("scales", [])
)
scales = tuple(_scale_from_dict(s) for s in ring.get("scales", []))
transitions = tuple(
Transition(file=t["file"], model=t.get("model", ""))
for t in ring.get("transitions", [])
@@ -97,27 +107,46 @@ def load_ring(path: str | Path) -> ScaleRing | None:
def ring_to_dict(ring: ScaleRing, clips: list[Clip]) -> dict:
"""JSON form of the ring for the API: ordered scales (with their clip title)
and the per-edge transitions."""
and the per-edge transitions.
Each scale carries its full rotating `pool` (clip id + title per member) plus
the primary `clip_id`/`title` for back-compat. The client lands on a member
chosen at the API boundary (`/api/ring/advance` → `target_clip_id`)."""
titles = {c.id: c.title for c in clips}
return {
"scales": [
{"id": s.id, "clip_id": s.clip_id, "title": titles.get(s.clip_id, s.id)}
{
"id": s.id,
"clip_id": s.clip_id,
"title": titles.get(s.clip_id, s.id),
"pool": [
{"clip_id": cid, "title": titles.get(cid, cid)}
for cid in s.members
],
}
for s in ring.scales
],
"transitions": [{"file": t.file, "model": t.model} for t in ring.transitions],
}
def ring_move_to_dict(move: RingMove, ring: ScaleRing) -> dict:
def ring_move_to_dict(
move: RingMove, ring: ScaleRing, chosen_clip_id: str | None = None
) -> dict:
"""JSON form of an encoder move: the landing scale's clip and the ordered
transition clips to play (with direction). `fast` flags a collapsed fast-spin
pass; the single step then carries `blended` so the renderer plays it quick."""
pass; the single step then carries `blended` so the renderer plays it quick.
`target_clip_id` is the pool member the player should load on landing. The
caller passes the random pick (`pick_clip_id(landed_scale, random.random())`,
content-pipeline §11.1); when omitted it falls back to the scale's primary
`clip_id` (deterministic — keeps pre-pool callers/tests working)."""
return {
"from_index": move.from_index,
"to_index": move.to_index,
"wrapped": move.wrapped,
"fast": move.fast,
"target_clip_id": scale_at(ring, move.to_index).clip_id,
"target_clip_id": chosen_clip_id or scale_at(ring, move.to_index).clip_id,
"steps": [
{
"edge": st.edge,
File diff suppressed because it is too large Load Diff
+8
View File
@@ -1,5 +1,13 @@
"""Generate cheap placeholder media for the full 5-scale RING (scales design §3).
SUPERSEDED (session 0016): the ring now uses real strict-PD footage in a ROTATING
POOL per scale, regenerated by `simulator/build_pool_manifest.py` (which also makes
the new coast-edge transition placeholders). This generator built the older
one-base-per-scale forest ring (cosmos/orbit/forest/reef/abyss) and is kept only
for reference / a clean-room placeholder rebuild; it does NOT match the current
pool manifest or ring order (cosmos → orbit → coast → reef → abyss).
Produces offline synthetic placeholders for all five ring scales — **cosmos**
(NASA/Hubble), **orbit** (NASA/ISS), **forest** (NPS/USGS), **reef** (NOAA Ocean
Exploration), and **abyss** (NOAA Ocean Exploration) — plus the short zoom/warp
+478 -24
View File
@@ -31,8 +31,10 @@ window.addEventListener("error", (e) =>
showError(`${e.message} @ ${(e.filename || "").split("/").pop()}:${e.lineno}`));
let clipsById = {}; // id -> clip manifest entry
let ring = null; // {scales:[{id,clip_id,title}], transitions:[...]} or null
let ring = null; // {scales:[{id,clip_id,title,pool:[...]}], transitions:[...]} or null
let serverRing = false; // true when /api/ring served a real ring (vs the synthesized fallback)
let ringIndex = 0; // current scale position on the ring
let activeClipId = null; // the pool member chosen for the current scale landing
let currentClipId = null; // base media currently loaded (reset to force a reload)
let busy = false; // true while a ring transition is playing
let lastOverlay = null; // {level, intensity} from the most-recent renderOverlay call; drives per-frame track animation
@@ -40,21 +42,137 @@ let lastOverlay = null; // {level, intensity} from the most-recent renderOver
async function loadData() {
const clips = (await (await fetch("/api/clips")).json()).clips || [];
clipsById = Object.fromEntries(clips.map((c) => [c.id, c]));
// Per-file content-hash tokens → appended to /media URLs as ?v=<hash> so a
// re-baked clip (new bytes, same path) gets a fresh URL the browser can't serve
// stale. Best-effort: an empty map just yields un-versioned URLs.
try { mediaVersions = (await (await fetch("/api/media-versions")).json()).versions || {}; }
catch (_) { mediaVersions = {}; }
const r = await fetch("/api/ring");
serverRing = r.ok;
ring = r.ok ? await r.json() : null;
if (!ring && clips.length) {
// No ring: single-clip mode — synthesize a 1-scale ring over clip 0.
ring = { scales: [{ id: clips[0].id, clip_id: clips[0].id, title: clips[0].title }], transitions: [] };
// No ring: single-clip mode — synthesize a 1-scale ring (pool of one) over clip 0.
ring = { scales: [{ id: clips[0].id, clip_id: clips[0].id, title: clips[0].title, pool: [{ clip_id: clips[0].id, title: clips[0].title }] }], transitions: [] };
}
ringIndex = 0;
}
function activeClip() {
if (!ring) return null;
return clipsById[ring.scales[ringIndex].clip_id] || null;
// Server-canonical random pick of a pool member for the current scale: a delta=0
// advance (content-pipeline §11.1, Python owns the randomness). The synthesized
// fallback ring has a pool of one, so it just returns that member. Returns a
// clip_id or null. Shared by the initial landing AND the Dev Mode re-roll button.
async function pickRandomMember() {
const scale = ring && ring.scales[ringIndex];
if (!scale) return null;
if (serverRing) {
try {
const resp = await fetch("/api/ring/advance", {
method: "POST", headers: { "content-type": "application/json" },
body: JSON.stringify({ from_index: ringIndex, delta: 0 }),
});
if (resp.ok) return (await resp.json()).target_clip_id;
} catch (_) { /* fall through to the primary member */ }
}
return scale.clip_id;
}
function mediaUrl(file) { return "/media/" + file; }
// Land on the current scale: pick a random pool member and force its media to load.
async function landScale() {
activeClipId = await pickRandomMember();
currentClipId = null;
}
function activeClip() {
return clipsById[activeClipId] || null;
}
// In-memory preload cache: media file path -> blob object URL. Once a clip's bytes
// are cached, mediaUrl() hands the <video> an in-memory blob instead of a network
// path, so swapping the altitude scale loads near-instantly (no fetch round-trip).
// Falls back to the network path for anything not yet (or never) cached.
const mediaBlobs = {};
let mediaVersions = {}; // file -> content-hash token (from /api/media-versions)
// Network path for a media file, content-hash-versioned so a re-baked clip's URL
// changes with its bytes (permanent cache-bust). The blob cache, when present,
// wins — those bytes are already the current ones for this session.
function mediaNetUrl(file) { const v = mediaVersions[file]; return "/media/" + file + (v ? "?v=" + v : ""); }
function mediaUrl(file) { return mediaBlobs[file] || mediaNetUrl(file); }
// Every media file the ring can show: each scale-pool clip's base footage plus the
// per-edge transition clips. This is the full preload set (~two dozen files).
function preloadList() {
const files = new Set();
for (const c of Object.values(clipsById)) if (c && c.base_file) files.add(c.base_file);
// Each ring edge ships both directions: the forward (zoom-in) clip and its baked
// reverse (zoom-out) companion — cache both so up and down moves are instant.
if (ring && ring.transitions) for (const t of ring.transitions) {
if (t && t.file) { files.add(t.file); files.add(reverseFile(t.file)); }
}
return [...files];
}
// Order the preload so the clips most likely to be shown next come first: the
// current scale's pool, then outward to neighboring scales, then transitions.
function preloadOrder() {
if (!ring || !ring.scales) return preloadList();
const n = ring.scales.length, seen = new Set(), ordered = [];
const add = (f) => { if (f && !seen.has(f)) { seen.add(f); ordered.push(f); } };
for (let d = 0; d < n; d++) {
for (const sign of d === 0 ? [0] : [1, -1]) {
const s = ring.scales[((ringIndex + sign * d) % n + n) % n];
if (s && s.pool) for (const m of s.pool) add((clipsById[m.clip_id] || {}).base_file);
else if (s) add((clipsById[s.clip_id] || {}).base_file);
}
}
if (ring.transitions) for (const t of ring.transitions) { add(t.file); add(reverseFile(t.file)); }
for (const f of preloadList()) add(f); // sweep up anything not on the ring
return ordered;
}
// Fetch every clip into the blob cache in the background — non-blocking, so the
// first scale plays immediately while the rest stream in. Bounded concurrency keeps
// the ~350 MB fetch from stampeding. A tiny progress chip self-removes when done.
async function preloadAllMedia(concurrency = 4) {
const files = preloadOrder();
if (!files.length) return;
let done = 0;
const total = files.length;
const chip = preloadChip();
const tick = () => { if (chip) chip.textContent = `⬇ caching clips ${done}/${total}`; };
tick();
let i = 0;
async function worker() {
while (i < files.length) {
const file = files[i++];
if (!mediaBlobs[file]) {
try {
// The content-hash `?v=` already makes a re-baked clip's URL unique, but
// `cache: "reload"` is belt-and-suspenders: it bypasses the HTTP cache for
// this fetch (so even an un-versioned or immutable-pinned prior entry can't
// serve stale) and refreshes the entry. The blob cache above still gives
// instant in-session swaps; this only affects the one fetch per (re)load.
const blob = await (await fetch(mediaNetUrl(file), { cache: "reload" })).blob();
mediaBlobs[file] = URL.createObjectURL(blob);
} catch (_) { /* leave it to the network path on demand */ }
}
done++; tick();
}
}
await Promise.all(Array.from({ length: Math.min(concurrency, files.length) }, worker));
if (chip) chip.remove();
}
function preloadChip() {
let c = document.getElementById("preload-chip");
if (!c) {
c = document.createElement("div");
c.id = "preload-chip";
c.style.cssText = "position:fixed;bottom:8px;right:10px;z-index:9998;background:rgba(0,0,0,.55);" +
"color:#9fe;font:11px/1.4 monospace;padding:4px 8px;border-radius:4px;pointer-events:none;";
document.body.appendChild(c);
}
return c;
}
// Load the active scale's BASE footage into the video (the painterly canvas reads
// its frames live). The Right knob no longer swaps the source — it only drives the
@@ -183,10 +301,12 @@ function paintLoop() {
gl.drawArrays(gl.TRIANGLES, 0, 3);
paint.style.filter = busy ? "none" : gradeFilter;
}
// Animate keyframed annotation tracks: re-place boxes against playback time.
// Animate per-frame: re-place keyframed tracks AND re-evaluate time-windowed
// labels (a label enters/leaves frame as the loop plays) against playback time.
const c = activeClip();
if (lastOverlay && lastOverlay.level > 0 && c &&
c.annotations.some((a) => a.track && a.track.length)) {
c.annotations.some((a) => (a.track && a.track.length) ||
typeof a.appear === "number" || typeof a.disappear === "number")) {
renderOverlay(lastOverlay.level, lastOverlay.intensity);
}
requestAnimationFrame(paintLoop);
@@ -261,17 +381,50 @@ function loopT() {
return vid && vid.duration ? (vid.currentTime % vid.duration) / vid.duration : 0;
}
// --- Progressive labels & emotions (content-pipeline §11.3 / §11.4) ---
const clamp = (v, lo, hi) => Math.max(lo, Math.min(hi, v));
// The Left level at which a label first appears. From `salience` (1..4): high
// salience appears early (first_level = 5 - salience), so raising Left brings in
// lower-salience objects. Falls back to a legacy flat `min_level`, else 1.
function firstLevel(a) {
if (typeof a.salience === "number") return clamp(5 - a.salience, 1, 4);
if (typeof a.min_level === "number") return a.min_level;
return 1;
}
// A tiered string escalates with a knob: a LIST is indexed by tier-1 (clamped to
// the available tiers); a plain string is static (tier ignored — back-compat).
function pickTier(value, tier) {
if (Array.isArray(value)) return value[clamp(tier - 1, 0, value.length - 1)];
return value;
}
function tierCount(value) { return Array.isArray(value) ? value.length : 1; }
// Time-windowed presence (§11.2): a label with `appear`/`disappear` (loop-norm t)
// shows only while on screen. A `disappear < appear` window wraps across the loop
// seam (visible when t >= appear OR t <= disappear). No window => always present.
function inWindow(a, t) {
const hasA = typeof a.appear === "number", hasD = typeof a.disappear === "number";
if (!hasA && !hasD) return true;
const ap = hasA ? a.appear : 0, dis = hasD ? a.disappear : 1;
return ap <= dis ? (t >= ap && t <= dis) : (t >= ap || t <= dis);
}
function renderOverlay(level, intensity) {
const clip = activeClip();
overlay.innerHTML = "";
if (!clip || level <= 0) { overlay.style.opacity = "0"; return; }
overlay.style.opacity = String(intensity);
const strings = (clip.strings && clip.strings.en) || {};
const t = loopT();
let shown = 0;
for (const a of clip.annotations) {
if (a.min_level > level) continue;
const first = firstLevel(a);
if (level < first) continue; // salience gating: low Left = fewer/general labels
if (!inWindow(a, t)) continue; // time-windowed: only while the object is on screen
shown++;
const [x, y, w, h] = boxAt(a, loopT()).map((n) => n * 100);
const [x, y, w, h] = boxAt(a, t).map((n) => n * 100);
const measure = a.key.startsWith("measure.");
const kind = measure ? "measure" : "detect";
reticle(x, y, w, h, "hud-reticle " + kind, overlay);
@@ -281,7 +434,12 @@ function renderOverlay(level, intensity) {
svg("line", { x1: cx - 1, y1: cyc, x2: cx + 1, y2: cyc, class: "hud-reticle measure" }, overlay);
svg("line", { x1: cx, y1: cyc - 1, x2: cx, y2: cyc + 1, class: "hud-reticle measure" }, overlay);
}
chip(x, y, strings[a.key] || a.key, measure ? null : confidence(a.key), kind, overlay);
// LEFT detail tier: a detection escalates general -> scientific+fact as Left
// rises above where it first appeared; measurements stay single-tier.
const raw = strings[a.key];
const tier = measure ? 1 : clamp(level - first + 1, 1, tierCount(raw));
const label = pickTier(raw !== undefined ? raw : a.key, tier);
chip(x, y, label, measure ? null : confidence(a.key), kind, overlay);
}
// Global status tag — the machine announcing it is analyzing, escalating with Left.
// Right-anchored at the frame edge; the plate is sized from the text's real bbox
@@ -302,7 +460,7 @@ function renderOverlay(level, intensity) {
// up. `strength` = min(left, right); `intensity` is the layer opacity. Words are
// placed at authored scene points (no boxes — feelings are scene-level) and read
// softer than the clinical reticles — the dream leaking into the machine's read.
function renderAffect(strength, intensity) {
function renderAffect(strength, intensity, right) {
const clip = activeClip();
if (!affectLayer) return; // tolerate a stale page missing the affect layer
affectLayer.innerHTML = "";
@@ -313,14 +471,23 @@ function renderAffect(strength, intensity) {
if (f.min_level > strength) continue;
const [x, y] = f.at.map((n) => n * 100);
const t = svg("text", { x, y, "text-anchor": "middle", class: "hud-affect" }, affectLayer);
t.textContent = strings[f.key] || f.key;
// RIGHT emotion tier: the vocabulary escalates basic -> compound as the Right
// knob rises (appearance still gated by min(left,right) above).
const raw = strings[f.key];
t.textContent = pickTier(raw !== undefined ? raw : f.key, clamp(right || 0, 1, tierCount(raw)));
}
}
function renderScaleReadout() {
if (!ring) return;
const s = ring.scales[ringIndex];
$("scale-name").textContent = `${s.title} (${ringIndex + 1}/${ring.scales.length})`;
const poolN = (s.pool && s.pool.length) || 1;
const member = (activeClip() && activeClip().title) || s.title;
// scale id · the chosen pool member · position on the ring (pool size if >1)
const poolTag = poolN > 1 ? ` · pool ${poolN}` : "";
$("scale-name").textContent = `${s.id} · ${member} (${ringIndex + 1}/${ring.scales.length}${poolTag})`;
renderDial();
refreshDevClip(); // keep the Dev Mode pool picker + clip data in sync with the landing
}
function controls() {
@@ -346,13 +513,13 @@ async function update() {
if (!resp.ok) { readout.textContent = "invalid: " + resp.status; return; }
const data = await resp.json();
readout.textContent = JSON.stringify(data, null, 2);
if (!data.content.video) { black.classList.remove("hidden"); return; }
if (!data.content.video) { black.style.opacity = "1"; black.classList.remove("hidden"); return; }
black.classList.add("hidden");
try {
ensureClipMedia();
applyVideoLook(data.plan.grade.tone, data.plan.dream.intensity);
renderOverlay(data.plan.overlay.level, data.plan.overlay.intensity);
renderAffect(data.plan.affect.strength, data.plan.affect.intensity);
renderAffect(data.plan.affect.strength, data.plan.affect.intensity, data.plan.dream.strength);
const b = document.getElementById("err-banner");
if (b) b.remove(); // render succeeded — clear any prior error
} catch (err) {
@@ -367,11 +534,17 @@ function debounced() { clearTimeout(timer); timer = setTimeout(update, 80); }
const FAST_BLEND_RATE = 2.5; // playback speed for a collapsed fast-spin pass
function playTransition(file, blended) {
// The zoom-OUT companion of an edge clip: the baked `<edge>.rev.mp4` reverse. A
// `reversed` step (an ascending / zoom-out ring move) plays this instead of the
// forward zoom-in clip, so going up recedes from the current scale.
function reverseFile(file) { return file.replace(/\.mp4$/, ".rev.mp4"); }
function playTransition(file, blended, reversed) {
// Play one zoom/warp transition clip start-to-finish, with the alteration
// layers hidden. A `blended` (fast-spin) pass runs at FAST_BLEND_RATE so a
// quick encoder spin lands fast instead of grinding through every morph.
// Resolves on 'ended' (or a safety timeout that scales with playback rate).
// Zoom-out moves (`reversed`) play the baked reverse clip. Resolves on 'ended'
// (or a safety timeout that scales with playback rate).
return new Promise((resolve) => {
overlay.style.opacity = "0";
affectLayer.style.opacity = "0";
@@ -379,7 +552,7 @@ function playTransition(file, blended) {
vid.style.filter = "none";
vid.loop = false;
vid.style.opacity = "1";
vid.src = mediaUrl(file);
vid.src = mediaUrl(reversed ? reverseFile(file) : file);
vid.playbackRate = blended ? FAST_BLEND_RATE : 1;
let done = false;
const finish = () => {
@@ -395,6 +568,29 @@ function playTransition(file, blended) {
});
}
// Briefly fade through the #black overlay while `swap` changes the displayed clip,
// so a content swap reads as a deliberate cut, not a jarring jump. Restores #black
// to its hidden/opaque resting state (the video-off use) afterward.
function fadeThroughBlack(swap) {
return new Promise((resolve) => {
black.style.opacity = "0";
black.classList.remove("hidden");
void black.offsetWidth; // reflow so the 0 -> 1 transition runs
black.style.opacity = "1";
setTimeout(() => {
try { swap(); } catch (_) {}
setTimeout(() => {
black.style.opacity = "0";
setTimeout(() => {
black.classList.add("hidden");
black.style.opacity = ""; // back to CSS default for the video-off use
resolve();
}, 210);
}, 160); // let the chosen clip decode under black
}, 210);
});
}
async function advance(delta) {
if (busy || !ring || ring.scales.length < 2) return;
busy = true;
@@ -408,11 +604,20 @@ async function advance(delta) {
// A fast spin comes back collapsed to a single blended step (move.fast);
// a slow spin chains one full transition per scale crossed.
for (const step of move.steps) {
await playTransition(step.file, step.blended);
await playTransition(step.file, step.blended, step.reversed);
}
ringIndex = move.to_index;
currentClipId = null; // force the target scale's base media to (re)load
activeClipId = move.target_clip_id; // the randomly-picked pool member to load
renderScaleReadout();
// The baked transition lands on the scale PRIMARY; if the rotating pool picked a
// DIFFERENT member, mask the swap behind a fade to black so it doesn't hard-cut
// from the primary to the chosen clip (e.g. coast birdrock -> surfgrass).
const landedPrimary = (ring.scales[move.to_index] || {}).clip_id;
if (move.target_clip_id && move.target_clip_id !== landedPrimary) {
await fadeThroughBlack(() => { currentClipId = null; ensureClipMedia(); });
} else {
currentClipId = null; // same clip the transition ended on — plain (re)load
}
} finally {
busy = false;
update();
@@ -431,6 +636,248 @@ function onWheel(e) {
}, 90);
}
// --- Altitude knob: the endless rotary encoder, drawn as a turnable dial ---
// cosmos (highest) sits at the top; turning CLOCKWISE descends through the scales
// (cosmos → orbit → coast → reef → abyss) and past the deepest WRAPS back to the
// top — the same endless ring the server already models. The drag accumulates a
// rotation and commits whole detents on release (so a big spin becomes one fast
// blended pass, exactly like the wheel); a tap on a label jumps to that scale.
const dial = $("dial");
const DIAL_C = 50, DIAL_LABEL_R = 41, DIAL_BODY_R = 27;
let needleDeg = 0;
let dialDrag = null; // {lastAng, accum, moved} while turning
function dialStep() { return ring && ring.scales.length ? 360 / ring.scales.length : 360; }
function _xy(r, deg) {
const a = (deg - 90) * Math.PI / 180; // -90° so 0° points UP
return [DIAL_C + r * Math.cos(a), DIAL_C + r * Math.sin(a)];
}
function buildDial() {
if (!dial || !ring) return;
dial.innerHTML = "";
const n = ring.scales.length, step = 360 / n;
svg("circle", { cx: DIAL_C, cy: DIAL_C, r: DIAL_LABEL_R + 6, class: "dial-rim" }, dial);
svg("circle", { cx: DIAL_C, cy: DIAL_C, r: DIAL_BODY_R, class: "dial-body" }, dial);
for (let i = 0; i < n; i++) {
const deg = i * step;
const [tx0, ty0] = _xy(DIAL_BODY_R - 1.5, deg);
const [tx1, ty1] = _xy(DIAL_BODY_R + 2.5, deg);
svg("line", { x1: tx0, y1: ty0, x2: tx1, y2: ty1, class: "dial-tick" }, dial);
const [lx, ly] = _xy(DIAL_LABEL_R, deg);
const t = svg("text", {
x: lx, y: ly, "text-anchor": "middle", "dominant-baseline": "central",
class: "dial-label", "data-index": String(i),
}, dial);
t.textContent = ring.scales[i].id;
}
svg("text", { x: DIAL_C, y: DIAL_C - 7, "text-anchor": "middle",
"dominant-baseline": "central", class: "dial-caption" }, dial)
.textContent = "ALTITUDE";
const needle = svg("g", { id: "needle", class: "dial-needle" }, dial);
const tip = DIAL_C - (DIAL_BODY_R - 4);
svg("polygon", { points: `${DIAL_C - 2.4},${DIAL_C} ${DIAL_C + 2.4},${DIAL_C} ${DIAL_C},${tip}` }, needle);
svg("circle", { cx: DIAL_C, cy: DIAL_C, r: 3, class: "dial-hub" }, dial);
renderDial();
}
function setNeedle(deg) {
const needle = $("needle");
if (needle) needle.setAttribute("transform", `rotate(${deg} ${DIAL_C} ${DIAL_C})`);
}
function renderDial() {
if (!dial || !ring) return;
needleDeg = ringIndex * dialStep();
setNeedle(needleDeg);
for (const el of dial.querySelectorAll(".dial-label")) {
el.classList.toggle("active", +el.getAttribute("data-index") === ringIndex);
}
}
function dialAngle(e) {
const r = dial.getBoundingClientRect();
const dx = e.clientX - (r.left + r.width / 2);
const dy = e.clientY - (r.top + r.height / 2);
return (Math.atan2(dx, -dy) * 180 / Math.PI + 360) % 360; // 0 at top, clockwise +
}
function angDelta(a, b) {
let d = a - b;
while (d > 180) d -= 360;
while (d < -180) d += 360;
return d;
}
function onDialDown(e) {
if (busy || !ring || ring.scales.length < 2) return;
e.preventDefault();
dialDrag = { lastAng: dialAngle(e), accum: 0, moved: 0, target: e.target };
}
function onDialMove(e) {
if (!dialDrag) return;
const a = dialAngle(e);
const d = angDelta(a, dialDrag.lastAng);
dialDrag.lastAng = a;
dialDrag.accum += d;
dialDrag.moved += Math.abs(d);
setNeedle(ringIndex * dialStep() + dialDrag.accum); // live feedback while turning
}
function onDialUp(e) {
if (!dialDrag) return;
const { accum, moved, target } = dialDrag;
dialDrag = null;
if (moved < 6) { // a tap, not a turn
if (target && target.classList && target.classList.contains("dial-label")) {
jumpToScale(+target.getAttribute("data-index"));
}
renderDial();
return;
}
const detents = Math.round(accum / dialStep());
if (detents) advance(detents); else renderDial(); // snap back if it didn't cross a detent
}
// Click a label → travel the SHORTEST signed way around the ring to that scale.
function jumpToScale(idx) {
if (!ring) return;
const n = ring.scales.length;
let d = (idx - ringIndex) % n;
if (d > n / 2) d -= n;
if (d < -n / 2) d += n;
if (d) advance(d);
}
// --- Dev Mode: pool picker + live analysis, all under one toggle ---
// Off by default, state persisted in localStorage. Everything here is read from
// data the renderer already has (clips, ring, the alteration response, the preload
// cache) — no server endpoints. Editing labels stays in /author.html.
const DEV_KEY = "hef.devMode";
let devMode = false;
let devStatsTimer = null;
const escapeHtml = (s) => String(s).replace(/[&<>"]/g,
(c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;" }[c]));
function applyDevVisibility() {
const panel = $("dev-panel");
if (panel) panel.classList.toggle("hidden", !devMode);
if (devMode) { refreshDevClip(); renderDevStats(); }
}
function initDev() {
const toggle = $("dev-mode");
if (!toggle) return;
try { devMode = localStorage.getItem(DEV_KEY) === "1"; } catch (_) { devMode = false; }
toggle.checked = devMode;
toggle.addEventListener("change", () => {
devMode = toggle.checked;
try { localStorage.setItem(DEV_KEY, devMode ? "1" : "0"); } catch (_) {}
applyDevVisibility();
});
$("pool-select").addEventListener("change", onPoolSelect);
$("pool-reroll").addEventListener("click", reroll);
// Cache + playback stats are live-ish; cheap to repaint twice a second when shown.
devStatsTimer = setInterval(() => { if (devMode) renderDevStats(); }, 500);
applyDevVisibility();
}
// Rebuild the pool picker for the current scale and repaint the clip data. Cheap
// (pools are <= a handful), so just rebuild on every landing/pick.
function refreshDevClip() {
if (!devMode) return;
const sel = $("pool-select");
const scale = ring && ring.scales[ringIndex];
if (sel && scale) {
const pool = (scale.pool && scale.pool.length) ? scale.pool : [{ clip_id: scale.clip_id, title: scale.title }];
sel.innerHTML = "";
for (const m of pool) {
const o = document.createElement("option");
o.value = m.clip_id;
o.textContent = `${m.clip_id} · ${m.title || (clipsById[m.clip_id] || {}).title || ""}`;
sel.appendChild(o);
}
if (activeClipId) sel.value = activeClipId;
sel.disabled = pool.length < 2;
}
renderDevMeta();
renderDevAnno();
}
function renderDevMeta() {
const dl = $("clip-meta");
const c = activeClip();
if (!dl) return;
if (!c) { dl.innerHTML = "<dt>—</dt><dd>no clip</dd>"; return; }
const rows = [["id", c.id], ["title", c.title], ["source", c.source],
["license", c.license], ["base_file", c.base_file]];
dl.innerHTML = rows.map(([k, v]) =>
`<dt>${escapeHtml(k)}</dt><dd>${escapeHtml(v != null ? v : "—")}</dd>`).join("");
}
function renderDevAnno() {
const box = $("clip-anno");
const c = activeClip();
if (!box) return;
const strings = (c && c.strings && c.strings.en) || {};
const win = (a) => (typeof a.appear === "number" || typeof a.disappear === "number")
? ` win ${(a.appear ?? 0).toFixed(2)}${(a.disappear ?? 1).toFixed(2)}` : "";
const rows = [];
for (const a of (c && c.annotations) || []) {
const measure = a.key.startsWith("measure.");
const sal = typeof a.salience === "number" ? `sal ${a.salience}`
: typeof a.min_level === "number" ? `min L${a.min_level}` : "sal 1";
const tiers = tierCount(strings[a.key]);
rows.push(`<div class="anno-row"><span class="anno-key ${measure ? "measure" : ""}">${escapeHtml(a.key)}</span>` +
`<span class="anno-meta"> · ${sal} · ${tiers} tier${tiers > 1 ? "s" : ""}${win(a)}</span></div>`);
}
for (const f of (c && c.affect) || []) {
const tiers = tierCount(strings[f.key]);
rows.push(`<div class="anno-row"><span class="anno-key affect">${escapeHtml(f.key)}</span>` +
`<span class="anno-meta"> · min L${f.min_level ?? 1} · ${tiers} tier${tiers > 1 ? "s" : ""}</span></div>`);
}
box.innerHTML = rows.length ? rows.join("") : `<div class="anno-empty">no annotations or affect</div>`;
}
function renderDevStats() {
const dl = $("dev-stats");
if (!dl) return;
const c = activeClip();
const total = preloadList().length;
const cached = Object.keys(mediaBlobs).length;
const fromMem = !!(c && mediaBlobs[c.base_file]);
const res = vid.videoWidth ? `${vid.videoWidth}×${vid.videoHeight}` : "—";
const dur = vid.duration ? `${vid.duration.toFixed(1)}s` : "—";
const loop = vid.duration ? `${Math.round(loopT() * 100)}%` : "—";
dl.innerHTML =
`<dt>cache</dt><dd>${cached}/${total} cached</dd>` +
`<dt>active source</dt><dd class="${fromMem ? "mem" : "net"}">${fromMem ? "memory (blob)" : "network"}</dd>` +
`<dt>resolution</dt><dd>${res}</dd>` +
`<dt>duration</dt><dd>${dur}</dd>` +
`<dt>loop pos</dt><dd>${loop}</dd>`;
}
// Pick a specific pool member (Dev Mode override of the random landing pick).
function onPoolSelect() {
const id = $("pool-select").value;
if (!id || id === activeClipId) return;
activeClipId = id;
currentClipId = null; // force the base media to reload
renderScaleReadout(); // refresh scale name + dev clip data
update(); // reload media + plan
}
// Re-roll: a fresh server-canonical random pick for the current scale.
async function reroll() {
if (busy) return;
const id = await pickRandomMember();
if (!id) return;
activeClipId = id;
currentClipId = null;
renderScaleReadout();
update();
}
// Dev live-reload: poll the asset version and reload when it changes, so an open
// tab never keeps running a stale renderer while we iterate (the readout updates
// from the live API and masks it otherwise). Reloads on my edits AND on a server
@@ -450,12 +897,19 @@ async function main() {
devLiveReload();
try { initPaint(); } catch (e) { paintOK = false; paint.style.display = "none"; showError("WebGL init: " + e.message); }
await loadData();
await landScale(); // pick the initial scale's pool member before first render
preloadAllMedia(); // background: cache every clip in memory for instant altitude swaps
buildDial(); // draw the altitude knob from the ring's scales
initDev(); // wire the Dev Mode toggle + pool picker (reads persisted state)
renderScaleReadout();
for (const id of ["content", "left", "right", "mood"]) {
$(id).addEventListener("input", debounced);
}
$("zoom-in").addEventListener("click", () => advance(1));
$("zoom-out").addEventListener("click", () => advance(-1));
// Altitude knob: drag to turn (commit detents on release), scroll to step, tap a label to jump.
dial.addEventListener("pointerdown", onDialDown);
window.addEventListener("pointermove", onDialMove);
window.addEventListener("pointerup", onDialUp);
dial.addEventListener("wheel", onWheel, { passive: false });
$("stage").addEventListener("wheel", onWheel, { passive: false });
update();
}
+64
View File
@@ -0,0 +1,64 @@
/* Author-mode-only styling (content-pipeline §11.5). Reuses style.css for the
shared stage/panel chrome; this adds the box-draw layer + editor widgets. */
.author #draw {
position: absolute;
inset: 0;
width: 100%;
height: 100%;
cursor: crosshair;
}
.author #draw .seed-box {
fill: rgba(80, 200, 255, 0.12);
stroke: #50c8ff;
stroke-width: 0.4;
vector-effect: non-scaling-stroke;
}
.author #draw .track-box {
fill: rgba(120, 255, 160, 0.10);
stroke: #78ffa0;
stroke-width: 0.4;
stroke-dasharray: 1.5 1;
vector-effect: non-scaling-stroke;
}
.author #draw .affect-dot {
fill: rgba(190, 150, 255, 0.85);
stroke: #fff;
stroke-width: 0.25;
vector-effect: non-scaling-stroke;
}
.author .scrub {
display: flex;
align-items: center;
gap: 8px;
margin-top: 8px;
}
.author .scrub input[type="range"] { flex: 1; }
.author .mono { font-family: ui-monospace, monospace; font-size: 12px; opacity: 0.85; }
.author .row { display: flex; gap: 8px; margin-top: 6px; flex-wrap: wrap; }
.author fieldset label { display: block; margin: 4px 0; font-size: 13px; }
.author fieldset input[type="text"],
.author fieldset input[type="number"] { width: 100%; box-sizing: border-box; }
.author fieldset input[type="number"] { width: 5em; }
.author ul.authored { list-style: none; padding: 0; margin: 6px 0; }
.author ul.authored li {
font-size: 12px;
padding: 3px 6px;
margin: 2px 0;
background: rgba(255, 255, 255, 0.05);
border-radius: 3px;
display: flex;
justify-content: space-between;
gap: 6px;
}
.author ul.authored .rm {
background: none;
border: none;
color: #f88;
cursor: pointer;
font-size: 12px;
}
.author header .hint { font-size: 12px; opacity: 0.8; max-width: 70ch; }
+79
View File
@@ -0,0 +1,79 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>HEF — Label Author Mode</title>
<link rel="stylesheet" href="/style.css" />
<link rel="stylesheet" href="/author.css" />
</head>
<body class="author">
<header><h1>HEF — Label Author Mode</h1>
<p class="hint">Scrub a pool clip · drag a box on a subject · run the tracker · author the tiers · save to the manifest. Geometry is propagated; <strong>semantics are hand-authored</strong> (content-pipeline §11.5). <a href="/">← back to preview</a></p>
</header>
<main>
<section class="stage" id="stage">
<div class="screen">
<video id="vid" muted playsinline></video>
<svg id="draw" viewBox="0 0 100 100" preserveAspectRatio="none"></svg>
</div>
<div class="scrub">
<button type="button" id="play">▶/⏸</button>
<input type="range" id="seek" min="0" max="1000" value="0" />
<span id="time" class="mono">0.00</span>
</div>
</section>
<section class="panel">
<fieldset>
<legend>Clip</legend>
<select id="clip"></select>
<p class="hint" id="clip-meta"></p>
</fieldset>
<fieldset>
<legend>Seed box (drag on the video)</legend>
<p class="mono" id="seed-readout">no box — drag on the stage</p>
<label>label key <input type="text" id="ann-key" placeholder="detected.jelly" /></label>
<label>salience (14; 4 = shows first/at low Left)
<input type="number" id="ann-salience" min="1" max="4" value="4" /></label>
<label>tier 1 — general <input type="text" id="t1" placeholder="jelly" /></label>
<label>tier 2 — specific <input type="text" id="t2" placeholder="comb jelly" /></label>
<label>tier 3 — scientific <input type="text" id="t3" placeholder="Ctenophora" /></label>
<label>tier 4 — +fact <input type="text" id="t4" placeholder="Ctenophora · beats rows of cilia" /></label>
<label>keyframes <input type="number" id="ann-keyframes" min="2" max="20" value="5" /></label>
<div class="row">
<button type="button" id="run-track">Run tracker → track</button>
<button type="button" id="add-static">Add as static box</button>
</div>
<p class="mono" id="track-readout"></p>
<button type="button" id="add-ann" disabled>+ Add label</button>
</fieldset>
<fieldset>
<legend>Affect anchor (click the stage to place)</legend>
<p class="mono" id="affect-readout">no point — shift-click the stage</p>
<label>feel key <input type="text" id="aff-key" placeholder="feel.unease" /></label>
<label>min strength (14) <input type="number" id="aff-min" min="1" max="4" value="1" /></label>
<label>tier 1 — basic <input type="text" id="a1" placeholder="uh" /></label>
<label>tier 2 <input type="text" id="a2" placeholder="unease" /></label>
<label>tier 3 <input type="text" id="a3" placeholder="disquiet" /></label>
<label>tier 4 — compound <input type="text" id="a4" placeholder="a creeping disquiet" /></label>
<button type="button" id="add-aff" disabled>+ Add emotion</button>
</fieldset>
<fieldset>
<legend>Authored for this clip</legend>
<ul id="ann-list" class="authored"></ul>
<ul id="aff-list" class="authored"></ul>
<div class="row">
<button type="button" id="load-existing">Load existing</button>
<button type="button" id="save">Save to manifest</button>
</div>
<p class="mono" id="save-readout"></p>
</fieldset>
</section>
</main>
<script src="/author.js"></script>
</body>
</html>
+254
View File
@@ -0,0 +1,254 @@
// Label author mode (content-pipeline §11.5): scrub a pool clip, drag a seed box
// on a subject, run the classical tracker to propagate a keyframed track, author
// the LEFT detail tiers + RIGHT emotion tiers, and save the entry to the manifest.
// Geometry comes from the tracker; SEMANTICS (keys, strings, tiers) are hand-typed.
const $ = (id) => document.getElementById(id);
const vid = $("vid"), draw = $("draw");
const SVGNS = "http://www.w3.org/2000/svg";
let clips = [];
let currentClip = null;
let seedBox = null; // [x,y,w,h] normalized, from a drag
let pendingTrack = null; // {track, appear, disappear} from the tracker (or static)
let affectPoint = null; // [x,y] normalized, from a shift-click
let annotations = []; // authored annotation dicts (carry _tiers)
let affect = []; // authored affect dicts (carry _tiers)
function svg(tag, attrs, parent) {
const el = document.createElementNS(SVGNS, tag);
for (const k in attrs) el.setAttribute(k, attrs[k]);
if (parent) parent.appendChild(el);
return el;
}
const mono = (n) => Number(n).toFixed(3);
async function loadClips() {
clips = (await (await fetch("/api/clips")).json()).clips || [];
const sel = $("clip");
sel.innerHTML = "";
for (const c of clips) {
const o = document.createElement("option");
o.value = c.id; o.textContent = c.id;
sel.appendChild(o);
}
if (clips.length) selectClip(clips[0].id);
}
function selectClip(id) {
currentClip = clips.find((c) => c.id === id) || null;
if (!currentClip) return;
$("clip-meta").textContent = `${currentClip.title}${currentClip.license}`;
vid.src = "/media/" + currentClip.base_file;
vid.currentTime = 0;
annotations = []; affect = []; seedBox = null; pendingTrack = null; affectPoint = null;
renderLists(); renderDrawLayer();
$("seed-readout").textContent = "no box — drag on the stage";
$("track-readout").textContent = "";
$("affect-readout").textContent = "no point — shift-click the stage";
}
// --- scrub ---
function seekT() { return vid.duration ? vid.currentTime / vid.duration : 0; }
$("play").addEventListener("click", () => { vid.paused ? vid.play() : vid.pause(); });
$("seek").addEventListener("input", () => {
if (vid.duration) vid.currentTime = (+$("seek").value / 1000) * vid.duration;
});
vid.addEventListener("timeupdate", () => {
if (!vid.duration) return;
$("seek").value = String(Math.round(seekT() * 1000));
$("time").textContent = seekT().toFixed(3);
});
// --- draw a seed box / place an affect point on the stage ---
function evtNorm(e) {
const r = draw.getBoundingClientRect();
return [(e.clientX - r.left) / r.width, (e.clientY - r.top) / r.height];
}
let dragStart = null;
draw.addEventListener("mousedown", (e) => {
if (e.shiftKey) { // shift-click = affect anchor
affectPoint = evtNorm(e).map((v) => Math.min(Math.max(v, 0), 1));
$("affect-readout").textContent = `affect @ ${affectPoint.map(mono).join(", ")}`;
$("add-aff").disabled = false;
renderDrawLayer();
return;
}
dragStart = evtNorm(e);
});
draw.addEventListener("mousemove", (e) => {
if (!dragStart) return;
seedBox = boxFrom(dragStart, evtNorm(e));
renderDrawLayer();
});
window.addEventListener("mouseup", (e) => {
if (!dragStart) return;
seedBox = boxFrom(dragStart, evtNorm(e));
dragStart = null;
pendingTrack = null;
$("seed-readout").textContent = `seed [${seedBox.map(mono).join(", ")}] @ t=${seekT().toFixed(3)}`;
$("add-ann").disabled = false;
renderDrawLayer();
});
function boxFrom(a, b) {
const x = Math.min(a[0], b[0]), y = Math.min(a[1], b[1]);
const w = Math.abs(b[0] - a[0]), h = Math.abs(b[1] - a[1]);
const cl = (v) => Math.min(Math.max(v, 0), 1);
return [cl(x), cl(y), Math.min(w, 1 - cl(x)), Math.min(h, 1 - cl(y))];
}
function renderDrawLayer() {
draw.innerHTML = "";
if (seedBox) {
const [x, y, w, h] = seedBox.map((n) => n * 100);
svg("rect", { x, y, width: w, height: h, class: "seed-box" }, draw);
}
if (pendingTrack && pendingTrack.track && vid.duration) {
const b = boxAt(pendingTrack.track, seekT());
if (b) {
const [x, y, w, h] = b.map((n) => n * 100);
svg("rect", { x, y, width: w, height: h, class: "track-box" }, draw);
}
}
if (affectPoint) {
const [x, y] = affectPoint.map((n) => n * 100);
svg("circle", { cx: x, cy: y, r: 1.2, class: "affect-dot" }, draw);
}
}
// Interpolate a track box at loop-normalized t (mirrors the preview renderer).
function boxAt(track, t) {
if (!track || !track.length) return null;
if (t <= track[0].t) return track[0].box;
if (t >= track[track.length - 1].t) return track[track.length - 1].box;
for (let i = 1; i < track.length; i++) {
if (t <= track[i].t) {
const a = track[i - 1], b = track[i], f = (t - a.t) / (b.t - a.t);
return a.box.map((v, j) => v + (b.box[j] - v) * f);
}
}
return track[track.length - 1].box;
}
// Animate the pending track preview while playing.
function previewLoop() { renderDrawLayer(); requestAnimationFrame(previewLoop); }
// --- run the tracker on the seed box ---
$("run-track").addEventListener("click", async () => {
if (!seedBox || !currentClip) { $("track-readout").textContent = "draw a seed box first"; return; }
const key = $("ann-key").value.trim() || "detected.object";
$("track-readout").textContent = "tracking…";
const resp = await fetch("/api/author/track", {
method: "POST", headers: { "content-type": "application/json" },
body: JSON.stringify({
clip_id: currentClip.id, key, seed_box: seedBox, seed_t: seekT(),
salience: +$("ann-salience").value, n_keyframes: +$("ann-keyframes").value,
}),
});
if (!resp.ok) { $("track-readout").textContent = "tracker error " + resp.status; return; }
const ann = await resp.json();
pendingTrack = { track: ann.track, appear: ann.appear, disappear: ann.disappear };
$("track-readout").textContent =
`track: ${ann.track.length} keyframes · window ${mono(ann.appear)}${mono(ann.disappear)}`;
$("add-ann").disabled = false;
});
// Add the current seed as a STATIC (untracked) box at the scrub position.
$("add-static").addEventListener("click", () => {
if (!seedBox) return;
pendingTrack = { static: true, box: seedBox.slice() };
$("track-readout").textContent = "static box (no track) at the drawn position";
$("add-ann").disabled = false;
});
function tiers(ids) {
const vals = ids.map((id) => $(id).value.trim());
while (vals.length && !vals[vals.length - 1]) vals.pop(); // drop empty trailing tiers
return vals.length ? vals : null;
}
$("add-ann").addEventListener("click", () => {
const key = $("ann-key").value.trim();
if (!key) { $("track-readout").textContent = "a label key is required"; return; }
const t = tiers(["t1", "t2", "t3", "t4"]);
const ann = { key, salience: +$("ann-salience").value, _tiers: t || key };
if (pendingTrack && pendingTrack.track) {
ann.track = pendingTrack.track;
ann.appear = pendingTrack.appear; ann.disappear = pendingTrack.disappear;
} else if (pendingTrack && pendingTrack.static) {
ann.box = pendingTrack.box;
} else if (seedBox) {
ann.box = seedBox.slice();
} else { $("track-readout").textContent = "draw + (optionally) track a box first"; return; }
annotations.push(ann);
seedBox = null; pendingTrack = null; $("add-ann").disabled = true;
renderLists(); renderDrawLayer();
});
$("add-aff").addEventListener("click", () => {
const key = $("aff-key").value.trim();
if (!key || !affectPoint) return;
const t = tiers(["a1", "a2", "a3", "a4"]);
affect.push({ key, at: affectPoint.slice(), min_level: +$("aff-min").value, _tiers: t || key });
affectPoint = null; $("add-aff").disabled = true;
renderLists(); renderDrawLayer();
});
function renderLists() {
const al = $("ann-list"); al.innerHTML = "";
annotations.forEach((a, i) => {
const li = document.createElement("li");
const kind = a.track ? `track ${a.track.length}kf ${mono(a.appear)}${mono(a.disappear)}` : "static";
const tip = Array.isArray(a._tiers) ? a._tiers.join(" → ") : a._tiers;
li.textContent = `${a.key} (s${a.salience}, ${kind}) — ${tip}`;
li.appendChild(rm(() => { annotations.splice(i, 1); renderLists(); }));
al.appendChild(li);
});
const fl = $("aff-list"); fl.innerHTML = "";
affect.forEach((f, i) => {
const li = document.createElement("li");
const tip = Array.isArray(f._tiers) ? f._tiers.join(" → ") : f._tiers;
li.textContent = `${f.key} (min ${f.min_level}) — ${tip}`;
li.appendChild(rm(() => { affect.splice(i, 1); renderLists(); }));
fl.appendChild(li);
});
}
function rm(fn) {
const b = document.createElement("button");
b.textContent = "✕"; b.className = "rm"; b.onclick = fn;
return b;
}
// Load whatever is already authored for this clip into the editor.
$("load-existing").addEventListener("click", () => {
if (!currentClip) return;
const strings = (currentClip.strings && currentClip.strings.en) || {};
annotations = (currentClip.annotations || []).map((a) => ({ ...a, _tiers: strings[a.key] || a.key }));
affect = (currentClip.affect || []).map((f) => ({ ...f, _tiers: strings[f.key] || f.key }));
renderLists();
});
// Assemble strings.en from each entry's _tiers and POST to the manifest.
$("save").addEventListener("click", async () => {
if (!currentClip) return;
const en = {};
const strip = (x) => { const { _tiers, ...rest } = x; return rest; };
const anns = annotations.map((a) => { en[a.key] = a._tiers; return strip(a); });
const affs = affect.map((f) => { en[f.key] = f._tiers; return strip(f); });
const resp = await fetch("/api/author/clip", {
method: "POST", headers: { "content-type": "application/json" },
body: JSON.stringify({ clip_id: currentClip.id, annotations: anns, affect: affs, strings: { en } }),
});
$("save-readout").textContent = resp.ok
? `saved ${anns.length} labels + ${affs.length} emotions to the manifest ✓`
: "save error " + resp.status;
if (resp.ok) { // refresh the in-memory clip copy
clips = (await (await fetch("/api/clips")).json()).clips || [];
currentClip = clips.find((c) => c.id === currentClip.id) || currentClip;
}
});
$("clip").addEventListener("change", (e) => selectClip(e.target.value));
async function main() {
await loadClips();
requestAnimationFrame(previewLoop);
}
main();
+38 -10
View File
@@ -35,13 +35,12 @@
</fieldset>
<fieldset>
<legend>Scale ring (endless encoder)</legend>
<div class="ring-control">
<button type="button" id="zoom-out" title="zoom out (toward cosmos)">⊖ out</button>
<span id="scale-name" class="scale-name"></span>
<button type="button" id="zoom-in" title="zoom in (toward microscopic)">in ⊕</button>
<legend>Altitude</legend>
<div class="dial-wrap">
<svg id="dial" viewBox="0 0 100 100" aria-label="Altitude knob (turn to change scale)"></svg>
</div>
<p class="hint">Relative, endless — wraps past the smallest back to the largest. Scroll the stage too.</p>
<span id="scale-name" class="scale-name"></span>
<p class="hint">Turn the knob (drag it, or scroll) to change altitude — endless: past the deepest it wraps back up to the highest. Click a label to jump there.</p>
</fieldset>
<fieldset>
@@ -51,11 +50,40 @@
<label>Mood — dark ◀ 0 ▶ light <input type="range" id="mood" min="-4" max="4" value="0" /></label>
</fieldset>
<!-- Dev Mode: a single toggle; everything dev (pool picker + analysis) lives below it. -->
<label class="dev-switch" for="dev-mode">
<input type="checkbox" id="dev-mode" />
<span class="dev-switch-track"><span class="dev-switch-thumb"></span></span>
<span class="dev-switch-label">Dev Mode</span>
</label>
<fieldset>
<legend>RenderPlan readout</legend>
<pre id="readout"></pre>
</fieldset>
<div id="dev-panel" class="dev-panel hidden">
<fieldset>
<legend>Pool — pick a clip</legend>
<select id="pool-select" aria-label="Choose a clip from this altitude's pool"></select>
<button type="button" id="pool-reroll" class="dev-btn">🎲 Re-roll random</button>
</fieldset>
<fieldset>
<legend>Active clip</legend>
<dl id="clip-meta" class="dev-dl"></dl>
</fieldset>
<fieldset>
<legend>Annotations &amp; affect</legend>
<div id="clip-anno" class="dev-anno"></div>
</fieldset>
<fieldset>
<legend>RenderPlan</legend>
<pre id="readout"></pre>
</fieldset>
<fieldset>
<legend>Cache &amp; playback</legend>
<dl id="dev-stats" class="dev-dl"></dl>
</fieldset>
</div>
</section>
</main>
<script src="/app.js"></script>
+52 -7
View File
@@ -39,7 +39,7 @@ main { display: flex; gap: 1rem; padding: 1rem; flex-wrap: wrap; }
.hud-chip-text.measure { fill: #ffd79a; }
.hud-conf { fill: #6cf; opacity: 0.8; }
.hud-status { fill: #8fdcff; font: 2.4px monospace; opacity: 0.9; letter-spacing: 0.15px; }
.black { position: absolute; inset: 0; background: #000; }
.black { position: absolute; inset: 0; background: #000; opacity: 1; transition: opacity 200ms ease; }
.hidden { display: none; }
.panel { flex: 0 0 280px; display: flex; flex-direction: column; gap: 0.8rem; }
fieldset { border: 1px solid #333; border-radius: 6px; }
@@ -48,10 +48,55 @@ label { display: block; margin: 0.4rem 0; }
input[type=range], select { width: 100%; }
#readout { background: #000; padding: 0.5rem; border-radius: 4px; font-size: 12px;
white-space: pre-wrap; max-height: 240px; overflow: auto; }
.ring-control { display: flex; align-items: center; gap: 0.4rem; }
.ring-control button { flex: 0 0 auto; background: #1a2436; color: #9af;
border: 1px solid #345; border-radius: 4px; padding: 0.3rem 0.5rem;
cursor: pointer; font-size: 13px; }
.ring-control button:hover { background: #243352; }
.scale-name { flex: 1 1 auto; text-align: center; font-size: 12px; color: #cde; }
/* Altitude knob */
.dial-wrap { display: flex; justify-content: center; padding: 0.3rem 0 0.1rem; }
#dial { width: 190px; height: 190px; touch-action: none; cursor: grab; user-select: none; }
#dial:active { cursor: grabbing; }
.dial-rim { fill: #0d1320; stroke: #243352; stroke-width: 1.2; }
.dial-body { fill: #16203200; stroke: #2c3c5c; stroke-width: 1; }
.dial-tick { stroke: #3a4d70; stroke-width: 0.8; }
.dial-label { fill: #789ac0; font-size: 6px; font-family: ui-monospace, monospace;
letter-spacing: 0.2px; cursor: pointer; }
.dial-label:hover { fill: #cde; }
.dial-label.active { fill: #9cf; font-weight: 700; }
.dial-caption { fill: #4d6184; font-size: 4.4px; letter-spacing: 1.2px;
font-family: ui-monospace, monospace; }
.dial-needle { fill: #9cf; }
.dial-needle polygon { filter: drop-shadow(0 0 1px #9cf); }
.dial-hub { fill: #2c3c5c; stroke: #9cf; stroke-width: 0.6; }
.scale-name { display: block; text-align: center; font-size: 12px; color: #cde; }
.hint { margin: 0.3rem 0 0; font-size: 11px; color: #789; }
/* --- Dev Mode --- a single switch; all dev controls/data live in #dev-panel below it. */
.dev-switch { display: flex; align-items: center; gap: 0.5rem; cursor: pointer;
padding: 0.5rem 0.2rem 0.2rem; border-top: 1px solid #222; user-select: none; }
.dev-switch input { position: absolute; opacity: 0; width: 0; height: 0; }
.dev-switch-track { position: relative; width: 34px; height: 18px; border-radius: 9px;
background: #2a2a2a; border: 1px solid #444; transition: background 0.15s; }
.dev-switch-thumb { position: absolute; top: 1px; left: 1px; width: 14px; height: 14px;
border-radius: 50%; background: #888; transition: transform 0.15s, background 0.15s; }
.dev-switch input:checked + .dev-switch-track { background: #1d3a2a; border-color: #2e6; }
.dev-switch input:checked + .dev-switch-track .dev-switch-thumb { transform: translateX(16px); background: #4e9; }
.dev-switch input:focus-visible + .dev-switch-track { outline: 2px solid #9af; outline-offset: 1px; }
.dev-switch-label { color: #9af; font-weight: 600; letter-spacing: 0.3px; }
.dev-panel { display: flex; flex-direction: column; gap: 0.8rem; }
/* Beat the generic `.hidden` (defined earlier, equal specificity): two classes
win, so the panel truly collapses when Dev Mode is off. */
.dev-panel.hidden { display: none; }
.dev-panel legend { color: #4e9; }
.dev-btn { margin-top: 0.5rem; width: 100%; padding: 0.4rem; cursor: pointer;
background: #182028; color: #cde; border: 1px solid #2c3c5c; border-radius: 4px; }
.dev-btn:hover { background: #20303f; }
.dev-dl { margin: 0; display: grid; grid-template-columns: auto 1fr; gap: 0.15rem 0.6rem; font-size: 12px; }
.dev-dl dt { color: #789; }
.dev-dl dd { margin: 0; color: #dde; word-break: break-word; }
.dev-dl dd.mem { color: #4e9; }
.dev-dl dd.net { color: #fc6; }
.dev-anno { font-size: 11px; max-height: 220px; overflow: auto;
font-family: ui-monospace, monospace; }
.dev-anno .anno-row { padding: 0.2rem 0; border-bottom: 1px solid #1d1d1d; }
.dev-anno .anno-key { color: #aee6ff; }
.dev-anno .anno-key.affect { color: #e3cfff; }
.dev-anno .anno-key.measure { color: #ffd79a; }
.dev-anno .anno-meta { color: #678; }
.dev-anno .anno-empty { color: #567; font-style: italic; }
+10 -10
View File
@@ -4,7 +4,9 @@ import pytest
from tools.pipeline.run import probe_duration, process_clip, resolve_ffmpeg
_FOREST = Path("simulator/sample_media/forest/base.mp4")
# A real pool base (forest was retired with the rotating-pool rename, session 0016);
# cosmos/base.mp4 is the cosmos pool primary and the most stable PD clip on disk.
_BASE = Path("simulator/sample_media/cosmos/base.mp4")
def _ffmpeg_available() -> bool:
@@ -18,20 +20,18 @@ def _ffmpeg_available() -> bool:
_HAS_FF = _ffmpeg_available()
@pytest.mark.skipif(not _FOREST.exists(),
reason="forest POC base absent (run simulator/setup_sample_media.py)")
def test_probe_duration_reads_forest_base():
if not _FOREST.exists():
pytest.skip("forest POC base absent")
assert probe_duration(_FOREST) > 0
@pytest.mark.skipif(not _BASE.exists(),
reason="cosmos base absent (run simulator/build_pool_manifest.py)")
def test_probe_duration_reads_real_base():
assert probe_duration(_BASE) > 0
@pytest.mark.skipif(not _HAS_FF,
reason="neither system ffmpeg nor imageio-ffmpeg available")
@pytest.mark.skipif(not _FOREST.exists(),
reason="forest POC base absent (run simulator/setup_sample_media.py)")
@pytest.mark.skipif(not _BASE.exists(),
reason="cosmos base absent (run simulator/build_pool_manifest.py)")
def test_process_clip_emits_proxy_and_master(tmp_path):
out = process_clip(_FOREST, tmp_path, overlap=0.5, start=0.0, duration=4.0)
out = process_clip(_BASE, tmp_path, overlap=0.5, start=0.0, duration=4.0)
assert out["proxy"].exists() and out["master"].exists()
# Proxy is exactly 1920×1080 — verified with cv2, no ffprobe dependency.
import cv2
+120
View File
@@ -0,0 +1,120 @@
"""tools/pipeline/track.py — the hybrid motion-track pass (content-pipeline §11.5).
Pure geometry helpers are unit-tested here; the cv2 optical-flow propagation is an
opt-in integration test that runs on a real pool clip when present."""
from pathlib import Path
import pytest
from tools.pipeline.track import (
clamp01_box,
denormalize_box,
infer_window,
loop_t,
normalize_box,
sample_track,
track_to_annotation,
)
# --- pure helpers -----------------------------------------------------------
def test_clamp01_box_keeps_box_on_screen():
assert clamp01_box((-0.2, 0.1, 0.5, 0.5)) == [0.0, 0.1, 0.5, 0.5]
# size trimmed so the box never runs past the right/bottom edge
assert clamp01_box((0.8, 0.8, 0.5, 0.5)) == [0.8, 0.8, pytest.approx(0.2), pytest.approx(0.2)]
def test_normalize_denormalize_round_trip():
px = (192, 108, 384, 216)
norm = normalize_box(px, 1920, 1080)
assert norm == [pytest.approx(0.1), pytest.approx(0.1), pytest.approx(0.2), pytest.approx(0.2)]
assert denormalize_box(norm, 1920, 1080) == (192, 108, 384, 216)
def test_loop_t_is_frame_over_total():
assert loop_t(0, 100) == 0.0
assert loop_t(50, 100) == 0.5
assert loop_t(5, 0) == 0.0 # guard divide-by-zero
def test_infer_window_spans_tracked_frames():
appear, disappear = infer_window([10, 11, 30, 31], 100)
assert appear == pytest.approx(0.1)
assert disappear == pytest.approx(0.31)
# no frames -> full clip
assert infer_window([], 100) == (0.0, 1.0)
def test_sample_track_downsamples_to_sparse_keyframes():
# 21 dense frames -> at most n_keyframes, including the first and last
per_frame = {i: [i / 100, 0.2, 0.1, 0.1] for i in range(0, 21)}
track = sample_track(per_frame, total_frames=100, n_keyframes=5)
assert len(track) <= 5
ts = [k["t"] for k in track]
assert ts == sorted(ts) # sorted by t
assert ts[0] == pytest.approx(0.0) # first tracked frame
assert ts[-1] == pytest.approx(0.2) # last tracked frame (frame 20 / 100)
assert len(set(ts)) == len(ts) # de-duplicated on t
def test_sample_track_keeps_all_when_already_sparse():
per_frame = {0: [0.1, 0.1, 0.1, 0.1], 60: [0.5, 0.3, 0.1, 0.1]}
track = sample_track(per_frame, total_frames=120, n_keyframes=5)
assert [k["t"] for k in track] == [pytest.approx(0.0), pytest.approx(0.5)]
def test_sample_track_empty_is_empty():
assert sample_track({}, total_frames=100) == []
def test_track_to_annotation_assembles_geometry_only():
track = [{"t": 0.0, "box": [0.1, 0.2, 0.1, 0.1]}, {"t": 0.5, "box": [0.4, 0.2, 0.1, 0.1]}]
ann = track_to_annotation("detected.jelly", track, salience=3, appear=0.0, disappear=0.55)
assert ann["key"] == "detected.jelly"
assert ann["salience"] == 3
assert ann["track"] == track
assert ann["appear"] == 0.0 and ann["disappear"] == 0.55
# no semantics leaked in — strings/tiers are the author's, added elsewhere
assert "strings" not in ann and "_tiers" not in ann
def test_track_to_annotation_omits_window_when_absent():
ann = track_to_annotation("detected.x", [], salience=4)
assert "appear" not in ann and "disappear" not in ann
# --- opt-in: classical optical-flow propagation on a real clip --------------
_CLIP = Path("simulator/sample_media/abyss_wow/base.mp4")
def _cv2_available() -> bool:
try:
import cv2 # noqa: F401
return True
except Exception:
return False
@pytest.mark.skipif(not _CLIP.exists(),
reason="abyss_wow base absent (run simulator/build_pool_manifest.py)")
@pytest.mark.skipif(not _cv2_available(), reason="opencv-python not available")
def test_track_seed_produces_a_sparse_annotation_on_real_footage():
from tools.pipeline.track import track_box, track_seed
seed = (0.2, 0.3, 0.15, 0.18)
per_frame, total = track_box(_CLIP, seed, seed_frame=0, max_frames=40)
assert total > 0
assert per_frame[0] == pytest.approx(list(seed), abs=0.02)
assert len(per_frame) > 1 # propagated beyond the seed frame
ann = track_seed(_CLIP, "detected.jelly", seed, seed_frame=0, n_keyframes=5, max_frames=40)
assert ann["key"] == "detected.jelly"
assert 2 <= len(ann["track"]) <= 5 # sparse keyframes
for kf in ann["track"]:
assert 0.0 <= kf["t"] <= 1.0
assert all(0.0 <= v <= 1.0 for v in kf["box"])
assert 0.0 <= ann["appear"] <= ann["disappear"] <= 1.0
+34
View File
@@ -8,6 +8,7 @@ from player.ring import (
Transition,
TransitionStep,
advance_ring,
pick_clip_id,
scale_at,
)
@@ -206,3 +207,36 @@ def test_fast_spin_noop_on_degenerate_ring():
move = advance_ring(r, from_index=0, delta=9, fast_spin_threshold=3)
assert move.fast is False
assert move.steps == ()
# --- rotating clip pool (content-pipeline design §11.1) ---
def test_scale_without_pool_is_a_pool_of_one():
s = Scale(id="cosmos", clip_id="cosmos")
assert s.members == ("cosmos",)
# any r maps to the single member
assert pick_clip_id(s, 0.0) == "cosmos"
assert pick_clip_id(s, 0.99) == "cosmos"
def test_scale_with_pool_exposes_all_members():
s = Scale(id="coast", clip_id="coast_a", pool=("coast_a", "coast_b", "coast_c"))
assert s.members == ("coast_a", "coast_b", "coast_c")
def test_pick_clip_id_selects_member_by_injected_r():
s = Scale(id="coast", clip_id="a", pool=("a", "b", "c", "d"))
# r in [0,1) partitions the pool into len equal buckets
assert pick_clip_id(s, 0.0) == "a"
assert pick_clip_id(s, 0.24) == "a"
assert pick_clip_id(s, 0.25) == "b"
assert pick_clip_id(s, 0.5) == "c"
assert pick_clip_id(s, 0.75) == "d"
def test_pick_clip_id_clamps_out_of_range_r():
s = Scale(id="coast", clip_id="a", pool=("a", "b", "c"))
assert pick_clip_id(s, -5.0) == "a" # below 0 clamps to first
assert pick_clip_id(s, 1.0) == "c" # 1.0 must not index past the end
assert pick_clip_id(s, 999.0) == "c"
+189
View File
@@ -76,6 +76,30 @@ def test_clips_returns_the_manifest(client):
assert data["clips"][0]["annotations"][0]["key"] == "detected.water"
def test_media_versions_endpoint_shape(client):
resp = client.get("/api/media-versions")
assert resp.status_code == 200
assert isinstance(resp.json()["versions"], dict)
def test_media_version_is_a_content_hash(tmp_path, monkeypatch):
import hashlib
import simulator.app as appmod
# Point the media root at a temp dir (the served dir is otherwise fixed).
monkeypatch.setattr(appmod, "MEDIA_DIR", tmp_path)
(tmp_path / "clip").mkdir()
f = tmp_path / "clip" / "base.mp4"
f.write_bytes(b"hello-bytes")
assert appmod._media_version("clip/base.mp4") == hashlib.sha1(b"hello-bytes").hexdigest()[:12]
# Absent file -> None (it's simply omitted from the versions map).
assert appmod._media_version("clip/missing.mp4") is None
# Re-baked under the same path (new bytes) -> a new token, not the cached one.
f.write_bytes(b"different-bytes-entirely")
assert appmod._media_version("clip/base.mp4") == hashlib.sha1(b"different-bytes-entirely").hexdigest()[:12]
def test_retired_selection_endpoints_are_gone(client):
# The route no longer exists; the static catch-all yields 404 on GET and
# 405 on the (now-unrouted) POST. Either proves the endpoint is gone.
@@ -91,6 +115,35 @@ def test_index_is_served():
assert "Alteration" in resp.text
def test_app_assets_are_not_cached(client):
# Dev iteration relies on the browser never serving stale renderer code.
resp = client.get("/")
assert "no-store" in resp.headers.get("cache-control", "")
def test_media_is_cacheable(tmp_path, monkeypatch):
# Media is static: it must be browser-cacheable so the preloader's fetch (and
# any fallback) doesn't re-hit the network on every altitude change.
import simulator.app as appmod
media = tmp_path / "media"
(media / "cosmos").mkdir(parents=True)
(media / "cosmos" / "base.mp4").write_bytes(b"\x00\x00\x00\x18ftyp")
monkeypatch.setattr(appmod, "MEDIA_DIR", media)
manifest = tmp_path / "manifest.json"
manifest.write_text(json.dumps({"clips": []}))
client = TestClient(create_app(manifest_path=manifest))
resp = client.get("/media/cosmos/base.mp4")
assert resp.status_code == 200
cc = resp.headers.get("cache-control", "")
# Stored but revalidated: not no-store (so the blob preload can cache it), and
# not pinned immutable (so a re-sourced/re-cropped clip shows on a plain reload).
assert "no-store" not in cc
assert "immutable" not in cc
assert "no-cache" in cc
@pytest.fixture
def ring_manifest_path(tmp_path):
p = tmp_path / "manifest.json"
@@ -189,3 +242,139 @@ def test_ring_404_when_no_ring_in_manifest(client):
def test_ring_advance_rejects_bad_delta(ring_client):
resp = ring_client.post("/api/ring/advance", json={"from_index": 0, "delta": "x"})
assert resp.status_code == 422
# --- rotating clip pool (content-pipeline §11.1) ---
@pytest.fixture
def pool_client(tmp_path):
p = tmp_path / "manifest.json"
members = ["coast_a", "coast_b", "coast_c", "coast_d"]
clips = [{"id": "cosmos", "title": "Cosmos", "base_file": "cosmos/base.mp4",
"license": "PD", "source": "NASA", "right_variants": {},
"annotations": [], "affect": [], "strings": {}}]
clips += [{"id": m, "title": m, "base_file": f"{m}/base.mp4", "license": "PD",
"source": "NPS", "right_variants": {}, "annotations": [], "affect": [],
"strings": {}} for m in members]
p.write_text(json.dumps({
"clips": clips,
"ring": {
"scales": [
{"id": "cosmos", "clip_id": "cosmos"},
{"id": "coast", "clip_id": members[0], "pool": members},
],
"transitions": [
{"file": "transitions/cosmos-coast.mp4"},
{"file": "transitions/coast-cosmos.mp4"},
],
},
}))
return TestClient(create_app(manifest_path=p)), members
def test_ring_exposes_each_scales_pool(pool_client):
client, members = pool_client
data = client.get("/api/ring").json()
coast = next(s for s in data["scales"] if s["id"] == "coast")
assert [m["clip_id"] for m in coast["pool"]] == members
# a single-clip scale still reports a pool of one
cosmos = next(s for s in data["scales"] if s["id"] == "cosmos")
assert [m["clip_id"] for m in cosmos["pool"]] == ["cosmos"]
def test_advance_into_a_pool_lands_on_a_member(pool_client):
client, members = pool_client
# landing on the coast scale (index 1) must pick one of its pool members
seen = set()
for _ in range(40):
data = client.post("/api/ring/advance", json={"from_index": 0, "delta": 1}).json()
assert data["to_index"] == 1
assert data["target_clip_id"] in members
seen.add(data["target_clip_id"])
# over many landings the random pick should cover more than one member
assert len(seen) > 1
def test_delta_zero_is_an_initial_pool_pick(pool_client):
client, members = pool_client
# a no-op (delta 0) advance on the pool scale is the initial / re-roll pick
data = client.post("/api/ring/advance", json={"from_index": 1, "delta": 0}).json()
assert data["to_index"] == 1
assert data["steps"] == []
assert data["target_clip_id"] in members
# --- author mode (content-pipeline §11.5) ---
@pytest.fixture
def author_client(tmp_path):
p = tmp_path / "manifest.json"
p.write_text(json.dumps({
"clips": [{
"id": "abyss_wow", "title": "World of Water", "base_file": "abyss_wow/base.mp4",
"license": "PD", "source": "NOAA", "right_variants": {},
"annotations": [], "affect": [], "strings": {"en": {}},
}],
}))
return TestClient(create_app(manifest_path=p)), p
def test_author_clip_persists_labels_and_keeps_provenance(author_client):
client, path = author_client
body = {
"clip_id": "abyss_wow",
"annotations": [{"key": "detected.jelly", "salience": 4,
"appear": 0.0, "disappear": 0.5,
"track": [{"t": 0.0, "box": [0.1, 0.2, 0.1, 0.1]}]}],
"affect": [{"key": "feel.unease", "at": [0.5, 0.5], "min_level": 1}],
"strings": {"en": {"detected.jelly": ["jelly", "comb jelly", "Ctenophora", "Ctenophora · cilia rows"],
"feel.unease": ["uh", "unease", "disquiet", "a creeping disquiet"]}},
}
resp = client.post("/api/author/clip", json=body)
assert resp.status_code == 200 and resp.json()["ok"] is True
# persisted to disk
saved = json.loads(path.read_text())["clips"][0]
assert saved["annotations"][0]["key"] == "detected.jelly"
assert saved["affect"][0]["key"] == "feel.unease"
assert saved["strings"]["en"]["detected.jelly"][2] == "Ctenophora"
# provenance preserved (only authored content replaced)
assert saved["base_file"] == "abyss_wow/base.mp4"
assert saved["license"] == "PD"
# reflected by /api/clips without a restart
served = client.get("/api/clips").json()["clips"][0]
assert served["annotations"][0]["key"] == "detected.jelly"
def test_author_clip_unknown_clip_404(author_client):
client, _ = author_client
resp = client.post("/api/author/clip", json={"clip_id": "nope", "annotations": []})
assert resp.status_code == 404
def test_author_track_unknown_clip_404(author_client):
client, _ = author_client
resp = client.post("/api/author/track", json={
"clip_id": "nope", "key": "detected.x", "seed_box": [0.1, 0.1, 0.1, 0.1]})
assert resp.status_code == 404
_REAL_CLIP = __import__("pathlib").Path("simulator/sample_media/abyss_wow/base.mp4")
@pytest.mark.skipif(not _REAL_CLIP.exists(),
reason="abyss_wow base absent (run simulator/build_pool_manifest.py)")
def test_author_track_on_real_clip_returns_keyframes():
# Against the real default manifest + footage: the tracker yields a sparse track.
client = TestClient(create_app())
resp = client.post("/api/author/track", json={
"clip_id": "abyss_wow", "key": "detected.jelly",
"seed_box": [0.2, 0.3, 0.15, 0.18], "seed_t": 0.0,
"salience": 4, "n_keyframes": 5, "max_frames": 30,
})
assert resp.status_code == 200
ann = resp.json()
assert ann["key"] == "detected.jelly"
assert 2 <= len(ann["track"]) <= 5
assert 0.0 <= ann["appear"] <= ann["disappear"] <= 1.0
+297
View File
@@ -0,0 +1,297 @@
"""Stage 5 geometry — the hybrid motion-track pass (content-pipeline §3.5 / §11.5).
Turns a hand-seeded box (or an ML detection) on one keyframe into the sparse,
loop-normalized keyframed `track` the manifest stores (design §4) plus an
appear/disappear window (§11.2). SEMANTICS are never produced here — the author
owns the `key`, strings, `salience` and tiers; this module only propagates BOX
GEOMETRY.
Two geometry paths, same artifact:
- **classical** (always available): a hand-seeded box propagated by OpenCV
Lucas-Kanade optical flow (a CSRT tracker is used instead when a contrib
build provides one). Translation model — right for calm drifting subjects
(the abyss/reef creatures). Re-seeds features when too few survive.
- **ML detect+track** (optional, lazy-imported): a YOLO+tracker pass proposing
boxes the author maps to a key. Imported only on demand so the base install
needs no torch/ultralytics; a clear ImportError tells you what to `pip
install` if you ask for it without it present.
The PURE helpers (normalization, keyframe sampling, window inference, annotation
assembly) are unit-tested; the cv2/ML propagation is covered by the opt-in
integration test on real footage.
"""
from __future__ import annotations
from typing import Optional
Box = tuple[float, float, float, float] # (x, y, w, h)
# --- pure geometry helpers --------------------------------------------------
def clamp01_box(box: Box) -> list[float]:
"""Clamp a normalized box into the visible [0,1] frame, keeping it on-screen
(origin clamped, then size trimmed so x+w<=1, y+h<=1)."""
x, y, w, h = box
x = min(max(x, 0.0), 1.0)
y = min(max(y, 0.0), 1.0)
w = min(max(w, 0.0), 1.0 - x)
h = min(max(h, 0.0), 1.0 - y)
return [x, y, w, h]
def normalize_box(box_px: Box, width: int, height: int) -> list[float]:
"""Pixel box -> normalized [0,1] box (the manifest's resolution-independent
form). Clamped on-screen."""
x, y, w, h = box_px
return clamp01_box((x / width, y / height, w / width, h / height))
def denormalize_box(box_norm: Box, width: int, height: int) -> tuple[int, int, int, int]:
"""Normalized [0,1] box -> integer pixel box for the cv2 tracker."""
x, y, w, h = box_norm
return (round(x * width), round(y * height), round(w * width), round(h * height))
def loop_t(frame_idx: int, total_frames: int) -> float:
"""Loop-normalized time of a frame in [0,1), matching the client's
`currentTime / duration` clock (design §4)."""
if total_frames <= 0:
return 0.0
return frame_idx / total_frames
def infer_window(tracked_frames: list[int], total_frames: int) -> tuple[float, float]:
"""The appear/disappear window (§11.2) covering the frames where a box was
tracked, as loop-normalized t. The first/last tracked frame bound it."""
if not tracked_frames:
return (0.0, 1.0)
lo, hi = min(tracked_frames), max(tracked_frames)
return (loop_t(lo, total_frames), loop_t(hi, total_frames))
def sample_track(
per_frame: dict[int, Box], total_frames: int, n_keyframes: int = 5
) -> list[dict]:
"""Down-sample a dense per-frame box map to a SPARSE keyframed track
(design §4: sparse keyframes interpolated at runtime, never dense per-frame).
Picks `n_keyframes` frames evenly across the tracked span (always including
the first and last tracked frame) and emits `[{t, box}, ...]` sorted by t,
de-duplicated on t. Pure — takes already-normalized boxes."""
if not per_frame:
return []
frames = sorted(per_frame)
n = max(2, n_keyframes)
if len(frames) <= n:
picks = frames
else:
lo, hi = frames[0], frames[-1]
step = (hi - lo) / (n - 1)
targets = [lo + round(step * i) for i in range(n)]
# snap each target to the nearest actually-tracked frame
picks = sorted({min(frames, key=lambda f: abs(f - t)) for t in targets})
out, seen = [], set()
for f in picks:
t = round(loop_t(f, total_frames), 4)
if t in seen:
continue
seen.add(t)
out.append({"t": t, "box": clamp01_box(per_frame[f])})
return out
def track_to_annotation(
key: str,
track: list[dict],
*,
salience: int = 4,
appear: Optional[float] = None,
disappear: Optional[float] = None,
) -> dict:
"""Assemble a manifest annotation from a propagated track + the author's
semantics. Geometry (track + window) comes from this module; `key`,
`salience` and the tiered strings are the author's (added separately)."""
ann: dict = {"key": key, "salience": salience, "track": track}
if appear is not None:
ann["appear"] = round(appear, 4)
if disappear is not None:
ann["disappear"] = round(disappear, 4)
return ann
# --- classical propagation (cv2; covered by the opt-in integration test) -----
def _new_csrt():
"""A CSRT tracker if this OpenCV build ships one (contrib), else None — then
the optical-flow path is used. Kept tiny so the import stays lazy."""
import cv2
for factory in ("TrackerCSRT_create",):
if hasattr(cv2, factory):
return getattr(cv2, factory)()
legacy = getattr(cv2, "legacy", None)
if legacy is not None and hasattr(legacy, "TrackerCSRT_create"):
return legacy.TrackerCSRT_create()
return None
def _features_in_box(gray, box_px, max_corners: int = 60):
"""Good-feature points inside a pixel box, in full-frame coords (for LK)."""
import cv2
import numpy as np
x, y, w, h = (int(v) for v in box_px)
h_img, w_img = gray.shape[:2]
x0, y0 = max(x, 0), max(y, 0)
x1, y1 = min(x + w, w_img), min(y + h, h_img)
if x1 - x0 < 4 or y1 - y0 < 4:
return None
roi = gray[y0:y1, x0:x1]
pts = cv2.goodFeaturesToTrack(roi, max_corners, 0.01, 5)
if pts is None:
return None
pts = pts.reshape(-1, 2) + np.array([x0, y0], dtype="float32")
return pts.reshape(-1, 1, 2)
def track_box(
video_path,
seed_box_norm: Box,
*,
seed_frame: int = 0,
max_frames: Optional[int] = None,
min_features: int = 8,
) -> tuple[dict[int, list[float]], int]:
"""Propagate a hand-seeded normalized box FORWARD from `seed_frame` with
optical flow (or CSRT when available). Returns `(per_frame_norm_boxes,
total_frames)` — feed `per_frame` to `sample_track`. Translation model: the
box follows the median displacement of features inside it, re-seeding when too
few survive. Impure (reads the video) — opt-in integration test."""
import cv2
import numpy as np
cap = cv2.VideoCapture(str(video_path))
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
if seed_frame:
cap.set(cv2.CAP_PROP_POS_FRAMES, seed_frame)
ok, frame = cap.read()
if not ok:
cap.release()
raise ValueError(f"could not read frame {seed_frame} of {video_path}")
h_img, w_img = frame.shape[:2]
box = list(denormalize_box(seed_box_norm, w_img, h_img)) # px [x,y,w,h]
csrt = _new_csrt()
if csrt is not None:
csrt.init(frame, tuple(int(v) for v in box))
prev_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
pts = None if csrt is not None else _features_in_box(prev_gray, box)
out: dict[int, list[float]] = {seed_frame: normalize_box(box, w_img, h_img)}
fidx = seed_frame
while True:
ok, frame = cap.read()
if not ok:
break
fidx += 1
if csrt is not None:
ok2, b = csrt.update(frame)
if ok2:
box = [b[0], b[1], b[2], b[3]]
else:
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
if pts is None or len(pts) < min_features:
pts = _features_in_box(prev_gray, box)
if pts is not None and len(pts):
nxt, status, _ = cv2.calcOpticalFlowPyrLK(prev_gray, gray, pts, None)
if nxt is not None:
good_old = pts[status == 1]
good_new = nxt[status == 1]
if len(good_new) >= 3:
dx = float(np.median(good_new[:, 0] - good_old[:, 0]))
dy = float(np.median(good_new[:, 1] - good_old[:, 1]))
box[0] += dx
box[1] += dy
pts = good_new.reshape(-1, 1, 2)
else:
pts = None
prev_gray = gray
out[fidx] = normalize_box(box, w_img, h_img)
if max_frames is not None and fidx - seed_frame >= max_frames:
break
cap.release()
return out, total
def track_seed(
video_path,
key: str,
seed_box_norm: Box,
*,
seed_frame: int = 0,
salience: int = 4,
n_keyframes: int = 5,
max_frames: Optional[int] = None,
) -> dict:
"""End-to-end classical pass: seed -> propagate -> sample -> annotation
(geometry only; the author adds strings/tiers). Impure (reads the video)."""
per_frame, total = track_box(
video_path, seed_box_norm, seed_frame=seed_frame, max_frames=max_frames
)
track = sample_track(per_frame, total, n_keyframes=n_keyframes)
appear, disappear = infer_window(list(per_frame), total)
return track_to_annotation(
key, track, salience=salience, appear=appear, disappear=disappear
)
# --- optional ML detect+track (lazy-imported) --------------------------------
def detect_and_track(video_path, *, model: str = "yolov8n.pt", classes=None) -> list[dict]:
"""Optional ML path (content-pipeline §3.5): run a detect+track model and
return candidate tracks `[{track_id, label, salience, track:[{t,box}],
appear, disappear}]` for the author to map to keys. Heavy deps are
lazy-imported so the base install stays light.
Raises a clear ImportError naming the install if `ultralytics` is absent —
this path is opt-in by design; the classical `track_seed` needs only cv2."""
try:
from ultralytics import YOLO # noqa: F401
except ImportError as e: # pragma: no cover - exercised only without the dep
raise ImportError(
"the ML detect+track path needs ultralytics — "
"`pip install ultralytics` (optional; the classical track_seed path "
"uses only opencv-python)."
) from e
import cv2 # noqa: F401 (pragma: no cover below — needs the real model)
yolo = YOLO(model) # pragma: no cover
cap = cv2.VideoCapture(str(video_path)) # pragma: no cover
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0) # pragma: no cover
cap.release() # pragma: no cover
per_track: dict[int, dict[int, list[float]]] = {} # pragma: no cover
labels: dict[int, str] = {} # pragma: no cover
for fidx, res in enumerate( # pragma: no cover
yolo.track(source=str(video_path), persist=True, classes=classes, stream=True)
):
if res.boxes is None or res.boxes.id is None: # pragma: no cover
continue
w_img, h_img = res.orig_shape[1], res.orig_shape[0] # pragma: no cover
for box, tid, cls in zip( # pragma: no cover
res.boxes.xywh.tolist(), res.boxes.id.tolist(), res.boxes.cls.tolist()
):
cx, cy, bw, bh = box # pragma: no cover
nb = normalize_box((cx - bw / 2, cy - bh / 2, bw, bh), w_img, h_img)
per_track.setdefault(int(tid), {})[fidx] = nb # pragma: no cover
labels[int(tid)] = yolo.names.get(int(cls), str(cls))
out = [] # pragma: no cover
for tid, frames in per_track.items(): # pragma: no cover
track = sample_track(frames, total) # pragma: no cover
appear, disappear = infer_window(list(frames), total) # pragma: no cover
out.append({ # pragma: no cover
"track_id": tid, "label": labels.get(tid, ""), "salience": 4,
"track": track, "appear": round(appear, 4), "disappear": round(disappear, 4),
})
return out # pragma: no cover