Compare commits
39 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 0928988abc | |||
| a1c9c5c760 | |||
| ab2edf77bb | |||
| d19fe4e8fa | |||
| 9a83c0132d | |||
| 7fd9aabd0e | |||
| 7a32ff1acc | |||
| 5fae4f9005 | |||
| 7628a5321f | |||
| bcd4d32b19 | |||
| c4452396a1 | |||
| 70fd367c70 | |||
| 12408e505e | |||
| edca8a9615 | |||
| 4270c265e3 | |||
| b28f930e11 | |||
| 7be4f18e77 | |||
| 6256bc10e4 | |||
| 317a6be6b0 | |||
| bd4b65684c | |||
| 6369d0c5ab | |||
| 4b8ebf11ea | |||
| 73e5c4a3c7 | |||
| bb5ecf09b4 | |||
| e812f41ef1 | |||
| 8789d6a6d5 | |||
| 2933498c51 | |||
| 02c762fd45 | |||
| f1c83c1a9a | |||
| 38a1046433 | |||
| 60d27aa662 | |||
| dc122fa0bf | |||
| 0fd8b1203d | |||
| d3b6cd7bbd | |||
| 4b5fd20374 | |||
| 7217daeb44 | |||
| 642451925f | |||
| 750e87eb0b | |||
| e210de6f84 |
+3
-1
@@ -6,5 +6,7 @@ media/
|
||||
.superpowers/
|
||||
*.egg-info/
|
||||
# Simulator sample media (look-tuning only; populate via setup_sample_media.py
|
||||
# / setup_scales_media.py). All scale + transition binaries are gitignored.
|
||||
# / build_pool_manifest.py). All scale + transition binaries are gitignored.
|
||||
simulator/sample_media/**/*.mp4
|
||||
# Local-only candidate review gallery (re-buildable; not shipped content).
|
||||
simulator/static/review.html
|
||||
|
||||
+28
-15
@@ -170,21 +170,34 @@ every transition, so fast navigation stays responsive on placeholder media.
|
||||
contract** with the firmware (the keyboard/serial stand-in already works via a
|
||||
`Controls` stream).
|
||||
- **White-noise generation** — runtime white/pink noise for that content position.
|
||||
- **Offline v2v variant pipeline** — author the pre-baked Right restyle variants
|
||||
via the local flow-stabilized SD pipeline (now ~free, not a paid API — see the
|
||||
scales-library/right-axis design §1/§4); a real multi-strength flow-stabilized
|
||||
re-bake per base clip + the multilingual label/string tables + TTS. The
|
||||
**scale-ring transitions** (forward/reverse per-edge morphs) are part of this
|
||||
offline pipeline too.
|
||||
- **Vertical slice done (session 0012):** the Right-variant half is
|
||||
productionized and run for the **forest** scale — `simulator/bake_right_variants.py`
|
||||
(the POC flow-restyle algorithm, repo-resident) bakes real strengths 1–4 on a
|
||||
by-eye keyframe-strength ramp; forest now carries real Right variants in the
|
||||
sim, so the dreamlike-axis look is judgeable on real footage. **Still
|
||||
deferred:** real Right variants for cosmos/abyss (need their real strict-PD
|
||||
bases sourced first) and the **real i2v ring transitions** (need a 2nd real
|
||||
base + a generative-video model not in the POC). Plan:
|
||||
[`2026-06-08-v2v-vertical-slice-forest-right-variants.md`](./superpowers/plans/2026-06-08-v2v-vertical-slice-forest-right-variants.md).
|
||||
- **Content pipeline + label track** — source strict-PD neutral bases, run them
|
||||
through the offline pipeline, and author the annotation tracks that drive the
|
||||
Left-HUD overlay. The Right dream is now **real-time** (Kuwahara WebGL shader,
|
||||
session 0013 — no per-clip ML bake), so the pipeline is
|
||||
**source → crossfade-loop → master+proxy transcode → hand-authored /
|
||||
motion-tracked labels → manifest**, not ML baking. Historical note: the
|
||||
flow-stabilized SD-turbo v2v bake (session 0012, `simulator/bake_right_variants.py`)
|
||||
was superseded when the operator found the baked-SD Right "too fluid" and the
|
||||
design pivoted to the deterministic real-time dream; `bake_right_variants.py`
|
||||
is now parked. Spec:
|
||||
[`2026-06-24-content-pipeline-design.md`](./superpowers/specs/2026-06-24-content-pipeline-design.md).
|
||||
|
||||
- **Increment 1 — deterministic tooling + 5-scale ring (session 0014) ✅ Done.**
|
||||
`tools/pipeline/` (frame/loop/transcode/manifest, pure + unit-tested + an
|
||||
opt-in integration test on real footage); the backward-compatible keyframed
|
||||
annotation-track schema with client interpolation; and the **full 5-scale ring**
|
||||
(cosmos → orbit → forest → reef → abyss) with orbit + reef as placeholder
|
||||
scales alongside the existing cosmos/abyss/forest. Plan:
|
||||
[`2026-06-24-content-pipeline-increment-1.md`](./superpowers/plans/2026-06-24-content-pipeline-increment-1.md).
|
||||
|
||||
- **Increment 2 — hybrid motion-track pass + author mode (pending).** The
|
||||
`tools/pipeline/track.py` classical OpenCV tracker + optional ML detect+track,
|
||||
plus a simulator **author mode** for reviewing/editing annotation tracks in the
|
||||
browser. Real strict-PD bases for all five scales sourced per
|
||||
[`docs/content-sourcing.md`](./content-sourcing.md).
|
||||
|
||||
- **Still deferred:** real **i2v ring transitions** (need a generative-video
|
||||
model + adjacent real bases); Pi renderer + serial/firmware.
|
||||
- **Catalog model changes** — audio *source* + "neutral base" vs "altered
|
||||
variant" flag (sub-project 2 territory, design §13).
|
||||
|
||||
|
||||
+48
-37
@@ -296,24 +296,17 @@ exists. (The earlier selection-era "curator's X-ray" view was retired when the
|
||||
piece moved from *selecting* clips to *altering* them.)
|
||||
|
||||
**One-time setup — populate the sample footage** (look-tuning only; not shipped
|
||||
content):
|
||||
content). The ring uses real strict-PD footage in a **rotating pool** per scale
|
||||
(`docs/content-candidate-pool.md`); regenerate the manifest + new transition
|
||||
placeholders with:
|
||||
|
||||
python simulator/setup_sample_media.py # stage the forest neutral base
|
||||
python -m simulator.bake_right_variants --scale forest # real Right variants (SD on MPS, ~11 min)
|
||||
python simulator/setup_scales_media.py # the cosmos + abyss scales + ring transitions
|
||||
python simulator/build_pool_manifest.py --media # write the pool manifest + coast-edge transitions
|
||||
|
||||
`setup_sample_media.py` stages the forest neutral base from the session-0008 POC
|
||||
artifacts (`~/hef-poc/out/neutral.mp4`). `bake_right_variants.py` then produces
|
||||
the **real** flow-stabilized painterly Right variants (strengths 1–4) for the
|
||||
forest scale — SD img2img keyframes + Farneback optical-flow tweens on Apple MPS,
|
||||
the temporally-calm POC algorithm, productionized (scales design §1/§4). The
|
||||
keyframe-strength ramp per Right level is by-eye tunable in
|
||||
`simulator/bake_right_variants.py`. `setup_scales_media.py` makes the scale
|
||||
**ring** demonstrable: cheap synthetic placeholder bases for the two true-PD
|
||||
scales (`cosmos` = NASA/Hubble, `abyss` = NOAA Ocean Exploration) plus the
|
||||
per-edge zoom/warp transition clips — these scales carry a raw base only (no real
|
||||
Right variants yet; the baker can extend to them once their real footage is
|
||||
sourced). The `.mp4` binaries are gitignored.
|
||||
`build_pool_manifest.py` holds the hand-authored label/affect content and emits
|
||||
`simulator/sample_media/manifest.json` (the 19-clip pool baseline). The pool clips
|
||||
themselves are sourced via the content pipeline (`tools/pipeline/run.py
|
||||
process_clip`); the `.mp4` binaries are gitignored. (`setup_scales_media.py` is the
|
||||
older one-base-per-scale generator, now superseded.)
|
||||
|
||||
**Run it (Docker):**
|
||||
|
||||
@@ -331,27 +324,45 @@ then open http://localhost:8000.
|
||||
- **Content dial** — picks audio/video channel; "off" and audio-only positions go
|
||||
to black walls.
|
||||
- **Scale ring (endless encoder)** — `⊖ out` / `in ⊕` (or scroll the stage) walk a
|
||||
*closed ring* of neutral "scales of nature" clips — cosmos → forest → abyss and
|
||||
back around (diving past the smallest **wraps** to the largest). It is *relative*
|
||||
(an endless encoder), distinct from the absolute 0–4 knobs: each step plays a
|
||||
short placeholder zoom/warp **transition**, then settles on the next scale, with
|
||||
the current knob alteration still applied. The current scale is named beside the
|
||||
buttons (`name (i/N)`). A **fast spin** (scroll several detents at once, ≥3)
|
||||
collapses to a single quick **blended pass** straight to the destination scale
|
||||
instead of grinding through every transition (scales design §3); slow single
|
||||
steps still chain one full transition per scale crossed.
|
||||
- **Four experience knobs (0–4):**
|
||||
- **Dark / Light** — a live runtime color grade (cool/dark ↔ warm/bright; equal
|
||||
or zero = the raw footage).
|
||||
- **Right (dreamlike)** — selects a discrete pre-baked, flow-stabilized restyle
|
||||
variant and crossfades to it (strength 0 = raw base).
|
||||
- **Left (analytical)** — a live overlay: labelled boxes from the clip's authored
|
||||
annotation track, with more annotations appearing at higher levels. Text is
|
||||
shaped live (the simulator analogue of the Pi's Pango/HarfBuzz path).
|
||||
- **Calibration sliders** — adjust the grade/overlay gain curves live; once a look
|
||||
is liked, bake the values into `DEFAULT_CALIBRATION` in `player/alteration.py`.
|
||||
*closed ring* of neutral "scales of nature" — **cosmos → orbit → coast → reef →
|
||||
abyss** and back around (diving past the smallest **wraps** to the largest). Each
|
||||
scale is a **rotating pool** of vetted clips: landing on a scale plays a random
|
||||
pool member, so the ring feels fresh each pass. It is *relative* (an endless
|
||||
encoder), distinct from the absolute 0–4 knobs: each step plays a short
|
||||
placeholder zoom/warp **transition**, then settles, with the current alteration
|
||||
still applied. The current scale + chosen member is named beside the buttons
|
||||
(`scale · member (i/N · pool N)`). A **fast spin** (≥3 detents at once) collapses
|
||||
to one quick **blended pass** instead of grinding through every transition.
|
||||
- **Three experience knobs:**
|
||||
- **Mood (−4 dark .. 0 .. +4 light)** — a live runtime color grade (cool/dark ↔
|
||||
warm/bright; 0 = the raw footage).
|
||||
- **Right (dreamlike, 0–4)** — a **deterministic real-time painterly dream** (a
|
||||
WebGL Kuwahara restyle of the live frames; holds still across the loop, goes
|
||||
trippy at max). It also drives **progressive emotion tiers**: when both knobs
|
||||
are up, the affect words escalate from basic to compound as Right rises.
|
||||
- **Left (analytical, 0–4)** — a live HUD of labelled boxes from the clip's
|
||||
authored annotation track. Labels use **progressive detail tiers**: low Left
|
||||
shows fewer, more general labels (high-salience objects only); raising Left
|
||||
brings in more objects *and* escalates each label general → specific →
|
||||
scientific → +fact. Time-windowed tracked labels appear only while their
|
||||
subject is on screen and follow it.
|
||||
- **RenderPlan readout** — always shows the exact numbers the engine produced (the
|
||||
project's honesty "X-ray," now over the alteration model).
|
||||
project's honesty "X-ray," over the alteration model).
|
||||
|
||||
The base clips, Right variants, Left annotation track, and string tables come from
|
||||
The base clips, the Left annotation tracks (with tiers + appear/disappear
|
||||
windows), affect anchors, and per-tier string tables all come from
|
||||
`simulator/sample_media/manifest.json`.
|
||||
|
||||
### Authoring labels — the author mode
|
||||
|
||||
Open **http://localhost:8000/author.html** to author a clip's labels by eye
|
||||
(content-pipeline §11.5). Pick a pool clip, **scrub** to where a subject is on
|
||||
screen, **drag a box** over it, set the label key + salience + the four detail
|
||||
tiers, then **Run tracker** to propagate the box into a keyframed track (classical
|
||||
OpenCV optical flow) with an appear/disappear window — or **Add as static box**
|
||||
for a fixed label. Shift-click the stage to place an **affect anchor** and type its
|
||||
emotion tiers. **Save to manifest** writes the entry back via the pipeline's
|
||||
idempotent upsert (keeping the clip's media + provenance, replacing only the
|
||||
authored content). The tracker only propagates **box geometry** — every label
|
||||
string, scientific name, and fact is **hand-authored**, because a generic detector
|
||||
can't produce "*Gymnothorax*" or a lifespan.
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
# Content candidate pool — operator-selected, vetted (2026-06-24)
|
||||
|
||||
> **WIRED (session 0016).** The rotating pool below is now the live manifest:
|
||||
> `simulator/build_pool_manifest.py` regenerates `simulator/sample_media/manifest.json`
|
||||
> from this pool; the ring scale `forest`→`coast` rename + the cleanup list (bottom)
|
||||
> are done; the player picks a random pool member on landing
|
||||
> (`player.ring.pick_clip_id`). See content-pipeline design §11.
|
||||
|
||||
The 5 scales each get a **rotating pool** of clips. When the viewer lands on a
|
||||
scale by zooming the ring in/out, the player **randomly picks one clip from that
|
||||
scale's pool**. (This supersedes the one-clip-per-scale model — see
|
||||
[`docs/superpowers/specs/2026-06-24-content-pipeline-design.md`](./superpowers/specs/2026-06-24-content-pipeline-design.md).)
|
||||
|
||||
Every clip below was operator-selected from the candidate gallery, is **strict
|
||||
public-domain or CC‑BY‑class with attribution**, and was vetted **word‑free /
|
||||
human‑free** (windows trimmed past title cards, captions, telemetry burn‑in,
|
||||
divers, construction, logos). All processed via `tools/pipeline/run.py
|
||||
process_clip` (trim → crossfade‑loop → 1080p proxy). **Media is gitignored**;
|
||||
this file is the re‑sourceable record. Sim ids are the `simulator/sample_media/<id>/` dirs.
|
||||
|
||||
License rule confirmed by operator: **CC0 / CC‑BY / CC‑BY‑SA are acceptable with
|
||||
a credit line** (our overlays + dream are derivatives, which these allow);
|
||||
**CC‑BY‑ND is NOT usable** (no derivatives); **CC‑BY‑NC only if the installation
|
||||
stays non‑commercial**.
|
||||
|
||||
## 🌌 cosmos (pool of 4)
|
||||
|
||||
| id | clip | source | window (s) | license / credit |
|
||||
|----|------|--------|-----------|------------------|
|
||||
| `cosmos` | Orion Nebula flythrough | NASA/JPL‑Caltech — https://images.nasa.gov/details/JPL-20221122-SOLSYSf-0001-Orion%20Dust%20and%20Death | ~30–54 | PD (17 U.S.C. §105) |
|
||||
| `cosmos_galaxies` | Flying Through Galaxies | NASA SVS — https://svs.gsfc.nasa.gov/vis/a010000/a014900/a014950/14950_Galaxies_FlyThrough_4k.mp4 | 8–32 | PD |
|
||||
| `cosmos_hudf` | Hubble Ultra Deep Field zoom | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030600/a030687/hudf-b-1920x1080p30.mov | 20–44 | CC‑BY‑class — credit **NASA, ESA, STScI** |
|
||||
| `cosmos_xdf` | eXtreme Deep Field flythrough | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030600/a030681/hxdf_fly-b-1920x1080p30.mov | 2–26 | CC‑BY‑class — credit **NASA, ESA, STScI** |
|
||||
|
||||
> ⚠️ **DISCREPANCY to fix next session.** Operator selected cosmos **#1, #3
|
||||
> (Orion), #4, #5**. During processing, **#6 Galaxy Traverse was run by mistake
|
||||
> instead of #3 Orion**. The Orion clip (#3) is text‑free and already lives at
|
||||
> `simulator/sample_media/cosmos/base.mp4` (the prior real base), so the pool above
|
||||
> is correct. **`cosmos_traverse` is an UNSELECTED extra on disk — drop it** (or
|
||||
> confirm with operator). The current ring scale id is `cosmos`; when the pool is
|
||||
> wired, `cosmos` becomes one pool member.
|
||||
|
||||
## 🛰️ orbit (pool of 3)
|
||||
|
||||
| id | clip | source | window (s) | license |
|
||||
|----|------|--------|-----------|---------|
|
||||
| `orbit_planetearth` | ISS — View of Planet Earth | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030700/a030771/ISS_View_of_Planet_Earth_2160p.mp4 | 10–34 | PD |
|
||||
| `orbit_crewobs` | ISS — Crew Earth Observations | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030700/a030771/ISS_Crew_Earth_Observations_2160p.mp4 | 9–33 | PD |
|
||||
| `orbit_bluemarble` | NPP "Blue Marble" rotating globe (CG) | NASA SVS — https://svs.gsfc.nasa.gov/vis/a000000/a004500/a004550/BlueMarble_starfield_4k_2160p30_v2.mp4 | 10–34 | PD |
|
||||
|
||||
> Operator also selected aurora→Perth (#4) and west‑coast lights (#5); both
|
||||
> **rejected as UNUSABLE** — #4 has a date/time burned into every frame, #5 has the
|
||||
> ISS module + solar panel in frame throughout. No clean window exists in either.
|
||||
|
||||
## 🏖️ coast (pool of 4) — NEW scale, replaces the granite/waterfall `forest`
|
||||
|
||||
| id | clip | source | window (s) | license |
|
||||
|----|------|--------|-----------|---------|
|
||||
| `coast_birdrock` | Bird rock + ocean (aerial) | NPS Channel Islands — https://www.nps.gov/nps-audiovideo/audiovideo/663a2f13-6183-4b3e-a4a9-6698a84619d01080p.mp4 | 3–27 | PD |
|
||||
| `coast_surfgrass` | Surfgrass / coralline tidepool | NPS Cabrillo — https://www.nps.gov/nps-audiovideo/audiovideo/de7d1cf2-e422-45b5-aa66-4c69e29a4d811080p.mp4 | 2–26 | PD |
|
||||
| `coast_elkbeach` | Elk resting in coastal grass/fog | NPS Redwood — https://www.nps.gov/nps-audiovideo/legacy/redw/4634070E-F68D-4F74-E43174448168AC8A/redw-elkbeach_1280x720.mp4 | 20–38 | PD (720p source) |
|
||||
| `coast_drakesbeach` | Elephant seals on dark sand | NPS Point Reyes — https://www.nps.gov/nps-audiovideo/audiovideo/1cfd8165-1061-4d75-af53-b46c2e3ed2701080p.mp4 | 20–35 | PD |
|
||||
|
||||
> Operator also selected the Lifeboat Station "elephant seal timelapse" (more‑options
|
||||
> #3) — **FAILED**: the NPS page's video resolved to the wrong footage (an aerial of
|
||||
> the bay with road/parking/boats, no seals). An alternate source UUID
|
||||
> (`50a7a20a-9e52-4643-b186-c61ce1cef0f7`) appeared in the page HTML and could be
|
||||
> retried next session if the seal timelapse is still wanted.
|
||||
>
|
||||
> **Scale rename:** the live ring scale is still `forest` (Yosemite). Next session
|
||||
> renames it `coast` and points it at this pool. The old real Yosemite base
|
||||
> (`simulator/sample_media/forest/`) becomes unused.
|
||||
|
||||
## 🐠 reef (pool of 5)
|
||||
|
||||
All NOAA Fisheries b‑roll (PD, credit "NOAA Fisheries"); Brightcove download
|
||||
endpoint `https://download.gallery.brightcove.com/api/gallery/account/659677166001/site/site-600408/video/<VIDEO_ID>/download`.
|
||||
|
||||
| id | clip | VIDEO_ID | window (s) | notes |
|
||||
|----|------|----------|-----------|-------|
|
||||
| `reef_spawning` | Snapper school + yellow fish over sunlit reef | 1553921798001 | 50–78 | clean window from a 12‑min reel |
|
||||
| `reef_hawkfish` | Humphead parrotfish over reef (Hawaiian atoll) | 6039896460001 | 250–274 | |
|
||||
| `reef_lionfish` | Lionfish hovering + blue snapper | 4088881464001 | 12–36 | underwater clean; humans on land after ~190s |
|
||||
| `reef_snapper` | Red snapper schooling (Gulf, greenish water) | 5305427942001 | 7–31 | |
|
||||
| `reef_coralspacific` | Pacific coral macro, spotted fish | 5231464663001 | 30–54 | diver‑free window (divers at 6–25s) |
|
||||
|
||||
## 🦑 abyss (pool of 3)
|
||||
|
||||
All NOAA Ocean Exploration (PD, 17 U.S.C. §105). These have creatures drifting
|
||||
**into and out of frame** — the prime material for time‑windowed tracked labels.
|
||||
|
||||
| id | clip | source | window (s) | creatures |
|
||||
|----|------|--------|-----------|-----------|
|
||||
| `abyss_wow` | "World of Water" | https://oceanexplorer.noaa.gov/wp-content/uploads/2018/05/wow-1280x720-1.mp4 | 10–34 | jelly + ctenophore + siphonophore enter/exit |
|
||||
| `abyss_midwaterexp` | "Midwater Exploration" | https://oceanexplorer.noaa.gov/wp-content/uploads/2019/09/midwater-exploration-1920x1080-1.mp4 | 16–40 | white medusa + red worm enter/exit |
|
||||
| `abyss_hiding` | "Hiding in the Dark" | https://oceanexplorer.noaa.gov/wp-content/uploads/2018/05/dark-1280x720-1.mp4 | 20–44 | crimson jelly + ctenophore enter/exit |
|
||||
|
||||
> Operator also selected Crossota (#4) and comb jelly (#5); both **rejected** —
|
||||
> Crossota has only ~17s clean (too short for a 24s window), comb jelly has a
|
||||
> species caption covering the whole clip.
|
||||
|
||||
## On disk but NOT in any pool — ✅ CLEANED UP (session 0016)
|
||||
|
||||
Removed from local media (all were gitignored, so repo-invisible):
|
||||
|
||||
- `cosmos_traverse` — unselected (the #6/#3 mix‑up above). **dropped.**
|
||||
- `orbit_westcoast` — rejected (ISS hardware in frame). **dropped.**
|
||||
- `forest` — the old Yosemite base, superseded by the `coast` pool. **dropped.**
|
||||
- `reef` — the original placeholder/early reef base, superseded by `reef_*`. **dropped.**
|
||||
- `orbit` / `abyss` — the Increment‑1 single real bases (NASA ISS Exp65 / ctenophore),
|
||||
not in the new pools. **dropped** (the cosmos Orion base stays as the cosmos pool primary).
|
||||
- `review.html` gallery — now `.gitignore`d (local‑only, re‑buildable).
|
||||
@@ -0,0 +1,29 @@
|
||||
# Content sourcing — strict-PD neutral bases
|
||||
|
||||
For each scale, acquire a calm, neutral source clip from a **strict public-domain**
|
||||
US-government source, verify the license, then run it through the pipeline.
|
||||
|
||||
| Scale | Source | Where |
|
||||
|--------|--------------------------------|------------------------------------|
|
||||
| cosmos | NASA / Hubble / JWST | https://images.nasa.gov/ |
|
||||
| orbit | NASA / ISS | https://images.nasa.gov/ |
|
||||
| forest | NPS / USGS | https://www.nps.gov/ , https://www.usgs.gov/ |
|
||||
| reef | NOAA Ocean Exploration | https://oceanexplorer.noaa.gov/ |
|
||||
| abyss | NOAA Ocean Exploration | https://oceanexplorer.noaa.gov/ |
|
||||
|
||||
**License check (mandatory):** confirm the asset is US-gov PD (17 U.S.C. §105) or
|
||||
explicitly CC0. "Free stock" (Pexels/Pixabay/etc.) is royalty-free but NOT public
|
||||
domain — do not use. Record a `Provenance` (tools/pipeline/provenance.py) per clip.
|
||||
|
||||
**Run the pipeline** (replaces the placeholder bytes for a scale):
|
||||
|
||||
```bash
|
||||
python -c "from tools.pipeline.run import process_clip; \
|
||||
process_clip('downloads/<scale>.mp4', 'simulator/sample_media/<scale>', \
|
||||
start=<in_s>, duration=<len_s>, overlap=<0.5..1.0>)"
|
||||
```
|
||||
|
||||
This writes `simulator/sample_media/<scale>/base.mp4` (proxy) + `master.mp4`. Then
|
||||
author the labels (Increment 2: the hybrid track pass + simulator author mode) or
|
||||
hand-edit the annotation track for now (spec §4). The ring + manifest entry already
|
||||
exist; `process_clip` only replaces the media bytes.
|
||||
@@ -0,0 +1,929 @@
|
||||
# Content Pipeline — Increment 1 (real footage, no ML) Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Build the deterministic content-pipeline tooling (frame → crossfade-loop → master+proxy transcode → manifest assembly), add the backward-compatible keyframed annotation-track schema with client interpolation, and extend the scale ring to 5 scales — proving the pipeline end-to-end on real footage without any ML.
|
||||
|
||||
**Architecture:** A new pure-Python `tools/pipeline/` package whose stage functions return ffmpeg argument lists (unit-tested without running ffmpeg); a thin `run.py` executes them and is covered by an opt-in integration test gated on `ffmpeg` + the real forest POC base. Manifest assembly is pure dict transforms. The annotation `track` is optional data that rides through the existing pass-through `annotations` list (no Python model change); interpolation is client-side JS, consistent with the existing "alteration math in Python, label placement in the client" split. This implements §3 (stages 1–4, 6), §4 (track schema), §6 (format), §7 (tooling), and Increment 1 of §8 in [`docs/superpowers/specs/2026-06-24-content-pipeline-design.md`](../specs/2026-06-24-content-pipeline-design.md).
|
||||
|
||||
**Tech Stack:** Python 3.13, pytest, ffmpeg (libx264), vanilla JS (SVG overlay). Tests run with `pytest -q` from the repo root in the project `.venv`.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Scaffold `tools/pipeline/` + the frame stage
|
||||
|
||||
**Files:**
|
||||
- Create: `tools/pipeline/__init__.py`
|
||||
- Create: `tools/pipeline/ffmpeg_ops.py`
|
||||
- Test: `tests/test_pipeline_ffmpeg_ops.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
# tests/test_pipeline_ffmpeg_ops.py
|
||||
from tools.pipeline.ffmpeg_ops import frame_args
|
||||
|
||||
|
||||
def test_frame_args_trims_scales_pads_and_strips_audio():
|
||||
args = frame_args(
|
||||
"src.mp4", "out.mp4",
|
||||
start=2.0, duration=8.0, width=1920, height=1080, ff="ffmpeg",
|
||||
)
|
||||
assert args[0] == "ffmpeg"
|
||||
assert "-ss" in args and args[args.index("-ss") + 1] == "2.0"
|
||||
assert "-t" in args and args[args.index("-t") + 1] == "8.0"
|
||||
assert "-an" in args # audio stripped (bases are video-only)
|
||||
vf = args[args.index("-vf") + 1]
|
||||
assert "scale=1920:1080:force_original_aspect_ratio=decrease" in vf
|
||||
assert "pad=1920:1080" in vf
|
||||
assert "format=yuv420p" in vf
|
||||
assert args[-1] == "out.mp4"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pytest tests/test_pipeline_ffmpeg_ops.py -q`
|
||||
Expected: FAIL with `ModuleNotFoundError: No module named 'tools.pipeline'`
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
```python
|
||||
# tools/pipeline/__init__.py
|
||||
"""Content pipeline: source -> frame -> loop -> transcode -> manifest.
|
||||
|
||||
Pure ffmpeg-argument builders (ffmpeg_ops), a thin runner (run), provenance
|
||||
records, and manifest assembly. See docs/superpowers/specs/2026-06-24-content-pipeline-design.md.
|
||||
"""
|
||||
```
|
||||
|
||||
```python
|
||||
# tools/pipeline/ffmpeg_ops.py
|
||||
"""Pure ffmpeg argument builders (no I/O) for the content pipeline.
|
||||
|
||||
Each function returns a list[str] ready for ffmpeg, so command construction is
|
||||
unit-testable without running ffmpeg. tools/pipeline/run.py executes them. The
|
||||
ffmpeg binary name is injected (default "ffmpeg").
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def _fit(width: int, height: int) -> str:
|
||||
"""A scale+pad filter chain fitting any input into width×height (letterbox),
|
||||
fixing SAR and pixel format so downstream concat/xfade get identical geometry."""
|
||||
return (
|
||||
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
|
||||
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
|
||||
"setsar=1,format=yuv420p"
|
||||
)
|
||||
|
||||
|
||||
def frame_args(src, dst, *, start: float, duration: float,
|
||||
width: int, height: int, ff: str = "ffmpeg") -> list[str]:
|
||||
"""Stage 2 — trim [start, start+duration], fit to width×height, strip audio.
|
||||
Produces a high-quality mezzanine for the loop stage."""
|
||||
return [
|
||||
ff, "-y", "-ss", str(start), "-t", str(duration),
|
||||
"-i", str(src), "-vf", _fit(width, height), "-an",
|
||||
"-c:v", "libx264", "-preset", "medium", "-crf", "16",
|
||||
"-pix_fmt", "yuv420p", "-movflags", "+faststart", str(dst),
|
||||
]
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `pytest tests/test_pipeline_ffmpeg_ops.py -q`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tools/pipeline/__init__.py tools/pipeline/ffmpeg_ops.py tests/test_pipeline_ffmpeg_ops.py
|
||||
git commit -m "feat(pipeline): frame stage ffmpeg arg builder (content pipeline §3.2)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Crossfade seamless-loop stage
|
||||
|
||||
**Files:**
|
||||
- Modify: `tools/pipeline/ffmpeg_ops.py`
|
||||
- Test: `tests/test_pipeline_ffmpeg_ops.py`
|
||||
|
||||
The seamless loop (spec §3.3) crossfades the clip's tail over its head. For a clip
|
||||
of duration `D` with overlap `O`, the output (length `D − O`) is
|
||||
`concat( xfade(tail, head, O), mid )` where `head`=`[0,O]`, `mid`=`[O, D−O]`,
|
||||
`tail`=`[D−O, D]`. Both internal joins and the loop seam are then continuous, and
|
||||
the tail→head jump is blended away over `O`.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
# add to tests/test_pipeline_ffmpeg_ops.py
|
||||
from tools.pipeline.ffmpeg_ops import crossfade_loop_args
|
||||
|
||||
|
||||
def test_crossfade_loop_builds_head_mid_tail_xfade_concat():
|
||||
args = crossfade_loop_args("in.mp4", "loop.mp4", duration=8.0, overlap=1.0)
|
||||
fc = args[args.index("-filter_complex") + 1]
|
||||
assert "trim=0:1.0" in fc # head = first O secs
|
||||
assert "trim=1.0:7.0" in fc # mid = [O, D-O]
|
||||
assert "trim=7.0:8.0" in fc # tail = last O secs
|
||||
assert "xfade=transition=fade:duration=1.0:offset=0" in fc
|
||||
assert "concat=n=2:v=1" in fc
|
||||
assert "-an" in args
|
||||
assert args[-1] == "loop.mp4"
|
||||
|
||||
|
||||
def test_crossfade_loop_rejects_overlap_too_large():
|
||||
import pytest
|
||||
with pytest.raises(ValueError):
|
||||
crossfade_loop_args("in.mp4", "loop.mp4", duration=4.0, overlap=2.0)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pytest tests/test_pipeline_ffmpeg_ops.py -q`
|
||||
Expected: FAIL with `ImportError: cannot import name 'crossfade_loop_args'`
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
```python
|
||||
# add to tools/pipeline/ffmpeg_ops.py
|
||||
def crossfade_loop_args(src, dst, *, duration: float, overlap: float,
|
||||
ff: str = "ffmpeg") -> list[str]:
|
||||
"""Stage 3 — make `src` loop seamlessly by crossfading its tail over its head.
|
||||
Output length = duration − overlap. Requires overlap < duration/2 so a non-empty
|
||||
middle remains."""
|
||||
if overlap <= 0 or overlap >= duration / 2:
|
||||
raise ValueError(f"overlap {overlap} must be in (0, duration/2={duration/2})")
|
||||
d, o = duration, overlap
|
||||
fc = (
|
||||
f"[0:v]trim=0:{o},setpts=PTS-STARTPTS[head];"
|
||||
f"[0:v]trim={o}:{d - o},setpts=PTS-STARTPTS[mid];"
|
||||
f"[0:v]trim={d - o}:{d},setpts=PTS-STARTPTS[tail];"
|
||||
f"[tail][head]xfade=transition=fade:duration={o}:offset=0[xf];"
|
||||
f"[xf][mid]concat=n=2:v=1,format=yuv420p[out]"
|
||||
)
|
||||
return [
|
||||
ff, "-y", "-i", str(src), "-filter_complex", fc,
|
||||
"-map", "[out]", "-an",
|
||||
"-c:v", "libx264", "-preset", "medium", "-crf", "16",
|
||||
"-pix_fmt", "yuv420p", "-movflags", "+faststart", str(dst),
|
||||
]
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `pytest tests/test_pipeline_ffmpeg_ops.py -q`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tools/pipeline/ffmpeg_ops.py tests/test_pipeline_ffmpeg_ops.py
|
||||
git commit -m "feat(pipeline): crossfade seamless-loop arg builder (content pipeline §3.3)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Master + proxy transcode stage
|
||||
|
||||
**Files:**
|
||||
- Modify: `tools/pipeline/ffmpeg_ops.py`
|
||||
- Test: `tests/test_pipeline_ffmpeg_ops.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
# add to tests/test_pipeline_ffmpeg_ops.py
|
||||
from tools.pipeline.ffmpeg_ops import transcode_args
|
||||
|
||||
|
||||
def test_transcode_args_high_profile_even_dims_faststart():
|
||||
args = transcode_args("loop.mp4", "proxy.mp4", width=1920, height=1080,
|
||||
crf=20, fps=30)
|
||||
assert "-profile:v" in args and args[args.index("-profile:v") + 1] == "high"
|
||||
assert "-crf" in args and args[args.index("-crf") + 1] == "20"
|
||||
vf = args[args.index("-vf") + 1]
|
||||
assert "scale=1920:1080:force_original_aspect_ratio=decrease" in vf
|
||||
assert "fps=30" in vf
|
||||
assert "+faststart" in args
|
||||
assert "-an" in args
|
||||
assert args[-1] == "proxy.mp4"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pytest tests/test_pipeline_ffmpeg_ops.py -q`
|
||||
Expected: FAIL with `ImportError: cannot import name 'transcode_args'`
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
```python
|
||||
# add to tools/pipeline/ffmpeg_ops.py
|
||||
def transcode_args(src, dst, *, width: int, height: int, crf: int,
|
||||
fps: int = 30, ff: str = "ffmpeg") -> list[str]:
|
||||
"""Stage 4 — encode a final deliverable (master or proxy). Master: source res
|
||||
capped ~3840 wide, crf 18. Proxy: 1920×1080, crf 20. Both H.264 High, yuv420p,
|
||||
even dims, +faststart, video-only (spec §6)."""
|
||||
vf = (
|
||||
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
|
||||
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
|
||||
f"setsar=1,fps={fps},format=yuv420p"
|
||||
)
|
||||
return [
|
||||
ff, "-y", "-i", str(src), "-vf", vf, "-an",
|
||||
"-c:v", "libx264", "-profile:v", "high", "-preset", "slow",
|
||||
"-crf", str(crf), "-pix_fmt", "yuv420p",
|
||||
"-movflags", "+faststart", str(dst),
|
||||
]
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `pytest tests/test_pipeline_ffmpeg_ops.py -q`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tools/pipeline/ffmpeg_ops.py tests/test_pipeline_ffmpeg_ops.py
|
||||
git commit -m "feat(pipeline): master/proxy transcode arg builder (content pipeline §3.4/§6)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Provenance record (stage 1 output)
|
||||
|
||||
**Files:**
|
||||
- Create: `tools/pipeline/provenance.py`
|
||||
- Test: `tests/test_pipeline_provenance.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
# tests/test_pipeline_provenance.py
|
||||
from tools.pipeline.provenance import Provenance
|
||||
|
||||
|
||||
def test_provenance_to_dict_roundtrip():
|
||||
p = Provenance(
|
||||
scale="abyss",
|
||||
license="public-domain (US Gov, 17 U.S.C. §105)",
|
||||
source="NOAA Ocean Exploration",
|
||||
url="https://oceanexplorer.noaa.gov/",
|
||||
fetched_at="2026-06-24",
|
||||
)
|
||||
d = p.to_dict()
|
||||
assert d == {
|
||||
"scale": "abyss",
|
||||
"license": "public-domain (US Gov, 17 U.S.C. §105)",
|
||||
"source": "NOAA Ocean Exploration",
|
||||
"url": "https://oceanexplorer.noaa.gov/",
|
||||
"fetched_at": "2026-06-24",
|
||||
}
|
||||
assert Provenance.from_dict(d) == p
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pytest tests/test_pipeline_provenance.py -q`
|
||||
Expected: FAIL with `ModuleNotFoundError: No module named 'tools.pipeline.provenance'`
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
```python
|
||||
# tools/pipeline/provenance.py
|
||||
"""Stage 1 — the per-clip strict-PD provenance record (spec §3.1).
|
||||
|
||||
Captured at source time so license + source + URL travel with the media. The
|
||||
'no explicit license -> assume PD, verify' trap (scales design §2.1) applies:
|
||||
"free stock" (Pexels/Pixabay) is NOT public domain.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict, dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Provenance:
|
||||
scale: str
|
||||
license: str
|
||||
source: str
|
||||
url: str
|
||||
fetched_at: str # ISO date; passed in (no clock here -> deterministic)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, d: dict) -> "Provenance":
|
||||
return cls(
|
||||
scale=d["scale"], license=d["license"], source=d["source"],
|
||||
url=d["url"], fetched_at=d["fetched_at"],
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `pytest tests/test_pipeline_provenance.py -q`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tools/pipeline/provenance.py tests/test_pipeline_provenance.py
|
||||
git commit -m "feat(pipeline): strict-PD provenance record (content pipeline §3.1)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Manifest assembly (stage 6)
|
||||
|
||||
**Files:**
|
||||
- Create: `tools/pipeline/manifest.py`
|
||||
- Test: `tests/test_pipeline_manifest.py`
|
||||
|
||||
Pure dict transforms over a loaded manifest: upsert a clip entry, place a scale in
|
||||
the ring order, and rebuild the per-edge transition list so it always has exactly
|
||||
N entries for N scales (one transition per ring edge, including the wrap).
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
# tests/test_pipeline_manifest.py
|
||||
from tools.pipeline.manifest import upsert_clip, add_ring_scale, rebuild_ring_edges
|
||||
|
||||
|
||||
def _base():
|
||||
return {"clips": [{"id": "cosmos", "title": "Cosmos"}],
|
||||
"ring": {"scales": [{"id": "cosmos", "clip_id": "cosmos"}],
|
||||
"transitions": []}}
|
||||
|
||||
|
||||
def test_upsert_clip_replaces_by_id_else_appends():
|
||||
m = _base()
|
||||
m = upsert_clip(m, {"id": "reef", "title": "Reef"})
|
||||
assert [c["id"] for c in m["clips"]] == ["cosmos", "reef"]
|
||||
m = upsert_clip(m, {"id": "reef", "title": "Coral Reef"}) # replace, not dup
|
||||
assert [c["id"] for c in m["clips"]] == ["cosmos", "reef"]
|
||||
assert m["clips"][1]["title"] == "Coral Reef"
|
||||
|
||||
|
||||
def test_add_ring_scale_inserts_after_named_scale():
|
||||
m = _base()
|
||||
m = add_ring_scale(m, "orbit", "orbit", after="cosmos")
|
||||
assert [s["id"] for s in m["ring"]["scales"]] == ["cosmos", "orbit"]
|
||||
|
||||
|
||||
def test_rebuild_ring_edges_one_transition_per_edge_with_wrap():
|
||||
m = _base()
|
||||
m = add_ring_scale(m, "orbit", "orbit", after="cosmos")
|
||||
m = add_ring_scale(m, "abyss", "abyss", after="orbit")
|
||||
m = rebuild_ring_edges(m)
|
||||
files = [t["file"] for t in m["ring"]["transitions"]]
|
||||
assert files == [
|
||||
"transitions/cosmos-orbit.mp4",
|
||||
"transitions/orbit-abyss.mp4",
|
||||
"transitions/abyss-cosmos.mp4", # wrap edge
|
||||
]
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pytest tests/test_pipeline_manifest.py -q`
|
||||
Expected: FAIL with `ModuleNotFoundError: No module named 'tools.pipeline.manifest'`
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
```python
|
||||
# tools/pipeline/manifest.py
|
||||
"""Stage 6 — assemble manifest entries (spec §3.6). Pure dict transforms over a
|
||||
loaded manifest dict; the caller does the json read/write. Idempotent by id so
|
||||
re-running earlier stages never clobbers authored labels."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
|
||||
|
||||
def upsert_clip(manifest: dict, clip: dict) -> dict:
|
||||
"""Replace the clip with the same id, or append it. Returns a new manifest."""
|
||||
m = copy.deepcopy(manifest)
|
||||
clips = m.setdefault("clips", [])
|
||||
for i, c in enumerate(clips):
|
||||
if c["id"] == clip["id"]:
|
||||
clips[i] = clip
|
||||
return m
|
||||
clips.append(clip)
|
||||
return m
|
||||
|
||||
|
||||
def add_ring_scale(manifest: dict, scale_id: str, clip_id: str,
|
||||
after: str | None = None) -> dict:
|
||||
"""Insert a scale into ring.scales after the named scale (or append if `after`
|
||||
is None / not found). No-op if scale_id already present. Returns a new manifest."""
|
||||
m = copy.deepcopy(manifest)
|
||||
ring = m.setdefault("ring", {"scales": [], "transitions": []})
|
||||
scales = ring.setdefault("scales", [])
|
||||
if any(s["id"] == scale_id for s in scales):
|
||||
return m
|
||||
entry = {"id": scale_id, "clip_id": clip_id}
|
||||
idx = next((i for i, s in enumerate(scales) if s["id"] == after), None)
|
||||
if idx is None:
|
||||
scales.append(entry)
|
||||
else:
|
||||
scales.insert(idx + 1, entry)
|
||||
return m
|
||||
|
||||
|
||||
def rebuild_ring_edges(manifest: dict) -> dict:
|
||||
"""Rebuild ring.transitions to one placeholder entry per ring edge (N scales ->
|
||||
N edges incl. the wrap), preserving any existing model string for an edge.
|
||||
Returns a new manifest."""
|
||||
m = copy.deepcopy(manifest)
|
||||
ring = m.setdefault("ring", {"scales": [], "transitions": []})
|
||||
scales = ring.get("scales", [])
|
||||
existing = {t["file"]: t.get("model", "") for t in ring.get("transitions", [])}
|
||||
edges = []
|
||||
n = len(scales)
|
||||
for i in range(n):
|
||||
a, b = scales[i]["id"], scales[(i + 1) % n]["id"]
|
||||
file = f"transitions/{a}-{b}.mp4"
|
||||
edges.append({"file": file, "model": existing.get(file, "placeholder-zoom")})
|
||||
ring["transitions"] = edges
|
||||
return m
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `pytest tests/test_pipeline_manifest.py -q`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tools/pipeline/manifest.py tests/test_pipeline_manifest.py
|
||||
git commit -m "feat(pipeline): manifest assembly — upsert clip + ring scale/edges (content pipeline §3.6)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Runner + opt-in integration test on the real forest POC base
|
||||
|
||||
**Files:**
|
||||
- Create: `tools/pipeline/run.py`
|
||||
- Test: `tests/test_pipeline_integration.py`
|
||||
|
||||
The runner shells out ffmpeg with the builders from Tasks 1–3, ffprobing the source
|
||||
duration so the loop stage gets real `D`. The integration test runs the **real**
|
||||
forest POC base (`simulator/sample_media/forest/base.mp4`, populated by
|
||||
`simulator/setup_sample_media.py`) through frame → loop → transcode and ffprobes the
|
||||
proxy — gated on `ffmpeg`/`ffprobe` and on the base existing (it's gitignored media),
|
||||
matching `tests/test_tools_integration.py`'s gating.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
# tests/test_pipeline_integration.py
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tools.pipeline.run import process_clip
|
||||
|
||||
_HAS_FF = shutil.which("ffmpeg") is not None and shutil.which("ffprobe") is not None
|
||||
_FOREST = Path("simulator/sample_media/forest/base.mp4")
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _HAS_FF, reason="ffmpeg/ffprobe not installed")
|
||||
@pytest.mark.skipif(not _FOREST.exists(),
|
||||
reason="forest POC base absent (run simulator/setup_sample_media.py)")
|
||||
def test_process_clip_emits_proxy_and_master(tmp_path):
|
||||
out = process_clip(_FOREST, tmp_path, overlap=0.5, start=0.0, duration=4.0)
|
||||
assert out["proxy"].exists() and out["master"].exists()
|
||||
# Proxy is exactly 1920x1080.
|
||||
probe = subprocess.run(
|
||||
["ffprobe", "-v", "quiet", "-select_streams", "v:0",
|
||||
"-show_entries", "stream=width,height", "-of", "csv=p=0", str(out["proxy"])],
|
||||
capture_output=True, text=True, check=True,
|
||||
).stdout.strip()
|
||||
assert probe == "1920,1080"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pytest tests/test_pipeline_integration.py -q`
|
||||
Expected: FAIL with `ModuleNotFoundError: No module named 'tools.pipeline.run'`
|
||||
(or SKIP if ffmpeg/the base are absent — then verify by importing in `python -c`.)
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
```python
|
||||
# tools/pipeline/run.py
|
||||
"""Thin runner: ffprobe the source, then execute the frame -> loop -> transcode
|
||||
stages with tools.pipeline.ffmpeg_ops. The pure arg builders are unit-tested;
|
||||
this glue is covered by the opt-in integration test."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from .ffmpeg_ops import crossfade_loop_args, frame_args, transcode_args
|
||||
|
||||
MASTER_MAX_W = 3840
|
||||
|
||||
|
||||
def probe_duration(src, *, ff: str = "ffprobe") -> float:
|
||||
out = subprocess.run(
|
||||
[ff, "-v", "quiet", "-show_entries", "format=duration",
|
||||
"-of", "csv=p=0", str(src)],
|
||||
capture_output=True, text=True, check=True,
|
||||
).stdout.strip()
|
||||
return float(out)
|
||||
|
||||
|
||||
def _run(args: list[str]) -> None:
|
||||
subprocess.run(args, check=True, capture_output=True)
|
||||
|
||||
|
||||
def process_clip(src, out_dir, *, start: float = 0.0, duration: float | None = None,
|
||||
overlap: float = 1.0, master_w: int = 3840, master_h: int = 2160,
|
||||
ff: str = "ffmpeg") -> dict:
|
||||
"""Run stages 2–4 for one clip; return {'master','proxy','loop','mezzanine'} paths.
|
||||
`duration` defaults to the full source length (minus the trim start)."""
|
||||
src = Path(src)
|
||||
out_dir = Path(out_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
if duration is None:
|
||||
duration = probe_duration(src) - start
|
||||
|
||||
mezz = out_dir / "mezzanine.mp4"
|
||||
loop = out_dir / "loop.mp4"
|
||||
master = out_dir / "master.mp4"
|
||||
proxy = out_dir / "base.mp4" # the proxy is what the manifest's base_file points at
|
||||
|
||||
_run(frame_args(src, mezz, start=start, duration=duration,
|
||||
width=master_w, height=master_h, ff=ff))
|
||||
_run(crossfade_loop_args(mezz, loop, duration=duration, overlap=overlap, ff=ff))
|
||||
_run(transcode_args(loop, master, width=master_w, height=master_h, crf=18, ff=ff))
|
||||
_run(transcode_args(loop, proxy, width=1920, height=1080, crf=20, ff=ff))
|
||||
return {"mezzanine": mezz, "loop": loop, "master": master, "proxy": proxy}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes (or skips cleanly)**
|
||||
|
||||
Run: `pytest tests/test_pipeline_integration.py -q`
|
||||
Expected: PASS if ffmpeg + the forest base are present; otherwise SKIP with the gate
|
||||
reason. If skipped, also run `python -c "from tools.pipeline.run import process_clip"`
|
||||
and expect no error.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tools/pipeline/run.py tests/test_pipeline_integration.py
|
||||
git commit -m "feat(pipeline): runner + opt-in integration test on real forest base (content pipeline §3)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Client annotation-track interpolation + a hand-authored track
|
||||
|
||||
**Files:**
|
||||
- Modify: `simulator/static/app.js` (`renderOverlay` at lines ~235–269; `paintLoop` at ~173–186)
|
||||
- Modify: `simulator/sample_media/manifest.json` (add a `track` to the forest `detected.water` annotation)
|
||||
|
||||
The track is *data* on an existing annotation; the Python `Clip` already passes
|
||||
`annotations` through opaquely (`simulator/clips.py` `_clip_from_dict` does
|
||||
`d.get("annotations", [])`), so **no Python change is needed**. Interpolation is
|
||||
client-side (spec §4). Because the overlay currently re-renders only on knob change,
|
||||
a tracked box must update as the video plays — drive a re-render from the existing
|
||||
`paintLoop` rAF when the active clip has any track and the overlay is visible.
|
||||
|
||||
- [ ] **Step 1: Add the data — a hand-authored track on forest's water detection**
|
||||
|
||||
In `simulator/sample_media/manifest.json`, replace the forest `detected.water`
|
||||
annotation (currently `{"key": "detected.water", "box": [0.30, 0.10, 0.18, 0.70], "min_level": 1}`)
|
||||
with a keyframed track (a small horizontal drift, looping at t=0 and t=1 so it is
|
||||
seamless):
|
||||
|
||||
```json
|
||||
{
|
||||
"key": "detected.water",
|
||||
"min_level": 1,
|
||||
"track": [
|
||||
{"t": 0.0, "box": [0.30, 0.10, 0.18, 0.70]},
|
||||
{"t": 0.5, "box": [0.33, 0.10, 0.18, 0.70]},
|
||||
{"t": 1.0, "box": [0.30, 0.10, 0.18, 0.70]}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add the interpolation helper + use it in `renderOverlay`**
|
||||
|
||||
In `simulator/static/app.js`, add a helper above `renderOverlay` and use it instead
|
||||
of `a.box`:
|
||||
|
||||
```javascript
|
||||
// Interpolate an annotation's box at loop-normalized time t (0..1). Static labels
|
||||
// (no track) return their fixed box. Linear between sorted keyframes; clamps to ends.
|
||||
function boxAt(a, t) {
|
||||
if (!a.track || !a.track.length) return a.box;
|
||||
const ks = a.track;
|
||||
if (t <= ks[0].t) return ks[0].box;
|
||||
if (t >= ks[ks.length - 1].t) return ks[ks.length - 1].box;
|
||||
for (let i = 1; i < ks.length; i++) {
|
||||
if (t <= ks[i].t) {
|
||||
const a0 = ks[i - 1], a1 = ks[i];
|
||||
const f = (t - a0.t) / (a1.t - a0.t);
|
||||
return a0.box.map((v, j) => v + (a1.box[j] - v) * f);
|
||||
}
|
||||
}
|
||||
return ks[ks.length - 1].box;
|
||||
}
|
||||
|
||||
// Loop-normalized playback time of the base video (0..1), for track interpolation.
|
||||
function loopT() {
|
||||
return vid && vid.duration ? (vid.currentTime % vid.duration) / vid.duration : 0;
|
||||
}
|
||||
```
|
||||
|
||||
Then in `renderOverlay`, change the box line:
|
||||
|
||||
```javascript
|
||||
const [x, y, w, h] = boxAt(a, loopT()).map((n) => n * 100);
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Re-render tracked overlays each frame from `paintLoop`**
|
||||
|
||||
Store the last overlay args and, in `paintLoop`, re-run `renderOverlay` when the
|
||||
active clip has any tracked annotation and the overlay is visible. Add near the top
|
||||
of `app.js` (module scope): `let lastOverlay = null;`. At the end of `renderOverlay`,
|
||||
record the call: set `lastOverlay = { level, intensity };` (add this line just before
|
||||
the closing brace). In `paintLoop` (after the existing shader work, before
|
||||
`requestAnimationFrame(paintLoop)`), add:
|
||||
|
||||
```javascript
|
||||
// Animate keyframed annotation tracks: re-place boxes against playback time.
|
||||
const c = activeClip();
|
||||
if (lastOverlay && lastOverlay.level > 0 && c &&
|
||||
c.annotations.some((a) => a.track && a.track.length)) {
|
||||
renderOverlay(lastOverlay.level, lastOverlay.intensity);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify by eye (no JS unit harness in this repo)**
|
||||
|
||||
Run the sim and watch the forest water reticle drift with playback, looping
|
||||
seamlessly; static labels (rock_face, conifer, the measurement) stay put.
|
||||
|
||||
```bash
|
||||
python -m simulator.app # then open the printed localhost URL
|
||||
```
|
||||
|
||||
Expected: with Left up on forest, the `flowing water` box drifts right then back
|
||||
over the loop with no jump at the loop seam; affect/measure labels unchanged.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add simulator/static/app.js simulator/sample_media/manifest.json
|
||||
git commit -m "feat(sim): client annotation-track interpolation + hand-authored forest track (content pipeline §4)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Extend the ring to 5 scales (orbit + reef placeholders)
|
||||
|
||||
**Files:**
|
||||
- Modify: `simulator/setup_scales_media.py` (`SCALES` lines ~32–36; `EDGES` line ~39)
|
||||
- Modify: `simulator/sample_media/manifest.json` (add `orbit` + `reef` clips; reorder `ring.scales`; rebuild `ring.transitions`)
|
||||
|
||||
Real strict-PD bases are sourced separately (Task 9); to keep the 5-scale ring
|
||||
demonstrable now, generate labelled placeholders for the two new scales with the
|
||||
existing generator, exactly as cosmos/abyss are today.
|
||||
|
||||
- [ ] **Step 1: Add orbit + reef to the placeholder generator**
|
||||
|
||||
In `simulator/setup_scales_media.py`, extend `SCALES` and `EDGES` to the 5-scale
|
||||
ring (order: cosmos → orbit → forest → reef → abyss → wrap):
|
||||
|
||||
```python
|
||||
SCALES = {
|
||||
"cosmos": ("0x05060f", "COSMOS — NASA/Hubble (PD placeholder)"),
|
||||
"orbit": ("0x081427", "ORBIT — NASA/ISS (PD placeholder)"),
|
||||
"forest": ("0x12301a", "FOREST — NPS/USGS (POC/placeholder)"),
|
||||
"reef": ("0x06303a", "REEF — NOAA Ocean Exploration (PD placeholder)"),
|
||||
"abyss": ("0x021016", "ABYSS — NOAA Ocean Exploration (PD placeholder)"),
|
||||
}
|
||||
|
||||
EDGES = [("cosmos", "orbit"), ("orbit", "forest"), ("forest", "reef"),
|
||||
("reef", "abyss"), ("abyss", "cosmos")]
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add the orbit + reef clip entries to the manifest**
|
||||
|
||||
In `simulator/sample_media/manifest.json`, add two clip objects (after `cosmos` and
|
||||
after `forest` respectively) mirroring the existing placeholder clips' shape, e.g.
|
||||
orbit:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "orbit",
|
||||
"title": "Earth from orbit (NASA/ISS, neutral base)",
|
||||
"base_file": "orbit/base.mp4",
|
||||
"license": "public-domain (US Gov, 17 U.S.C. §105)",
|
||||
"source": "NASA/ISS — https://images.nasa.gov/ (true PD; placeholder bytes until ingest)",
|
||||
"right_variants": {},
|
||||
"annotations": [
|
||||
{"key": "detected.cloud_band", "box": [0.30, 0.30, 0.30, 0.20], "min_level": 1},
|
||||
{"key": "measure.altitude", "box": [0.06, 0.06, 0.18, 0.08], "min_level": 2}
|
||||
],
|
||||
"affect": [
|
||||
{"key": "feel.serenity", "at": [0.50, 0.46], "min_level": 1},
|
||||
{"key": "feel.fragility", "at": [0.22, 0.68], "min_level": 2},
|
||||
{"key": "feel.unity", "at": [0.66, 0.58], "min_level": 3},
|
||||
{"key": "feel.distance", "at": [0.42, 0.82], "min_level": 4}
|
||||
],
|
||||
"strings": {
|
||||
"en": {
|
||||
"detected.cloud_band": "cloud band",
|
||||
"measure.altitude": "~408 km",
|
||||
"feel.serenity": "serenity", "feel.fragility": "fragility",
|
||||
"feel.unity": "unity", "feel.distance": "distance"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
And reef:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "reef",
|
||||
"title": "Coral reef (NOAA Ocean Exploration, neutral base)",
|
||||
"base_file": "reef/base.mp4",
|
||||
"license": "public-domain (US Gov, 17 U.S.C. §105)",
|
||||
"source": "NOAA Ocean Exploration — https://oceanexplorer.noaa.gov/ (true PD, worldwide; placeholder bytes until ingest)",
|
||||
"right_variants": {},
|
||||
"annotations": [
|
||||
{"key": "detected.coral", "box": [0.20, 0.45, 0.30, 0.30], "min_level": 1},
|
||||
{"key": "detected.fish", "min_level": 2,
|
||||
"track": [
|
||||
{"t": 0.0, "box": [0.20, 0.30, 0.10, 0.08]},
|
||||
{"t": 0.5, "box": [0.62, 0.34, 0.10, 0.08]},
|
||||
{"t": 1.0, "box": [0.20, 0.30, 0.10, 0.08]}
|
||||
]},
|
||||
{"key": "measure.depth", "box": [0.06, 0.06, 0.16, 0.08], "min_level": 3}
|
||||
],
|
||||
"affect": [
|
||||
{"key": "feel.delight", "at": [0.50, 0.46], "min_level": 1},
|
||||
{"key": "feel.abundance", "at": [0.22, 0.68], "min_level": 2},
|
||||
{"key": "feel.curiosity", "at": [0.66, 0.58], "min_level": 3},
|
||||
{"key": "feel.immersion", "at": [0.42, 0.82], "min_level": 4}
|
||||
],
|
||||
"strings": {
|
||||
"en": {
|
||||
"detected.coral": "coral colony", "detected.fish": "reef fish",
|
||||
"measure.depth": "−18 m",
|
||||
"feel.delight": "delight", "feel.abundance": "abundance",
|
||||
"feel.curiosity": "curiosity", "feel.immersion": "immersion"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Reorder the ring + rebuild edges in the manifest**
|
||||
|
||||
Set `ring.scales` to the 5-scale order and `ring.transitions` to the 5 edges:
|
||||
|
||||
```json
|
||||
"ring": {
|
||||
"scales": [
|
||||
{"id": "cosmos", "clip_id": "cosmos"},
|
||||
{"id": "orbit", "clip_id": "orbit"},
|
||||
{"id": "forest", "clip_id": "forest"},
|
||||
{"id": "reef", "clip_id": "reef"},
|
||||
{"id": "abyss", "clip_id": "abyss"}
|
||||
],
|
||||
"transitions": [
|
||||
{"file": "transitions/cosmos-orbit.mp4", "model": "placeholder-zoom"},
|
||||
{"file": "transitions/orbit-forest.mp4", "model": "placeholder-zoom"},
|
||||
{"file": "transitions/forest-reef.mp4", "model": "placeholder-zoom"},
|
||||
{"file": "transitions/reef-abyss.mp4", "model": "placeholder-zoom"},
|
||||
{"file": "transitions/abyss-cosmos.mp4", "model": "placeholder-zoom"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Regenerate placeholder media + verify the 5-scale ring**
|
||||
|
||||
```bash
|
||||
python simulator/setup_scales_media.py
|
||||
python -m simulator.app # open the URL; spin the encoder full circle
|
||||
```
|
||||
|
||||
Expected: the encoder walks cosmos → orbit → forest → reef → abyss → (wrap) cosmos
|
||||
with a transition on every edge; `GET /api/ring` lists 5 scales + 5 transitions; the
|
||||
reef `reef fish` box drifts with playback (Task 7 interpolation) when Left ≥ 2.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add simulator/setup_scales_media.py simulator/sample_media/manifest.json
|
||||
git commit -m "feat(sim): extend scale ring to 5 (orbit + reef) (content pipeline §5)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Sourcing procedure doc + ROADMAP update
|
||||
|
||||
**Files:**
|
||||
- Create: `docs/content-sourcing.md`
|
||||
- Modify: `docs/ROADMAP.md` (sub-project 3 "Offline v2v variant pipeline" bullet, lines ~174–187)
|
||||
|
||||
Sourcing the real strict-PD bytes is a curatorial task, not a code step. Document the
|
||||
procedure so anyone can run the pipeline per clip, and record the new content-pipeline
|
||||
work on the roadmap.
|
||||
|
||||
- [ ] **Step 1: Write the sourcing procedure**
|
||||
|
||||
```markdown
|
||||
# Content sourcing — strict-PD neutral bases
|
||||
|
||||
For each scale, acquire a calm, neutral source clip from a **strict public-domain**
|
||||
US-government source, verify the license, then run it through the pipeline.
|
||||
|
||||
| Scale | Source | Where |
|
||||
|--------|--------------------------------|------------------------------------|
|
||||
| cosmos | NASA / Hubble / JWST | https://images.nasa.gov/ |
|
||||
| orbit | NASA / ISS | https://images.nasa.gov/ |
|
||||
| forest | NPS / USGS | https://www.nps.gov/ , https://www.usgs.gov/ |
|
||||
| reef | NOAA Ocean Exploration | https://oceanexplorer.noaa.gov/ |
|
||||
| abyss | NOAA Ocean Exploration | https://oceanexplorer.noaa.gov/ |
|
||||
|
||||
**License check (mandatory):** confirm the asset is US-gov PD (17 U.S.C. §105) or
|
||||
explicitly CC0. "Free stock" (Pexels/Pixabay/etc.) is royalty-free but NOT public
|
||||
domain — do not use. Record a `Provenance` (tools/pipeline/provenance.py) per clip.
|
||||
|
||||
**Run the pipeline** (replaces the placeholder bytes for a scale):
|
||||
|
||||
```bash
|
||||
python -c "from tools.pipeline.run import process_clip; \
|
||||
process_clip('downloads/<scale>.mp4', 'simulator/sample_media/<scale>', \
|
||||
start=<in_s>, duration=<len_s>, overlap=<0.5..1.0>)"
|
||||
```
|
||||
|
||||
This writes `simulator/sample_media/<scale>/base.mp4` (proxy) + `master.mp4`. Then
|
||||
author the labels (Increment 2: the hybrid track pass + simulator author mode) or
|
||||
hand-edit the annotation track for now (spec §4). The ring + manifest entry already
|
||||
exist (Task 8); `process_clip` only replaces the media bytes.
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Update the roadmap**
|
||||
|
||||
In `docs/ROADMAP.md`, under sub-project 3, replace the "Offline v2v variant pipeline"
|
||||
framing with the content-pipeline reality: the Right dream is real-time (no ML bake),
|
||||
so the content pipeline is **source → crossfade-loop → master+proxy transcode →
|
||||
hand-authored/motion-tracked labels → manifest**. Note Increment 1 (this plan: tooling
|
||||
+ track schema + 5-scale ring, no ML) done on merge; Increment 2 (hybrid track pass +
|
||||
simulator author mode) and real i2v ring transitions still pending. Cite the spec
|
||||
`docs/superpowers/specs/2026-06-24-content-pipeline-design.md`.
|
||||
|
||||
- [ ] **Step 3: Run the full suite**
|
||||
|
||||
Run: `pytest -q`
|
||||
Expected: all prior tests still pass + the new pipeline tests (ffmpeg_ops,
|
||||
provenance, manifest) pass; the integration test passes or skips cleanly.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/content-sourcing.md docs/ROADMAP.md
|
||||
git commit -m "docs: content-sourcing procedure + roadmap update (content pipeline Increment 1)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notes for the executor
|
||||
|
||||
- **Run from the main clone, not a nested worktree** — `resolve-app.py` errors
|
||||
"ambiguous" when a `.worktrees/` checkout shadows the main clone (session-0013
|
||||
gotcha). If you must use a worktree, expect the `WGL_APP_SCAN_DEPTH` / explicit
|
||||
`publish-transcript.sh` flag workarounds.
|
||||
- **PRs go through the Gitea API** (`wgl-gitea-admin` `gitea-api.sh`, host
|
||||
`git.benstull.org`) — no `tea`/`gh` here: create via `POST /repos/.../pulls`,
|
||||
merge via `.../merge`.
|
||||
- The project `.venv` has the deps (pytest, cv2/torch for later ML work). Activate it
|
||||
before running tests.
|
||||
- **Increment 2 (separate plan):** `tools/pipeline/track.py` (classical OpenCV
|
||||
tracker + optional lazy-imported ML detect+track) and the simulator **author mode**
|
||||
(draw/seed/correct boxes, assign keys/min_level/strings, place affect, write the
|
||||
manifest). Plus real i2v ring transitions once a generative-video model + adjacent
|
||||
real bases exist.
|
||||
```
|
||||
@@ -0,0 +1,355 @@
|
||||
# HEF — Content Pipeline (source → process → label → manifest)
|
||||
|
||||
**Date:** 2026-06-24
|
||||
**Status:** Graduated — operator-approved (session 0014); **Increment 1 built & merged** (PR #13); real PD sourcing + rotating pool curated (PR #14/#15). **Increment 2 BUILT** (session 0016 — pool model + tiers + windows in PR #16; `track.py` + author mode in PR B; see §11).
|
||||
**Repo:** `human-experience-filter-art`
|
||||
**Anchor:** `docs/ROADMAP.md` sub-project 3 (Player Runtime) content needs; the
|
||||
spiritual successor to sub-project 2 (Ingest & Tagging tools), retargeted from the
|
||||
retired coordinate-catalog model to the manifest content model.
|
||||
**Refines:** [`2026-06-07-scales-library-and-right-axis-pipeline-design.md`](./2026-06-07-scales-library-and-right-axis-pipeline-design.md)
|
||||
§2 (scales library), §2.1 (strict-PD sourcing), §6 (open sourcing questions).
|
||||
**Builds on:** [`2026-06-22-right-axis-deterministic-dream-design.md`](./2026-06-22-right-axis-deterministic-dream-design.md)
|
||||
(the real-time dream — **no per-clip ML bake**) and
|
||||
[`2026-06-22-affect-channel-hud-design.md`](./2026-06-22-affect-channel-hud-design.md)
|
||||
(scene-anchored affect words).
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this exists
|
||||
|
||||
The installation has an alteration engine, a scale ring, a Left analytical HUD, an
|
||||
affect channel, and a real-time Right "dream" — but it has **no real content**. The
|
||||
three ring scales today are: `cosmos` and `abyss` carrying *placeholder bytes* (PD
|
||||
provenance recorded, but the media is synthetic), and `forest` carrying a *POC
|
||||
sample* (a trimmed Yosemite clip licensed "look-tuning only; not shipped content").
|
||||
There is no repeatable way to turn a public-domain source clip into a shippable,
|
||||
seamless-looping, labelled neutral base.
|
||||
|
||||
This design defines that pipeline: **source → frame → seamless-loop → transcode →
|
||||
label/author → manifest entry.** One big anchor fact reshapes it versus the old
|
||||
plan — the Right dream is now a **real-time deterministic shader** (session 0013),
|
||||
so there is **no per-clip ML restyle bake** anymore. The pipeline's offline cost is
|
||||
therefore **transcode + human authoring**, not generative-video baking. The one
|
||||
deliberate exception is the motion-tracking label pass chosen this session (§3),
|
||||
which re-introduces a bounded offline compute step — by choice, for realism.
|
||||
|
||||
## 2. Decisions settled this session
|
||||
|
||||
| # | Decision | Choice | Consequence |
|
||||
|---|----------|--------|-------------|
|
||||
| 1 | Scale set | **Lean 5** — cosmos · orbit · forest/land · reef · abyss (microscopic dropped) | Ring wraps **abyss → cosmos**; replaces the POC forest + both placeholder bases with real strict-PD footage |
|
||||
| 2 | Seamless loop | **Crossfade loop** (tail→head overlap) | Any source clip becomes an indefinite seamless dwell loop; a transcode-time step |
|
||||
| 3 | Label placement | **Motion-tracked, hybrid** | ML proposes tracks where it can; hand-seed + classical tracker elsewhere. Re-introduces an offline pass + an authoring step. Semantics always hand-authored |
|
||||
| 4 | Format | **Master + 1080p proxy** | High-quality master retained per clip; a 1080p H.264 proxy drives the sim; Pi chooses later |
|
||||
|
||||
Microscopic was dropped because PD microscopy *video* (vs stills) is the thinnest
|
||||
pool and hardest to source well; it can be added later by re-running the pipeline.
|
||||
|
||||
## 3. The pipeline — six stages
|
||||
|
||||
Each stage is a deterministic unit with one job (handbook §4.2: deterministic tools,
|
||||
not prose an agent re-derives). Stages 1–4 and 6 are mechanical; stage 5 is the one
|
||||
human-in-the-loop step.
|
||||
|
||||
```
|
||||
(1) source ──▶ (2) frame ──▶ (3) loop ──▶ (4) transcode ──▶ (6) manifest
|
||||
provenance trim/crop crossfade master + proxy entry
|
||||
& scale seam │ ▲
|
||||
└──▶ (5) label/author (hybrid track)
|
||||
```
|
||||
|
||||
### 3.1 Stage 1 — source & provenance
|
||||
Acquire the strict-PD source clip and record license + source URL + capture notes.
|
||||
Reuse sub-project-2 fetchers where they map (a NASA fetcher already exists); add
|
||||
NOAA Ocean Exploration / NPS / USGS fetchers (or accept a manual download + a
|
||||
provenance record). The "no explicit license → assume PD, verify" trap from the
|
||||
scales design §2.1 applies: **"free stock" (Pexels/Pixabay) is NOT public domain.**
|
||||
Output: the raw source file + a provenance record (license, source, url, fetched_at).
|
||||
|
||||
### 3.2 Stage 2 — frame (trim / crop / scale)
|
||||
Pick the in/out points; crop to the target aspect; scale toward the format target
|
||||
(§6). The panoramic projector aspect is still unmeasured (open question), so v1
|
||||
frames to a standard wide 16:9 for the sim; the master retains maximum available
|
||||
width so a later pano crop/extension has pixels to work with. Output: a framed,
|
||||
silent (audio stripped — bases are video-only; audio is the separate content-dial
|
||||
channel) intermediate.
|
||||
|
||||
### 3.3 Stage 3 — seamless crossfade loop
|
||||
Make the clip loop indefinitely with no visible seam by overlapping the tail into
|
||||
the head with a short crossfade (~0.5–1.0 s). The final looped clip is shorter than
|
||||
the source by the overlap; during the overlap region motion is a blend (acceptable
|
||||
for calm ambient nature footage). This is the chosen loop strategy (clean-cut and
|
||||
palindrome were rejected — seam risk and reversed motion respectively). Output: a
|
||||
loop-clean clip whose first and last frames join seamlessly.
|
||||
|
||||
### 3.4 Stage 4 — transcode (master + proxy)
|
||||
Emit two encodes per clip (§6 for exact params): a **master** (high quality, up to
|
||||
4K, retained as the archival/Pi-candidate source) and a **1080p H.264 proxy** (what
|
||||
the sim plays today, matching current `base.mp4` behaviour). Both video-only,
|
||||
`yuv420p`, even dimensions, `+faststart`.
|
||||
|
||||
### 3.5 Stage 5 — label & author (the hybrid motion-track pass)
|
||||
The only human-in-the-loop stage, and the only one that adds offline ML compute.
|
||||
|
||||
- **Semantics are always hand-authored.** A generic detector cannot produce the
|
||||
installation's semantic keys — "spiral galaxy", "siphonophore", `z ≈ 0.04`,
|
||||
`~2.1 m³/s` are out-of-distribution. The author owns the label **keys**,
|
||||
**strings**, and `min_level`; tracking only propagates **box geometry**.
|
||||
- **Hybrid geometry (decision 3).** Where a model can lock onto a subject (a fish
|
||||
on the reef, an animal in the forest), an ML detect+track pass (YOLO+ByteTrack /
|
||||
SAM2-class, lazy-imported like the parked baker imported torch) proposes a track
|
||||
the author maps to a key. Where it can't (cosmos, abyss, microscopic-type
|
||||
content), the author **hand-seeds** a box on a keyframe and a **classical tracker**
|
||||
(OpenCV CSRT / optical-flow) propagates it; the author fixes drift. Both paths
|
||||
emit the same artifact.
|
||||
- **Channel rules.** `detected.*` object labels **track**; `measure.*` readouts and
|
||||
`feel.*` affect words stay **static / scene-anchored** (a depth readout or "awe"
|
||||
is not pinned to a moving pixel). This matches the affect-channel design (affect
|
||||
is scene-level) and keeps measurement chips legible.
|
||||
- Output: an **annotation track** per object label (§4) plus the static measurement
|
||||
and affect placements, all keyed and stringed.
|
||||
|
||||
### 3.6 Stage 6 — manifest assembly
|
||||
Write the clip entry (base/proxy/master paths, license, source, annotations with
|
||||
tracks, affect, strings) into `simulator/sample_media/manifest.json`, and register
|
||||
the scale in the `ring` section with its transition edges (§5). Idempotent: stage 6
|
||||
owns the manifest entry so re-running earlier stages never clobbers authored labels
|
||||
(mirrors how `setup_sample_media.py` was made to never clobber a real bake).
|
||||
|
||||
## 4. Data model — annotation tracks
|
||||
|
||||
Today an annotation is a single static box: `{"key", "box":[x,y,w,h], "min_level"}`.
|
||||
Add an **optional keyframed track**; static labels are unchanged (backward
|
||||
compatible):
|
||||
|
||||
```json
|
||||
{
|
||||
"key": "detected.fish",
|
||||
"min_level": 1,
|
||||
"track": [
|
||||
{"t": 0.0, "box": [0.30, 0.40, 0.10, 0.08]},
|
||||
{"t": 0.5, "box": [0.52, 0.38, 0.10, 0.08]},
|
||||
{"t": 1.0, "box": [0.30, 0.40, 0.10, 0.08]}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- `t` is **normalized 0→1 over the looped clip** — resolution- and fps-independent,
|
||||
so the same track is correct for both master and proxy and survives a re-transcode.
|
||||
(A track that loops cleanly has matching boxes at `t=0` and `t=1`.)
|
||||
- **Runtime:** if `track` is present, the client interpolates the box at the current
|
||||
loop-normalized playback time (linear between keyframes); otherwise it uses the
|
||||
static `box`. This lives **client-side**, consistent with the existing split
|
||||
(alteration math in Python; label selection/placement in the client). `Clip` gains
|
||||
the track only as data inside `annotations`; `player/alteration.py` is **unchanged**
|
||||
(labels are not part of `RenderPlan`).
|
||||
- `measure.*` keep a single static `box`; `feel.*` keep a single static `at:[x,y]`.
|
||||
|
||||
## 5. The 5 scales, sources, and ring
|
||||
|
||||
| Scale | Strict-PD source | Notes |
|
||||
|-------|------------------|-------|
|
||||
| `cosmos` | NASA / Hubble / JWST | 🟢 abundant true PD (17 U.S.C. §105); replaces placeholder |
|
||||
| `orbit` | NASA / ISS (Earth from orbit) | 🟢 true PD; **new** scale |
|
||||
| `forest` | NPS / USGS (US land/wildlife) | 🟢 true PD (US locations); replaces the non-shippable POC sample |
|
||||
| `reef` | NOAA Ocean Exploration (shallow) | 🟢 true PD worldwide; **new** scale |
|
||||
| `abyss` | NOAA Ocean Exploration (deep sea) | 🟢 true PD worldwide; replaces placeholder |
|
||||
|
||||
**Ring order (large → small, zooms inward; wraps small → large):**
|
||||
`cosmos → orbit → forest → reef → abyss → (wrap) cosmos`. Five scales → five ring
|
||||
edges. The wrap seam is now **abyss → cosmos** (deep sea back out to the cosmos) —
|
||||
the infinite-zoom payoff, minus the microscopic step.
|
||||
|
||||
**Transitions** between the new adjacent edges remain the **deferred i2v generative
|
||||
work** (scales design §3 / §6) — they need a generative-video model and two real
|
||||
adjacent bases, heavier than anything in this pipeline. Until then new edges carry
|
||||
**placeholder transitions** (the `setup_scales_media.py` ffmpeg `xfade=zoomin`
|
||||
generator already produces these), exactly as cosmos/forest/abyss do today. This
|
||||
pipeline's job is the **bases + labels**; transitions are tracked but out of scope
|
||||
here (§8).
|
||||
|
||||
## 6. Format target
|
||||
|
||||
| | Master | Proxy (sim) |
|
||||
|---|--------|-------------|
|
||||
| Resolution | source, capped ~3840 wide | 1920×1080 (fit wide) |
|
||||
| Codec | H.264 High | H.264 High |
|
||||
| Quality | CRF ~18 | CRF ~20 |
|
||||
| FPS | preserve source (cap 30) | 30 |
|
||||
| Pixel fmt | `yuv420p`, even dims | `yuv420p`, even dims |
|
||||
| Audio | none (stripped) | none (stripped) |
|
||||
| Flags | `+faststart` | `+faststart` |
|
||||
|
||||
The sim plays the **proxy** as today's `base_file`. The **master** is retained per
|
||||
clip for the eventual Pi/pano encode once the panoramic resolution is measured.
|
||||
Browser HEVC support is too spotty to make HEVC the sim path, which is why the proxy
|
||||
stays H.264 (the master can be re-encoded to HEVC for the Pi later without
|
||||
re-sourcing).
|
||||
|
||||
## 7. Tooling & where it lives
|
||||
|
||||
- **New `tools/pipeline/` package** — deterministic ffmpeg-backed verbs for stages
|
||||
1–4 and 6 (`fetch`/provenance, `frame`, `loop`, `transcode`, `manifest`), reusing
|
||||
sub-project-2 fetchers where valid. This **supersedes** the coordinate-era
|
||||
ingest *drafting* (the `left/right/dark/light` Record model retired with the
|
||||
machine-altered-perception pivot); the pipeline targets the **manifest**, not
|
||||
catalog `Record`s.
|
||||
- **Stage 5 tracking** — `tools/pipeline/track.py`: classical OpenCV tracker
|
||||
(always available) + an optional lazy-imported ML detect+track path. Emits
|
||||
candidate keyframed tracks for the author to accept/correct.
|
||||
- **Authoring surface** — extend the **simulator** with an **author mode** (it
|
||||
already renders the exact HUD overlay and is the by-eye judge): scrub the clip,
|
||||
draw/seed/adjust boxes, run the tracker, assign keys/`min_level`/strings, place
|
||||
affect anchors, write the manifest entry. Reusing the preview as the editor keeps
|
||||
"what you author is what plays." **This is the largest new piece** — see the
|
||||
delivery order (§8), which lets v1 prove the schema before the full editor exists.
|
||||
|
||||
## 8. Delivery / rollout order
|
||||
|
||||
Build order within this Feature (Solution-Design rollout strategy — not a task
|
||||
list):
|
||||
|
||||
1. **Increment 1 — real footage, no ML.** Stages 1–4 + 6 + the §4 track schema +
|
||||
client interpolation. Source the 5 strict-PD bases, crossfade-loop them, emit
|
||||
master+proxy, wire them into the ring. Author a *couple* of tracks **by hand as
|
||||
JSON** to prove the track schema and the runtime interpolation. Outcome: the
|
||||
whole ring is **real, seamless-looping footage** judgeable by eye — the
|
||||
highest-value, lowest-risk step, and it stands alone.
|
||||
2. **Increment 2 — the hybrid track pass + author mode.** `track.py` (classical +
|
||||
optional ML) and the simulator author mode; label all five scales properly.
|
||||
|
||||
**Deferred (tracked, not built here):** the real **i2v ring transitions** for the
|
||||
new edges (need a generative-video model + adjacent real bases); the Pi renderer +
|
||||
serial/firmware (per the simulator-first directive); the panoramic-resolution
|
||||
master encode (waits on a measured pano spec).
|
||||
|
||||
> Scope note: if Increment 2's author mode grows large enough to be its own
|
||||
> increment in practice, split it at execution time. The design is one coherent
|
||||
> pipeline; the increments above are its sensible build order, not separate
|
||||
> Features.
|
||||
|
||||
## 9. Open questions (for the plan, not blockers)
|
||||
|
||||
- **Panoramic resolution/aspect** — unmeasured (scales design §6). Drives the master
|
||||
crop/extension strategy; v1 frames 16:9 for the sim and retains max width.
|
||||
- **ML model choice** for the hybrid track path (YOLO+ByteTrack vs SAM2-class) and
|
||||
whether it earns its place given how few scales it can usefully fire on.
|
||||
- **Classical tracker drift** on long/fast clips — keyframe interval, re-seed cadence.
|
||||
- **Source fetchers** — which of NOAA / NPS / USGS get real fetchers vs manual
|
||||
download + provenance record for v1.
|
||||
- **Crossfade overlap length** per scale (slow cosmos vs busier reef) — by eye.
|
||||
|
||||
## 11. Increment 2 — schema decisions (session 0016)
|
||||
|
||||
Increment 2 is the second build order from §8. It is mostly **schema work**, so
|
||||
this section pins the concrete data shapes before the build (the operator asked for
|
||||
a short design pass first). Each subsection is a new, **backward-compatible**
|
||||
extension — Increment 1 manifests keep rendering unchanged.
|
||||
|
||||
### 11.1 Rotating clip pool (supersedes one-clip-per-scale)
|
||||
|
||||
Each ring scale references a **pool** of vetted clips
|
||||
(`docs/content-candidate-pool.md`); the player picks **one member at random when
|
||||
the viewer lands on that scale** (zooming the ring in/out). This supersedes
|
||||
decision §5's one-base-per-scale.
|
||||
|
||||
- **Manifest:** `ring.scales[i]` gains `"pool": ["clip_id", …]` (≥1). The legacy
|
||||
`"clip_id"` stays as the **primary** member for back-compat (= `pool[0]` when a
|
||||
pool is written); a scale with only `clip_id` and no `pool` behaves exactly as
|
||||
Increment 1 (pool of one).
|
||||
- **Canonical logic (`player/ring.py`):** `Scale` gains `pool: tuple[str, …]`;
|
||||
`members` = `pool or (clip_id,)`. A **pure** `pick_clip_id(scale, r)` takes an
|
||||
injected `r ∈ [0,1)` and returns `members[int(r·len)]` — randomness stays
|
||||
**injected** (DI), so the function is pure/testable and the Pi reuses it.
|
||||
- **Where the pick happens:** at the **API boundary** (`simulator/app.py`), which
|
||||
may be impure: the advance endpoint calls `pick_clip_id(landed_scale,
|
||||
random.random())` and returns it as `target_clip_id`. A `delta=0` advance is the
|
||||
**initial / re-roll** pick (no-op move, fresh random member). The client loads
|
||||
`move.target_clip_id` directly instead of deriving the clip from the index. One
|
||||
selection implementation, Python-canonical (browser still "only renders").
|
||||
- **Rename + cleanup:** ring scale `forest` → **`coast`** (new land↔sea scale
|
||||
between `orbit` and `reef`). Drop the leftovers `cosmos_traverse`,
|
||||
`orbit_westcoast`, old `forest`, old `reef` (per the pool doc's cleanup list);
|
||||
the cosmos pool's primary stays `cosmos` (Orion). New ring order:
|
||||
`cosmos → orbit → coast → reef → abyss → (wrap) cosmos`; edges regenerated
|
||||
(`orbit-coast`, `coast-reef` placeholders via `setup_scales_media.py`).
|
||||
|
||||
### 11.2 Time-windowed tracked labels
|
||||
|
||||
Objects drift **into and out of frame**, so a label appears only **while its
|
||||
object is on screen** and its box **tracks** the object (the §4 keyframed `track`).
|
||||
|
||||
- **Manifest:** an annotation gains optional `"appear"` and `"disappear"`
|
||||
(loop-normalized `t ∈ [0,1]`, same clock as `track`). The label renders only
|
||||
while `appear ≤ loopT ≤ disappear`. To let an object straddle the loop seam, a
|
||||
**`disappear < appear` window wraps** (visible when `loopT ≥ appear` **or**
|
||||
`loopT ≤ disappear`). Absent both fields → always-present (Increment-1 behavior).
|
||||
- **Runtime:** client-side in `renderOverlay` — a window test gates the annotation
|
||||
before the existing `boxAt(loopT)` placement. Engine unchanged (labels are not in
|
||||
`RenderPlan`). The **abyss + reef** pool clips (creatures entering/leaving) are
|
||||
the authored test material.
|
||||
|
||||
### 11.3 Progressive LEFT-brain detail tiers (replaces flat `min_level` gating)
|
||||
|
||||
The Left knob now drives **how many** labels show **and how analytical** each is —
|
||||
no fading. Low Left = **fewer, more general** labels; high Left = **more** labels +
|
||||
**more analytical** detail.
|
||||
|
||||
- **Per-object `salience` (1–4):** how prominent the object is. It sets the Left
|
||||
level at which the label first appears: `first_level = 5 − salience` (salience 4 →
|
||||
appears at Left 1; salience 1 → only at Left 4). So raising Left brings in
|
||||
lower-salience objects.
|
||||
- **Detail tiers (escalate with Left):** the displayed string escalates
|
||||
`tier 1 general ("eel") → 2 specific ("moray eel") → 3 scientific
|
||||
(*Gymnothorax*) → 4 scientific + a fact (lifespan)`. The tier shown =
|
||||
`clamp(left − first_level + 1, 1, n_tiers)` — an object shows tier 1 the moment
|
||||
it appears and climbs one tier per further Left notch until its tiers run out.
|
||||
- **Strings carry the tiers:** `strings.en[key]` may be a **list** (indexed by
|
||||
`tier−1`) instead of a string; a plain string stays static (back-compat, tier
|
||||
ignored). Keeps one key per object and the i18n table as the single source.
|
||||
- **Back-compat:** an annotation with `min_level` and no `salience` keeps flat
|
||||
gating (`first_level = min_level`, single-tier string). `measure.*` readouts stay
|
||||
single-tier, gated by their level (a measurement is not a tiered object).
|
||||
- **No engine change:** tier selection is client-side against the existing
|
||||
`overlay.level` (the Left knob) already in the `RenderPlan`.
|
||||
|
||||
### 11.4 Progressive RIGHT-brain emotion tiers
|
||||
|
||||
The affect **vocabulary** gets more sophisticated as Right rises — basic words at
|
||||
Right 1, nuanced/compound feelings at high Right — mirroring §11.3 for affect.
|
||||
|
||||
- **Appearance rule unchanged:** affect surfaces only when **both** knobs are up;
|
||||
an anchor's `min_level` is tested against `strength = min(left, right)` as today.
|
||||
- **Emotion tiers:** `strings.en[feel.key]` may be a **list**; the word shown =
|
||||
`clamp(right, 1, n_tiers)` (the Right knob, read from `dream.strength` already in
|
||||
the `RenderPlan`). A plain string stays static (back-compat).
|
||||
- **No engine change:** `AffectOverlay` already carries `strength`; the Right level
|
||||
comes from `dream.strength`. Tier selection is client-side.
|
||||
|
||||
### 11.5 `tools/pipeline/track.py` (hybrid) + simulator author mode
|
||||
|
||||
The stage-5 tooling from §3.5 / §7, delivered as a **second PR** after the
|
||||
experience PR (§8's split note).
|
||||
|
||||
- **`track.py`:** a **classical** path (OpenCV CSRT / optical-flow) seeded by a
|
||||
hand-drawn box on a keyframe, propagated and **sampled to a sparse keyframed
|
||||
`track`** (normalized `{t, box}` over the loop, §4); plus an **optional,
|
||||
lazy-imported ML** detect+track path (YOLO+ByteTrack-class) that proposes tracks
|
||||
the author maps to keys. Both emit the same `track` artifact + an `appear`/
|
||||
`disappear` window. Pure helpers (normalization, keyframe sampling,
|
||||
track→manifest) are unit-tested; an **opt-in** integration test runs the real
|
||||
tracker on a pool clip (mirrors the Increment-1 real-footage test).
|
||||
- **Author mode (simulator):** reuse the preview UI as the editor — scrub a pool
|
||||
clip, draw/seed/correct boxes, run the tracker, assign `key` / `salience` /
|
||||
tiered strings / `appear`-`disappear` / affect anchors, and **write the manifest
|
||||
entry** (an endpoint upserts via `tools/pipeline/manifest.upsert_clip` and
|
||||
persists). Semantics (keys, strings, scientific names, facts) are **always
|
||||
hand-authored** — the tracker only propagates **box geometry** (§3.5).
|
||||
|
||||
## 12. Out of scope (YAGNI)
|
||||
|
||||
- Generative i2v **ring transitions** for the new edges (deferred; placeholders hold).
|
||||
- **Audio** sourcing/alteration (separate content-dial channel; already deferred).
|
||||
- The **Pi/pano** encode + renderer and serial/firmware (simulator-first).
|
||||
- Reviving the retired coordinate-catalog `Record` drafting.
|
||||
- Per-frame **dense** tracks (sparse keyframed tracks interpolated at runtime instead).
|
||||
@@ -0,0 +1,64 @@
|
||||
# Simulator Dev Mode — design
|
||||
|
||||
> 2026-06-25 · session 0017 · planning-and-executing
|
||||
> Anchor: leaf task (simulator dev-tooling — R2b, no upstream design needed)
|
||||
|
||||
## Problem
|
||||
|
||||
While previewing the experience, the operator can't choose *which* clip in an
|
||||
altitude's rotating pool is shown — the server picks a random pool member on each
|
||||
landing (content-pipeline §11.1). For by-eye review they want to step through every
|
||||
clip in a pool on demand, and to see the engine/clip data behind what's on screen.
|
||||
The RenderPlan JSON readout is also always visible, cluttering what should read as
|
||||
an audience-facing preview.
|
||||
|
||||
## Goal
|
||||
|
||||
A single **Dev Mode** toggle at the bottom of the control panel. Off by default
|
||||
(persisted in `localStorage`). When on, a panel appears directly below it with all
|
||||
dev controls and data; when off, the normal view is just the experience + the three
|
||||
control fieldsets.
|
||||
|
||||
## Scope
|
||||
|
||||
Frontend only — `simulator/static/{index.html,app.js,style.css}`. No API changes:
|
||||
every datum Dev Mode shows is already available client-side (`/api/clips`,
|
||||
`/api/ring`, the `/api/alteration` response, and the in-memory preload cache). The
|
||||
separate `/author.html` remains the place to *edit* labels; Dev Mode is
|
||||
inspect-and-select only.
|
||||
|
||||
## Dev panel contents (top → bottom)
|
||||
|
||||
1. **Pool picker** — a `<select>` of the current altitude's pool members
|
||||
(`clip_id · title`), its value reflecting the clip currently playing. Choosing a
|
||||
member sets `activeClipId` to it and reloads the base media, overriding the random
|
||||
landing pick. A `🎲 Re-roll random` button re-runs the server-canonical random
|
||||
pick for the current scale. The dropdown is rebuilt on every scale landing, so it
|
||||
always lists the current pool; single-clip scales (cosmos/orbit) show one entry.
|
||||
2. **Active clip** — `id`, `title`, `source`, `license`, `base_file`.
|
||||
3. **Annotations & affect** — each annotation key with salience (or legacy
|
||||
`min_level`), time window (`appear`–`disappear`), and tier count; each affect key
|
||||
with `min_level` and tier count. Read-only.
|
||||
4. **RenderPlan** — the relocated `#readout` (the `/api/alteration` JSON).
|
||||
5. **Cache & playback** — preload status (`N/total cached`), whether the active
|
||||
clip is served **from memory** (a preloaded blob) vs the network, and live video
|
||||
resolution / duration / loop position.
|
||||
|
||||
## Behavior
|
||||
|
||||
- **Pool override is per-landing.** Changing altitude re-rolls randomly as today;
|
||||
the operator then picks again from the new scale's pool. The pick is not persisted
|
||||
across landings (YAGNI — "select any video while on an altitude").
|
||||
- **Refresh points.** Pool picker + clip metadata + annotations rebuild whenever the
|
||||
scale lands (hooked into `renderScaleReadout()`) and after a manual pick/re-roll.
|
||||
The RenderPlan readout updates on each `update()` (knob change) as today. The
|
||||
cache/playback stats refresh on a light interval (~500 ms) while Dev Mode is on,
|
||||
and the cache count also ticks up as the preloader fills.
|
||||
- **`pickRandomMember()`** is factored out of `landScale()` so the initial landing,
|
||||
ring advances, and the Re-roll button all share the one server-canonical pick.
|
||||
|
||||
## Testing
|
||||
|
||||
No JS unit harness in the repo and no API change, so verification is: `node --check`
|
||||
on `app.js`, the Python suite stays green (unaffected), and by-eye review by the
|
||||
operator (no headless browser yet — §9 E2E policy noted, machinery pending).
|
||||
+30
-1
@@ -50,10 +50,25 @@ class RingError(ValueError):
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Scale:
|
||||
"""A node on the ring: a scale of nature and the base clip it plays."""
|
||||
"""A node on the ring: a scale of nature and the clip(s) it can play.
|
||||
|
||||
A scale carries a rotating POOL of vetted clips (content-pipeline design
|
||||
§11.1): when the viewer lands on the scale, the player picks one member at
|
||||
random (`pick_clip_id`). `clip_id` is the PRIMARY member (the deterministic
|
||||
default / pool[0]); `pool` is the full set. A scale written with only
|
||||
`clip_id` and no `pool` is a pool of one — exactly the Increment-1 behavior,
|
||||
so old manifests are unchanged.
|
||||
"""
|
||||
|
||||
id: str
|
||||
clip_id: str
|
||||
pool: tuple[str, ...] = ()
|
||||
|
||||
@property
|
||||
def members(self) -> tuple[str, ...]:
|
||||
"""The clip ids this scale can play — the pool, or just `clip_id` when
|
||||
no pool was authored (a pool of one)."""
|
||||
return self.pool if self.pool else (self.clip_id,)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -134,6 +149,20 @@ def scale_at(ring: ScaleRing, index: int) -> Scale:
|
||||
return ring.scales[index % len(ring)]
|
||||
|
||||
|
||||
def pick_clip_id(scale: Scale, r: float) -> str:
|
||||
"""Pick one clip id from a scale's rotating pool, given an injected uniform
|
||||
`r` in [0, 1) (content-pipeline design §11.1).
|
||||
|
||||
Randomness is INJECTED so this stays a pure function — testable and shared
|
||||
with the Pi player; the impure draw (`random.random()`) happens once at the
|
||||
API/runtime boundary. `r` is clamped into [0, 1) so an off-by-epsilon caller
|
||||
never indexes out of range; a pool of one always returns its single member.
|
||||
"""
|
||||
members = scale.members
|
||||
r = min(max(r, 0.0), 0.999999)
|
||||
return members[int(r * len(members))]
|
||||
|
||||
|
||||
def advance_ring(
|
||||
ring: ScaleRing,
|
||||
from_index: int,
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
# Session 0014.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-24T06-58 (PST)
|
||||
> End: 2026-06-24T08-29 (PST)
|
||||
> Type: brainstorming
|
||||
> Posture: careful
|
||||
> Claude-Session: 50301aba-c534-4a7c-9f33-8b0881920ab2
|
||||
> Anchor: docs/ROADMAP.md sub-project 3 (Player Runtime) — content pipeline
|
||||
> Status: **FINALIZED**
|
||||
|
||||
## Launch prompt
|
||||
|
||||
```
|
||||
Launch: /goal next → resolved from memory to the recorded Next /goal:
|
||||
START A NEW SESSION for the content pipeline — sourcing full neutral video clips + processing + labeling. Brainstorm/scope it into a content-pipeline spec FIRST, deciding: (1) sourcing (strict-PD clips, which scales); (2) clip length + seamless-loop strategy; (3) labeling approach (static vs motion-tracked; hand-authored vs detection-assisted; same for affect); (4) format target (res/fps/codec for sim now + Pi later). Anchor fact: Right dream is real-time, no per-clip bake → pipeline is transcode + authoring, NOT ML baking.
|
||||
```
|
||||
|
||||
## Plan
|
||||
|
||||
> Anchor: docs/ROADMAP.md sub-project 3 content pipeline. Roadmap-anchored repo
|
||||
> (no Gitea issue tracker used 0001–0013); content pipeline is one cohesive
|
||||
> Feature → design-eligible. See Deferred decisions for the R1 deviation.
|
||||
|
||||
**Goal:** Explore the content pipeline (source full neutral PD clips → process →
|
||||
label/author) and write its Solution-Design spec. Anchor fact: the Right dream is
|
||||
real-time (session 0013), so the pipeline is **transcode + human authoring**, not
|
||||
ML baking.
|
||||
|
||||
## What happened (arc)
|
||||
|
||||
This opened as `/goal next`; the session gate resolved the stored `Next /goal:`
|
||||
from memory (content pipeline) and routed to **brainstorming** (the goal said
|
||||
"brainstorm/scope it into a spec first"). Init claimed session **0014** (careful
|
||||
posture, the brainstorming default), no concurrent sessions, clean `main`.
|
||||
|
||||
**Orientation.** Read the roadmap (sub-project 3), the scales-library/right-axis
|
||||
design (§1/§3/§4), the two session-0013 specs (real-time dream supersedes the SD
|
||||
bake; affect channel), and the current content shape (`simulator/clips.py`,
|
||||
`manifest.json`, `setup_scales_media.py`). Confirmed the key simplifier: the Right
|
||||
dream is real-time, so the pipeline is transcode + authoring, **not** ML baking.
|
||||
|
||||
**Brainstorming (careful, agree-up-front → draft-whole → review-whole).** Walked
|
||||
the four launch-prompt decisions as `AskUserQuestion` checkpoints:
|
||||
1. Scale set → **lean 5** (cosmos · orbit · forest/land · reef · abyss; microscopic
|
||||
dropped — thinnest PD-video pool).
|
||||
2. Loop → **crossfade loop** (tail→head overlap).
|
||||
3. Labels → **motion-tracked, HYBRID** (ML proposes where it can; hand-seed +
|
||||
classical tracker elsewhere). Surfaced that label *semantics* must always be
|
||||
hand-authored (a generic detector can't produce "siphonophore"/"z ≈ 0.04");
|
||||
tracking only propagates box geometry.
|
||||
4. Format → **master + 1080p H.264 proxy**.
|
||||
|
||||
Drafted the full spec `docs/superpowers/specs/2026-06-24-content-pipeline-design.md`,
|
||||
pushed to `feature/content-pipeline`, handed over the rendered Gitea URL for
|
||||
whole-document review. Operator chose **"Approve + plan now"**.
|
||||
|
||||
**Planning.** Wrote `docs/superpowers/plans/2026-06-24-content-pipeline-increment-1.md`
|
||||
(Increment 1 = real footage, no ML): 9 bite-sized TDD tasks. Self-reviewed against
|
||||
the spec (coverage, types, one dead-variable fix). Operator chose **"Execute now
|
||||
(subagent-driven)"**.
|
||||
|
||||
**Execution (subagent-driven, 2-stage review per group).** Implementer + reviewer
|
||||
subagents (sonnet), grouped by file coupling:
|
||||
- Tasks 1–3 (`tools/pipeline/ffmpeg_ops.py` pure builders) — DONE; review added an
|
||||
`overlap<=0` test.
|
||||
- Tasks 4–5 (`provenance.py`, `manifest.py`) — DONE; review added append-fallback +
|
||||
N=1 self-wrap tests.
|
||||
- Task 6 (`run.py` runner + integration test) — the test gated on `shutil.which`
|
||||
and SKIPPED (ffmpeg not on PATH). Increment 1's goal is to *prove the pipeline on
|
||||
real footage*, so I hardened the runner to resolve the **bundled imageio-ffmpeg**
|
||||
(matching `setup_scales_media.py`) and switched duration probing to **cv2** (no
|
||||
ffprobe dep — Pi-friendly). Re-run **actually exercised real ffmpeg on the real
|
||||
forest base and surfaced a genuine bug**: `trim` yields `1/0` framerate which
|
||||
`xfade` rejects (needs CFR) — fixed with `fps=30` after each trim segment. Test
|
||||
now PASSES (proxy 1920×1080 verified via cv2). Closed an ffprobe-gap minor too.
|
||||
- Task 7 (client annotation-track interpolation, `app.js`) — DONE; `node --check`
|
||||
clean, API serves the track. Visual drift deferred to operator by-eye.
|
||||
- Task 8 (ring 3→5: orbit + reef placeholders) — DONE; `/api/ring` shows 5 scales
|
||||
+ 5 edges. Ring spin deferred to operator by-eye.
|
||||
- Task 9 (sourcing doc + ROADMAP + stale-docstring fix) — DONE.
|
||||
|
||||
Final whole-branch review: **READY TO MERGE** (no critical/blocking). 246 passed /
|
||||
2 skipped. Operator chose **"Merge to main now"** → pushed, created **PR #13** via
|
||||
the Gitea API, merged (delete branch), synced main, deleted local branch. Stamped
|
||||
the spec **graduated** on main.
|
||||
|
||||
## Outcome
|
||||
|
||||
- Content-pipeline **spec** (graduated) + **Increment-1 plan** (archived in-repo)
|
||||
+ **Increment 1 implementation** all merged to `main` (PR #13, merge `bb5ecf0`;
|
||||
spec stamp `73e5c4a`). Suite **246 passed / 2 skipped**.
|
||||
- New `tools/pipeline/` package; keyframed annotation-`track` schema
|
||||
(client-interpolated, backward compatible); scale ring extended 3→5.
|
||||
- `docs/content-sourcing.md` (strict-PD sourcing procedure); ROADMAP updated.
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
- **R1 anchor (no Gitea issue):** the brainstorming gate wants a non-`epic`
|
||||
`type/*` tracker issue as the design anchor. This repo has never used a Gitea
|
||||
issue tracker (roadmap-driven, 0001–0013), so the spec is anchored to
|
||||
`docs/ROADMAP.md` sub-project 3 — matching every prior brainstorming session
|
||||
here (0005/0007/0008/0013). Flagged rather than force-filing an issue that would
|
||||
break established repo practice; operator can request issue-tracking if wanted.
|
||||
- **Session-type blur (operator-directed):** a brainstorming session normally ends
|
||||
at the spec; the operator explicitly extended it through plan → execute → merge.
|
||||
Each transition was an explicit operator choice via `AskUserQuestion`, so the
|
||||
careful-posture gates were honored, not bypassed.
|
||||
- **Crossfade duration source (minor, non-blocking):** `process_clip` probes the
|
||||
duration from the *source*, not the post-frame mezzanine; GOP alignment can make
|
||||
the tail segment ±~33 ms off. ffmpeg handles EOF gracefully; imperceptible on
|
||||
calm footage. Noted for a future fast-motion clip.
|
||||
|
||||
## Pending for next session
|
||||
|
||||
- **Operator by-eye review of Increment 1** in the sim: the track drift
|
||||
(water/fish boxes follow playback + loop seamlessly) and spinning the 5-scale
|
||||
ring cosmos→orbit→forest→reef→abyss→wrap. A dev sim (session-0013 live-reload)
|
||||
was running on :8000 during the session.
|
||||
|
||||
## Next /goal
|
||||
|
||||
```
|
||||
/goal content-pipeline Increment 2 — source real strict-PD bases per docs/content-sourcing.md (run each through tools/pipeline/run.py process_clip) + build the hybrid motion-track pass tools/pipeline/track.py + a simulator author mode; FIRST do the pending by-eye review of Increment 1. Per docs/superpowers/specs/2026-06-24-content-pipeline-design.md
|
||||
```
|
||||
|
||||
> Still-deferred (tracked in `docs/ROADMAP.md`, this repo's queue): real i2v ring
|
||||
> transitions (need a generative-video model + adjacent real bases); Pi renderer +
|
||||
> serial/firmware (simulator-first).
|
||||
@@ -1,26 +0,0 @@
|
||||
# Session 0014.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-24T06-58 (PST)
|
||||
> Type: brainstorming
|
||||
> Posture: careful
|
||||
> Claude-Session: 50301aba-c534-4a7c-9f33-8b0881920ab2
|
||||
> Status: **PLACEHOLDER — claimed at session start; finalized at session end.**
|
||||
>
|
||||
> This file reserves session ID 0014 for human-experience-filter-art. The driver replaces this
|
||||
> body with the full transcript and renames the file to its final
|
||||
> SESSION-0014.0-TRANSCRIPT-2026-06-24T06-58--<end>.md form at session end.
|
||||
|
||||
## Launch prompt
|
||||
|
||||
```
|
||||
Launch: /goal next → resolved from memory to the recorded Next /goal:
|
||||
START A NEW SESSION for the content pipeline — sourcing full neutral video clips + processing + labeling. Brainstorm/scope it into a content-pipeline spec FIRST, deciding: (1) sourcing (strict-PD clips, which scales); (2) clip length + seamless-loop strategy; (3) labeling approach (static vs motion-tracked; hand-authored vs detection-assisted; same for affect); (4) format target (res/fps/codec for sim now + Pi later). Anchor fact: Right dream is real-time, no per-clip bake → pipeline is transcode + authoring, NOT ML baking.
|
||||
|
||||
```
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
_Autonomous-mode low-confidence calls the driver made and would have
|
||||
liked operator input on. Appended as the session runs; surfaced at
|
||||
finalize. Empty if none._
|
||||
@@ -0,0 +1,85 @@
|
||||
# Session 0015.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-24T13-16 (PST)
|
||||
> Type: planning-and-executing
|
||||
> Posture: yolo
|
||||
> Claude-Session: 50301aba-c534-4a7c-9f33-8b0881920ab2
|
||||
> Status: **PLACEHOLDER — claimed at session start; finalized at session end.**
|
||||
>
|
||||
> This file reserves session ID 0015 for human-experience-filter-art. The driver replaces this
|
||||
> body with the full transcript and renames the file to its final
|
||||
> SESSION-0015.0-TRANSCRIPT-2026-06-24T13-16--<end>.md form at session end.
|
||||
|
||||
## Launch prompt
|
||||
|
||||
```
|
||||
Process 5 candidate ORBIT (Earth-from-space) clips for the human-experience-filter art installation: resolve URLs, download, vet for text + humans, trim to a word-free calm window, transcode. Clips: orbit_planetearth, orbit_crewobs, orbit_aurora_perth (SVS 31281), orbit_westcoast (SVS 30180), orbit_bluemarble (SVS 4550).
|
||||
|
||||
```
|
||||
|
||||
## Plan
|
||||
|
||||
Process 5 candidate ORBIT (Earth-from-space) clips: resolve URLs, download, vet for text/telemetry/humans via contact sheets, trim to word-free 24s window, transcode via process_clip.
|
||||
|
||||
Tasks:
|
||||
1. Resolve direct mp4 URLs for SVS pages (aurora_perth/31281, westcoast/30180, bluemarble/4550)
|
||||
2. Download all 5 clips
|
||||
3. Build contact sheets (overview + first-8s + last-8s) for each clip
|
||||
4. Vet each clip for text overlays, telemetry, ISS hardware, humans
|
||||
5. Determine clean window or mark UNUSABLE
|
||||
6. process_clip → simulator/sample_media/<id>/ for usable clips
|
||||
7. Generate preview frames
|
||||
|
||||
## Clip results
|
||||
|
||||
### orbit_planetearth
|
||||
- URL: https://svs.gsfc.nasa.gov/vis/a030000/a030700/a030771/ISS_View_of_Planet_Earth_2160p.mp4
|
||||
- Duration: 97.2s
|
||||
- Text/telemetry: "INTERNATIONAL SPACE STATION" title card t=0–8s; ISS logo end card t≈80–97s
|
||||
- Humans: none
|
||||
- Window: 10–34s (24s, clean daytime Earth clouds)
|
||||
- Status: USABLE
|
||||
- Output: simulator/sample_media/orbit_planetearth/
|
||||
- Preview: /tmp/hef-cand/orbit_planetearth/preview.png
|
||||
|
||||
### orbit_crewobs
|
||||
- URL: https://svs.gsfc.nasa.gov/vis/a030000/a030700/a030771/ISS_Crew_Earth_Observations_2160p.mp4
|
||||
- Duration: 66.4s
|
||||
- Text/telemetry: "INTERNATIONAL SPACE STATION" + date/location title card t=0–8s; ISS logo end card t≈55–66s
|
||||
- Humans: none
|
||||
- Window: 9–33s (24s, clean desert/cloud Earth)
|
||||
- Status: USABLE
|
||||
- Output: simulator/sample_media/orbit_crewobs/
|
||||
- Preview: /tmp/hef-cand/orbit_crewobs/preview.png
|
||||
|
||||
### orbit_aurora_perth
|
||||
- URL: https://svs.gsfc.nasa.gov/vis/a030000/a031200/a031281/ISS067_20220817_aurora_1080p25.mp4
|
||||
- Duration: 49.6s
|
||||
- Text/telemetry: Date+timestamp burned in top-left throughout (e.g. "2022-08-17 19:15:25", updating every second)
|
||||
- Humans: none
|
||||
- Window: none (telemetry throughout)
|
||||
- Status: UNUSABLE
|
||||
|
||||
### orbit_westcoast
|
||||
- URL: https://svs.gsfc.nasa.gov/vis/a030000/a030100/a030180/iss028_nighttime_20110819_720p.mp4
|
||||
- Duration: 54.7s
|
||||
- Text/telemetry: none
|
||||
- Humans: none
|
||||
- ISS hardware: cylindrical ISS module + solar panel grid visible in top-right corner throughout entire clip (confirmed by exhaustive per-second crop inspection t=14–44s)
|
||||
- Window: none (ISS hardware throughout)
|
||||
- Status: UNUSABLE — orbit_westcoast output dir deleted
|
||||
|
||||
### orbit_bluemarble
|
||||
- URL: https://svs.gsfc.nasa.gov/vis/a000000/a004500/a004550/BlueMarble_starfield_4k_2160p30_v2.mp4
|
||||
- Duration: 121.0s
|
||||
- Text/telemetry: none (CG visualization, no overlays anywhere)
|
||||
- Humans: none
|
||||
- Window: 10–34s (24s, rotating Earth globe on starfield)
|
||||
- Status: USABLE
|
||||
- Output: simulator/sample_media/orbit_bluemarble/
|
||||
- Preview: /tmp/hef-cand/orbit_bluemarble/preview.png
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
- orbit_westcoast output directory (simulator/sample_media/orbit_westcoast/) was written before the ISS hardware was confirmed throughout. It needs to be deleted. The rm -rf was attempted but permission was denied in the sandbox — operator should delete it manually: `rm -rf simulator/sample_media/orbit_westcoast`
|
||||
@@ -0,0 +1,117 @@
|
||||
# Session 0016.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-24T17-48 (PST)
|
||||
> Type: planning-and-executing
|
||||
> Posture: yolo
|
||||
> Claude-Session: 0fde1d36-7c64-414f-8f49-8669b5dd2263
|
||||
> Status: **FINALIZED**
|
||||
|
||||
## Launch prompt
|
||||
|
||||
`/goal` Content pipeline Increment 2 (expanded) — rotating clip pools + time-windowed
|
||||
TRACKED labels + progressive left-brain detail + progressive right-brain emotions.
|
||||
Short design pass first (schema changes), then build.
|
||||
|
||||
> Anchor: design `docs/superpowers/specs/2026-06-24-content-pipeline-design.md`
|
||||
> (graduated, R2a-eligible). Repo is roadmap-anchored, no Gitea issue tracker (R1
|
||||
> deviation, as all prior sessions). Increment 2 of that design.
|
||||
|
||||
## Pre-state
|
||||
|
||||
`main` at `b28f930` (PR #15 — operator-curated rotating-pool doc merged). All 19
|
||||
vetted pool clips on disk (gitignored) plus the leftovers to drop. The ring was the
|
||||
Increment-1 one-base-per-scale model (cosmos/orbit/forest/reef/abyss), flat
|
||||
`min_level` label gating, static or simple-track boxes, scene-level affect gated by
|
||||
`min(left,right)`. `tools/pipeline/` had stages 1–4 + 6 (no `track.py`). 246 tests.
|
||||
|
||||
## Arc
|
||||
|
||||
1. **Session gate + init.** Classified the `/goal` as planning-and-executing,
|
||||
claimed session **0016** (peeked an ENDED-UNFINALIZED 0015 — its PR #15 work was
|
||||
already merged, not live — and proceeded; no worktree needed, solo session).
|
||||
Verified clean `main`, read the pool doc + spec + sub-project-3 memory + the live
|
||||
code (ring, clips, app, app.js, alteration, pipeline).
|
||||
2. **Design pass (operator asked for it — schema changes).** Appended **§11** to the
|
||||
content-pipeline spec pinning the concrete shapes: rotating pool, time-windowed
|
||||
tracks, progressive Left detail tiers (salience + tiered strings), progressive
|
||||
Right emotion tiers, and the `track.py` + author-mode tooling. Key call: tiers
|
||||
need **no engine change** — the client reads `overlay.level` (Left) and
|
||||
`dream.strength` (Right) already in the `RenderPlan`.
|
||||
3. **PR #16 — the experience.** `Scale.pool` + pure `pick_clip_id` (injected
|
||||
randomness); server-side random pick at the API boundary (delta=0 = initial pick).
|
||||
`forest`→`coast` rename + ring reorder + orphan-media cleanup. Time-windowed
|
||||
tracked labels (`appear`/`disappear`, seam-wrapping). Salience-gated +
|
||||
tier-escalating Left labels; tier-escalating Right emotions; tiered-or-static
|
||||
strings (back-compat). New `build_pool_manifest.py` emits the 19-clip pool
|
||||
manifest (abyss+reef authored as tracked/windowed showcase). Merged via Gitea API.
|
||||
4. **PR #17 — the tooling.** `tools/pipeline/track.py`: classical LK optical-flow box
|
||||
tracker (this opencv build has no CSRT) → sparse keyframed track + window; optional
|
||||
lazy ML path. Simulator AUTHOR MODE (`/author.html` + `/api/author/{track,clip}`):
|
||||
scrub, drag a seed box, run tracker, author tiers/affect, save to manifest
|
||||
(idempotent upsert, keeps provenance). Repointed the pipeline integration test off
|
||||
the retired forest base; USER_GUIDE sim section brought current. Merged.
|
||||
|
||||
## Cut state
|
||||
|
||||
`main` at `c445239` (PR #17 merge). **267 passed / 2 skipped.** Clean tree, fully
|
||||
pushed, no open PRs. Both feature branches deleted. All 5 goal items delivered.
|
||||
|
||||
Verification was API/logic-level + JS syntax + live-server probes (5 scales with
|
||||
pools, rotating picks, new edges, 19 clips served, tracker returns keyframes on real
|
||||
abyss_wow footage, committed manifest untouched by the read-only probe). **The
|
||||
VISUAL look was not screenshot-verified — no Chrome on this box** — so operator
|
||||
by-eye review is the next step (matches the established deferred-review pattern).
|
||||
|
||||
## Follow-up (post-finalize, same conversation) — PR #18
|
||||
|
||||
Operator asked to replace the scale-ring `⊖ out / in ⊕` buttons with a **circular
|
||||
"Altitude" knob** that turns all the way around (including abyss→cosmos). Built a
|
||||
turnable SVG dial in `simulator/static/app.js` (`buildDial` / `renderDial` / drag +
|
||||
wheel handlers) + `style.css`: cosmos (highest) at top, clockwise descends
|
||||
cosmos→orbit→coast→reef→abyss and **wraps** back up. Dragging accumulates rotation
|
||||
and commits whole detents on release (a big spin → one fast blended pass, reusing
|
||||
the proven wheel/detent path); scroll the knob to step; tap a label to jump the
|
||||
shortest way around. **Pure frontend** — the endless ring + wrap already existed
|
||||
server-side, so no engine/API change. JS syntax-checked; 23 sim-API tests green;
|
||||
live probe confirmed the dial replaced the buttons and the abyss→cosmos wrap is
|
||||
intact. Merged as **PR #18** (`main` at `7a32ff1`). Visual look still operator-by-eye
|
||||
(no Chrome on this box).
|
||||
|
||||
## Deployment pipeline (§9)
|
||||
|
||||
No stage run: the HEF simulator is a **local dev tool** (simulator-first directive),
|
||||
not a deployed app — it has no PPE/prod infra. Nothing shippable through the §9
|
||||
pipeline this session; the session output is the merged code on `main`.
|
||||
|
||||
## Plan archival
|
||||
|
||||
No separate `superpowers:writing-plans` artifact was produced — the design pass was a
|
||||
spec §11 edit (merged in PR #16) and execution was direct TDD tracked by the
|
||||
transcript `## Plan` block + the task list. Nothing to `submit-plan.sh`; this app has
|
||||
no content repo anyway (roadmap-anchored).
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
- **Adopting/finalizing the ENDED-UNFINALIZED session 0015 placeholder** was skipped
|
||||
— its PR #15 work is already on `main` and its outcomes are fully recorded in
|
||||
memory; finalizing a prior dead session was judged an unnecessary detour. Low
|
||||
confidence; flagging for the operator.
|
||||
- **By-eye visual review deferred** to the operator (no Chrome here) — consistent with
|
||||
the prior-session pattern, but the look of the tiers/windows/pool rotation AND the
|
||||
new Altitude knob is unconfirmed visually.
|
||||
- **Light-vs-rich label authoring:** only abyss + reef + the scale primaries are
|
||||
richly authored; the other pool members carry lighter labels. Finishing them is the
|
||||
author-mode follow-up (next goal).
|
||||
|
||||
## Next-session prompt
|
||||
|
||||
`/goal` Operator by-eye review of Increment 2 + the Altitude knob in the sim
|
||||
(`make sim-local`, :8000): turn the Altitude knob (drag/scroll, wraps abyss→cosmos;
|
||||
pools rotate per landing), sweep Left (progressive label tiers
|
||||
general→scientific+fact), sweep Right with Left up (progressive emotions
|
||||
basic→compound), watch abyss/reef time-windowed tracked labels enter/leave + track,
|
||||
and exercise `/author.html`. Then author real labels for the remaining pool members
|
||||
via author mode. After the content is liked, the next build frontier is the deferred
|
||||
i2v ring transitions (generative-video model + adjacent real bases) and/or the Pi
|
||||
renderer + serial/firmware (per [[simulator-first-before-hardware]]), per ROADMAP.md.
|
||||
@@ -0,0 +1,22 @@
|
||||
# Session 0017.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-25T07-41 (PST)
|
||||
> Type: planning-and-executing
|
||||
> Posture: yolo
|
||||
> Claude-Session: bcb069f2-0fba-4a66-ae8e-f40b7917a48c
|
||||
> Status: **PLACEHOLDER — claimed at session start; finalized at session end.**
|
||||
>
|
||||
> This file reserves session ID 0017 for human-experience-filter-art. The driver replaces this
|
||||
> body with the full transcript and renames the file to its final
|
||||
> SESSION-0017.0-TRANSCRIPT-2026-06-25T07-41--<end>.md form at session end.
|
||||
|
||||
## Launch prompt
|
||||
|
||||
_(launch prompt not captured at claim time)_
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
_Autonomous-mode low-confidence calls the driver made and would have
|
||||
liked operator input on. Appended as the session runs; surfaced at
|
||||
finalize. Empty if none._
|
||||
@@ -40,5 +40,14 @@
|
||||
},
|
||||
"0014": {
|
||||
"title": ""
|
||||
},
|
||||
"0015": {
|
||||
"title": ""
|
||||
},
|
||||
"0016": {
|
||||
"title": ""
|
||||
},
|
||||
"0017": {
|
||||
"title": ""
|
||||
}
|
||||
}
|
||||
|
||||
+99
-7
@@ -8,7 +8,9 @@ endpoints (/api/select, /api/catalog/meta) and the X-ray are retired.
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
@@ -26,7 +28,12 @@ from player.alteration import (
|
||||
)
|
||||
from player.content import resolve_content
|
||||
from player.controls import CONTENT_POSITIONS
|
||||
from player.ring import DEFAULT_FAST_SPIN_THRESHOLD, advance_ring
|
||||
from player.ring import (
|
||||
DEFAULT_FAST_SPIN_THRESHOLD,
|
||||
advance_ring,
|
||||
pick_clip_id,
|
||||
scale_at,
|
||||
)
|
||||
from simulator.clips import load_manifest, load_ring, ring_move_to_dict, ring_to_dict
|
||||
|
||||
STATIC_DIR = Path(__file__).parent / "static"
|
||||
@@ -78,6 +85,29 @@ class RingAdvanceRequest(BaseModel):
|
||||
delta: int
|
||||
|
||||
|
||||
class AuthorTrackRequest(BaseModel):
|
||||
"""Author mode: propagate a hand-seeded box on a clip into a keyframed track
|
||||
(content-pipeline §11.5). `seed_box` + `seed_t` are normalized."""
|
||||
|
||||
clip_id: str
|
||||
key: str
|
||||
seed_box: list[float] = Field(min_length=4, max_length=4)
|
||||
seed_t: float = Field(default=0.0, ge=0.0, le=1.0)
|
||||
salience: int = Field(default=4, ge=1, le=4)
|
||||
n_keyframes: int = Field(default=5, ge=2, le=20)
|
||||
max_frames: Optional[int] = None
|
||||
|
||||
|
||||
class AuthorClipRequest(BaseModel):
|
||||
"""Author mode: persist the authored labels/affect/strings for an existing
|
||||
pool clip into the manifest (geometry from the tracker, semantics by hand)."""
|
||||
|
||||
clip_id: str
|
||||
annotations: list = []
|
||||
affect: list = []
|
||||
strings: dict = {}
|
||||
|
||||
|
||||
def _load_clips(manifest_path: Optional[Path]):
|
||||
path = Path(manifest_path) if manifest_path else DEFAULT_MANIFEST
|
||||
if path.exists():
|
||||
@@ -94,6 +124,7 @@ def _load_ring(manifest_path: Optional[Path]):
|
||||
|
||||
def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
|
||||
app = FastAPI(title="HEF Alteration Simulator")
|
||||
app.state.manifest_path = Path(manifest_path) if manifest_path else DEFAULT_MANIFEST
|
||||
app.state.clips = _load_clips(manifest_path)
|
||||
app.state.ring = _load_ring(manifest_path)
|
||||
|
||||
@@ -145,15 +176,76 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
|
||||
req.delta,
|
||||
fast_spin_threshold=DEFAULT_FAST_SPIN_THRESHOLD,
|
||||
)
|
||||
return ring_move_to_dict(move, app.state.ring)
|
||||
# Rotating pool: pick a random member of the LANDED scale (content-pipeline
|
||||
# §11.1). A delta=0 advance is the initial / re-roll pick (a no-op move that
|
||||
# still yields a fresh random clip). The pure pick takes injected randomness.
|
||||
landed = scale_at(app.state.ring, move.to_index)
|
||||
chosen = pick_clip_id(landed, random.random())
|
||||
return ring_move_to_dict(move, app.state.ring, chosen)
|
||||
|
||||
@app.post("/api/author/track")
|
||||
def api_author_track(req: AuthorTrackRequest):
|
||||
# Author mode (§11.5): run the classical optical-flow tracker on a clip's
|
||||
# base footage, returning the keyframed track geometry for the author to
|
||||
# accept/correct. Semantics (key/strings/tiers) stay the author's.
|
||||
clip = next((c for c in app.state.clips if c.id == req.clip_id), None)
|
||||
if clip is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown clip {req.clip_id!r}")
|
||||
base = MEDIA_DIR / clip.base_file
|
||||
if not base.exists():
|
||||
raise HTTPException(status_code=404, detail=f"media absent: {clip.base_file}")
|
||||
try:
|
||||
import cv2
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="opencv-python not installed")
|
||||
from tools.pipeline.track import track_seed
|
||||
|
||||
cap = cv2.VideoCapture(str(base))
|
||||
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
|
||||
cap.release()
|
||||
seed_frame = max(0, min(total - 1, round(req.seed_t * total))) if total else 0
|
||||
return track_seed(
|
||||
base, req.key, tuple(req.seed_box), seed_frame=seed_frame,
|
||||
salience=req.salience, n_keyframes=req.n_keyframes, max_frames=req.max_frames,
|
||||
)
|
||||
|
||||
@app.post("/api/author/clip")
|
||||
def api_author_clip(req: AuthorClipRequest):
|
||||
# Author mode (§11.5 / stage 6): persist authored labels/affect/strings for
|
||||
# an EXISTING pool clip — idempotent upsert that keeps the clip's media +
|
||||
# provenance and replaces only the authored content. Reloads in place so
|
||||
# the preview reflects the edit without a restart.
|
||||
from tools.pipeline.manifest import upsert_clip
|
||||
|
||||
path = app.state.manifest_path
|
||||
data = json.loads(path.read_text())
|
||||
existing = next((c for c in data.get("clips", []) if c["id"] == req.clip_id), None)
|
||||
if existing is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown clip {req.clip_id!r}")
|
||||
merged = dict(existing)
|
||||
merged["annotations"] = req.annotations
|
||||
merged["affect"] = req.affect
|
||||
if req.strings:
|
||||
merged["strings"] = req.strings
|
||||
data = upsert_clip(data, merged)
|
||||
path.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n")
|
||||
app.state.clips = load_manifest(path)
|
||||
app.state.ring = load_ring(path)
|
||||
return {"ok": True, "clip": merged}
|
||||
|
||||
@app.middleware("http")
|
||||
async def _no_cache(request, call_next):
|
||||
# Dev preview server: never let a browser serve a stale app.js/style.css.
|
||||
# The whole point is by-eye iteration, so a plain refresh must always get
|
||||
# the latest assets (the heuristic cache otherwise hides JS/CSS edits).
|
||||
async def _cache_policy(request, call_next):
|
||||
# Dev preview server: never let a browser serve a stale app.js/style.css —
|
||||
# by-eye iteration needs a plain refresh to always get the latest assets.
|
||||
# Media is the exception: the clips are static and large, so they must be
|
||||
# browser-cacheable. The renderer also preloads them into in-memory blob
|
||||
# URLs (static/app.js), but a cacheable header keeps the preload fetch and
|
||||
# any fallback off the network on repeat loads — instant altitude swaps.
|
||||
response = await call_next(request)
|
||||
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
|
||||
if request.url.path.startswith("/media/"):
|
||||
response.headers["Cache-Control"] = "public, max-age=31536000, immutable"
|
||||
else:
|
||||
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
|
||||
return response
|
||||
|
||||
if MEDIA_DIR.exists():
|
||||
|
||||
@@ -0,0 +1,403 @@
|
||||
"""Build the rotating-pool manifest for the 5 ring scales (content-pipeline §11).
|
||||
|
||||
This is the reproducible, hand-authored BASELINE for Increment 2's manifest:
|
||||
each ring scale carries a POOL of vetted clips (docs/content-candidate-pool.md);
|
||||
the player picks one member at random on landing (player.ring.pick_clip_id). The
|
||||
authored label/affect content lives here in readable Python and is emitted to
|
||||
`simulator/sample_media/manifest.json` — far less error-prone than hand-editing 19
|
||||
clips of JSON, and a clean place for the simulator author mode to regenerate from.
|
||||
|
||||
Authoring model (design §11.3 / §11.4):
|
||||
- LEFT detail tiers: each label carries `salience` (1-4; first shows at Left
|
||||
level `5 - salience`) and a tiered string list (general -> specific ->
|
||||
scientific -> +fact) that escalates with the Left knob.
|
||||
- RIGHT emotion tiers: affect words are a tiered list per feeling (basic ->
|
||||
compound) escalating with the Right knob; still gated on min(Left, Right).
|
||||
- TIME-WINDOWED + TRACKED labels: a label may carry `appear`/`disappear`
|
||||
(loop-normalized) and a keyframed `track`; it shows only while on screen and
|
||||
its box follows the subject. The abyss + reef pools are the test material.
|
||||
|
||||
Strings: a value may be a LIST (indexed by tier-1) or a plain string (static,
|
||||
back-compat). Affect appears only when both knobs are up.
|
||||
|
||||
Usage:
|
||||
python simulator/build_pool_manifest.py # write manifest.json
|
||||
python simulator/build_pool_manifest.py --media # + generate new transition placeholders
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
MEDIA = Path(__file__).parent / "sample_media"
|
||||
MANIFEST = MEDIA / "manifest.json"
|
||||
|
||||
# --- Pools: scale id -> ordered clip ids (primary first) (content-candidate-pool.md) ---
|
||||
POOLS: dict[str, list[str]] = {
|
||||
"cosmos": ["cosmos", "cosmos_galaxies", "cosmos_hudf", "cosmos_xdf"],
|
||||
"orbit": ["orbit_planetearth", "orbit_crewobs", "orbit_bluemarble"],
|
||||
"coast": ["coast_birdrock", "coast_surfgrass", "coast_elkbeach", "coast_drakesbeach"],
|
||||
"reef": ["reef_lionfish", "reef_spawning", "reef_hawkfish", "reef_snapper", "reef_coralspacific"],
|
||||
"abyss": ["abyss_wow", "abyss_midwaterexp", "abyss_hiding"],
|
||||
}
|
||||
|
||||
# Ring order (large -> small, wraps): cosmos -> orbit -> coast -> reef -> abyss -> cosmos.
|
||||
RING_ORDER = ["cosmos", "orbit", "coast", "reef", "abyss"]
|
||||
|
||||
PD = "public-domain (US Gov, 17 U.S.C. §105)"
|
||||
PD_NPS = "public-domain (NPS, no © — 17 U.S.C. §105)"
|
||||
CCBY_STSCI = "CC-BY-class — credit NASA, ESA, STScI"
|
||||
|
||||
# --- Per-clip provenance: id -> (title, license, source) ---
|
||||
META: dict[str, tuple[str, str, str]] = {
|
||||
# cosmos
|
||||
"cosmos": ("Orion Nebula flythrough (NASA/JPL-Caltech)", PD,
|
||||
"NASA/JPL-Caltech — images.nasa.gov JPL-20221122-SOLSYSf-0001 (Orion); trim ~30–54s, crossfade-loop"),
|
||||
"cosmos_galaxies": ("Flying Through Galaxies (NASA SVS)", PD,
|
||||
"NASA SVS a014950 14950_Galaxies_FlyThrough_4k; trim 8–32s, crossfade-loop"),
|
||||
"cosmos_hudf": ("Hubble Ultra Deep Field zoom (NASA/ESA/STScI)", CCBY_STSCI,
|
||||
"NASA SVS a030687 hudf-b-1920x1080p30; trim 20–44s, crossfade-loop"),
|
||||
"cosmos_xdf": ("eXtreme Deep Field flythrough (NASA/ESA/STScI)", CCBY_STSCI,
|
||||
"NASA SVS a030681 hxdf_fly-b-1920x1080p30; trim 2–26s, crossfade-loop"),
|
||||
# orbit
|
||||
"orbit_planetearth": ("ISS — View of Planet Earth (NASA SVS)", PD,
|
||||
"NASA SVS a030771 ISS_View_of_Planet_Earth_2160p; trim 10–34s, crossfade-loop"),
|
||||
"orbit_crewobs": ("ISS — Crew Earth Observations (NASA SVS)", PD,
|
||||
"NASA SVS a030771 ISS_Crew_Earth_Observations_2160p; trim 9–33s, crossfade-loop"),
|
||||
"orbit_bluemarble": ("NPP “Blue Marble” rotating globe (NASA SVS)", PD,
|
||||
"NASA SVS a004550 BlueMarble_starfield_4k_2160p30_v2; trim 10–34s, crossfade-loop"),
|
||||
# coast
|
||||
"coast_birdrock": ("Bird rock + ocean, aerial (NPS Channel Islands)", PD_NPS,
|
||||
"NPS Channel Islands AV 663a2f13; trim 3–27s, crossfade-loop"),
|
||||
"coast_surfgrass": ("Surfgrass / coralline tidepool (NPS Cabrillo)", PD_NPS,
|
||||
"NPS Cabrillo AV de7d1cf2; trim 2–26s, crossfade-loop"),
|
||||
"coast_elkbeach": ("Elk resting in coastal fog (NPS Redwood)", PD_NPS,
|
||||
"NPS Redwood redw-elkbeach_1280x720; trim 20–38s, crossfade-loop (720p source)"),
|
||||
"coast_drakesbeach": ("Elephant seals on dark sand (NPS Point Reyes)", PD_NPS,
|
||||
"NPS Point Reyes AV 1cfd8165; trim 20–35s, crossfade-loop"),
|
||||
# reef
|
||||
"reef_lionfish": ("Lionfish hovering over reef (NOAA Fisheries)", PD,
|
||||
"NOAA Fisheries b-roll VIDEO_ID 4088881464001; trim 12–36s, crossfade-loop"),
|
||||
"reef_spawning": ("Snapper school over sunlit reef (NOAA Fisheries)", PD,
|
||||
"NOAA Fisheries b-roll VIDEO_ID 1553921798001; trim 50–78s, crossfade-loop"),
|
||||
"reef_hawkfish": ("Humphead parrotfish over reef (NOAA Fisheries)", PD,
|
||||
"NOAA Fisheries b-roll VIDEO_ID 6039896460001; trim 250–274s, crossfade-loop"),
|
||||
"reef_snapper": ("Red snapper schooling (NOAA Fisheries)", PD,
|
||||
"NOAA Fisheries b-roll VIDEO_ID 5305427942001; trim 7–31s, crossfade-loop"),
|
||||
"reef_coralspacific": ("Pacific coral macro (NOAA Fisheries)", PD,
|
||||
"NOAA Fisheries b-roll VIDEO_ID 5231464663001; trim 30–54s, crossfade-loop"),
|
||||
# abyss
|
||||
"abyss_wow": ("“World of Water” (NOAA Ocean Exploration)", PD,
|
||||
"NOAA Ocean Exploration wow-1280x720-1; trim 10–34s, crossfade-loop"),
|
||||
"abyss_midwaterexp": ("“Midwater Exploration” (NOAA Ocean Exploration)", PD,
|
||||
"NOAA Ocean Exploration midwater-exploration-1920x1080-1; trim 16–40s, crossfade-loop"),
|
||||
"abyss_hiding": ("“Hiding in the Dark” (NOAA Ocean Exploration)", PD,
|
||||
"NOAA Ocean Exploration dark-1280x720-1; trim 20–44s, crossfade-loop"),
|
||||
}
|
||||
|
||||
# --- Affect vocabulary per SCALE (shared by all the scale's pool members) -------
|
||||
# Each: (key, [x, y], min_level, [tier1 basic .. tier4 compound]). The word shown
|
||||
# escalates with the RIGHT knob; appears only when both knobs are up (§11.4).
|
||||
AFFECT: dict[str, list[tuple]] = {
|
||||
"cosmos": [
|
||||
("feel.wonder", [0.50, 0.44], 1, ["wow", "wonder", "awe", "transcendent awe"]),
|
||||
("feel.vastness", [0.20, 0.70], 2, ["big", "vastness", "immensity", "a dizzying immensity"]),
|
||||
("feel.insignificance", [0.66, 0.60], 3, ["small", "smallness", "insignificance", "a humbling insignificance"]),
|
||||
("feel.longing", [0.42, 0.84], 4, ["want", "longing", "yearning", "an aching yearning"]),
|
||||
],
|
||||
"orbit": [
|
||||
("feel.serenity", [0.50, 0.44], 1, ["calm", "serenity", "peace", "a quiet, weightless peace"]),
|
||||
("feel.fragility", [0.22, 0.68], 2, ["thin", "fragility", "vulnerability", "a tender fragility"]),
|
||||
("feel.unity", [0.66, 0.58], 3, ["one", "unity", "belonging", "a borderless belonging"]),
|
||||
("feel.distance", [0.42, 0.82], 4, ["far", "distance", "remoteness", "an exquisite remoteness"]),
|
||||
],
|
||||
"coast": [
|
||||
("feel.ease", [0.50, 0.44], 1, ["nice", "ease", "calm", "an unhurried calm"]),
|
||||
("feel.nostalgia", [0.22, 0.68], 2, ["miss", "nostalgia", "wistfulness", "a salt-air wistfulness"]),
|
||||
("feel.belonging", [0.66, 0.58], 3, ["home", "belonging", "rootedness", "a tidal rootedness"]),
|
||||
("feel.melancholy", [0.42, 0.82], 4, ["sad", "melancholy", "longing", "a soft seaward longing"]),
|
||||
],
|
||||
"reef": [
|
||||
("feel.delight", [0.50, 0.44], 1, ["fun", "delight", "joy", "a darting, bright joy"]),
|
||||
("feel.abundance", [0.22, 0.68], 2, ["full", "abundance", "richness", "a teeming richness"]),
|
||||
("feel.curiosity", [0.66, 0.58], 3, ["look", "curiosity", "fascination", "an absorbed fascination"]),
|
||||
("feel.immersion", [0.42, 0.82], 4, ["in", "immersion", "absorption", "a held, breathless absorption"]),
|
||||
],
|
||||
"abyss": [
|
||||
("feel.unease", [0.50, 0.44], 1, ["uh", "unease", "disquiet", "a creeping disquiet"]),
|
||||
("feel.fascination", [0.22, 0.68], 2, ["ooh", "fascination", "wonder", "a forbidden wonder"]),
|
||||
("feel.isolation", [0.66, 0.58], 3, ["alone", "isolation", "solitude", "a crushing solitude"]),
|
||||
("feel.dread", [0.42, 0.82], 4, ["fear", "dread", "foreboding", "a slow, cold foreboding"]),
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def static_label(key, salience, tiers, box):
|
||||
"""A fixed-box tiered label (always on-screen, salience-gated by Left)."""
|
||||
return {"key": key, "salience": salience, "box": box, "_tiers": tiers}
|
||||
|
||||
|
||||
def tracked_label(key, salience, tiers, appear, disappear, track):
|
||||
"""A time-windowed tracked label: shows only in [appear, disappear] and its
|
||||
box follows the subject (§11.2). `track` = [(t, [x,y,w,h]), ...]."""
|
||||
return {
|
||||
"key": key, "salience": salience, "appear": appear, "disappear": disappear,
|
||||
"track": [{"t": t, "box": b} for t, b in track], "_tiers": tiers,
|
||||
}
|
||||
|
||||
|
||||
def measure(key, value, box, min_level):
|
||||
"""A single-tier measurement readout (not a tiered object), gated by Left."""
|
||||
return {"key": key, "box": box, "min_level": min_level, "_tiers": value}
|
||||
|
||||
|
||||
# --- Labels per clip. `_tiers` is lifted into the strings table by build(). ------
|
||||
# Showcase tracked/windowed labels live on the abyss + reef pools (the creatures
|
||||
# drift through frame). Other clips get a few salience-gated static tiered labels.
|
||||
LABELS: dict[str, list[dict]] = {
|
||||
# ---------- cosmos ----------
|
||||
"cosmos": [
|
||||
static_label("detected.nebula", 4, ["cloud", "nebula", "emission nebula", "emission nebula · ionized H II region"], [0.32, 0.28, 0.34, 0.36]),
|
||||
static_label("detected.star", 2, ["star", "young star", "protostar", "protostar · <1 Myr old"], [0.60, 0.30, 0.10, 0.12]),
|
||||
measure("measure.redshift", "z ≈ 0.0", [0.36, 0.70, 0.18, 0.08], 4),
|
||||
],
|
||||
"cosmos_galaxies": [
|
||||
static_label("detected.galaxy", 4, ["galaxy", "spiral galaxy", "barred spiral", "barred spiral · ~10¹¹ stars"], [0.34, 0.30, 0.30, 0.34]),
|
||||
measure("measure.distance", "~Mly", [0.06, 0.06, 0.18, 0.08], 3),
|
||||
],
|
||||
"cosmos_hudf": [
|
||||
static_label("detected.galaxy", 4, ["smudge", "galaxy", "early galaxy", "early galaxy · z ≈ 1–6, light Gyr-old"], [0.40, 0.36, 0.20, 0.22]),
|
||||
measure("measure.field", "deep field", [0.06, 0.06, 0.2, 0.08], 3),
|
||||
],
|
||||
"cosmos_xdf": [
|
||||
static_label("detected.galaxy", 4, ["dot", "galaxy", "faint galaxy", "faint galaxy · among the earliest seen"], [0.42, 0.38, 0.18, 0.2]),
|
||||
],
|
||||
# ---------- orbit ----------
|
||||
"orbit_planetearth": [
|
||||
static_label("detected.cloud_band", 4, ["clouds", "cloud band", "cumulus field", "cumulus field · tropical convection"], [0.30, 0.30, 0.30, 0.20]),
|
||||
static_label("detected.limb", 2, ["edge", "Earth’s limb", "atmospheric limb", "atmospheric limb · ~100 km of air"], [0.05, 0.70, 0.9, 0.12]),
|
||||
measure("measure.altitude", "~408 km", [0.06, 0.06, 0.18, 0.08], 3),
|
||||
],
|
||||
"orbit_crewobs": [
|
||||
static_label("detected.coastline", 4, ["land", "coastline", "continental margin", "continental margin · land meets sea"], [0.30, 0.34, 0.34, 0.26]),
|
||||
measure("measure.altitude", "~408 km", [0.06, 0.06, 0.18, 0.08], 3),
|
||||
],
|
||||
"orbit_bluemarble": [
|
||||
static_label("detected.globe", 4, ["Earth", "the globe", "terrestrial planet", "terrestrial planet · 12,742 km across"], [0.28, 0.18, 0.44, 0.6]),
|
||||
],
|
||||
# ---------- coast ----------
|
||||
"coast_birdrock": [
|
||||
static_label("detected.surf", 4, ["waves", "surf", "breaking swell", "breaking swell · wind-driven, ~10 s period"], [0.20, 0.55, 0.6, 0.3]),
|
||||
static_label("detected.searock", 2, ["rock", "sea stack", "coastal sea stack", "sea stack · wave-cut residual rock"], [0.30, 0.25, 0.22, 0.3]),
|
||||
],
|
||||
"coast_surfgrass": [
|
||||
static_label("detected.surfgrass", 4, ["grass", "surfgrass", "Phyllospadix", "Phyllospadix · a marine seagrass, not algae"], [0.18, 0.40, 0.5, 0.4]),
|
||||
static_label("detected.coralline", 2, ["pink", "coralline algae", "crustose coralline", "crustose coralline · calcified red algae"], [0.6, 0.55, 0.2, 0.2]),
|
||||
],
|
||||
"coast_elkbeach": [
|
||||
static_label("detected.elk", 4, ["elk", "Roosevelt elk", "Cervus canadensis roosevelti", "C. c. roosevelti · largest elk subspecies"], [0.34, 0.42, 0.3, 0.3]),
|
||||
static_label("detected.fog", 2, ["fog", "coastal fog", "advection fog", "advection fog · warm air over cold upwelling"], [0.05, 0.05, 0.9, 0.25]),
|
||||
],
|
||||
"coast_drakesbeach": [
|
||||
static_label("detected.seal", 4, ["seals", "elephant seals", "Mirounga angustirostris", "M. angustirostris · males to 2,000 kg"], [0.25, 0.55, 0.5, 0.3]),
|
||||
static_label("detected.sand", 1, ["sand", "dark sand", "mineral-dark beach", "dark beach · eroded coastal sediment"], [0.05, 0.8, 0.9, 0.15]),
|
||||
],
|
||||
# ---------- reef (showcase: tracked + windowed) ----------
|
||||
"reef_lionfish": [
|
||||
tracked_label(
|
||||
"detected.lionfish", 4,
|
||||
["fish", "lionfish", "Pterois volitans", "Pterois volitans · venomous spines, invasive in the Atlantic"],
|
||||
0.05, 0.95,
|
||||
[(0.05, [0.15, 0.30, 0.16, 0.18]), (0.5, [0.50, 0.38, 0.16, 0.18]), (0.95, [0.72, 0.34, 0.16, 0.18])],
|
||||
),
|
||||
tracked_label(
|
||||
"detected.snapper", 2,
|
||||
["fish", "blue snapper", "Lutjanus", "Lutjanus · schools over reef structure"],
|
||||
0.0, 0.6,
|
||||
[(0.0, [0.05, 0.55, 0.12, 0.1]), (0.3, [0.22, 0.5, 0.12, 0.1]), (0.6, [0.40, 0.56, 0.12, 0.1])],
|
||||
),
|
||||
measure("measure.depth", "−18 m", [0.06, 0.06, 0.16, 0.08], 3),
|
||||
],
|
||||
"reef_spawning": [
|
||||
tracked_label(
|
||||
"detected.school", 4,
|
||||
["fish", "fish school", "snapper aggregation", "snapper aggregation · synchronized spawning run"],
|
||||
0.0, 1.0,
|
||||
[(0.0, [0.20, 0.30, 0.2, 0.18]), (0.5, [0.55, 0.34, 0.2, 0.18]), (1.0, [0.20, 0.30, 0.2, 0.18])],
|
||||
),
|
||||
static_label("detected.coral", 2, ["coral", "reef coral", "scleractinian", "scleractinian · reef-building stony coral"], [0.15, 0.6, 0.35, 0.3]),
|
||||
measure("measure.depth", "−22 m", [0.06, 0.06, 0.16, 0.08], 3),
|
||||
],
|
||||
"reef_hawkfish": [
|
||||
tracked_label(
|
||||
"detected.parrotfish", 4,
|
||||
["fish", "parrotfish", "Bolbometopon muricatum", "B. muricatum · humphead, grazes reef into sand"],
|
||||
0.1, 0.9,
|
||||
[(0.1, [0.55, 0.25, 0.2, 0.22]), (0.5, [0.35, 0.34, 0.2, 0.22]), (0.9, [0.12, 0.40, 0.2, 0.22])],
|
||||
),
|
||||
measure("measure.depth", "−12 m", [0.06, 0.06, 0.16, 0.08], 3),
|
||||
],
|
||||
"reef_snapper": [
|
||||
tracked_label(
|
||||
"detected.snapper", 4,
|
||||
["fish", "red snapper", "Lutjanus campechanus", "L. campechanus · long-lived, can reach 50+ yr"],
|
||||
0.0, 0.7,
|
||||
[(0.0, [0.10, 0.3, 0.16, 0.14]), (0.35, [0.40, 0.36, 0.16, 0.14]), (0.7, [0.68, 0.30, 0.16, 0.14])],
|
||||
),
|
||||
],
|
||||
"reef_coralspacific": [
|
||||
static_label("detected.coral", 4, ["coral", "coral colony", "Pacific scleractinian", "Pacific scleractinian · a symbiosis with algae"], [0.2, 0.4, 0.4, 0.4]),
|
||||
tracked_label(
|
||||
"detected.spotfish", 2,
|
||||
["fish", "spotted fish", "reef damselfish", "reef damselfish · territorial over its coral"],
|
||||
0.2, 0.8,
|
||||
[(0.2, [0.6, 0.25, 0.1, 0.09]), (0.5, [0.45, 0.3, 0.1, 0.09]), (0.8, [0.3, 0.26, 0.1, 0.09])],
|
||||
),
|
||||
],
|
||||
# ---------- abyss (showcase: tracked + windowed; creatures drift through) ----------
|
||||
"abyss_wow": [
|
||||
tracked_label(
|
||||
"detected.jelly", 4,
|
||||
["jelly", "comb jelly", "Ctenophora", "Ctenophora · swims by beating rows of cilia"],
|
||||
0.0, 0.55,
|
||||
[(0.0, [0.05, 0.25, 0.14, 0.18]), (0.3, [0.30, 0.32, 0.14, 0.18]), (0.55, [0.52, 0.28, 0.14, 0.18])],
|
||||
),
|
||||
tracked_label(
|
||||
"detected.siphonophore", 3,
|
||||
["chain", "siphonophore", "Siphonophorae", "Siphonophorae · a colony of clones, can exceed 40 m"],
|
||||
0.45, 1.0,
|
||||
[(0.45, [0.80, 0.6, 0.1, 0.3]), (0.7, [0.6, 0.45, 0.1, 0.34]), (1.0, [0.45, 0.3, 0.1, 0.38])],
|
||||
),
|
||||
measure("measure.depth", "−1,200 m", [0.06, 0.06, 0.16, 0.08], 2),
|
||||
],
|
||||
"abyss_midwaterexp": [
|
||||
tracked_label(
|
||||
"detected.medusa", 4,
|
||||
["jelly", "white medusa", "Scyphozoa", "Scyphozoa · the swimming medusa stage of a jellyfish"],
|
||||
0.0, 0.6,
|
||||
[(0.0, [0.55, 0.20, 0.16, 0.2]), (0.3, [0.40, 0.34, 0.16, 0.2]), (0.6, [0.22, 0.30, 0.16, 0.2])],
|
||||
),
|
||||
tracked_label(
|
||||
"detected.worm", 2,
|
||||
["worm", "red worm", "polychaete", "polychaete · a free-swimming bristle worm"],
|
||||
0.5, 1.0,
|
||||
[(0.5, [0.10, 0.7, 0.12, 0.1]), (0.75, [0.3, 0.6, 0.12, 0.1]), (1.0, [0.5, 0.66, 0.12, 0.1])],
|
||||
),
|
||||
measure("measure.depth", "−1,600 m", [0.06, 0.06, 0.16, 0.08], 2),
|
||||
],
|
||||
"abyss_hiding": [
|
||||
tracked_label(
|
||||
"detected.jelly", 4,
|
||||
["jelly", "crimson jelly", "Scyphozoa", "Scyphozoa · red is invisible in the lightless deep"],
|
||||
0.0, 0.7,
|
||||
[(0.0, [0.15, 0.28, 0.16, 0.2]), (0.35, [0.42, 0.36, 0.16, 0.2]), (0.7, [0.66, 0.30, 0.16, 0.2])],
|
||||
),
|
||||
measure("measure.depth", "−2,140 m", [0.06, 0.06, 0.16, 0.08], 2),
|
||||
],
|
||||
}
|
||||
|
||||
# Human-readable strings for affect/measurement keys are produced from the tiered
|
||||
# data above; this maps measurement keys to nothing extra (their value IS the string).
|
||||
|
||||
|
||||
def _affect_for_clip(scale: str) -> tuple[list, dict]:
|
||||
"""Build the affect list + its strings for a scale's pool member."""
|
||||
entries, strings = [], {}
|
||||
for key, at, min_level, tiers in AFFECT[scale]:
|
||||
entries.append({"key": key, "at": at, "min_level": min_level})
|
||||
strings[key] = tiers
|
||||
return entries, strings
|
||||
|
||||
|
||||
def _labels_for_clip(clip_id: str) -> tuple[list, dict]:
|
||||
"""Build the annotation list + its strings, lifting `_tiers` into the table."""
|
||||
anns, strings = [], {}
|
||||
for raw in LABELS.get(clip_id, []):
|
||||
a = {k: v for k, v in raw.items() if k != "_tiers"}
|
||||
anns.append(a)
|
||||
strings[a["key"]] = raw["_tiers"]
|
||||
return anns, strings
|
||||
|
||||
|
||||
def _clip_entry(scale: str, clip_id: str) -> dict:
|
||||
title, license_, source = META[clip_id]
|
||||
anns, lab_strings = _labels_for_clip(clip_id)
|
||||
affect, aff_strings = _affect_for_clip(scale)
|
||||
return {
|
||||
"id": clip_id,
|
||||
"title": title,
|
||||
"base_file": f"{clip_id}/base.mp4",
|
||||
"license": license_,
|
||||
"source": source,
|
||||
"right_variants": {},
|
||||
"annotations": anns,
|
||||
"affect": affect,
|
||||
"strings": {"en": {**lab_strings, **aff_strings}},
|
||||
}
|
||||
|
||||
|
||||
def build_manifest() -> dict:
|
||||
"""Assemble the full rotating-pool manifest dict."""
|
||||
clips = []
|
||||
for scale in RING_ORDER:
|
||||
for clip_id in POOLS[scale]:
|
||||
clips.append(_clip_entry(scale, clip_id))
|
||||
scales = [{"id": s, "clip_id": POOLS[s][0], "pool": POOLS[s]} for s in RING_ORDER]
|
||||
transitions = []
|
||||
n = len(RING_ORDER)
|
||||
for i in range(n):
|
||||
a, b = RING_ORDER[i], RING_ORDER[(i + 1) % n]
|
||||
transitions.append({"file": f"transitions/{a}-{b}.mp4", "model": "placeholder-zoom"})
|
||||
return {"clips": clips, "ring": {"scales": scales, "transitions": transitions}}
|
||||
|
||||
|
||||
# --- Optional: generate the NEW transition placeholders (orbit-coast, coast-reef) ---
|
||||
|
||||
def _ffmpeg() -> str:
|
||||
if shutil.which("ffmpeg"):
|
||||
return "ffmpeg"
|
||||
import imageio_ffmpeg
|
||||
return imageio_ffmpeg.get_ffmpeg_exe()
|
||||
|
||||
|
||||
def _make_transition(ff: str, scale_a: str, scale_b: str) -> Path:
|
||||
"""A placeholder zoom/warp morph between two scales, from their PRIMARY pool
|
||||
members' bases (mirrors setup_scales_media.py but keyed by scale->primary)."""
|
||||
a = MEDIA / POOLS[scale_a][0] / "base.mp4"
|
||||
b = MEDIA / POOLS[scale_b][0] / "base.mp4"
|
||||
out = MEDIA / "transitions" / f"{scale_a}-{scale_b}.mp4"
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
norm = "trim=0:3,setpts=PTS-STARTPTS,scale=1280:720,fps=25,setsar=1,format=yuv420p"
|
||||
subprocess.run([
|
||||
ff, "-y", "-i", str(a), "-i", str(b), "-filter_complex",
|
||||
f"[0:v]{norm}[a];[1:v]{norm}[b];"
|
||||
"[a][b]xfade=transition=zoomin:duration=1.5:offset=0.75,format=yuv420p[v]",
|
||||
"-map", "[v]", "-an", str(out),
|
||||
], check=True, capture_output=True)
|
||||
return out
|
||||
|
||||
|
||||
def generate_media() -> None:
|
||||
"""Generate the new ring edges introduced by the coast scale; the cosmos-orbit,
|
||||
reef-abyss and abyss-cosmos edges already exist from the prior 5-scale ring."""
|
||||
ff = _ffmpeg()
|
||||
for a, b in [("orbit", "coast"), ("coast", "reef")]:
|
||||
_make_transition(ff, a, b)
|
||||
print(f"generated transitions/{a}-{b}.mp4 (placeholder zoom)")
|
||||
|
||||
|
||||
def main(argv: list[str]) -> None:
|
||||
manifest = build_manifest()
|
||||
MANIFEST.write_text(json.dumps(manifest, indent=2, ensure_ascii=False) + "\n")
|
||||
print(f"wrote {MANIFEST} — {len(manifest['clips'])} clips, "
|
||||
f"{len(manifest['ring']['scales'])} ring scales")
|
||||
if "--media" in argv:
|
||||
generate_media()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main(sys.argv[1:])
|
||||
+37
-8
@@ -73,6 +73,18 @@ def load_manifest(path: str | Path) -> list[Clip]:
|
||||
return [_clip_from_dict(c) for c in data["clips"]]
|
||||
|
||||
|
||||
def _scale_from_dict(s: dict[str, Any]) -> Scale:
|
||||
"""A ring scale node, with its rotating clip pool (content-pipeline §11.1).
|
||||
|
||||
Accepts a `pool` list (the rotating model) and/or a legacy single `clip_id`.
|
||||
The primary `clip_id` is the explicit one if given, else the first pool
|
||||
member; `pool` defaults to `(clip_id,)` for a pre-pool manifest.
|
||||
"""
|
||||
pool = tuple(s.get("pool", []))
|
||||
clip_id = s.get("clip_id") or (pool[0] if pool else "")
|
||||
return Scale(id=s["id"], clip_id=clip_id, pool=pool)
|
||||
|
||||
|
||||
def load_ring(path: str | Path) -> ScaleRing | None:
|
||||
"""Load the optional `ring` section as a `ScaleRing`, or None if absent.
|
||||
|
||||
@@ -85,9 +97,7 @@ def load_ring(path: str | Path) -> ScaleRing | None:
|
||||
ring = data.get("ring")
|
||||
if not ring:
|
||||
return None
|
||||
scales = tuple(
|
||||
Scale(id=s["id"], clip_id=s["clip_id"]) for s in ring.get("scales", [])
|
||||
)
|
||||
scales = tuple(_scale_from_dict(s) for s in ring.get("scales", []))
|
||||
transitions = tuple(
|
||||
Transition(file=t["file"], model=t.get("model", ""))
|
||||
for t in ring.get("transitions", [])
|
||||
@@ -97,27 +107,46 @@ def load_ring(path: str | Path) -> ScaleRing | None:
|
||||
|
||||
def ring_to_dict(ring: ScaleRing, clips: list[Clip]) -> dict:
|
||||
"""JSON form of the ring for the API: ordered scales (with their clip title)
|
||||
and the per-edge transitions."""
|
||||
and the per-edge transitions.
|
||||
|
||||
Each scale carries its full rotating `pool` (clip id + title per member) plus
|
||||
the primary `clip_id`/`title` for back-compat. The client lands on a member
|
||||
chosen at the API boundary (`/api/ring/advance` → `target_clip_id`)."""
|
||||
titles = {c.id: c.title for c in clips}
|
||||
return {
|
||||
"scales": [
|
||||
{"id": s.id, "clip_id": s.clip_id, "title": titles.get(s.clip_id, s.id)}
|
||||
{
|
||||
"id": s.id,
|
||||
"clip_id": s.clip_id,
|
||||
"title": titles.get(s.clip_id, s.id),
|
||||
"pool": [
|
||||
{"clip_id": cid, "title": titles.get(cid, cid)}
|
||||
for cid in s.members
|
||||
],
|
||||
}
|
||||
for s in ring.scales
|
||||
],
|
||||
"transitions": [{"file": t.file, "model": t.model} for t in ring.transitions],
|
||||
}
|
||||
|
||||
|
||||
def ring_move_to_dict(move: RingMove, ring: ScaleRing) -> dict:
|
||||
def ring_move_to_dict(
|
||||
move: RingMove, ring: ScaleRing, chosen_clip_id: str | None = None
|
||||
) -> dict:
|
||||
"""JSON form of an encoder move: the landing scale's clip and the ordered
|
||||
transition clips to play (with direction). `fast` flags a collapsed fast-spin
|
||||
pass; the single step then carries `blended` so the renderer plays it quick."""
|
||||
pass; the single step then carries `blended` so the renderer plays it quick.
|
||||
|
||||
`target_clip_id` is the pool member the player should load on landing. The
|
||||
caller passes the random pick (`pick_clip_id(landed_scale, random.random())`,
|
||||
content-pipeline §11.1); when omitted it falls back to the scale's primary
|
||||
`clip_id` (deterministic — keeps pre-pool callers/tests working)."""
|
||||
return {
|
||||
"from_index": move.from_index,
|
||||
"to_index": move.to_index,
|
||||
"wrapped": move.wrapped,
|
||||
"fast": move.fast,
|
||||
"target_clip_id": scale_at(ring, move.to_index).clip_id,
|
||||
"target_clip_id": chosen_clip_id or scale_at(ring, move.to_index).clip_id,
|
||||
"steps": [
|
||||
{
|
||||
"edge": st.edge,
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,16 +1,26 @@
|
||||
"""Generate cheap placeholder media for the scale RING (scales design §3).
|
||||
"""Generate cheap placeholder media for the full 5-scale RING (scales design §3).
|
||||
|
||||
The ring needs >=2 neutral scales to be demonstrable. This script produces
|
||||
cheap, offline, synthetic placeholders for the two true-PD scales the ring adds —
|
||||
**cosmos** (NASA/Hubble) and **abyss** (NOAA Ocean Exploration) — plus the short
|
||||
zoom/warp **transition** clips between adjacent scales. Forest reuses the real
|
||||
POC base from `setup_sample_media.py` (or a synthetic placeholder if absent).
|
||||
SUPERSEDED (session 0016): the ring now uses real strict-PD footage in a ROTATING
|
||||
POOL per scale, regenerated by `simulator/build_pool_manifest.py` (which also makes
|
||||
the new coast-edge transition placeholders). This generator built the older
|
||||
one-base-per-scale forest ring (cosmos/orbit/forest/reef/abyss) and is kept only
|
||||
for reference / a clean-room placeholder rebuild; it does NOT match the current
|
||||
pool manifest or ring order (cosmos → orbit → coast → reef → abyss).
|
||||
|
||||
|
||||
Produces offline synthetic placeholders for all five ring scales — **cosmos**
|
||||
(NASA/Hubble), **orbit** (NASA/ISS), **forest** (NPS/USGS), **reef** (NOAA Ocean
|
||||
Exploration), and **abyss** (NOAA Ocean Exploration) — plus the short zoom/warp
|
||||
**transition** clips between adjacent scales. Forest reuses the real POC base from
|
||||
`setup_sample_media.py` (or a synthetic placeholder if absent); all others are
|
||||
generated as labelled solid-tint clips.
|
||||
|
||||
This is deliberately the CHEAP path: the manifest records the real true-PD
|
||||
provenance (NASA 17 U.S.C. §105 / NOAA PD), but the bytes here are labelled
|
||||
synthetic placeholders for look-tuning the ring navigation. The expensive real
|
||||
multi-strength flow-stabilized Right re-bake is DEFERRED until the ring is liked,
|
||||
so the new scales carry a raw base only (no Right variants).
|
||||
provenance (NASA / NPS / USGS / NOAA, 17 U.S.C. §105), but the bytes here are
|
||||
labelled synthetic placeholders for look-tuning the ring navigation. The Right dream
|
||||
is real-time (Kuwahara WebGL shader), so no per-clip ML bake is needed; each scale
|
||||
carries a raw base only. Real strict-PD bases replace these placeholders via the
|
||||
content pipeline (see `docs/content-sourcing.md`).
|
||||
|
||||
Usage: python simulator/setup_scales_media.py
|
||||
Requires: ffmpeg on PATH (or `pip install imageio-ffmpeg`).
|
||||
@@ -31,12 +41,15 @@ FPS = 25
|
||||
# Each scale: (dir, base color, on-screen placeholder label).
|
||||
SCALES = {
|
||||
"cosmos": ("0x05060f", "COSMOS — NASA/Hubble (PD placeholder)"),
|
||||
"forest": ("0x12301a", "FOREST — Yosemite (POC/placeholder)"),
|
||||
"orbit": ("0x081427", "ORBIT — NASA/ISS (PD placeholder)"),
|
||||
"forest": ("0x12301a", "FOREST — NPS/USGS (POC/placeholder)"),
|
||||
"reef": ("0x06303a", "REEF — NOAA Ocean Exploration (PD placeholder)"),
|
||||
"abyss": ("0x021016", "ABYSS — NOAA Ocean Exploration (PD placeholder)"),
|
||||
}
|
||||
|
||||
# Ring edges (adjacent pairs + the abyss->cosmos wrap), matching the manifest.
|
||||
EDGES = [("cosmos", "forest"), ("forest", "abyss"), ("abyss", "cosmos")]
|
||||
EDGES = [("cosmos", "orbit"), ("orbit", "forest"), ("forest", "reef"),
|
||||
("reef", "abyss"), ("abyss", "cosmos")]
|
||||
|
||||
|
||||
def _ffmpeg() -> str:
|
||||
|
||||
+445
-17
@@ -31,29 +31,129 @@ window.addEventListener("error", (e) =>
|
||||
showError(`${e.message} @ ${(e.filename || "").split("/").pop()}:${e.lineno}`));
|
||||
|
||||
let clipsById = {}; // id -> clip manifest entry
|
||||
let ring = null; // {scales:[{id,clip_id,title}], transitions:[...]} or null
|
||||
let ring = null; // {scales:[{id,clip_id,title,pool:[...]}], transitions:[...]} or null
|
||||
let serverRing = false; // true when /api/ring served a real ring (vs the synthesized fallback)
|
||||
let ringIndex = 0; // current scale position on the ring
|
||||
let activeClipId = null; // the pool member chosen for the current scale landing
|
||||
let currentClipId = null; // base media currently loaded (reset to force a reload)
|
||||
let busy = false; // true while a ring transition is playing
|
||||
let lastOverlay = null; // {level, intensity} from the most-recent renderOverlay call; drives per-frame track animation
|
||||
|
||||
async function loadData() {
|
||||
const clips = (await (await fetch("/api/clips")).json()).clips || [];
|
||||
clipsById = Object.fromEntries(clips.map((c) => [c.id, c]));
|
||||
const r = await fetch("/api/ring");
|
||||
serverRing = r.ok;
|
||||
ring = r.ok ? await r.json() : null;
|
||||
if (!ring && clips.length) {
|
||||
// No ring: single-clip mode — synthesize a 1-scale ring over clip 0.
|
||||
ring = { scales: [{ id: clips[0].id, clip_id: clips[0].id, title: clips[0].title }], transitions: [] };
|
||||
// No ring: single-clip mode — synthesize a 1-scale ring (pool of one) over clip 0.
|
||||
ring = { scales: [{ id: clips[0].id, clip_id: clips[0].id, title: clips[0].title, pool: [{ clip_id: clips[0].id, title: clips[0].title }] }], transitions: [] };
|
||||
}
|
||||
ringIndex = 0;
|
||||
}
|
||||
|
||||
function activeClip() {
|
||||
if (!ring) return null;
|
||||
return clipsById[ring.scales[ringIndex].clip_id] || null;
|
||||
// Server-canonical random pick of a pool member for the current scale: a delta=0
|
||||
// advance (content-pipeline §11.1, Python owns the randomness). The synthesized
|
||||
// fallback ring has a pool of one, so it just returns that member. Returns a
|
||||
// clip_id or null. Shared by the initial landing AND the Dev Mode re-roll button.
|
||||
async function pickRandomMember() {
|
||||
const scale = ring && ring.scales[ringIndex];
|
||||
if (!scale) return null;
|
||||
if (serverRing) {
|
||||
try {
|
||||
const resp = await fetch("/api/ring/advance", {
|
||||
method: "POST", headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ from_index: ringIndex, delta: 0 }),
|
||||
});
|
||||
if (resp.ok) return (await resp.json()).target_clip_id;
|
||||
} catch (_) { /* fall through to the primary member */ }
|
||||
}
|
||||
return scale.clip_id;
|
||||
}
|
||||
|
||||
function mediaUrl(file) { return "/media/" + file; }
|
||||
// Land on the current scale: pick a random pool member and force its media to load.
|
||||
async function landScale() {
|
||||
activeClipId = await pickRandomMember();
|
||||
currentClipId = null;
|
||||
}
|
||||
|
||||
function activeClip() {
|
||||
return clipsById[activeClipId] || null;
|
||||
}
|
||||
|
||||
// In-memory preload cache: media file path -> blob object URL. Once a clip's bytes
|
||||
// are cached, mediaUrl() hands the <video> an in-memory blob instead of a network
|
||||
// path, so swapping the altitude scale loads near-instantly (no fetch round-trip).
|
||||
// Falls back to the network path for anything not yet (or never) cached.
|
||||
const mediaBlobs = {};
|
||||
function mediaUrl(file) { return mediaBlobs[file] || ("/media/" + file); }
|
||||
|
||||
// Every media file the ring can show: each scale-pool clip's base footage plus the
|
||||
// per-edge transition clips. This is the full preload set (~two dozen files).
|
||||
function preloadList() {
|
||||
const files = new Set();
|
||||
for (const c of Object.values(clipsById)) if (c && c.base_file) files.add(c.base_file);
|
||||
if (ring && ring.transitions) for (const t of ring.transitions) if (t && t.file) files.add(t.file);
|
||||
return [...files];
|
||||
}
|
||||
|
||||
// Order the preload so the clips most likely to be shown next come first: the
|
||||
// current scale's pool, then outward to neighboring scales, then transitions.
|
||||
function preloadOrder() {
|
||||
if (!ring || !ring.scales) return preloadList();
|
||||
const n = ring.scales.length, seen = new Set(), ordered = [];
|
||||
const add = (f) => { if (f && !seen.has(f)) { seen.add(f); ordered.push(f); } };
|
||||
for (let d = 0; d < n; d++) {
|
||||
for (const sign of d === 0 ? [0] : [1, -1]) {
|
||||
const s = ring.scales[((ringIndex + sign * d) % n + n) % n];
|
||||
if (s && s.pool) for (const m of s.pool) add((clipsById[m.clip_id] || {}).base_file);
|
||||
else if (s) add((clipsById[s.clip_id] || {}).base_file);
|
||||
}
|
||||
}
|
||||
if (ring.transitions) for (const t of ring.transitions) add(t.file);
|
||||
for (const f of preloadList()) add(f); // sweep up anything not on the ring
|
||||
return ordered;
|
||||
}
|
||||
|
||||
// Fetch every clip into the blob cache in the background — non-blocking, so the
|
||||
// first scale plays immediately while the rest stream in. Bounded concurrency keeps
|
||||
// the ~350 MB fetch from stampeding. A tiny progress chip self-removes when done.
|
||||
async function preloadAllMedia(concurrency = 4) {
|
||||
const files = preloadOrder();
|
||||
if (!files.length) return;
|
||||
let done = 0;
|
||||
const total = files.length;
|
||||
const chip = preloadChip();
|
||||
const tick = () => { if (chip) chip.textContent = `⬇ caching clips ${done}/${total}`; };
|
||||
tick();
|
||||
let i = 0;
|
||||
async function worker() {
|
||||
while (i < files.length) {
|
||||
const file = files[i++];
|
||||
if (!mediaBlobs[file]) {
|
||||
try {
|
||||
const blob = await (await fetch(mediaUrl(file))).blob();
|
||||
mediaBlobs[file] = URL.createObjectURL(blob);
|
||||
} catch (_) { /* leave it to the network path on demand */ }
|
||||
}
|
||||
done++; tick();
|
||||
}
|
||||
}
|
||||
await Promise.all(Array.from({ length: Math.min(concurrency, files.length) }, worker));
|
||||
if (chip) chip.remove();
|
||||
}
|
||||
|
||||
function preloadChip() {
|
||||
let c = document.getElementById("preload-chip");
|
||||
if (!c) {
|
||||
c = document.createElement("div");
|
||||
c.id = "preload-chip";
|
||||
c.style.cssText = "position:fixed;bottom:8px;right:10px;z-index:9998;background:rgba(0,0,0,.55);" +
|
||||
"color:#9fe;font:11px/1.4 monospace;padding:4px 8px;border-radius:4px;pointer-events:none;";
|
||||
document.body.appendChild(c);
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
// Load the active scale's BASE footage into the video (the painterly canvas reads
|
||||
// its frames live). The Right knob no longer swaps the source — it only drives the
|
||||
@@ -182,6 +282,14 @@ function paintLoop() {
|
||||
gl.drawArrays(gl.TRIANGLES, 0, 3);
|
||||
paint.style.filter = busy ? "none" : gradeFilter;
|
||||
}
|
||||
// Animate per-frame: re-place keyframed tracks AND re-evaluate time-windowed
|
||||
// labels (a label enters/leaves frame as the loop plays) against playback time.
|
||||
const c = activeClip();
|
||||
if (lastOverlay && lastOverlay.level > 0 && c &&
|
||||
c.annotations.some((a) => (a.track && a.track.length) ||
|
||||
typeof a.appear === "number" || typeof a.disappear === "number")) {
|
||||
renderOverlay(lastOverlay.level, lastOverlay.intensity);
|
||||
}
|
||||
requestAnimationFrame(paintLoop);
|
||||
}
|
||||
|
||||
@@ -232,17 +340,72 @@ function chip(x, y, label, conf, kind, parent) {
|
||||
}
|
||||
}
|
||||
|
||||
// Interpolate an annotation's box at loop-normalized time t (0..1). Static labels
|
||||
// (no track) return their fixed box. Linear between sorted keyframes; clamps to ends.
|
||||
function boxAt(a, t) {
|
||||
if (!a.track || !a.track.length) return a.box;
|
||||
const ks = a.track;
|
||||
if (t <= ks[0].t) return ks[0].box;
|
||||
if (t >= ks[ks.length - 1].t) return ks[ks.length - 1].box;
|
||||
for (let i = 1; i < ks.length; i++) {
|
||||
if (t <= ks[i].t) {
|
||||
const a0 = ks[i - 1], a1 = ks[i];
|
||||
const f = (t - a0.t) / (a1.t - a0.t);
|
||||
return a0.box.map((v, j) => v + (a1.box[j] - v) * f);
|
||||
}
|
||||
}
|
||||
return ks[ks.length - 1].box;
|
||||
}
|
||||
|
||||
// Loop-normalized playback time of the base video (0..1), for track interpolation.
|
||||
function loopT() {
|
||||
return vid && vid.duration ? (vid.currentTime % vid.duration) / vid.duration : 0;
|
||||
}
|
||||
|
||||
// --- Progressive labels & emotions (content-pipeline §11.3 / §11.4) ---
|
||||
const clamp = (v, lo, hi) => Math.max(lo, Math.min(hi, v));
|
||||
|
||||
// The Left level at which a label first appears. From `salience` (1..4): high
|
||||
// salience appears early (first_level = 5 - salience), so raising Left brings in
|
||||
// lower-salience objects. Falls back to a legacy flat `min_level`, else 1.
|
||||
function firstLevel(a) {
|
||||
if (typeof a.salience === "number") return clamp(5 - a.salience, 1, 4);
|
||||
if (typeof a.min_level === "number") return a.min_level;
|
||||
return 1;
|
||||
}
|
||||
|
||||
// A tiered string escalates with a knob: a LIST is indexed by tier-1 (clamped to
|
||||
// the available tiers); a plain string is static (tier ignored — back-compat).
|
||||
function pickTier(value, tier) {
|
||||
if (Array.isArray(value)) return value[clamp(tier - 1, 0, value.length - 1)];
|
||||
return value;
|
||||
}
|
||||
function tierCount(value) { return Array.isArray(value) ? value.length : 1; }
|
||||
|
||||
// Time-windowed presence (§11.2): a label with `appear`/`disappear` (loop-norm t)
|
||||
// shows only while on screen. A `disappear < appear` window wraps across the loop
|
||||
// seam (visible when t >= appear OR t <= disappear). No window => always present.
|
||||
function inWindow(a, t) {
|
||||
const hasA = typeof a.appear === "number", hasD = typeof a.disappear === "number";
|
||||
if (!hasA && !hasD) return true;
|
||||
const ap = hasA ? a.appear : 0, dis = hasD ? a.disappear : 1;
|
||||
return ap <= dis ? (t >= ap && t <= dis) : (t >= ap || t <= dis);
|
||||
}
|
||||
|
||||
function renderOverlay(level, intensity) {
|
||||
const clip = activeClip();
|
||||
overlay.innerHTML = "";
|
||||
if (!clip || level <= 0) { overlay.style.opacity = "0"; return; }
|
||||
overlay.style.opacity = String(intensity);
|
||||
const strings = (clip.strings && clip.strings.en) || {};
|
||||
const t = loopT();
|
||||
let shown = 0;
|
||||
for (const a of clip.annotations) {
|
||||
if (a.min_level > level) continue;
|
||||
const first = firstLevel(a);
|
||||
if (level < first) continue; // salience gating: low Left = fewer/general labels
|
||||
if (!inWindow(a, t)) continue; // time-windowed: only while the object is on screen
|
||||
shown++;
|
||||
const [x, y, w, h] = a.box.map((n) => n * 100);
|
||||
const [x, y, w, h] = boxAt(a, t).map((n) => n * 100);
|
||||
const measure = a.key.startsWith("measure.");
|
||||
const kind = measure ? "measure" : "detect";
|
||||
reticle(x, y, w, h, "hud-reticle " + kind, overlay);
|
||||
@@ -252,7 +415,12 @@ function renderOverlay(level, intensity) {
|
||||
svg("line", { x1: cx - 1, y1: cyc, x2: cx + 1, y2: cyc, class: "hud-reticle measure" }, overlay);
|
||||
svg("line", { x1: cx, y1: cyc - 1, x2: cx, y2: cyc + 1, class: "hud-reticle measure" }, overlay);
|
||||
}
|
||||
chip(x, y, strings[a.key] || a.key, measure ? null : confidence(a.key), kind, overlay);
|
||||
// LEFT detail tier: a detection escalates general -> scientific+fact as Left
|
||||
// rises above where it first appeared; measurements stay single-tier.
|
||||
const raw = strings[a.key];
|
||||
const tier = measure ? 1 : clamp(level - first + 1, 1, tierCount(raw));
|
||||
const label = pickTier(raw !== undefined ? raw : a.key, tier);
|
||||
chip(x, y, label, measure ? null : confidence(a.key), kind, overlay);
|
||||
}
|
||||
// Global status tag — the machine announcing it is analyzing, escalating with Left.
|
||||
// Right-anchored at the frame edge; the plate is sized from the text's real bbox
|
||||
@@ -266,13 +434,14 @@ function renderOverlay(level, intensity) {
|
||||
height: b.height + pad * 1.2, rx: 0.4, class: "hud-chip-bg detect" });
|
||||
overlay.insertBefore(bg, st);
|
||||
}
|
||||
lastOverlay = { level, intensity };
|
||||
}
|
||||
|
||||
// Affect channel: soft, glowing emotion-words surfaced only when BOTH knobs are
|
||||
// up. `strength` = min(left, right); `intensity` is the layer opacity. Words are
|
||||
// placed at authored scene points (no boxes — feelings are scene-level) and read
|
||||
// softer than the clinical reticles — the dream leaking into the machine's read.
|
||||
function renderAffect(strength, intensity) {
|
||||
function renderAffect(strength, intensity, right) {
|
||||
const clip = activeClip();
|
||||
if (!affectLayer) return; // tolerate a stale page missing the affect layer
|
||||
affectLayer.innerHTML = "";
|
||||
@@ -283,14 +452,23 @@ function renderAffect(strength, intensity) {
|
||||
if (f.min_level > strength) continue;
|
||||
const [x, y] = f.at.map((n) => n * 100);
|
||||
const t = svg("text", { x, y, "text-anchor": "middle", class: "hud-affect" }, affectLayer);
|
||||
t.textContent = strings[f.key] || f.key;
|
||||
// RIGHT emotion tier: the vocabulary escalates basic -> compound as the Right
|
||||
// knob rises (appearance still gated by min(left,right) above).
|
||||
const raw = strings[f.key];
|
||||
t.textContent = pickTier(raw !== undefined ? raw : f.key, clamp(right || 0, 1, tierCount(raw)));
|
||||
}
|
||||
}
|
||||
|
||||
function renderScaleReadout() {
|
||||
if (!ring) return;
|
||||
const s = ring.scales[ringIndex];
|
||||
$("scale-name").textContent = `${s.title} (${ringIndex + 1}/${ring.scales.length})`;
|
||||
const poolN = (s.pool && s.pool.length) || 1;
|
||||
const member = (activeClip() && activeClip().title) || s.title;
|
||||
// scale id · the chosen pool member · position on the ring (pool size if >1)
|
||||
const poolTag = poolN > 1 ? ` · pool ${poolN}` : "";
|
||||
$("scale-name").textContent = `${s.id} · ${member} (${ringIndex + 1}/${ring.scales.length}${poolTag})`;
|
||||
renderDial();
|
||||
refreshDevClip(); // keep the Dev Mode pool picker + clip data in sync with the landing
|
||||
}
|
||||
|
||||
function controls() {
|
||||
@@ -322,7 +500,7 @@ async function update() {
|
||||
ensureClipMedia();
|
||||
applyVideoLook(data.plan.grade.tone, data.plan.dream.intensity);
|
||||
renderOverlay(data.plan.overlay.level, data.plan.overlay.intensity);
|
||||
renderAffect(data.plan.affect.strength, data.plan.affect.intensity);
|
||||
renderAffect(data.plan.affect.strength, data.plan.affect.intensity, data.plan.dream.strength);
|
||||
const b = document.getElementById("err-banner");
|
||||
if (b) b.remove(); // render succeeded — clear any prior error
|
||||
} catch (err) {
|
||||
@@ -381,7 +559,8 @@ async function advance(delta) {
|
||||
await playTransition(step.file, step.blended);
|
||||
}
|
||||
ringIndex = move.to_index;
|
||||
currentClipId = null; // force the target scale's base media to (re)load
|
||||
activeClipId = move.target_clip_id; // the randomly-picked pool member to load
|
||||
currentClipId = null; // force the target scale's base media to (re)load
|
||||
renderScaleReadout();
|
||||
} finally {
|
||||
busy = false;
|
||||
@@ -401,6 +580,248 @@ function onWheel(e) {
|
||||
}, 90);
|
||||
}
|
||||
|
||||
// --- Altitude knob: the endless rotary encoder, drawn as a turnable dial ---
|
||||
// cosmos (highest) sits at the top; turning CLOCKWISE descends through the scales
|
||||
// (cosmos → orbit → coast → reef → abyss) and past the deepest WRAPS back to the
|
||||
// top — the same endless ring the server already models. The drag accumulates a
|
||||
// rotation and commits whole detents on release (so a big spin becomes one fast
|
||||
// blended pass, exactly like the wheel); a tap on a label jumps to that scale.
|
||||
const dial = $("dial");
|
||||
const DIAL_C = 50, DIAL_LABEL_R = 41, DIAL_BODY_R = 27;
|
||||
let needleDeg = 0;
|
||||
let dialDrag = null; // {lastAng, accum, moved} while turning
|
||||
|
||||
function dialStep() { return ring && ring.scales.length ? 360 / ring.scales.length : 360; }
|
||||
|
||||
function _xy(r, deg) {
|
||||
const a = (deg - 90) * Math.PI / 180; // -90° so 0° points UP
|
||||
return [DIAL_C + r * Math.cos(a), DIAL_C + r * Math.sin(a)];
|
||||
}
|
||||
|
||||
function buildDial() {
|
||||
if (!dial || !ring) return;
|
||||
dial.innerHTML = "";
|
||||
const n = ring.scales.length, step = 360 / n;
|
||||
svg("circle", { cx: DIAL_C, cy: DIAL_C, r: DIAL_LABEL_R + 6, class: "dial-rim" }, dial);
|
||||
svg("circle", { cx: DIAL_C, cy: DIAL_C, r: DIAL_BODY_R, class: "dial-body" }, dial);
|
||||
for (let i = 0; i < n; i++) {
|
||||
const deg = i * step;
|
||||
const [tx0, ty0] = _xy(DIAL_BODY_R - 1.5, deg);
|
||||
const [tx1, ty1] = _xy(DIAL_BODY_R + 2.5, deg);
|
||||
svg("line", { x1: tx0, y1: ty0, x2: tx1, y2: ty1, class: "dial-tick" }, dial);
|
||||
const [lx, ly] = _xy(DIAL_LABEL_R, deg);
|
||||
const t = svg("text", {
|
||||
x: lx, y: ly, "text-anchor": "middle", "dominant-baseline": "central",
|
||||
class: "dial-label", "data-index": String(i),
|
||||
}, dial);
|
||||
t.textContent = ring.scales[i].id;
|
||||
}
|
||||
svg("text", { x: DIAL_C, y: DIAL_C - 7, "text-anchor": "middle",
|
||||
"dominant-baseline": "central", class: "dial-caption" }, dial)
|
||||
.textContent = "ALTITUDE";
|
||||
const needle = svg("g", { id: "needle", class: "dial-needle" }, dial);
|
||||
const tip = DIAL_C - (DIAL_BODY_R - 4);
|
||||
svg("polygon", { points: `${DIAL_C - 2.4},${DIAL_C} ${DIAL_C + 2.4},${DIAL_C} ${DIAL_C},${tip}` }, needle);
|
||||
svg("circle", { cx: DIAL_C, cy: DIAL_C, r: 3, class: "dial-hub" }, dial);
|
||||
renderDial();
|
||||
}
|
||||
|
||||
function setNeedle(deg) {
|
||||
const needle = $("needle");
|
||||
if (needle) needle.setAttribute("transform", `rotate(${deg} ${DIAL_C} ${DIAL_C})`);
|
||||
}
|
||||
|
||||
function renderDial() {
|
||||
if (!dial || !ring) return;
|
||||
needleDeg = ringIndex * dialStep();
|
||||
setNeedle(needleDeg);
|
||||
for (const el of dial.querySelectorAll(".dial-label")) {
|
||||
el.classList.toggle("active", +el.getAttribute("data-index") === ringIndex);
|
||||
}
|
||||
}
|
||||
|
||||
function dialAngle(e) {
|
||||
const r = dial.getBoundingClientRect();
|
||||
const dx = e.clientX - (r.left + r.width / 2);
|
||||
const dy = e.clientY - (r.top + r.height / 2);
|
||||
return (Math.atan2(dx, -dy) * 180 / Math.PI + 360) % 360; // 0 at top, clockwise +
|
||||
}
|
||||
function angDelta(a, b) {
|
||||
let d = a - b;
|
||||
while (d > 180) d -= 360;
|
||||
while (d < -180) d += 360;
|
||||
return d;
|
||||
}
|
||||
|
||||
function onDialDown(e) {
|
||||
if (busy || !ring || ring.scales.length < 2) return;
|
||||
e.preventDefault();
|
||||
dialDrag = { lastAng: dialAngle(e), accum: 0, moved: 0, target: e.target };
|
||||
}
|
||||
function onDialMove(e) {
|
||||
if (!dialDrag) return;
|
||||
const a = dialAngle(e);
|
||||
const d = angDelta(a, dialDrag.lastAng);
|
||||
dialDrag.lastAng = a;
|
||||
dialDrag.accum += d;
|
||||
dialDrag.moved += Math.abs(d);
|
||||
setNeedle(ringIndex * dialStep() + dialDrag.accum); // live feedback while turning
|
||||
}
|
||||
function onDialUp(e) {
|
||||
if (!dialDrag) return;
|
||||
const { accum, moved, target } = dialDrag;
|
||||
dialDrag = null;
|
||||
if (moved < 6) { // a tap, not a turn
|
||||
if (target && target.classList && target.classList.contains("dial-label")) {
|
||||
jumpToScale(+target.getAttribute("data-index"));
|
||||
}
|
||||
renderDial();
|
||||
return;
|
||||
}
|
||||
const detents = Math.round(accum / dialStep());
|
||||
if (detents) advance(detents); else renderDial(); // snap back if it didn't cross a detent
|
||||
}
|
||||
|
||||
// Click a label → travel the SHORTEST signed way around the ring to that scale.
|
||||
function jumpToScale(idx) {
|
||||
if (!ring) return;
|
||||
const n = ring.scales.length;
|
||||
let d = (idx - ringIndex) % n;
|
||||
if (d > n / 2) d -= n;
|
||||
if (d < -n / 2) d += n;
|
||||
if (d) advance(d);
|
||||
}
|
||||
|
||||
// --- Dev Mode: pool picker + live analysis, all under one toggle ---
|
||||
// Off by default, state persisted in localStorage. Everything here is read from
|
||||
// data the renderer already has (clips, ring, the alteration response, the preload
|
||||
// cache) — no server endpoints. Editing labels stays in /author.html.
|
||||
const DEV_KEY = "hef.devMode";
|
||||
let devMode = false;
|
||||
let devStatsTimer = null;
|
||||
|
||||
const escapeHtml = (s) => String(s).replace(/[&<>"]/g,
|
||||
(c) => ({ "&": "&", "<": "<", ">": ">", '"': """ }[c]));
|
||||
|
||||
function applyDevVisibility() {
|
||||
const panel = $("dev-panel");
|
||||
if (panel) panel.classList.toggle("hidden", !devMode);
|
||||
if (devMode) { refreshDevClip(); renderDevStats(); }
|
||||
}
|
||||
|
||||
function initDev() {
|
||||
const toggle = $("dev-mode");
|
||||
if (!toggle) return;
|
||||
try { devMode = localStorage.getItem(DEV_KEY) === "1"; } catch (_) { devMode = false; }
|
||||
toggle.checked = devMode;
|
||||
toggle.addEventListener("change", () => {
|
||||
devMode = toggle.checked;
|
||||
try { localStorage.setItem(DEV_KEY, devMode ? "1" : "0"); } catch (_) {}
|
||||
applyDevVisibility();
|
||||
});
|
||||
$("pool-select").addEventListener("change", onPoolSelect);
|
||||
$("pool-reroll").addEventListener("click", reroll);
|
||||
// Cache + playback stats are live-ish; cheap to repaint twice a second when shown.
|
||||
devStatsTimer = setInterval(() => { if (devMode) renderDevStats(); }, 500);
|
||||
applyDevVisibility();
|
||||
}
|
||||
|
||||
// Rebuild the pool picker for the current scale and repaint the clip data. Cheap
|
||||
// (pools are <= a handful), so just rebuild on every landing/pick.
|
||||
function refreshDevClip() {
|
||||
if (!devMode) return;
|
||||
const sel = $("pool-select");
|
||||
const scale = ring && ring.scales[ringIndex];
|
||||
if (sel && scale) {
|
||||
const pool = (scale.pool && scale.pool.length) ? scale.pool : [{ clip_id: scale.clip_id, title: scale.title }];
|
||||
sel.innerHTML = "";
|
||||
for (const m of pool) {
|
||||
const o = document.createElement("option");
|
||||
o.value = m.clip_id;
|
||||
o.textContent = `${m.clip_id} · ${m.title || (clipsById[m.clip_id] || {}).title || ""}`;
|
||||
sel.appendChild(o);
|
||||
}
|
||||
if (activeClipId) sel.value = activeClipId;
|
||||
sel.disabled = pool.length < 2;
|
||||
}
|
||||
renderDevMeta();
|
||||
renderDevAnno();
|
||||
}
|
||||
|
||||
function renderDevMeta() {
|
||||
const dl = $("clip-meta");
|
||||
const c = activeClip();
|
||||
if (!dl) return;
|
||||
if (!c) { dl.innerHTML = "<dt>—</dt><dd>no clip</dd>"; return; }
|
||||
const rows = [["id", c.id], ["title", c.title], ["source", c.source],
|
||||
["license", c.license], ["base_file", c.base_file]];
|
||||
dl.innerHTML = rows.map(([k, v]) =>
|
||||
`<dt>${escapeHtml(k)}</dt><dd>${escapeHtml(v != null ? v : "—")}</dd>`).join("");
|
||||
}
|
||||
|
||||
function renderDevAnno() {
|
||||
const box = $("clip-anno");
|
||||
const c = activeClip();
|
||||
if (!box) return;
|
||||
const strings = (c && c.strings && c.strings.en) || {};
|
||||
const win = (a) => (typeof a.appear === "number" || typeof a.disappear === "number")
|
||||
? ` win ${(a.appear ?? 0).toFixed(2)}–${(a.disappear ?? 1).toFixed(2)}` : "";
|
||||
const rows = [];
|
||||
for (const a of (c && c.annotations) || []) {
|
||||
const measure = a.key.startsWith("measure.");
|
||||
const sal = typeof a.salience === "number" ? `sal ${a.salience}`
|
||||
: typeof a.min_level === "number" ? `min L${a.min_level}` : "sal 1";
|
||||
const tiers = tierCount(strings[a.key]);
|
||||
rows.push(`<div class="anno-row"><span class="anno-key ${measure ? "measure" : ""}">${escapeHtml(a.key)}</span>` +
|
||||
`<span class="anno-meta"> · ${sal} · ${tiers} tier${tiers > 1 ? "s" : ""}${win(a)}</span></div>`);
|
||||
}
|
||||
for (const f of (c && c.affect) || []) {
|
||||
const tiers = tierCount(strings[f.key]);
|
||||
rows.push(`<div class="anno-row"><span class="anno-key affect">${escapeHtml(f.key)}</span>` +
|
||||
`<span class="anno-meta"> · min L${f.min_level ?? 1} · ${tiers} tier${tiers > 1 ? "s" : ""}</span></div>`);
|
||||
}
|
||||
box.innerHTML = rows.length ? rows.join("") : `<div class="anno-empty">no annotations or affect</div>`;
|
||||
}
|
||||
|
||||
function renderDevStats() {
|
||||
const dl = $("dev-stats");
|
||||
if (!dl) return;
|
||||
const c = activeClip();
|
||||
const total = preloadList().length;
|
||||
const cached = Object.keys(mediaBlobs).length;
|
||||
const fromMem = !!(c && mediaBlobs[c.base_file]);
|
||||
const res = vid.videoWidth ? `${vid.videoWidth}×${vid.videoHeight}` : "—";
|
||||
const dur = vid.duration ? `${vid.duration.toFixed(1)}s` : "—";
|
||||
const loop = vid.duration ? `${Math.round(loopT() * 100)}%` : "—";
|
||||
dl.innerHTML =
|
||||
`<dt>cache</dt><dd>${cached}/${total} cached</dd>` +
|
||||
`<dt>active source</dt><dd class="${fromMem ? "mem" : "net"}">${fromMem ? "memory (blob)" : "network"}</dd>` +
|
||||
`<dt>resolution</dt><dd>${res}</dd>` +
|
||||
`<dt>duration</dt><dd>${dur}</dd>` +
|
||||
`<dt>loop pos</dt><dd>${loop}</dd>`;
|
||||
}
|
||||
|
||||
// Pick a specific pool member (Dev Mode override of the random landing pick).
|
||||
function onPoolSelect() {
|
||||
const id = $("pool-select").value;
|
||||
if (!id || id === activeClipId) return;
|
||||
activeClipId = id;
|
||||
currentClipId = null; // force the base media to reload
|
||||
renderScaleReadout(); // refresh scale name + dev clip data
|
||||
update(); // reload media + plan
|
||||
}
|
||||
|
||||
// Re-roll: a fresh server-canonical random pick for the current scale.
|
||||
async function reroll() {
|
||||
if (busy) return;
|
||||
const id = await pickRandomMember();
|
||||
if (!id) return;
|
||||
activeClipId = id;
|
||||
currentClipId = null;
|
||||
renderScaleReadout();
|
||||
update();
|
||||
}
|
||||
|
||||
// Dev live-reload: poll the asset version and reload when it changes, so an open
|
||||
// tab never keeps running a stale renderer while we iterate (the readout updates
|
||||
// from the live API and masks it otherwise). Reloads on my edits AND on a server
|
||||
@@ -420,12 +841,19 @@ async function main() {
|
||||
devLiveReload();
|
||||
try { initPaint(); } catch (e) { paintOK = false; paint.style.display = "none"; showError("WebGL init: " + e.message); }
|
||||
await loadData();
|
||||
await landScale(); // pick the initial scale's pool member before first render
|
||||
preloadAllMedia(); // background: cache every clip in memory for instant altitude swaps
|
||||
buildDial(); // draw the altitude knob from the ring's scales
|
||||
initDev(); // wire the Dev Mode toggle + pool picker (reads persisted state)
|
||||
renderScaleReadout();
|
||||
for (const id of ["content", "left", "right", "mood"]) {
|
||||
$(id).addEventListener("input", debounced);
|
||||
}
|
||||
$("zoom-in").addEventListener("click", () => advance(1));
|
||||
$("zoom-out").addEventListener("click", () => advance(-1));
|
||||
// Altitude knob: drag to turn (commit detents on release), scroll to step, tap a label to jump.
|
||||
dial.addEventListener("pointerdown", onDialDown);
|
||||
window.addEventListener("pointermove", onDialMove);
|
||||
window.addEventListener("pointerup", onDialUp);
|
||||
dial.addEventListener("wheel", onWheel, { passive: false });
|
||||
$("stage").addEventListener("wheel", onWheel, { passive: false });
|
||||
update();
|
||||
}
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
/* Author-mode-only styling (content-pipeline §11.5). Reuses style.css for the
|
||||
shared stage/panel chrome; this adds the box-draw layer + editor widgets. */
|
||||
|
||||
.author #draw {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
cursor: crosshair;
|
||||
}
|
||||
.author #draw .seed-box {
|
||||
fill: rgba(80, 200, 255, 0.12);
|
||||
stroke: #50c8ff;
|
||||
stroke-width: 0.4;
|
||||
vector-effect: non-scaling-stroke;
|
||||
}
|
||||
.author #draw .track-box {
|
||||
fill: rgba(120, 255, 160, 0.10);
|
||||
stroke: #78ffa0;
|
||||
stroke-width: 0.4;
|
||||
stroke-dasharray: 1.5 1;
|
||||
vector-effect: non-scaling-stroke;
|
||||
}
|
||||
.author #draw .affect-dot {
|
||||
fill: rgba(190, 150, 255, 0.85);
|
||||
stroke: #fff;
|
||||
stroke-width: 0.25;
|
||||
vector-effect: non-scaling-stroke;
|
||||
}
|
||||
|
||||
.author .scrub {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-top: 8px;
|
||||
}
|
||||
.author .scrub input[type="range"] { flex: 1; }
|
||||
.author .mono { font-family: ui-monospace, monospace; font-size: 12px; opacity: 0.85; }
|
||||
|
||||
.author .row { display: flex; gap: 8px; margin-top: 6px; flex-wrap: wrap; }
|
||||
.author fieldset label { display: block; margin: 4px 0; font-size: 13px; }
|
||||
.author fieldset input[type="text"],
|
||||
.author fieldset input[type="number"] { width: 100%; box-sizing: border-box; }
|
||||
.author fieldset input[type="number"] { width: 5em; }
|
||||
|
||||
.author ul.authored { list-style: none; padding: 0; margin: 6px 0; }
|
||||
.author ul.authored li {
|
||||
font-size: 12px;
|
||||
padding: 3px 6px;
|
||||
margin: 2px 0;
|
||||
background: rgba(255, 255, 255, 0.05);
|
||||
border-radius: 3px;
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
gap: 6px;
|
||||
}
|
||||
.author ul.authored .rm {
|
||||
background: none;
|
||||
border: none;
|
||||
color: #f88;
|
||||
cursor: pointer;
|
||||
font-size: 12px;
|
||||
}
|
||||
.author header .hint { font-size: 12px; opacity: 0.8; max-width: 70ch; }
|
||||
@@ -0,0 +1,79 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>HEF — Label Author Mode</title>
|
||||
<link rel="stylesheet" href="/style.css" />
|
||||
<link rel="stylesheet" href="/author.css" />
|
||||
</head>
|
||||
<body class="author">
|
||||
<header><h1>HEF — Label Author Mode</h1>
|
||||
<p class="hint">Scrub a pool clip · drag a box on a subject · run the tracker · author the tiers · save to the manifest. Geometry is propagated; <strong>semantics are hand-authored</strong> (content-pipeline §11.5). <a href="/">← back to preview</a></p>
|
||||
</header>
|
||||
<main>
|
||||
<section class="stage" id="stage">
|
||||
<div class="screen">
|
||||
<video id="vid" muted playsinline></video>
|
||||
<svg id="draw" viewBox="0 0 100 100" preserveAspectRatio="none"></svg>
|
||||
</div>
|
||||
<div class="scrub">
|
||||
<button type="button" id="play">▶/⏸</button>
|
||||
<input type="range" id="seek" min="0" max="1000" value="0" />
|
||||
<span id="time" class="mono">0.00</span>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="panel">
|
||||
<fieldset>
|
||||
<legend>Clip</legend>
|
||||
<select id="clip"></select>
|
||||
<p class="hint" id="clip-meta">—</p>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Seed box (drag on the video)</legend>
|
||||
<p class="mono" id="seed-readout">no box — drag on the stage</p>
|
||||
<label>label key <input type="text" id="ann-key" placeholder="detected.jelly" /></label>
|
||||
<label>salience (1–4; 4 = shows first/at low Left)
|
||||
<input type="number" id="ann-salience" min="1" max="4" value="4" /></label>
|
||||
<label>tier 1 — general <input type="text" id="t1" placeholder="jelly" /></label>
|
||||
<label>tier 2 — specific <input type="text" id="t2" placeholder="comb jelly" /></label>
|
||||
<label>tier 3 — scientific <input type="text" id="t3" placeholder="Ctenophora" /></label>
|
||||
<label>tier 4 — +fact <input type="text" id="t4" placeholder="Ctenophora · beats rows of cilia" /></label>
|
||||
<label>keyframes <input type="number" id="ann-keyframes" min="2" max="20" value="5" /></label>
|
||||
<div class="row">
|
||||
<button type="button" id="run-track">Run tracker → track</button>
|
||||
<button type="button" id="add-static">Add as static box</button>
|
||||
</div>
|
||||
<p class="mono" id="track-readout"></p>
|
||||
<button type="button" id="add-ann" disabled>+ Add label</button>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Affect anchor (click the stage to place)</legend>
|
||||
<p class="mono" id="affect-readout">no point — shift-click the stage</p>
|
||||
<label>feel key <input type="text" id="aff-key" placeholder="feel.unease" /></label>
|
||||
<label>min strength (1–4) <input type="number" id="aff-min" min="1" max="4" value="1" /></label>
|
||||
<label>tier 1 — basic <input type="text" id="a1" placeholder="uh" /></label>
|
||||
<label>tier 2 <input type="text" id="a2" placeholder="unease" /></label>
|
||||
<label>tier 3 <input type="text" id="a3" placeholder="disquiet" /></label>
|
||||
<label>tier 4 — compound <input type="text" id="a4" placeholder="a creeping disquiet" /></label>
|
||||
<button type="button" id="add-aff" disabled>+ Add emotion</button>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Authored for this clip</legend>
|
||||
<ul id="ann-list" class="authored"></ul>
|
||||
<ul id="aff-list" class="authored"></ul>
|
||||
<div class="row">
|
||||
<button type="button" id="load-existing">Load existing</button>
|
||||
<button type="button" id="save">Save to manifest</button>
|
||||
</div>
|
||||
<p class="mono" id="save-readout"></p>
|
||||
</fieldset>
|
||||
</section>
|
||||
</main>
|
||||
<script src="/author.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,254 @@
|
||||
// Label author mode (content-pipeline §11.5): scrub a pool clip, drag a seed box
|
||||
// on a subject, run the classical tracker to propagate a keyframed track, author
|
||||
// the LEFT detail tiers + RIGHT emotion tiers, and save the entry to the manifest.
|
||||
// Geometry comes from the tracker; SEMANTICS (keys, strings, tiers) are hand-typed.
|
||||
const $ = (id) => document.getElementById(id);
|
||||
const vid = $("vid"), draw = $("draw");
|
||||
const SVGNS = "http://www.w3.org/2000/svg";
|
||||
|
||||
let clips = [];
|
||||
let currentClip = null;
|
||||
let seedBox = null; // [x,y,w,h] normalized, from a drag
|
||||
let pendingTrack = null; // {track, appear, disappear} from the tracker (or static)
|
||||
let affectPoint = null; // [x,y] normalized, from a shift-click
|
||||
let annotations = []; // authored annotation dicts (carry _tiers)
|
||||
let affect = []; // authored affect dicts (carry _tiers)
|
||||
|
||||
function svg(tag, attrs, parent) {
|
||||
const el = document.createElementNS(SVGNS, tag);
|
||||
for (const k in attrs) el.setAttribute(k, attrs[k]);
|
||||
if (parent) parent.appendChild(el);
|
||||
return el;
|
||||
}
|
||||
const mono = (n) => Number(n).toFixed(3);
|
||||
|
||||
async function loadClips() {
|
||||
clips = (await (await fetch("/api/clips")).json()).clips || [];
|
||||
const sel = $("clip");
|
||||
sel.innerHTML = "";
|
||||
for (const c of clips) {
|
||||
const o = document.createElement("option");
|
||||
o.value = c.id; o.textContent = c.id;
|
||||
sel.appendChild(o);
|
||||
}
|
||||
if (clips.length) selectClip(clips[0].id);
|
||||
}
|
||||
|
||||
function selectClip(id) {
|
||||
currentClip = clips.find((c) => c.id === id) || null;
|
||||
if (!currentClip) return;
|
||||
$("clip-meta").textContent = `${currentClip.title} — ${currentClip.license}`;
|
||||
vid.src = "/media/" + currentClip.base_file;
|
||||
vid.currentTime = 0;
|
||||
annotations = []; affect = []; seedBox = null; pendingTrack = null; affectPoint = null;
|
||||
renderLists(); renderDrawLayer();
|
||||
$("seed-readout").textContent = "no box — drag on the stage";
|
||||
$("track-readout").textContent = "";
|
||||
$("affect-readout").textContent = "no point — shift-click the stage";
|
||||
}
|
||||
|
||||
// --- scrub ---
|
||||
function seekT() { return vid.duration ? vid.currentTime / vid.duration : 0; }
|
||||
$("play").addEventListener("click", () => { vid.paused ? vid.play() : vid.pause(); });
|
||||
$("seek").addEventListener("input", () => {
|
||||
if (vid.duration) vid.currentTime = (+$("seek").value / 1000) * vid.duration;
|
||||
});
|
||||
vid.addEventListener("timeupdate", () => {
|
||||
if (!vid.duration) return;
|
||||
$("seek").value = String(Math.round(seekT() * 1000));
|
||||
$("time").textContent = seekT().toFixed(3);
|
||||
});
|
||||
|
||||
// --- draw a seed box / place an affect point on the stage ---
|
||||
function evtNorm(e) {
|
||||
const r = draw.getBoundingClientRect();
|
||||
return [(e.clientX - r.left) / r.width, (e.clientY - r.top) / r.height];
|
||||
}
|
||||
let dragStart = null;
|
||||
draw.addEventListener("mousedown", (e) => {
|
||||
if (e.shiftKey) { // shift-click = affect anchor
|
||||
affectPoint = evtNorm(e).map((v) => Math.min(Math.max(v, 0), 1));
|
||||
$("affect-readout").textContent = `affect @ ${affectPoint.map(mono).join(", ")}`;
|
||||
$("add-aff").disabled = false;
|
||||
renderDrawLayer();
|
||||
return;
|
||||
}
|
||||
dragStart = evtNorm(e);
|
||||
});
|
||||
draw.addEventListener("mousemove", (e) => {
|
||||
if (!dragStart) return;
|
||||
seedBox = boxFrom(dragStart, evtNorm(e));
|
||||
renderDrawLayer();
|
||||
});
|
||||
window.addEventListener("mouseup", (e) => {
|
||||
if (!dragStart) return;
|
||||
seedBox = boxFrom(dragStart, evtNorm(e));
|
||||
dragStart = null;
|
||||
pendingTrack = null;
|
||||
$("seed-readout").textContent = `seed [${seedBox.map(mono).join(", ")}] @ t=${seekT().toFixed(3)}`;
|
||||
$("add-ann").disabled = false;
|
||||
renderDrawLayer();
|
||||
});
|
||||
function boxFrom(a, b) {
|
||||
const x = Math.min(a[0], b[0]), y = Math.min(a[1], b[1]);
|
||||
const w = Math.abs(b[0] - a[0]), h = Math.abs(b[1] - a[1]);
|
||||
const cl = (v) => Math.min(Math.max(v, 0), 1);
|
||||
return [cl(x), cl(y), Math.min(w, 1 - cl(x)), Math.min(h, 1 - cl(y))];
|
||||
}
|
||||
|
||||
function renderDrawLayer() {
|
||||
draw.innerHTML = "";
|
||||
if (seedBox) {
|
||||
const [x, y, w, h] = seedBox.map((n) => n * 100);
|
||||
svg("rect", { x, y, width: w, height: h, class: "seed-box" }, draw);
|
||||
}
|
||||
if (pendingTrack && pendingTrack.track && vid.duration) {
|
||||
const b = boxAt(pendingTrack.track, seekT());
|
||||
if (b) {
|
||||
const [x, y, w, h] = b.map((n) => n * 100);
|
||||
svg("rect", { x, y, width: w, height: h, class: "track-box" }, draw);
|
||||
}
|
||||
}
|
||||
if (affectPoint) {
|
||||
const [x, y] = affectPoint.map((n) => n * 100);
|
||||
svg("circle", { cx: x, cy: y, r: 1.2, class: "affect-dot" }, draw);
|
||||
}
|
||||
}
|
||||
// Interpolate a track box at loop-normalized t (mirrors the preview renderer).
|
||||
function boxAt(track, t) {
|
||||
if (!track || !track.length) return null;
|
||||
if (t <= track[0].t) return track[0].box;
|
||||
if (t >= track[track.length - 1].t) return track[track.length - 1].box;
|
||||
for (let i = 1; i < track.length; i++) {
|
||||
if (t <= track[i].t) {
|
||||
const a = track[i - 1], b = track[i], f = (t - a.t) / (b.t - a.t);
|
||||
return a.box.map((v, j) => v + (b.box[j] - v) * f);
|
||||
}
|
||||
}
|
||||
return track[track.length - 1].box;
|
||||
}
|
||||
// Animate the pending track preview while playing.
|
||||
function previewLoop() { renderDrawLayer(); requestAnimationFrame(previewLoop); }
|
||||
|
||||
// --- run the tracker on the seed box ---
|
||||
$("run-track").addEventListener("click", async () => {
|
||||
if (!seedBox || !currentClip) { $("track-readout").textContent = "draw a seed box first"; return; }
|
||||
const key = $("ann-key").value.trim() || "detected.object";
|
||||
$("track-readout").textContent = "tracking…";
|
||||
const resp = await fetch("/api/author/track", {
|
||||
method: "POST", headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
clip_id: currentClip.id, key, seed_box: seedBox, seed_t: seekT(),
|
||||
salience: +$("ann-salience").value, n_keyframes: +$("ann-keyframes").value,
|
||||
}),
|
||||
});
|
||||
if (!resp.ok) { $("track-readout").textContent = "tracker error " + resp.status; return; }
|
||||
const ann = await resp.json();
|
||||
pendingTrack = { track: ann.track, appear: ann.appear, disappear: ann.disappear };
|
||||
$("track-readout").textContent =
|
||||
`track: ${ann.track.length} keyframes · window ${mono(ann.appear)}–${mono(ann.disappear)}`;
|
||||
$("add-ann").disabled = false;
|
||||
});
|
||||
|
||||
// Add the current seed as a STATIC (untracked) box at the scrub position.
|
||||
$("add-static").addEventListener("click", () => {
|
||||
if (!seedBox) return;
|
||||
pendingTrack = { static: true, box: seedBox.slice() };
|
||||
$("track-readout").textContent = "static box (no track) at the drawn position";
|
||||
$("add-ann").disabled = false;
|
||||
});
|
||||
|
||||
function tiers(ids) {
|
||||
const vals = ids.map((id) => $(id).value.trim());
|
||||
while (vals.length && !vals[vals.length - 1]) vals.pop(); // drop empty trailing tiers
|
||||
return vals.length ? vals : null;
|
||||
}
|
||||
|
||||
$("add-ann").addEventListener("click", () => {
|
||||
const key = $("ann-key").value.trim();
|
||||
if (!key) { $("track-readout").textContent = "a label key is required"; return; }
|
||||
const t = tiers(["t1", "t2", "t3", "t4"]);
|
||||
const ann = { key, salience: +$("ann-salience").value, _tiers: t || key };
|
||||
if (pendingTrack && pendingTrack.track) {
|
||||
ann.track = pendingTrack.track;
|
||||
ann.appear = pendingTrack.appear; ann.disappear = pendingTrack.disappear;
|
||||
} else if (pendingTrack && pendingTrack.static) {
|
||||
ann.box = pendingTrack.box;
|
||||
} else if (seedBox) {
|
||||
ann.box = seedBox.slice();
|
||||
} else { $("track-readout").textContent = "draw + (optionally) track a box first"; return; }
|
||||
annotations.push(ann);
|
||||
seedBox = null; pendingTrack = null; $("add-ann").disabled = true;
|
||||
renderLists(); renderDrawLayer();
|
||||
});
|
||||
|
||||
$("add-aff").addEventListener("click", () => {
|
||||
const key = $("aff-key").value.trim();
|
||||
if (!key || !affectPoint) return;
|
||||
const t = tiers(["a1", "a2", "a3", "a4"]);
|
||||
affect.push({ key, at: affectPoint.slice(), min_level: +$("aff-min").value, _tiers: t || key });
|
||||
affectPoint = null; $("add-aff").disabled = true;
|
||||
renderLists(); renderDrawLayer();
|
||||
});
|
||||
|
||||
function renderLists() {
|
||||
const al = $("ann-list"); al.innerHTML = "";
|
||||
annotations.forEach((a, i) => {
|
||||
const li = document.createElement("li");
|
||||
const kind = a.track ? `track ${a.track.length}kf ${mono(a.appear)}–${mono(a.disappear)}` : "static";
|
||||
const tip = Array.isArray(a._tiers) ? a._tiers.join(" → ") : a._tiers;
|
||||
li.textContent = `${a.key} (s${a.salience}, ${kind}) — ${tip}`;
|
||||
li.appendChild(rm(() => { annotations.splice(i, 1); renderLists(); }));
|
||||
al.appendChild(li);
|
||||
});
|
||||
const fl = $("aff-list"); fl.innerHTML = "";
|
||||
affect.forEach((f, i) => {
|
||||
const li = document.createElement("li");
|
||||
const tip = Array.isArray(f._tiers) ? f._tiers.join(" → ") : f._tiers;
|
||||
li.textContent = `${f.key} (min ${f.min_level}) — ${tip}`;
|
||||
li.appendChild(rm(() => { affect.splice(i, 1); renderLists(); }));
|
||||
fl.appendChild(li);
|
||||
});
|
||||
}
|
||||
function rm(fn) {
|
||||
const b = document.createElement("button");
|
||||
b.textContent = "✕"; b.className = "rm"; b.onclick = fn;
|
||||
return b;
|
||||
}
|
||||
|
||||
// Load whatever is already authored for this clip into the editor.
|
||||
$("load-existing").addEventListener("click", () => {
|
||||
if (!currentClip) return;
|
||||
const strings = (currentClip.strings && currentClip.strings.en) || {};
|
||||
annotations = (currentClip.annotations || []).map((a) => ({ ...a, _tiers: strings[a.key] || a.key }));
|
||||
affect = (currentClip.affect || []).map((f) => ({ ...f, _tiers: strings[f.key] || f.key }));
|
||||
renderLists();
|
||||
});
|
||||
|
||||
// Assemble strings.en from each entry's _tiers and POST to the manifest.
|
||||
$("save").addEventListener("click", async () => {
|
||||
if (!currentClip) return;
|
||||
const en = {};
|
||||
const strip = (x) => { const { _tiers, ...rest } = x; return rest; };
|
||||
const anns = annotations.map((a) => { en[a.key] = a._tiers; return strip(a); });
|
||||
const affs = affect.map((f) => { en[f.key] = f._tiers; return strip(f); });
|
||||
const resp = await fetch("/api/author/clip", {
|
||||
method: "POST", headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ clip_id: currentClip.id, annotations: anns, affect: affs, strings: { en } }),
|
||||
});
|
||||
$("save-readout").textContent = resp.ok
|
||||
? `saved ${anns.length} labels + ${affs.length} emotions to the manifest ✓`
|
||||
: "save error " + resp.status;
|
||||
if (resp.ok) { // refresh the in-memory clip copy
|
||||
clips = (await (await fetch("/api/clips")).json()).clips || [];
|
||||
currentClip = clips.find((c) => c.id === currentClip.id) || currentClip;
|
||||
}
|
||||
});
|
||||
|
||||
$("clip").addEventListener("change", (e) => selectClip(e.target.value));
|
||||
|
||||
async function main() {
|
||||
await loadClips();
|
||||
requestAnimationFrame(previewLoop);
|
||||
}
|
||||
main();
|
||||
+38
-10
@@ -35,13 +35,12 @@
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Scale ring (endless encoder)</legend>
|
||||
<div class="ring-control">
|
||||
<button type="button" id="zoom-out" title="zoom out (toward cosmos)">⊖ out</button>
|
||||
<span id="scale-name" class="scale-name">—</span>
|
||||
<button type="button" id="zoom-in" title="zoom in (toward microscopic)">in ⊕</button>
|
||||
<legend>Altitude</legend>
|
||||
<div class="dial-wrap">
|
||||
<svg id="dial" viewBox="0 0 100 100" aria-label="Altitude knob (turn to change scale)"></svg>
|
||||
</div>
|
||||
<p class="hint">Relative, endless — wraps past the smallest back to the largest. Scroll the stage too.</p>
|
||||
<span id="scale-name" class="scale-name">—</span>
|
||||
<p class="hint">Turn the knob (drag it, or scroll) to change altitude — endless: past the deepest it wraps back up to the highest. Click a label to jump there.</p>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
@@ -51,11 +50,40 @@
|
||||
<label>Mood — dark ◀ 0 ▶ light <input type="range" id="mood" min="-4" max="4" value="0" /></label>
|
||||
</fieldset>
|
||||
|
||||
<!-- Dev Mode: a single toggle; everything dev (pool picker + analysis) lives below it. -->
|
||||
<label class="dev-switch" for="dev-mode">
|
||||
<input type="checkbox" id="dev-mode" />
|
||||
<span class="dev-switch-track"><span class="dev-switch-thumb"></span></span>
|
||||
<span class="dev-switch-label">Dev Mode</span>
|
||||
</label>
|
||||
|
||||
<fieldset>
|
||||
<legend>RenderPlan readout</legend>
|
||||
<pre id="readout">—</pre>
|
||||
</fieldset>
|
||||
<div id="dev-panel" class="dev-panel hidden">
|
||||
<fieldset>
|
||||
<legend>Pool — pick a clip</legend>
|
||||
<select id="pool-select" aria-label="Choose a clip from this altitude's pool"></select>
|
||||
<button type="button" id="pool-reroll" class="dev-btn">🎲 Re-roll random</button>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Active clip</legend>
|
||||
<dl id="clip-meta" class="dev-dl"></dl>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Annotations & affect</legend>
|
||||
<div id="clip-anno" class="dev-anno"></div>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>RenderPlan</legend>
|
||||
<pre id="readout">—</pre>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Cache & playback</legend>
|
||||
<dl id="dev-stats" class="dev-dl"></dl>
|
||||
</fieldset>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
<script src="/app.js"></script>
|
||||
|
||||
@@ -48,10 +48,55 @@ label { display: block; margin: 0.4rem 0; }
|
||||
input[type=range], select { width: 100%; }
|
||||
#readout { background: #000; padding: 0.5rem; border-radius: 4px; font-size: 12px;
|
||||
white-space: pre-wrap; max-height: 240px; overflow: auto; }
|
||||
.ring-control { display: flex; align-items: center; gap: 0.4rem; }
|
||||
.ring-control button { flex: 0 0 auto; background: #1a2436; color: #9af;
|
||||
border: 1px solid #345; border-radius: 4px; padding: 0.3rem 0.5rem;
|
||||
cursor: pointer; font-size: 13px; }
|
||||
.ring-control button:hover { background: #243352; }
|
||||
.scale-name { flex: 1 1 auto; text-align: center; font-size: 12px; color: #cde; }
|
||||
/* Altitude knob */
|
||||
.dial-wrap { display: flex; justify-content: center; padding: 0.3rem 0 0.1rem; }
|
||||
#dial { width: 190px; height: 190px; touch-action: none; cursor: grab; user-select: none; }
|
||||
#dial:active { cursor: grabbing; }
|
||||
.dial-rim { fill: #0d1320; stroke: #243352; stroke-width: 1.2; }
|
||||
.dial-body { fill: #16203200; stroke: #2c3c5c; stroke-width: 1; }
|
||||
.dial-tick { stroke: #3a4d70; stroke-width: 0.8; }
|
||||
.dial-label { fill: #789ac0; font-size: 6px; font-family: ui-monospace, monospace;
|
||||
letter-spacing: 0.2px; cursor: pointer; }
|
||||
.dial-label:hover { fill: #cde; }
|
||||
.dial-label.active { fill: #9cf; font-weight: 700; }
|
||||
.dial-caption { fill: #4d6184; font-size: 4.4px; letter-spacing: 1.2px;
|
||||
font-family: ui-monospace, monospace; }
|
||||
.dial-needle { fill: #9cf; }
|
||||
.dial-needle polygon { filter: drop-shadow(0 0 1px #9cf); }
|
||||
.dial-hub { fill: #2c3c5c; stroke: #9cf; stroke-width: 0.6; }
|
||||
.scale-name { display: block; text-align: center; font-size: 12px; color: #cde; }
|
||||
.hint { margin: 0.3rem 0 0; font-size: 11px; color: #789; }
|
||||
|
||||
/* --- Dev Mode --- a single switch; all dev controls/data live in #dev-panel below it. */
|
||||
.dev-switch { display: flex; align-items: center; gap: 0.5rem; cursor: pointer;
|
||||
padding: 0.5rem 0.2rem 0.2rem; border-top: 1px solid #222; user-select: none; }
|
||||
.dev-switch input { position: absolute; opacity: 0; width: 0; height: 0; }
|
||||
.dev-switch-track { position: relative; width: 34px; height: 18px; border-radius: 9px;
|
||||
background: #2a2a2a; border: 1px solid #444; transition: background 0.15s; }
|
||||
.dev-switch-thumb { position: absolute; top: 1px; left: 1px; width: 14px; height: 14px;
|
||||
border-radius: 50%; background: #888; transition: transform 0.15s, background 0.15s; }
|
||||
.dev-switch input:checked + .dev-switch-track { background: #1d3a2a; border-color: #2e6; }
|
||||
.dev-switch input:checked + .dev-switch-track .dev-switch-thumb { transform: translateX(16px); background: #4e9; }
|
||||
.dev-switch input:focus-visible + .dev-switch-track { outline: 2px solid #9af; outline-offset: 1px; }
|
||||
.dev-switch-label { color: #9af; font-weight: 600; letter-spacing: 0.3px; }
|
||||
.dev-panel { display: flex; flex-direction: column; gap: 0.8rem; }
|
||||
/* Beat the generic `.hidden` (defined earlier, equal specificity): two classes
|
||||
win, so the panel truly collapses when Dev Mode is off. */
|
||||
.dev-panel.hidden { display: none; }
|
||||
.dev-panel legend { color: #4e9; }
|
||||
.dev-btn { margin-top: 0.5rem; width: 100%; padding: 0.4rem; cursor: pointer;
|
||||
background: #182028; color: #cde; border: 1px solid #2c3c5c; border-radius: 4px; }
|
||||
.dev-btn:hover { background: #20303f; }
|
||||
.dev-dl { margin: 0; display: grid; grid-template-columns: auto 1fr; gap: 0.15rem 0.6rem; font-size: 12px; }
|
||||
.dev-dl dt { color: #789; }
|
||||
.dev-dl dd { margin: 0; color: #dde; word-break: break-word; }
|
||||
.dev-dl dd.mem { color: #4e9; }
|
||||
.dev-dl dd.net { color: #fc6; }
|
||||
.dev-anno { font-size: 11px; max-height: 220px; overflow: auto;
|
||||
font-family: ui-monospace, monospace; }
|
||||
.dev-anno .anno-row { padding: 0.2rem 0; border-bottom: 1px solid #1d1d1d; }
|
||||
.dev-anno .anno-key { color: #aee6ff; }
|
||||
.dev-anno .anno-key.affect { color: #e3cfff; }
|
||||
.dev-anno .anno-key.measure { color: #ffd79a; }
|
||||
.dev-anno .anno-meta { color: #678; }
|
||||
.dev-anno .anno-empty { color: #567; font-style: italic; }
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
from tools.pipeline.ffmpeg_ops import frame_args, crossfade_loop_args, transcode_args
|
||||
|
||||
|
||||
def test_frame_args_trims_scales_pads_and_strips_audio():
|
||||
args = frame_args(
|
||||
"src.mp4", "out.mp4",
|
||||
start=2.0, duration=8.0, width=1920, height=1080, ff="ffmpeg",
|
||||
)
|
||||
assert args[0] == "ffmpeg"
|
||||
assert "-ss" in args and args[args.index("-ss") + 1] == "2.0"
|
||||
assert "-t" in args and args[args.index("-t") + 1] == "8.0"
|
||||
assert "-an" in args # audio stripped (bases are video-only)
|
||||
vf = args[args.index("-vf") + 1]
|
||||
assert "scale=1920:1080:force_original_aspect_ratio=decrease" in vf
|
||||
assert "pad=1920:1080" in vf
|
||||
assert "format=yuv420p" in vf
|
||||
assert args[-1] == "out.mp4"
|
||||
|
||||
|
||||
def test_crossfade_loop_builds_head_mid_tail_xfade_concat():
|
||||
args = crossfade_loop_args("in.mp4", "loop.mp4", duration=8.0, overlap=1.0)
|
||||
fc = args[args.index("-filter_complex") + 1]
|
||||
assert "trim=0:1.0" in fc # head = first O secs
|
||||
assert "trim=1.0:7.0" in fc # mid = [O, D-O]
|
||||
assert "trim=7.0:8.0" in fc # tail = last O secs
|
||||
assert "xfade=transition=fade:duration=1.0:offset=0" in fc
|
||||
assert "concat=n=2:v=1" in fc
|
||||
assert "-an" in args
|
||||
assert args[-1] == "loop.mp4"
|
||||
|
||||
|
||||
def test_crossfade_loop_rejects_overlap_too_large():
|
||||
import pytest
|
||||
with pytest.raises(ValueError):
|
||||
crossfade_loop_args("in.mp4", "loop.mp4", duration=4.0, overlap=2.0)
|
||||
|
||||
|
||||
def test_crossfade_loop_rejects_nonpositive_overlap():
|
||||
import pytest
|
||||
with pytest.raises(ValueError):
|
||||
crossfade_loop_args("in.mp4", "loop.mp4", duration=8.0, overlap=0.0)
|
||||
|
||||
|
||||
def test_transcode_args_high_profile_even_dims_faststart():
|
||||
args = transcode_args("loop.mp4", "proxy.mp4", width=1920, height=1080,
|
||||
crf=20, fps=30)
|
||||
assert "-profile:v" in args and args[args.index("-profile:v") + 1] == "high"
|
||||
assert "-crf" in args and args[args.index("-crf") + 1] == "20"
|
||||
vf = args[args.index("-vf") + 1]
|
||||
assert "scale=1920:1080:force_original_aspect_ratio=decrease" in vf
|
||||
assert "fps=30" in vf
|
||||
assert "+faststart" in args
|
||||
assert "-an" in args
|
||||
assert args[-1] == "proxy.mp4"
|
||||
@@ -0,0 +1,42 @@
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tools.pipeline.run import probe_duration, process_clip, resolve_ffmpeg
|
||||
|
||||
# A real pool base (forest was retired with the rotating-pool rename, session 0016);
|
||||
# cosmos/base.mp4 is the cosmos pool primary and the most stable PD clip on disk.
|
||||
_BASE = Path("simulator/sample_media/cosmos/base.mp4")
|
||||
|
||||
|
||||
def _ffmpeg_available() -> bool:
|
||||
try:
|
||||
resolve_ffmpeg()
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
_HAS_FF = _ffmpeg_available()
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _BASE.exists(),
|
||||
reason="cosmos base absent (run simulator/build_pool_manifest.py)")
|
||||
def test_probe_duration_reads_real_base():
|
||||
assert probe_duration(_BASE) > 0
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _HAS_FF,
|
||||
reason="neither system ffmpeg nor imageio-ffmpeg available")
|
||||
@pytest.mark.skipif(not _BASE.exists(),
|
||||
reason="cosmos base absent (run simulator/build_pool_manifest.py)")
|
||||
def test_process_clip_emits_proxy_and_master(tmp_path):
|
||||
out = process_clip(_BASE, tmp_path, overlap=0.5, start=0.0, duration=4.0)
|
||||
assert out["proxy"].exists() and out["master"].exists()
|
||||
# Proxy is exactly 1920×1080 — verified with cv2, no ffprobe dependency.
|
||||
import cv2
|
||||
cap = cv2.VideoCapture(str(out["proxy"]))
|
||||
w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
|
||||
h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
|
||||
cap.release()
|
||||
assert (w, h) == (1920, 1080)
|
||||
@@ -0,0 +1,47 @@
|
||||
from tools.pipeline.manifest import upsert_clip, add_ring_scale, rebuild_ring_edges
|
||||
|
||||
|
||||
def _base():
|
||||
return {"clips": [{"id": "cosmos", "title": "Cosmos"}],
|
||||
"ring": {"scales": [{"id": "cosmos", "clip_id": "cosmos"}],
|
||||
"transitions": []}}
|
||||
|
||||
|
||||
def test_upsert_clip_replaces_by_id_else_appends():
|
||||
m = _base()
|
||||
m = upsert_clip(m, {"id": "reef", "title": "Reef"})
|
||||
assert [c["id"] for c in m["clips"]] == ["cosmos", "reef"]
|
||||
m = upsert_clip(m, {"id": "reef", "title": "Coral Reef"}) # replace, not dup
|
||||
assert [c["id"] for c in m["clips"]] == ["cosmos", "reef"]
|
||||
assert m["clips"][1]["title"] == "Coral Reef"
|
||||
|
||||
|
||||
def test_add_ring_scale_inserts_after_named_scale():
|
||||
m = _base()
|
||||
m = add_ring_scale(m, "orbit", "orbit", after="cosmos")
|
||||
assert [s["id"] for s in m["ring"]["scales"]] == ["cosmos", "orbit"]
|
||||
|
||||
|
||||
def test_rebuild_ring_edges_one_transition_per_edge_with_wrap():
|
||||
m = _base()
|
||||
m = add_ring_scale(m, "orbit", "orbit", after="cosmos")
|
||||
m = add_ring_scale(m, "abyss", "abyss", after="orbit")
|
||||
m = rebuild_ring_edges(m)
|
||||
files = [t["file"] for t in m["ring"]["transitions"]]
|
||||
assert files == [
|
||||
"transitions/cosmos-orbit.mp4",
|
||||
"transitions/orbit-abyss.mp4",
|
||||
"transitions/abyss-cosmos.mp4", # wrap edge
|
||||
]
|
||||
|
||||
|
||||
def test_add_ring_scale_appends_when_after_missing():
|
||||
m = _base()
|
||||
m = add_ring_scale(m, "abyss", "abyss", after="nope")
|
||||
assert [s["id"] for s in m["ring"]["scales"]] == ["cosmos", "abyss"]
|
||||
|
||||
|
||||
def test_rebuild_ring_edges_single_scale_self_wrap():
|
||||
m = _base() # one scale: cosmos
|
||||
m = rebuild_ring_edges(m)
|
||||
assert [t["file"] for t in m["ring"]["transitions"]] == ["transitions/cosmos-cosmos.mp4"]
|
||||
@@ -0,0 +1,20 @@
|
||||
from tools.pipeline.provenance import Provenance
|
||||
|
||||
|
||||
def test_provenance_to_dict_roundtrip():
|
||||
p = Provenance(
|
||||
scale="abyss",
|
||||
license="public-domain (US Gov, 17 U.S.C. §105)",
|
||||
source="NOAA Ocean Exploration",
|
||||
url="https://oceanexplorer.noaa.gov/",
|
||||
fetched_at="2026-06-24",
|
||||
)
|
||||
d = p.to_dict()
|
||||
assert d == {
|
||||
"scale": "abyss",
|
||||
"license": "public-domain (US Gov, 17 U.S.C. §105)",
|
||||
"source": "NOAA Ocean Exploration",
|
||||
"url": "https://oceanexplorer.noaa.gov/",
|
||||
"fetched_at": "2026-06-24",
|
||||
}
|
||||
assert Provenance.from_dict(d) == p
|
||||
@@ -0,0 +1,120 @@
|
||||
"""tools/pipeline/track.py — the hybrid motion-track pass (content-pipeline §11.5).
|
||||
|
||||
Pure geometry helpers are unit-tested here; the cv2 optical-flow propagation is an
|
||||
opt-in integration test that runs on a real pool clip when present."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tools.pipeline.track import (
|
||||
clamp01_box,
|
||||
denormalize_box,
|
||||
infer_window,
|
||||
loop_t,
|
||||
normalize_box,
|
||||
sample_track,
|
||||
track_to_annotation,
|
||||
)
|
||||
|
||||
|
||||
# --- pure helpers -----------------------------------------------------------
|
||||
|
||||
|
||||
def test_clamp01_box_keeps_box_on_screen():
|
||||
assert clamp01_box((-0.2, 0.1, 0.5, 0.5)) == [0.0, 0.1, 0.5, 0.5]
|
||||
# size trimmed so the box never runs past the right/bottom edge
|
||||
assert clamp01_box((0.8, 0.8, 0.5, 0.5)) == [0.8, 0.8, pytest.approx(0.2), pytest.approx(0.2)]
|
||||
|
||||
|
||||
def test_normalize_denormalize_round_trip():
|
||||
px = (192, 108, 384, 216)
|
||||
norm = normalize_box(px, 1920, 1080)
|
||||
assert norm == [pytest.approx(0.1), pytest.approx(0.1), pytest.approx(0.2), pytest.approx(0.2)]
|
||||
assert denormalize_box(norm, 1920, 1080) == (192, 108, 384, 216)
|
||||
|
||||
|
||||
def test_loop_t_is_frame_over_total():
|
||||
assert loop_t(0, 100) == 0.0
|
||||
assert loop_t(50, 100) == 0.5
|
||||
assert loop_t(5, 0) == 0.0 # guard divide-by-zero
|
||||
|
||||
|
||||
def test_infer_window_spans_tracked_frames():
|
||||
appear, disappear = infer_window([10, 11, 30, 31], 100)
|
||||
assert appear == pytest.approx(0.1)
|
||||
assert disappear == pytest.approx(0.31)
|
||||
# no frames -> full clip
|
||||
assert infer_window([], 100) == (0.0, 1.0)
|
||||
|
||||
|
||||
def test_sample_track_downsamples_to_sparse_keyframes():
|
||||
# 21 dense frames -> at most n_keyframes, including the first and last
|
||||
per_frame = {i: [i / 100, 0.2, 0.1, 0.1] for i in range(0, 21)}
|
||||
track = sample_track(per_frame, total_frames=100, n_keyframes=5)
|
||||
assert len(track) <= 5
|
||||
ts = [k["t"] for k in track]
|
||||
assert ts == sorted(ts) # sorted by t
|
||||
assert ts[0] == pytest.approx(0.0) # first tracked frame
|
||||
assert ts[-1] == pytest.approx(0.2) # last tracked frame (frame 20 / 100)
|
||||
assert len(set(ts)) == len(ts) # de-duplicated on t
|
||||
|
||||
|
||||
def test_sample_track_keeps_all_when_already_sparse():
|
||||
per_frame = {0: [0.1, 0.1, 0.1, 0.1], 60: [0.5, 0.3, 0.1, 0.1]}
|
||||
track = sample_track(per_frame, total_frames=120, n_keyframes=5)
|
||||
assert [k["t"] for k in track] == [pytest.approx(0.0), pytest.approx(0.5)]
|
||||
|
||||
|
||||
def test_sample_track_empty_is_empty():
|
||||
assert sample_track({}, total_frames=100) == []
|
||||
|
||||
|
||||
def test_track_to_annotation_assembles_geometry_only():
|
||||
track = [{"t": 0.0, "box": [0.1, 0.2, 0.1, 0.1]}, {"t": 0.5, "box": [0.4, 0.2, 0.1, 0.1]}]
|
||||
ann = track_to_annotation("detected.jelly", track, salience=3, appear=0.0, disappear=0.55)
|
||||
assert ann["key"] == "detected.jelly"
|
||||
assert ann["salience"] == 3
|
||||
assert ann["track"] == track
|
||||
assert ann["appear"] == 0.0 and ann["disappear"] == 0.55
|
||||
# no semantics leaked in — strings/tiers are the author's, added elsewhere
|
||||
assert "strings" not in ann and "_tiers" not in ann
|
||||
|
||||
|
||||
def test_track_to_annotation_omits_window_when_absent():
|
||||
ann = track_to_annotation("detected.x", [], salience=4)
|
||||
assert "appear" not in ann and "disappear" not in ann
|
||||
|
||||
|
||||
# --- opt-in: classical optical-flow propagation on a real clip --------------
|
||||
|
||||
_CLIP = Path("simulator/sample_media/abyss_wow/base.mp4")
|
||||
|
||||
|
||||
def _cv2_available() -> bool:
|
||||
try:
|
||||
import cv2 # noqa: F401
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _CLIP.exists(),
|
||||
reason="abyss_wow base absent (run simulator/build_pool_manifest.py)")
|
||||
@pytest.mark.skipif(not _cv2_available(), reason="opencv-python not available")
|
||||
def test_track_seed_produces_a_sparse_annotation_on_real_footage():
|
||||
from tools.pipeline.track import track_box, track_seed
|
||||
|
||||
seed = (0.2, 0.3, 0.15, 0.18)
|
||||
per_frame, total = track_box(_CLIP, seed, seed_frame=0, max_frames=40)
|
||||
assert total > 0
|
||||
assert per_frame[0] == pytest.approx(list(seed), abs=0.02)
|
||||
assert len(per_frame) > 1 # propagated beyond the seed frame
|
||||
|
||||
ann = track_seed(_CLIP, "detected.jelly", seed, seed_frame=0, n_keyframes=5, max_frames=40)
|
||||
assert ann["key"] == "detected.jelly"
|
||||
assert 2 <= len(ann["track"]) <= 5 # sparse keyframes
|
||||
for kf in ann["track"]:
|
||||
assert 0.0 <= kf["t"] <= 1.0
|
||||
assert all(0.0 <= v <= 1.0 for v in kf["box"])
|
||||
assert 0.0 <= ann["appear"] <= ann["disappear"] <= 1.0
|
||||
@@ -8,6 +8,7 @@ from player.ring import (
|
||||
Transition,
|
||||
TransitionStep,
|
||||
advance_ring,
|
||||
pick_clip_id,
|
||||
scale_at,
|
||||
)
|
||||
|
||||
@@ -206,3 +207,36 @@ def test_fast_spin_noop_on_degenerate_ring():
|
||||
move = advance_ring(r, from_index=0, delta=9, fast_spin_threshold=3)
|
||||
assert move.fast is False
|
||||
assert move.steps == ()
|
||||
|
||||
|
||||
# --- rotating clip pool (content-pipeline design §11.1) ---
|
||||
|
||||
|
||||
def test_scale_without_pool_is_a_pool_of_one():
|
||||
s = Scale(id="cosmos", clip_id="cosmos")
|
||||
assert s.members == ("cosmos",)
|
||||
# any r maps to the single member
|
||||
assert pick_clip_id(s, 0.0) == "cosmos"
|
||||
assert pick_clip_id(s, 0.99) == "cosmos"
|
||||
|
||||
|
||||
def test_scale_with_pool_exposes_all_members():
|
||||
s = Scale(id="coast", clip_id="coast_a", pool=("coast_a", "coast_b", "coast_c"))
|
||||
assert s.members == ("coast_a", "coast_b", "coast_c")
|
||||
|
||||
|
||||
def test_pick_clip_id_selects_member_by_injected_r():
|
||||
s = Scale(id="coast", clip_id="a", pool=("a", "b", "c", "d"))
|
||||
# r in [0,1) partitions the pool into len equal buckets
|
||||
assert pick_clip_id(s, 0.0) == "a"
|
||||
assert pick_clip_id(s, 0.24) == "a"
|
||||
assert pick_clip_id(s, 0.25) == "b"
|
||||
assert pick_clip_id(s, 0.5) == "c"
|
||||
assert pick_clip_id(s, 0.75) == "d"
|
||||
|
||||
|
||||
def test_pick_clip_id_clamps_out_of_range_r():
|
||||
s = Scale(id="coast", clip_id="a", pool=("a", "b", "c"))
|
||||
assert pick_clip_id(s, -5.0) == "a" # below 0 clamps to first
|
||||
assert pick_clip_id(s, 1.0) == "c" # 1.0 must not index past the end
|
||||
assert pick_clip_id(s, 999.0) == "c"
|
||||
|
||||
@@ -91,6 +91,32 @@ def test_index_is_served():
|
||||
assert "Alteration" in resp.text
|
||||
|
||||
|
||||
def test_app_assets_are_not_cached(client):
|
||||
# Dev iteration relies on the browser never serving stale renderer code.
|
||||
resp = client.get("/")
|
||||
assert "no-store" in resp.headers.get("cache-control", "")
|
||||
|
||||
|
||||
def test_media_is_cacheable(tmp_path, monkeypatch):
|
||||
# Media is static: it must be browser-cacheable so the preloader's fetch (and
|
||||
# any fallback) doesn't re-hit the network on every altitude change.
|
||||
import simulator.app as appmod
|
||||
|
||||
media = tmp_path / "media"
|
||||
(media / "cosmos").mkdir(parents=True)
|
||||
(media / "cosmos" / "base.mp4").write_bytes(b"\x00\x00\x00\x18ftyp")
|
||||
monkeypatch.setattr(appmod, "MEDIA_DIR", media)
|
||||
manifest = tmp_path / "manifest.json"
|
||||
manifest.write_text(json.dumps({"clips": []}))
|
||||
|
||||
client = TestClient(create_app(manifest_path=manifest))
|
||||
resp = client.get("/media/cosmos/base.mp4")
|
||||
assert resp.status_code == 200
|
||||
cc = resp.headers.get("cache-control", "")
|
||||
assert "no-store" not in cc
|
||||
assert "max-age" in cc
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def ring_manifest_path(tmp_path):
|
||||
p = tmp_path / "manifest.json"
|
||||
@@ -189,3 +215,139 @@ def test_ring_404_when_no_ring_in_manifest(client):
|
||||
def test_ring_advance_rejects_bad_delta(ring_client):
|
||||
resp = ring_client.post("/api/ring/advance", json={"from_index": 0, "delta": "x"})
|
||||
assert resp.status_code == 422
|
||||
|
||||
|
||||
# --- rotating clip pool (content-pipeline §11.1) ---
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def pool_client(tmp_path):
|
||||
p = tmp_path / "manifest.json"
|
||||
members = ["coast_a", "coast_b", "coast_c", "coast_d"]
|
||||
clips = [{"id": "cosmos", "title": "Cosmos", "base_file": "cosmos/base.mp4",
|
||||
"license": "PD", "source": "NASA", "right_variants": {},
|
||||
"annotations": [], "affect": [], "strings": {}}]
|
||||
clips += [{"id": m, "title": m, "base_file": f"{m}/base.mp4", "license": "PD",
|
||||
"source": "NPS", "right_variants": {}, "annotations": [], "affect": [],
|
||||
"strings": {}} for m in members]
|
||||
p.write_text(json.dumps({
|
||||
"clips": clips,
|
||||
"ring": {
|
||||
"scales": [
|
||||
{"id": "cosmos", "clip_id": "cosmos"},
|
||||
{"id": "coast", "clip_id": members[0], "pool": members},
|
||||
],
|
||||
"transitions": [
|
||||
{"file": "transitions/cosmos-coast.mp4"},
|
||||
{"file": "transitions/coast-cosmos.mp4"},
|
||||
],
|
||||
},
|
||||
}))
|
||||
return TestClient(create_app(manifest_path=p)), members
|
||||
|
||||
|
||||
def test_ring_exposes_each_scales_pool(pool_client):
|
||||
client, members = pool_client
|
||||
data = client.get("/api/ring").json()
|
||||
coast = next(s for s in data["scales"] if s["id"] == "coast")
|
||||
assert [m["clip_id"] for m in coast["pool"]] == members
|
||||
# a single-clip scale still reports a pool of one
|
||||
cosmos = next(s for s in data["scales"] if s["id"] == "cosmos")
|
||||
assert [m["clip_id"] for m in cosmos["pool"]] == ["cosmos"]
|
||||
|
||||
|
||||
def test_advance_into_a_pool_lands_on_a_member(pool_client):
|
||||
client, members = pool_client
|
||||
# landing on the coast scale (index 1) must pick one of its pool members
|
||||
seen = set()
|
||||
for _ in range(40):
|
||||
data = client.post("/api/ring/advance", json={"from_index": 0, "delta": 1}).json()
|
||||
assert data["to_index"] == 1
|
||||
assert data["target_clip_id"] in members
|
||||
seen.add(data["target_clip_id"])
|
||||
# over many landings the random pick should cover more than one member
|
||||
assert len(seen) > 1
|
||||
|
||||
|
||||
def test_delta_zero_is_an_initial_pool_pick(pool_client):
|
||||
client, members = pool_client
|
||||
# a no-op (delta 0) advance on the pool scale is the initial / re-roll pick
|
||||
data = client.post("/api/ring/advance", json={"from_index": 1, "delta": 0}).json()
|
||||
assert data["to_index"] == 1
|
||||
assert data["steps"] == []
|
||||
assert data["target_clip_id"] in members
|
||||
|
||||
|
||||
# --- author mode (content-pipeline §11.5) ---
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def author_client(tmp_path):
|
||||
p = tmp_path / "manifest.json"
|
||||
p.write_text(json.dumps({
|
||||
"clips": [{
|
||||
"id": "abyss_wow", "title": "World of Water", "base_file": "abyss_wow/base.mp4",
|
||||
"license": "PD", "source": "NOAA", "right_variants": {},
|
||||
"annotations": [], "affect": [], "strings": {"en": {}},
|
||||
}],
|
||||
}))
|
||||
return TestClient(create_app(manifest_path=p)), p
|
||||
|
||||
|
||||
def test_author_clip_persists_labels_and_keeps_provenance(author_client):
|
||||
client, path = author_client
|
||||
body = {
|
||||
"clip_id": "abyss_wow",
|
||||
"annotations": [{"key": "detected.jelly", "salience": 4,
|
||||
"appear": 0.0, "disappear": 0.5,
|
||||
"track": [{"t": 0.0, "box": [0.1, 0.2, 0.1, 0.1]}]}],
|
||||
"affect": [{"key": "feel.unease", "at": [0.5, 0.5], "min_level": 1}],
|
||||
"strings": {"en": {"detected.jelly": ["jelly", "comb jelly", "Ctenophora", "Ctenophora · cilia rows"],
|
||||
"feel.unease": ["uh", "unease", "disquiet", "a creeping disquiet"]}},
|
||||
}
|
||||
resp = client.post("/api/author/clip", json=body)
|
||||
assert resp.status_code == 200 and resp.json()["ok"] is True
|
||||
# persisted to disk
|
||||
saved = json.loads(path.read_text())["clips"][0]
|
||||
assert saved["annotations"][0]["key"] == "detected.jelly"
|
||||
assert saved["affect"][0]["key"] == "feel.unease"
|
||||
assert saved["strings"]["en"]["detected.jelly"][2] == "Ctenophora"
|
||||
# provenance preserved (only authored content replaced)
|
||||
assert saved["base_file"] == "abyss_wow/base.mp4"
|
||||
assert saved["license"] == "PD"
|
||||
# reflected by /api/clips without a restart
|
||||
served = client.get("/api/clips").json()["clips"][0]
|
||||
assert served["annotations"][0]["key"] == "detected.jelly"
|
||||
|
||||
|
||||
def test_author_clip_unknown_clip_404(author_client):
|
||||
client, _ = author_client
|
||||
resp = client.post("/api/author/clip", json={"clip_id": "nope", "annotations": []})
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
def test_author_track_unknown_clip_404(author_client):
|
||||
client, _ = author_client
|
||||
resp = client.post("/api/author/track", json={
|
||||
"clip_id": "nope", "key": "detected.x", "seed_box": [0.1, 0.1, 0.1, 0.1]})
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
_REAL_CLIP = __import__("pathlib").Path("simulator/sample_media/abyss_wow/base.mp4")
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _REAL_CLIP.exists(),
|
||||
reason="abyss_wow base absent (run simulator/build_pool_manifest.py)")
|
||||
def test_author_track_on_real_clip_returns_keyframes():
|
||||
# Against the real default manifest + footage: the tracker yields a sparse track.
|
||||
client = TestClient(create_app())
|
||||
resp = client.post("/api/author/track", json={
|
||||
"clip_id": "abyss_wow", "key": "detected.jelly",
|
||||
"seed_box": [0.2, 0.3, 0.15, 0.18], "seed_t": 0.0,
|
||||
"salience": 4, "n_keyframes": 5, "max_frames": 30,
|
||||
})
|
||||
assert resp.status_code == 200
|
||||
ann = resp.json()
|
||||
assert ann["key"] == "detected.jelly"
|
||||
assert 2 <= len(ann["track"]) <= 5
|
||||
assert 0.0 <= ann["appear"] <= ann["disappear"] <= 1.0
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Content pipeline: source -> frame -> loop -> transcode -> manifest.
|
||||
|
||||
Pure ffmpeg-argument builders (ffmpeg_ops), a thin runner (run), provenance
|
||||
records, and manifest assembly. See docs/superpowers/specs/2026-06-24-content-pipeline-design.md.
|
||||
"""
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Pure ffmpeg argument builders (no I/O) for the content pipeline.
|
||||
|
||||
Each function returns a list[str] ready for ffmpeg, so command construction is
|
||||
unit-testable without running ffmpeg. tools/pipeline/run.py executes them. The
|
||||
ffmpeg binary name is injected (default "ffmpeg").
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def _fit(width: int, height: int) -> str:
|
||||
"""A scale+pad filter chain fitting any input into width×height (letterbox),
|
||||
fixing SAR and pixel format so downstream concat/xfade get identical geometry."""
|
||||
return (
|
||||
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
|
||||
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
|
||||
"setsar=1,format=yuv420p"
|
||||
)
|
||||
|
||||
|
||||
def frame_args(src, dst, *, start: float, duration: float,
|
||||
width: int, height: int, ff: str = "ffmpeg") -> list[str]:
|
||||
"""Stage 2 — trim [start, start+duration], fit to width×height, strip audio.
|
||||
Produces a high-quality mezzanine for the loop stage."""
|
||||
return [
|
||||
ff, "-y", "-ss", str(start), "-t", str(duration),
|
||||
"-i", str(src), "-vf", _fit(width, height), "-an",
|
||||
"-c:v", "libx264", "-preset", "medium", "-crf", "16",
|
||||
"-pix_fmt", "yuv420p", "-movflags", "+faststart", str(dst),
|
||||
]
|
||||
|
||||
|
||||
def crossfade_loop_args(src, dst, *, duration: float, overlap: float,
|
||||
fps: int = 30, ff: str = "ffmpeg") -> list[str]:
|
||||
"""Stage 3 — make `src` loop seamlessly by crossfading its tail over its head.
|
||||
Output length = duration − overlap. Requires overlap < duration/2 so a non-empty
|
||||
middle remains.
|
||||
|
||||
fps is applied after each trim so xfade receives a constant-frame-rate stream
|
||||
(xfade requires CFR; a raw trim output reports rate 1/0 which xfade rejects)."""
|
||||
if overlap <= 0 or overlap >= duration / 2:
|
||||
raise ValueError(f"overlap {overlap} must be in (0, duration/2={duration/2})")
|
||||
d, o = duration, overlap
|
||||
fc = (
|
||||
f"[0:v]trim=0:{o},setpts=PTS-STARTPTS,fps={fps}[head];"
|
||||
f"[0:v]trim={o}:{d - o},setpts=PTS-STARTPTS,fps={fps}[mid];"
|
||||
f"[0:v]trim={d - o}:{d},setpts=PTS-STARTPTS,fps={fps}[tail];"
|
||||
f"[tail][head]xfade=transition=fade:duration={o}:offset=0[xf];"
|
||||
f"[xf][mid]concat=n=2:v=1,format=yuv420p[out]"
|
||||
)
|
||||
return [
|
||||
ff, "-y", "-i", str(src), "-filter_complex", fc,
|
||||
"-map", "[out]", "-an",
|
||||
"-c:v", "libx264", "-preset", "medium", "-crf", "16",
|
||||
"-pix_fmt", "yuv420p", "-movflags", "+faststart", str(dst),
|
||||
]
|
||||
|
||||
|
||||
def transcode_args(src, dst, *, width: int, height: int, crf: int,
|
||||
fps: int = 30, ff: str = "ffmpeg") -> list[str]:
|
||||
"""Stage 4 — encode a final deliverable (master or proxy). Master: source res
|
||||
capped ~3840 wide, crf 18. Proxy: 1920×1080, crf 20. Both H.264 High, yuv420p,
|
||||
even dims, +faststart, video-only (spec §6)."""
|
||||
vf = (
|
||||
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
|
||||
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
|
||||
f"setsar=1,fps={fps},format=yuv420p"
|
||||
)
|
||||
return [
|
||||
ff, "-y", "-i", str(src), "-vf", vf, "-an",
|
||||
"-c:v", "libx264", "-profile:v", "high", "-preset", "slow",
|
||||
"-crf", str(crf), "-pix_fmt", "yuv420p",
|
||||
"-movflags", "+faststart", str(dst),
|
||||
]
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Stage 6 — assemble manifest entries (spec §3.6). Pure dict transforms over a
|
||||
loaded manifest dict; the caller does the json read/write. Idempotent by id so
|
||||
re-running earlier stages never clobbers authored labels."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
|
||||
|
||||
def upsert_clip(manifest: dict, clip: dict) -> dict:
|
||||
"""Replace the clip with the same id, or append it. Returns a new manifest."""
|
||||
m = copy.deepcopy(manifest)
|
||||
clips = m.setdefault("clips", [])
|
||||
for i, c in enumerate(clips):
|
||||
if c["id"] == clip["id"]:
|
||||
clips[i] = clip
|
||||
return m
|
||||
clips.append(clip)
|
||||
return m
|
||||
|
||||
|
||||
def add_ring_scale(manifest: dict, scale_id: str, clip_id: str,
|
||||
after: str | None = None) -> dict:
|
||||
"""Insert a scale into ring.scales after the named scale (or append if `after`
|
||||
is None / not found). No-op if scale_id already present. Returns a new manifest."""
|
||||
m = copy.deepcopy(manifest)
|
||||
ring = m.setdefault("ring", {"scales": [], "transitions": []})
|
||||
scales = ring.setdefault("scales", [])
|
||||
if any(s["id"] == scale_id for s in scales):
|
||||
return m
|
||||
entry = {"id": scale_id, "clip_id": clip_id}
|
||||
idx = next((i for i, s in enumerate(scales) if s["id"] == after), None)
|
||||
if idx is None:
|
||||
scales.append(entry)
|
||||
else:
|
||||
scales.insert(idx + 1, entry)
|
||||
return m
|
||||
|
||||
|
||||
def rebuild_ring_edges(manifest: dict) -> dict:
|
||||
"""Rebuild ring.transitions to one placeholder entry per ring edge (N scales ->
|
||||
N edges incl. the wrap), preserving any existing model string for an edge.
|
||||
Returns a new manifest."""
|
||||
m = copy.deepcopy(manifest)
|
||||
ring = m.setdefault("ring", {"scales": [], "transitions": []})
|
||||
scales = ring.get("scales", [])
|
||||
existing = {t["file"]: t.get("model", "") for t in ring.get("transitions", [])}
|
||||
edges = []
|
||||
n = len(scales)
|
||||
for i in range(n):
|
||||
a, b = scales[i]["id"], scales[(i + 1) % n]["id"]
|
||||
file = f"transitions/{a}-{b}.mp4"
|
||||
edges.append({"file": file, "model": existing.get(file, "placeholder-zoom")})
|
||||
ring["transitions"] = edges
|
||||
return m
|
||||
@@ -0,0 +1,29 @@
|
||||
"""Stage 1 — the per-clip strict-PD provenance record (spec §3.1).
|
||||
|
||||
Captured at source time so license + source + URL travel with the media. The
|
||||
'no explicit license -> assume PD, verify' trap (scales design §2.1) applies:
|
||||
"free stock" (Pexels/Pixabay) is NOT public domain.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict, dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Provenance:
|
||||
scale: str
|
||||
license: str
|
||||
source: str
|
||||
url: str
|
||||
fetched_at: str # ISO date; passed in (no clock here -> deterministic)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, d: dict) -> "Provenance":
|
||||
return cls(
|
||||
scale=d["scale"], license=d["license"], source=d["source"],
|
||||
url=d["url"], fetched_at=d["fetched_at"],
|
||||
)
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Thin runner: probe the source duration, then execute the frame -> loop ->
|
||||
transcode stages with tools.pipeline.ffmpeg_ops. The pure arg builders are
|
||||
unit-tested; this glue is covered by the opt-in integration test."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from .ffmpeg_ops import crossfade_loop_args, frame_args, transcode_args
|
||||
|
||||
MASTER_MAX_W = 3840
|
||||
|
||||
|
||||
def resolve_ffmpeg() -> str:
|
||||
"""ffmpeg from PATH, else the bundled imageio-ffmpeg binary (matches
|
||||
simulator/setup_scales_media.py). The Pi/dev box may lack a system ffmpeg."""
|
||||
if shutil.which("ffmpeg"):
|
||||
return "ffmpeg"
|
||||
import imageio_ffmpeg
|
||||
return imageio_ffmpeg.get_ffmpeg_exe()
|
||||
|
||||
|
||||
def probe_duration(src) -> float:
|
||||
"""Clip duration in seconds via cv2 (frame_count / fps) — avoids a system
|
||||
ffprobe dependency (imageio-ffmpeg ships ffmpeg only; the Pi has no ffprobe)."""
|
||||
import cv2
|
||||
cap = cv2.VideoCapture(str(src))
|
||||
fps = cap.get(cv2.CAP_PROP_FPS) or 0.0
|
||||
frames = cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0.0
|
||||
cap.release()
|
||||
if fps <= 0:
|
||||
raise ValueError(f"could not read fps from {src}")
|
||||
return frames / fps
|
||||
|
||||
|
||||
def _run(args: list[str]) -> None:
|
||||
subprocess.run(args, check=True, capture_output=True)
|
||||
|
||||
|
||||
def process_clip(src, out_dir, *, start: float = 0.0, duration: float | None = None,
|
||||
overlap: float = 1.0, master_w: int = MASTER_MAX_W, master_h: int = 2160,
|
||||
ff: str | None = None) -> dict:
|
||||
"""Run stages 2–4 for one clip; return {'master','proxy','loop','mezzanine'} paths.
|
||||
`duration` defaults to the full source length (minus the trim start).
|
||||
Note: `crossfade_loop_args` uses a fixed fps only to satisfy xfade's
|
||||
constant-frame-rate requirement; the final output fps is set by `transcode_args`
|
||||
(both default 30 — keep in sync if overridden)."""
|
||||
ff = ff or resolve_ffmpeg()
|
||||
src = Path(src)
|
||||
out_dir = Path(out_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
if duration is None:
|
||||
duration = probe_duration(src) - start
|
||||
|
||||
mezz = out_dir / "mezzanine.mp4"
|
||||
loop = out_dir / "loop.mp4"
|
||||
master = out_dir / "master.mp4"
|
||||
proxy = out_dir / "base.mp4" # the proxy is what the manifest's base_file points at
|
||||
|
||||
_run(frame_args(src, mezz, start=start, duration=duration,
|
||||
width=master_w, height=master_h, ff=ff))
|
||||
_run(crossfade_loop_args(mezz, loop, duration=duration, overlap=overlap, ff=ff))
|
||||
_run(transcode_args(loop, master, width=master_w, height=master_h, crf=18, ff=ff))
|
||||
_run(transcode_args(loop, proxy, width=1920, height=1080, crf=20, ff=ff))
|
||||
return {"mezzanine": mezz, "loop": loop, "master": master, "proxy": proxy}
|
||||
@@ -0,0 +1,297 @@
|
||||
"""Stage 5 geometry — the hybrid motion-track pass (content-pipeline §3.5 / §11.5).
|
||||
|
||||
Turns a hand-seeded box (or an ML detection) on one keyframe into the sparse,
|
||||
loop-normalized keyframed `track` the manifest stores (design §4) plus an
|
||||
appear/disappear window (§11.2). SEMANTICS are never produced here — the author
|
||||
owns the `key`, strings, `salience` and tiers; this module only propagates BOX
|
||||
GEOMETRY.
|
||||
|
||||
Two geometry paths, same artifact:
|
||||
- **classical** (always available): a hand-seeded box propagated by OpenCV
|
||||
Lucas-Kanade optical flow (a CSRT tracker is used instead when a contrib
|
||||
build provides one). Translation model — right for calm drifting subjects
|
||||
(the abyss/reef creatures). Re-seeds features when too few survive.
|
||||
- **ML detect+track** (optional, lazy-imported): a YOLO+tracker pass proposing
|
||||
boxes the author maps to a key. Imported only on demand so the base install
|
||||
needs no torch/ultralytics; a clear ImportError tells you what to `pip
|
||||
install` if you ask for it without it present.
|
||||
|
||||
The PURE helpers (normalization, keyframe sampling, window inference, annotation
|
||||
assembly) are unit-tested; the cv2/ML propagation is covered by the opt-in
|
||||
integration test on real footage.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Optional
|
||||
|
||||
Box = tuple[float, float, float, float] # (x, y, w, h)
|
||||
|
||||
|
||||
# --- pure geometry helpers --------------------------------------------------
|
||||
|
||||
def clamp01_box(box: Box) -> list[float]:
|
||||
"""Clamp a normalized box into the visible [0,1] frame, keeping it on-screen
|
||||
(origin clamped, then size trimmed so x+w<=1, y+h<=1)."""
|
||||
x, y, w, h = box
|
||||
x = min(max(x, 0.0), 1.0)
|
||||
y = min(max(y, 0.0), 1.0)
|
||||
w = min(max(w, 0.0), 1.0 - x)
|
||||
h = min(max(h, 0.0), 1.0 - y)
|
||||
return [x, y, w, h]
|
||||
|
||||
|
||||
def normalize_box(box_px: Box, width: int, height: int) -> list[float]:
|
||||
"""Pixel box -> normalized [0,1] box (the manifest's resolution-independent
|
||||
form). Clamped on-screen."""
|
||||
x, y, w, h = box_px
|
||||
return clamp01_box((x / width, y / height, w / width, h / height))
|
||||
|
||||
|
||||
def denormalize_box(box_norm: Box, width: int, height: int) -> tuple[int, int, int, int]:
|
||||
"""Normalized [0,1] box -> integer pixel box for the cv2 tracker."""
|
||||
x, y, w, h = box_norm
|
||||
return (round(x * width), round(y * height), round(w * width), round(h * height))
|
||||
|
||||
|
||||
def loop_t(frame_idx: int, total_frames: int) -> float:
|
||||
"""Loop-normalized time of a frame in [0,1), matching the client's
|
||||
`currentTime / duration` clock (design §4)."""
|
||||
if total_frames <= 0:
|
||||
return 0.0
|
||||
return frame_idx / total_frames
|
||||
|
||||
|
||||
def infer_window(tracked_frames: list[int], total_frames: int) -> tuple[float, float]:
|
||||
"""The appear/disappear window (§11.2) covering the frames where a box was
|
||||
tracked, as loop-normalized t. The first/last tracked frame bound it."""
|
||||
if not tracked_frames:
|
||||
return (0.0, 1.0)
|
||||
lo, hi = min(tracked_frames), max(tracked_frames)
|
||||
return (loop_t(lo, total_frames), loop_t(hi, total_frames))
|
||||
|
||||
|
||||
def sample_track(
|
||||
per_frame: dict[int, Box], total_frames: int, n_keyframes: int = 5
|
||||
) -> list[dict]:
|
||||
"""Down-sample a dense per-frame box map to a SPARSE keyframed track
|
||||
(design §4: sparse keyframes interpolated at runtime, never dense per-frame).
|
||||
|
||||
Picks `n_keyframes` frames evenly across the tracked span (always including
|
||||
the first and last tracked frame) and emits `[{t, box}, ...]` sorted by t,
|
||||
de-duplicated on t. Pure — takes already-normalized boxes."""
|
||||
if not per_frame:
|
||||
return []
|
||||
frames = sorted(per_frame)
|
||||
n = max(2, n_keyframes)
|
||||
if len(frames) <= n:
|
||||
picks = frames
|
||||
else:
|
||||
lo, hi = frames[0], frames[-1]
|
||||
step = (hi - lo) / (n - 1)
|
||||
targets = [lo + round(step * i) for i in range(n)]
|
||||
# snap each target to the nearest actually-tracked frame
|
||||
picks = sorted({min(frames, key=lambda f: abs(f - t)) for t in targets})
|
||||
out, seen = [], set()
|
||||
for f in picks:
|
||||
t = round(loop_t(f, total_frames), 4)
|
||||
if t in seen:
|
||||
continue
|
||||
seen.add(t)
|
||||
out.append({"t": t, "box": clamp01_box(per_frame[f])})
|
||||
return out
|
||||
|
||||
|
||||
def track_to_annotation(
|
||||
key: str,
|
||||
track: list[dict],
|
||||
*,
|
||||
salience: int = 4,
|
||||
appear: Optional[float] = None,
|
||||
disappear: Optional[float] = None,
|
||||
) -> dict:
|
||||
"""Assemble a manifest annotation from a propagated track + the author's
|
||||
semantics. Geometry (track + window) comes from this module; `key`,
|
||||
`salience` and the tiered strings are the author's (added separately)."""
|
||||
ann: dict = {"key": key, "salience": salience, "track": track}
|
||||
if appear is not None:
|
||||
ann["appear"] = round(appear, 4)
|
||||
if disappear is not None:
|
||||
ann["disappear"] = round(disappear, 4)
|
||||
return ann
|
||||
|
||||
|
||||
# --- classical propagation (cv2; covered by the opt-in integration test) -----
|
||||
|
||||
def _new_csrt():
|
||||
"""A CSRT tracker if this OpenCV build ships one (contrib), else None — then
|
||||
the optical-flow path is used. Kept tiny so the import stays lazy."""
|
||||
import cv2
|
||||
for factory in ("TrackerCSRT_create",):
|
||||
if hasattr(cv2, factory):
|
||||
return getattr(cv2, factory)()
|
||||
legacy = getattr(cv2, "legacy", None)
|
||||
if legacy is not None and hasattr(legacy, "TrackerCSRT_create"):
|
||||
return legacy.TrackerCSRT_create()
|
||||
return None
|
||||
|
||||
|
||||
def _features_in_box(gray, box_px, max_corners: int = 60):
|
||||
"""Good-feature points inside a pixel box, in full-frame coords (for LK)."""
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
x, y, w, h = (int(v) for v in box_px)
|
||||
h_img, w_img = gray.shape[:2]
|
||||
x0, y0 = max(x, 0), max(y, 0)
|
||||
x1, y1 = min(x + w, w_img), min(y + h, h_img)
|
||||
if x1 - x0 < 4 or y1 - y0 < 4:
|
||||
return None
|
||||
roi = gray[y0:y1, x0:x1]
|
||||
pts = cv2.goodFeaturesToTrack(roi, max_corners, 0.01, 5)
|
||||
if pts is None:
|
||||
return None
|
||||
pts = pts.reshape(-1, 2) + np.array([x0, y0], dtype="float32")
|
||||
return pts.reshape(-1, 1, 2)
|
||||
|
||||
|
||||
def track_box(
|
||||
video_path,
|
||||
seed_box_norm: Box,
|
||||
*,
|
||||
seed_frame: int = 0,
|
||||
max_frames: Optional[int] = None,
|
||||
min_features: int = 8,
|
||||
) -> tuple[dict[int, list[float]], int]:
|
||||
"""Propagate a hand-seeded normalized box FORWARD from `seed_frame` with
|
||||
optical flow (or CSRT when available). Returns `(per_frame_norm_boxes,
|
||||
total_frames)` — feed `per_frame` to `sample_track`. Translation model: the
|
||||
box follows the median displacement of features inside it, re-seeding when too
|
||||
few survive. Impure (reads the video) — opt-in integration test."""
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
cap = cv2.VideoCapture(str(video_path))
|
||||
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
|
||||
if seed_frame:
|
||||
cap.set(cv2.CAP_PROP_POS_FRAMES, seed_frame)
|
||||
ok, frame = cap.read()
|
||||
if not ok:
|
||||
cap.release()
|
||||
raise ValueError(f"could not read frame {seed_frame} of {video_path}")
|
||||
h_img, w_img = frame.shape[:2]
|
||||
box = list(denormalize_box(seed_box_norm, w_img, h_img)) # px [x,y,w,h]
|
||||
|
||||
csrt = _new_csrt()
|
||||
if csrt is not None:
|
||||
csrt.init(frame, tuple(int(v) for v in box))
|
||||
|
||||
prev_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
|
||||
pts = None if csrt is not None else _features_in_box(prev_gray, box)
|
||||
out: dict[int, list[float]] = {seed_frame: normalize_box(box, w_img, h_img)}
|
||||
|
||||
fidx = seed_frame
|
||||
while True:
|
||||
ok, frame = cap.read()
|
||||
if not ok:
|
||||
break
|
||||
fidx += 1
|
||||
if csrt is not None:
|
||||
ok2, b = csrt.update(frame)
|
||||
if ok2:
|
||||
box = [b[0], b[1], b[2], b[3]]
|
||||
else:
|
||||
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
|
||||
if pts is None or len(pts) < min_features:
|
||||
pts = _features_in_box(prev_gray, box)
|
||||
if pts is not None and len(pts):
|
||||
nxt, status, _ = cv2.calcOpticalFlowPyrLK(prev_gray, gray, pts, None)
|
||||
if nxt is not None:
|
||||
good_old = pts[status == 1]
|
||||
good_new = nxt[status == 1]
|
||||
if len(good_new) >= 3:
|
||||
dx = float(np.median(good_new[:, 0] - good_old[:, 0]))
|
||||
dy = float(np.median(good_new[:, 1] - good_old[:, 1]))
|
||||
box[0] += dx
|
||||
box[1] += dy
|
||||
pts = good_new.reshape(-1, 1, 2)
|
||||
else:
|
||||
pts = None
|
||||
prev_gray = gray
|
||||
out[fidx] = normalize_box(box, w_img, h_img)
|
||||
if max_frames is not None and fidx - seed_frame >= max_frames:
|
||||
break
|
||||
cap.release()
|
||||
return out, total
|
||||
|
||||
|
||||
def track_seed(
|
||||
video_path,
|
||||
key: str,
|
||||
seed_box_norm: Box,
|
||||
*,
|
||||
seed_frame: int = 0,
|
||||
salience: int = 4,
|
||||
n_keyframes: int = 5,
|
||||
max_frames: Optional[int] = None,
|
||||
) -> dict:
|
||||
"""End-to-end classical pass: seed -> propagate -> sample -> annotation
|
||||
(geometry only; the author adds strings/tiers). Impure (reads the video)."""
|
||||
per_frame, total = track_box(
|
||||
video_path, seed_box_norm, seed_frame=seed_frame, max_frames=max_frames
|
||||
)
|
||||
track = sample_track(per_frame, total, n_keyframes=n_keyframes)
|
||||
appear, disappear = infer_window(list(per_frame), total)
|
||||
return track_to_annotation(
|
||||
key, track, salience=salience, appear=appear, disappear=disappear
|
||||
)
|
||||
|
||||
|
||||
# --- optional ML detect+track (lazy-imported) --------------------------------
|
||||
|
||||
def detect_and_track(video_path, *, model: str = "yolov8n.pt", classes=None) -> list[dict]:
|
||||
"""Optional ML path (content-pipeline §3.5): run a detect+track model and
|
||||
return candidate tracks `[{track_id, label, salience, track:[{t,box}],
|
||||
appear, disappear}]` for the author to map to keys. Heavy deps are
|
||||
lazy-imported so the base install stays light.
|
||||
|
||||
Raises a clear ImportError naming the install if `ultralytics` is absent —
|
||||
this path is opt-in by design; the classical `track_seed` needs only cv2."""
|
||||
try:
|
||||
from ultralytics import YOLO # noqa: F401
|
||||
except ImportError as e: # pragma: no cover - exercised only without the dep
|
||||
raise ImportError(
|
||||
"the ML detect+track path needs ultralytics — "
|
||||
"`pip install ultralytics` (optional; the classical track_seed path "
|
||||
"uses only opencv-python)."
|
||||
) from e
|
||||
import cv2 # noqa: F401 (pragma: no cover below — needs the real model)
|
||||
|
||||
yolo = YOLO(model) # pragma: no cover
|
||||
cap = cv2.VideoCapture(str(video_path)) # pragma: no cover
|
||||
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0) # pragma: no cover
|
||||
cap.release() # pragma: no cover
|
||||
per_track: dict[int, dict[int, list[float]]] = {} # pragma: no cover
|
||||
labels: dict[int, str] = {} # pragma: no cover
|
||||
for fidx, res in enumerate( # pragma: no cover
|
||||
yolo.track(source=str(video_path), persist=True, classes=classes, stream=True)
|
||||
):
|
||||
if res.boxes is None or res.boxes.id is None: # pragma: no cover
|
||||
continue
|
||||
w_img, h_img = res.orig_shape[1], res.orig_shape[0] # pragma: no cover
|
||||
for box, tid, cls in zip( # pragma: no cover
|
||||
res.boxes.xywh.tolist(), res.boxes.id.tolist(), res.boxes.cls.tolist()
|
||||
):
|
||||
cx, cy, bw, bh = box # pragma: no cover
|
||||
nb = normalize_box((cx - bw / 2, cy - bh / 2, bw, bh), w_img, h_img)
|
||||
per_track.setdefault(int(tid), {})[fidx] = nb # pragma: no cover
|
||||
labels[int(tid)] = yolo.names.get(int(cls), str(cls))
|
||||
out = [] # pragma: no cover
|
||||
for tid, frames in per_track.items(): # pragma: no cover
|
||||
track = sample_track(frames, total) # pragma: no cover
|
||||
appear, disappear = infer_window(list(frames), total) # pragma: no cover
|
||||
out.append({ # pragma: no cover
|
||||
"track_id": tid, "label": labels.get(tid, ""), "salience": 4,
|
||||
"track": track, "appear": round(appear, 4), "disappear": round(disappear, 4),
|
||||
})
|
||||
return out # pragma: no cover
|
||||
Reference in New Issue
Block a user