Compare commits

...

32 Commits

Author SHA1 Message Date
BenStullsBets edca8a9615 content(sim): rotating clip pools + progressive tracked labels & emotions
Content-pipeline Increment 2, part 1 (the experience). Implements design §11
(docs/superpowers/specs/2026-06-24-content-pipeline-design.md):

- Rotating pool (§11.1): each ring scale references a POOL of vetted clips
  (docs/content-candidate-pool.md); the player picks a random member on landing.
  player.ring.Scale gains `pool` + pure `pick_clip_id(scale, r)` (injected
  randomness); the pick happens at the API boundary (random.random()), a delta=0
  advance is the initial/re-roll pick. clips.py parses/exposes the pool; the
  client lands on move.target_clip_id. Back-compat: a scale with only clip_id is
  a pool of one.
- Rename ring scale forest -> coast (new land<->sea scale, orbit..reef); ring
  order cosmos -> orbit -> coast -> reef -> abyss -> wrap. Cleanup: dropped the
  leftover local media (cosmos_traverse, orbit_westcoast, old forest/reef, the
  Increment-1 orbit/abyss single bases); new coast-edge transition placeholders.
- Time-windowed tracked labels (§11.2): annotations gain appear/disappear
  (loop-normalized, wraps across the seam); a label shows only while on screen
  and its box tracks the subject. abyss + reef pools authored as test material.
- Progressive LEFT detail tiers (§11.3): per-object `salience` (first shows at
  Left 5-salience) + tiered strings (general -> specific -> scientific -> +fact)
  escalating with the Left knob; replaces flat min_level gating. Strings may be a
  tier list or a static string (back-compat). No engine change (client reads
  overlay.level).
- Progressive RIGHT emotion tiers (§11.4): affect vocabulary escalates basic ->
  compound with the Right knob (dream.strength); appearance still gated by
  min(left,right). No engine change.

New simulator/build_pool_manifest.py holds the hand-authored content and emits
the 19-clip pool manifest (the reproducible baseline). setup_scales_media.py
marked superseded. 251 passed / 4 skipped (+ ring pool + API pool/window tests).
By-eye visual review deferred to the operator (no Chrome on this box).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 18:05:13 -07:00
BenStullsBets 4270c265e3 claim human-experience-filter-art session 0016 (placeholder) + sessions.json entry 2026-06-24 17:49:11 -07:00
benstull b28f930e11 Merge pull request 'docs: operator-selected content candidate pool' (#15) from content-candidate-pool into main 2026-06-25 00:39:59 +00:00
BenStullsBets 7be4f18e77 docs: operator-selected content candidate pool (rotating-pool model)
Records the 5-scale rotating clip pools the operator selected + vetted from the
candidate gallery (19 clips: cosmos 4, orbit 3, coast 4, reef 5, abyss 3), each
strict-PD/CC-BY-class, word-free, human-free, with source URL + trim window so
they are re-sourceable (media stays gitignored). Captures: the rotating-pool model
(random clip per scale on ring landing), the forest→coast rename, the cosmos
#3-Orion vs #6-Traverse processing mix-up to fix, and the rejected/leftover clips
to clean up. Feeds the next session (Increment 2 expanded: pool wiring + tracked
progressive labels + progressive emotions).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 17:39:54 -07:00
BenStullsBets 6256bc10e4 update sessions/0015/SESSION-0015.0-TRANSCRIPT-2026-06-24T13-16--INPROGRESS.md 2026-06-24 13:29:09 -07:00
BenStullsBets 317a6be6b0 claim human-experience-filter-art session 0015 (placeholder) + sessions.json entry 2026-06-24 13:16:43 -07:00
benstull bd4b65684c Merge pull request 'content(sim): real strict-PD bases for all 5 scales' (#14) from content-real-pd-bases into main 2026-06-24 16:16:37 +00:00
BenStullsBets 6369d0c5ab content(sim): real strict-PD bases for all 5 scales (provenance + license)
Sourced, license-verified, and ran through tools/pipeline/run.py process_clip
(trim → crossfade-loop → 1080p proxy) the five real public-domain neutral bases,
replacing the synthetic placeholders. Media stays gitignored; this records the
real provenance/license + trim window per clip so it is re-sourceable.

- cosmos — NASA/JPL-Caltech, Orion Nebula flythrough (17 U.S.C. §105)
- orbit  — NASA/JSC, Earth from ISS (Expedition 65) (§105)
- forest — NPS Yosemite stock b-roll (PD, no ©)
- reef   — NOAA Fisheries coral b-roll (§105)
- abyss  — NOAA Ocean Exploration comb jelly / ctenophore (§105)

All transcoded in ~2.6 min total (well under the 1-hour budget). Labels/tracks
remain the Increment-1 placeholders (real per-clip authoring is Increment 2).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 09:16:21 -07:00
BenStullsBets 4b8ebf11ea add sessions/0014/SESSION-0014.0-TRANSCRIPT-2026-06-24T06-58--2026-06-24T08-29.md + replace placeholder/variant SESSION-0014.0-TRANSCRIPT-2026-06-24T06-58--INPROGRESS.md 2026-06-24 08:30:02 -07:00
BenStullsBets 73e5c4a3c7 docs(spec): stamp content-pipeline design graduated + Increment 1 shipped (session 0014)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 08:28:02 -07:00
benstull bb5ecf09b4 Merge pull request 'feat(content): content pipeline Increment 1 — tooling + annotation tracks + 5-scale ring' (#13) from feature/content-pipeline into main 2026-06-24 15:26:17 +00:00
BenStullsBets e812f41ef1 docs: content-sourcing procedure + roadmap update + setup docstring (content pipeline Increment 1)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-24 08:19:41 -07:00
BenStullsBets 8789d6a6d5 feat(sim): extend scale ring to 5 (orbit + reef) (content pipeline §5)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-24 08:15:41 -07:00
BenStullsBets 2933498c51 feat(sim): client annotation-track interpolation + hand-authored forest track (content pipeline §4) 2026-06-24 08:11:40 -07:00
BenStullsBets 02c762fd45 fix(pipeline): probe duration via cv2 (no ffprobe dep); wire MASTER_MAX_W; fps-sync note
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-24 08:09:01 -07:00
BenStullsBets f1c83c1a9a feat(pipeline): resolve bundled ffmpeg fallback; integration test runs on real forest base
- run.py: add resolve_ffmpeg() mirroring setup_scales_media.py; change
  process_clip ff param to None and resolve at call time
- test_pipeline_integration.py: gate on resolve_ffmpeg() not PATH ffmpeg;
  verify proxy dims with cv2 instead of ffprobe
- ffmpeg_ops.py: add fps= to trim segments in crossfade_loop_args so xfade
  receives CFR input (xfade rejects 1/0 rate from raw trim output)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-24 08:05:11 -07:00
BenStullsBets 38a1046433 feat(pipeline): runner + opt-in integration test on real forest base (content pipeline §3)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-24 08:01:25 -07:00
BenStullsBets 60d27aa662 test(pipeline): cover ring-scale append fallback + N=1 self-wrap edge 2026-06-24 07:59:34 -07:00
BenStullsBets dc122fa0bf feat(pipeline): manifest assembly — upsert clip + ring scale/edges (content pipeline §3.6) 2026-06-24 07:57:38 -07:00
BenStullsBets 0fd8b1203d feat(pipeline): strict-PD provenance record (content pipeline §3.1) 2026-06-24 07:57:07 -07:00
BenStullsBets d3b6cd7bbd test(pipeline): cover non-positive overlap guard in crossfade_loop_args 2026-06-24 07:56:03 -07:00
BenStullsBets 4b5fd20374 feat(pipeline): master/proxy transcode arg builder (content pipeline §3.4/§6) 2026-06-24 07:54:36 -07:00
BenStullsBets 7217daeb44 feat(pipeline): crossfade seamless-loop arg builder (content pipeline §3.3) 2026-06-24 07:54:01 -07:00
BenStullsBets 642451925f feat(pipeline): frame stage ffmpeg arg builder (content pipeline §3.2) 2026-06-24 07:53:28 -07:00
BenStullsBets 750e87eb0b docs(plan): content pipeline Increment 1 implementation plan (session 0014)
Bite-sized TDD plan for Increment 1 (real footage, no ML): the deterministic
tools/pipeline/ package (frame → crossfade-loop → master+proxy transcode →
manifest assembly, pure ffmpeg arg builders unit-tested + an opt-in integration
test on the real forest POC base), the backward-compatible keyframed annotation
track schema with client-side interpolation, and the 5-scale ring (orbit + reef).
9 tasks. Implements spec §3 (stages 1–4,6), §4, §5, §6, §7, and §8 Increment 1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 07:47:00 -07:00
Ben Stull e210de6f84 docs(spec): content pipeline design — source → process → label → manifest (session 0014)
Brainstormed + scoped the content pipeline that turns strict-PD source clips into
shippable, seamless-looping, labelled neutral bases for the scale ring.

Decisions settled this session:
- Scale set: lean 5 (cosmos · orbit · forest/land · reef · abyss; microscopic dropped)
- Loop: crossfade tail→head seam
- Labels: motion-tracked, hybrid (ML proposes where it can, hand-seed + classical
  tracker elsewhere); label semantics always hand-authored
- Format: master + 1080p H.264 proxy

Anchor fact: the Right dream is real-time (no per-clip ML bake, session 0013), so
the pipeline is transcode + authoring — the one offline ML step is the chosen
motion-track pass. Six stages + an optional keyframed annotation-track schema
(backward compatible) + a 2-increment rollout (real footage no-ML first, then the
hybrid track pass + simulator author mode). i2v ring transitions + Pi/pano deferred.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 07:12:52 -07:00
Ben Stull daa6bddca6 claim human-experience-filter-art session 0014 (placeholder) + sessions.json entry 2026-06-24 06:59:52 -07:00
Ben Stull cfd8317965 add sessions/0013/SESSION-0013.0-TRANSCRIPT-2026-06-22T08-34--2026-06-23T11-29.md + replace placeholder/variant SESSION-0013.0-TRANSCRIPT-2026-06-22T08-34--INPROGRESS.md 2026-06-23 11:30:58 -07:00
benstull 23306fa140 Merge pull request 'feat(sim): Left HUD redesign · emotion channel · deterministic painterly Right dream · 3-knob control' (#12) from feature/left-hud-look-and-feel into main 2026-06-23 18:27:20 +00:00
Ben Stull f5d8a4f96b feat(sim): collapse Dark/Light into one bipolar Mood dial; drop calibration UI
The engine already models Dark/Light as the two poles of one signed axis
(tone = (light - dark)/4, equal = neutral), so the simulator now exposes a single
bipolar Mood dial (-4 dark .. 0 neutral .. +4 light) that maps onto dark/light in
controls(). Removes the Calibration section entirely: mood_gain and overlay_gain
were always-locked 1.0 dev knobs (full dial = full effect), so they bake in as the
DEFAULT_CALIBRATION constants and the sim sends no calibration. Experience panel is
now just Left, Right, Mood.

Engine/tests unchanged (DEFAULT_CALIBRATION already carries the 1.0 constants).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 09:58:43 -07:00
Ben Stull cbb7c9f53d feat(sim): Right axis = deterministic real-time painterly dream
Replace the pre-baked SD restyle variants (too fluid — the painting re-rolled
every keyframe and the flow-warp melted frames) with a deterministic, real-time
dream that holds STILL across the loop; only the footage's own motion moves.
Design: docs/superpowers/specs/2026-06-22-right-axis-deterministic-dream-design.md

Engine (player/alteration.py): Restyle(variant) -> Dream(strength, intensity);
Calibration.right_variant_map -> dream_gain. player/state.py: a Right change is a
LIVE_UPDATE now, not a crossfade (only a clip swap crossfades). Plan dict
restyle -> dream. Tests migrated; 233 green. (No new Python behavior for the
client-render swap below.)

Renderer (simulator front-end): the Right dream is a WebGL2 Kuwahara shader on a
<canvas> over the live video — edge-preserving, so motion/structure stay crisp;
the dream is the same footage, just stylized (no blur). Ramped by the knob; max
goes "trippy" (vivid saturation + ~45deg hue drift + posterize, concentrated near
max via t=intensity^2). Phase-1 luminous-haze (CSS+bloom) was built then retired
by eye (blur read as out-of-focus video). Mood grade rides as a CSS filter on the
canvas; bloom layer removed.

Dev ergonomics that fell out of the iteration:
- /dev/version + a 1s client poll = live-reload (open tab never runs stale
  renderer code; reloads on my edits AND server restarts).
- no-cache on the dev server; an on-screen error banner + crash-proof render so a
  silent throw can't masquerade as "nothing is changing".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 09:55:35 -07:00
Ben Stull 6adb93407b feat(sim): analytical Left HUD redesign + affect channel + no-cache
Left-brain HUD look-and-feel pass (judged by eye in the sim):
- corner-bracket targeting reticles (capped arms) replace plain boxes
- translucent label chips, legible over any frame
- two sensor channels: cyan detections (with deterministic confidence
  scores) vs amber measurements (center crosshair)
- bbox-sized "ANALYSIS . L{n} . {k} OBJ" status tag, escalating with Left

Affect channel (emotions in the HUD), per
docs/superpowers/specs/2026-06-22-affect-channel-hud-design.md:
- surfaces only when BOTH knobs up; strength = min(left, right)
- stable per-scene palette, more words appear with combined strength
- soft glowing violet words on their own opacity layer (the dream
  leaking into the machine's read); math in player/alteration.py
- forest/cosmos/abyss palettes in the manifest; 5 new unit tests

Simulator no-cache middleware so by-eye iteration never serves a stale
app.js / api response (fixes the latent "plan has affect but clip.affect
empty" stale-/api/clips bug).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:11:02 -07:00
37 changed files with 5885 additions and 217 deletions
+3 -1
View File
@@ -6,5 +6,7 @@ media/
.superpowers/
*.egg-info/
# Simulator sample media (look-tuning only; populate via setup_sample_media.py
# / setup_scales_media.py). All scale + transition binaries are gitignored.
# / build_pool_manifest.py). All scale + transition binaries are gitignored.
simulator/sample_media/**/*.mp4
# Local-only candidate review gallery (re-buildable; not shipped content).
simulator/static/review.html
+28 -15
View File
@@ -170,21 +170,34 @@ every transition, so fast navigation stays responsive on placeholder media.
contract** with the firmware (the keyboard/serial stand-in already works via a
`Controls` stream).
- **White-noise generation** — runtime white/pink noise for that content position.
- **Offline v2v variant pipeline** — author the pre-baked Right restyle variants
via the local flow-stabilized SD pipeline (now ~free, not a paid API — see the
scales-library/right-axis design §1/§4); a real multi-strength flow-stabilized
re-bake per base clip + the multilingual label/string tables + TTS. The
**scale-ring transitions** (forward/reverse per-edge morphs) are part of this
offline pipeline too.
- **Vertical slice done (session 0012):** the Right-variant half is
productionized and run for the **forest** scale — `simulator/bake_right_variants.py`
(the POC flow-restyle algorithm, repo-resident) bakes real strengths 14 on a
by-eye keyframe-strength ramp; forest now carries real Right variants in the
sim, so the dreamlike-axis look is judgeable on real footage. **Still
deferred:** real Right variants for cosmos/abyss (need their real strict-PD
bases sourced first) and the **real i2v ring transitions** (need a 2nd real
base + a generative-video model not in the POC). Plan:
[`2026-06-08-v2v-vertical-slice-forest-right-variants.md`](./superpowers/plans/2026-06-08-v2v-vertical-slice-forest-right-variants.md).
- **Content pipeline + label track** — source strict-PD neutral bases, run them
through the offline pipeline, and author the annotation tracks that drive the
Left-HUD overlay. The Right dream is now **real-time** (Kuwahara WebGL shader,
session 0013 — no per-clip ML bake), so the pipeline is
**source → crossfade-loop → master+proxy transcode → hand-authored /
motion-tracked labels → manifest**, not ML baking. Historical note: the
flow-stabilized SD-turbo v2v bake (session 0012, `simulator/bake_right_variants.py`)
was superseded when the operator found the baked-SD Right "too fluid" and the
design pivoted to the deterministic real-time dream; `bake_right_variants.py`
is now parked. Spec:
[`2026-06-24-content-pipeline-design.md`](./superpowers/specs/2026-06-24-content-pipeline-design.md).
- **Increment 1 — deterministic tooling + 5-scale ring (session 0014) ✅ Done.**
`tools/pipeline/` (frame/loop/transcode/manifest, pure + unit-tested + an
opt-in integration test on real footage); the backward-compatible keyframed
annotation-track schema with client interpolation; and the **full 5-scale ring**
(cosmos → orbit → forest → reef → abyss) with orbit + reef as placeholder
scales alongside the existing cosmos/abyss/forest. Plan:
[`2026-06-24-content-pipeline-increment-1.md`](./superpowers/plans/2026-06-24-content-pipeline-increment-1.md).
- **Increment 2 — hybrid motion-track pass + author mode (pending).** The
`tools/pipeline/track.py` classical OpenCV tracker + optional ML detect+track,
plus a simulator **author mode** for reviewing/editing annotation tracks in the
browser. Real strict-PD bases for all five scales sourced per
[`docs/content-sourcing.md`](./content-sourcing.md).
- **Still deferred:** real **i2v ring transitions** (need a generative-video
model + adjacent real bases); Pi renderer + serial/firmware.
- **Catalog model changes** — audio *source* + "neutral base" vs "altered
variant" flag (sub-project 2 territory, design §13).
+112
View File
@@ -0,0 +1,112 @@
# Content candidate pool — operator-selected, vetted (2026-06-24)
> **WIRED (session 0016).** The rotating pool below is now the live manifest:
> `simulator/build_pool_manifest.py` regenerates `simulator/sample_media/manifest.json`
> from this pool; the ring scale `forest`→`coast` rename + the cleanup list (bottom)
> are done; the player picks a random pool member on landing
> (`player.ring.pick_clip_id`). See content-pipeline design §11.
The 5 scales each get a **rotating pool** of clips. When the viewer lands on a
scale by zooming the ring in/out, the player **randomly picks one clip from that
scale's pool**. (This supersedes the one-clip-per-scale model — see
[`docs/superpowers/specs/2026-06-24-content-pipeline-design.md`](./superpowers/specs/2026-06-24-content-pipeline-design.md).)
Every clip below was operator-selected from the candidate gallery, is **strict
public-domain or CCBYclass with attribution**, and was vetted **wordfree /
humanfree** (windows trimmed past title cards, captions, telemetry burnin,
divers, construction, logos). All processed via `tools/pipeline/run.py
process_clip` (trim → crossfadeloop → 1080p proxy). **Media is gitignored**;
this file is the resourceable record. Sim ids are the `simulator/sample_media/<id>/` dirs.
License rule confirmed by operator: **CC0 / CCBY / CCBYSA are acceptable with
a credit line** (our overlays + dream are derivatives, which these allow);
**CCBYND is NOT usable** (no derivatives); **CCBYNC only if the installation
stays noncommercial**.
## 🌌 cosmos (pool of 4)
| id | clip | source | window (s) | license / credit |
|----|------|--------|-----------|------------------|
| `cosmos` | Orion Nebula flythrough | NASA/JPLCaltech — https://images.nasa.gov/details/JPL-20221122-SOLSYSf-0001-Orion%20Dust%20and%20Death | ~3054 | PD (17 U.S.C. §105) |
| `cosmos_galaxies` | Flying Through Galaxies | NASA SVS — https://svs.gsfc.nasa.gov/vis/a010000/a014900/a014950/14950_Galaxies_FlyThrough_4k.mp4 | 832 | PD |
| `cosmos_hudf` | Hubble Ultra Deep Field zoom | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030600/a030687/hudf-b-1920x1080p30.mov | 2044 | CCBYclass — credit **NASA, ESA, STScI** |
| `cosmos_xdf` | eXtreme Deep Field flythrough | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030600/a030681/hxdf_fly-b-1920x1080p30.mov | 226 | CCBYclass — credit **NASA, ESA, STScI** |
> ⚠️ **DISCREPANCY to fix next session.** Operator selected cosmos **#1, #3
> (Orion), #4, #5**. During processing, **#6 Galaxy Traverse was run by mistake
> instead of #3 Orion**. The Orion clip (#3) is textfree and already lives at
> `simulator/sample_media/cosmos/base.mp4` (the prior real base), so the pool above
> is correct. **`cosmos_traverse` is an UNSELECTED extra on disk — drop it** (or
> confirm with operator). The current ring scale id is `cosmos`; when the pool is
> wired, `cosmos` becomes one pool member.
## 🛰️ orbit (pool of 3)
| id | clip | source | window (s) | license |
|----|------|--------|-----------|---------|
| `orbit_planetearth` | ISS — View of Planet Earth | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030700/a030771/ISS_View_of_Planet_Earth_2160p.mp4 | 1034 | PD |
| `orbit_crewobs` | ISS — Crew Earth Observations | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030700/a030771/ISS_Crew_Earth_Observations_2160p.mp4 | 933 | PD |
| `orbit_bluemarble` | NPP "Blue Marble" rotating globe (CG) | NASA SVS — https://svs.gsfc.nasa.gov/vis/a000000/a004500/a004550/BlueMarble_starfield_4k_2160p30_v2.mp4 | 1034 | PD |
> Operator also selected aurora→Perth (#4) and westcoast lights (#5); both
> **rejected as UNUSABLE** — #4 has a date/time burned into every frame, #5 has the
> ISS module + solar panel in frame throughout. No clean window exists in either.
## 🏖️ coast (pool of 4) — NEW scale, replaces the granite/waterfall `forest`
| id | clip | source | window (s) | license |
|----|------|--------|-----------|---------|
| `coast_birdrock` | Bird rock + ocean (aerial) | NPS Channel Islands — https://www.nps.gov/nps-audiovideo/audiovideo/663a2f13-6183-4b3e-a4a9-6698a84619d01080p.mp4 | 327 | PD |
| `coast_surfgrass` | Surfgrass / coralline tidepool | NPS Cabrillo — https://www.nps.gov/nps-audiovideo/audiovideo/de7d1cf2-e422-45b5-aa66-4c69e29a4d811080p.mp4 | 226 | PD |
| `coast_elkbeach` | Elk resting in coastal grass/fog | NPS Redwood — https://www.nps.gov/nps-audiovideo/legacy/redw/4634070E-F68D-4F74-E43174448168AC8A/redw-elkbeach_1280x720.mp4 | 2038 | PD (720p source) |
| `coast_drakesbeach` | Elephant seals on dark sand | NPS Point Reyes — https://www.nps.gov/nps-audiovideo/audiovideo/1cfd8165-1061-4d75-af53-b46c2e3ed2701080p.mp4 | 2035 | PD |
> Operator also selected the Lifeboat Station "elephant seal timelapse" (moreoptions
> #3) — **FAILED**: the NPS page's video resolved to the wrong footage (an aerial of
> the bay with road/parking/boats, no seals). An alternate source UUID
> (`50a7a20a-9e52-4643-b186-c61ce1cef0f7`) appeared in the page HTML and could be
> retried next session if the seal timelapse is still wanted.
>
> **Scale rename:** the live ring scale is still `forest` (Yosemite). Next session
> renames it `coast` and points it at this pool. The old real Yosemite base
> (`simulator/sample_media/forest/`) becomes unused.
## 🐠 reef (pool of 5)
All NOAA Fisheries broll (PD, credit "NOAA Fisheries"); Brightcove download
endpoint `https://download.gallery.brightcove.com/api/gallery/account/659677166001/site/site-600408/video/<VIDEO_ID>/download`.
| id | clip | VIDEO_ID | window (s) | notes |
|----|------|----------|-----------|-------|
| `reef_spawning` | Snapper school + yellow fish over sunlit reef | 1553921798001 | 5078 | clean window from a 12min reel |
| `reef_hawkfish` | Humphead parrotfish over reef (Hawaiian atoll) | 6039896460001 | 250274 | |
| `reef_lionfish` | Lionfish hovering + blue snapper | 4088881464001 | 1236 | underwater clean; humans on land after ~190s |
| `reef_snapper` | Red snapper schooling (Gulf, greenish water) | 5305427942001 | 731 | |
| `reef_coralspacific` | Pacific coral macro, spotted fish | 5231464663001 | 3054 | diverfree window (divers at 625s) |
## 🦑 abyss (pool of 3)
All NOAA Ocean Exploration (PD, 17 U.S.C. §105). These have creatures drifting
**into and out of frame** — the prime material for timewindowed tracked labels.
| id | clip | source | window (s) | creatures |
|----|------|--------|-----------|-----------|
| `abyss_wow` | "World of Water" | https://oceanexplorer.noaa.gov/wp-content/uploads/2018/05/wow-1280x720-1.mp4 | 1034 | jelly + ctenophore + siphonophore enter/exit |
| `abyss_midwaterexp` | "Midwater Exploration" | https://oceanexplorer.noaa.gov/wp-content/uploads/2019/09/midwater-exploration-1920x1080-1.mp4 | 1640 | white medusa + red worm enter/exit |
| `abyss_hiding` | "Hiding in the Dark" | https://oceanexplorer.noaa.gov/wp-content/uploads/2018/05/dark-1280x720-1.mp4 | 2044 | crimson jelly + ctenophore enter/exit |
> Operator also selected Crossota (#4) and comb jelly (#5); both **rejected** —
> Crossota has only ~17s clean (too short for a 24s window), comb jelly has a
> species caption covering the whole clip.
## On disk but NOT in any pool — ✅ CLEANED UP (session 0016)
Removed from local media (all were gitignored, so repo-invisible):
- `cosmos_traverse` — unselected (the #6/#3 mixup above). **dropped.**
- `orbit_westcoast` — rejected (ISS hardware in frame). **dropped.**
- `forest` — the old Yosemite base, superseded by the `coast` pool. **dropped.**
- `reef` — the original placeholder/early reef base, superseded by `reef_*`. **dropped.**
- `orbit` / `abyss` — the Increment1 single real bases (NASA ISS Exp65 / ctenophore),
not in the new pools. **dropped** (the cosmos Orion base stays as the cosmos pool primary).
- `review.html` gallery — now `.gitignore`d (localonly, rebuildable).
+29
View File
@@ -0,0 +1,29 @@
# Content sourcing — strict-PD neutral bases
For each scale, acquire a calm, neutral source clip from a **strict public-domain**
US-government source, verify the license, then run it through the pipeline.
| Scale | Source | Where |
|--------|--------------------------------|------------------------------------|
| cosmos | NASA / Hubble / JWST | https://images.nasa.gov/ |
| orbit | NASA / ISS | https://images.nasa.gov/ |
| forest | NPS / USGS | https://www.nps.gov/ , https://www.usgs.gov/ |
| reef | NOAA Ocean Exploration | https://oceanexplorer.noaa.gov/ |
| abyss | NOAA Ocean Exploration | https://oceanexplorer.noaa.gov/ |
**License check (mandatory):** confirm the asset is US-gov PD (17 U.S.C. §105) or
explicitly CC0. "Free stock" (Pexels/Pixabay/etc.) is royalty-free but NOT public
domain — do not use. Record a `Provenance` (tools/pipeline/provenance.py) per clip.
**Run the pipeline** (replaces the placeholder bytes for a scale):
```bash
python -c "from tools.pipeline.run import process_clip; \
process_clip('downloads/<scale>.mp4', 'simulator/sample_media/<scale>', \
start=<in_s>, duration=<len_s>, overlap=<0.5..1.0>)"
```
This writes `simulator/sample_media/<scale>/base.mp4` (proxy) + `master.mp4`. Then
author the labels (Increment 2: the hybrid track pass + simulator author mode) or
hand-edit the annotation track for now (spec §4). The ring + manifest entry already
exist; `process_clip` only replaces the media bytes.
@@ -0,0 +1,929 @@
# Content Pipeline — Increment 1 (real footage, no ML) Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build the deterministic content-pipeline tooling (frame → crossfade-loop → master+proxy transcode → manifest assembly), add the backward-compatible keyframed annotation-track schema with client interpolation, and extend the scale ring to 5 scales — proving the pipeline end-to-end on real footage without any ML.
**Architecture:** A new pure-Python `tools/pipeline/` package whose stage functions return ffmpeg argument lists (unit-tested without running ffmpeg); a thin `run.py` executes them and is covered by an opt-in integration test gated on `ffmpeg` + the real forest POC base. Manifest assembly is pure dict transforms. The annotation `track` is optional data that rides through the existing pass-through `annotations` list (no Python model change); interpolation is client-side JS, consistent with the existing "alteration math in Python, label placement in the client" split. This implements §3 (stages 14, 6), §4 (track schema), §6 (format), §7 (tooling), and Increment 1 of §8 in [`docs/superpowers/specs/2026-06-24-content-pipeline-design.md`](../specs/2026-06-24-content-pipeline-design.md).
**Tech Stack:** Python 3.13, pytest, ffmpeg (libx264), vanilla JS (SVG overlay). Tests run with `pytest -q` from the repo root in the project `.venv`.
---
### Task 1: Scaffold `tools/pipeline/` + the frame stage
**Files:**
- Create: `tools/pipeline/__init__.py`
- Create: `tools/pipeline/ffmpeg_ops.py`
- Test: `tests/test_pipeline_ffmpeg_ops.py`
- [ ] **Step 1: Write the failing test**
```python
# tests/test_pipeline_ffmpeg_ops.py
from tools.pipeline.ffmpeg_ops import frame_args
def test_frame_args_trims_scales_pads_and_strips_audio():
args = frame_args(
"src.mp4", "out.mp4",
start=2.0, duration=8.0, width=1920, height=1080, ff="ffmpeg",
)
assert args[0] == "ffmpeg"
assert "-ss" in args and args[args.index("-ss") + 1] == "2.0"
assert "-t" in args and args[args.index("-t") + 1] == "8.0"
assert "-an" in args # audio stripped (bases are video-only)
vf = args[args.index("-vf") + 1]
assert "scale=1920:1080:force_original_aspect_ratio=decrease" in vf
assert "pad=1920:1080" in vf
assert "format=yuv420p" in vf
assert args[-1] == "out.mp4"
```
- [ ] **Step 2: Run test to verify it fails**
Run: `pytest tests/test_pipeline_ffmpeg_ops.py -q`
Expected: FAIL with `ModuleNotFoundError: No module named 'tools.pipeline'`
- [ ] **Step 3: Write minimal implementation**
```python
# tools/pipeline/__init__.py
"""Content pipeline: source -> frame -> loop -> transcode -> manifest.
Pure ffmpeg-argument builders (ffmpeg_ops), a thin runner (run), provenance
records, and manifest assembly. See docs/superpowers/specs/2026-06-24-content-pipeline-design.md.
"""
```
```python
# tools/pipeline/ffmpeg_ops.py
"""Pure ffmpeg argument builders (no I/O) for the content pipeline.
Each function returns a list[str] ready for ffmpeg, so command construction is
unit-testable without running ffmpeg. tools/pipeline/run.py executes them. The
ffmpeg binary name is injected (default "ffmpeg").
"""
from __future__ import annotations
def _fit(width: int, height: int) -> str:
"""A scale+pad filter chain fitting any input into width×height (letterbox),
fixing SAR and pixel format so downstream concat/xfade get identical geometry."""
return (
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
"setsar=1,format=yuv420p"
)
def frame_args(src, dst, *, start: float, duration: float,
width: int, height: int, ff: str = "ffmpeg") -> list[str]:
"""Stage 2 — trim [start, start+duration], fit to width×height, strip audio.
Produces a high-quality mezzanine for the loop stage."""
return [
ff, "-y", "-ss", str(start), "-t", str(duration),
"-i", str(src), "-vf", _fit(width, height), "-an",
"-c:v", "libx264", "-preset", "medium", "-crf", "16",
"-pix_fmt", "yuv420p", "-movflags", "+faststart", str(dst),
]
```
- [ ] **Step 4: Run test to verify it passes**
Run: `pytest tests/test_pipeline_ffmpeg_ops.py -q`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add tools/pipeline/__init__.py tools/pipeline/ffmpeg_ops.py tests/test_pipeline_ffmpeg_ops.py
git commit -m "feat(pipeline): frame stage ffmpeg arg builder (content pipeline §3.2)"
```
---
### Task 2: Crossfade seamless-loop stage
**Files:**
- Modify: `tools/pipeline/ffmpeg_ops.py`
- Test: `tests/test_pipeline_ffmpeg_ops.py`
The seamless loop (spec §3.3) crossfades the clip's tail over its head. For a clip
of duration `D` with overlap `O`, the output (length `D O`) is
`concat( xfade(tail, head, O), mid )` where `head`=`[0,O]`, `mid`=`[O, DO]`,
`tail`=`[DO, D]`. Both internal joins and the loop seam are then continuous, and
the tail→head jump is blended away over `O`.
- [ ] **Step 1: Write the failing test**
```python
# add to tests/test_pipeline_ffmpeg_ops.py
from tools.pipeline.ffmpeg_ops import crossfade_loop_args
def test_crossfade_loop_builds_head_mid_tail_xfade_concat():
args = crossfade_loop_args("in.mp4", "loop.mp4", duration=8.0, overlap=1.0)
fc = args[args.index("-filter_complex") + 1]
assert "trim=0:1.0" in fc # head = first O secs
assert "trim=1.0:7.0" in fc # mid = [O, D-O]
assert "trim=7.0:8.0" in fc # tail = last O secs
assert "xfade=transition=fade:duration=1.0:offset=0" in fc
assert "concat=n=2:v=1" in fc
assert "-an" in args
assert args[-1] == "loop.mp4"
def test_crossfade_loop_rejects_overlap_too_large():
import pytest
with pytest.raises(ValueError):
crossfade_loop_args("in.mp4", "loop.mp4", duration=4.0, overlap=2.0)
```
- [ ] **Step 2: Run test to verify it fails**
Run: `pytest tests/test_pipeline_ffmpeg_ops.py -q`
Expected: FAIL with `ImportError: cannot import name 'crossfade_loop_args'`
- [ ] **Step 3: Write minimal implementation**
```python
# add to tools/pipeline/ffmpeg_ops.py
def crossfade_loop_args(src, dst, *, duration: float, overlap: float,
ff: str = "ffmpeg") -> list[str]:
"""Stage 3 — make `src` loop seamlessly by crossfading its tail over its head.
Output length = duration overlap. Requires overlap < duration/2 so a non-empty
middle remains."""
if overlap <= 0 or overlap >= duration / 2:
raise ValueError(f"overlap {overlap} must be in (0, duration/2={duration/2})")
d, o = duration, overlap
fc = (
f"[0:v]trim=0:{o},setpts=PTS-STARTPTS[head];"
f"[0:v]trim={o}:{d - o},setpts=PTS-STARTPTS[mid];"
f"[0:v]trim={d - o}:{d},setpts=PTS-STARTPTS[tail];"
f"[tail][head]xfade=transition=fade:duration={o}:offset=0[xf];"
f"[xf][mid]concat=n=2:v=1,format=yuv420p[out]"
)
return [
ff, "-y", "-i", str(src), "-filter_complex", fc,
"-map", "[out]", "-an",
"-c:v", "libx264", "-preset", "medium", "-crf", "16",
"-pix_fmt", "yuv420p", "-movflags", "+faststart", str(dst),
]
```
- [ ] **Step 4: Run test to verify it passes**
Run: `pytest tests/test_pipeline_ffmpeg_ops.py -q`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add tools/pipeline/ffmpeg_ops.py tests/test_pipeline_ffmpeg_ops.py
git commit -m "feat(pipeline): crossfade seamless-loop arg builder (content pipeline §3.3)"
```
---
### Task 3: Master + proxy transcode stage
**Files:**
- Modify: `tools/pipeline/ffmpeg_ops.py`
- Test: `tests/test_pipeline_ffmpeg_ops.py`
- [ ] **Step 1: Write the failing test**
```python
# add to tests/test_pipeline_ffmpeg_ops.py
from tools.pipeline.ffmpeg_ops import transcode_args
def test_transcode_args_high_profile_even_dims_faststart():
args = transcode_args("loop.mp4", "proxy.mp4", width=1920, height=1080,
crf=20, fps=30)
assert "-profile:v" in args and args[args.index("-profile:v") + 1] == "high"
assert "-crf" in args and args[args.index("-crf") + 1] == "20"
vf = args[args.index("-vf") + 1]
assert "scale=1920:1080:force_original_aspect_ratio=decrease" in vf
assert "fps=30" in vf
assert "+faststart" in args
assert "-an" in args
assert args[-1] == "proxy.mp4"
```
- [ ] **Step 2: Run test to verify it fails**
Run: `pytest tests/test_pipeline_ffmpeg_ops.py -q`
Expected: FAIL with `ImportError: cannot import name 'transcode_args'`
- [ ] **Step 3: Write minimal implementation**
```python
# add to tools/pipeline/ffmpeg_ops.py
def transcode_args(src, dst, *, width: int, height: int, crf: int,
fps: int = 30, ff: str = "ffmpeg") -> list[str]:
"""Stage 4 — encode a final deliverable (master or proxy). Master: source res
capped ~3840 wide, crf 18. Proxy: 1920×1080, crf 20. Both H.264 High, yuv420p,
even dims, +faststart, video-only (spec §6)."""
vf = (
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
f"setsar=1,fps={fps},format=yuv420p"
)
return [
ff, "-y", "-i", str(src), "-vf", vf, "-an",
"-c:v", "libx264", "-profile:v", "high", "-preset", "slow",
"-crf", str(crf), "-pix_fmt", "yuv420p",
"-movflags", "+faststart", str(dst),
]
```
- [ ] **Step 4: Run test to verify it passes**
Run: `pytest tests/test_pipeline_ffmpeg_ops.py -q`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add tools/pipeline/ffmpeg_ops.py tests/test_pipeline_ffmpeg_ops.py
git commit -m "feat(pipeline): master/proxy transcode arg builder (content pipeline §3.4/§6)"
```
---
### Task 4: Provenance record (stage 1 output)
**Files:**
- Create: `tools/pipeline/provenance.py`
- Test: `tests/test_pipeline_provenance.py`
- [ ] **Step 1: Write the failing test**
```python
# tests/test_pipeline_provenance.py
from tools.pipeline.provenance import Provenance
def test_provenance_to_dict_roundtrip():
p = Provenance(
scale="abyss",
license="public-domain (US Gov, 17 U.S.C. §105)",
source="NOAA Ocean Exploration",
url="https://oceanexplorer.noaa.gov/",
fetched_at="2026-06-24",
)
d = p.to_dict()
assert d == {
"scale": "abyss",
"license": "public-domain (US Gov, 17 U.S.C. §105)",
"source": "NOAA Ocean Exploration",
"url": "https://oceanexplorer.noaa.gov/",
"fetched_at": "2026-06-24",
}
assert Provenance.from_dict(d) == p
```
- [ ] **Step 2: Run test to verify it fails**
Run: `pytest tests/test_pipeline_provenance.py -q`
Expected: FAIL with `ModuleNotFoundError: No module named 'tools.pipeline.provenance'`
- [ ] **Step 3: Write minimal implementation**
```python
# tools/pipeline/provenance.py
"""Stage 1 — the per-clip strict-PD provenance record (spec §3.1).
Captured at source time so license + source + URL travel with the media. The
'no explicit license -> assume PD, verify' trap (scales design §2.1) applies:
"free stock" (Pexels/Pixabay) is NOT public domain.
"""
from __future__ import annotations
from dataclasses import asdict, dataclass
@dataclass(frozen=True)
class Provenance:
scale: str
license: str
source: str
url: str
fetched_at: str # ISO date; passed in (no clock here -> deterministic)
def to_dict(self) -> dict:
return asdict(self)
@classmethod
def from_dict(cls, d: dict) -> "Provenance":
return cls(
scale=d["scale"], license=d["license"], source=d["source"],
url=d["url"], fetched_at=d["fetched_at"],
)
```
- [ ] **Step 4: Run test to verify it passes**
Run: `pytest tests/test_pipeline_provenance.py -q`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add tools/pipeline/provenance.py tests/test_pipeline_provenance.py
git commit -m "feat(pipeline): strict-PD provenance record (content pipeline §3.1)"
```
---
### Task 5: Manifest assembly (stage 6)
**Files:**
- Create: `tools/pipeline/manifest.py`
- Test: `tests/test_pipeline_manifest.py`
Pure dict transforms over a loaded manifest: upsert a clip entry, place a scale in
the ring order, and rebuild the per-edge transition list so it always has exactly
N entries for N scales (one transition per ring edge, including the wrap).
- [ ] **Step 1: Write the failing test**
```python
# tests/test_pipeline_manifest.py
from tools.pipeline.manifest import upsert_clip, add_ring_scale, rebuild_ring_edges
def _base():
return {"clips": [{"id": "cosmos", "title": "Cosmos"}],
"ring": {"scales": [{"id": "cosmos", "clip_id": "cosmos"}],
"transitions": []}}
def test_upsert_clip_replaces_by_id_else_appends():
m = _base()
m = upsert_clip(m, {"id": "reef", "title": "Reef"})
assert [c["id"] for c in m["clips"]] == ["cosmos", "reef"]
m = upsert_clip(m, {"id": "reef", "title": "Coral Reef"}) # replace, not dup
assert [c["id"] for c in m["clips"]] == ["cosmos", "reef"]
assert m["clips"][1]["title"] == "Coral Reef"
def test_add_ring_scale_inserts_after_named_scale():
m = _base()
m = add_ring_scale(m, "orbit", "orbit", after="cosmos")
assert [s["id"] for s in m["ring"]["scales"]] == ["cosmos", "orbit"]
def test_rebuild_ring_edges_one_transition_per_edge_with_wrap():
m = _base()
m = add_ring_scale(m, "orbit", "orbit", after="cosmos")
m = add_ring_scale(m, "abyss", "abyss", after="orbit")
m = rebuild_ring_edges(m)
files = [t["file"] for t in m["ring"]["transitions"]]
assert files == [
"transitions/cosmos-orbit.mp4",
"transitions/orbit-abyss.mp4",
"transitions/abyss-cosmos.mp4", # wrap edge
]
```
- [ ] **Step 2: Run test to verify it fails**
Run: `pytest tests/test_pipeline_manifest.py -q`
Expected: FAIL with `ModuleNotFoundError: No module named 'tools.pipeline.manifest'`
- [ ] **Step 3: Write minimal implementation**
```python
# tools/pipeline/manifest.py
"""Stage 6 — assemble manifest entries (spec §3.6). Pure dict transforms over a
loaded manifest dict; the caller does the json read/write. Idempotent by id so
re-running earlier stages never clobbers authored labels."""
from __future__ import annotations
import copy
def upsert_clip(manifest: dict, clip: dict) -> dict:
"""Replace the clip with the same id, or append it. Returns a new manifest."""
m = copy.deepcopy(manifest)
clips = m.setdefault("clips", [])
for i, c in enumerate(clips):
if c["id"] == clip["id"]:
clips[i] = clip
return m
clips.append(clip)
return m
def add_ring_scale(manifest: dict, scale_id: str, clip_id: str,
after: str | None = None) -> dict:
"""Insert a scale into ring.scales after the named scale (or append if `after`
is None / not found). No-op if scale_id already present. Returns a new manifest."""
m = copy.deepcopy(manifest)
ring = m.setdefault("ring", {"scales": [], "transitions": []})
scales = ring.setdefault("scales", [])
if any(s["id"] == scale_id for s in scales):
return m
entry = {"id": scale_id, "clip_id": clip_id}
idx = next((i for i, s in enumerate(scales) if s["id"] == after), None)
if idx is None:
scales.append(entry)
else:
scales.insert(idx + 1, entry)
return m
def rebuild_ring_edges(manifest: dict) -> dict:
"""Rebuild ring.transitions to one placeholder entry per ring edge (N scales ->
N edges incl. the wrap), preserving any existing model string for an edge.
Returns a new manifest."""
m = copy.deepcopy(manifest)
ring = m.setdefault("ring", {"scales": [], "transitions": []})
scales = ring.get("scales", [])
existing = {t["file"]: t.get("model", "") for t in ring.get("transitions", [])}
edges = []
n = len(scales)
for i in range(n):
a, b = scales[i]["id"], scales[(i + 1) % n]["id"]
file = f"transitions/{a}-{b}.mp4"
edges.append({"file": file, "model": existing.get(file, "placeholder-zoom")})
ring["transitions"] = edges
return m
```
- [ ] **Step 4: Run test to verify it passes**
Run: `pytest tests/test_pipeline_manifest.py -q`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add tools/pipeline/manifest.py tests/test_pipeline_manifest.py
git commit -m "feat(pipeline): manifest assembly — upsert clip + ring scale/edges (content pipeline §3.6)"
```
---
### Task 6: Runner + opt-in integration test on the real forest POC base
**Files:**
- Create: `tools/pipeline/run.py`
- Test: `tests/test_pipeline_integration.py`
The runner shells out ffmpeg with the builders from Tasks 13, ffprobing the source
duration so the loop stage gets real `D`. The integration test runs the **real**
forest POC base (`simulator/sample_media/forest/base.mp4`, populated by
`simulator/setup_sample_media.py`) through frame → loop → transcode and ffprobes the
proxy — gated on `ffmpeg`/`ffprobe` and on the base existing (it's gitignored media),
matching `tests/test_tools_integration.py`'s gating.
- [ ] **Step 1: Write the failing test**
```python
# tests/test_pipeline_integration.py
import shutil
import subprocess
from pathlib import Path
import pytest
from tools.pipeline.run import process_clip
_HAS_FF = shutil.which("ffmpeg") is not None and shutil.which("ffprobe") is not None
_FOREST = Path("simulator/sample_media/forest/base.mp4")
@pytest.mark.skipif(not _HAS_FF, reason="ffmpeg/ffprobe not installed")
@pytest.mark.skipif(not _FOREST.exists(),
reason="forest POC base absent (run simulator/setup_sample_media.py)")
def test_process_clip_emits_proxy_and_master(tmp_path):
out = process_clip(_FOREST, tmp_path, overlap=0.5, start=0.0, duration=4.0)
assert out["proxy"].exists() and out["master"].exists()
# Proxy is exactly 1920x1080.
probe = subprocess.run(
["ffprobe", "-v", "quiet", "-select_streams", "v:0",
"-show_entries", "stream=width,height", "-of", "csv=p=0", str(out["proxy"])],
capture_output=True, text=True, check=True,
).stdout.strip()
assert probe == "1920,1080"
```
- [ ] **Step 2: Run test to verify it fails**
Run: `pytest tests/test_pipeline_integration.py -q`
Expected: FAIL with `ModuleNotFoundError: No module named 'tools.pipeline.run'`
(or SKIP if ffmpeg/the base are absent — then verify by importing in `python -c`.)
- [ ] **Step 3: Write minimal implementation**
```python
# tools/pipeline/run.py
"""Thin runner: ffprobe the source, then execute the frame -> loop -> transcode
stages with tools.pipeline.ffmpeg_ops. The pure arg builders are unit-tested;
this glue is covered by the opt-in integration test."""
from __future__ import annotations
import subprocess
from pathlib import Path
from .ffmpeg_ops import crossfade_loop_args, frame_args, transcode_args
MASTER_MAX_W = 3840
def probe_duration(src, *, ff: str = "ffprobe") -> float:
out = subprocess.run(
[ff, "-v", "quiet", "-show_entries", "format=duration",
"-of", "csv=p=0", str(src)],
capture_output=True, text=True, check=True,
).stdout.strip()
return float(out)
def _run(args: list[str]) -> None:
subprocess.run(args, check=True, capture_output=True)
def process_clip(src, out_dir, *, start: float = 0.0, duration: float | None = None,
overlap: float = 1.0, master_w: int = 3840, master_h: int = 2160,
ff: str = "ffmpeg") -> dict:
"""Run stages 24 for one clip; return {'master','proxy','loop','mezzanine'} paths.
`duration` defaults to the full source length (minus the trim start)."""
src = Path(src)
out_dir = Path(out_dir)
out_dir.mkdir(parents=True, exist_ok=True)
if duration is None:
duration = probe_duration(src) - start
mezz = out_dir / "mezzanine.mp4"
loop = out_dir / "loop.mp4"
master = out_dir / "master.mp4"
proxy = out_dir / "base.mp4" # the proxy is what the manifest's base_file points at
_run(frame_args(src, mezz, start=start, duration=duration,
width=master_w, height=master_h, ff=ff))
_run(crossfade_loop_args(mezz, loop, duration=duration, overlap=overlap, ff=ff))
_run(transcode_args(loop, master, width=master_w, height=master_h, crf=18, ff=ff))
_run(transcode_args(loop, proxy, width=1920, height=1080, crf=20, ff=ff))
return {"mezzanine": mezz, "loop": loop, "master": master, "proxy": proxy}
```
- [ ] **Step 4: Run test to verify it passes (or skips cleanly)**
Run: `pytest tests/test_pipeline_integration.py -q`
Expected: PASS if ffmpeg + the forest base are present; otherwise SKIP with the gate
reason. If skipped, also run `python -c "from tools.pipeline.run import process_clip"`
and expect no error.
- [ ] **Step 5: Commit**
```bash
git add tools/pipeline/run.py tests/test_pipeline_integration.py
git commit -m "feat(pipeline): runner + opt-in integration test on real forest base (content pipeline §3)"
```
---
### Task 7: Client annotation-track interpolation + a hand-authored track
**Files:**
- Modify: `simulator/static/app.js` (`renderOverlay` at lines ~235269; `paintLoop` at ~173186)
- Modify: `simulator/sample_media/manifest.json` (add a `track` to the forest `detected.water` annotation)
The track is *data* on an existing annotation; the Python `Clip` already passes
`annotations` through opaquely (`simulator/clips.py` `_clip_from_dict` does
`d.get("annotations", [])`), so **no Python change is needed**. Interpolation is
client-side (spec §4). Because the overlay currently re-renders only on knob change,
a tracked box must update as the video plays — drive a re-render from the existing
`paintLoop` rAF when the active clip has any track and the overlay is visible.
- [ ] **Step 1: Add the data — a hand-authored track on forest's water detection**
In `simulator/sample_media/manifest.json`, replace the forest `detected.water`
annotation (currently `{"key": "detected.water", "box": [0.30, 0.10, 0.18, 0.70], "min_level": 1}`)
with a keyframed track (a small horizontal drift, looping at t=0 and t=1 so it is
seamless):
```json
{
"key": "detected.water",
"min_level": 1,
"track": [
{"t": 0.0, "box": [0.30, 0.10, 0.18, 0.70]},
{"t": 0.5, "box": [0.33, 0.10, 0.18, 0.70]},
{"t": 1.0, "box": [0.30, 0.10, 0.18, 0.70]}
]
}
```
- [ ] **Step 2: Add the interpolation helper + use it in `renderOverlay`**
In `simulator/static/app.js`, add a helper above `renderOverlay` and use it instead
of `a.box`:
```javascript
// Interpolate an annotation's box at loop-normalized time t (0..1). Static labels
// (no track) return their fixed box. Linear between sorted keyframes; clamps to ends.
function boxAt(a, t) {
if (!a.track || !a.track.length) return a.box;
const ks = a.track;
if (t <= ks[0].t) return ks[0].box;
if (t >= ks[ks.length - 1].t) return ks[ks.length - 1].box;
for (let i = 1; i < ks.length; i++) {
if (t <= ks[i].t) {
const a0 = ks[i - 1], a1 = ks[i];
const f = (t - a0.t) / (a1.t - a0.t);
return a0.box.map((v, j) => v + (a1.box[j] - v) * f);
}
}
return ks[ks.length - 1].box;
}
// Loop-normalized playback time of the base video (0..1), for track interpolation.
function loopT() {
return vid && vid.duration ? (vid.currentTime % vid.duration) / vid.duration : 0;
}
```
Then in `renderOverlay`, change the box line:
```javascript
const [x, y, w, h] = boxAt(a, loopT()).map((n) => n * 100);
```
- [ ] **Step 3: Re-render tracked overlays each frame from `paintLoop`**
Store the last overlay args and, in `paintLoop`, re-run `renderOverlay` when the
active clip has any tracked annotation and the overlay is visible. Add near the top
of `app.js` (module scope): `let lastOverlay = null;`. At the end of `renderOverlay`,
record the call: set `lastOverlay = { level, intensity };` (add this line just before
the closing brace). In `paintLoop` (after the existing shader work, before
`requestAnimationFrame(paintLoop)`), add:
```javascript
// Animate keyframed annotation tracks: re-place boxes against playback time.
const c = activeClip();
if (lastOverlay && lastOverlay.level > 0 && c &&
c.annotations.some((a) => a.track && a.track.length)) {
renderOverlay(lastOverlay.level, lastOverlay.intensity);
}
```
- [ ] **Step 4: Verify by eye (no JS unit harness in this repo)**
Run the sim and watch the forest water reticle drift with playback, looping
seamlessly; static labels (rock_face, conifer, the measurement) stay put.
```bash
python -m simulator.app # then open the printed localhost URL
```
Expected: with Left up on forest, the `flowing water` box drifts right then back
over the loop with no jump at the loop seam; affect/measure labels unchanged.
- [ ] **Step 5: Commit**
```bash
git add simulator/static/app.js simulator/sample_media/manifest.json
git commit -m "feat(sim): client annotation-track interpolation + hand-authored forest track (content pipeline §4)"
```
---
### Task 8: Extend the ring to 5 scales (orbit + reef placeholders)
**Files:**
- Modify: `simulator/setup_scales_media.py` (`SCALES` lines ~3236; `EDGES` line ~39)
- Modify: `simulator/sample_media/manifest.json` (add `orbit` + `reef` clips; reorder `ring.scales`; rebuild `ring.transitions`)
Real strict-PD bases are sourced separately (Task 9); to keep the 5-scale ring
demonstrable now, generate labelled placeholders for the two new scales with the
existing generator, exactly as cosmos/abyss are today.
- [ ] **Step 1: Add orbit + reef to the placeholder generator**
In `simulator/setup_scales_media.py`, extend `SCALES` and `EDGES` to the 5-scale
ring (order: cosmos → orbit → forest → reef → abyss → wrap):
```python
SCALES = {
"cosmos": ("0x05060f", "COSMOS — NASA/Hubble (PD placeholder)"),
"orbit": ("0x081427", "ORBIT — NASA/ISS (PD placeholder)"),
"forest": ("0x12301a", "FOREST — NPS/USGS (POC/placeholder)"),
"reef": ("0x06303a", "REEF — NOAA Ocean Exploration (PD placeholder)"),
"abyss": ("0x021016", "ABYSS — NOAA Ocean Exploration (PD placeholder)"),
}
EDGES = [("cosmos", "orbit"), ("orbit", "forest"), ("forest", "reef"),
("reef", "abyss"), ("abyss", "cosmos")]
```
- [ ] **Step 2: Add the orbit + reef clip entries to the manifest**
In `simulator/sample_media/manifest.json`, add two clip objects (after `cosmos` and
after `forest` respectively) mirroring the existing placeholder clips' shape, e.g.
orbit:
```json
{
"id": "orbit",
"title": "Earth from orbit (NASA/ISS, neutral base)",
"base_file": "orbit/base.mp4",
"license": "public-domain (US Gov, 17 U.S.C. §105)",
"source": "NASA/ISS — https://images.nasa.gov/ (true PD; placeholder bytes until ingest)",
"right_variants": {},
"annotations": [
{"key": "detected.cloud_band", "box": [0.30, 0.30, 0.30, 0.20], "min_level": 1},
{"key": "measure.altitude", "box": [0.06, 0.06, 0.18, 0.08], "min_level": 2}
],
"affect": [
{"key": "feel.serenity", "at": [0.50, 0.46], "min_level": 1},
{"key": "feel.fragility", "at": [0.22, 0.68], "min_level": 2},
{"key": "feel.unity", "at": [0.66, 0.58], "min_level": 3},
{"key": "feel.distance", "at": [0.42, 0.82], "min_level": 4}
],
"strings": {
"en": {
"detected.cloud_band": "cloud band",
"measure.altitude": "~408 km",
"feel.serenity": "serenity", "feel.fragility": "fragility",
"feel.unity": "unity", "feel.distance": "distance"
}
}
}
```
And reef:
```json
{
"id": "reef",
"title": "Coral reef (NOAA Ocean Exploration, neutral base)",
"base_file": "reef/base.mp4",
"license": "public-domain (US Gov, 17 U.S.C. §105)",
"source": "NOAA Ocean Exploration — https://oceanexplorer.noaa.gov/ (true PD, worldwide; placeholder bytes until ingest)",
"right_variants": {},
"annotations": [
{"key": "detected.coral", "box": [0.20, 0.45, 0.30, 0.30], "min_level": 1},
{"key": "detected.fish", "min_level": 2,
"track": [
{"t": 0.0, "box": [0.20, 0.30, 0.10, 0.08]},
{"t": 0.5, "box": [0.62, 0.34, 0.10, 0.08]},
{"t": 1.0, "box": [0.20, 0.30, 0.10, 0.08]}
]},
{"key": "measure.depth", "box": [0.06, 0.06, 0.16, 0.08], "min_level": 3}
],
"affect": [
{"key": "feel.delight", "at": [0.50, 0.46], "min_level": 1},
{"key": "feel.abundance", "at": [0.22, 0.68], "min_level": 2},
{"key": "feel.curiosity", "at": [0.66, 0.58], "min_level": 3},
{"key": "feel.immersion", "at": [0.42, 0.82], "min_level": 4}
],
"strings": {
"en": {
"detected.coral": "coral colony", "detected.fish": "reef fish",
"measure.depth": "18 m",
"feel.delight": "delight", "feel.abundance": "abundance",
"feel.curiosity": "curiosity", "feel.immersion": "immersion"
}
}
}
```
- [ ] **Step 3: Reorder the ring + rebuild edges in the manifest**
Set `ring.scales` to the 5-scale order and `ring.transitions` to the 5 edges:
```json
"ring": {
"scales": [
{"id": "cosmos", "clip_id": "cosmos"},
{"id": "orbit", "clip_id": "orbit"},
{"id": "forest", "clip_id": "forest"},
{"id": "reef", "clip_id": "reef"},
{"id": "abyss", "clip_id": "abyss"}
],
"transitions": [
{"file": "transitions/cosmos-orbit.mp4", "model": "placeholder-zoom"},
{"file": "transitions/orbit-forest.mp4", "model": "placeholder-zoom"},
{"file": "transitions/forest-reef.mp4", "model": "placeholder-zoom"},
{"file": "transitions/reef-abyss.mp4", "model": "placeholder-zoom"},
{"file": "transitions/abyss-cosmos.mp4", "model": "placeholder-zoom"}
]
}
```
- [ ] **Step 4: Regenerate placeholder media + verify the 5-scale ring**
```bash
python simulator/setup_scales_media.py
python -m simulator.app # open the URL; spin the encoder full circle
```
Expected: the encoder walks cosmos → orbit → forest → reef → abyss → (wrap) cosmos
with a transition on every edge; `GET /api/ring` lists 5 scales + 5 transitions; the
reef `reef fish` box drifts with playback (Task 7 interpolation) when Left ≥ 2.
- [ ] **Step 5: Commit**
```bash
git add simulator/setup_scales_media.py simulator/sample_media/manifest.json
git commit -m "feat(sim): extend scale ring to 5 (orbit + reef) (content pipeline §5)"
```
---
### Task 9: Sourcing procedure doc + ROADMAP update
**Files:**
- Create: `docs/content-sourcing.md`
- Modify: `docs/ROADMAP.md` (sub-project 3 "Offline v2v variant pipeline" bullet, lines ~174187)
Sourcing the real strict-PD bytes is a curatorial task, not a code step. Document the
procedure so anyone can run the pipeline per clip, and record the new content-pipeline
work on the roadmap.
- [ ] **Step 1: Write the sourcing procedure**
```markdown
# Content sourcing — strict-PD neutral bases
For each scale, acquire a calm, neutral source clip from a **strict public-domain**
US-government source, verify the license, then run it through the pipeline.
| Scale | Source | Where |
|--------|--------------------------------|------------------------------------|
| cosmos | NASA / Hubble / JWST | https://images.nasa.gov/ |
| orbit | NASA / ISS | https://images.nasa.gov/ |
| forest | NPS / USGS | https://www.nps.gov/ , https://www.usgs.gov/ |
| reef | NOAA Ocean Exploration | https://oceanexplorer.noaa.gov/ |
| abyss | NOAA Ocean Exploration | https://oceanexplorer.noaa.gov/ |
**License check (mandatory):** confirm the asset is US-gov PD (17 U.S.C. §105) or
explicitly CC0. "Free stock" (Pexels/Pixabay/etc.) is royalty-free but NOT public
domain — do not use. Record a `Provenance` (tools/pipeline/provenance.py) per clip.
**Run the pipeline** (replaces the placeholder bytes for a scale):
```bash
python -c "from tools.pipeline.run import process_clip; \
process_clip('downloads/<scale>.mp4', 'simulator/sample_media/<scale>', \
start=<in_s>, duration=<len_s>, overlap=<0.5..1.0>)"
```
This writes `simulator/sample_media/<scale>/base.mp4` (proxy) + `master.mp4`. Then
author the labels (Increment 2: the hybrid track pass + simulator author mode) or
hand-edit the annotation track for now (spec §4). The ring + manifest entry already
exist (Task 8); `process_clip` only replaces the media bytes.
```
- [ ] **Step 2: Update the roadmap**
In `docs/ROADMAP.md`, under sub-project 3, replace the "Offline v2v variant pipeline"
framing with the content-pipeline reality: the Right dream is real-time (no ML bake),
so the content pipeline is **source → crossfade-loop → master+proxy transcode →
hand-authored/motion-tracked labels → manifest**. Note Increment 1 (this plan: tooling
+ track schema + 5-scale ring, no ML) done on merge; Increment 2 (hybrid track pass +
simulator author mode) and real i2v ring transitions still pending. Cite the spec
`docs/superpowers/specs/2026-06-24-content-pipeline-design.md`.
- [ ] **Step 3: Run the full suite**
Run: `pytest -q`
Expected: all prior tests still pass + the new pipeline tests (ffmpeg_ops,
provenance, manifest) pass; the integration test passes or skips cleanly.
- [ ] **Step 4: Commit**
```bash
git add docs/content-sourcing.md docs/ROADMAP.md
git commit -m "docs: content-sourcing procedure + roadmap update (content pipeline Increment 1)"
```
---
## Notes for the executor
- **Run from the main clone, not a nested worktree** — `resolve-app.py` errors
"ambiguous" when a `.worktrees/` checkout shadows the main clone (session-0013
gotcha). If you must use a worktree, expect the `WGL_APP_SCAN_DEPTH` / explicit
`publish-transcript.sh` flag workarounds.
- **PRs go through the Gitea API** (`wgl-gitea-admin` `gitea-api.sh`, host
`git.benstull.org`) — no `tea`/`gh` here: create via `POST /repos/.../pulls`,
merge via `.../merge`.
- The project `.venv` has the deps (pytest, cv2/torch for later ML work). Activate it
before running tests.
- **Increment 2 (separate plan):** `tools/pipeline/track.py` (classical OpenCV
tracker + optional lazy-imported ML detect+track) and the simulator **author mode**
(draw/seed/correct boxes, assign keys/min_level/strings, place affect, write the
manifest). Plus real i2v ring transitions once a generative-video model + adjacent
real bases exist.
```
@@ -0,0 +1,85 @@
# Affect channel — emotions in the Left-brain HUD
**Status:** approved (session 0013, 2026-06-22)
**Surface:** simulator preview (`simulator/`, `player/alteration.py`)
**Builds on:** the machine-altered-perception design SPEC; the Left HUD look-and-feel
pass (same session).
## Idea
A third HUD channel beside the cyan *detections* and amber *measurements*: soft,
glowing **emotion-words** that surface the feelings the experience may invoke.
They appear only when *both* the analytical (Left) and dreamlike (Right) knobs are
up — the dream leaking into the machine's reading. This is the uncanny edge of the
thesis: the machine trying to quantify not just what is there, but how you are
meant to feel.
## Behaviour
- **Trigger / intensity.** Affect strength = `min(left, right)` (04). Either knob
at 0 → no emotions at all. Opacity scales with that strength × `overlay_gain`.
- **Escalation.** A **stable emotional palette per scene** — higher combined
strength reveals *more* words at higher opacity. Each word carries a `min_level`
(the same mechanic the detection channel uses), but compared against the
*combined* `min(left, right)` strength rather than Left alone.
- **Rendering.** Floating words at authored scene positions — **no boxes or
reticles**, lowercase, semi-transparent with a soft glow, in a distinct **violet**
register so they read as affective and clearly *softer* than the hard clinical
reticles.
## Where the math lives
Consistent with the existing split (alteration math in Python, label-selection in
the client):
- `player/alteration.py` gains an `AffectOverlay(strength, intensity)` on the
`RenderPlan`. `strength = min(left, right)`; `intensity = clamp(overlay_gain *
min(left,right) / KNOB_MAX, 0, 1)`. Returned in `render_plan_to_dict` under
`"affect"`. Pure, unit-tested.
- The simulator client picks *which* words to show by comparing each affect
entry's `min_level` to `plan.affect.strength`, and renders them at
`plan.affect.intensity` opacity.
`is_identity` is unaffected: `min(left,right) > 0` implies `left > 0`, which already
makes `overlay.level > 0`.
## Data
New optional per-clip `affect` list in the manifest, parallel to `annotations`:
```json
"affect": [
{"key": "feel.awe", "at": [0.5, 0.4], "min_level": 1},
{"key": "feel.serenity", "at": [0.2, 0.62], "min_level": 2}
],
"strings": {"en": {"feel.awe": "awe", "feel.serenity": "serenity"}}
```
`at` is a normalized `[x, y]` anchor point (not a box — feelings are scene-level).
The words live in the existing `strings.en` map. `Clip` gains an `affect` field
(defaulting to `[]`) threaded through `_clip_from_dict` and `to_dict`.
Starter palettes (words are art-direction, easily tuned):
| scale | palette (min_level 1 → 4) |
|--------|-----------------------------------------------|
| forest | awe · serenity · reverence · stillness |
| cosmos | wonder · vastness · insignificance · longing |
| abyss | unease · fascination · isolation · dread |
All three scales carry the data; forest is fully realized (it has the real Right
variants).
## Testing
- Python unit tests: affect `strength = min(left, right)` across knob combos;
intensity scaling and `overlay_gain`; `0` when either knob is 0; presence in the
serialized plan.
- By-eye in the live simulator (forest, Left & Right both up) — the established
judge-by-eye loop for this surface.
## Out of scope
- No drift-toward-unsettling arc (rejected in favour of a stable palette).
- No per-object emotion tags; no animated drift (words are statically placed for
v1; gentle motion can come later if wanted).
@@ -0,0 +1,109 @@
# Right axis as a real-time deterministic dream
**Status:** built (session 0013, 2026-06-22) — Phase 1 superseded by Phase 2 in the
same session after by-eye review (the haze "looked like blurred video, not stylized").
The shipped Right dream is the **Phase 2 painterly** path.
**Surface:** simulator preview (`simulator/`, `player/alteration.py`)
**Supersedes:** the "discrete pre-baked, flow-stabilized restyle variant" definition
of the Right axis in the machine-altered-perception design SPEC §4.1 — a deliberate
change to a core decision (see Why).
## Why
The pre-baked SD img2img variants (`bake_right_variants.py`) are **too fluid**: the
painting re-rolls every keyframe (~2 s) and the optical-flow tween warps frames into
each other, so across the loop the dream churns and melts. The right brain should be
a **still altered state** — a calm, holistic counterpart to the busy, accreting
analytical Left — not a restless morph.
## Concept
The Right knob no longer selects a pre-baked variant. It drives a **deterministic,
real-time filter** applied live to the base footage, scaled 0→4. The same filter runs
every frame, so the look is **stable by construction** — locked to scene content,
moving only where the footage already moves. Bonus: no baked variant videos to ship,
which simplifies the eventual hardware (Pi) story.
Staged delivery (both built this session; Phase 2 is what shipped):
- **Phase 1: luminous haze** — soft-focus, blooming highlights, pastel desaturation.
CSS + a bloom layer. Built, then **retired** by eye: blur reads as *out-of-focus
video*, not stylized, and softens the base's crisp motion.
- **Phase 2: painterly flatten (shipped)** — a WebGL2 **Kuwahara** shader on a
`<canvas>` restyling the live video frames. Edge-preserving, so motion/structure
stay as crisp as the source — the dream is the same footage, just stylized. No blur.
## Phase 2 — the shipped painterly dream
- **Renderer:** `<canvas id="paint">` over `<video>`; a `requestAnimationFrame` loop
uploads each video frame to a texture and runs the Kuwahara fragment shader (radius
`R=6`). `intensity = 0` ⇒ passthrough (raw base); ring transitions render passthrough
(`busy`). WebGL-less fallback: canvas hidden, mood grade applied to `#vid`.
- **Painterly:** `mix(base, kuwahara, intensity)` — flatten into painted regions, edges
preserved.
- **"Trippy" at max** (operator-tuned by eye): the psychedelic terms ramp with
`t = intensity²` so low/mid Right stays gently dreamy and only **max** goes
psychedelic — vivid saturation (`×(1 + 1.1·t)`), a surreal **hue drift** (~45° at max),
and **posterize** banding (`mix(255, 6, t)` levels), plus a slight luminous lift.
- **Mood grade** (Dark/Light) rides as a CSS filter on the canvas; `#tint` unchanged.
- The Python plan is unchanged — Phase 2 is a client-render swap; `dream.intensity`
drives the shader.
## Phase 1 — components
All scale with the Right strength `s = intensity` (0→1):
- **soft-focus** — gentle blur, ~`1.5·s` px.
- **pastel** — `saturate(1 0.35·s)`, `contrast(1 0.15·s)` (lift shadows / flatten).
- **luminous** — `brightness(1 + 0.08·s)`.
- **bloom** — a second, synced copy of the video, blurred + brightened and
`screen`-blended over the main, opacity `~0.6·s` — highlights glow and bleed.
Starting coefficients are by-eye seeds; the look is tuned live in the sim.
## Engine — `player/alteration.py`
- Replace `Restyle(variant)` with `Dream(strength, intensity)`:
`strength = right` (04, discrete), `intensity = clamp(dream_gain · right / KNOB_MAX,
0, 1)`.
- `Calibration` gains `dream_gain: float = 1.0`; `DEFAULT_CALIBRATION` sets it 1.0.
- `render_plan_to_dict`: `"restyle": {"variant"}``"dream": {"strength", "intensity"}`.
- `is_identity`: Right contributes when `strength > 0` (replaces the
`restyle.variant == 0` clause).
- Pure + unit-tested. The client maps `dream.intensity` → CSS params, mirroring how
`grade.tone` → video filter is mapped client-side today.
## Rendering — simulator
- **Remove** the variant-crossfade machinery (`loadVariant`, `variantFile`, the
`right_variants` / `currentVariant` handling). The main video always plays
`base_file`.
- **Compose one video filter** from *both* the mood grade (`grade.tone`) and the dream
(`dream.intensity`) — today `applyGrade` owns `vid.style.filter`; it becomes a single
`applyVideoLook(tone, dreamIntensity)` so the two don't clobber each other.
- **Add a bloom layer**: a second `<video id="bloom">` (same `base_file`, muted,
looped, started in sync), `screen`-blended, blurred/brightened, opacity from
`dream.intensity`. Both videos' `src` are set together on scale settle / ring move.
- `#tint` (the Dark cool wash) is unchanged.
## Migration
- The baked SD variants (`*/right*.mp4`) and `bake_right_variants.py` become **legacy:
parked in-repo, not deleted, unused at runtime**. `right_variants` stays in the
manifest/`clips.py` (ignored by the client) to avoid churn; it can be removed later.
- The scale ring and its transitions are untouched.
- The affect channel is unaffected — Right is still a 04 knob, so `min(left, right)`
emotions keep working.
## Testing
- Python unit tests: `Dream.strength = right`, `intensity` scaling + `dream_gain`,
clamping, `0` at `right=0`, presence/shape in the serialized plan, `is_identity`.
- By-eye in the live sim (forest, Right up, then Left+Right for the whole picture) —
the established judge-by-eye loop. Success = the dream holds still across the loop;
only real motion (the waterfall) moves.
## Out of scope
- Removing the legacy baker / baked media (left parked, unused at runtime).
- Any change to the scale ring, transitions, Left HUD, or affect channel.
- A Pi-GPU performance pass on the Kuwahara shader (desktop sim is the current target).
@@ -0,0 +1,355 @@
# HEF — Content Pipeline (source → process → label → manifest)
**Date:** 2026-06-24
**Status:** Graduated — operator-approved (session 0014); **Increment 1 built & merged** (PR #13); real PD sourcing + rotating pool curated (PR #14/#15). **Increment 2 schema pinned + building** (session 0016 — see §11).
**Repo:** `human-experience-filter-art`
**Anchor:** `docs/ROADMAP.md` sub-project 3 (Player Runtime) content needs; the
spiritual successor to sub-project 2 (Ingest & Tagging tools), retargeted from the
retired coordinate-catalog model to the manifest content model.
**Refines:** [`2026-06-07-scales-library-and-right-axis-pipeline-design.md`](./2026-06-07-scales-library-and-right-axis-pipeline-design.md)
§2 (scales library), §2.1 (strict-PD sourcing), §6 (open sourcing questions).
**Builds on:** [`2026-06-22-right-axis-deterministic-dream-design.md`](./2026-06-22-right-axis-deterministic-dream-design.md)
(the real-time dream — **no per-clip ML bake**) and
[`2026-06-22-affect-channel-hud-design.md`](./2026-06-22-affect-channel-hud-design.md)
(scene-anchored affect words).
---
## 1. Why this exists
The installation has an alteration engine, a scale ring, a Left analytical HUD, an
affect channel, and a real-time Right "dream" — but it has **no real content**. The
three ring scales today are: `cosmos` and `abyss` carrying *placeholder bytes* (PD
provenance recorded, but the media is synthetic), and `forest` carrying a *POC
sample* (a trimmed Yosemite clip licensed "look-tuning only; not shipped content").
There is no repeatable way to turn a public-domain source clip into a shippable,
seamless-looping, labelled neutral base.
This design defines that pipeline: **source → frame → seamless-loop → transcode →
label/author → manifest entry.** One big anchor fact reshapes it versus the old
plan — the Right dream is now a **real-time deterministic shader** (session 0013),
so there is **no per-clip ML restyle bake** anymore. The pipeline's offline cost is
therefore **transcode + human authoring**, not generative-video baking. The one
deliberate exception is the motion-tracking label pass chosen this session (§3),
which re-introduces a bounded offline compute step — by choice, for realism.
## 2. Decisions settled this session
| # | Decision | Choice | Consequence |
|---|----------|--------|-------------|
| 1 | Scale set | **Lean 5** — cosmos · orbit · forest/land · reef · abyss (microscopic dropped) | Ring wraps **abyss → cosmos**; replaces the POC forest + both placeholder bases with real strict-PD footage |
| 2 | Seamless loop | **Crossfade loop** (tail→head overlap) | Any source clip becomes an indefinite seamless dwell loop; a transcode-time step |
| 3 | Label placement | **Motion-tracked, hybrid** | ML proposes tracks where it can; hand-seed + classical tracker elsewhere. Re-introduces an offline pass + an authoring step. Semantics always hand-authored |
| 4 | Format | **Master + 1080p proxy** | High-quality master retained per clip; a 1080p H.264 proxy drives the sim; Pi chooses later |
Microscopic was dropped because PD microscopy *video* (vs stills) is the thinnest
pool and hardest to source well; it can be added later by re-running the pipeline.
## 3. The pipeline — six stages
Each stage is a deterministic unit with one job (handbook §4.2: deterministic tools,
not prose an agent re-derives). Stages 14 and 6 are mechanical; stage 5 is the one
human-in-the-loop step.
```
(1) source ──▶ (2) frame ──▶ (3) loop ──▶ (4) transcode ──▶ (6) manifest
provenance trim/crop crossfade master + proxy entry
& scale seam │ ▲
└──▶ (5) label/author (hybrid track)
```
### 3.1 Stage 1 — source & provenance
Acquire the strict-PD source clip and record license + source URL + capture notes.
Reuse sub-project-2 fetchers where they map (a NASA fetcher already exists); add
NOAA Ocean Exploration / NPS / USGS fetchers (or accept a manual download + a
provenance record). The "no explicit license → assume PD, verify" trap from the
scales design §2.1 applies: **"free stock" (Pexels/Pixabay) is NOT public domain.**
Output: the raw source file + a provenance record (license, source, url, fetched_at).
### 3.2 Stage 2 — frame (trim / crop / scale)
Pick the in/out points; crop to the target aspect; scale toward the format target
(§6). The panoramic projector aspect is still unmeasured (open question), so v1
frames to a standard wide 16:9 for the sim; the master retains maximum available
width so a later pano crop/extension has pixels to work with. Output: a framed,
silent (audio stripped — bases are video-only; audio is the separate content-dial
channel) intermediate.
### 3.3 Stage 3 — seamless crossfade loop
Make the clip loop indefinitely with no visible seam by overlapping the tail into
the head with a short crossfade (~0.51.0 s). The final looped clip is shorter than
the source by the overlap; during the overlap region motion is a blend (acceptable
for calm ambient nature footage). This is the chosen loop strategy (clean-cut and
palindrome were rejected — seam risk and reversed motion respectively). Output: a
loop-clean clip whose first and last frames join seamlessly.
### 3.4 Stage 4 — transcode (master + proxy)
Emit two encodes per clip (§6 for exact params): a **master** (high quality, up to
4K, retained as the archival/Pi-candidate source) and a **1080p H.264 proxy** (what
the sim plays today, matching current `base.mp4` behaviour). Both video-only,
`yuv420p`, even dimensions, `+faststart`.
### 3.5 Stage 5 — label & author (the hybrid motion-track pass)
The only human-in-the-loop stage, and the only one that adds offline ML compute.
- **Semantics are always hand-authored.** A generic detector cannot produce the
installation's semantic keys — "spiral galaxy", "siphonophore", `z ≈ 0.04`,
`~2.1 m³/s` are out-of-distribution. The author owns the label **keys**,
**strings**, and `min_level`; tracking only propagates **box geometry**.
- **Hybrid geometry (decision 3).** Where a model can lock onto a subject (a fish
on the reef, an animal in the forest), an ML detect+track pass (YOLO+ByteTrack /
SAM2-class, lazy-imported like the parked baker imported torch) proposes a track
the author maps to a key. Where it can't (cosmos, abyss, microscopic-type
content), the author **hand-seeds** a box on a keyframe and a **classical tracker**
(OpenCV CSRT / optical-flow) propagates it; the author fixes drift. Both paths
emit the same artifact.
- **Channel rules.** `detected.*` object labels **track**; `measure.*` readouts and
`feel.*` affect words stay **static / scene-anchored** (a depth readout or "awe"
is not pinned to a moving pixel). This matches the affect-channel design (affect
is scene-level) and keeps measurement chips legible.
- Output: an **annotation track** per object label (§4) plus the static measurement
and affect placements, all keyed and stringed.
### 3.6 Stage 6 — manifest assembly
Write the clip entry (base/proxy/master paths, license, source, annotations with
tracks, affect, strings) into `simulator/sample_media/manifest.json`, and register
the scale in the `ring` section with its transition edges (§5). Idempotent: stage 6
owns the manifest entry so re-running earlier stages never clobbers authored labels
(mirrors how `setup_sample_media.py` was made to never clobber a real bake).
## 4. Data model — annotation tracks
Today an annotation is a single static box: `{"key", "box":[x,y,w,h], "min_level"}`.
Add an **optional keyframed track**; static labels are unchanged (backward
compatible):
```json
{
"key": "detected.fish",
"min_level": 1,
"track": [
{"t": 0.0, "box": [0.30, 0.40, 0.10, 0.08]},
{"t": 0.5, "box": [0.52, 0.38, 0.10, 0.08]},
{"t": 1.0, "box": [0.30, 0.40, 0.10, 0.08]}
]
}
```
- `t` is **normalized 0→1 over the looped clip** — resolution- and fps-independent,
so the same track is correct for both master and proxy and survives a re-transcode.
(A track that loops cleanly has matching boxes at `t=0` and `t=1`.)
- **Runtime:** if `track` is present, the client interpolates the box at the current
loop-normalized playback time (linear between keyframes); otherwise it uses the
static `box`. This lives **client-side**, consistent with the existing split
(alteration math in Python; label selection/placement in the client). `Clip` gains
the track only as data inside `annotations`; `player/alteration.py` is **unchanged**
(labels are not part of `RenderPlan`).
- `measure.*` keep a single static `box`; `feel.*` keep a single static `at:[x,y]`.
## 5. The 5 scales, sources, and ring
| Scale | Strict-PD source | Notes |
|-------|------------------|-------|
| `cosmos` | NASA / Hubble / JWST | 🟢 abundant true PD (17 U.S.C. §105); replaces placeholder |
| `orbit` | NASA / ISS (Earth from orbit) | 🟢 true PD; **new** scale |
| `forest` | NPS / USGS (US land/wildlife) | 🟢 true PD (US locations); replaces the non-shippable POC sample |
| `reef` | NOAA Ocean Exploration (shallow) | 🟢 true PD worldwide; **new** scale |
| `abyss` | NOAA Ocean Exploration (deep sea) | 🟢 true PD worldwide; replaces placeholder |
**Ring order (large → small, zooms inward; wraps small → large):**
`cosmos → orbit → forest → reef → abyss → (wrap) cosmos`. Five scales → five ring
edges. The wrap seam is now **abyss → cosmos** (deep sea back out to the cosmos) —
the infinite-zoom payoff, minus the microscopic step.
**Transitions** between the new adjacent edges remain the **deferred i2v generative
work** (scales design §3 / §6) — they need a generative-video model and two real
adjacent bases, heavier than anything in this pipeline. Until then new edges carry
**placeholder transitions** (the `setup_scales_media.py` ffmpeg `xfade=zoomin`
generator already produces these), exactly as cosmos/forest/abyss do today. This
pipeline's job is the **bases + labels**; transitions are tracked but out of scope
here (§8).
## 6. Format target
| | Master | Proxy (sim) |
|---|--------|-------------|
| Resolution | source, capped ~3840 wide | 1920×1080 (fit wide) |
| Codec | H.264 High | H.264 High |
| Quality | CRF ~18 | CRF ~20 |
| FPS | preserve source (cap 30) | 30 |
| Pixel fmt | `yuv420p`, even dims | `yuv420p`, even dims |
| Audio | none (stripped) | none (stripped) |
| Flags | `+faststart` | `+faststart` |
The sim plays the **proxy** as today's `base_file`. The **master** is retained per
clip for the eventual Pi/pano encode once the panoramic resolution is measured.
Browser HEVC support is too spotty to make HEVC the sim path, which is why the proxy
stays H.264 (the master can be re-encoded to HEVC for the Pi later without
re-sourcing).
## 7. Tooling & where it lives
- **New `tools/pipeline/` package** — deterministic ffmpeg-backed verbs for stages
14 and 6 (`fetch`/provenance, `frame`, `loop`, `transcode`, `manifest`), reusing
sub-project-2 fetchers where valid. This **supersedes** the coordinate-era
ingest *drafting* (the `left/right/dark/light` Record model retired with the
machine-altered-perception pivot); the pipeline targets the **manifest**, not
catalog `Record`s.
- **Stage 5 tracking** — `tools/pipeline/track.py`: classical OpenCV tracker
(always available) + an optional lazy-imported ML detect+track path. Emits
candidate keyframed tracks for the author to accept/correct.
- **Authoring surface** — extend the **simulator** with an **author mode** (it
already renders the exact HUD overlay and is the by-eye judge): scrub the clip,
draw/seed/adjust boxes, run the tracker, assign keys/`min_level`/strings, place
affect anchors, write the manifest entry. Reusing the preview as the editor keeps
"what you author is what plays." **This is the largest new piece** — see the
delivery order (§8), which lets v1 prove the schema before the full editor exists.
## 8. Delivery / rollout order
Build order within this Feature (Solution-Design rollout strategy — not a task
list):
1. **Increment 1 — real footage, no ML.** Stages 14 + 6 + the §4 track schema +
client interpolation. Source the 5 strict-PD bases, crossfade-loop them, emit
master+proxy, wire them into the ring. Author a *couple* of tracks **by hand as
JSON** to prove the track schema and the runtime interpolation. Outcome: the
whole ring is **real, seamless-looping footage** judgeable by eye — the
highest-value, lowest-risk step, and it stands alone.
2. **Increment 2 — the hybrid track pass + author mode.** `track.py` (classical +
optional ML) and the simulator author mode; label all five scales properly.
**Deferred (tracked, not built here):** the real **i2v ring transitions** for the
new edges (need a generative-video model + adjacent real bases); the Pi renderer +
serial/firmware (per the simulator-first directive); the panoramic-resolution
master encode (waits on a measured pano spec).
> Scope note: if Increment 2's author mode grows large enough to be its own
> increment in practice, split it at execution time. The design is one coherent
> pipeline; the increments above are its sensible build order, not separate
> Features.
## 9. Open questions (for the plan, not blockers)
- **Panoramic resolution/aspect** — unmeasured (scales design §6). Drives the master
crop/extension strategy; v1 frames 16:9 for the sim and retains max width.
- **ML model choice** for the hybrid track path (YOLO+ByteTrack vs SAM2-class) and
whether it earns its place given how few scales it can usefully fire on.
- **Classical tracker drift** on long/fast clips — keyframe interval, re-seed cadence.
- **Source fetchers** — which of NOAA / NPS / USGS get real fetchers vs manual
download + provenance record for v1.
- **Crossfade overlap length** per scale (slow cosmos vs busier reef) — by eye.
## 11. Increment 2 — schema decisions (session 0016)
Increment 2 is the second build order from §8. It is mostly **schema work**, so
this section pins the concrete data shapes before the build (the operator asked for
a short design pass first). Each subsection is a new, **backward-compatible**
extension — Increment 1 manifests keep rendering unchanged.
### 11.1 Rotating clip pool (supersedes one-clip-per-scale)
Each ring scale references a **pool** of vetted clips
(`docs/content-candidate-pool.md`); the player picks **one member at random when
the viewer lands on that scale** (zooming the ring in/out). This supersedes
decision §5's one-base-per-scale.
- **Manifest:** `ring.scales[i]` gains `"pool": ["clip_id", …]` (≥1). The legacy
`"clip_id"` stays as the **primary** member for back-compat (= `pool[0]` when a
pool is written); a scale with only `clip_id` and no `pool` behaves exactly as
Increment 1 (pool of one).
- **Canonical logic (`player/ring.py`):** `Scale` gains `pool: tuple[str, …]`;
`members` = `pool or (clip_id,)`. A **pure** `pick_clip_id(scale, r)` takes an
injected `r ∈ [0,1)` and returns `members[int(r·len)]` — randomness stays
**injected** (DI), so the function is pure/testable and the Pi reuses it.
- **Where the pick happens:** at the **API boundary** (`simulator/app.py`), which
may be impure: the advance endpoint calls `pick_clip_id(landed_scale,
random.random())` and returns it as `target_clip_id`. A `delta=0` advance is the
**initial / re-roll** pick (no-op move, fresh random member). The client loads
`move.target_clip_id` directly instead of deriving the clip from the index. One
selection implementation, Python-canonical (browser still "only renders").
- **Rename + cleanup:** ring scale `forest`**`coast`** (new land↔sea scale
between `orbit` and `reef`). Drop the leftovers `cosmos_traverse`,
`orbit_westcoast`, old `forest`, old `reef` (per the pool doc's cleanup list);
the cosmos pool's primary stays `cosmos` (Orion). New ring order:
`cosmos → orbit → coast → reef → abyss → (wrap) cosmos`; edges regenerated
(`orbit-coast`, `coast-reef` placeholders via `setup_scales_media.py`).
### 11.2 Time-windowed tracked labels
Objects drift **into and out of frame**, so a label appears only **while its
object is on screen** and its box **tracks** the object (the §4 keyframed `track`).
- **Manifest:** an annotation gains optional `"appear"` and `"disappear"`
(loop-normalized `t ∈ [0,1]`, same clock as `track`). The label renders only
while `appear ≤ loopT ≤ disappear`. To let an object straddle the loop seam, a
**`disappear < appear` window wraps** (visible when `loopT ≥ appear` **or**
`loopT ≤ disappear`). Absent both fields → always-present (Increment-1 behavior).
- **Runtime:** client-side in `renderOverlay` — a window test gates the annotation
before the existing `boxAt(loopT)` placement. Engine unchanged (labels are not in
`RenderPlan`). The **abyss + reef** pool clips (creatures entering/leaving) are
the authored test material.
### 11.3 Progressive LEFT-brain detail tiers (replaces flat `min_level` gating)
The Left knob now drives **how many** labels show **and how analytical** each is —
no fading. Low Left = **fewer, more general** labels; high Left = **more** labels +
**more analytical** detail.
- **Per-object `salience` (14):** how prominent the object is. It sets the Left
level at which the label first appears: `first_level = 5 salience` (salience 4 →
appears at Left 1; salience 1 → only at Left 4). So raising Left brings in
lower-salience objects.
- **Detail tiers (escalate with Left):** the displayed string escalates
`tier 1 general ("eel") → 2 specific ("moray eel") → 3 scientific
(*Gymnothorax*) → 4 scientific + a fact (lifespan)`. The tier shown =
`clamp(left first_level + 1, 1, n_tiers)` — an object shows tier 1 the moment
it appears and climbs one tier per further Left notch until its tiers run out.
- **Strings carry the tiers:** `strings.en[key]` may be a **list** (indexed by
`tier1`) instead of a string; a plain string stays static (back-compat, tier
ignored). Keeps one key per object and the i18n table as the single source.
- **Back-compat:** an annotation with `min_level` and no `salience` keeps flat
gating (`first_level = min_level`, single-tier string). `measure.*` readouts stay
single-tier, gated by their level (a measurement is not a tiered object).
- **No engine change:** tier selection is client-side against the existing
`overlay.level` (the Left knob) already in the `RenderPlan`.
### 11.4 Progressive RIGHT-brain emotion tiers
The affect **vocabulary** gets more sophisticated as Right rises — basic words at
Right 1, nuanced/compound feelings at high Right — mirroring §11.3 for affect.
- **Appearance rule unchanged:** affect surfaces only when **both** knobs are up;
an anchor's `min_level` is tested against `strength = min(left, right)` as today.
- **Emotion tiers:** `strings.en[feel.key]` may be a **list**; the word shown =
`clamp(right, 1, n_tiers)` (the Right knob, read from `dream.strength` already in
the `RenderPlan`). A plain string stays static (back-compat).
- **No engine change:** `AffectOverlay` already carries `strength`; the Right level
comes from `dream.strength`. Tier selection is client-side.
### 11.5 `tools/pipeline/track.py` (hybrid) + simulator author mode
The stage-5 tooling from §3.5 / §7, delivered as a **second PR** after the
experience PR (§8's split note).
- **`track.py`:** a **classical** path (OpenCV CSRT / optical-flow) seeded by a
hand-drawn box on a keyframe, propagated and **sampled to a sparse keyframed
`track`** (normalized `{t, box}` over the loop, §4); plus an **optional,
lazy-imported ML** detect+track path (YOLO+ByteTrack-class) that proposes tracks
the author maps to keys. Both emit the same `track` artifact + an `appear`/
`disappear` window. Pure helpers (normalization, keyframe sampling,
track→manifest) are unit-tested; an **opt-in** integration test runs the real
tracker on a pool clip (mirrors the Increment-1 real-footage test).
- **Author mode (simulator):** reuse the preview UI as the editor — scrub a pool
clip, draw/seed/correct boxes, run the tracker, assign `key` / `salience` /
tiered strings / `appear`-`disappear` / affect anchors, and **write the manifest
entry** (an endpoint upserts via `tools/pipeline/manifest.upsert_clip` and
persists). Semantics (keys, strings, scientific names, facts) are **always
hand-authored** — the tracker only propagates **box geometry** (§3.5).
## 12. Out of scope (YAGNI)
- Generative i2v **ring transitions** for the new edges (deferred; placeholders hold).
- **Audio** sourcing/alteration (separate content-dial channel; already deferred).
- The **Pi/pano** encode + renderer and serial/firmware (simulator-first).
- Reviving the retired coordinate-catalog `Record` drafting.
- Per-frame **dense** tracks (sparse keyframed tracks interpolated at runtime instead).
+67 -30
View File
@@ -1,18 +1,19 @@
"""The alteration engine: a knob vector -> a layered RenderPlan (design §4, §5).
Reconciled slice (2026-06-07): the Right axis is a DISCRETE selection of a
pre-baked, flow-stabilized restyle variant (not a continuous blend), the Left
axis carries its knob LEVEL so a runtime annotation track can pick which labels
show, and a frozen `Calibration` parameterizes the knob->strength curves so they
can be tuned by eye in the simulator and baked into DEFAULT_CALIBRATION.
Right axis (reframed session 0013): a DETERMINISTIC, real-time dream applied live
to the base footage — a STILL altered state, not the old pre-baked SD variant that
churned across the loop (Right-axis dream design,
docs/superpowers/specs/2026-06-22-right-axis-deterministic-dream-design.md). The
Left axis carries its knob LEVEL so a runtime annotation track can pick which
labels show, and a frozen `Calibration` parameterizes the knob->strength gains so
they can be tuned by eye in the simulator and baked into DEFAULT_CALIBRATION.
Layers compose per §4.2:
- Substrate: ColorGrade (Dark/Light mood, center = identity §5) + a pre-baked
Right restyle variant.
- Overlay: AnalyticalOverlay (Left), composited on top at runtime.
- Substrate: ColorGrade (Dark/Light mood, center = identity §5) + the Right Dream
(deterministic luminous haze, applied client-side from `dream.intensity`).
- Overlay: AnalyticalOverlay (Left) + the affect channel, composited on top.
Left and Right stack (different layers); Dark/Light are the two poles of one
mood grade. See docs/superpowers/specs/2026-06-07-reconciled-simulator-
alteration-slice-design.md.
mood grade.
"""
from __future__ import annotations
@@ -33,13 +34,13 @@ class Calibration:
"""Tunable knob->strength curves (settled by eye in the sim, then baked).
- mood_gain: scales the signed Dark/Light tone (result clamped to [-1, 1]).
- overlay_gain: scales the Left overlay intensity (clamped to [0, 1]).
- right_variant_map: knob value (0..4) -> pre-baked Right variant index.
- overlay_gain: scales the Left overlay (and affect) intensity (clamped [0, 1]).
- dream_gain: scales the Right deterministic-dream intensity (clamped [0, 1]).
"""
mood_gain: float = 1.0
overlay_gain: float = 1.0
right_variant_map: tuple = (0, 1, 2, 3, 4)
dream_gain: float = 1.0
# The LOCKED calibration (session 0010, 2026-06-07) — settled by eye in the
@@ -51,15 +52,15 @@ class Calibration:
# full tilt is peaceful on every axis, so no softening is warranted.
# - overlay_gain = 1.0: full Left = opacity 1.0; the HUD is legible, not
# overwhelming, sitting above the grade.
# - right_variant_map linear: the 5 knob notches map 1:1 onto the 5 discrete
# pre-baked Right strengths (0 = raw base).
# These are unity/linear by deliberate choice, not as placeholders. The curve
# stays parameterized so a future re-bake or a different feel is one edit away;
# test_default_calibration_is_locked guards the values from drifting silently.
# - dream_gain = 1.0: full Right = dream intensity 1.0; the deterministic
# luminous haze (Right-axis dream design, session 0013) reaches full strength.
# These are unity by deliberate choice, not as placeholders. The gains stay
# parameterized so a different feel is one edit away; test_default_calibration_is_locked
# guards the values from drifting silently.
DEFAULT_CALIBRATION = Calibration(
mood_gain=1.0,
overlay_gain=1.0,
right_variant_map=(0, 1, 2, 3, 4),
dream_gain=1.0,
)
@@ -86,11 +87,27 @@ class AnalyticalOverlay:
@dataclass(frozen=True)
class Restyle:
"""Right axis (§4.1): selects a pre-baked, flow-stabilized restyle variant.
`variant` is a discrete index (0 = raw base, no restyle)."""
class AffectOverlay:
"""Affect channel (Left x Right): emotion-words surfaced only when BOTH the
analytical (Left) and dreamlike (Right) knobs are up. `strength` is
`min(left, right)` (0..4) — a runtime affect track picks which feeling words
appear by it; `intensity` 0..1 is their opacity. The dream leaking into the
machine's reading (affect-channel design, session 0013)."""
variant: int
strength: int
intensity: float
@dataclass(frozen=True)
class Dream:
"""Right axis: a deterministic, real-time dream applied live to the base
footage — a STILL altered state (Right-axis dream design, session 0013),
NOT the old pre-baked SD variant that churned across the loop. `strength` is
the Right knob (0..4); `intensity` 0..1 is the dream's strength (the client
maps it to the luminous-haze look, the way it maps `grade.tone` to a filter)."""
strength: int
intensity: float
@dataclass(frozen=True)
@@ -99,15 +116,18 @@ class RenderPlan:
grade: ColorGrade
overlay: AnalyticalOverlay
restyle: Restyle
affect: AffectOverlay
dream: Dream
@property
def is_identity(self) -> bool:
"""True when the plan leaves the neutral base un-altered."""
"""True when the plan leaves the neutral base un-altered. Affect needs no
clause: `min(left,right) > 0` implies `left > 0`, so `overlay.level > 0`
already makes a plan with active affect non-identity."""
return (
self.grade.is_identity
and self.overlay.level == 0
and self.restyle.variant == 0
and self.dream.strength == 0
)
@@ -115,8 +135,17 @@ def _overlay_intensity(left: int, cal: Calibration) -> float:
return _clamp(cal.overlay_gain * left / KNOB_MAX, 0.0, 1.0)
def _right_variant(right: int, cal: Calibration) -> int:
return cal.right_variant_map[right]
def _affect_strength(left: int, right: int) -> int:
"""Affect surfaces only when BOTH knobs are up — gated by the smaller one."""
return min(left, right)
def _affect_intensity(left: int, right: int, cal: Calibration) -> float:
return _clamp(cal.overlay_gain * _affect_strength(left, right) / KNOB_MAX, 0.0, 1.0)
def _dream_intensity(right: int, cal: Calibration) -> float:
return _clamp(cal.dream_gain * right / KNOB_MAX, 0.0, 1.0)
def _mood_tone(dark: int, light: int, cal: Calibration) -> float:
@@ -133,7 +162,14 @@ def plan_alteration(
level=coord.left,
intensity=_overlay_intensity(coord.left, calibration),
),
restyle=Restyle(variant=_right_variant(coord.right, calibration)),
affect=AffectOverlay(
strength=_affect_strength(coord.left, coord.right),
intensity=_affect_intensity(coord.left, coord.right, calibration),
),
dream=Dream(
strength=coord.right,
intensity=_dream_intensity(coord.right, calibration),
),
)
@@ -142,6 +178,7 @@ def render_plan_to_dict(plan: RenderPlan) -> dict:
return {
"grade": {"tone": plan.grade.tone},
"overlay": {"level": plan.overlay.level, "intensity": plan.overlay.intensity},
"restyle": {"variant": plan.restyle.variant},
"affect": {"strength": plan.affect.strength, "intensity": plan.affect.intensity},
"dream": {"strength": plan.dream.strength, "intensity": plan.dream.intensity},
"is_identity": plan.is_identity,
}
+30 -1
View File
@@ -50,10 +50,25 @@ class RingError(ValueError):
@dataclass(frozen=True)
class Scale:
"""A node on the ring: a scale of nature and the base clip it plays."""
"""A node on the ring: a scale of nature and the clip(s) it can play.
A scale carries a rotating POOL of vetted clips (content-pipeline design
§11.1): when the viewer lands on the scale, the player picks one member at
random (`pick_clip_id`). `clip_id` is the PRIMARY member (the deterministic
default / pool[0]); `pool` is the full set. A scale written with only
`clip_id` and no `pool` is a pool of one — exactly the Increment-1 behavior,
so old manifests are unchanged.
"""
id: str
clip_id: str
pool: tuple[str, ...] = ()
@property
def members(self) -> tuple[str, ...]:
"""The clip ids this scale can play — the pool, or just `clip_id` when
no pool was authored (a pool of one)."""
return self.pool if self.pool else (self.clip_id,)
@dataclass(frozen=True)
@@ -134,6 +149,20 @@ def scale_at(ring: ScaleRing, index: int) -> Scale:
return ring.scales[index % len(ring)]
def pick_clip_id(scale: Scale, r: float) -> str:
"""Pick one clip id from a scale's rotating pool, given an injected uniform
`r` in [0, 1) (content-pipeline design §11.1).
Randomness is INJECTED so this stays a pure function — testable and shared
with the Pi player; the impure draw (`random.random()`) happens once at the
API/runtime boundary. `r` is clamped into [0, 1) so an off-by-epsilon caller
never indexes out of range; a pool of one always returns its single member.
"""
members = scale.members
r = min(max(r, 0.0), 0.999999)
return members[int(r * len(members))]
def advance_ring(
ring: ScaleRing,
from_index: int,
+7 -5
View File
@@ -3,10 +3,10 @@
Pure decision logic — no mpv, no audio, no serial. Each update() resolves the
desired Playback (which neutral base clip, how it is altered, what audio plays,
at what levels) and returns the Transition from the previous Playback. The
transition KIND encodes design §4.3: the Dark/Light grade and the Left overlay
are continuous runtime ops (LIVE_UPDATE), whereas swapping the clip or the
pre-baked Right restyle variant (a discrete index) needs a CROSSFADE, and
toggling video on/off fades to/from black.
transition KIND encodes design §4.3: the Dark/Light grade, the Left overlay AND
the Right dream (a deterministic, live luminous haze since session 0013) are all
continuous runtime ops (LIVE_UPDATE); only swapping the clip needs a CROSSFADE,
and toggling video on/off fades to/from black.
Knobs no longer *select* a clip (the base library is neutral by construction);
they *transform* it. Which neutral base to show is an injected policy
@@ -107,7 +107,9 @@ class Player:
if next_video and not prev_video:
return TransitionKind.FADE_FROM_BLACK
if next_video and prev_video:
if nxt.clip_id != prev.clip_id or nxt.plan.restyle != prev.plan.restyle:
# The Right dream is a live filter now (not a substrate swap), so it
# no longer forces a crossfade — only a clip (scale) change does.
if nxt.clip_id != prev.clip_id:
return TransitionKind.CROSSFADE
return TransitionKind.LIVE_UPDATE
# both black: only audio/levels could have changed
@@ -0,0 +1,115 @@
# Session 0013.0 — Transcript
> App: human-experience-filter-art
> Start: 2026-06-22T08-34 (PST)
> End: 2026-06-23T11-29 (PST)
> Type: planning-and-executing
> Posture: yolo (autonomous)
> Claude-Session: f0a3044a-e9cb-467e-991f-f1ce618fc418
> Status: **FINALIZED**
> Lands: PR #12 → `main` (merge `23306fa`). Suite 233 passed / 2 skipped.
## Launch prompt
"Let's keep working on the simulation. What's next? Figuring out the labeling for left
brain HUD?" — opened against the stored 0012 next-goal (judge the real forest Right look,
tune the v2v curve). Reconciled live: operator said "I liked the forest Right," which
closed the judgment half and mooted the curve tuning, freeing the session for HUD + a
Right-axis rethink.
## Pre-state
Through session 0012: engine (`player/`) + simulator built; scale ring (cosmos/forest/abyss);
forest carried real flow-stabilized **pre-baked SD Right variants** (14); Left HUD was
functional but never art-directed (plain cyan boxes); Dark/Light were two separate knobs with
`mood_gain`/`overlay_gain` calibration sliders. 228 tests.
## Arc (turn by turn, wrong turns included)
1. **Session gate + claim.** Classified planning-and-executing; claimed ID 0013 from the main
clone (no nested-worktree resolver issue this time). Clean `main` baseline.
2. **Left HUD art-direction pass.** Operator chose "look & feel first". Redesigned the analytical
overlay: corner-bracket targeting reticles (capped arm length), translucent legible label
chips, two sensor channels — cyan `detected.*` with deterministic per-key confidence scores
vs amber `measure.*` with a center crosshair — and a bbox-sized `◉ ANALYSIS · L{n} · {k} OBJ`
status tag that escalates with the Left knob. Judged by eye via headless-Chrome screenshots.
Fixed two self-spotted flaws (status-tag clipping → measure from real `getBBox`; over-long
reticle arms → cap).
3. **Affect / emotion channel (brainstormed → built).** Operator wanted emotions surfaced when
Left AND Right are both up. Brainstormed: trigger = `min(left,right)`; soft ambient violet
words (not boxed); stable per-scene palette, more words as combined strength rises. Built:
`AffectOverlay(strength,intensity)` in the engine (math in Python), own SVG layer in the
client, palettes in the manifest. 5 new unit tests. Design doc committed.
4. **mood_gain "does nothing" (not a bug).** Operator reported it. Traced: it's a gain on the
Dark/Light grade — invisible with both at 0. Added a clarifying hint, then (later) removed the
whole calibration UI anyway.
5. **Stale-cache saga (several rounds — the time sink).** "I don't see the emotions" / "Nothing
is changing" — repeatedly diagnosed as the open tab running stale `app.js` (the live RenderPlan
readout masked it). Added `no-cache` to the dev server; then a real `/dev/version` + 1 s client
poll **live-reload**; then an on-screen error banner + crash-proof `update()`. Confirmed by the
operator: "Works in incognito mode" — code was always correct; environment state was stale.
6. **Right axis reframe — the centerpiece.** Operator: the dreamlike knob is "too fluid... changes
across the loop more than it should." Brainstormed the concept: Right should be a STILL altered
state. Chose a **deterministic real-time filter** over the baked SD churn. Phase 1 (CSS luminous
haze) built — then the operator clarified "same movement as the main video, just stylized": the
haze read as blurred video, not stylized. Pivoted to **Phase 2: a WebGL2 Kuwahara painterly
shader** on the live frames (edge-preserving → crisp motion, holds still across the loop). Caught
and fixed a UV-mapping bug (lower-half streaking) mid-build. Operator: "That looks great. Let's
get even a little more stylized. Max should be pretty trippy" → added saturation + ~45° hue-drift
+ posterize, ramped by `intensity²` so only max goes psychedelic. "Perfect."
7. **Control surface 4→3 knobs.** Operator: make Dark/Light one dial; fold in mood_gain; drop
overlay_gain. Collapsed Dark+Light into one bipolar Mood dial; removed the Calibration section
(gains pinned to the locked `1.0` constants). Sim panel is now Left · Right · Mood.
8. **Processing-time question.** Operator asked compute/min for the coming content pipeline.
Measured bundled ffmpeg at ~8× realtime (~10 s/min). Headline: the reframe killed the dominant
per-clip cost (the SD bake) — per-minute compute is now trivial; the real cost is human authoring.
9. **Wrap.** Committed in three feature commits; PR #12`main`, merged; branch deleted; memory
updated; transcript published.
## Cut state
`main` @ `23306fa`. 233 passed / 2 skipped. Sim runs locally (uvicorn :8000); Right dream is
real-time WebGL (no bake); 3-knob control. Legacy `bake_right_variants.py` + baked `right*.mp4`
parked, unused. Three design docs in `docs/superpowers/specs/` (affect channel; right-axis
deterministic dream; + the Phase-1/2 staging recorded there).
## Deferred decisions
- **No formal implementation-plan artifact** — built by-eye in the fast iteration loop; the two new
design docs are the record. No content repo configured (`CONTENT_REMOTE` empty), so nothing to
`submit-plan.sh`; docs live in-repo and merged.
- **Pipeline (§9):** no PPE/prod stage exists for the simulator (simulator-first phase) — merge to
`main` is the completion; no deploy ran. Not a skipped gate; there is no stage yet.
- **4→3 knob change** is a real experience-design change not yet reflected in the canonical design
SPEC (still describes 4 knobs). Engine keeps dark/light internally. To record in the SPEC later.
- **Legacy SD baker parked, not deleted** — kept for reference; can be removed once the real-time
dream is fully settled.
## Operator plate (what the operator did / decided)
Drove every aesthetic call by eye: liked forest Right; chose HUD look-and-feel; specified emotions
on Left+Right; rejected the "too fluid" SD dream and the blurred haze; approved the painterly
direction and asked for a trippy max; asked to merge Dark/Light into one Mood dial and remove the
calibration knobs; asked the processing-time question; directed commit + push + finalize and to
start a fresh session for the content pipeline.
## Next-session prompt
`/goal` Start the **content pipeline** for the installation: source full neutral video clips +
processing + labeling. **Brainstorm/scope into a content-pipeline spec first**, deciding:
(1) sourcing — which strict-PD clips, which scales (beyond cosmos/forest/abyss); reuse the 0008
strict-PD map (NASA/Hubble, NOAA Ocean Exploration, NPS/USGS = true PD; "free stock" is NOT PD);
(2) clip length + seamless-loop strategy; (3) labeling approach — STATIC boxes (current, 0 compute)
vs MOTION-TRACKED (a detection+tracking pass), hand-authored vs detection-assisted, same for affect
placement; (4) format target (res/fps/codec for the sim now + the Pi later). Anchor fact: the Right
dream is **real-time, no per-clip bake**, so the pipeline is transcode + authoring, not ML baking.
Read `memory/sub-project-3-player-progress.md` + the scales design
(`docs/superpowers/specs/2026-06-07-scales-library-and-right-axis-pipeline-design.md` §1/§3/§4) first.
@@ -0,0 +1,129 @@
# Session 0014.0 — Transcript
> App: human-experience-filter-art
> Start: 2026-06-24T06-58 (PST)
> End: 2026-06-24T08-29 (PST)
> Type: brainstorming
> Posture: careful
> Claude-Session: 50301aba-c534-4a7c-9f33-8b0881920ab2
> Anchor: docs/ROADMAP.md sub-project 3 (Player Runtime) — content pipeline
> Status: **FINALIZED**
## Launch prompt
```
Launch: /goal next → resolved from memory to the recorded Next /goal:
START A NEW SESSION for the content pipeline — sourcing full neutral video clips + processing + labeling. Brainstorm/scope it into a content-pipeline spec FIRST, deciding: (1) sourcing (strict-PD clips, which scales); (2) clip length + seamless-loop strategy; (3) labeling approach (static vs motion-tracked; hand-authored vs detection-assisted; same for affect); (4) format target (res/fps/codec for sim now + Pi later). Anchor fact: Right dream is real-time, no per-clip bake → pipeline is transcode + authoring, NOT ML baking.
```
## Plan
> Anchor: docs/ROADMAP.md sub-project 3 content pipeline. Roadmap-anchored repo
> (no Gitea issue tracker used 00010013); content pipeline is one cohesive
> Feature → design-eligible. See Deferred decisions for the R1 deviation.
**Goal:** Explore the content pipeline (source full neutral PD clips → process →
label/author) and write its Solution-Design spec. Anchor fact: the Right dream is
real-time (session 0013), so the pipeline is **transcode + human authoring**, not
ML baking.
## What happened (arc)
This opened as `/goal next`; the session gate resolved the stored `Next /goal:`
from memory (content pipeline) and routed to **brainstorming** (the goal said
"brainstorm/scope it into a spec first"). Init claimed session **0014** (careful
posture, the brainstorming default), no concurrent sessions, clean `main`.
**Orientation.** Read the roadmap (sub-project 3), the scales-library/right-axis
design (§1/§3/§4), the two session-0013 specs (real-time dream supersedes the SD
bake; affect channel), and the current content shape (`simulator/clips.py`,
`manifest.json`, `setup_scales_media.py`). Confirmed the key simplifier: the Right
dream is real-time, so the pipeline is transcode + authoring, **not** ML baking.
**Brainstorming (careful, agree-up-front → draft-whole → review-whole).** Walked
the four launch-prompt decisions as `AskUserQuestion` checkpoints:
1. Scale set → **lean 5** (cosmos · orbit · forest/land · reef · abyss; microscopic
dropped — thinnest PD-video pool).
2. Loop → **crossfade loop** (tail→head overlap).
3. Labels → **motion-tracked, HYBRID** (ML proposes where it can; hand-seed +
classical tracker elsewhere). Surfaced that label *semantics* must always be
hand-authored (a generic detector can't produce "siphonophore"/"z ≈ 0.04");
tracking only propagates box geometry.
4. Format → **master + 1080p H.264 proxy**.
Drafted the full spec `docs/superpowers/specs/2026-06-24-content-pipeline-design.md`,
pushed to `feature/content-pipeline`, handed over the rendered Gitea URL for
whole-document review. Operator chose **"Approve + plan now"**.
**Planning.** Wrote `docs/superpowers/plans/2026-06-24-content-pipeline-increment-1.md`
(Increment 1 = real footage, no ML): 9 bite-sized TDD tasks. Self-reviewed against
the spec (coverage, types, one dead-variable fix). Operator chose **"Execute now
(subagent-driven)"**.
**Execution (subagent-driven, 2-stage review per group).** Implementer + reviewer
subagents (sonnet), grouped by file coupling:
- Tasks 13 (`tools/pipeline/ffmpeg_ops.py` pure builders) — DONE; review added an
`overlap<=0` test.
- Tasks 45 (`provenance.py`, `manifest.py`) — DONE; review added append-fallback +
N=1 self-wrap tests.
- Task 6 (`run.py` runner + integration test) — the test gated on `shutil.which`
and SKIPPED (ffmpeg not on PATH). Increment 1's goal is to *prove the pipeline on
real footage*, so I hardened the runner to resolve the **bundled imageio-ffmpeg**
(matching `setup_scales_media.py`) and switched duration probing to **cv2** (no
ffprobe dep — Pi-friendly). Re-run **actually exercised real ffmpeg on the real
forest base and surfaced a genuine bug**: `trim` yields `1/0` framerate which
`xfade` rejects (needs CFR) — fixed with `fps=30` after each trim segment. Test
now PASSES (proxy 1920×1080 verified via cv2). Closed an ffprobe-gap minor too.
- Task 7 (client annotation-track interpolation, `app.js`) — DONE; `node --check`
clean, API serves the track. Visual drift deferred to operator by-eye.
- Task 8 (ring 3→5: orbit + reef placeholders) — DONE; `/api/ring` shows 5 scales
+ 5 edges. Ring spin deferred to operator by-eye.
- Task 9 (sourcing doc + ROADMAP + stale-docstring fix) — DONE.
Final whole-branch review: **READY TO MERGE** (no critical/blocking). 246 passed /
2 skipped. Operator chose **"Merge to main now"** → pushed, created **PR #13** via
the Gitea API, merged (delete branch), synced main, deleted local branch. Stamped
the spec **graduated** on main.
## Outcome
- Content-pipeline **spec** (graduated) + **Increment-1 plan** (archived in-repo)
+ **Increment 1 implementation** all merged to `main` (PR #13, merge `bb5ecf0`;
spec stamp `73e5c4a`). Suite **246 passed / 2 skipped**.
- New `tools/pipeline/` package; keyframed annotation-`track` schema
(client-interpolated, backward compatible); scale ring extended 3→5.
- `docs/content-sourcing.md` (strict-PD sourcing procedure); ROADMAP updated.
## Deferred decisions
- **R1 anchor (no Gitea issue):** the brainstorming gate wants a non-`epic`
`type/*` tracker issue as the design anchor. This repo has never used a Gitea
issue tracker (roadmap-driven, 00010013), so the spec is anchored to
`docs/ROADMAP.md` sub-project 3 — matching every prior brainstorming session
here (0005/0007/0008/0013). Flagged rather than force-filing an issue that would
break established repo practice; operator can request issue-tracking if wanted.
- **Session-type blur (operator-directed):** a brainstorming session normally ends
at the spec; the operator explicitly extended it through plan → execute → merge.
Each transition was an explicit operator choice via `AskUserQuestion`, so the
careful-posture gates were honored, not bypassed.
- **Crossfade duration source (minor, non-blocking):** `process_clip` probes the
duration from the *source*, not the post-frame mezzanine; GOP alignment can make
the tail segment ±~33 ms off. ffmpeg handles EOF gracefully; imperceptible on
calm footage. Noted for a future fast-motion clip.
## Pending for next session
- **Operator by-eye review of Increment 1** in the sim: the track drift
(water/fish boxes follow playback + loop seamlessly) and spinning the 5-scale
ring cosmos→orbit→forest→reef→abyss→wrap. A dev sim (session-0013 live-reload)
was running on :8000 during the session.
## Next /goal
```
/goal content-pipeline Increment 2 — source real strict-PD bases per docs/content-sourcing.md (run each through tools/pipeline/run.py process_clip) + build the hybrid motion-track pass tools/pipeline/track.py + a simulator author mode; FIRST do the pending by-eye review of Increment 1. Per docs/superpowers/specs/2026-06-24-content-pipeline-design.md
```
> Still-deferred (tracked in `docs/ROADMAP.md`, this repo's queue): real i2v ring
> transitions (need a generative-video model + adjacent real bases); Pi renderer +
> serial/firmware (simulator-first).
@@ -0,0 +1,85 @@
# Session 0015.0 — Transcript
> App: human-experience-filter-art
> Start: 2026-06-24T13-16 (PST)
> Type: planning-and-executing
> Posture: yolo
> Claude-Session: 50301aba-c534-4a7c-9f33-8b0881920ab2
> Status: **PLACEHOLDER — claimed at session start; finalized at session end.**
>
> This file reserves session ID 0015 for human-experience-filter-art. The driver replaces this
> body with the full transcript and renames the file to its final
> SESSION-0015.0-TRANSCRIPT-2026-06-24T13-16--<end>.md form at session end.
## Launch prompt
```
Process 5 candidate ORBIT (Earth-from-space) clips for the human-experience-filter art installation: resolve URLs, download, vet for text + humans, trim to a word-free calm window, transcode. Clips: orbit_planetearth, orbit_crewobs, orbit_aurora_perth (SVS 31281), orbit_westcoast (SVS 30180), orbit_bluemarble (SVS 4550).
```
## Plan
Process 5 candidate ORBIT (Earth-from-space) clips: resolve URLs, download, vet for text/telemetry/humans via contact sheets, trim to word-free 24s window, transcode via process_clip.
Tasks:
1. Resolve direct mp4 URLs for SVS pages (aurora_perth/31281, westcoast/30180, bluemarble/4550)
2. Download all 5 clips
3. Build contact sheets (overview + first-8s + last-8s) for each clip
4. Vet each clip for text overlays, telemetry, ISS hardware, humans
5. Determine clean window or mark UNUSABLE
6. process_clip → simulator/sample_media/<id>/ for usable clips
7. Generate preview frames
## Clip results
### orbit_planetearth
- URL: https://svs.gsfc.nasa.gov/vis/a030000/a030700/a030771/ISS_View_of_Planet_Earth_2160p.mp4
- Duration: 97.2s
- Text/telemetry: "INTERNATIONAL SPACE STATION" title card t=08s; ISS logo end card t≈8097s
- Humans: none
- Window: 1034s (24s, clean daytime Earth clouds)
- Status: USABLE
- Output: simulator/sample_media/orbit_planetearth/
- Preview: /tmp/hef-cand/orbit_planetearth/preview.png
### orbit_crewobs
- URL: https://svs.gsfc.nasa.gov/vis/a030000/a030700/a030771/ISS_Crew_Earth_Observations_2160p.mp4
- Duration: 66.4s
- Text/telemetry: "INTERNATIONAL SPACE STATION" + date/location title card t=08s; ISS logo end card t≈5566s
- Humans: none
- Window: 933s (24s, clean desert/cloud Earth)
- Status: USABLE
- Output: simulator/sample_media/orbit_crewobs/
- Preview: /tmp/hef-cand/orbit_crewobs/preview.png
### orbit_aurora_perth
- URL: https://svs.gsfc.nasa.gov/vis/a030000/a031200/a031281/ISS067_20220817_aurora_1080p25.mp4
- Duration: 49.6s
- Text/telemetry: Date+timestamp burned in top-left throughout (e.g. "2022-08-17 19:15:25", updating every second)
- Humans: none
- Window: none (telemetry throughout)
- Status: UNUSABLE
### orbit_westcoast
- URL: https://svs.gsfc.nasa.gov/vis/a030000/a030100/a030180/iss028_nighttime_20110819_720p.mp4
- Duration: 54.7s
- Text/telemetry: none
- Humans: none
- ISS hardware: cylindrical ISS module + solar panel grid visible in top-right corner throughout entire clip (confirmed by exhaustive per-second crop inspection t=1444s)
- Window: none (ISS hardware throughout)
- Status: UNUSABLE — orbit_westcoast output dir deleted
### orbit_bluemarble
- URL: https://svs.gsfc.nasa.gov/vis/a000000/a004500/a004550/BlueMarble_starfield_4k_2160p30_v2.mp4
- Duration: 121.0s
- Text/telemetry: none (CG visualization, no overlays anywhere)
- Humans: none
- Window: 1034s (24s, rotating Earth globe on starfield)
- Status: USABLE
- Output: simulator/sample_media/orbit_bluemarble/
- Preview: /tmp/hef-cand/orbit_bluemarble/preview.png
## Deferred decisions
- orbit_westcoast output directory (simulator/sample_media/orbit_westcoast/) was written before the ISS hardware was confirmed throughout. It needs to be deleted. The rm -rf was attempted but permission was denied in the sandbox — operator should delete it manually: `rm -rf simulator/sample_media/orbit_westcoast`
@@ -1,15 +1,15 @@
# Session 0013.0 — Transcript
# Session 0016.0 — Transcript
> App: human-experience-filter-art
> Start: 2026-06-22T08-34 (PST)
> Start: 2026-06-24T17-48 (PST)
> Type: planning-and-executing
> Posture: yolo
> Claude-Session: f0a3044a-e9cb-467e-991f-f1ce618fc418
> Claude-Session: 0fde1d36-7c64-414f-8f49-8669b5dd2263
> Status: **PLACEHOLDER — claimed at session start; finalized at session end.**
>
> This file reserves session ID 0013 for human-experience-filter-art. The driver replaces this
> This file reserves session ID 0016 for human-experience-filter-art. The driver replaces this
> body with the full transcript and renames the file to its final
> SESSION-0013.0-TRANSCRIPT-2026-06-22T08-34--<end>.md form at session end.
> SESSION-0016.0-TRANSCRIPT-2026-06-24T17-48--<end>.md form at session end.
## Launch prompt
+9
View File
@@ -37,5 +37,14 @@
},
"0013": {
"title": ""
},
"0014": {
"title": ""
},
"0015": {
"title": ""
},
"0016": {
"title": ""
}
}
+51 -4
View File
@@ -7,6 +7,10 @@ endpoints (/api/select, /api/catalog/meta) and the X-ray are retired.
from __future__ import annotations
import hashlib
import os
import random
import time
from pathlib import Path
from typing import Optional
@@ -23,13 +27,36 @@ from player.alteration import (
)
from player.content import resolve_content
from player.controls import CONTENT_POSITIONS
from player.ring import DEFAULT_FAST_SPIN_THRESHOLD, advance_ring
from player.ring import (
DEFAULT_FAST_SPIN_THRESHOLD,
advance_ring,
pick_clip_id,
scale_at,
)
from simulator.clips import load_manifest, load_ring, ring_move_to_dict, ring_to_dict
STATIC_DIR = Path(__file__).parent / "static"
MEDIA_DIR = Path(__file__).parent / "sample_media"
DEFAULT_MANIFEST = MEDIA_DIR / "manifest.json"
# Per-process boot token: changes on every server restart so the dev live-reload
# (below) also fires when Python code changes (which needs a restart), not just
# when a static asset's mtime changes.
_BOOT_ID = f"{os.getpid()}-{int(time.time())}"
def _asset_version() -> str:
"""A short digest of the renderer assets' mtimes + this process's boot token.
Changes whenever app.js/style.css/index.html change on disk OR the server is
restarted — the signal the browser polls to know it's running stale code."""
parts = [_BOOT_ID]
for name in ("app.js", "style.css", "index.html"):
try:
parts.append(f"{name}:{(STATIC_DIR / name).stat().st_mtime_ns}")
except OSError:
parts.append(f"{name}:?")
return hashlib.sha1("|".join(parts).encode()).hexdigest()[:16]
class ControlsModel(BaseModel):
content: str
@@ -44,7 +71,7 @@ class ControlsModel(BaseModel):
class CalibrationModel(BaseModel):
mood_gain: float = 1.0
overlay_gain: float = 1.0
right_variant_map: list[int] = [0, 1, 2, 3, 4]
dream_gain: float = 1.0
class AlterationRequest(BaseModel):
@@ -86,7 +113,7 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
Calibration(
mood_gain=req.calibration.mood_gain,
overlay_gain=req.calibration.overlay_gain,
right_variant_map=tuple(req.calibration.right_variant_map),
dream_gain=req.calibration.dream_gain,
)
if req.calibration
else DEFAULT_CALIBRATION
@@ -98,6 +125,12 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
"content": {"audio_source": content.audio_source, "video": content.video},
}
@app.get("/dev/version")
def dev_version():
# Dev live-reload signal: the browser polls this and reloads when it
# changes, so it never silently runs a stale renderer during iteration.
return {"version": _asset_version()}
@app.get("/api/clips")
def api_clips():
return {"clips": [c.to_dict() for c in app.state.clips]}
@@ -118,7 +151,21 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
req.delta,
fast_spin_threshold=DEFAULT_FAST_SPIN_THRESHOLD,
)
return ring_move_to_dict(move, app.state.ring)
# Rotating pool: pick a random member of the LANDED scale (content-pipeline
# §11.1). A delta=0 advance is the initial / re-roll pick (a no-op move that
# still yields a fresh random clip). The pure pick takes injected randomness.
landed = scale_at(app.state.ring, move.to_index)
chosen = pick_clip_id(landed, random.random())
return ring_move_to_dict(move, app.state.ring, chosen)
@app.middleware("http")
async def _no_cache(request, call_next):
# Dev preview server: never let a browser serve a stale app.js/style.css.
# The whole point is by-eye iteration, so a plain refresh must always get
# the latest assets (the heuristic cache otherwise hides JS/CSS edits).
response = await call_next(request)
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
return response
if MEDIA_DIR.exists():
app.mount("/media", StaticFiles(directory=MEDIA_DIR), name="media")
+403
View File
@@ -0,0 +1,403 @@
"""Build the rotating-pool manifest for the 5 ring scales (content-pipeline §11).
This is the reproducible, hand-authored BASELINE for Increment 2's manifest:
each ring scale carries a POOL of vetted clips (docs/content-candidate-pool.md);
the player picks one member at random on landing (player.ring.pick_clip_id). The
authored label/affect content lives here in readable Python and is emitted to
`simulator/sample_media/manifest.json` — far less error-prone than hand-editing 19
clips of JSON, and a clean place for the simulator author mode to regenerate from.
Authoring model (design §11.3 / §11.4):
- LEFT detail tiers: each label carries `salience` (1-4; first shows at Left
level `5 - salience`) and a tiered string list (general -> specific ->
scientific -> +fact) that escalates with the Left knob.
- RIGHT emotion tiers: affect words are a tiered list per feeling (basic ->
compound) escalating with the Right knob; still gated on min(Left, Right).
- TIME-WINDOWED + TRACKED labels: a label may carry `appear`/`disappear`
(loop-normalized) and a keyframed `track`; it shows only while on screen and
its box follows the subject. The abyss + reef pools are the test material.
Strings: a value may be a LIST (indexed by tier-1) or a plain string (static,
back-compat). Affect appears only when both knobs are up.
Usage:
python simulator/build_pool_manifest.py # write manifest.json
python simulator/build_pool_manifest.py --media # + generate new transition placeholders
"""
from __future__ import annotations
import json
import shutil
import subprocess
import sys
from pathlib import Path
MEDIA = Path(__file__).parent / "sample_media"
MANIFEST = MEDIA / "manifest.json"
# --- Pools: scale id -> ordered clip ids (primary first) (content-candidate-pool.md) ---
POOLS: dict[str, list[str]] = {
"cosmos": ["cosmos", "cosmos_galaxies", "cosmos_hudf", "cosmos_xdf"],
"orbit": ["orbit_planetearth", "orbit_crewobs", "orbit_bluemarble"],
"coast": ["coast_birdrock", "coast_surfgrass", "coast_elkbeach", "coast_drakesbeach"],
"reef": ["reef_lionfish", "reef_spawning", "reef_hawkfish", "reef_snapper", "reef_coralspacific"],
"abyss": ["abyss_wow", "abyss_midwaterexp", "abyss_hiding"],
}
# Ring order (large -> small, wraps): cosmos -> orbit -> coast -> reef -> abyss -> cosmos.
RING_ORDER = ["cosmos", "orbit", "coast", "reef", "abyss"]
PD = "public-domain (US Gov, 17 U.S.C. §105)"
PD_NPS = "public-domain (NPS, no © — 17 U.S.C. §105)"
CCBY_STSCI = "CC-BY-class — credit NASA, ESA, STScI"
# --- Per-clip provenance: id -> (title, license, source) ---
META: dict[str, tuple[str, str, str]] = {
# cosmos
"cosmos": ("Orion Nebula flythrough (NASA/JPL-Caltech)", PD,
"NASA/JPL-Caltech — images.nasa.gov JPL-20221122-SOLSYSf-0001 (Orion); trim ~3054s, crossfade-loop"),
"cosmos_galaxies": ("Flying Through Galaxies (NASA SVS)", PD,
"NASA SVS a014950 14950_Galaxies_FlyThrough_4k; trim 832s, crossfade-loop"),
"cosmos_hudf": ("Hubble Ultra Deep Field zoom (NASA/ESA/STScI)", CCBY_STSCI,
"NASA SVS a030687 hudf-b-1920x1080p30; trim 2044s, crossfade-loop"),
"cosmos_xdf": ("eXtreme Deep Field flythrough (NASA/ESA/STScI)", CCBY_STSCI,
"NASA SVS a030681 hxdf_fly-b-1920x1080p30; trim 226s, crossfade-loop"),
# orbit
"orbit_planetearth": ("ISS — View of Planet Earth (NASA SVS)", PD,
"NASA SVS a030771 ISS_View_of_Planet_Earth_2160p; trim 1034s, crossfade-loop"),
"orbit_crewobs": ("ISS — Crew Earth Observations (NASA SVS)", PD,
"NASA SVS a030771 ISS_Crew_Earth_Observations_2160p; trim 933s, crossfade-loop"),
"orbit_bluemarble": ("NPP “Blue Marble” rotating globe (NASA SVS)", PD,
"NASA SVS a004550 BlueMarble_starfield_4k_2160p30_v2; trim 1034s, crossfade-loop"),
# coast
"coast_birdrock": ("Bird rock + ocean, aerial (NPS Channel Islands)", PD_NPS,
"NPS Channel Islands AV 663a2f13; trim 327s, crossfade-loop"),
"coast_surfgrass": ("Surfgrass / coralline tidepool (NPS Cabrillo)", PD_NPS,
"NPS Cabrillo AV de7d1cf2; trim 226s, crossfade-loop"),
"coast_elkbeach": ("Elk resting in coastal fog (NPS Redwood)", PD_NPS,
"NPS Redwood redw-elkbeach_1280x720; trim 2038s, crossfade-loop (720p source)"),
"coast_drakesbeach": ("Elephant seals on dark sand (NPS Point Reyes)", PD_NPS,
"NPS Point Reyes AV 1cfd8165; trim 2035s, crossfade-loop"),
# reef
"reef_lionfish": ("Lionfish hovering over reef (NOAA Fisheries)", PD,
"NOAA Fisheries b-roll VIDEO_ID 4088881464001; trim 1236s, crossfade-loop"),
"reef_spawning": ("Snapper school over sunlit reef (NOAA Fisheries)", PD,
"NOAA Fisheries b-roll VIDEO_ID 1553921798001; trim 5078s, crossfade-loop"),
"reef_hawkfish": ("Humphead parrotfish over reef (NOAA Fisheries)", PD,
"NOAA Fisheries b-roll VIDEO_ID 6039896460001; trim 250274s, crossfade-loop"),
"reef_snapper": ("Red snapper schooling (NOAA Fisheries)", PD,
"NOAA Fisheries b-roll VIDEO_ID 5305427942001; trim 731s, crossfade-loop"),
"reef_coralspacific": ("Pacific coral macro (NOAA Fisheries)", PD,
"NOAA Fisheries b-roll VIDEO_ID 5231464663001; trim 3054s, crossfade-loop"),
# abyss
"abyss_wow": ("“World of Water” (NOAA Ocean Exploration)", PD,
"NOAA Ocean Exploration wow-1280x720-1; trim 1034s, crossfade-loop"),
"abyss_midwaterexp": ("“Midwater Exploration” (NOAA Ocean Exploration)", PD,
"NOAA Ocean Exploration midwater-exploration-1920x1080-1; trim 1640s, crossfade-loop"),
"abyss_hiding": ("“Hiding in the Dark” (NOAA Ocean Exploration)", PD,
"NOAA Ocean Exploration dark-1280x720-1; trim 2044s, crossfade-loop"),
}
# --- Affect vocabulary per SCALE (shared by all the scale's pool members) -------
# Each: (key, [x, y], min_level, [tier1 basic .. tier4 compound]). The word shown
# escalates with the RIGHT knob; appears only when both knobs are up (§11.4).
AFFECT: dict[str, list[tuple]] = {
"cosmos": [
("feel.wonder", [0.50, 0.44], 1, ["wow", "wonder", "awe", "transcendent awe"]),
("feel.vastness", [0.20, 0.70], 2, ["big", "vastness", "immensity", "a dizzying immensity"]),
("feel.insignificance", [0.66, 0.60], 3, ["small", "smallness", "insignificance", "a humbling insignificance"]),
("feel.longing", [0.42, 0.84], 4, ["want", "longing", "yearning", "an aching yearning"]),
],
"orbit": [
("feel.serenity", [0.50, 0.44], 1, ["calm", "serenity", "peace", "a quiet, weightless peace"]),
("feel.fragility", [0.22, 0.68], 2, ["thin", "fragility", "vulnerability", "a tender fragility"]),
("feel.unity", [0.66, 0.58], 3, ["one", "unity", "belonging", "a borderless belonging"]),
("feel.distance", [0.42, 0.82], 4, ["far", "distance", "remoteness", "an exquisite remoteness"]),
],
"coast": [
("feel.ease", [0.50, 0.44], 1, ["nice", "ease", "calm", "an unhurried calm"]),
("feel.nostalgia", [0.22, 0.68], 2, ["miss", "nostalgia", "wistfulness", "a salt-air wistfulness"]),
("feel.belonging", [0.66, 0.58], 3, ["home", "belonging", "rootedness", "a tidal rootedness"]),
("feel.melancholy", [0.42, 0.82], 4, ["sad", "melancholy", "longing", "a soft seaward longing"]),
],
"reef": [
("feel.delight", [0.50, 0.44], 1, ["fun", "delight", "joy", "a darting, bright joy"]),
("feel.abundance", [0.22, 0.68], 2, ["full", "abundance", "richness", "a teeming richness"]),
("feel.curiosity", [0.66, 0.58], 3, ["look", "curiosity", "fascination", "an absorbed fascination"]),
("feel.immersion", [0.42, 0.82], 4, ["in", "immersion", "absorption", "a held, breathless absorption"]),
],
"abyss": [
("feel.unease", [0.50, 0.44], 1, ["uh", "unease", "disquiet", "a creeping disquiet"]),
("feel.fascination", [0.22, 0.68], 2, ["ooh", "fascination", "wonder", "a forbidden wonder"]),
("feel.isolation", [0.66, 0.58], 3, ["alone", "isolation", "solitude", "a crushing solitude"]),
("feel.dread", [0.42, 0.82], 4, ["fear", "dread", "foreboding", "a slow, cold foreboding"]),
],
}
def static_label(key, salience, tiers, box):
"""A fixed-box tiered label (always on-screen, salience-gated by Left)."""
return {"key": key, "salience": salience, "box": box, "_tiers": tiers}
def tracked_label(key, salience, tiers, appear, disappear, track):
"""A time-windowed tracked label: shows only in [appear, disappear] and its
box follows the subject (§11.2). `track` = [(t, [x,y,w,h]), ...]."""
return {
"key": key, "salience": salience, "appear": appear, "disappear": disappear,
"track": [{"t": t, "box": b} for t, b in track], "_tiers": tiers,
}
def measure(key, value, box, min_level):
"""A single-tier measurement readout (not a tiered object), gated by Left."""
return {"key": key, "box": box, "min_level": min_level, "_tiers": value}
# --- Labels per clip. `_tiers` is lifted into the strings table by build(). ------
# Showcase tracked/windowed labels live on the abyss + reef pools (the creatures
# drift through frame). Other clips get a few salience-gated static tiered labels.
LABELS: dict[str, list[dict]] = {
# ---------- cosmos ----------
"cosmos": [
static_label("detected.nebula", 4, ["cloud", "nebula", "emission nebula", "emission nebula · ionized H II region"], [0.32, 0.28, 0.34, 0.36]),
static_label("detected.star", 2, ["star", "young star", "protostar", "protostar · <1 Myr old"], [0.60, 0.30, 0.10, 0.12]),
measure("measure.redshift", "z ≈ 0.0", [0.36, 0.70, 0.18, 0.08], 4),
],
"cosmos_galaxies": [
static_label("detected.galaxy", 4, ["galaxy", "spiral galaxy", "barred spiral", "barred spiral · ~10¹¹ stars"], [0.34, 0.30, 0.30, 0.34]),
measure("measure.distance", "~Mly", [0.06, 0.06, 0.18, 0.08], 3),
],
"cosmos_hudf": [
static_label("detected.galaxy", 4, ["smudge", "galaxy", "early galaxy", "early galaxy · z ≈ 16, light Gyr-old"], [0.40, 0.36, 0.20, 0.22]),
measure("measure.field", "deep field", [0.06, 0.06, 0.2, 0.08], 3),
],
"cosmos_xdf": [
static_label("detected.galaxy", 4, ["dot", "galaxy", "faint galaxy", "faint galaxy · among the earliest seen"], [0.42, 0.38, 0.18, 0.2]),
],
# ---------- orbit ----------
"orbit_planetearth": [
static_label("detected.cloud_band", 4, ["clouds", "cloud band", "cumulus field", "cumulus field · tropical convection"], [0.30, 0.30, 0.30, 0.20]),
static_label("detected.limb", 2, ["edge", "Earths limb", "atmospheric limb", "atmospheric limb · ~100 km of air"], [0.05, 0.70, 0.9, 0.12]),
measure("measure.altitude", "~408 km", [0.06, 0.06, 0.18, 0.08], 3),
],
"orbit_crewobs": [
static_label("detected.coastline", 4, ["land", "coastline", "continental margin", "continental margin · land meets sea"], [0.30, 0.34, 0.34, 0.26]),
measure("measure.altitude", "~408 km", [0.06, 0.06, 0.18, 0.08], 3),
],
"orbit_bluemarble": [
static_label("detected.globe", 4, ["Earth", "the globe", "terrestrial planet", "terrestrial planet · 12,742 km across"], [0.28, 0.18, 0.44, 0.6]),
],
# ---------- coast ----------
"coast_birdrock": [
static_label("detected.surf", 4, ["waves", "surf", "breaking swell", "breaking swell · wind-driven, ~10 s period"], [0.20, 0.55, 0.6, 0.3]),
static_label("detected.searock", 2, ["rock", "sea stack", "coastal sea stack", "sea stack · wave-cut residual rock"], [0.30, 0.25, 0.22, 0.3]),
],
"coast_surfgrass": [
static_label("detected.surfgrass", 4, ["grass", "surfgrass", "Phyllospadix", "Phyllospadix · a marine seagrass, not algae"], [0.18, 0.40, 0.5, 0.4]),
static_label("detected.coralline", 2, ["pink", "coralline algae", "crustose coralline", "crustose coralline · calcified red algae"], [0.6, 0.55, 0.2, 0.2]),
],
"coast_elkbeach": [
static_label("detected.elk", 4, ["elk", "Roosevelt elk", "Cervus canadensis roosevelti", "C. c. roosevelti · largest elk subspecies"], [0.34, 0.42, 0.3, 0.3]),
static_label("detected.fog", 2, ["fog", "coastal fog", "advection fog", "advection fog · warm air over cold upwelling"], [0.05, 0.05, 0.9, 0.25]),
],
"coast_drakesbeach": [
static_label("detected.seal", 4, ["seals", "elephant seals", "Mirounga angustirostris", "M. angustirostris · males to 2,000 kg"], [0.25, 0.55, 0.5, 0.3]),
static_label("detected.sand", 1, ["sand", "dark sand", "mineral-dark beach", "dark beach · eroded coastal sediment"], [0.05, 0.8, 0.9, 0.15]),
],
# ---------- reef (showcase: tracked + windowed) ----------
"reef_lionfish": [
tracked_label(
"detected.lionfish", 4,
["fish", "lionfish", "Pterois volitans", "Pterois volitans · venomous spines, invasive in the Atlantic"],
0.05, 0.95,
[(0.05, [0.15, 0.30, 0.16, 0.18]), (0.5, [0.50, 0.38, 0.16, 0.18]), (0.95, [0.72, 0.34, 0.16, 0.18])],
),
tracked_label(
"detected.snapper", 2,
["fish", "blue snapper", "Lutjanus", "Lutjanus · schools over reef structure"],
0.0, 0.6,
[(0.0, [0.05, 0.55, 0.12, 0.1]), (0.3, [0.22, 0.5, 0.12, 0.1]), (0.6, [0.40, 0.56, 0.12, 0.1])],
),
measure("measure.depth", "18 m", [0.06, 0.06, 0.16, 0.08], 3),
],
"reef_spawning": [
tracked_label(
"detected.school", 4,
["fish", "fish school", "snapper aggregation", "snapper aggregation · synchronized spawning run"],
0.0, 1.0,
[(0.0, [0.20, 0.30, 0.2, 0.18]), (0.5, [0.55, 0.34, 0.2, 0.18]), (1.0, [0.20, 0.30, 0.2, 0.18])],
),
static_label("detected.coral", 2, ["coral", "reef coral", "scleractinian", "scleractinian · reef-building stony coral"], [0.15, 0.6, 0.35, 0.3]),
measure("measure.depth", "22 m", [0.06, 0.06, 0.16, 0.08], 3),
],
"reef_hawkfish": [
tracked_label(
"detected.parrotfish", 4,
["fish", "parrotfish", "Bolbometopon muricatum", "B. muricatum · humphead, grazes reef into sand"],
0.1, 0.9,
[(0.1, [0.55, 0.25, 0.2, 0.22]), (0.5, [0.35, 0.34, 0.2, 0.22]), (0.9, [0.12, 0.40, 0.2, 0.22])],
),
measure("measure.depth", "12 m", [0.06, 0.06, 0.16, 0.08], 3),
],
"reef_snapper": [
tracked_label(
"detected.snapper", 4,
["fish", "red snapper", "Lutjanus campechanus", "L. campechanus · long-lived, can reach 50+ yr"],
0.0, 0.7,
[(0.0, [0.10, 0.3, 0.16, 0.14]), (0.35, [0.40, 0.36, 0.16, 0.14]), (0.7, [0.68, 0.30, 0.16, 0.14])],
),
],
"reef_coralspacific": [
static_label("detected.coral", 4, ["coral", "coral colony", "Pacific scleractinian", "Pacific scleractinian · a symbiosis with algae"], [0.2, 0.4, 0.4, 0.4]),
tracked_label(
"detected.spotfish", 2,
["fish", "spotted fish", "reef damselfish", "reef damselfish · territorial over its coral"],
0.2, 0.8,
[(0.2, [0.6, 0.25, 0.1, 0.09]), (0.5, [0.45, 0.3, 0.1, 0.09]), (0.8, [0.3, 0.26, 0.1, 0.09])],
),
],
# ---------- abyss (showcase: tracked + windowed; creatures drift through) ----------
"abyss_wow": [
tracked_label(
"detected.jelly", 4,
["jelly", "comb jelly", "Ctenophora", "Ctenophora · swims by beating rows of cilia"],
0.0, 0.55,
[(0.0, [0.05, 0.25, 0.14, 0.18]), (0.3, [0.30, 0.32, 0.14, 0.18]), (0.55, [0.52, 0.28, 0.14, 0.18])],
),
tracked_label(
"detected.siphonophore", 3,
["chain", "siphonophore", "Siphonophorae", "Siphonophorae · a colony of clones, can exceed 40 m"],
0.45, 1.0,
[(0.45, [0.80, 0.6, 0.1, 0.3]), (0.7, [0.6, 0.45, 0.1, 0.34]), (1.0, [0.45, 0.3, 0.1, 0.38])],
),
measure("measure.depth", "1,200 m", [0.06, 0.06, 0.16, 0.08], 2),
],
"abyss_midwaterexp": [
tracked_label(
"detected.medusa", 4,
["jelly", "white medusa", "Scyphozoa", "Scyphozoa · the swimming medusa stage of a jellyfish"],
0.0, 0.6,
[(0.0, [0.55, 0.20, 0.16, 0.2]), (0.3, [0.40, 0.34, 0.16, 0.2]), (0.6, [0.22, 0.30, 0.16, 0.2])],
),
tracked_label(
"detected.worm", 2,
["worm", "red worm", "polychaete", "polychaete · a free-swimming bristle worm"],
0.5, 1.0,
[(0.5, [0.10, 0.7, 0.12, 0.1]), (0.75, [0.3, 0.6, 0.12, 0.1]), (1.0, [0.5, 0.66, 0.12, 0.1])],
),
measure("measure.depth", "1,600 m", [0.06, 0.06, 0.16, 0.08], 2),
],
"abyss_hiding": [
tracked_label(
"detected.jelly", 4,
["jelly", "crimson jelly", "Scyphozoa", "Scyphozoa · red is invisible in the lightless deep"],
0.0, 0.7,
[(0.0, [0.15, 0.28, 0.16, 0.2]), (0.35, [0.42, 0.36, 0.16, 0.2]), (0.7, [0.66, 0.30, 0.16, 0.2])],
),
measure("measure.depth", "2,140 m", [0.06, 0.06, 0.16, 0.08], 2),
],
}
# Human-readable strings for affect/measurement keys are produced from the tiered
# data above; this maps measurement keys to nothing extra (their value IS the string).
def _affect_for_clip(scale: str) -> tuple[list, dict]:
"""Build the affect list + its strings for a scale's pool member."""
entries, strings = [], {}
for key, at, min_level, tiers in AFFECT[scale]:
entries.append({"key": key, "at": at, "min_level": min_level})
strings[key] = tiers
return entries, strings
def _labels_for_clip(clip_id: str) -> tuple[list, dict]:
"""Build the annotation list + its strings, lifting `_tiers` into the table."""
anns, strings = [], {}
for raw in LABELS.get(clip_id, []):
a = {k: v for k, v in raw.items() if k != "_tiers"}
anns.append(a)
strings[a["key"]] = raw["_tiers"]
return anns, strings
def _clip_entry(scale: str, clip_id: str) -> dict:
title, license_, source = META[clip_id]
anns, lab_strings = _labels_for_clip(clip_id)
affect, aff_strings = _affect_for_clip(scale)
return {
"id": clip_id,
"title": title,
"base_file": f"{clip_id}/base.mp4",
"license": license_,
"source": source,
"right_variants": {},
"annotations": anns,
"affect": affect,
"strings": {"en": {**lab_strings, **aff_strings}},
}
def build_manifest() -> dict:
"""Assemble the full rotating-pool manifest dict."""
clips = []
for scale in RING_ORDER:
for clip_id in POOLS[scale]:
clips.append(_clip_entry(scale, clip_id))
scales = [{"id": s, "clip_id": POOLS[s][0], "pool": POOLS[s]} for s in RING_ORDER]
transitions = []
n = len(RING_ORDER)
for i in range(n):
a, b = RING_ORDER[i], RING_ORDER[(i + 1) % n]
transitions.append({"file": f"transitions/{a}-{b}.mp4", "model": "placeholder-zoom"})
return {"clips": clips, "ring": {"scales": scales, "transitions": transitions}}
# --- Optional: generate the NEW transition placeholders (orbit-coast, coast-reef) ---
def _ffmpeg() -> str:
if shutil.which("ffmpeg"):
return "ffmpeg"
import imageio_ffmpeg
return imageio_ffmpeg.get_ffmpeg_exe()
def _make_transition(ff: str, scale_a: str, scale_b: str) -> Path:
"""A placeholder zoom/warp morph between two scales, from their PRIMARY pool
members' bases (mirrors setup_scales_media.py but keyed by scale->primary)."""
a = MEDIA / POOLS[scale_a][0] / "base.mp4"
b = MEDIA / POOLS[scale_b][0] / "base.mp4"
out = MEDIA / "transitions" / f"{scale_a}-{scale_b}.mp4"
out.parent.mkdir(parents=True, exist_ok=True)
norm = "trim=0:3,setpts=PTS-STARTPTS,scale=1280:720,fps=25,setsar=1,format=yuv420p"
subprocess.run([
ff, "-y", "-i", str(a), "-i", str(b), "-filter_complex",
f"[0:v]{norm}[a];[1:v]{norm}[b];"
"[a][b]xfade=transition=zoomin:duration=1.5:offset=0.75,format=yuv420p[v]",
"-map", "[v]", "-an", str(out),
], check=True, capture_output=True)
return out
def generate_media() -> None:
"""Generate the new ring edges introduced by the coast scale; the cosmos-orbit,
reef-abyss and abyss-cosmos edges already exist from the prior 5-scale ring."""
ff = _ffmpeg()
for a, b in [("orbit", "coast"), ("coast", "reef")]:
_make_transition(ff, a, b)
print(f"generated transitions/{a}-{b}.mp4 (placeholder zoom)")
def main(argv: list[str]) -> None:
manifest = build_manifest()
MANIFEST.write_text(json.dumps(manifest, indent=2, ensure_ascii=False) + "\n")
print(f"wrote {MANIFEST}{len(manifest['clips'])} clips, "
f"{len(manifest['ring']['scales'])} ring scales")
if "--media" in argv:
generate_media()
if __name__ == "__main__":
main(sys.argv[1:])
+40 -8
View File
@@ -26,6 +26,7 @@ class Clip:
source: str
right_variants: dict # {"1": {"file": ...}, "4": {...}} (no "0")
annotations: list # [{"key", "box":[x,y,w,h], "min_level"}, ...]
affect: list # [{"key", "at":[x,y], "min_level"}, ...] (Left x Right)
strings: dict # {"en": {key: text}}
def variant_file(self, strength: int) -> str:
@@ -46,6 +47,7 @@ class Clip:
"source": self.source,
"right_variants": variants,
"annotations": self.annotations,
"affect": self.affect,
"strings": self.strings,
}
@@ -59,6 +61,7 @@ def _clip_from_dict(d: dict[str, Any]) -> Clip:
source=d.get("source", ""),
right_variants=d.get("right_variants", {}),
annotations=d.get("annotations", []),
affect=d.get("affect", []),
strings=d.get("strings", {}),
)
@@ -70,6 +73,18 @@ def load_manifest(path: str | Path) -> list[Clip]:
return [_clip_from_dict(c) for c in data["clips"]]
def _scale_from_dict(s: dict[str, Any]) -> Scale:
"""A ring scale node, with its rotating clip pool (content-pipeline §11.1).
Accepts a `pool` list (the rotating model) and/or a legacy single `clip_id`.
The primary `clip_id` is the explicit one if given, else the first pool
member; `pool` defaults to `(clip_id,)` for a pre-pool manifest.
"""
pool = tuple(s.get("pool", []))
clip_id = s.get("clip_id") or (pool[0] if pool else "")
return Scale(id=s["id"], clip_id=clip_id, pool=pool)
def load_ring(path: str | Path) -> ScaleRing | None:
"""Load the optional `ring` section as a `ScaleRing`, or None if absent.
@@ -82,9 +97,7 @@ def load_ring(path: str | Path) -> ScaleRing | None:
ring = data.get("ring")
if not ring:
return None
scales = tuple(
Scale(id=s["id"], clip_id=s["clip_id"]) for s in ring.get("scales", [])
)
scales = tuple(_scale_from_dict(s) for s in ring.get("scales", []))
transitions = tuple(
Transition(file=t["file"], model=t.get("model", ""))
for t in ring.get("transitions", [])
@@ -94,27 +107,46 @@ def load_ring(path: str | Path) -> ScaleRing | None:
def ring_to_dict(ring: ScaleRing, clips: list[Clip]) -> dict:
"""JSON form of the ring for the API: ordered scales (with their clip title)
and the per-edge transitions."""
and the per-edge transitions.
Each scale carries its full rotating `pool` (clip id + title per member) plus
the primary `clip_id`/`title` for back-compat. The client lands on a member
chosen at the API boundary (`/api/ring/advance` → `target_clip_id`)."""
titles = {c.id: c.title for c in clips}
return {
"scales": [
{"id": s.id, "clip_id": s.clip_id, "title": titles.get(s.clip_id, s.id)}
{
"id": s.id,
"clip_id": s.clip_id,
"title": titles.get(s.clip_id, s.id),
"pool": [
{"clip_id": cid, "title": titles.get(cid, cid)}
for cid in s.members
],
}
for s in ring.scales
],
"transitions": [{"file": t.file, "model": t.model} for t in ring.transitions],
}
def ring_move_to_dict(move: RingMove, ring: ScaleRing) -> dict:
def ring_move_to_dict(
move: RingMove, ring: ScaleRing, chosen_clip_id: str | None = None
) -> dict:
"""JSON form of an encoder move: the landing scale's clip and the ordered
transition clips to play (with direction). `fast` flags a collapsed fast-spin
pass; the single step then carries `blended` so the renderer plays it quick."""
pass; the single step then carries `blended` so the renderer plays it quick.
`target_clip_id` is the pool member the player should load on landing. The
caller passes the random pick (`pick_clip_id(landed_scale, random.random())`,
content-pipeline §11.1); when omitted it falls back to the scale's primary
`clip_id` (deterministic — keeps pre-pool callers/tests working)."""
return {
"from_index": move.from_index,
"to_index": move.to_index,
"wrapped": move.wrapped,
"fast": move.fast,
"target_clip_id": scale_at(ring, move.to_index).clip_id,
"target_clip_id": chosen_clip_id or scale_at(ring, move.to_index).clip_id,
"steps": [
{
"edge": st.edge,
File diff suppressed because it is too large Load Diff
+25 -12
View File
@@ -1,16 +1,26 @@
"""Generate cheap placeholder media for the scale RING (scales design §3).
"""Generate cheap placeholder media for the full 5-scale RING (scales design §3).
The ring needs >=2 neutral scales to be demonstrable. This script produces
cheap, offline, synthetic placeholders for the two true-PD scales the ring adds —
**cosmos** (NASA/Hubble) and **abyss** (NOAA Ocean Exploration) — plus the short
zoom/warp **transition** clips between adjacent scales. Forest reuses the real
POC base from `setup_sample_media.py` (or a synthetic placeholder if absent).
SUPERSEDED (session 0016): the ring now uses real strict-PD footage in a ROTATING
POOL per scale, regenerated by `simulator/build_pool_manifest.py` (which also makes
the new coast-edge transition placeholders). This generator built the older
one-base-per-scale forest ring (cosmos/orbit/forest/reef/abyss) and is kept only
for reference / a clean-room placeholder rebuild; it does NOT match the current
pool manifest or ring order (cosmos → orbit → coast → reef → abyss).
Produces offline synthetic placeholders for all five ring scales — **cosmos**
(NASA/Hubble), **orbit** (NASA/ISS), **forest** (NPS/USGS), **reef** (NOAA Ocean
Exploration), and **abyss** (NOAA Ocean Exploration) — plus the short zoom/warp
**transition** clips between adjacent scales. Forest reuses the real POC base from
`setup_sample_media.py` (or a synthetic placeholder if absent); all others are
generated as labelled solid-tint clips.
This is deliberately the CHEAP path: the manifest records the real true-PD
provenance (NASA 17 U.S.C. §105 / NOAA PD), but the bytes here are labelled
synthetic placeholders for look-tuning the ring navigation. The expensive real
multi-strength flow-stabilized Right re-bake is DEFERRED until the ring is liked,
so the new scales carry a raw base only (no Right variants).
provenance (NASA / NPS / USGS / NOAA, 17 U.S.C. §105), but the bytes here are
labelled synthetic placeholders for look-tuning the ring navigation. The Right dream
is real-time (Kuwahara WebGL shader), so no per-clip ML bake is needed; each scale
carries a raw base only. Real strict-PD bases replace these placeholders via the
content pipeline (see `docs/content-sourcing.md`).
Usage: python simulator/setup_scales_media.py
Requires: ffmpeg on PATH (or `pip install imageio-ffmpeg`).
@@ -31,12 +41,15 @@ FPS = 25
# Each scale: (dir, base color, on-screen placeholder label).
SCALES = {
"cosmos": ("0x05060f", "COSMOS — NASA/Hubble (PD placeholder)"),
"forest": ("0x12301a", "FOREST — Yosemite (POC/placeholder)"),
"orbit": ("0x081427", "ORBIT — NASA/ISS (PD placeholder)"),
"forest": ("0x12301a", "FOREST — NPS/USGS (POC/placeholder)"),
"reef": ("0x06303a", "REEF — NOAA Ocean Exploration (PD placeholder)"),
"abyss": ("0x021016", "ABYSS — NOAA Ocean Exploration (PD placeholder)"),
}
# Ring edges (adjacent pairs + the abyss->cosmos wrap), matching the manifest.
EDGES = [("cosmos", "forest"), ("forest", "abyss"), ("abyss", "cosmos")]
EDGES = [("cosmos", "orbit"), ("orbit", "forest"), ("forest", "reef"),
("reef", "abyss"), ("abyss", "cosmos")]
def _ffmpeg() -> str:
+385 -58
View File
@@ -1,66 +1,317 @@
// Thin renderer: post controls+calibration -> RenderPlan; render grade, Right
// variant crossfade, and the live Left overlay. All alteration math stays in
// Python. The scale RING (endless encoder) is navigated via /api/ring/advance —
// Python owns the step/wrap/transition-selection math; the browser only plays
// the returned transition clip(s) then settles on the target scale's clip.
// Thin renderer: post controls+calibration -> RenderPlan; render the mood grade,
// the live Right DREAM (a deterministic luminous haze, NOT a baked variant — it
// holds still across the loop), and the live Left overlay + affect. All alteration
// math stays in Python. The scale RING (endless encoder) is navigated via
// /api/ring/advance — Python owns the step/wrap/transition-selection math; the
// browser only plays the returned transition clip(s) then settles on the target.
const $ = (id) => document.getElementById(id);
const vid = $("vid"), tint = $("tint"), overlay = $("overlay"), black = $("black"), readout = $("readout");
const vid = $("vid"), paint = $("paint"), tint = $("tint"), overlay = $("overlay"),
black = $("black"), readout = $("readout");
const affectLayer = $("affect");
// Right-dream state, read by the painterly render loop each frame. update() (debounced
// on knob moves) writes these; the WebGL loop renders the live video continuously.
let dreamIntensity = 0; // 0..1 — painterly strength (and dream pastel/luminous)
let gradeFilter = "none"; // CSS filter string for the Dark/Light mood grade
// Surface any uncaught JS error on-screen (a silent throw in update() otherwise
// looks like "nothing is changing" — the render dies but the page sits there).
function showError(msg) {
let b = document.getElementById("err-banner");
if (!b) {
b = document.createElement("div");
b.id = "err-banner";
b.style.cssText = "position:fixed;top:0;left:0;right:0;z-index:9999;background:#a00;" +
"color:#fff;font:12px/1.4 monospace;padding:6px 10px;white-space:pre-wrap;";
document.body.appendChild(b);
}
b.textContent = "⚠ " + msg;
}
window.addEventListener("error", (e) =>
showError(`${e.message} @ ${(e.filename || "").split("/").pop()}:${e.lineno}`));
let clipsById = {}; // id -> clip manifest entry
let ring = null; // {scales:[{id,clip_id,title}], transitions:[...]} or null
let ring = null; // {scales:[{id,clip_id,title,pool:[...]}], transitions:[...]} or null
let serverRing = false; // true when /api/ring served a real ring (vs the synthesized fallback)
let ringIndex = 0; // current scale position on the ring
let currentVariant = -1; // last loaded Right strength (reset on clip change)
let activeClipId = null; // the pool member chosen for the current scale landing
let currentClipId = null; // base media currently loaded (reset to force a reload)
let busy = false; // true while a ring transition is playing
let lastOverlay = null; // {level, intensity} from the most-recent renderOverlay call; drives per-frame track animation
async function loadData() {
const clips = (await (await fetch("/api/clips")).json()).clips || [];
clipsById = Object.fromEntries(clips.map((c) => [c.id, c]));
const r = await fetch("/api/ring");
serverRing = r.ok;
ring = r.ok ? await r.json() : null;
if (!ring && clips.length) {
// No ring: single-clip mode — synthesize a 1-scale ring over clip 0.
ring = { scales: [{ id: clips[0].id, clip_id: clips[0].id, title: clips[0].title }], transitions: [] };
// No ring: single-clip mode — synthesize a 1-scale ring (pool of one) over clip 0.
ring = { scales: [{ id: clips[0].id, clip_id: clips[0].id, title: clips[0].title, pool: [{ clip_id: clips[0].id, title: clips[0].title }] }], transitions: [] };
}
ringIndex = 0;
}
// Land on the current scale: pick a random pool member (content-pipeline §11.1).
// A real server ring picks via a delta=0 advance (Python-canonical pick); the
// synthesized fallback ring has a pool of one, so it just takes that member.
async function landScale() {
const scale = ring && ring.scales[ringIndex];
if (!scale) { activeClipId = null; return; }
if (serverRing) {
try {
const resp = await fetch("/api/ring/advance", {
method: "POST", headers: { "content-type": "application/json" },
body: JSON.stringify({ from_index: ringIndex, delta: 0 }),
});
if (resp.ok) { activeClipId = (await resp.json()).target_clip_id; currentClipId = null; return; }
} catch (_) { /* fall through to the primary member */ }
}
activeClipId = scale.clip_id;
currentClipId = null;
}
function activeClip() {
if (!ring) return null;
return clipsById[ring.scales[ringIndex].clip_id] || null;
return clipsById[activeClipId] || null;
}
function mediaUrl(file) { return "/media/" + file; }
function variantFile(strength) {
// Load the active scale's BASE footage into the video (the painterly canvas reads
// its frames live). The Right knob no longer swaps the source — it only drives the
// deterministic dream restyle — so media reloads only when the SCALE changes.
function ensureClipMedia() {
const clip = activeClip();
if (!clip) return "";
const v = clip.right_variants[String(strength)];
return v ? v.file : clip.base_file;
if (!clip || clip.id === currentClipId) return;
currentClipId = clip.id;
vid.src = mediaUrl(clip.base_file);
vid.loop = true;
vid.muted = true;
vid.play().catch(() => {});
vid.style.opacity = "1";
}
function applyGrade(tone) {
// Light: warm + brighten (sepia). Dark: cool + darken via a multiply-blended
// blue wash (#tint) that lifts shadows toward blue while keeping natural
// greens — the peaceful POC dark look, NOT a full-frame hue spin.
// Compose the video look. The Right DREAM is now a painterly WebGL restyle of the
// live frames (see the render loop below) — so here we only stash its intensity for
// that loop, and build the Dark/Light mood grade as a CSS filter. The grade rides
// on the painterly canvas (or #vid in the WebGL-less fallback). No blur: the dream
// must keep the base's crisp motion, just stylized.
function applyVideoLook(tone, dream) {
const warm = tone > 0 ? tone : 0, cool = tone < 0 ? -tone : 0;
const bright = 1 + 0.25 * warm - 0.35 * cool;
const sat = 1 + 0.15 * warm - 0.30 * cool;
vid.style.filter =
`brightness(${bright.toFixed(3)}) saturate(${sat.toFixed(3)}) ` +
`sepia(${(warm * 0.5).toFixed(3)})`;
const sepia = warm * 0.5;
gradeFilter = `brightness(${bright.toFixed(3)}) saturate(${sat.toFixed(3)}) sepia(${sepia.toFixed(3)})`;
dreamIntensity = dream;
tint.style.opacity = (cool * 0.6).toFixed(3);
// WebGL-less fallback: the canvas is hidden, so grade the visible #vid directly.
if (!paintOK) vid.style.filter = gradeFilter;
}
function loadVariant(strength) {
if (strength === currentVariant) return;
currentVariant = strength;
vid.style.opacity = "0";
setTimeout(() => {
vid.src = mediaUrl(variantFile(strength));
vid.loop = true;
vid.play().catch(() => {});
vid.style.opacity = "1";
}, 150);
// --- Right dream: real-time painterly restyle (WebGL2 Kuwahara) ---
// An edge-preserving "oil painting" filter on the LIVE video frames: each pixel
// becomes the mean of whichever surrounding quadrant is most uniform, flattening
// the image into painted regions WHILE KEEPING EDGES — so motion stays as crisp as
// the source. Strength + a gentle pastel/luminous dream-grade scale with intensity.
// Deterministic (a pure function of each frame) => the dream holds still across the
// loop; only the footage's own motion moves.
let paintOK = false;
const PAINT_VS = `#version 300 es
in vec2 p; out vec2 v_uv;
void main(){
// Fullscreen triangle (verts to +3); map the visible [-1,1] to uv [0,1] with Y
// flipped for the video's top-left origin. (Visible region stays in range; the
// off-screen excess is clipped.)
v_uv = vec2(p.x * 0.5 + 0.5, 0.5 - p.y * 0.5);
gl_Position = vec4(p, 0.0, 1.0);
}`;
const PAINT_FS = `#version 300 es
precision highp float;
in vec2 v_uv; out vec4 frag;
uniform sampler2D u_tex; uniform vec2 u_texel; uniform float u_amt;
#define R 6
#define QUAD(x0,x1,y0,y1,MO,VO) { vec3 m=vec3(0.0),s=vec3(0.0); \
for(int j=y0;j<=y1;j++){ for(int i=x0;i<=x1;i++){ \
vec3 c=texture(u_tex, v_uv+vec2(float(i),float(j))*u_texel).rgb; m+=c; s+=c*c; } } \
float n=float((x1-x0+1)*(y1-y0+1)); m/=n; vec3 vv=s/n-m*m; MO=m; VO=vv.r+vv.g+vv.b; }
vec3 hueRotate(vec3 c, float a){
float co=cos(a), si=sin(a);
return vec3(
c.r*(0.299+0.701*co+0.168*si) + c.g*(0.587-0.587*co+0.330*si) + c.b*(0.114-0.114*co-0.497*si),
c.r*(0.299-0.299*co-0.328*si) + c.g*(0.587+0.413*co+0.035*si) + c.b*(0.114-0.114*co+0.292*si),
c.r*(0.299-0.300*co+1.250*si) + c.g*(0.587-0.588*co-1.050*si) + c.b*(0.114+0.886*co-0.203*si));
}
void main(){
vec3 base = texture(u_tex, v_uv).rgb;
if (u_amt <= 0.001) { frag = vec4(base, 1.0); return; }
vec3 m0,m1,m2,m3; float v0,v1,v2,v3;
QUAD(-R,0,-R,0,m0,v0) QUAD(0,R,-R,0,m1,v1) QUAD(-R,0,0,R,m2,v2) QUAD(0,R,0,R,m3,v3)
vec3 painted=m0; float mv=v0;
if(v1<mv){mv=v1;painted=m1;} if(v2<mv){mv=v2;painted=m2;} if(v3<mv){mv=v3;painted=m3;}
vec3 styl = mix(base, painted, u_amt); // painterly restyle, linear in the knob
// The "trippy" terms ramp with t = amt^2, so low Right stays gently dreamy and
// only MAX goes psychedelic: vivid saturation, a surreal hue drift, poster banding.
float t = u_amt * u_amt;
float luma = dot(styl, vec3(0.299, 0.587, 0.114));
styl = mix(vec3(luma), styl, 1.0 + 1.1 * t); // saturation: vivid toward max
styl = hueRotate(styl, 0.8 * t); // surreal hue drift (~45deg at max)
float levels = mix(255.0, 6.0, t); // posterize: color banding toward max
styl = floor(styl * levels + 0.5) / levels;
styl *= 1.0 + 0.06 * u_amt; // luminous lift
frag = vec4(clamp(styl, 0.0, 1.0), 1.0);
}`;
let gl = null, uAmt = null, uTexel = null;
function _shader(kind, src) {
const s = gl.createShader(kind);
gl.shaderSource(s, src); gl.compileShader(s);
if (!gl.getShaderParameter(s, gl.COMPILE_STATUS)) throw new Error(gl.getShaderInfoLog(s));
return s;
}
function initPaint() {
gl = paint.getContext("webgl2");
if (!gl) { paint.style.display = "none"; return false; } // CSS fallback grades #vid
const prog = gl.createProgram();
gl.attachShader(prog, _shader(gl.VERTEX_SHADER, PAINT_VS));
gl.attachShader(prog, _shader(gl.FRAGMENT_SHADER, PAINT_FS));
gl.bindAttribLocation(prog, 0, "p"); gl.linkProgram(prog);
if (!gl.getProgramParameter(prog, gl.LINK_STATUS)) throw new Error(gl.getProgramInfoLog(prog));
gl.useProgram(prog);
const buf = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, buf);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-1, -1, 3, -1, -1, 3]), gl.STATIC_DRAW);
gl.enableVertexAttribArray(0); gl.vertexAttribPointer(0, 2, gl.FLOAT, false, 0, 0);
const t = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, t);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
uAmt = gl.getUniformLocation(prog, "u_amt");
uTexel = gl.getUniformLocation(prog, "u_texel");
paintOK = true;
requestAnimationFrame(paintLoop);
return true;
}
function paintLoop() {
if (paintOK && vid.readyState >= 2 && vid.videoWidth) {
const w = vid.videoWidth, h = vid.videoHeight;
if (paint.width !== w || paint.height !== h) { paint.width = w; paint.height = h; }
gl.viewport(0, 0, w, h);
try { gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGB, gl.RGB, gl.UNSIGNED_BYTE, vid); } catch (_) {}
// No painterly restyle during a ring transition (busy) — show it raw.
gl.uniform1f(uAmt, busy ? 0 : dreamIntensity);
gl.uniform2f(uTexel, 1 / w, 1 / h);
gl.drawArrays(gl.TRIANGLES, 0, 3);
paint.style.filter = busy ? "none" : gradeFilter;
}
// Animate per-frame: re-place keyframed tracks AND re-evaluate time-windowed
// labels (a label enters/leaves frame as the loop plays) against playback time.
const c = activeClip();
if (lastOverlay && lastOverlay.level > 0 && c &&
c.annotations.some((a) => (a.track && a.track.length) ||
typeof a.appear === "number" || typeof a.disappear === "number")) {
renderOverlay(lastOverlay.level, lastOverlay.intensity);
}
requestAnimationFrame(paintLoop);
}
const SVGNS = "http://www.w3.org/2000/svg";
function svg(tag, attrs, parent) {
const el = document.createElementNS(SVGNS, tag);
for (const k in attrs) el.setAttribute(k, attrs[k]);
if (parent) parent.appendChild(el);
return el;
}
// Deterministic per-key "confidence" in [0.78, 0.99] — stable across renders so
// a detection always reports the same score (looks like a real model, not noise).
function confidence(key) {
let h = 0;
for (let i = 0; i < key.length; i++) h = (h * 31 + key.charCodeAt(i)) >>> 0;
return (0.78 + (h % 2200) / 10000).toFixed(2);
}
// Four L-shaped corner brackets framing [x,y,w,h] — a targeting reticle rather
// than a plain rectangle. Arm length scales to the box but is clamped legible.
function reticle(x, y, w, h, cls, parent) {
const L = Math.min(Math.max(Math.min(w, h) * 0.28, 1.4), 4);
const corners = [
[[x + L, y], [x, y], [x, y + L]],
[[x + w - L, y], [x + w, y], [x + w, y + L]],
[[x + L, y + h], [x, y + h], [x, y + h - L]],
[[x + w - L, y + h], [x + w, y + h], [x + w, y + h - L]],
];
for (const pts of corners) {
svg("polyline", { points: pts.map((p) => p.join(",")).join(" "), class: cls }, parent);
}
}
// A label chip: translucent plate + monospace text, anchored top-left of the box.
// `conf` (detections only) is appended as a dimmer score tspan.
function chip(x, y, label, conf, kind, parent) {
const fs = 2.4, pad = 0.6;
const chars = label.length + (conf ? conf.length + 1 : 0);
const w = chars * fs * 0.6 + pad * 2, h = fs + pad * 1.4;
const cy = Math.max(y - h - 0.6, 0.4);
svg("rect", { x, y: cy, width: w, height: h, rx: 0.4, class: "hud-chip-bg " + kind }, parent);
const t = svg("text", { x: x + pad, y: cy + h - pad * 1.1, class: "hud-chip-text " + kind }, parent);
t.appendChild(document.createTextNode(label));
if (conf) {
const sp = svg("tspan", { class: "hud-conf" }, t);
sp.textContent = " " + conf;
}
}
// Interpolate an annotation's box at loop-normalized time t (0..1). Static labels
// (no track) return their fixed box. Linear between sorted keyframes; clamps to ends.
function boxAt(a, t) {
if (!a.track || !a.track.length) return a.box;
const ks = a.track;
if (t <= ks[0].t) return ks[0].box;
if (t >= ks[ks.length - 1].t) return ks[ks.length - 1].box;
for (let i = 1; i < ks.length; i++) {
if (t <= ks[i].t) {
const a0 = ks[i - 1], a1 = ks[i];
const f = (t - a0.t) / (a1.t - a0.t);
return a0.box.map((v, j) => v + (a1.box[j] - v) * f);
}
}
return ks[ks.length - 1].box;
}
// Loop-normalized playback time of the base video (0..1), for track interpolation.
function loopT() {
return vid && vid.duration ? (vid.currentTime % vid.duration) / vid.duration : 0;
}
// --- Progressive labels & emotions (content-pipeline §11.3 / §11.4) ---
const clamp = (v, lo, hi) => Math.max(lo, Math.min(hi, v));
// The Left level at which a label first appears. From `salience` (1..4): high
// salience appears early (first_level = 5 - salience), so raising Left brings in
// lower-salience objects. Falls back to a legacy flat `min_level`, else 1.
function firstLevel(a) {
if (typeof a.salience === "number") return clamp(5 - a.salience, 1, 4);
if (typeof a.min_level === "number") return a.min_level;
return 1;
}
// A tiered string escalates with a knob: a LIST is indexed by tier-1 (clamped to
// the available tiers); a plain string is static (tier ignored — back-compat).
function pickTier(value, tier) {
if (Array.isArray(value)) return value[clamp(tier - 1, 0, value.length - 1)];
return value;
}
function tierCount(value) { return Array.isArray(value) ? value.length : 1; }
// Time-windowed presence (§11.2): a label with `appear`/`disappear` (loop-norm t)
// shows only while on screen. A `disappear < appear` window wraps across the loop
// seam (visible when t >= appear OR t <= disappear). No window => always present.
function inWindow(a, t) {
const hasA = typeof a.appear === "number", hasD = typeof a.disappear === "number";
if (!hasA && !hasD) return true;
const ap = hasA ? a.appear : 0, dis = hasD ? a.disappear : 1;
return ap <= dis ? (t >= ap && t <= dis) : (t >= ap || t <= dis);
}
function renderOverlay(level, intensity) {
@@ -69,57 +320,113 @@ function renderOverlay(level, intensity) {
if (!clip || level <= 0) { overlay.style.opacity = "0"; return; }
overlay.style.opacity = String(intensity);
const strings = (clip.strings && clip.strings.en) || {};
const t = loopT();
let shown = 0;
for (const a of clip.annotations) {
if (a.min_level > level) continue;
const [x, y, w, h] = a.box.map((n) => n * 100);
const rect = document.createElementNS("http://www.w3.org/2000/svg", "rect");
rect.setAttribute("x", x); rect.setAttribute("y", y);
rect.setAttribute("width", w); rect.setAttribute("height", h);
rect.setAttribute("class", "anno-box");
overlay.appendChild(rect);
const text = document.createElementNS("http://www.w3.org/2000/svg", "text");
text.setAttribute("x", x + 0.5); text.setAttribute("y", Math.max(y - 0.5, 2));
text.setAttribute("class", "anno-label");
text.textContent = strings[a.key] || a.key;
overlay.appendChild(text);
const first = firstLevel(a);
if (level < first) continue; // salience gating: low Left = fewer/general labels
if (!inWindow(a, t)) continue; // time-windowed: only while the object is on screen
shown++;
const [x, y, w, h] = boxAt(a, t).map((n) => n * 100);
const measure = a.key.startsWith("measure.");
const kind = measure ? "measure" : "detect";
reticle(x, y, w, h, "hud-reticle " + kind, overlay);
if (measure) {
// Point measurement: a center crosshair tick to read as "sampled here".
const cx = x + w / 2, cyc = y + h / 2;
svg("line", { x1: cx - 1, y1: cyc, x2: cx + 1, y2: cyc, class: "hud-reticle measure" }, overlay);
svg("line", { x1: cx, y1: cyc - 1, x2: cx, y2: cyc + 1, class: "hud-reticle measure" }, overlay);
}
// LEFT detail tier: a detection escalates general -> scientific+fact as Left
// rises above where it first appeared; measurements stay single-tier.
const raw = strings[a.key];
const tier = measure ? 1 : clamp(level - first + 1, 1, tierCount(raw));
const label = pickTier(raw !== undefined ? raw : a.key, tier);
chip(x, y, label, measure ? null : confidence(a.key), kind, overlay);
}
// Global status tag — the machine announcing it is analyzing, escalating with Left.
// Right-anchored at the frame edge; the plate is sized from the text's real bbox
// (the ◉/· glyphs are wider than monospace, so an estimate clips).
if (shown) {
const pad = 0.8;
const st = svg("text", { x: 98, y: 5, "text-anchor": "end", class: "hud-status" }, overlay);
st.textContent = "◉ ANALYSIS · L" + level + " · " + shown + " OBJ";
const b = st.getBBox();
const bg = svg("rect", { x: b.x - pad, y: b.y - pad * 0.6, width: b.width + pad * 2,
height: b.height + pad * 1.2, rx: 0.4, class: "hud-chip-bg detect" });
overlay.insertBefore(bg, st);
}
lastOverlay = { level, intensity };
}
// Affect channel: soft, glowing emotion-words surfaced only when BOTH knobs are
// up. `strength` = min(left, right); `intensity` is the layer opacity. Words are
// placed at authored scene points (no boxes — feelings are scene-level) and read
// softer than the clinical reticles — the dream leaking into the machine's read.
function renderAffect(strength, intensity, right) {
const clip = activeClip();
if (!affectLayer) return; // tolerate a stale page missing the affect layer
affectLayer.innerHTML = "";
if (!clip || !clip.affect || strength <= 0) { affectLayer.style.opacity = "0"; return; }
affectLayer.style.opacity = String(intensity);
const strings = (clip.strings && clip.strings.en) || {};
for (const f of clip.affect) {
if (f.min_level > strength) continue;
const [x, y] = f.at.map((n) => n * 100);
const t = svg("text", { x, y, "text-anchor": "middle", class: "hud-affect" }, affectLayer);
// RIGHT emotion tier: the vocabulary escalates basic -> compound as the Right
// knob rises (appearance still gated by min(left,right) above).
const raw = strings[f.key];
t.textContent = pickTier(raw !== undefined ? raw : f.key, clamp(right || 0, 1, tierCount(raw)));
}
}
function renderScaleReadout() {
if (!ring) return;
const s = ring.scales[ringIndex];
$("scale-name").textContent = `${s.title} (${ringIndex + 1}/${ring.scales.length})`;
const poolN = (s.pool && s.pool.length) || 1;
const member = (activeClip() && activeClip().title) || s.title;
// scale id · the chosen pool member · position on the ring (pool size if >1)
const poolTag = poolN > 1 ? ` · pool ${poolN}` : "";
$("scale-name").textContent = `${s.id} · ${member} (${ringIndex + 1}/${ring.scales.length}${poolTag})`;
}
function controls() {
// One bipolar Mood dial (-4 dark .. 0 neutral .. +4 light) maps onto the engine's
// two poles. The calibration gains are pinned to their locked 1.0 (full dial = full
// effect), so the simulator sends no calibration — the server uses DEFAULT_CALIBRATION.
const mood = +$("mood").value;
return {
content: $("content").value,
left: +$("left").value, right: +$("right").value,
dark: +$("dark").value, light: +$("light").value,
dark: mood < 0 ? -mood : 0, light: mood > 0 ? mood : 0,
volume: 2, brightness: 2,
};
}
function calibration() {
return { mood_gain: +$("mood_gain").value, overlay_gain: +$("overlay_gain").value,
right_variant_map: [0, 1, 2, 3, 4] };
}
let timer = null;
async function update() {
if (busy) return;
const resp = await fetch("/api/alteration", {
method: "POST", headers: { "content-type": "application/json" },
body: JSON.stringify({ controls: controls(), calibration: calibration() }),
body: JSON.stringify({ controls: controls() }),
});
if (!resp.ok) { readout.textContent = "invalid: " + resp.status; return; }
const data = await resp.json();
readout.textContent = JSON.stringify(data, null, 2);
if (!data.content.video) { black.classList.remove("hidden"); return; }
black.classList.add("hidden");
applyGrade(data.plan.grade.tone);
loadVariant(data.plan.restyle.variant);
renderOverlay(data.plan.overlay.level, data.plan.overlay.intensity);
try {
ensureClipMedia();
applyVideoLook(data.plan.grade.tone, data.plan.dream.intensity);
renderOverlay(data.plan.overlay.level, data.plan.overlay.intensity);
renderAffect(data.plan.affect.strength, data.plan.affect.intensity, data.plan.dream.strength);
const b = document.getElementById("err-banner");
if (b) b.remove(); // render succeeded — clear any prior error
} catch (err) {
showError("render failed: " + (err && err.message ? err.message : err));
throw err;
}
}
function debounced() { clearTimeout(timer); timer = setTimeout(update, 80); }
@@ -135,6 +442,7 @@ function playTransition(file, blended) {
// Resolves on 'ended' (or a safety timeout that scales with playback rate).
return new Promise((resolve) => {
overlay.style.opacity = "0";
affectLayer.style.opacity = "0";
tint.style.opacity = "0";
vid.style.filter = "none";
vid.loop = false;
@@ -171,7 +479,8 @@ async function advance(delta) {
await playTransition(step.file, step.blended);
}
ringIndex = move.to_index;
currentVariant = -1; // force the target clip's variant to (re)load
activeClipId = move.target_clip_id; // the randomly-picked pool member to load
currentClipId = null; // force the target scale's base media to (re)load
renderScaleReadout();
} finally {
busy = false;
@@ -191,10 +500,28 @@ function onWheel(e) {
}, 90);
}
// Dev live-reload: poll the asset version and reload when it changes, so an open
// tab never keeps running a stale renderer while we iterate (the readout updates
// from the live API and masks it otherwise). Reloads on my edits AND on a server
// restart. Dev-only convenience; harmless if the endpoint is absent.
function devLiveReload() {
let seen = null;
setInterval(async () => {
try {
const v = (await (await fetch("/dev/version", { cache: "no-store" })).json()).version;
if (seen === null) seen = v;
else if (v !== seen) location.reload();
} catch (_) { /* server restarting / endpoint absent — ignore */ }
}, 1000);
}
async function main() {
devLiveReload();
try { initPaint(); } catch (e) { paintOK = false; paint.style.display = "none"; showError("WebGL init: " + e.message); }
await loadData();
await landScale(); // pick the initial scale's pool member before first render
renderScaleReadout();
for (const id of ["content", "left", "right", "dark", "light", "mood_gain", "overlay_gain"]) {
for (const id of ["content", "left", "right", "mood"]) {
$(id).addEventListener("input", debounced);
}
$("zoom-in").addEventListener("click", () => advance(1));
+3 -7
View File
@@ -12,8 +12,10 @@
<section class="stage" id="stage">
<div class="screen">
<video id="vid" loop muted playsinline></video>
<canvas id="paint"></canvas>
<div id="tint"></div>
<svg id="overlay" viewBox="0 0 100 100" preserveAspectRatio="none"></svg>
<svg id="affect" viewBox="0 0 100 100" preserveAspectRatio="none"></svg>
<div id="black" class="black hidden"></div>
</div>
</section>
@@ -46,15 +48,9 @@
<legend>Experience knobs (04)</legend>
<label>Left (analytical) <input type="range" id="left" min="0" max="4" value="0" /></label>
<label>Right (dreamlike) <input type="range" id="right" min="0" max="4" value="0" /></label>
<label>Dark <input type="range" id="dark" min="0" max="4" value="0" /></label>
<label>Light <input type="range" id="light" min="0" max="4" value="0" /></label>
<label>Mood — dark ◀ 0 ▶ light <input type="range" id="mood" min="-4" max="4" value="0" /></label>
</fieldset>
<fieldset>
<legend>Calibration</legend>
<label>mood gain <input type="range" id="mood_gain" min="0" max="2" step="0.05" value="1" /></label>
<label>overlay gain <input type="range" id="overlay_gain" min="0" max="2" step="0.05" value="1" /></label>
</fieldset>
<fieldset>
<legend>RenderPlan readout</legend>
+27 -2
View File
@@ -7,13 +7,38 @@ main { display: flex; gap: 1rem; padding: 1rem; flex-wrap: wrap; }
.screen { position: relative; width: 100%; aspect-ratio: 16 / 9; background: #000;
border-radius: 6px; overflow: hidden; }
#vid { width: 100%; height: 100%; object-fit: cover; transition: opacity 0.15s ease; }
/* Right-dream painterly canvas: a WebGL Kuwahara restyle of the LIVE video frames,
drawn over the base. Edge-preserving, so motion stays as crisp as the source —
the dream is the same footage, just stylized. The mood grade rides as a CSS
filter here. Hidden when WebGL is unavailable (CSS fallback shows #vid). */
#paint { position: absolute; inset: 0; width: 100%; height: 100%; object-fit: cover;
pointer-events: none; transition: filter 0.2s ease; }
#tint { position: absolute; inset: 0; pointer-events: none; opacity: 0;
background: #28425f; mix-blend-mode: multiply;
transition: opacity 0.2s ease; }
#overlay { position: absolute; inset: 0; width: 100%; height: 100%;
pointer-events: none; transition: opacity 0.2s ease; }
.anno-box { fill: none; stroke: #6cf; stroke-width: 0.4; vector-effect: non-scaling-stroke; }
.anno-label { fill: #6cf; font-size: 3px; font-family: monospace; }
/* Affect channel — its own layer so its opacity tracks min(left,right), not Left.
Soft violet glow, distinct from the clinical reticles (the dream leaking in). */
#affect { position: absolute; inset: 0; width: 100%; height: 100%;
pointer-events: none; transition: opacity 0.35s ease;
filter: drop-shadow(0 0 0.9px rgba(176, 120, 255, 0.95)); }
.hud-affect { fill: #e3cfff; font: italic 600 3.6px Georgia, "Times New Roman", serif;
letter-spacing: 0.06px; }
/* Left-brain analytical HUD. Two sensor channels: detection (cyan) + measurement
(amber). Corner-bracket reticles, translucent label chips, a status tag. */
.hud-reticle { fill: none; stroke-width: 0.6; vector-effect: non-scaling-stroke;
stroke-linecap: square; }
.hud-reticle.detect { stroke: #6cf; }
.hud-reticle.measure { stroke: #ffc46b; }
.hud-chip-bg { fill-opacity: 0.72; }
.hud-chip-bg.detect { fill: #001722; }
.hud-chip-bg.measure { fill: #1c1200; }
.hud-chip-text { font: 2.4px monospace; letter-spacing: 0.04px; }
.hud-chip-text.detect { fill: #aee6ff; }
.hud-chip-text.measure { fill: #ffd79a; }
.hud-conf { fill: #6cf; opacity: 0.8; }
.hud-status { fill: #8fdcff; font: 2.4px monospace; opacity: 0.9; letter-spacing: 0.15px; }
.black { position: absolute; inset: 0; background: #000; }
.hidden { display: none; }
.panel { flex: 0 0 280px; display: flex; flex-direction: column; gap: 0.8rem; }
+54
View File
@@ -0,0 +1,54 @@
from tools.pipeline.ffmpeg_ops import frame_args, crossfade_loop_args, transcode_args
def test_frame_args_trims_scales_pads_and_strips_audio():
args = frame_args(
"src.mp4", "out.mp4",
start=2.0, duration=8.0, width=1920, height=1080, ff="ffmpeg",
)
assert args[0] == "ffmpeg"
assert "-ss" in args and args[args.index("-ss") + 1] == "2.0"
assert "-t" in args and args[args.index("-t") + 1] == "8.0"
assert "-an" in args # audio stripped (bases are video-only)
vf = args[args.index("-vf") + 1]
assert "scale=1920:1080:force_original_aspect_ratio=decrease" in vf
assert "pad=1920:1080" in vf
assert "format=yuv420p" in vf
assert args[-1] == "out.mp4"
def test_crossfade_loop_builds_head_mid_tail_xfade_concat():
args = crossfade_loop_args("in.mp4", "loop.mp4", duration=8.0, overlap=1.0)
fc = args[args.index("-filter_complex") + 1]
assert "trim=0:1.0" in fc # head = first O secs
assert "trim=1.0:7.0" in fc # mid = [O, D-O]
assert "trim=7.0:8.0" in fc # tail = last O secs
assert "xfade=transition=fade:duration=1.0:offset=0" in fc
assert "concat=n=2:v=1" in fc
assert "-an" in args
assert args[-1] == "loop.mp4"
def test_crossfade_loop_rejects_overlap_too_large():
import pytest
with pytest.raises(ValueError):
crossfade_loop_args("in.mp4", "loop.mp4", duration=4.0, overlap=2.0)
def test_crossfade_loop_rejects_nonpositive_overlap():
import pytest
with pytest.raises(ValueError):
crossfade_loop_args("in.mp4", "loop.mp4", duration=8.0, overlap=0.0)
def test_transcode_args_high_profile_even_dims_faststart():
args = transcode_args("loop.mp4", "proxy.mp4", width=1920, height=1080,
crf=20, fps=30)
assert "-profile:v" in args and args[args.index("-profile:v") + 1] == "high"
assert "-crf" in args and args[args.index("-crf") + 1] == "20"
vf = args[args.index("-vf") + 1]
assert "scale=1920:1080:force_original_aspect_ratio=decrease" in vf
assert "fps=30" in vf
assert "+faststart" in args
assert "-an" in args
assert args[-1] == "proxy.mp4"
+42
View File
@@ -0,0 +1,42 @@
from pathlib import Path
import pytest
from tools.pipeline.run import probe_duration, process_clip, resolve_ffmpeg
_FOREST = Path("simulator/sample_media/forest/base.mp4")
def _ffmpeg_available() -> bool:
try:
resolve_ffmpeg()
return True
except Exception:
return False
_HAS_FF = _ffmpeg_available()
@pytest.mark.skipif(not _FOREST.exists(),
reason="forest POC base absent (run simulator/setup_sample_media.py)")
def test_probe_duration_reads_forest_base():
if not _FOREST.exists():
pytest.skip("forest POC base absent")
assert probe_duration(_FOREST) > 0
@pytest.mark.skipif(not _HAS_FF,
reason="neither system ffmpeg nor imageio-ffmpeg available")
@pytest.mark.skipif(not _FOREST.exists(),
reason="forest POC base absent (run simulator/setup_sample_media.py)")
def test_process_clip_emits_proxy_and_master(tmp_path):
out = process_clip(_FOREST, tmp_path, overlap=0.5, start=0.0, duration=4.0)
assert out["proxy"].exists() and out["master"].exists()
# Proxy is exactly 1920×1080 — verified with cv2, no ffprobe dependency.
import cv2
cap = cv2.VideoCapture(str(out["proxy"]))
w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
cap.release()
assert (w, h) == (1920, 1080)
+47
View File
@@ -0,0 +1,47 @@
from tools.pipeline.manifest import upsert_clip, add_ring_scale, rebuild_ring_edges
def _base():
return {"clips": [{"id": "cosmos", "title": "Cosmos"}],
"ring": {"scales": [{"id": "cosmos", "clip_id": "cosmos"}],
"transitions": []}}
def test_upsert_clip_replaces_by_id_else_appends():
m = _base()
m = upsert_clip(m, {"id": "reef", "title": "Reef"})
assert [c["id"] for c in m["clips"]] == ["cosmos", "reef"]
m = upsert_clip(m, {"id": "reef", "title": "Coral Reef"}) # replace, not dup
assert [c["id"] for c in m["clips"]] == ["cosmos", "reef"]
assert m["clips"][1]["title"] == "Coral Reef"
def test_add_ring_scale_inserts_after_named_scale():
m = _base()
m = add_ring_scale(m, "orbit", "orbit", after="cosmos")
assert [s["id"] for s in m["ring"]["scales"]] == ["cosmos", "orbit"]
def test_rebuild_ring_edges_one_transition_per_edge_with_wrap():
m = _base()
m = add_ring_scale(m, "orbit", "orbit", after="cosmos")
m = add_ring_scale(m, "abyss", "abyss", after="orbit")
m = rebuild_ring_edges(m)
files = [t["file"] for t in m["ring"]["transitions"]]
assert files == [
"transitions/cosmos-orbit.mp4",
"transitions/orbit-abyss.mp4",
"transitions/abyss-cosmos.mp4", # wrap edge
]
def test_add_ring_scale_appends_when_after_missing():
m = _base()
m = add_ring_scale(m, "abyss", "abyss", after="nope")
assert [s["id"] for s in m["ring"]["scales"]] == ["cosmos", "abyss"]
def test_rebuild_ring_edges_single_scale_self_wrap():
m = _base() # one scale: cosmos
m = rebuild_ring_edges(m)
assert [t["file"] for t in m["ring"]["transitions"]] == ["transitions/cosmos-cosmos.mp4"]
+20
View File
@@ -0,0 +1,20 @@
from tools.pipeline.provenance import Provenance
def test_provenance_to_dict_roundtrip():
p = Provenance(
scale="abyss",
license="public-domain (US Gov, 17 U.S.C. §105)",
source="NOAA Ocean Exploration",
url="https://oceanexplorer.noaa.gov/",
fetched_at="2026-06-24",
)
d = p.to_dict()
assert d == {
"scale": "abyss",
"license": "public-domain (US Gov, 17 U.S.C. §105)",
"source": "NOAA Ocean Exploration",
"url": "https://oceanexplorer.noaa.gov/",
"fetched_at": "2026-06-24",
}
assert Provenance.from_dict(d) == p
+66 -16
View File
@@ -3,11 +3,12 @@ import pytest
from hef.selection import Coordinate
from player.alteration import (
DEFAULT_CALIBRATION,
AffectOverlay,
AnalyticalOverlay,
Calibration,
ColorGrade,
Dream,
RenderPlan,
Restyle,
plan_alteration,
render_plan_to_dict,
)
@@ -22,7 +23,8 @@ def test_all_zero_knobs_is_the_unaltered_base():
assert plan.is_identity
assert plan.overlay.level == 0
assert plan.overlay.intensity == 0.0
assert plan.restyle.variant == 0
assert plan.dream.strength == 0
assert plan.dream.intensity == 0.0
assert plan.grade.tone == 0.0
assert plan.grade.is_identity
@@ -31,21 +33,65 @@ def test_left_drives_the_analytical_overlay_only():
plan = plan_alteration(_coord(left=4))
assert plan.overlay.level == 4
assert plan.overlay.intensity == 1.0
assert plan.restyle.variant == 0 # Left does not touch the substrate
assert plan.dream.strength == 0 # Left does not touch the substrate
assert plan.dream.intensity == 0.0
assert plan.grade.tone == 0.0
def test_right_selects_a_discrete_restyle_variant_only():
def test_right_drives_the_deterministic_dream_only():
# Right-axis dream reframe (session 0013): Right is a live deterministic
# haze, carried as strength (0..4) + intensity (0..1), not a baked variant.
plan = plan_alteration(_coord(right=2))
assert plan.restyle.variant == 2
assert plan.dream.strength == 2
assert plan.dream.intensity == pytest.approx(0.5)
assert plan.overlay.level == 0 # Right does not add overlay
def test_left_and_right_stack_not_cancel():
# design §4.2: whole-brain corner = dreamlike substrate WITH labels on top
# design §4.2: whole-brain corner = dream substrate WITH labels on top
plan = plan_alteration(_coord(left=4, right=4))
assert plan.overlay.level == 4
assert plan.restyle.variant == 4
assert plan.dream.strength == 4
assert plan.dream.intensity == 1.0
def test_affect_needs_both_knobs_up():
# affect-channel design (session 0013): emotion-words surface only when BOTH
# the Left and Right knobs are up — either at 0 means no affect at all.
assert plan_alteration(_coord(left=4, right=0)).affect.strength == 0
assert plan_alteration(_coord(left=0, right=4)).affect.strength == 0
assert plan_alteration(_coord(left=4, right=0)).affect.intensity == 0.0
assert plan_alteration(_coord(left=0, right=4)).affect.intensity == 0.0
def test_affect_strength_is_the_smaller_knob():
# strength = min(left, right) — the smaller knob gates the channel.
assert plan_alteration(_coord(left=2, right=4)).affect.strength == 2
assert plan_alteration(_coord(left=4, right=2)).affect.strength == 2
assert plan_alteration(_coord(left=4, right=4)).affect.strength == 4
assert plan_alteration(_coord(left=3, right=1)).affect.strength == 1
def test_affect_intensity_scales_with_strength_over_full_scale():
for lo in range(5):
plan = plan_alteration(_coord(left=lo, right=4))
assert plan.affect.intensity == pytest.approx(lo / 4)
assert plan_alteration(_coord(left=4, right=4)).affect.intensity == 1.0
def test_affect_intensity_honors_overlay_gain():
# affect rides the same overlay_gain as the analytical HUD it belongs to.
cal = Calibration(overlay_gain=0.5)
assert plan_alteration(_coord(left=4, right=4), cal).affect.intensity == pytest.approx(0.5)
clamp = Calibration(overlay_gain=10.0)
assert plan_alteration(_coord(left=4, right=4), clamp).affect.intensity == 1.0
def test_affect_active_implies_non_identity():
# min(left,right) > 0 implies left > 0, so an active affect plan is never identity.
plan = plan_alteration(_coord(left=2, right=3))
assert plan.affect.strength == 2
assert not plan.is_identity
def test_light_pole_grades_warm_positive_tone():
@@ -72,7 +118,8 @@ def test_dark_minus_light_sets_intermediate_tone():
def test_whole_brain_dark_corner_stacks_grade_substrate_and_overlay():
plan = plan_alteration(_coord(left=4, right=2, dark=4, light=0))
assert plan.overlay.level == 4
assert plan.restyle.variant == 2
assert plan.dream.strength == 2
assert plan.dream.intensity == pytest.approx(0.5)
assert plan.grade.tone == -1.0
assert not plan.is_identity
@@ -84,32 +131,34 @@ def test_default_calibration_is_locked():
# locked feel is a deliberate edit here + in alteration.py.
assert DEFAULT_CALIBRATION.mood_gain == 1.0
assert DEFAULT_CALIBRATION.overlay_gain == 1.0
assert DEFAULT_CALIBRATION.right_variant_map == (0, 1, 2, 3, 4)
assert DEFAULT_CALIBRATION.dream_gain == 1.0
def test_default_calibration_is_behavior_preserving():
# DEFAULT_CALIBRATION must reproduce the original three helpers exactly.
# DEFAULT_CALIBRATION reproduces the per-axis helpers exactly.
for left in range(5):
assert plan_alteration(_coord(left=left)).overlay.intensity == pytest.approx(left / 4)
for right in range(5):
assert plan_alteration(_coord(right=right)).restyle.variant == right
assert plan_alteration(_coord(right=right)).dream.strength == right
assert plan_alteration(_coord(right=right)).dream.intensity == pytest.approx(right / 4)
for dark in range(5):
for light in range(5):
expected = (light - dark) / 4
assert plan_alteration(_coord(dark=dark, light=light)).grade.tone == pytest.approx(expected)
def test_custom_calibration_scales_mood_and_overlay():
cal = Calibration(mood_gain=0.5, overlay_gain=0.5, right_variant_map=(0, 0, 1, 1, 2))
def test_custom_calibration_scales_mood_overlay_and_dream():
cal = Calibration(mood_gain=0.5, overlay_gain=0.5, dream_gain=0.5)
assert plan_alteration(_coord(light=4), cal).grade.tone == pytest.approx(0.5)
assert plan_alteration(_coord(left=4), cal).overlay.intensity == pytest.approx(0.5)
assert plan_alteration(_coord(right=3), cal).restyle.variant == 1
assert plan_alteration(_coord(right=4), cal).dream.intensity == pytest.approx(0.5)
def test_calibration_gain_is_clamped_to_unit_range():
cal = Calibration(mood_gain=10.0, overlay_gain=10.0)
cal = Calibration(mood_gain=10.0, overlay_gain=10.0, dream_gain=10.0)
assert plan_alteration(_coord(light=4), cal).grade.tone == 1.0 # clamped, not 10
assert plan_alteration(_coord(left=4), cal).overlay.intensity == 1.0
assert plan_alteration(_coord(right=4), cal).dream.intensity == 1.0
def test_render_plan_to_dict_round_trips_the_numbers():
@@ -117,7 +166,8 @@ def test_render_plan_to_dict_round_trips_the_numbers():
assert d == {
"grade": {"tone": -1.0},
"overlay": {"level": 4, "intensity": 1.0},
"restyle": {"variant": 2},
"affect": {"strength": 2, "intensity": 0.5},
"dream": {"strength": 2, "intensity": 0.5},
"is_identity": False,
}
+34
View File
@@ -8,6 +8,7 @@ from player.ring import (
Transition,
TransitionStep,
advance_ring,
pick_clip_id,
scale_at,
)
@@ -206,3 +207,36 @@ def test_fast_spin_noop_on_degenerate_ring():
move = advance_ring(r, from_index=0, delta=9, fast_spin_threshold=3)
assert move.fast is False
assert move.steps == ()
# --- rotating clip pool (content-pipeline design §11.1) ---
def test_scale_without_pool_is_a_pool_of_one():
s = Scale(id="cosmos", clip_id="cosmos")
assert s.members == ("cosmos",)
# any r maps to the single member
assert pick_clip_id(s, 0.0) == "cosmos"
assert pick_clip_id(s, 0.99) == "cosmos"
def test_scale_with_pool_exposes_all_members():
s = Scale(id="coast", clip_id="coast_a", pool=("coast_a", "coast_b", "coast_c"))
assert s.members == ("coast_a", "coast_b", "coast_c")
def test_pick_clip_id_selects_member_by_injected_r():
s = Scale(id="coast", clip_id="a", pool=("a", "b", "c", "d"))
# r in [0,1) partitions the pool into len equal buckets
assert pick_clip_id(s, 0.0) == "a"
assert pick_clip_id(s, 0.24) == "a"
assert pick_clip_id(s, 0.25) == "b"
assert pick_clip_id(s, 0.5) == "c"
assert pick_clip_id(s, 0.75) == "d"
def test_pick_clip_id_clamps_out_of_range_r():
s = Scale(id="coast", clip_id="a", pool=("a", "b", "c"))
assert pick_clip_id(s, -5.0) == "a" # below 0 clamps to first
assert pick_clip_id(s, 1.0) == "c" # 1.0 must not index past the end
assert pick_clip_id(s, 999.0) == "c"
+6 -4
View File
@@ -59,13 +59,15 @@ def test_overlay_change_is_a_live_update():
assert t.playback.plan.overlay.level == 4
def test_restyle_change_crossfades_the_substrate():
# design §4.3: the Right v2v substrate is a pre-baked variant swap
def test_dream_change_is_a_live_update_not_a_crossfade():
# Right-axis dream reframe (session 0013): the Right dream is a deterministic
# LIVE filter now, so changing it no longer crossfades — only a clip swap does.
p = Player(LIB)
p.update(_controls(content="video", right=0))
t = p.update(_controls(content="video", right=4))
assert t.kind == TransitionKind.CROSSFADE
assert t.playback.plan.restyle.variant == 4
assert t.kind == TransitionKind.LIVE_UPDATE
assert t.playback.plan.dream.strength == 4
assert t.playback.plan.dream.intensity == 1.0
def test_volume_only_change_is_a_live_update():
+64 -2
View File
@@ -38,7 +38,8 @@ def test_alteration_returns_the_engine_plan(client):
assert resp.status_code == 200
data = resp.json()
assert data["plan"]["overlay"]["level"] == 4
assert data["plan"]["restyle"]["variant"] == 2
assert data["plan"]["dream"]["strength"] == 2
assert data["plan"]["dream"]["intensity"] == 0.5
assert data["plan"]["grade"]["tone"] == -1.0
assert data["content"]["video"] is True
@@ -51,7 +52,7 @@ def test_alteration_honors_off_as_black(client):
def test_alteration_accepts_calibration(client):
body = {"controls": _controls(light=4),
"calibration": {"mood_gain": 0.5, "overlay_gain": 1.0, "right_variant_map": [0, 1, 2, 3, 4]}}
"calibration": {"mood_gain": 0.5, "overlay_gain": 1.0, "dream_gain": 1.0}}
resp = client.post("/api/alteration", json=body)
assert resp.json()["plan"]["grade"]["tone"] == 0.5
@@ -188,3 +189,64 @@ def test_ring_404_when_no_ring_in_manifest(client):
def test_ring_advance_rejects_bad_delta(ring_client):
resp = ring_client.post("/api/ring/advance", json={"from_index": 0, "delta": "x"})
assert resp.status_code == 422
# --- rotating clip pool (content-pipeline §11.1) ---
@pytest.fixture
def pool_client(tmp_path):
p = tmp_path / "manifest.json"
members = ["coast_a", "coast_b", "coast_c", "coast_d"]
clips = [{"id": "cosmos", "title": "Cosmos", "base_file": "cosmos/base.mp4",
"license": "PD", "source": "NASA", "right_variants": {},
"annotations": [], "affect": [], "strings": {}}]
clips += [{"id": m, "title": m, "base_file": f"{m}/base.mp4", "license": "PD",
"source": "NPS", "right_variants": {}, "annotations": [], "affect": [],
"strings": {}} for m in members]
p.write_text(json.dumps({
"clips": clips,
"ring": {
"scales": [
{"id": "cosmos", "clip_id": "cosmos"},
{"id": "coast", "clip_id": members[0], "pool": members},
],
"transitions": [
{"file": "transitions/cosmos-coast.mp4"},
{"file": "transitions/coast-cosmos.mp4"},
],
},
}))
return TestClient(create_app(manifest_path=p)), members
def test_ring_exposes_each_scales_pool(pool_client):
client, members = pool_client
data = client.get("/api/ring").json()
coast = next(s for s in data["scales"] if s["id"] == "coast")
assert [m["clip_id"] for m in coast["pool"]] == members
# a single-clip scale still reports a pool of one
cosmos = next(s for s in data["scales"] if s["id"] == "cosmos")
assert [m["clip_id"] for m in cosmos["pool"]] == ["cosmos"]
def test_advance_into_a_pool_lands_on_a_member(pool_client):
client, members = pool_client
# landing on the coast scale (index 1) must pick one of its pool members
seen = set()
for _ in range(40):
data = client.post("/api/ring/advance", json={"from_index": 0, "delta": 1}).json()
assert data["to_index"] == 1
assert data["target_clip_id"] in members
seen.add(data["target_clip_id"])
# over many landings the random pick should cover more than one member
assert len(seen) > 1
def test_delta_zero_is_an_initial_pool_pick(pool_client):
client, members = pool_client
# a no-op (delta 0) advance on the pool scale is the initial / re-roll pick
data = client.post("/api/ring/advance", json={"from_index": 1, "delta": 0}).json()
assert data["to_index"] == 1
assert data["steps"] == []
assert data["target_clip_id"] in members
+5
View File
@@ -0,0 +1,5 @@
"""Content pipeline: source -> frame -> loop -> transcode -> manifest.
Pure ffmpeg-argument builders (ffmpeg_ops), a thin runner (run), provenance
records, and manifest assembly. See docs/superpowers/specs/2026-06-24-content-pipeline-design.md.
"""
+74
View File
@@ -0,0 +1,74 @@
"""Pure ffmpeg argument builders (no I/O) for the content pipeline.
Each function returns a list[str] ready for ffmpeg, so command construction is
unit-testable without running ffmpeg. tools/pipeline/run.py executes them. The
ffmpeg binary name is injected (default "ffmpeg").
"""
from __future__ import annotations
def _fit(width: int, height: int) -> str:
"""A scale+pad filter chain fitting any input into width×height (letterbox),
fixing SAR and pixel format so downstream concat/xfade get identical geometry."""
return (
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
"setsar=1,format=yuv420p"
)
def frame_args(src, dst, *, start: float, duration: float,
width: int, height: int, ff: str = "ffmpeg") -> list[str]:
"""Stage 2 — trim [start, start+duration], fit to width×height, strip audio.
Produces a high-quality mezzanine for the loop stage."""
return [
ff, "-y", "-ss", str(start), "-t", str(duration),
"-i", str(src), "-vf", _fit(width, height), "-an",
"-c:v", "libx264", "-preset", "medium", "-crf", "16",
"-pix_fmt", "yuv420p", "-movflags", "+faststart", str(dst),
]
def crossfade_loop_args(src, dst, *, duration: float, overlap: float,
fps: int = 30, ff: str = "ffmpeg") -> list[str]:
"""Stage 3 — make `src` loop seamlessly by crossfading its tail over its head.
Output length = duration overlap. Requires overlap < duration/2 so a non-empty
middle remains.
fps is applied after each trim so xfade receives a constant-frame-rate stream
(xfade requires CFR; a raw trim output reports rate 1/0 which xfade rejects)."""
if overlap <= 0 or overlap >= duration / 2:
raise ValueError(f"overlap {overlap} must be in (0, duration/2={duration/2})")
d, o = duration, overlap
fc = (
f"[0:v]trim=0:{o},setpts=PTS-STARTPTS,fps={fps}[head];"
f"[0:v]trim={o}:{d - o},setpts=PTS-STARTPTS,fps={fps}[mid];"
f"[0:v]trim={d - o}:{d},setpts=PTS-STARTPTS,fps={fps}[tail];"
f"[tail][head]xfade=transition=fade:duration={o}:offset=0[xf];"
f"[xf][mid]concat=n=2:v=1,format=yuv420p[out]"
)
return [
ff, "-y", "-i", str(src), "-filter_complex", fc,
"-map", "[out]", "-an",
"-c:v", "libx264", "-preset", "medium", "-crf", "16",
"-pix_fmt", "yuv420p", "-movflags", "+faststart", str(dst),
]
def transcode_args(src, dst, *, width: int, height: int, crf: int,
fps: int = 30, ff: str = "ffmpeg") -> list[str]:
"""Stage 4 — encode a final deliverable (master or proxy). Master: source res
capped ~3840 wide, crf 18. Proxy: 1920×1080, crf 20. Both H.264 High, yuv420p,
even dims, +faststart, video-only (spec §6)."""
vf = (
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
f"setsar=1,fps={fps},format=yuv420p"
)
return [
ff, "-y", "-i", str(src), "-vf", vf, "-an",
"-c:v", "libx264", "-profile:v", "high", "-preset", "slow",
"-crf", str(crf), "-pix_fmt", "yuv420p",
"-movflags", "+faststart", str(dst),
]
+55
View File
@@ -0,0 +1,55 @@
"""Stage 6 — assemble manifest entries (spec §3.6). Pure dict transforms over a
loaded manifest dict; the caller does the json read/write. Idempotent by id so
re-running earlier stages never clobbers authored labels."""
from __future__ import annotations
import copy
def upsert_clip(manifest: dict, clip: dict) -> dict:
"""Replace the clip with the same id, or append it. Returns a new manifest."""
m = copy.deepcopy(manifest)
clips = m.setdefault("clips", [])
for i, c in enumerate(clips):
if c["id"] == clip["id"]:
clips[i] = clip
return m
clips.append(clip)
return m
def add_ring_scale(manifest: dict, scale_id: str, clip_id: str,
after: str | None = None) -> dict:
"""Insert a scale into ring.scales after the named scale (or append if `after`
is None / not found). No-op if scale_id already present. Returns a new manifest."""
m = copy.deepcopy(manifest)
ring = m.setdefault("ring", {"scales": [], "transitions": []})
scales = ring.setdefault("scales", [])
if any(s["id"] == scale_id for s in scales):
return m
entry = {"id": scale_id, "clip_id": clip_id}
idx = next((i for i, s in enumerate(scales) if s["id"] == after), None)
if idx is None:
scales.append(entry)
else:
scales.insert(idx + 1, entry)
return m
def rebuild_ring_edges(manifest: dict) -> dict:
"""Rebuild ring.transitions to one placeholder entry per ring edge (N scales ->
N edges incl. the wrap), preserving any existing model string for an edge.
Returns a new manifest."""
m = copy.deepcopy(manifest)
ring = m.setdefault("ring", {"scales": [], "transitions": []})
scales = ring.get("scales", [])
existing = {t["file"]: t.get("model", "") for t in ring.get("transitions", [])}
edges = []
n = len(scales)
for i in range(n):
a, b = scales[i]["id"], scales[(i + 1) % n]["id"]
file = f"transitions/{a}-{b}.mp4"
edges.append({"file": file, "model": existing.get(file, "placeholder-zoom")})
ring["transitions"] = edges
return m
+29
View File
@@ -0,0 +1,29 @@
"""Stage 1 — the per-clip strict-PD provenance record (spec §3.1).
Captured at source time so license + source + URL travel with the media. The
'no explicit license -> assume PD, verify' trap (scales design §2.1) applies:
"free stock" (Pexels/Pixabay) is NOT public domain.
"""
from __future__ import annotations
from dataclasses import asdict, dataclass
@dataclass(frozen=True)
class Provenance:
scale: str
license: str
source: str
url: str
fetched_at: str # ISO date; passed in (no clock here -> deterministic)
def to_dict(self) -> dict:
return asdict(self)
@classmethod
def from_dict(cls, d: dict) -> "Provenance":
return cls(
scale=d["scale"], license=d["license"], source=d["source"],
url=d["url"], fetched_at=d["fetched_at"],
)
+67
View File
@@ -0,0 +1,67 @@
"""Thin runner: probe the source duration, then execute the frame -> loop ->
transcode stages with tools.pipeline.ffmpeg_ops. The pure arg builders are
unit-tested; this glue is covered by the opt-in integration test."""
from __future__ import annotations
import shutil
import subprocess
from pathlib import Path
from .ffmpeg_ops import crossfade_loop_args, frame_args, transcode_args
MASTER_MAX_W = 3840
def resolve_ffmpeg() -> str:
"""ffmpeg from PATH, else the bundled imageio-ffmpeg binary (matches
simulator/setup_scales_media.py). The Pi/dev box may lack a system ffmpeg."""
if shutil.which("ffmpeg"):
return "ffmpeg"
import imageio_ffmpeg
return imageio_ffmpeg.get_ffmpeg_exe()
def probe_duration(src) -> float:
"""Clip duration in seconds via cv2 (frame_count / fps) — avoids a system
ffprobe dependency (imageio-ffmpeg ships ffmpeg only; the Pi has no ffprobe)."""
import cv2
cap = cv2.VideoCapture(str(src))
fps = cap.get(cv2.CAP_PROP_FPS) or 0.0
frames = cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0.0
cap.release()
if fps <= 0:
raise ValueError(f"could not read fps from {src}")
return frames / fps
def _run(args: list[str]) -> None:
subprocess.run(args, check=True, capture_output=True)
def process_clip(src, out_dir, *, start: float = 0.0, duration: float | None = None,
overlap: float = 1.0, master_w: int = MASTER_MAX_W, master_h: int = 2160,
ff: str | None = None) -> dict:
"""Run stages 24 for one clip; return {'master','proxy','loop','mezzanine'} paths.
`duration` defaults to the full source length (minus the trim start).
Note: `crossfade_loop_args` uses a fixed fps only to satisfy xfade's
constant-frame-rate requirement; the final output fps is set by `transcode_args`
(both default 30 — keep in sync if overridden)."""
ff = ff or resolve_ffmpeg()
src = Path(src)
out_dir = Path(out_dir)
out_dir.mkdir(parents=True, exist_ok=True)
if duration is None:
duration = probe_duration(src) - start
mezz = out_dir / "mezzanine.mp4"
loop = out_dir / "loop.mp4"
master = out_dir / "master.mp4"
proxy = out_dir / "base.mp4" # the proxy is what the manifest's base_file points at
_run(frame_args(src, mezz, start=start, duration=duration,
width=master_w, height=master_h, ff=ff))
_run(crossfade_loop_args(mezz, loop, duration=duration, overlap=overlap, ff=ff))
_run(transcode_args(loop, master, width=master_w, height=master_h, crf=18, ff=ff))
_run(transcode_args(loop, proxy, width=1920, height=1080, crf=20, ff=ff))
return {"mezzanine": mezz, "loop": loop, "master": master, "proxy": proxy}