Compare commits
56 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 625d32c0f3 | |||
| 4ac39c9034 | |||
| 30ec0c9c26 | |||
| 3d46b96ecb | |||
| 3b19d3eb92 | |||
| fddb6d65d4 | |||
| 2722949805 | |||
| d446238b79 | |||
| c51fdc422a | |||
| aa76cd1c9f | |||
| c07067492b | |||
| fd0aad6610 | |||
| 68f9d7342a | |||
| 29c6d822bd | |||
| 1f9b2a0bec | |||
| 39fb9108db | |||
| 062a283de4 | |||
| adc39fe648 | |||
| 99a751cb46 | |||
| 6f2d1c4f7b | |||
| a3ec09af7e | |||
| e0b29c032c | |||
| 1b96e7065b | |||
| f28a06cc7f | |||
| 229af60f1f | |||
| c389e2a834 | |||
| 32e74ca446 | |||
| e3a9b7cff6 | |||
| 7a2f37af7e | |||
| d0ad9a9ac9 | |||
| c3aa419c53 | |||
| 5bb762ae53 | |||
| 2eb752b5bc | |||
| aa56dfe145 | |||
| ed267c554c | |||
| 7da7446740 | |||
| d24f829276 | |||
| de33cbe86d | |||
| 129bb23cb9 | |||
| a2ef792f0f | |||
| 53cb89b4c6 | |||
| 0201690ce0 | |||
| 0928988abc | |||
| a1c9c5c760 | |||
| ab2edf77bb | |||
| d19fe4e8fa | |||
| 9a83c0132d | |||
| 7fd9aabd0e | |||
| 7a32ff1acc | |||
| 5fae4f9005 | |||
| 7628a5321f | |||
| bcd4d32b19 | |||
| c4452396a1 | |||
| 70fd367c70 | |||
| 12408e505e | |||
| edca8a9615 |
+13
-1
@@ -1,10 +1,22 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.DS_Store
|
||||
.pytest_cache/
|
||||
.venv/
|
||||
media/
|
||||
.superpowers/
|
||||
*.egg-info/
|
||||
# Simulator sample media (look-tuning only; populate via setup_sample_media.py
|
||||
# / setup_scales_media.py). All scale + transition binaries are gitignored.
|
||||
# / build_pool_manifest.py). All scale + transition binaries are gitignored.
|
||||
simulator/sample_media/**/*.mp4
|
||||
# Audio candidate clips (re-sourceable via docs/audio-candidate-pool.md).
|
||||
simulator/sample_media/**/*.wav
|
||||
simulator/sample_media/**/*.mp3
|
||||
simulator/sample_media/**/*.ogg
|
||||
simulator/sample_media/**/*.m4a
|
||||
# Local-only candidate review galleries (re-buildable; not shipped content).
|
||||
simulator/static/review.html
|
||||
simulator/static/review_audio.html
|
||||
|
||||
# Editor annotation/comment-thread metadata (not project content)
|
||||
.threads/
|
||||
|
||||
@@ -1,7 +1,11 @@
|
||||
.PHONY: sim sim-local
|
||||
|
||||
# Prefer the project venv's interpreter (this box has no bare `python` on PATH),
|
||||
# fall back to python3, then python.
|
||||
PYTHON ?= $(firstword $(wildcard .venv/bin/python) python3 python)
|
||||
|
||||
sim:
|
||||
docker compose -f simulator/docker-compose.yml up --build
|
||||
|
||||
sim-local:
|
||||
python -m uvicorn simulator.app:app --reload --port 8000
|
||||
$(PYTHON) -m uvicorn simulator.app:app --reload --port 8000
|
||||
|
||||
+16
-1
@@ -196,8 +196,23 @@ every transition, so fast navigation stays responsive on placeholder media.
|
||||
browser. Real strict-PD bases for all five scales sourced per
|
||||
[`docs/content-sourcing.md`](./content-sourcing.md).
|
||||
|
||||
- **Audio + separated Visual/Audio controls (session 0020) ✅ Done.** The
|
||||
simulator now has **sound**: per-altitude soundtracks that follow the Altitude
|
||||
dial + a synthesized pink **white-noise** bed, behind two orthogonal controls —
|
||||
**Visual** (on/off) × **Audio** (off / soundtrack / white_noise) — replacing
|
||||
the bundled 7-way Content dial. `tools/pipeline/audio_ops` + `audio_run` +
|
||||
`simulator/build_audio_media.py` (ffmpeg loop/loudnorm/noise, gitignored
|
||||
assets), `Scale.audio` in the manifest, a pure `player/audio.py` resolver
|
||||
(retiring `player/content.py`), server-resolved `render.audio.url`, and a client
|
||||
A/B `<audio>` crossfade layer. This is the "Content step" of the future control
|
||||
surface (it reframes the proposed networked-control-surface spec §3/§5). Spec:
|
||||
[`2026-06-26-audio-video-separated-controls-design.md`](./superpowers/specs/2026-06-26-audio-video-separated-controls-design.md);
|
||||
plan:
|
||||
[`2026-06-26-audio-video-separated-controls.md`](./superpowers/plans/2026-06-26-audio-video-separated-controls.md).
|
||||
|
||||
- **Still deferred:** real **i2v ring transitions** (need a generative-video
|
||||
model + adjacent real bases); Pi renderer + serial/firmware.
|
||||
model + adjacent real bases); Pi renderer + serial/firmware; the **music**
|
||||
audio layer (reserved dial position, no assets yet).
|
||||
- **Catalog model changes** — audio *source* + "neutral base" vs "altered
|
||||
variant" flag (sub-project 2 territory, design §13).
|
||||
|
||||
|
||||
+48
-37
@@ -296,24 +296,17 @@ exists. (The earlier selection-era "curator's X-ray" view was retired when the
|
||||
piece moved from *selecting* clips to *altering* them.)
|
||||
|
||||
**One-time setup — populate the sample footage** (look-tuning only; not shipped
|
||||
content):
|
||||
content). The ring uses real strict-PD footage in a **rotating pool** per scale
|
||||
(`docs/content-candidate-pool.md`); regenerate the manifest + new transition
|
||||
placeholders with:
|
||||
|
||||
python simulator/setup_sample_media.py # stage the forest neutral base
|
||||
python -m simulator.bake_right_variants --scale forest # real Right variants (SD on MPS, ~11 min)
|
||||
python simulator/setup_scales_media.py # the cosmos + abyss scales + ring transitions
|
||||
python simulator/build_pool_manifest.py --media # write the pool manifest + coast-edge transitions
|
||||
|
||||
`setup_sample_media.py` stages the forest neutral base from the session-0008 POC
|
||||
artifacts (`~/hef-poc/out/neutral.mp4`). `bake_right_variants.py` then produces
|
||||
the **real** flow-stabilized painterly Right variants (strengths 1–4) for the
|
||||
forest scale — SD img2img keyframes + Farneback optical-flow tweens on Apple MPS,
|
||||
the temporally-calm POC algorithm, productionized (scales design §1/§4). The
|
||||
keyframe-strength ramp per Right level is by-eye tunable in
|
||||
`simulator/bake_right_variants.py`. `setup_scales_media.py` makes the scale
|
||||
**ring** demonstrable: cheap synthetic placeholder bases for the two true-PD
|
||||
scales (`cosmos` = NASA/Hubble, `abyss` = NOAA Ocean Exploration) plus the
|
||||
per-edge zoom/warp transition clips — these scales carry a raw base only (no real
|
||||
Right variants yet; the baker can extend to them once their real footage is
|
||||
sourced). The `.mp4` binaries are gitignored.
|
||||
`build_pool_manifest.py` holds the hand-authored label/affect content and emits
|
||||
`simulator/sample_media/manifest.json` (the 19-clip pool baseline). The pool clips
|
||||
themselves are sourced via the content pipeline (`tools/pipeline/run.py
|
||||
process_clip`); the `.mp4` binaries are gitignored. (`setup_scales_media.py` is the
|
||||
older one-base-per-scale generator, now superseded.)
|
||||
|
||||
**Run it (Docker):**
|
||||
|
||||
@@ -331,27 +324,45 @@ then open http://localhost:8000.
|
||||
- **Content dial** — picks audio/video channel; "off" and audio-only positions go
|
||||
to black walls.
|
||||
- **Scale ring (endless encoder)** — `⊖ out` / `in ⊕` (or scroll the stage) walk a
|
||||
*closed ring* of neutral "scales of nature" clips — cosmos → forest → abyss and
|
||||
back around (diving past the smallest **wraps** to the largest). It is *relative*
|
||||
(an endless encoder), distinct from the absolute 0–4 knobs: each step plays a
|
||||
short placeholder zoom/warp **transition**, then settles on the next scale, with
|
||||
the current knob alteration still applied. The current scale is named beside the
|
||||
buttons (`name (i/N)`). A **fast spin** (scroll several detents at once, ≥3)
|
||||
collapses to a single quick **blended pass** straight to the destination scale
|
||||
instead of grinding through every transition (scales design §3); slow single
|
||||
steps still chain one full transition per scale crossed.
|
||||
- **Four experience knobs (0–4):**
|
||||
- **Dark / Light** — a live runtime color grade (cool/dark ↔ warm/bright; equal
|
||||
or zero = the raw footage).
|
||||
- **Right (dreamlike)** — selects a discrete pre-baked, flow-stabilized restyle
|
||||
variant and crossfades to it (strength 0 = raw base).
|
||||
- **Left (analytical)** — a live overlay: labelled boxes from the clip's authored
|
||||
annotation track, with more annotations appearing at higher levels. Text is
|
||||
shaped live (the simulator analogue of the Pi's Pango/HarfBuzz path).
|
||||
- **Calibration sliders** — adjust the grade/overlay gain curves live; once a look
|
||||
is liked, bake the values into `DEFAULT_CALIBRATION` in `player/alteration.py`.
|
||||
*closed ring* of neutral "scales of nature" — **cosmos → orbit → coast → reef →
|
||||
abyss** and back around (diving past the smallest **wraps** to the largest). Each
|
||||
scale is a **rotating pool** of vetted clips: landing on a scale plays a random
|
||||
pool member, so the ring feels fresh each pass. It is *relative* (an endless
|
||||
encoder), distinct from the absolute 0–4 knobs: each step plays a short
|
||||
placeholder zoom/warp **transition**, then settles, with the current alteration
|
||||
still applied. The current scale + chosen member is named beside the buttons
|
||||
(`scale · member (i/N · pool N)`). A **fast spin** (≥3 detents at once) collapses
|
||||
to one quick **blended pass** instead of grinding through every transition.
|
||||
- **Three experience knobs:**
|
||||
- **Mood (−4 dark .. 0 .. +4 light)** — a live runtime color grade (cool/dark ↔
|
||||
warm/bright; 0 = the raw footage).
|
||||
- **Right (dreamlike, 0–4)** — a **deterministic real-time painterly dream** (a
|
||||
WebGL Kuwahara restyle of the live frames; holds still across the loop, goes
|
||||
trippy at max). It also drives **progressive emotion tiers**: when both knobs
|
||||
are up, the affect words escalate from basic to compound as Right rises.
|
||||
- **Left (analytical, 0–4)** — a live HUD of labelled boxes from the clip's
|
||||
authored annotation track. Labels use **progressive detail tiers**: low Left
|
||||
shows fewer, more general labels (high-salience objects only); raising Left
|
||||
brings in more objects *and* escalates each label general → specific →
|
||||
scientific → +fact. Time-windowed tracked labels appear only while their
|
||||
subject is on screen and follow it.
|
||||
- **RenderPlan readout** — always shows the exact numbers the engine produced (the
|
||||
project's honesty "X-ray," now over the alteration model).
|
||||
project's honesty "X-ray," over the alteration model).
|
||||
|
||||
The base clips, Right variants, Left annotation track, and string tables come from
|
||||
The base clips, the Left annotation tracks (with tiers + appear/disappear
|
||||
windows), affect anchors, and per-tier string tables all come from
|
||||
`simulator/sample_media/manifest.json`.
|
||||
|
||||
### Authoring labels — the author mode
|
||||
|
||||
Open **http://localhost:8000/author.html** to author a clip's labels by eye
|
||||
(content-pipeline §11.5). Pick a pool clip, **scrub** to where a subject is on
|
||||
screen, **drag a box** over it, set the label key + salience + the four detail
|
||||
tiers, then **Run tracker** to propagate the box into a keyframed track (classical
|
||||
OpenCV optical flow) with an appear/disappear window — or **Add as static box**
|
||||
for a fixed label. Shift-click the stage to place an **affect anchor** and type its
|
||||
emotion tiers. **Save to manifest** writes the entry back via the pipeline's
|
||||
idempotent upsert (keeping the clip's media + provenance, replacing only the
|
||||
authored content). The tracker only propagates **box geometry** — every label
|
||||
string, scientific name, and fact is **hand-authored**, because a generic detector
|
||||
can't produce "*Gymnothorax*" or a lifespan.
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
# Audio candidate pool — rotating pools per altitude (2026-06-26)
|
||||
|
||||
Candidate soundtrack/ambience for the **"audio + video" setting**. Following operator
|
||||
feedback, each altitude now has a **rotating pool** (mirrors the video `Scale.pool`
|
||||
model — random pick per landing, or pairable per-clip).
|
||||
|
||||
## Two audio layers per altitude (operator, 2026-06-26)
|
||||
|
||||
The "audio + video" setting layers **two** kinds of audio per altitude:
|
||||
1. **Ambience / audio track** — the soundscape (waves, reef crackle, chorus, etc.).
|
||||
**This is what we're sourcing now.**
|
||||
2. **Music track** — a separate composed musical layer. **Deferred** until the
|
||||
ambience tracks are settled across all altitudes.
|
||||
|
||||
Pools below are the **ambience** layer unless marked `music`. **Distinction (operator):**
|
||||
the ambience/soundtrack layer is an *environmental, real-world soundscape* (waves, reef,
|
||||
whale, room-tone hum) — **not composed music**. A musical pad/track, however nice, belongs
|
||||
in the music layer, even for "atmospheric" altitudes like cosmos/orbit.
|
||||
|
||||
**Media is gitignored** (`simulator/sample_media/**/*.{wav,mp3,…}`); this file is the
|
||||
re-sourceable record. Files: `simulator/sample_media/audio/<altitude>/<name>.<ext>`,
|
||||
served at `/media/audio/<altitude>/<name>.<ext>`. Review gallery (gitignored):
|
||||
`simulator/static/review_audio.html` → `/review_audio.html` with the sim running.
|
||||
|
||||
## Round-2 feedback applied (operator, session 2026-06-26)
|
||||
|
||||
- **abyss** ✓ kept (blue-whale moan) · **coast** ✓ kept (waves + terns)
|
||||
- **cosmos** ✗ plasma static "hurt my ears" → replaced with 3 NASA **sonifications**
|
||||
- **orbit** ✗ whistlers "sound like lasers" → replaced with gentle Earth **chorus** (+ wind)
|
||||
- **reef** ✗ "shouldn't have whales (too deep for a reef)" → whale removed; now crackle + diver
|
||||
- Operator: *"try all three… rotation of audio like the video, or per-video audio track."*
|
||||
|
||||
## Design arc
|
||||
|
||||
Descending the dial is one journey: **cosmos & abyss** = vast/dark/sparse siblings;
|
||||
**coast** = warm human middle; **orbit & reef** = transitional membranes. The science
|
||||
recordings (NASA/NOAA) keep the "machines reshaping perception" thesis literal.
|
||||
|
||||
## Pools
|
||||
|
||||
| altitude | id | clip | source | dur | license |
|
||||
|---|---|---|---|---|---|
|
||||
| 🌌 cosmos | `cosmos/pillars` | **✓ KEEP** — Pillars of Creation (M16), twinkly sweep | NASA/CXC/SAO — https://chandra.si.edu/sound/m16.html (`sounds/m16_all.mp4`) | 30s | PD (NASA, credit) |
|
||||
| 🌌 cosmos | `cosmos/music` | 🎵 **music track — TBD** (deferred). Black-hole drone DROPPED (too creepy, not awe-inspiring); Galactic Center dropped earlier | — | — | — |
|
||||
| 🛰 orbit | `orbit/spaceamb` | **✓ KEEP** — station room-tone hum ("inside the orbiting craft") | Sonicfreak — https://freesound.org/s/174450/ (`cdn.freesound.org/previews/174/174450_746632-hq.mp3`) | 2:31 | CC0 |
|
||||
| 🛰 (music) | _deferred_ | 🎵 "Frozen Star" warm pad — moved to MUSIC layer (it's music, not ambience) | Kevin MacLeod / incompetech — https://incompetech.com/music/royalty-free/mp3-royaltyfree/Frozen%20Star.mp3 | 3:41 | CC-BY |
|
||||
| 🐠 reef | `reef/soundscape` | **✓** richer reef (round 4) — 6-layer bake: submerged underwater bed (bubbles/sloosh) + shrimp crackle + fish chorus + toadfish + grouper + damselfish | NOAA SanctSound (FK+GR `*_snappingshrimp_*`, `FK02_01_fishchorus`, `GR01_01_toadfish`, `FK03_01_redgrouper`, `PM05_01_damselfish`) + DCSFX "Underwater [Loop] AMB" https://freesound.org/s/366159/ (`cdn.freesound.org/previews/366/366159_6725579-hq.mp3`); loop-to-60 + amix + dynaudnorm | 60s | PD (NOAA) + CC0 (underwater) |
|
||||
| 🏖 coast | `coast/waves` | Sea waves + tern calls | BigSoundBank #0267 — https://bigsoundbank.com/sea-waves-and-seagulls-s0267.html (`/UPLOAD/mp3/0267.mp3`) | 57s | CC0 |
|
||||
| 🕳 abyss | `abyss/whale` | NE Pacific blue-whale "AB call" | NOAA PMEL — https://www.pmel.noaa.gov/acoustics/multimedia/NETS_bluWhale.wav | 13s | PD (NOAA) |
|
||||
|
||||
## Sourcing rule (learned 2026-06-26)
|
||||
|
||||
**Source only from a browsable page the operator can open and whose license is stated.**
|
||||
The Mixkit clips were pulled by raw CDN asset id (`assets.mixkit.co/.../<id>-preview.mp3`)
|
||||
with no findable catalog page — operator couldn't see them, so they were removed. Good
|
||||
hubs: NASA SVS / Chandra, NOAA PMEL/SanctSound (PD), BigSoundBank (CC0), Pixabay
|
||||
(visible license). Avoid asset-id grabs and ND/NC licenses.
|
||||
|
||||
## Open items / next pass
|
||||
|
||||
- **orbit** — ✅ SETTLED: `orbit/spaceamb` (Sonicfreak station room-tone hum, CC0). Rejected
|
||||
en route: laser whistlers, Earth chorus, plain wind, "Frozen Star" pad (music). A real NASA
|
||||
ISS recording (Hadfield "Space Station Noise", SoundCloud) remains the authentic upgrade if
|
||||
ever wanted — needs a harder fetch (no yt-dlp; brew install denied).
|
||||
- **ambience layer COMPLETE** — all 5 picked: cosmos/pillars · orbit/spaceamb · coast/waves ·
|
||||
reef/soundscape · abyss/whale. Next is the production pass + wiring (below) and the music layer.
|
||||
- **reef** — ✅ round-4 6-layer bake (`reef/soundscape`): NOAA PD biological layers + a CC0
|
||||
submerged underwater "you're under" bed (DCSFX freesound #366159, hq preview). License clean.
|
||||
SanctSound clips: `files/SanctSound_*.mp4` at `https://sanctsound.ioos.us/files/` (extract audio).
|
||||
- **music layer** — separate composed music track per altitude, deferred (cosmos slot reserved).
|
||||
- **Production (after picks):** loop seams, length normalization, cross-altitude crossfade
|
||||
(mirror video); cosmos/abyss clips are short → loop.
|
||||
|
||||
## Re-download / re-bake
|
||||
|
||||
```
|
||||
A=simulator/sample_media/audio; FF=$(python -c "import imageio_ffmpeg;print(imageio_ffmpeg.get_ffmpeg_exe())")
|
||||
mkdir -p $A/cosmos $A/orbit $A/reef $A/coast $A/abyss
|
||||
# NASA sonifications (mp4 -> mp3)
|
||||
for u in perseus_sonification:blackhole m16_all:pillars galactic_all:galaxy; do
|
||||
f=${u%%:*}; o=${u##*:}; curl -fsSL "https://chandra.si.edu/sound/sounds/$f.mp4" -o /tmp/$o.mp4
|
||||
"$FF" -y -i /tmp/$o.mp4 -vn -q:a 4 $A/cosmos/$o.mp3; done
|
||||
curl -fsSL "https://svs.gsfc.nasa.gov/vis/a010000/a011000/a011073/Earthsong-540-MASTER_high.mp4" -o /tmp/es.mp4
|
||||
"$FF" -y -i /tmp/es.mp4 -vn -q:a 4 $A/orbit/earthsong.mp3
|
||||
curl -fsSL "https://assets.mixkit.co/active_storage/sfx/1162/1162-preview.mp3" -o $A/orbit/wind.mp3
|
||||
curl -fsSL "https://www.nhm.ac.uk/content/dam/nhm-www/discover/audio/sound-of-coral-reef.mp3" -o $A/reef/crackle.mp3
|
||||
curl -fsSL "https://assets.mixkit.co/active_storage/sfx/1242/1242-preview.mp3" -o $A/reef/scuba.mp3
|
||||
curl -fsSL "https://bigsoundbank.com/UPLOAD/mp3/0267.mp3" -o $A/coast/waves.mp3
|
||||
curl -fsSL "https://www.pmel.noaa.gov/acoustics/multimedia/NETS_bluWhale.wav" -o $A/abyss/whale.wav
|
||||
# layered reef
|
||||
"$FF" -y -i $A/reef/crackle.mp3 -i $A/reef/scuba.mp3 \
|
||||
-filter_complex "[1:a]volume=0.55[s];[0:a][s]amix=inputs=2:duration=first,dynaudnorm" -q:a 4 $A/reef/layered.mp3
|
||||
```
|
||||
@@ -1,5 +1,11 @@
|
||||
# Content candidate pool — operator-selected, vetted (2026-06-24)
|
||||
|
||||
> **WIRED (session 0016).** The rotating pool below is now the live manifest:
|
||||
> `simulator/build_pool_manifest.py` regenerates `simulator/sample_media/manifest.json`
|
||||
> from this pool; the ring scale `forest`→`coast` rename + the cleanup list (bottom)
|
||||
> are done; the player picks a random pool member on landing
|
||||
> (`player.ring.pick_clip_id`). See content-pipeline design §11.
|
||||
|
||||
The 5 scales each get a **rotating pool** of clips. When the viewer lands on a
|
||||
scale by zooming the ring in/out, the player **randomly picks one clip from that
|
||||
scale's pool**. (This supersedes the one-clip-per-scale model — see
|
||||
@@ -17,20 +23,43 @@ a credit line** (our overlays + dream are derivatives, which these allow);
|
||||
**CC‑BY‑ND is NOT usable** (no derivatives); **CC‑BY‑NC only if the installation
|
||||
stays non‑commercial**.
|
||||
|
||||
## 🌌 cosmos (pool of 4)
|
||||
## 🌌 cosmos (pool of 5)
|
||||
|
||||
| id | clip | source | window (s) | license / credit |
|
||||
|----|------|--------|-----------|------------------|
|
||||
| `cosmos` | Orion Nebula flythrough | NASA/JPL‑Caltech — https://images.nasa.gov/details/JPL-20221122-SOLSYSf-0001-Orion%20Dust%20and%20Death | ~30–54 | PD (17 U.S.C. §105) |
|
||||
| `cosmos` | Cosmic Cliffs — Carina Nebula flythrough | NASA SVS 31348 — https://svs.gsfc.nasa.gov/31348/ (`Clifs-3d-STScI.mp4`, *Exploring the Cosmic Cliffs in 3D*, Webb NIRCam, **4K**) | 10–34 | CC‑BY‑class — credit **NASA, ESA, CSA, STScI** |
|
||||
| `cosmos_miri` | Cosmic Cliffs — Carina Nebula, **mid-infrared** (same towers, muted gray/teal/gold dust palette) | ESA/Webb weic2205c — https://esawebb.org/videos/weic2205c/ (*Pan of the Carina Nebula, combined NIRCam + MIRI*, **4K** `weic2205c.mp4`) | 4–26 | CC‑BY‑class — credit **NASA, ESA, CSA, STScI** |
|
||||
|
||||
> ℹ️ **Cosmos primary swapped to the Webb "Cosmic Cliffs" flythrough (2026-06-25, session 0018).**
|
||||
> History: the original primary was NASA/JPL's *"Orion: Dust and Death"* (captions
|
||||
> burned in → cropping lost framing); it was then swapped to STScI's text-free
|
||||
> *Flight Through the Orion Nebula* (SVS 30957, visible light). The operator found
|
||||
> the Orion clip merely good and asked for a more dramatic nebula, so the primary is
|
||||
> now STScI's **Exploring the Cosmic Cliffs in 3D** (Carina Nebula, Webb NIRCam,
|
||||
> **SVS 31348**) — a full-frame, text-free 3D flythrough sourced from its **4K**
|
||||
> master (`Clifs-3d-STScI.mp4`, 3840×2160) → supersampled 1080p base + 4K master.
|
||||
> Trim 10–34s clears the opening title card (gone by ~8s) and the end-credit card
|
||||
> (~102s). The earlier `decaption_cosmos` crop helper remains removed.
|
||||
| `cosmos_galaxies` | Flying Through Galaxies | NASA SVS — https://svs.gsfc.nasa.gov/vis/a010000/a014900/a014950/14950_Galaxies_FlyThrough_4k.mp4 | 8–32 | PD |
|
||||
| `cosmos_hudf` | Hubble Ultra Deep Field zoom | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030600/a030687/hudf-b-1920x1080p30.mov | 20–44 | CC‑BY‑class — credit **NASA, ESA, STScI** |
|
||||
| `cosmos_xdf` | eXtreme Deep Field flythrough | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030600/a030681/hxdf_fly-b-1920x1080p30.mov | 2–26 | CC‑BY‑class — credit **NASA, ESA, STScI** |
|
||||
|
||||
> ➕ **`cosmos_miri` added (2026-06-26).** Operator wanted more *Cosmic Cliffs*
|
||||
> footage and a different-color rendering. The keep-the-flythrough-primary +
|
||||
> add-the-mid-infrared-pan pair: ESA/Webb's **weic2205c** is the *same* Carina
|
||||
> towers shot with NIRCam **+ MIRI**, so it renders in a muted gray/teal/gold dust
|
||||
> palette instead of the iconic orange/blue — a distinct look from the same scene.
|
||||
> Sourced text-free/human-free from the **4K** master, trim 4–26s (clears the
|
||||
> opening fade-from-black, ends before any tail fade) → 1080p base + 4K master,
|
||||
> 1 s crossfade-loop (~21 s). ESA/Webb's narration-free *"Diving Into the Cosmic
|
||||
> Cliffs"* pan (`carina_revisit_pan`, NIRCam palette) was the runner-up — left
|
||||
> out for now since it repeats the existing palette; easy to add if wanted.
|
||||
|
||||
> ⚠️ **DISCREPANCY to fix next session.** Operator selected cosmos **#1, #3
|
||||
> (Orion), #4, #5**. During processing, **#6 Galaxy Traverse was run by mistake
|
||||
> instead of #3 Orion**. The Orion clip (#3) is text‑free and already lives at
|
||||
> `simulator/sample_media/cosmos/base.mp4` (the prior real base), so the pool above
|
||||
> is correct. **`cosmos_traverse` is an UNSELECTED extra on disk — drop it** (or
|
||||
> instead of #3 Orion**. The cosmos primary now lives at
|
||||
> `simulator/sample_media/cosmos/base.mp4` as the Webb Cosmic Cliffs flythrough
|
||||
> (SVS 31348 — see the cosmos-primary note above). **`cosmos_traverse` is an UNSELECTED extra on disk — drop it** (or
|
||||
> confirm with operator). The current ring scale id is `cosmos`; when the pool is
|
||||
> wired, `cosmos` becomes one pool member.
|
||||
|
||||
@@ -93,11 +122,14 @@ All NOAA Ocean Exploration (PD, 17 U.S.C. §105). These have creatures drifting
|
||||
> Crossota has only ~17s clean (too short for a 24s window), comb jelly has a
|
||||
> species caption covering the whole clip.
|
||||
|
||||
## On disk but NOT in any pool (clean up next session)
|
||||
## On disk but NOT in any pool — ✅ CLEANED UP (session 0016)
|
||||
|
||||
- `cosmos_traverse` — unselected (the #6/#3 mix‑up above).
|
||||
- `orbit_westcoast` — rejected (ISS hardware in frame); leftover output dir.
|
||||
- `forest` — the old Yosemite base, superseded by the `coast` pool.
|
||||
- `reef` — the original placeholder/early reef base, superseded by the `reef_*` pool.
|
||||
- The `review.html` gallery + all `cosmos/orbit/coast/reef/abyss` candidate dirs are
|
||||
local‑only (gitignored).
|
||||
Removed from local media (all were gitignored, so repo-invisible):
|
||||
|
||||
- `cosmos_traverse` — unselected (the #6/#3 mix‑up above). **dropped.**
|
||||
- `orbit_westcoast` — rejected (ISS hardware in frame). **dropped.**
|
||||
- `forest` — the old Yosemite base, superseded by the `coast` pool. **dropped.**
|
||||
- `reef` — the original placeholder/early reef base, superseded by `reef_*`. **dropped.**
|
||||
- `orbit` / `abyss` — the Increment‑1 single real bases (NASA ISS Exp65 / ctenophore),
|
||||
not in the new pools. **dropped** (the cosmos base — now the Webb Cosmic Cliffs flythrough, SVS 31348 — stays as the cosmos pool primary).
|
||||
- `review.html` gallery — now `.gitignore`d (local‑only, re‑buildable).
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,27 +1,36 @@
|
||||
# Affect channel — emotions in the Left-brain HUD
|
||||
|
||||
**Status:** approved (session 0013, 2026-06-22)
|
||||
**Status:** approved (session 0013, 2026-06-22); **revised session 0018, 2026-06-26**
|
||||
**Surface:** simulator preview (`simulator/`, `player/alteration.py`)
|
||||
**Builds on:** the machine-altered-perception design SPEC; the Left HUD look-and-feel
|
||||
pass (same session).
|
||||
|
||||
> **Revision (session 0018) — affect is a RIGHT-brain output.** The original design
|
||||
> gated emotions on *both* knobs (`strength = min(left, right)`). Per operator
|
||||
> direction, **emotions are now controlled by the Right (dreamlike) knob alone** —
|
||||
> the analytical Left knob no longer gates them. This sharpens the left/right split:
|
||||
> Left = measurement (detections + measurements), Right = the dream **and** the
|
||||
> feeling it evokes. Every `min(left, right)` below now reads `right`. The data
|
||||
> shape, rendering, and per-word `min_level` mechanic are unchanged — only the knob
|
||||
> the strength is keyed off.
|
||||
|
||||
## Idea
|
||||
|
||||
A third HUD channel beside the cyan *detections* and amber *measurements*: soft,
|
||||
glowing **emotion-words** that surface the feelings the experience may invoke.
|
||||
They appear only when *both* the analytical (Left) and dreamlike (Right) knobs are
|
||||
up — the dream leaking into the machine's reading. This is the uncanny edge of the
|
||||
thesis: the machine trying to quantify not just what is there, but how you are
|
||||
meant to feel.
|
||||
They are a **right-brain output: driven by the dreamlike Right knob alone** (the
|
||||
analytical Left knob does not gate them) — the dream surfacing as the machine's
|
||||
felt reading. This is the uncanny edge of the thesis: the machine trying to
|
||||
quantify not just what is there, but how you are meant to feel.
|
||||
|
||||
## Behaviour
|
||||
|
||||
- **Trigger / intensity.** Affect strength = `min(left, right)` (0–4). Either knob
|
||||
at 0 → no emotions at all. Opacity scales with that strength × `overlay_gain`.
|
||||
- **Escalation.** A **stable emotional palette per scene** — higher combined
|
||||
strength reveals *more* words at higher opacity. Each word carries a `min_level`
|
||||
(the same mechanic the detection channel uses), but compared against the
|
||||
*combined* `min(left, right)` strength rather than Left alone.
|
||||
- **Trigger / intensity.** Affect strength = `right` (0–4). Right at 0 → no
|
||||
emotions at all, regardless of Left. Opacity scales with that strength ×
|
||||
`overlay_gain`.
|
||||
- **Escalation.** A **stable emotional palette per scene** — higher Right reveals
|
||||
*more* words at higher opacity. Each word carries a `min_level` (the same
|
||||
mechanic the detection channel uses), compared against the `right` strength.
|
||||
- **Rendering.** Floating words at authored scene positions — **no boxes or
|
||||
reticles**, lowercase, semi-transparent with a soft glow, in a distinct **violet**
|
||||
register so they read as affective and clearly *softer* than the hard clinical
|
||||
@@ -33,15 +42,15 @@ Consistent with the existing split (alteration math in Python, label-selection i
|
||||
the client):
|
||||
|
||||
- `player/alteration.py` gains an `AffectOverlay(strength, intensity)` on the
|
||||
`RenderPlan`. `strength = min(left, right)`; `intensity = clamp(overlay_gain *
|
||||
min(left,right) / KNOB_MAX, 0, 1)`. Returned in `render_plan_to_dict` under
|
||||
`"affect"`. Pure, unit-tested.
|
||||
`RenderPlan`. `strength = right`; `intensity = clamp(overlay_gain * right /
|
||||
KNOB_MAX, 0, 1)`. Returned in `render_plan_to_dict` under `"affect"`. Pure,
|
||||
unit-tested.
|
||||
- The simulator client picks *which* words to show by comparing each affect
|
||||
entry's `min_level` to `plan.affect.strength`, and renders them at
|
||||
`plan.affect.intensity` opacity.
|
||||
|
||||
`is_identity` is unaffected: `min(left,right) > 0` implies `left > 0`, which already
|
||||
makes `overlay.level > 0`.
|
||||
`is_identity` is unaffected: `affect.strength == right`, and `right > 0` already
|
||||
makes `dream.strength > 0`.
|
||||
|
||||
## Data
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# HEF — Content Pipeline (source → process → label → manifest)
|
||||
|
||||
**Date:** 2026-06-24
|
||||
**Status:** Graduated — operator-approved (session 0014); **Increment 1 built & merged** (PR #13). Increment 2 (hybrid track pass + author mode) + real PD sourcing pending.
|
||||
**Status:** Graduated — operator-approved (session 0014); **Increment 1 built & merged** (PR #13); real PD sourcing + rotating pool curated (PR #14/#15). **Increment 2 BUILT** (session 0016 — pool model + tiers + windows in PR #16; `track.py` + author mode in PR B; see §11).
|
||||
**Repo:** `human-experience-filter-art`
|
||||
**Anchor:** `docs/ROADMAP.md` sub-project 3 (Player Runtime) content needs; the
|
||||
spiritual successor to sub-project 2 (Ingest & Tagging tools), retargeted from the
|
||||
@@ -239,7 +239,114 @@ master encode (waits on a measured pano spec).
|
||||
download + provenance record for v1.
|
||||
- **Crossfade overlap length** per scale (slow cosmos vs busier reef) — by eye.
|
||||
|
||||
## 10. Out of scope (YAGNI)
|
||||
## 11. Increment 2 — schema decisions (session 0016)
|
||||
|
||||
Increment 2 is the second build order from §8. It is mostly **schema work**, so
|
||||
this section pins the concrete data shapes before the build (the operator asked for
|
||||
a short design pass first). Each subsection is a new, **backward-compatible**
|
||||
extension — Increment 1 manifests keep rendering unchanged.
|
||||
|
||||
### 11.1 Rotating clip pool (supersedes one-clip-per-scale)
|
||||
|
||||
Each ring scale references a **pool** of vetted clips
|
||||
(`docs/content-candidate-pool.md`); the player picks **one member at random when
|
||||
the viewer lands on that scale** (zooming the ring in/out). This supersedes
|
||||
decision §5's one-base-per-scale.
|
||||
|
||||
- **Manifest:** `ring.scales[i]` gains `"pool": ["clip_id", …]` (≥1). The legacy
|
||||
`"clip_id"` stays as the **primary** member for back-compat (= `pool[0]` when a
|
||||
pool is written); a scale with only `clip_id` and no `pool` behaves exactly as
|
||||
Increment 1 (pool of one).
|
||||
- **Canonical logic (`player/ring.py`):** `Scale` gains `pool: tuple[str, …]`;
|
||||
`members` = `pool or (clip_id,)`. A **pure** `pick_clip_id(scale, r)` takes an
|
||||
injected `r ∈ [0,1)` and returns `members[int(r·len)]` — randomness stays
|
||||
**injected** (DI), so the function is pure/testable and the Pi reuses it.
|
||||
- **Where the pick happens:** at the **API boundary** (`simulator/app.py`), which
|
||||
may be impure: the advance endpoint calls `pick_clip_id(landed_scale,
|
||||
random.random())` and returns it as `target_clip_id`. A `delta=0` advance is the
|
||||
**initial / re-roll** pick (no-op move, fresh random member). The client loads
|
||||
`move.target_clip_id` directly instead of deriving the clip from the index. One
|
||||
selection implementation, Python-canonical (browser still "only renders").
|
||||
- **Rename + cleanup:** ring scale `forest` → **`coast`** (new land↔sea scale
|
||||
between `orbit` and `reef`). Drop the leftovers `cosmos_traverse`,
|
||||
`orbit_westcoast`, old `forest`, old `reef` (per the pool doc's cleanup list);
|
||||
the cosmos pool's primary stays `cosmos` (Orion). New ring order:
|
||||
`cosmos → orbit → coast → reef → abyss → (wrap) cosmos`; edges regenerated
|
||||
(`orbit-coast`, `coast-reef` placeholders via `setup_scales_media.py`).
|
||||
|
||||
### 11.2 Time-windowed tracked labels
|
||||
|
||||
Objects drift **into and out of frame**, so a label appears only **while its
|
||||
object is on screen** and its box **tracks** the object (the §4 keyframed `track`).
|
||||
|
||||
- **Manifest:** an annotation gains optional `"appear"` and `"disappear"`
|
||||
(loop-normalized `t ∈ [0,1]`, same clock as `track`). The label renders only
|
||||
while `appear ≤ loopT ≤ disappear`. To let an object straddle the loop seam, a
|
||||
**`disappear < appear` window wraps** (visible when `loopT ≥ appear` **or**
|
||||
`loopT ≤ disappear`). Absent both fields → always-present (Increment-1 behavior).
|
||||
- **Runtime:** client-side in `renderOverlay` — a window test gates the annotation
|
||||
before the existing `boxAt(loopT)` placement. Engine unchanged (labels are not in
|
||||
`RenderPlan`). The **abyss + reef** pool clips (creatures entering/leaving) are
|
||||
the authored test material.
|
||||
|
||||
### 11.3 Progressive LEFT-brain detail tiers (replaces flat `min_level` gating)
|
||||
|
||||
The Left knob now drives **how many** labels show **and how analytical** each is —
|
||||
no fading. Low Left = **fewer, more general** labels; high Left = **more** labels +
|
||||
**more analytical** detail.
|
||||
|
||||
- **Per-object `salience` (1–4):** how prominent the object is. It sets the Left
|
||||
level at which the label first appears: `first_level = 5 − salience` (salience 4 →
|
||||
appears at Left 1; salience 1 → only at Left 4). So raising Left brings in
|
||||
lower-salience objects.
|
||||
- **Detail tiers (escalate with Left):** the displayed string escalates
|
||||
`tier 1 general ("eel") → 2 specific ("moray eel") → 3 scientific
|
||||
(*Gymnothorax*) → 4 scientific + a fact (lifespan)`. The tier shown =
|
||||
`clamp(left − first_level + 1, 1, n_tiers)` — an object shows tier 1 the moment
|
||||
it appears and climbs one tier per further Left notch until its tiers run out.
|
||||
- **Strings carry the tiers:** `strings.en[key]` may be a **list** (indexed by
|
||||
`tier−1`) instead of a string; a plain string stays static (back-compat, tier
|
||||
ignored). Keeps one key per object and the i18n table as the single source.
|
||||
- **Back-compat:** an annotation with `min_level` and no `salience` keeps flat
|
||||
gating (`first_level = min_level`, single-tier string). `measure.*` readouts stay
|
||||
single-tier, gated by their level (a measurement is not a tiered object).
|
||||
- **No engine change:** tier selection is client-side against the existing
|
||||
`overlay.level` (the Left knob) already in the `RenderPlan`.
|
||||
|
||||
### 11.4 Progressive RIGHT-brain emotion tiers
|
||||
|
||||
The affect **vocabulary** gets more sophisticated as Right rises — basic words at
|
||||
Right 1, nuanced/compound feelings at high Right — mirroring §11.3 for affect.
|
||||
|
||||
- **Appearance rule unchanged:** affect surfaces only when **both** knobs are up;
|
||||
an anchor's `min_level` is tested against `strength = min(left, right)` as today.
|
||||
- **Emotion tiers:** `strings.en[feel.key]` may be a **list**; the word shown =
|
||||
`clamp(right, 1, n_tiers)` (the Right knob, read from `dream.strength` already in
|
||||
the `RenderPlan`). A plain string stays static (back-compat).
|
||||
- **No engine change:** `AffectOverlay` already carries `strength`; the Right level
|
||||
comes from `dream.strength`. Tier selection is client-side.
|
||||
|
||||
### 11.5 `tools/pipeline/track.py` (hybrid) + simulator author mode
|
||||
|
||||
The stage-5 tooling from §3.5 / §7, delivered as a **second PR** after the
|
||||
experience PR (§8's split note).
|
||||
|
||||
- **`track.py`:** a **classical** path (OpenCV CSRT / optical-flow) seeded by a
|
||||
hand-drawn box on a keyframe, propagated and **sampled to a sparse keyframed
|
||||
`track`** (normalized `{t, box}` over the loop, §4); plus an **optional,
|
||||
lazy-imported ML** detect+track path (YOLO+ByteTrack-class) that proposes tracks
|
||||
the author maps to keys. Both emit the same `track` artifact + an `appear`/
|
||||
`disappear` window. Pure helpers (normalization, keyframe sampling,
|
||||
track→manifest) are unit-tested; an **opt-in** integration test runs the real
|
||||
tracker on a pool clip (mirrors the Increment-1 real-footage test).
|
||||
- **Author mode (simulator):** reuse the preview UI as the editor — scrub a pool
|
||||
clip, draw/seed/correct boxes, run the tracker, assign `key` / `salience` /
|
||||
tiered strings / `appear`-`disappear` / affect anchors, and **write the manifest
|
||||
entry** (an endpoint upserts via `tools/pipeline/manifest.upsert_clip` and
|
||||
persists). Semantics (keys, strings, scientific names, facts) are **always
|
||||
hand-authored** — the tracker only propagates **box geometry** (§3.5).
|
||||
|
||||
## 12. Out of scope (YAGNI)
|
||||
|
||||
- Generative i2v **ring transitions** for the new edges (deferred; placeholders hold).
|
||||
- **Audio** sourcing/alteration (separate content-dial channel; already deferred).
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# Simulator Dev Mode — design
|
||||
|
||||
> 2026-06-25 · session 0017 · planning-and-executing
|
||||
> Anchor: leaf task (simulator dev-tooling — R2b, no upstream design needed)
|
||||
|
||||
## Problem
|
||||
|
||||
While previewing the experience, the operator can't choose *which* clip in an
|
||||
altitude's rotating pool is shown — the server picks a random pool member on each
|
||||
landing (content-pipeline §11.1). For by-eye review they want to step through every
|
||||
clip in a pool on demand, and to see the engine/clip data behind what's on screen.
|
||||
The RenderPlan JSON readout is also always visible, cluttering what should read as
|
||||
an audience-facing preview.
|
||||
|
||||
## Goal
|
||||
|
||||
A single **Dev Mode** toggle at the bottom of the control panel. Off by default
|
||||
(persisted in `localStorage`). When on, a panel appears directly below it with all
|
||||
dev controls and data; when off, the normal view is just the experience + the three
|
||||
control fieldsets.
|
||||
|
||||
## Scope
|
||||
|
||||
Frontend only — `simulator/static/{index.html,app.js,style.css}`. No API changes:
|
||||
every datum Dev Mode shows is already available client-side (`/api/clips`,
|
||||
`/api/ring`, the `/api/alteration` response, and the in-memory preload cache). The
|
||||
separate `/author.html` remains the place to *edit* labels; Dev Mode is
|
||||
inspect-and-select only.
|
||||
|
||||
## Dev panel contents (top → bottom)
|
||||
|
||||
1. **Pool picker** — a `<select>` of the current altitude's pool members
|
||||
(`clip_id · title`), its value reflecting the clip currently playing. Choosing a
|
||||
member sets `activeClipId` to it and reloads the base media, overriding the random
|
||||
landing pick. A `🎲 Re-roll random` button re-runs the server-canonical random
|
||||
pick for the current scale. The dropdown is rebuilt on every scale landing, so it
|
||||
always lists the current pool; single-clip scales (cosmos/orbit) show one entry.
|
||||
2. **Active clip** — `id`, `title`, `source`, `license`, `base_file`.
|
||||
3. **Annotations & affect** — each annotation key with salience (or legacy
|
||||
`min_level`), time window (`appear`–`disappear`), and tier count; each affect key
|
||||
with `min_level` and tier count. Read-only.
|
||||
4. **RenderPlan** — the relocated `#readout` (the `/api/alteration` JSON).
|
||||
5. **Cache & playback** — preload status (`N/total cached`), whether the active
|
||||
clip is served **from memory** (a preloaded blob) vs the network, and live video
|
||||
resolution / duration / loop position.
|
||||
|
||||
## Behavior
|
||||
|
||||
- **Pool override is per-landing.** Changing altitude re-rolls randomly as today;
|
||||
the operator then picks again from the new scale's pool. The pick is not persisted
|
||||
across landings (YAGNI — "select any video while on an altitude").
|
||||
- **Refresh points.** Pool picker + clip metadata + annotations rebuild whenever the
|
||||
scale lands (hooked into `renderScaleReadout()`) and after a manual pick/re-roll.
|
||||
The RenderPlan readout updates on each `update()` (knob change) as today. The
|
||||
cache/playback stats refresh on a light interval (~500 ms) while Dev Mode is on,
|
||||
and the cache count also ticks up as the preloader fills.
|
||||
- **`pickRandomMember()`** is factored out of `landScale()` so the initial landing,
|
||||
ring advances, and the Re-roll button all share the one server-canonical pick.
|
||||
|
||||
## Testing
|
||||
|
||||
No JS unit harness in the repo and no API change, so verification is: `node --check`
|
||||
on `app.js`, the Python suite stays green (unaffected), and by-eye review by the
|
||||
operator (no headless browser yet — §9 E2E policy noted, machinery pending).
|
||||
@@ -0,0 +1,274 @@
|
||||
# Audio + Video — separated Visual & Audio controls — design
|
||||
|
||||
**Status:** proposed → built (sessions 0020 build + 0021 revision)
|
||||
**Date:** 2026-06-26
|
||||
**Session:** 0020 (audio) · 0021 (live-UI revision)
|
||||
|
||||
> **v1 build revision (session 0021, operator-driven):** the live **Audio** control
|
||||
> ships as a simple **off / soundtrack** toggle (styled like the Dev Mode switch),
|
||||
> and **Video** is the matching on/off toggle. **White-noise is deferred** with music
|
||||
> (the orthogonal Visual × Audio model and the synthesized-pink-noise machinery
|
||||
> remain in the codebase for when it's wanted). Two real-browser bugs were fixed in
|
||||
> this revision: video-off now blanks GPU-composited layers (z-index + explicit layer
|
||||
> hide), and the soundtrack now plays on Safari/iOS (audio elements unlocked
|
||||
> synchronously inside the start gesture). Sections below describe the fuller
|
||||
> off/soundtrack/white-noise design; v1 live = off/soundtrack.
|
||||
>
|
||||
> **Startup revision (session 0022):** the tap-to-start wall is **removed** — the app
|
||||
> opens behind a **"Loading Universe…"** splash (shown until all media is preloaded
|
||||
> and the experience can run smoothly), then both toggles start **off** (black +
|
||||
> silent). Autoplay is satisfied without a wall: a **single** `<audio>` element is
|
||||
> played **synchronously inside the toggle's own click** (the reliable Safari unlock),
|
||||
> replacing the A/B-crossfade-after-gesture approach that failed on real Safari. The
|
||||
> first time **Video** is turned on, **Audio** turns on with it (one flip = the full
|
||||
> experience); turning Audio on first stays audio-only. Altitude changes swap the
|
||||
> single element's source with a brief fade.
|
||||
**Extends / reframes:** [`2026-06-26-networked-control-surface-design.md`](./2026-06-26-networked-control-surface-design.md)
|
||||
§3 (control inventory) and §5 (server contract) — its bundled **Content** selector
|
||||
is split into two orthogonal controls. Everything else in that spec (server-of-truth
|
||||
+ SSE, the two pages, the Arduino seam, transition lifecycle) stands unchanged.
|
||||
**Asset record:** [`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md).
|
||||
|
||||
---
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Give the experience **sound**, and make audio and video **independently
|
||||
controllable**. Today the control surface bundles audio+video into a single 7-way
|
||||
**Content** selector (*video / audio+video / music+video / off / white noise /
|
||||
music / audio track*). That bundle is both awkward to operate and **incomplete** —
|
||||
it has no "white noise **with** video," for instance.
|
||||
|
||||
Replace it with **two orthogonal dials** — **Visual** and **Audio** — so every
|
||||
combination is reachable, including the ones the bundle omitted. Operator framing
|
||||
(2026-06-26): *"separate these dials… it's ok to let people have e.g. white noise
|
||||
with video."*
|
||||
|
||||
## 2. Core idea: orthogonalize Content into Visual × Audio
|
||||
|
||||
```
|
||||
BEFORE (bundled) AFTER (orthogonal)
|
||||
Content: 7-way selector Visual: on / off
|
||||
video Audio: off / soundtrack / white noise
|
||||
audio+video (· music — deferred)
|
||||
music+video ──▶ ───────────────────────────────────────
|
||||
off Any Visual × Any Audio is now reachable:
|
||||
white noise video + soundtrack (was "audio+video")
|
||||
music video + white noise (NEW — was missing)
|
||||
audio track black + white noise (was "white noise")
|
||||
black + soundtrack (was "audio track")
|
||||
```
|
||||
|
||||
The 7 bundled modes were really points in a **2 × N grid** (visual on/off ×
|
||||
audio source). Making the grid explicit removes the awkwardness and fills the gap.
|
||||
The **Altitude** dial is unchanged and remains the single scale selector.
|
||||
|
||||
## 3. The controls (reframes control-surface §3)
|
||||
|
||||
| Control | Positions (v1) | Drives | Class (§4 of control-surface spec) |
|
||||
|---|---|---|---|
|
||||
| **Altitude** | endless dial, cosmos→abyss (wraps) | the visual **scale** **and**, when Audio=soundtrack, the matching ambience | transitional |
|
||||
| **Visual** | on / off | video shown vs. black | transitional (video fade) |
|
||||
| **Audio** | off / soundtrack / white noise (· *music* deferred) | the audio **source** | live + short crossfade (§7) |
|
||||
| **Left** / **Right** / **Mood** | unchanged | the alteration engine | live |
|
||||
|
||||
So the panel grows from 5 controls to **6** (Content → Visual + Audio). The
|
||||
physical form (later): Visual = a toggle/2-pos switch; Audio = a small rotary
|
||||
selector (3-pos in v1, 4 with music).
|
||||
|
||||
## 4. Audio model (the coupling decision)
|
||||
|
||||
**Decision (operator):** the audio dial is a **source** selector; the per-altitude
|
||||
soundtrack **follows the single Altitude dial**. There is **no** separate audio
|
||||
altitude.
|
||||
|
||||
- **`soundtrack`** → plays the ambience of the **current Altitude scale**. Turning
|
||||
Altitude to coast shows coast video *and* plays coast waves; the soundtrack
|
||||
**crossfades** as the altitude transition settles (mirrors the video ring
|
||||
crossfade). One dial, two coupled outputs.
|
||||
- **`white noise`** → a single global bed, **altitude-independent**. Changing
|
||||
Altitude does not change it. This is what makes "white noise + any video" work.
|
||||
- **`off`** → silence. Video (if Visual=on) plays mute.
|
||||
- **`music`** *(deferred)* → the composed per-altitude music layer. Reserved as a
|
||||
future Audio position; **not in v1** (no assets yet — see
|
||||
[`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md) "two audio layers").
|
||||
|
||||
Independence summary: **Visual ⟂ Audio** (any pairing allowed); **Audio source ⟂
|
||||
Altitude** for white noise/music, **coupled to Altitude** for soundtrack.
|
||||
|
||||
## 5. Assets & manifest
|
||||
|
||||
### 5.1 Per-scale soundtrack
|
||||
|
||||
The five ambiences are sourced and reviewed (PD/CC0; record in
|
||||
[`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md), local-only review
|
||||
gallery `simulator/static/review_audio.html`):
|
||||
|
||||
| scale | soundtrack | license |
|
||||
|---|---|---|
|
||||
| cosmos | Pillars of Creation sonification | PD (NASA) |
|
||||
| orbit | station room-tone hum | CC0 |
|
||||
| coast | sea waves + terns | CC0 |
|
||||
| reef | 6-layer reef soundscape | PD (NOAA) + CC0 |
|
||||
| abyss | blue-whale moan | PD (NOAA) |
|
||||
|
||||
The manifest's `Scale` gains an **`audio`** field (path to the scale's soundtrack),
|
||||
parallel to how clips are referenced; media stays gitignored and is served at
|
||||
`/media/audio/<scale>/<file>`. Soundtracks are a **per-scale single** (not a
|
||||
rotating pool) in v1 — one bed per altitude. (A rotating audio pool, like the clip
|
||||
pool, is a possible later extension; out of v1.)
|
||||
|
||||
### 5.2 White noise
|
||||
|
||||
Altitude-independent global asset. **Synthesized, not sourced** — a calm
|
||||
**pink/brown** noise (gentler than pure white) generated deterministically with
|
||||
`ffmpeg -f lavfi -i anoisesrc=color=pink` → a clean ~60 s loop. Zero licensing
|
||||
concerns. (Colloquially "white noise"; spec a calm colored noise.)
|
||||
|
||||
### 5.3 Production pass (assets)
|
||||
|
||||
Before ship the source clips get a light, deterministic pass (the existing
|
||||
`tools/pipeline` ffmpeg approach): **seamless loop** (crossfade the loop point),
|
||||
**normalize loudness** across the five so altitudes match, and a consistent format
|
||||
(mp3/ogg). Several are short (abyss 13 s, cosmos 30 s) and must loop cleanly. The
|
||||
6-layer reef bake is already loop-length (60 s).
|
||||
|
||||
## 6. Server contract (reframes control-surface §5)
|
||||
|
||||
The control-surface spec's authoritative `SessionState` + four endpoints are
|
||||
unchanged in shape; only the **controls** payload changes: `content` is replaced
|
||||
by `visual` + `audio`, and the derived `render.content` is split.
|
||||
|
||||
### 6.1 `GET /api/state` — updated `controls` + `render`
|
||||
|
||||
```json
|
||||
{
|
||||
"seq": 42,
|
||||
"controls": { "visual": "on", "audio": "soundtrack", "left": 3, "right": 1, "mood": -2 },
|
||||
"altitude": { "index": 2, "scale": "coast", "clip_id": "coast_07" },
|
||||
"transitions": { "altitude": "idle", "visual": "idle" },
|
||||
"render": {
|
||||
"plan": { },
|
||||
"video": { "shown": true },
|
||||
"audio": { "source": "soundtrack", "url": "/media/audio/coast/waves.mp3", "altitude_coupled": true }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `audio.url` is resolved **server-side** from `(audio source, altitude)`:
|
||||
`soundtrack` → the current scale's `audio`; `white_noise` → the global noise
|
||||
asset; `off` → `null`. The renderer just plays `audio.url` (or nothing).
|
||||
- `transitions` tracks `visual` instead of `content` (Audio is not transitional —
|
||||
§7).
|
||||
|
||||
### 6.2 `POST /api/control`
|
||||
|
||||
```json
|
||||
{ "set": { "audio": "white_noise" } } // audio source change
|
||||
{ "set": { "visual": "off" } } // visual fade to black
|
||||
{ "set": {}, "altitude_delta": -1 } // altitude tick (also re-resolves audio.url when source=soundtrack)
|
||||
```
|
||||
|
||||
Server applies, bumps `seq`, recomputes `render.audio.url`, and broadcasts.
|
||||
Absolute `set` values keep retries idempotent (control-surface §5.6).
|
||||
|
||||
### 6.3 SSE events
|
||||
|
||||
- `state` — carries the new controls/render; emitted on Audio change and on any
|
||||
live change (Audio source change rides the `state` event — §7).
|
||||
- `ring` — altitude move; now **also** carries the new `audio.url` so the renderer
|
||||
can crossfade the soundtrack as it plays the visual transition (when
|
||||
source=soundtrack).
|
||||
- `mode` — repurposed to the **Visual** fade (on/off). (Same renderer fade machinery
|
||||
the bundled Content used.)
|
||||
|
||||
## 7. Live vs. transitional classification
|
||||
|
||||
- **Visual on/off → transitional.** It's a visible video fade (to/from black); it
|
||||
uses the renderer's existing fade + `settled` ack, exactly as the bundled
|
||||
Content's fade did. `transitions.visual` flips `transitioning`→`idle`.
|
||||
- **Audio source → live, with a short renderer-side crossfade.** Swapping
|
||||
soundtrack/white-noise/off is a ~0.5–1 s **gain crossfade** between two `<audio>`
|
||||
elements — perceptible but **non-blocking** and not a renderer animation with
|
||||
unknown timing. So it's classified **live** (no busy spinner, no `settled` ack),
|
||||
like Left/Right/Mood. *Flagged decision:* if the operator wants the Audio knob to
|
||||
feel "busy" during its fade, it can be promoted to transitional later; v1 keeps it
|
||||
live for simplicity.
|
||||
- **Altitude → transitional** (unchanged). When source=soundtrack, the renderer
|
||||
crossfades the audio bed in step with the ring transition; the audio crossfade is
|
||||
slaved to the visual transition and needs no separate ack.
|
||||
|
||||
## 8. Renderer behavior (extends control-surface §6.1)
|
||||
|
||||
The display page adds an **audio layer**:
|
||||
|
||||
- Two `<audio>` elements (A/B) for gapless gain crossfades; a tiny crossfade helper.
|
||||
- On `state`: reconcile `audio.url` — if it changed, crossfade A↔B to the new url
|
||||
(or fade to silence for `off`). White-noise and soundtrack are the same mechanism;
|
||||
only the url differs.
|
||||
- On `ring` (altitude move) with source=soundtrack: start the soundtrack crossfade
|
||||
to the event's new `audio.url` so it lands with the visual scale.
|
||||
- On `mode` (Visual): fade the **video** to/from black; audio is untouched (you can
|
||||
hear soundtrack/white-noise over black).
|
||||
- Each soundtrack loops (the production-pass clean loop); white-noise loops.
|
||||
- Autoplay: the renderer is launched full-screen by the operator; a one-time user
|
||||
gesture (or `--autoplay`/muted-then-unmute) satisfies browser autoplay policy.
|
||||
*Flagged:* document the launch gesture so the projection starts audio reliably.
|
||||
|
||||
## 9. Testing (mandatory pipeline tiers)
|
||||
|
||||
- **Pure unit** — `render.audio.url` resolution from `(source, altitude)`:
|
||||
soundtrack→scale asset, white_noise→global, off→null; altitude tick re-resolves
|
||||
only when source=soundtrack; `seq` bump. No I/O.
|
||||
- **API/contract** — `POST /api/control {visual}` flips `transitions.visual`;
|
||||
`{audio}` rides a `state` event with the new url and **no** transition flag; bad
|
||||
enum → 422; `GET /api/state` shape.
|
||||
- **E2E (Playwright)** — renderer + `/remote`: set Audio=white_noise → assert the
|
||||
renderer's audio element src is the noise asset; tick Altitude with
|
||||
Audio=soundtrack → assert the soundtrack url crossfades to the new scale; set
|
||||
Visual=off → assert the video fades and audio keeps playing. (Audio *playback*
|
||||
asserted via element `src`/`paused`, not waveform.)
|
||||
|
||||
## 10. Scope
|
||||
|
||||
**In v1:**
|
||||
|
||||
- Split **Content** → **Visual** (on/off) + **Audio** (off / soundtrack / white
|
||||
noise) across `SessionState`, `/api/state`, `/api/control`, SSE, and both pages.
|
||||
- Per-scale `audio` in the manifest + the 5 sourced soundtracks (production-passed
|
||||
loops); synthesized white-noise asset.
|
||||
- Server-side `audio.url` resolution; renderer audio layer with crossfades;
|
||||
altitude-coupled soundtrack crossfade.
|
||||
|
||||
**Explicitly NOT in v1 (deferred):**
|
||||
|
||||
- **Music layer** (`audio=music`) — reserved dial position; no assets yet.
|
||||
- **Separate audio altitude** (hear reef while seeing coast) — rejected coupling
|
||||
option; soundtrack follows the one Altitude dial.
|
||||
- **Rotating audio pools** per scale (one bed per altitude in v1).
|
||||
- Per-altitude white-noise variants (one global noise).
|
||||
- Physical Visual/Audio switches (Arduino) — contract only, per control-surface §7.
|
||||
|
||||
## 11. Spec & roadmap impact
|
||||
|
||||
- **Reframes** control-surface §3/§5: `content` → `visual` + `audio`;
|
||||
`transitions.content` → `transitions.visual`; `render.content` → `render.video`
|
||||
+ `render.audio`. That spec is still *proposed* (unbuilt), so this is a clean
|
||||
pre-implementation edit, not a migration. Both can be built together, or this
|
||||
one folds into the control-surface build as its Content step.
|
||||
- The **music layer** becomes a tracked follow-on (compose + source per-altitude
|
||||
music; flip on the reserved Audio position).
|
||||
- `ROADMAP.md` to note the audio capability at implementation time.
|
||||
|
||||
## 12. Open questions / flagged — RESOLVED (session 0020 build)
|
||||
|
||||
- **Audio knob feel** (§7) — **RESOLVED: live**, with a ~0.6 s renderer-side gain
|
||||
crossfade (`XFADE_MS=600`), no `settled` ack. Promote to transitional later only
|
||||
if the iPad makes it feel like it needs a "busy" state.
|
||||
- **White-noise color** — **RESOLVED: pink** (`anoisesrc=color=pink`), a balanced
|
||||
calm bed. Brown is a one-parameter alternative (`color=brown`) if a deeper rumble
|
||||
is wanted; the build is parameterized for it.
|
||||
- **Autoplay gesture** (§8) — **RESOLVED: a one-time tap-to-start overlay**
|
||||
("▶ tap to start sound", `#start-gesture`). The first click unlocks both `<audio>`
|
||||
elements within the user gesture; thereafter audio follows state. This is the
|
||||
documented launch step for the full-screen projection.
|
||||
@@ -0,0 +1,331 @@
|
||||
# Networked Control Surface — design
|
||||
|
||||
**Status:** proposed
|
||||
**Date:** 2026-06-26
|
||||
**Session:** 0019
|
||||
**Supersedes/reframes:** ROADMAP sub-project 4 (Arduino Firmware / control panel) and
|
||||
the "serial input" slice of sub-project 3 (Player Runtime).
|
||||
|
||||
---
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Split the existing simulator into a **renderer** (the laptop full-screen
|
||||
projection — "the experience") and a separate **controller** (a touch remote),
|
||||
talking over the local network, so the experience can be driven from a handheld
|
||||
device the way the real installation's panel drives its screen.
|
||||
|
||||
This is **v1 hardware**, deliberately simulator-friendly:
|
||||
|
||||
- The **laptop/computer renders** (no Raspberry Pi yet).
|
||||
- The controller is a **separate device** over **wifi** (LAN), not a USB-serial
|
||||
link to a Pi.
|
||||
- The controller is **prototyped as a web page** (an iPad/tablet/phone twin),
|
||||
and a **physical Arduino remote** drops in later as just another control source
|
||||
speaking the same contract.
|
||||
|
||||
Two goals, equally weighted (operator, session 0019): **decouple control from
|
||||
display cleanly**, *and* get the **handheld "stand back and drive it" feel**.
|
||||
|
||||
## 2. Core idea: one server of truth, interchangeable control sources
|
||||
|
||||
```
|
||||
┌──────────────────┐ POST /api/control ┌─────────────────────┐
|
||||
│ CONTROL SOURCE │ ───── (knob change) ────▶ │ FastAPI server │
|
||||
│ │ │ (laptop) │
|
||||
│ • iPad web remote│ ◀──── GET /api/state ──── │ ┌───────────────┐ │
|
||||
│ (v1) │ (sync on load) │ │ SessionState │ │
|
||||
│ • Arduino (later)│ │ │ controls + │ │
|
||||
└──────────────────┘ │ │ ring index + │ │
|
||||
│ │ transitions + │ │
|
||||
┌──────────────────┐ GET /api/events (SSE) │ │ seq (version) │ │
|
||||
│ RENDERER │ ◀═══ state/ring/mode ════ │ └───────┬───────┘ │
|
||||
│ laptop full- │ (live push) └──────────┼──────────┘
|
||||
│ screen │ │
|
||||
│ (the experience) │ ──── POST /api/render/event ────────▶│ (settled ack)
|
||||
└──────────────────┘ recompute RenderPlan + ring move
|
||||
```
|
||||
|
||||
- **Server** owns the single authoritative `SessionState` (current controls,
|
||||
altitude/ring index, per-field transition state, a monotonic `seq`). Every
|
||||
change recomputes the derived view and broadcasts it.
|
||||
- **Control source** = anything that `POST`s a control change. v1 has exactly
|
||||
one: the iPad web remote. A future Arduino is *another instance of the same
|
||||
role* — no new server concept.
|
||||
- **Renderer** = the laptop projection. It holds **no** input UI; it subscribes
|
||||
to the SSE stream and renders whatever state arrives. It is today's
|
||||
`simulator/static/index.html`, stripped of its sliders, given a `state`
|
||||
listener plus one small upstream ack channel.
|
||||
|
||||
The property that matters: controller and renderer are now **separate web pages
|
||||
off the same server**, decoupled exactly as a real installation's panel and
|
||||
screen are. (Chosen architecture: server-authoritative session + **Server-Sent
|
||||
Events** push. SSE fits because the directions are asymmetric — the renderer only
|
||||
*receives*, the control source only *sends* — so no bidirectional WebSocket is
|
||||
needed. Alternatives considered and rejected: renderer polling — laggy/wasteful;
|
||||
peer-to-peer WebRTC — most plumbing and it breaks the clean single-contract seam
|
||||
that makes the Arduino a drop-in.)
|
||||
|
||||
## 3. Control surface: the six controls (the faithful twin)
|
||||
|
||||
The iPad remote is a **faithful on-screen twin** of the intended physical panel —
|
||||
what you like on the iPad is the spec for the Arduino panel. The control
|
||||
inventory is exactly what the sim exposes today:
|
||||
|
||||
| Control | Sim today | Physical form (later) |
|
||||
|---|---|---|
|
||||
| **Visual** | toggle (show video on/off) | 2-position switch |
|
||||
| **Audio** | on/off toggle (on = per-altitude soundtrack; ·white-noise & music deferred) | 2-position switch (rotary selector when more sources land) |
|
||||
| **Altitude** | endless circular dial — walks the 5 scales (cosmos→abyss), wraps | rotary encoder (endless, detented) |
|
||||
| **Left** (analytical) | slider 0–4 | knob/pot, 5 detents |
|
||||
| **Right** (dreamlike) | slider 0–4 | knob/pot, 5 detents |
|
||||
| **Mood** | slider −4…+4 (dark↔light) | center-detented knob |
|
||||
|
||||
(This is the original sub-project-4 inventory, *updated*: Dark+Light collapsed
|
||||
into one **Mood** knob in session 0013, the **Altitude** encoder added in session
|
||||
0018, and the bundled 7-way **Content** dial split into orthogonal **Visual** ×
|
||||
**Audio** controls in session 0020 — see
|
||||
[`2026-06-26-audio-video-separated-controls-design.md`](./2026-06-26-audio-video-separated-controls-design.md).)
|
||||
|
||||
## 4. Live vs. transitional controls (per-knob transition state)
|
||||
|
||||
The five controls split into two classes by how a **human** perceives the change:
|
||||
|
||||
- **Live controls — Left, Right, Mood, Audio.** The alteration engine treats
|
||||
Left/Right/Mood as `LIVE_UPDATE` (the WebGL Kuwahara dream + HUD overlay + color
|
||||
grade re-tune within a frame). **Audio** source changes are also live: a short
|
||||
(~0.6 s) renderer-side gain crossfade between two `<audio>` elements —
|
||||
perceptible but non-blocking, with no `settled` ack (audio spec §7). These are
|
||||
*always idle*; the remote just shows the value.
|
||||
- **Transitional controls — Altitude, Visual.** These trigger a renderer
|
||||
animation that takes real, perceptible time: Altitude plays ring-transition
|
||||
clips (zoom/warp between scales); **Visual** on/off does fade-to-black /
|
||||
fade-from-black. During that window the control is **busy**, and the control
|
||||
source must model that so its feedback is honest.
|
||||
|
||||
Per-transitional-control lifecycle:
|
||||
|
||||
```
|
||||
idle ──(source sends)──▶ pending ──(renderer starts)──▶ transitioning ──(renderer settles)──▶ idle
|
||||
```
|
||||
|
||||
- **pending** — set locally by the remote the instant the knob is turned
|
||||
(immediate tactile feedback before the server replies).
|
||||
- **transitioning** — the server flags the field once the renderer actually
|
||||
begins the animation; broadcast to *all* control sources.
|
||||
- **idle** — cleared when the **renderer acks completion** (the only party that
|
||||
knows real playback timing), with a server-side max-duration timeout as a
|
||||
safety net so a dropped renderer can't wedge a knob.
|
||||
|
||||
Hardware payoff: a physical Altitude encoder can light an LED while
|
||||
`transitioning`; the iPad twin shows the matching spinner/disabled-detent.
|
||||
|
||||
`SessionState` therefore carries
|
||||
`transitions: { altitude: idle|transitioning, visual: idle|transitioning }`
|
||||
alongside the controls and `seq`. (Live controls — Left/Right/Mood/Audio — need
|
||||
no entry.)
|
||||
|
||||
## 5. Server contract
|
||||
|
||||
Four endpoints carry the whole thing. The existing `/api/alteration`,
|
||||
`/api/ring`, `/api/ring/advance` remain as **internal helpers the server calls**;
|
||||
control sources never hit them directly.
|
||||
|
||||
### 5.1 `GET /api/state` — snapshot (sync-on-load / polling fallback)
|
||||
|
||||
```json
|
||||
{
|
||||
"seq": 42,
|
||||
"controls": { "visual": "on", "audio": "soundtrack", "left": 3, "right": 1, "mood": -2 },
|
||||
"altitude": { "index": 2, "scale": "coast", "clip_id": "coast_07" },
|
||||
"transitions": { "altitude": "idle", "visual": "idle" },
|
||||
"render": {
|
||||
"plan": { },
|
||||
"video": { "shown": true },
|
||||
"audio": { "source": "soundtrack", "url": "/media/audio/coast/waves.loop.mp3", "altitude_coupled": true }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(The `render.video` + `render.audio` split and server-side `audio.url` resolution
|
||||
are specified in the audio spec §6.1.)
|
||||
|
||||
A control source GETs this on load to draw its knobs in the right positions; the
|
||||
renderer can use it to cold-start or resync after a reconnect.
|
||||
|
||||
### 5.2 `POST /api/control` — the one control-source contract
|
||||
|
||||
A partial patch — only the fields that changed:
|
||||
|
||||
```json
|
||||
{ "set": { "left": 4 }, "altitude_delta": 0 } // live knob
|
||||
{ "set": { "audio": "white_noise" }, "altitude_delta": 0 } // live (audio source)
|
||||
{ "set": { "visual": "off" }, "altitude_delta": 0 } // transitional (video fade)
|
||||
{ "set": {}, "altitude_delta": -1 } // one encoder tick down
|
||||
```
|
||||
|
||||
Server applies it, bumps `seq`, recomputes, and:
|
||||
|
||||
- **live field** (Left/Right/Mood, **or an `audio` source change**) → broadcasts
|
||||
a `state` event immediately (the renderer crossfades audio on its own, §7);
|
||||
- **transitional field** (a `visual` change, or a non-zero `altitude_delta`) →
|
||||
computes the ring move / video fade, marks that field `transitioning`,
|
||||
broadcasts a `ring` (or `mode`) event for the renderer to animate.
|
||||
|
||||
Returns the new snapshot (an immediate authoritative echo for the poster).
|
||||
Absolute `set` values make retries idempotent; `altitude_delta` is relative
|
||||
(encoder ticks) and the server coalesces rapid ticks via the existing
|
||||
`advance_ring` fast-spin logic.
|
||||
|
||||
### 5.3 `GET /api/events` — SSE stream (renderer subscribes)
|
||||
|
||||
Event types:
|
||||
|
||||
- `state` — full snapshot; sent on connect and on every live change (including an
|
||||
**audio** source change, which the renderer crossfades — §7).
|
||||
- `ring` — an altitude move: the transition clip(s) to play + the landed
|
||||
scale/clip, **plus the new `audio.url`** so the renderer can crossfade the
|
||||
soundtrack in step (when `audio=soundtrack`). Renderer plays them, then settles.
|
||||
- `mode` — a **Visual** on/off change: the video fade the renderer should run.
|
||||
- `ping` — keepalive.
|
||||
|
||||
The SSE event payloads reuse the existing serializers
|
||||
(`ring_move_to_dict`, `render_plan_to_dict`, etc.).
|
||||
|
||||
### 5.4 `POST /api/render/event` — the renderer's one upstream ack
|
||||
|
||||
```json
|
||||
{ "type": "settled", "seq": 42, "field": "altitude" }
|
||||
```
|
||||
|
||||
Sent when the renderer finishes a transition. Server flips that field `idle` and
|
||||
broadcasts a `state` so every remote clears its spinner. A server-side timeout
|
||||
(max transition duration + margin) clears it anyway if the renderer never acks.
|
||||
|
||||
### 5.5 End-to-end flow for one altitude tick
|
||||
|
||||
```
|
||||
iPad: turn dial → field=pending locally → POST /api/control {altitude_delta:-1}
|
||||
server: apply, seq++, transitions.altitude="transitioning"
|
||||
→ SSE "ring" to renderer → SSE "state" to all remotes (altitude busy)
|
||||
renderer: play transition clip(s), settle on forest_07
|
||||
→ POST /api/render/event {type:"settled", field:"altitude"}
|
||||
server: transitions.altitude="idle" → SSE "state" to all → iPad clears spinner
|
||||
```
|
||||
|
||||
### 5.6 Decisions flagged (not assumed silently)
|
||||
|
||||
- **Concurrency = last-write-wins.** Multiple control sources are allowed;
|
||||
per-field last write wins, `seq` orders everything. No locking in v1.
|
||||
- **No auth.** v1 trusts the LAN — any device that can reach the laptop can drive
|
||||
it. A room-token is deferred until wanted.
|
||||
|
||||
## 6. The two pages
|
||||
|
||||
### 6.1 Renderer — `/` (display), refactored from today's `index.html`
|
||||
|
||||
- **Removes** the input controls (the Visual toggle, the Audio `<select>`, the
|
||||
Altitude SVG, the three sliders) and their handlers.
|
||||
- **Keeps** everything that draws the experience: the video element, the **A/B
|
||||
`<audio>` crossfade layer**, the WebGL Kuwahara dream shader, the
|
||||
HUD/annotation + affect overlay, the ring-transition playback, the media
|
||||
preload pool, the crossfade-loop logic.
|
||||
- **Adds** a thin `state` client: open `EventSource('/api/events')`; on `state`
|
||||
apply controls→render live (incl. the audio crossfade); on `ring` play the
|
||||
transition clip(s) — crossfading the soundtrack to the event's `audio.url` —
|
||||
then settle and `POST /api/render/event {settled}`; on `mode` run the video
|
||||
fade. On (re)connect,
|
||||
pull `GET /api/state` to resync. SSE auto-reconnects, so a flaky link
|
||||
self-heals.
|
||||
- Runs full-screen on the laptop driving the projector; no keyboard/mouse needed
|
||||
once launched.
|
||||
|
||||
### 6.2 Controller — `/remote`, a new touch-first page (the faithful twin)
|
||||
|
||||
- The six controls laid out like the intended physical panel: **Visual** toggle +
|
||||
**Audio** selector, **Altitude** encoder dial (reuse the existing SVG dial
|
||||
component), **Left** / **Right** / **Mood** knobs.
|
||||
- On any input → `POST /api/control` with just the changed field. Live knobs post
|
||||
on drag; Altitude posts encoder ticks; Visual/Audio post on change.
|
||||
- Per-knob transition state (§4): a transitional control goes **pending** on
|
||||
touch, reflects **transitioning** from the SSE feed, clears on **settled**.
|
||||
Altitude shows a settling indicator and its detents soft-lock (further ticks
|
||||
queue as the next target rather than stacking); **Visual** shows the toggle as
|
||||
pending until the video fade completes; **Audio** is live (no pending state).
|
||||
- It **also** subscribes to `/api/events` so it stays in sync with other
|
||||
sources (turn the future Arduino's knob and the iPad reflects it, and
|
||||
vice-versa). On load it GETs `/api/state` to draw current positions.
|
||||
- Touch-sized hit targets, no hover dependence, works in mobile Safari with no
|
||||
install.
|
||||
|
||||
## 7. The future-Arduino seam (contract only in v1)
|
||||
|
||||
No firmware in this feature, but the design **proves the seam** so sub-project 4
|
||||
is a true drop-in:
|
||||
|
||||
- **ESP32 (wifi Arduino)** → posts straight to `POST /api/control` over the LAN.
|
||||
Zero server change. Recommended hardware target when the time comes.
|
||||
- **Classic USB Arduino** → emits serial frames to a tiny `bridge.py` on the
|
||||
laptop that translates each frame into a `POST /api/control`. The old roadmap
|
||||
"3⇄4 serial framing contract" shrinks to *just* the Arduino↔bridge link; the
|
||||
renderer/server never know the difference.
|
||||
- Either way the Arduino sends **absolute pot values** (`set`) + **encoder
|
||||
ticks** (`altitude_delta`) — exactly the iPad's message shape.
|
||||
|
||||
This is what makes "faithful twin" real: the iPad and the Arduino are the same
|
||||
control source speaking the same contract.
|
||||
|
||||
## 8. Error handling / resilience
|
||||
|
||||
- **Renderer drops:** SSE auto-reconnects; on reconnect it GETs `/api/state` and
|
||||
resyncs. The per-field transition timeout clears any knob left `transitioning`
|
||||
by a renderer that vanished mid-animation.
|
||||
- **Controller drops / flaky wifi:** POSTs are fire-and-retry; absolute `set`
|
||||
values are idempotent, so a retried knob value is harmless. `altitude_delta` is
|
||||
the one relative input — on a failed POST the remote keeps the tick in its
|
||||
local pending target and resends rather than double-counting.
|
||||
- **No renderer connected:** the server still accepts control changes and holds
|
||||
state; a renderer that connects later catches up from the snapshot.
|
||||
- **Stale control source:** every event carries `seq`; a source ignores events
|
||||
older than what it has applied.
|
||||
|
||||
## 9. Testing (satisfies the mandatory pipeline tiers)
|
||||
|
||||
- **Pure unit** — a `SessionState` store in Python: apply patch → live vs.
|
||||
transitional classification; `altitude_delta` → ring move via the existing
|
||||
`advance_ring`; `seq` bump; transition flag set/clear; timeout. No I/O, tested
|
||||
like the rest of `player/`.
|
||||
- **API/contract** — `GET /api/state`; `POST /api/control` (live + transitional +
|
||||
bad input → 422); `POST /api/render/event` clears the flag; the SSE
|
||||
event-builder tested as a pure function.
|
||||
- **E2E (Playwright, required for the UI surface)** — open the renderer and
|
||||
`/remote` as two pages: move a live knob → assert the renderer's render
|
||||
updates; tick Altitude → assert a transition plays, the remote shows
|
||||
`transitioning`, then clears on settle.
|
||||
|
||||
## 10. Scope
|
||||
|
||||
**In v1:**
|
||||
|
||||
- Server `SessionState` + the four endpoints (`/api/state`, `/api/control`,
|
||||
`/api/events`, `/api/render/event`).
|
||||
- Renderer refactor to display-only + SSE client + settled ack.
|
||||
- The `/remote` iPad twin: the five controls + per-knob transition state.
|
||||
- Wifi/LAN only; laptop renders.
|
||||
|
||||
**Explicitly NOT in v1 (deferred):**
|
||||
|
||||
- Any Arduino/firmware (contract defined here; hardware later).
|
||||
- Bluetooth / USB transport.
|
||||
- Auth / room-token.
|
||||
- The Pi renderer.
|
||||
- Multi-room / multi-renderer fan-out (the SSE design allows it; not a v1 goal).
|
||||
|
||||
## 11. Roadmap impact
|
||||
|
||||
- **Sub-project 4** is reframed: the panel is now "any control source over the
|
||||
`/api/control` contract," not USB-serial-to-Pi first. The iPad twin is the v1
|
||||
deliverable; the Arduino is a later drop-in.
|
||||
- **Sub-project 3's "serial input" slice** is absorbed into a transport detail
|
||||
(the Arduino↔bridge link), no longer a Pi-coupled concern.
|
||||
- `ROADMAP.md` to be updated at implementation time.
|
||||
+17
-14
@@ -88,11 +88,12 @@ class AnalyticalOverlay:
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AffectOverlay:
|
||||
"""Affect channel (Left x Right): emotion-words surfaced only when BOTH the
|
||||
analytical (Left) and dreamlike (Right) knobs are up. `strength` is
|
||||
`min(left, right)` (0..4) — a runtime affect track picks which feeling words
|
||||
appear by it; `intensity` 0..1 is their opacity. The dream leaking into the
|
||||
machine's reading (affect-channel design, session 0013)."""
|
||||
"""Affect channel (RIGHT brain): emotion-words surfaced by the Right (dreamlike)
|
||||
knob alone — the analytical Left knob does NOT gate them. `strength` is the Right
|
||||
knob (0..4) — a runtime affect track picks which feeling words appear by it;
|
||||
`intensity` 0..1 is their opacity. Feeling is the right brain's output, surfaced
|
||||
as the machine's felt reading (affect-channel design, session 0013; Right-only
|
||||
since session 0018)."""
|
||||
|
||||
strength: int
|
||||
intensity: float
|
||||
@@ -122,8 +123,9 @@ class RenderPlan:
|
||||
@property
|
||||
def is_identity(self) -> bool:
|
||||
"""True when the plan leaves the neutral base un-altered. Affect needs no
|
||||
clause: `min(left,right) > 0` implies `left > 0`, so `overlay.level > 0`
|
||||
already makes a plan with active affect non-identity."""
|
||||
clause: `affect.strength == right`, and `right > 0` already makes
|
||||
`dream.strength > 0`, so an active-affect plan is non-identity via the
|
||||
dream check below."""
|
||||
return (
|
||||
self.grade.is_identity
|
||||
and self.overlay.level == 0
|
||||
@@ -135,13 +137,14 @@ def _overlay_intensity(left: int, cal: Calibration) -> float:
|
||||
return _clamp(cal.overlay_gain * left / KNOB_MAX, 0.0, 1.0)
|
||||
|
||||
|
||||
def _affect_strength(left: int, right: int) -> int:
|
||||
"""Affect surfaces only when BOTH knobs are up — gated by the smaller one."""
|
||||
return min(left, right)
|
||||
def _affect_strength(right: int) -> int:
|
||||
"""Affect is a RIGHT-brain output: the emotion channel is driven by the Right
|
||||
(dreamlike) knob alone — the analytical Left knob no longer gates it."""
|
||||
return right
|
||||
|
||||
|
||||
def _affect_intensity(left: int, right: int, cal: Calibration) -> float:
|
||||
return _clamp(cal.overlay_gain * _affect_strength(left, right) / KNOB_MAX, 0.0, 1.0)
|
||||
def _affect_intensity(right: int, cal: Calibration) -> float:
|
||||
return _clamp(cal.overlay_gain * _affect_strength(right) / KNOB_MAX, 0.0, 1.0)
|
||||
|
||||
|
||||
def _dream_intensity(right: int, cal: Calibration) -> float:
|
||||
@@ -163,8 +166,8 @@ def plan_alteration(
|
||||
intensity=_overlay_intensity(coord.left, calibration),
|
||||
),
|
||||
affect=AffectOverlay(
|
||||
strength=_affect_strength(coord.left, coord.right),
|
||||
intensity=_affect_intensity(coord.left, coord.right, calibration),
|
||||
strength=_affect_strength(coord.right),
|
||||
intensity=_affect_intensity(coord.right, calibration),
|
||||
),
|
||||
dream=Dream(
|
||||
strength=coord.right,
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Resolve the orthogonal Visual × Audio controls into render outputs (audio spec
|
||||
§4/§6). The single source of truth for: whether the projector shows video, and
|
||||
which audio url plays for a given (audio source, current altitude).
|
||||
|
||||
Replaces player/content.py's 7-way bundled table. Soundtrack follows the single
|
||||
Altitude dial; off is silence. `white_noise` and `music` are deferred sources —
|
||||
not in v1 (the live control is a simple Audio on/off toggle where on=soundtrack)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
VISUAL_POSITIONS = frozenset({"on", "off"})
|
||||
# v1 audio sources. The live UI is an Audio on/off toggle (on=soundtrack).
|
||||
# "white_noise" and "music" are deferred (no live control) and intentionally absent.
|
||||
AUDIO_SOURCES = frozenset({"off", "soundtrack"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AudioResolution:
|
||||
source: str
|
||||
url: str | None
|
||||
altitude_coupled: bool
|
||||
|
||||
|
||||
def resolve_visual(position: str) -> bool:
|
||||
"""Visual on/off → whether the renderer shows video (vs fade-to-black)."""
|
||||
if position not in VISUAL_POSITIONS:
|
||||
raise ValueError(
|
||||
f"unknown visual position {position!r}; expected one of {sorted(VISUAL_POSITIONS)}"
|
||||
)
|
||||
return position == "on"
|
||||
|
||||
|
||||
def resolve_audio_url(source: str, *, scale_audio: str) -> str | None:
|
||||
"""The audio url for (source, current altitude): soundtrack → the current
|
||||
scale's asset (or None if none authored); off → None. `scale_audio` is the ring
|
||||
scale's `audio` path (relative to /media/audio/)."""
|
||||
if source not in AUDIO_SOURCES:
|
||||
raise ValueError(
|
||||
f"unknown audio source {source!r}; expected one of {sorted(AUDIO_SOURCES)}"
|
||||
)
|
||||
if source == "off":
|
||||
return None
|
||||
# soundtrack — couple to the current altitude's asset
|
||||
return f"/media/audio/{scale_audio}" if scale_audio else None
|
||||
|
||||
|
||||
def resolve_audio(source: str, *, scale_audio: str) -> AudioResolution:
|
||||
"""The full render.audio view: source, resolved url, and whether it follows the
|
||||
Altitude dial (true only for soundtrack)."""
|
||||
url = resolve_audio_url(source, scale_audio=scale_audio)
|
||||
return AudioResolution(source=source, url=url, altitude_coupled=(source == "soundtrack"))
|
||||
@@ -1,42 +0,0 @@
|
||||
"""Resolve the 7-way content dial into an audio source + video on/off (§6).
|
||||
|
||||
The single source of truth for design §6's table: which audio source plays and
|
||||
whether the projector shows video, for each dial position.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
# The four distinct audio sources. "white_noise" is generated at runtime;
|
||||
# "music" is the public-domain classical pool; "audio_track" is the clip's own
|
||||
# field audio; "none" is silence.
|
||||
AUDIO_SOURCES = frozenset({"none", "white_noise", "music", "audio_track"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ContentResolution:
|
||||
audio_source: str
|
||||
video: bool
|
||||
|
||||
|
||||
# Design §6, one row per dial position.
|
||||
_TABLE = {
|
||||
"off": ContentResolution("none", False),
|
||||
"white_noise": ContentResolution("white_noise", False),
|
||||
"music": ContentResolution("music", False),
|
||||
"audio_track": ContentResolution("audio_track", False),
|
||||
"video": ContentResolution("none", True),
|
||||
"music_video": ContentResolution("music", True),
|
||||
"audio_video": ContentResolution("audio_track", True),
|
||||
}
|
||||
|
||||
|
||||
def resolve_content(position: str) -> ContentResolution:
|
||||
"""Map a content-dial position to its audio source and video flag (§6)."""
|
||||
try:
|
||||
return _TABLE[position]
|
||||
except KeyError:
|
||||
raise ValueError(
|
||||
f"unknown content position {position!r}; expected one of {sorted(_TABLE)}"
|
||||
) from None
|
||||
+15
-12
@@ -1,20 +1,17 @@
|
||||
"""Control-panel state: the data shape read from the Arduino over serial.
|
||||
|
||||
This is the serial-contract payload shared with sub-project 4 (firmware). It
|
||||
models the full panel from design §6/§7: the 7-way content dial, the four
|
||||
experience knobs (0..4), and the two intensity levels (volume, brightness).
|
||||
The wire framing itself is the separate 3<->4 serial contract; this module is
|
||||
the *decoded* form.
|
||||
models the full panel from the audio spec §3: a Visual toggle (on/off) + an
|
||||
Audio source selector (off/soundtrack; white-noise/music deferred), the four experience knobs
|
||||
(0..4), and the two intensity levels (volume, brightness). The wire framing
|
||||
itself is the separate 3<->4 serial contract; this module is the *decoded* form.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, fields
|
||||
|
||||
# The seven positions of the content dial (design §6).
|
||||
CONTENT_POSITIONS = frozenset(
|
||||
{"off", "white_noise", "music", "audio_track", "video", "music_video", "audio_video"}
|
||||
)
|
||||
from player.audio import AUDIO_SOURCES, VISUAL_POSITIONS
|
||||
|
||||
KNOB_FIELDS = ("left", "right", "dark", "light", "volume", "brightness")
|
||||
KNOB_MIN = 0
|
||||
@@ -27,7 +24,8 @@ class ControlsError(ValueError):
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Controls:
|
||||
content: str
|
||||
visual: str
|
||||
audio: str
|
||||
left: int
|
||||
right: int
|
||||
dark: int
|
||||
@@ -38,10 +36,15 @@ class Controls:
|
||||
|
||||
def validate_controls(c: Controls) -> None:
|
||||
"""Raise ControlsError if the payload is structurally invalid."""
|
||||
if c.content not in CONTENT_POSITIONS:
|
||||
if c.visual not in VISUAL_POSITIONS:
|
||||
raise ControlsError(
|
||||
f"invalid content position {c.content!r}; "
|
||||
f"expected one of {sorted(CONTENT_POSITIONS)}"
|
||||
f"invalid visual position {c.visual!r}; "
|
||||
f"expected one of {sorted(VISUAL_POSITIONS)}"
|
||||
)
|
||||
if c.audio not in AUDIO_SOURCES:
|
||||
raise ControlsError(
|
||||
f"invalid audio source {c.audio!r}; "
|
||||
f"expected one of {sorted(AUDIO_SOURCES)}"
|
||||
)
|
||||
for name in KNOB_FIELDS:
|
||||
value = getattr(c, name)
|
||||
|
||||
+31
-1
@@ -50,10 +50,26 @@ class RingError(ValueError):
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Scale:
|
||||
"""A node on the ring: a scale of nature and the base clip it plays."""
|
||||
"""A node on the ring: a scale of nature and the clip(s) it can play.
|
||||
|
||||
A scale carries a rotating POOL of vetted clips (content-pipeline design
|
||||
§11.1): when the viewer lands on the scale, the player picks one member at
|
||||
random (`pick_clip_id`). `clip_id` is the PRIMARY member (the deterministic
|
||||
default / pool[0]); `pool` is the full set. A scale written with only
|
||||
`clip_id` and no `pool` is a pool of one — exactly the Increment-1 behavior,
|
||||
so old manifests are unchanged.
|
||||
"""
|
||||
|
||||
id: str
|
||||
clip_id: str
|
||||
pool: tuple[str, ...] = ()
|
||||
audio: str = "" # per-scale soundtrack, relative to /media/audio/ (audio spec §5.1)
|
||||
|
||||
@property
|
||||
def members(self) -> tuple[str, ...]:
|
||||
"""The clip ids this scale can play — the pool, or just `clip_id` when
|
||||
no pool was authored (a pool of one)."""
|
||||
return self.pool if self.pool else (self.clip_id,)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -134,6 +150,20 @@ def scale_at(ring: ScaleRing, index: int) -> Scale:
|
||||
return ring.scales[index % len(ring)]
|
||||
|
||||
|
||||
def pick_clip_id(scale: Scale, r: float) -> str:
|
||||
"""Pick one clip id from a scale's rotating pool, given an injected uniform
|
||||
`r` in [0, 1) (content-pipeline design §11.1).
|
||||
|
||||
Randomness is INJECTED so this stays a pure function — testable and shared
|
||||
with the Pi player; the impure draw (`random.random()`) happens once at the
|
||||
API/runtime boundary. `r` is clamped into [0, 1) so an off-by-epsilon caller
|
||||
never indexes out of range; a pool of one always returns its single member.
|
||||
"""
|
||||
members = scale.members
|
||||
r = min(max(r, 0.0), 0.999999)
|
||||
return members[int(r * len(members))]
|
||||
|
||||
|
||||
def advance_ring(
|
||||
ring: ScaleRing,
|
||||
from_index: int,
|
||||
|
||||
+14
-8
@@ -20,7 +20,7 @@ from typing import Callable, Optional
|
||||
|
||||
from hef.selection import Coordinate
|
||||
from player.alteration import RenderPlan, plan_alteration
|
||||
from player.content import ContentResolution, resolve_content
|
||||
from player.audio import resolve_visual
|
||||
from player.controls import Controls
|
||||
|
||||
|
||||
@@ -34,11 +34,14 @@ class TransitionKind:
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Playback:
|
||||
"""What should currently be playing."""
|
||||
"""What should currently be playing. Visual and audio are orthogonal (audio
|
||||
spec §2): `video` is whether the projector shows footage, `audio_source` is
|
||||
the independent audio dial (off / soundtrack)."""
|
||||
|
||||
clip_id: Optional[str] # None = black walls
|
||||
plan: Optional[RenderPlan] # None when black
|
||||
content: ContentResolution
|
||||
video: bool # whether footage is shown (Visual on/off)
|
||||
audio_source: str # the Audio dial source (off/soundtrack)
|
||||
volume: int
|
||||
brightness: int
|
||||
|
||||
@@ -58,7 +61,8 @@ def _first(library):
|
||||
_BLACK = Playback(
|
||||
clip_id=None,
|
||||
plan=None,
|
||||
content=ContentResolution("none", False),
|
||||
video=False,
|
||||
audio_source="off",
|
||||
volume=0,
|
||||
brightness=0,
|
||||
)
|
||||
@@ -78,12 +82,13 @@ class Player:
|
||||
self._current = _BLACK
|
||||
|
||||
def _resolve(self, controls: Controls) -> Playback:
|
||||
content = resolve_content(controls.content)
|
||||
if not content.video:
|
||||
video = resolve_visual(controls.visual)
|
||||
if not video:
|
||||
return Playback(
|
||||
clip_id=None,
|
||||
plan=None,
|
||||
content=content,
|
||||
video=False,
|
||||
audio_source=controls.audio,
|
||||
volume=controls.volume,
|
||||
brightness=controls.brightness,
|
||||
)
|
||||
@@ -92,7 +97,8 @@ class Player:
|
||||
return Playback(
|
||||
clip_id=clip.id,
|
||||
plan=plan_alteration(coord),
|
||||
content=content,
|
||||
video=True,
|
||||
audio_source=controls.audio,
|
||||
volume=controls.volume,
|
||||
brightness=controls.brightness,
|
||||
)
|
||||
|
||||
@@ -24,3 +24,9 @@ sim = [
|
||||
"uvicorn[standard]>=0.29",
|
||||
"httpx>=0.27",
|
||||
]
|
||||
# Playwright E2E browser tests (audio spec §9). Install with the browser binary:
|
||||
# pip install -e '.[e2e]' && python -m playwright install chromium
|
||||
# The E2E suite skips cleanly when Playwright/Chromium is absent (headless box).
|
||||
e2e = [
|
||||
"pytest-playwright>=0.4",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
# Session 0016.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-24T17-48 (PST)
|
||||
> Type: planning-and-executing
|
||||
> Posture: yolo
|
||||
> Claude-Session: 0fde1d36-7c64-414f-8f49-8669b5dd2263
|
||||
> Status: **FINALIZED**
|
||||
|
||||
## Launch prompt
|
||||
|
||||
`/goal` Content pipeline Increment 2 (expanded) — rotating clip pools + time-windowed
|
||||
TRACKED labels + progressive left-brain detail + progressive right-brain emotions.
|
||||
Short design pass first (schema changes), then build.
|
||||
|
||||
> Anchor: design `docs/superpowers/specs/2026-06-24-content-pipeline-design.md`
|
||||
> (graduated, R2a-eligible). Repo is roadmap-anchored, no Gitea issue tracker (R1
|
||||
> deviation, as all prior sessions). Increment 2 of that design.
|
||||
|
||||
## Pre-state
|
||||
|
||||
`main` at `b28f930` (PR #15 — operator-curated rotating-pool doc merged). All 19
|
||||
vetted pool clips on disk (gitignored) plus the leftovers to drop. The ring was the
|
||||
Increment-1 one-base-per-scale model (cosmos/orbit/forest/reef/abyss), flat
|
||||
`min_level` label gating, static or simple-track boxes, scene-level affect gated by
|
||||
`min(left,right)`. `tools/pipeline/` had stages 1–4 + 6 (no `track.py`). 246 tests.
|
||||
|
||||
## Arc
|
||||
|
||||
1. **Session gate + init.** Classified the `/goal` as planning-and-executing,
|
||||
claimed session **0016** (peeked an ENDED-UNFINALIZED 0015 — its PR #15 work was
|
||||
already merged, not live — and proceeded; no worktree needed, solo session).
|
||||
Verified clean `main`, read the pool doc + spec + sub-project-3 memory + the live
|
||||
code (ring, clips, app, app.js, alteration, pipeline).
|
||||
2. **Design pass (operator asked for it — schema changes).** Appended **§11** to the
|
||||
content-pipeline spec pinning the concrete shapes: rotating pool, time-windowed
|
||||
tracks, progressive Left detail tiers (salience + tiered strings), progressive
|
||||
Right emotion tiers, and the `track.py` + author-mode tooling. Key call: tiers
|
||||
need **no engine change** — the client reads `overlay.level` (Left) and
|
||||
`dream.strength` (Right) already in the `RenderPlan`.
|
||||
3. **PR #16 — the experience.** `Scale.pool` + pure `pick_clip_id` (injected
|
||||
randomness); server-side random pick at the API boundary (delta=0 = initial pick).
|
||||
`forest`→`coast` rename + ring reorder + orphan-media cleanup. Time-windowed
|
||||
tracked labels (`appear`/`disappear`, seam-wrapping). Salience-gated +
|
||||
tier-escalating Left labels; tier-escalating Right emotions; tiered-or-static
|
||||
strings (back-compat). New `build_pool_manifest.py` emits the 19-clip pool
|
||||
manifest (abyss+reef authored as tracked/windowed showcase). Merged via Gitea API.
|
||||
4. **PR #17 — the tooling.** `tools/pipeline/track.py`: classical LK optical-flow box
|
||||
tracker (this opencv build has no CSRT) → sparse keyframed track + window; optional
|
||||
lazy ML path. Simulator AUTHOR MODE (`/author.html` + `/api/author/{track,clip}`):
|
||||
scrub, drag a seed box, run tracker, author tiers/affect, save to manifest
|
||||
(idempotent upsert, keeps provenance). Repointed the pipeline integration test off
|
||||
the retired forest base; USER_GUIDE sim section brought current. Merged.
|
||||
|
||||
## Cut state
|
||||
|
||||
`main` at `c445239` (PR #17 merge). **267 passed / 2 skipped.** Clean tree, fully
|
||||
pushed, no open PRs. Both feature branches deleted. All 5 goal items delivered.
|
||||
|
||||
Verification was API/logic-level + JS syntax + live-server probes (5 scales with
|
||||
pools, rotating picks, new edges, 19 clips served, tracker returns keyframes on real
|
||||
abyss_wow footage, committed manifest untouched by the read-only probe). **The
|
||||
VISUAL look was not screenshot-verified — no Chrome on this box** — so operator
|
||||
by-eye review is the next step (matches the established deferred-review pattern).
|
||||
|
||||
## Follow-up (post-finalize, same conversation) — PR #18
|
||||
|
||||
Operator asked to replace the scale-ring `⊖ out / in ⊕` buttons with a **circular
|
||||
"Altitude" knob** that turns all the way around (including abyss→cosmos). Built a
|
||||
turnable SVG dial in `simulator/static/app.js` (`buildDial` / `renderDial` / drag +
|
||||
wheel handlers) + `style.css`: cosmos (highest) at top, clockwise descends
|
||||
cosmos→orbit→coast→reef→abyss and **wraps** back up. Dragging accumulates rotation
|
||||
and commits whole detents on release (a big spin → one fast blended pass, reusing
|
||||
the proven wheel/detent path); scroll the knob to step; tap a label to jump the
|
||||
shortest way around. **Pure frontend** — the endless ring + wrap already existed
|
||||
server-side, so no engine/API change. JS syntax-checked; 23 sim-API tests green;
|
||||
live probe confirmed the dial replaced the buttons and the abyss→cosmos wrap is
|
||||
intact. Merged as **PR #18** (`main` at `7a32ff1`). Visual look still operator-by-eye
|
||||
(no Chrome on this box).
|
||||
|
||||
## Deployment pipeline (§9)
|
||||
|
||||
No stage run: the HEF simulator is a **local dev tool** (simulator-first directive),
|
||||
not a deployed app — it has no PPE/prod infra. Nothing shippable through the §9
|
||||
pipeline this session; the session output is the merged code on `main`.
|
||||
|
||||
## Plan archival
|
||||
|
||||
No separate `superpowers:writing-plans` artifact was produced — the design pass was a
|
||||
spec §11 edit (merged in PR #16) and execution was direct TDD tracked by the
|
||||
transcript `## Plan` block + the task list. Nothing to `submit-plan.sh`; this app has
|
||||
no content repo anyway (roadmap-anchored).
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
- **Adopting/finalizing the ENDED-UNFINALIZED session 0015 placeholder** was skipped
|
||||
— its PR #15 work is already on `main` and its outcomes are fully recorded in
|
||||
memory; finalizing a prior dead session was judged an unnecessary detour. Low
|
||||
confidence; flagging for the operator.
|
||||
- **By-eye visual review deferred** to the operator (no Chrome here) — consistent with
|
||||
the prior-session pattern, but the look of the tiers/windows/pool rotation AND the
|
||||
new Altitude knob is unconfirmed visually.
|
||||
- **Light-vs-rich label authoring:** only abyss + reef + the scale primaries are
|
||||
richly authored; the other pool members carry lighter labels. Finishing them is the
|
||||
author-mode follow-up (next goal).
|
||||
|
||||
## Next-session prompt
|
||||
|
||||
`/goal` Operator by-eye review of Increment 2 + the Altitude knob in the sim
|
||||
(`make sim-local`, :8000): turn the Altitude knob (drag/scroll, wraps abyss→cosmos;
|
||||
pools rotate per landing), sweep Left (progressive label tiers
|
||||
general→scientific+fact), sweep Right with Left up (progressive emotions
|
||||
basic→compound), watch abyss/reef time-windowed tracked labels enter/leave + track,
|
||||
and exercise `/author.html`. Then author real labels for the remaining pool members
|
||||
via author mode. After the content is liked, the next build frontier is the deferred
|
||||
i2v ring transitions (generative-video model + adjacent real bases) and/or the Pi
|
||||
renderer + serial/firmware (per [[simulator-first-before-hardware]]), per ROADMAP.md.
|
||||
+5
-5
@@ -1,15 +1,15 @@
|
||||
# Session 0016.0 — Transcript
|
||||
# Session 0017.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-24T17-48 (PST)
|
||||
> Start: 2026-06-25T07-41 (PST)
|
||||
> Type: planning-and-executing
|
||||
> Posture: yolo
|
||||
> Claude-Session: 0fde1d36-7c64-414f-8f49-8669b5dd2263
|
||||
> Claude-Session: bcb069f2-0fba-4a66-ae8e-f40b7917a48c
|
||||
> Status: **PLACEHOLDER — claimed at session start; finalized at session end.**
|
||||
>
|
||||
> This file reserves session ID 0016 for human-experience-filter-art. The driver replaces this
|
||||
> This file reserves session ID 0017 for human-experience-filter-art. The driver replaces this
|
||||
> body with the full transcript and renames the file to its final
|
||||
> SESSION-0016.0-TRANSCRIPT-2026-06-24T17-48--<end>.md form at session end.
|
||||
> SESSION-0017.0-TRANSCRIPT-2026-06-25T07-41--<end>.md form at session end.
|
||||
|
||||
## Launch prompt
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
# Session 0018.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-25T09-42 (PST)
|
||||
> End: 2026-06-26T05-56 (PST)
|
||||
> Type: planning-and-executing
|
||||
> Posture: yolo
|
||||
> Claude-Session: b96041e5-c682-49f5-88ee-380ea7abb6af
|
||||
> Status: **FINALIZED**
|
||||
|
||||
## Launch prompt
|
||||
|
||||
```
|
||||
Find a better public-domain nebula flythrough video for the cosmos scale (the current text-free SVS Orion clip lost quality when cropped to remove its text overlay), or upscale the existing one if no better source exists.
|
||||
```
|
||||
|
||||
A content-and-polish session on the simulator's cosmos scale that grew, by
|
||||
operator follow-ups, into four merged PRs: the nebula swap, two cache fixes, a
|
||||
ring-landing artifact fix, and a Right-brain affect rework.
|
||||
|
||||
## Pre-state
|
||||
|
||||
- Branch `feature/ring-transitions-real` (later merged): HEAD `129bb23` had just
|
||||
swapped the cosmos primary off the captioned/cropped JPL "Orion: Dust and Death"
|
||||
clip onto the text-free SVS 30957 Orion flythrough.
|
||||
- Sim served by a manually-launched `uvicorn ... --port 8000` (no `--reload`),
|
||||
booted before the session's media changes.
|
||||
- Affect channel gated on `min(left, right)` (session 0013 rule).
|
||||
- Concurrent unfinalized sessions present (0015, 0017).
|
||||
|
||||
## Arc (turn by turn)
|
||||
|
||||
1. **Anchor + research.** Classified as planning-and-executing, claimed session
|
||||
0018. Found the current cosmos clip was already the uncropped text-free SVS
|
||||
30957 Orion (the crop damage the prompt described was already fixed at HEAD).
|
||||
Researched NASA SVS: (a) the same Orion clip exists in 4K; (b) the Webb
|
||||
**"Cosmic Cliffs"** Carina flythrough (SVS 31348) is a 4K PD alternative.
|
||||
Offered three options; **operator chose #2 — switch to Cosmic Cliffs.**
|
||||
|
||||
2. **Cosmos swap (PR #21).** Downloaded the Carina 4K master
|
||||
(`Clifs-3d-STScI.mp4`, 3840×2160, 60 fps, 103 s), sampled frames to find a
|
||||
clean window, trimmed **10–34 s** (clears the title card ~8 s and the
|
||||
end-credit card ~102 s), ran `tools.pipeline.run.process_clip` → 4K master +
|
||||
supersampled 1080p base (5.5 Mbps, crisper than the old 2.9 Mbps Orion encode).
|
||||
Updated `build_pool_manifest.py` META (new `CCBY_WEBB` credit, Carina labels:
|
||||
emission nebula / protostar, redshift→distance ≈7,500 ly), regenerated
|
||||
`manifest.json` (also corrected stale JPL-Orion provenance) + all ring-edge
|
||||
transitions, and `docs/content-candidate-pool.md`. Operator authorized the
|
||||
merge → opened + merged **PR #21** (the whole `feature/ring-transitions-real`
|
||||
line), synced main, deleted the local branch. Remote branch delete returned
|
||||
HTTP 405 (known Gitea perm limit).
|
||||
|
||||
3. **"Still seeing Orion" — caching (PRs #22, #23).** Diagnosed: the sim server
|
||||
(PID 33672) had been running since before the media swap and was launched
|
||||
**without `--reload`**, holding the pre-swap manifest in memory; the browser
|
||||
also held the old clip via the in-memory blob preload and a possible pre-
|
||||
`no-cache` immutable HTTP entry. Restarted the server with `make sim-local`
|
||||
(`--reload`). Shipped **PR #22** (`cache: "reload"` on the preload fetch), then
|
||||
— at operator request for a permanent fix — **PR #23**: a content-hash
|
||||
`GET /api/media-versions` endpoint (sha1[:12] per served file, cached by
|
||||
mtime/size) and client `?v=<hash>` on every /media URL, so a re-baked clip's
|
||||
URL changes with its bytes.
|
||||
|
||||
4. **Coast landing artifact (in PR #23).** Operator noticed zooming into coast
|
||||
showed birdrock then switched to another clip. Root cause: the baked ring
|
||||
transition lands on the scale **primary** (birdrock), but the rotating pool
|
||||
picks a **random member** on landing — a non-primary pick hard-cut from primary
|
||||
to chosen. Fix: `advance()` masks the swap behind a short fade through the
|
||||
existing `#black` overlay when `target_clip_id !== ring.scales[to_index].clip_id`;
|
||||
general across all pooled scales.
|
||||
|
||||
5. **Right-brain affect (PR #24).** Operator: emotions should be controlled by the
|
||||
right brain, not the left. Reversed `affect.strength = min(left, right)` →
|
||||
`strength = right`; intensity keyed off Right; Left no longer gates emotions.
|
||||
Updated engine + client comments + the affect design spec (revision note) +
|
||||
tests. Live-verified: left=0/right=4 → strength 4; left=4/right=0 → 0.
|
||||
|
||||
## Cut state
|
||||
|
||||
- **`main` @ (session-0018 merges all landed):** PRs **#21** (`7da7446`), **#22**
|
||||
(`aa56dfe`), **#23** (`5bb762a`), **#24** (`d0ad9a9`) — all confirmed in main's
|
||||
ancestry. Session 0019 has since advanced main further (cosmos_miri pool member,
|
||||
networked-control-surface spec, sim-local venv fix).
|
||||
- **Tests:** 271 passed, 2 skipped (+ media-version tests + rewritten affect
|
||||
tests).
|
||||
- **Sim:** running on :8000 as `uvicorn --reload` (PID 50199, restarted this
|
||||
session). The cosmos Cosmic Cliffs clip, content-hash cache-bust, coast fade,
|
||||
and Right-only affect are all live and API/logic-verified.
|
||||
- **No plan artifact** archived — leaf content/polish task executed directly
|
||||
(no `superpowers:writing-plans` doc).
|
||||
|
||||
## Operator plate
|
||||
|
||||
- **Cosmic Cliffs looks "incredible"** (operator) — the cosmos primary is now the
|
||||
Webb Carina flythrough.
|
||||
- **Unverified by eye** (no Chrome on the box): the coast landing fade timing, and
|
||||
the Right-only affect look (Left=0 + Right up should show emotion words over an
|
||||
un-annotated scene). Worth a by-eye pass in `make sim-local`.
|
||||
- **Pipeline §9:** changes merged to `main` (= done); no PPE/prod run — the E2E +
|
||||
PPE machinery is still being built (§10.6), and the simulator is a local dev
|
||||
surface. Not shipped to prod.
|
||||
- **Left untouched (not this session's work):** session 0019's in-flight audio
|
||||
work is dirty in the tree (`docs/audio-candidate-pool.md`, `.gitignore` audio
|
||||
entries) — left for 0019 to finalize. Pre-existing unfinalized sessions 0015,
|
||||
0017, 0019 remain `--INPROGRESS`.
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
None logged — every call this session was operator-directed (the option-2 clip
|
||||
choice, the permanent cache-bust request, the Right-brain affect direction).
|
||||
|
||||
## Next-session prompt
|
||||
|
||||
```
|
||||
/goal By-eye review of session 0018's sim changes in `make sim-local` (:8000): the
|
||||
Webb Cosmic Cliffs cosmos clip, the coast/pool landing fade (no hard cut to the
|
||||
random member), and Right-only emotions (Left=0 + Right up shows feeling words over
|
||||
an un-annotated scene). Tune the ~200ms fade timing if it feels off. Then continue
|
||||
the experience/content line — deferred i2v ring transitions and/or the Pi renderer
|
||||
(per [[simulator-first-before-hardware]]). Note: session 0019 separately owns the
|
||||
audio pool + networked-control-surface work.
|
||||
```
|
||||
@@ -0,0 +1,95 @@
|
||||
# Session 0019.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-26T04-47 (PST)
|
||||
> End: 2026-06-26T05-47 (PST)
|
||||
> Type: brainstorming
|
||||
> Posture: careful
|
||||
> Claude-Session: d6ed1203-92b6-471f-80a4-4c773b645ca1
|
||||
> Anchor: ROADMAP sub-project 4 (Arduino Firmware / control panel) — reframed
|
||||
> Status: **FINALIZED**
|
||||
|
||||
## Launch prompt
|
||||
|
||||
```
|
||||
Let's start talking about hardware. Let's assume for the first version I'm ok having my computer or laptop do the rendering. We'd still want a controller but it could be connected to the computer via wifi, USB, or bluetooth and would send commands to the software we've already written that would adjust left brain, right brain, etc. We'd prototype this by having a controller simulator run on another device, such as an ipad (just riffing here) but eventually move to a physical remote running on an Arduino or similar
|
||||
```
|
||||
|
||||
## Pre-state
|
||||
|
||||
- Simulator is a **FastAPI** app, **stateless**: the single browser holds knob
|
||||
state in JS and POSTs to `/api/alteration` for a `RenderPlan`. Renderer and
|
||||
controller are the same browser tab.
|
||||
- Operator-facing control surface (sim today): Content (7-way `<select>`),
|
||||
Altitude (endless circular dial across 5 scales), Left/Right (sliders 0–4),
|
||||
Mood (slider −4…+4). (Dark+Light → Mood in s0013; Altitude dial in s0018.)
|
||||
- ROADMAP sub-project 4 = "Arduino Firmware (control panel)" assuming a **Pi**
|
||||
runs the room and the panel talks **USB serial** to it.
|
||||
- Standing directive: simulator-first before hardware.
|
||||
- Checkout shared with a concurrent audio-sourcing session (0017/0018), with
|
||||
uncommitted `.gitignore` + untracked files on `main`.
|
||||
|
||||
## Arc
|
||||
|
||||
1. **Session gate / routing.** Classified the opening ("let's talk about
|
||||
hardware… just riffing") as **brainstorming**; invoked `wgl-brainstorming`,
|
||||
claimed session **0019** (`--type brainstorming`, careful posture). Three stale
|
||||
`--INPROGRESS` sessions (0015/0017/0018) noted as leftovers, not live overlaps;
|
||||
no worktree (the audio session's edits are unrelated/uncommitted).
|
||||
|
||||
2. **Orientation.** Read `docs/ROADMAP.md` + the simulator (`app.py`, `index.html`)
|
||||
to ground the discussion; confirmed the real control surface and the stateless
|
||||
architecture. Recognized the request as a **reframe** of sub-project 4.
|
||||
|
||||
3. **Brainstorming dialogue** (`superpowers:brainstorming`), decisions reached:
|
||||
- **Intent:** both — decouple control/display cleanly AND want the handheld feel.
|
||||
- **Transport (v1):** wifi/web (iPad opens a controller page off the FastAPI
|
||||
server; least friction; Arduino transport left open).
|
||||
- **Remote fidelity:** faithful twin — decide the physical panel inventory now,
|
||||
iPad mirrors it.
|
||||
- Operator refinement: **per-knob transition state** — live controls
|
||||
(Left/Right/Mood) are instant; transitional ones (Altitude, Content) carry a
|
||||
lifecycle so the remote's feedback is honest.
|
||||
- **Architecture:** approach A — server-authoritative `SessionState` + **SSE**
|
||||
push (chosen over polling / WebRTC).
|
||||
|
||||
4. **Design presented section by section** (roles → transition model → server
|
||||
contract → two pages + Arduino seam → error/test/scope), each approved.
|
||||
|
||||
5. **Spec written** to
|
||||
`docs/superpowers/specs/2026-06-26-networked-control-surface-design.md`,
|
||||
committed on `feat/networked-control-surface`, pushed; operator approved going
|
||||
straight to finalize.
|
||||
|
||||
6. **Finalize.** Single-repo app (CONTENT_REMOTE empty) → merging the branch IS
|
||||
the submission. Discovered the shared checkout had switched to `main` (the
|
||||
audio session) and `main` had advanced to PR #25; landed the spec via a safe
|
||||
local `--no-ff` merge that added only the new spec file (their uncommitted work
|
||||
untouched), pushed `main` (`229af60..f28a06c`), deleted the feature branch.
|
||||
|
||||
## Cut state
|
||||
|
||||
- `main` @ `f28a06c` — spec merged. Feature branch deleted.
|
||||
- Spec `status: proposed` (careful posture); operator reviewed it live, approving
|
||||
every section before write.
|
||||
- No code written (brainstorming session). 0 new tests (none applicable).
|
||||
- Concurrent audio session's uncommitted `.gitignore`/untracked files left intact.
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
_None — no autonomous low-confidence calls. The two flagged design choices
|
||||
(last-write-wins concurrency; no auth in v1) were presented to the operator and
|
||||
approved in-session, not deferred._
|
||||
|
||||
## Operator plate
|
||||
|
||||
- Review the merged spec if desired:
|
||||
`docs/superpowers/specs/2026-06-26-networked-control-surface-design.md` on `main`.
|
||||
- Next: open a **writing-plans** session to turn the spec into an implementation
|
||||
plan (the `/goal` below).
|
||||
|
||||
## Next /goal
|
||||
|
||||
```
|
||||
/goal Write the implementation plan for the networked control surface, per docs/superpowers/specs/2026-06-26-networked-control-surface-design.md
|
||||
```
|
||||
@@ -0,0 +1,109 @@
|
||||
# Session 0020.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-26T06-47 (PST)
|
||||
> End: 2026-06-26T07-58 (PST)
|
||||
> Type: writing-plans (extended into execution — operator approved "then build it" at the plan gate)
|
||||
> Posture: careful (execution proceeded under explicit operator approval)
|
||||
> Claude-Session: e0cc053a-41fa-416d-a4f5-493f14984e75
|
||||
> Anchor: design docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md (R2a)
|
||||
> Status: FINALIZED
|
||||
|
||||
## Launch prompt
|
||||
|
||||
> Write the implementation plan for the "Audio + Video — separated Visual & Audio
|
||||
> controls" feature, then build it. (Spec:
|
||||
> docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md.)
|
||||
> Split the bundled 7-way Content selector into orthogonal Visual (on/off) ×
|
||||
> Audio (off / soundtrack / white noise). Audio follows the single Altitude dial;
|
||||
> white noise synthesized (ffmpeg anoisesrc). Reframe the (unbuilt) networked-
|
||||
> control-surface spec §3/§5 as a clean pre-implementation edit. Honor the testing
|
||||
> tiers; resolve the three §12 flagged decisions.
|
||||
|
||||
## Pre-state
|
||||
|
||||
- `main` clean at 99a751c. Two stale `--INPROGRESS` placeholders (0015, 0017) from
|
||||
2026-06-24/25 — prior dead sessions, not live; left untouched.
|
||||
- The networked-control-surface architecture (SSE / `/remote` / renderer split) was
|
||||
**unbuilt** — the sim is a single fused page (`simulator/static/index.html`) talking
|
||||
to FastAPI via `POST /api/alteration` (knobs) + `POST /api/ring/advance` (altitude).
|
||||
- Audio assets (5 sourced ambiences) sat gitignored under `simulator/sample_media/audio/<scale>/`,
|
||||
unreferenced by the manifest, unserved; no `<audio>` element anywhere; video muted.
|
||||
|
||||
## Session arc (turn by turn)
|
||||
|
||||
1. **Session gate → writing-plans.** Classified the two-part "write the plan, then
|
||||
build it" prompt as the build track's entry; claimed session **0020** (`--type
|
||||
writing-plans`, careful posture). Baseline clean; created branch `session-0020`.
|
||||
2. **Scoping.** Read both specs + the audio candidate pool; dispatched an Explore
|
||||
agent to map the current simulator. Key finding: the control-surface architecture
|
||||
is unbuilt, so the audio spec's §6 server contract had no host. **Resolved by the
|
||||
standing simulator-first directive:** build the audio feature into the current
|
||||
single-page sim (ships sound now); reframe the control-surface spec on paper only;
|
||||
do NOT build the SSE/`/remote`/renderer split (it's "v1 hardware," deferred). This
|
||||
build is the "Content step" the audio spec §11 anticipates.
|
||||
3. **Plan.** Wrote `docs/superpowers/plans/2026-06-26-audio-video-separated-controls.md`
|
||||
(10 tasks, all testing tiers). Committed + pushed; presented at the careful-posture
|
||||
plan gate. Operator answered **"Approve — build it."** §12 decisions resolved:
|
||||
audio=live (~0.6 s crossfade), noise=pink, autoplay=tap-to-start overlay; asset
|
||||
params −18 LUFS / mp3 / loop mirrors the proven video crossfade recipe.
|
||||
4. **Build (TDD, commit per task):**
|
||||
- T1–2: pure ffmpeg audio builders (`tools/pipeline/audio_ops.py`) + runner
|
||||
(`audio_run.py`) + `simulator/build_audio_media.py`; produced the 5 normalized
|
||||
loops + pink-noise bed (gitignored).
|
||||
- T3: `Scale.audio` manifest field + read/emit/serialize + `SCALE_AUDIO`; manifest
|
||||
regenerated (clean diff — only 5 audio lines).
|
||||
- T4–6: pure `player/audio.py` resolver; retired `player/content.py` (+ test);
|
||||
reframed `player/controls.py` + `player/state.py`; `/api/alteration` validates
|
||||
visual+audio and returns `render.video` + server-resolved `render.audio.url` from
|
||||
`altitude_index`. Full Python suite green.
|
||||
- T7–8: client — Content `<select>` → Visual toggle + Audio selector; A/B `<audio>`
|
||||
gain-crossfade layer; tap-to-start gesture. Smoke-tested page + media (200s).
|
||||
- T9: reframed the networked-control-surface spec §3/§4/§5/§6; recorded §12
|
||||
resolutions; ROADMAP note.
|
||||
- T10: Playwright E2E (`[e2e]` extra). Installed Chromium and ran it for real —
|
||||
surfaced + fixed three client issues (see Cut state). **3 E2E pass**, full suite
|
||||
**288 passed, 2 skipped**.
|
||||
5. **Verify + ship.** Screenshotted the running app (cosmos plays, soundtrack audibly
|
||||
playing `cosmos/pillars.loop.mp3`, split controls visible, zero JS errors). Merged
|
||||
`session-0020` → `main` (`--no-ff`) and pushed. (`gh` PR create denied — Gitea host;
|
||||
merged locally with branch→merge discipline.)
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
_No low-confidence autonomous calls. The scope decision (current-sim vs full control
|
||||
surface) was approved at the plan gate; the three §12 resolutions were operator-
|
||||
delegated by the launch prompt ("resolve the three flagged decisions"); asset params
|
||||
(−18 LUFS, mp3, pink) follow the spec's stated direction._
|
||||
|
||||
## Cut state
|
||||
|
||||
- `main` @ `aa76cd1` (merge of session-0020), clean, synced with origin.
|
||||
- **288 tests + 3 Playwright E2E pass; 2 skipped** (opt-in media integration).
|
||||
- Three client robustness fixes the E2E drove out (worth remembering):
|
||||
- `fadeVolume` uses `setInterval`, not `requestAnimationFrame` (rAF stalls when the
|
||||
tab is backgrounded → the gain ramp hung in headless).
|
||||
- The start gesture drops per-element "priming": a single page click grants Chrome
|
||||
autoplay document-wide; priming srcless `<audio>` only hung the handler / polluted
|
||||
the paused-state.
|
||||
- `applyAudio` does not `await` `play()` (awaiting a srcless/buffering play can hang).
|
||||
- Plan archival: this app has **no content repo** (`CONTENT_REMOTE` empty), so the plan
|
||||
was not archived to a `plans/` collection — it lives in-repo at
|
||||
`docs/superpowers/plans/` and is already on `main`.
|
||||
- §9 pipeline: localhost + E2E stage is **green**; there is no PPE/prod infra for this
|
||||
simulator-first art app (deferred with the Pi renderer), so no further ship stage ran.
|
||||
|
||||
## Operator plate (what's true now / next)
|
||||
|
||||
- The simulator has **sound**: per-altitude soundtracks that follow the Altitude dial +
|
||||
a pink white-noise bed, behind orthogonal **Visual** × **Audio** controls. The unheard
|
||||
audio wants a by-ear review/tune.
|
||||
- Deferred (already roadmap-tracked, not re-parked in memory): the **music** layer
|
||||
(reserved Audio position, no assets), the networked-control-surface SSE/`/remote`
|
||||
build, i2v ring transitions, the Pi renderer.
|
||||
|
||||
## Next /goal
|
||||
|
||||
```
|
||||
/goal review/tune the new audio by ear in the sim (per-altitude soundtracks + pink white-noise + the ~0.6s A/B crossfade; tap-to-start), then source/compose the deferred per-altitude music layer (the reserved Audio position), per docs/audio-candidate-pool.md
|
||||
```
|
||||
@@ -0,0 +1,88 @@
|
||||
# Session 0021.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-26T08-12 (PST)
|
||||
> End: 2026-06-26T08-20 (PST)
|
||||
> Type: executing-plans (operator-driven hotfix/revision of session 0020's audio feature)
|
||||
> Posture: yolo
|
||||
> Claude-Session: e0cc053a-41fa-416d-a4f5-493f14984e75
|
||||
> Status: FINALIZED
|
||||
|
||||
## Launch prompt
|
||||
|
||||
> Show video unchecked doesn't turn off the video. Make "show video" just "Video"
|
||||
> and have a toggle like "Dev Mode" has. I can't hear any audio when "soundtrack
|
||||
> (follows altitude)" is selected. Let's just have Audio as off/on like Video for
|
||||
> now and get rid of white noise.
|
||||
|
||||
## Pre-state
|
||||
|
||||
- Session 0020 had just merged the audio feature to `main` (Visual×Audio split,
|
||||
per-altitude soundtracks, pink white-noise, A/B crossfade). The operator tested it
|
||||
live and reported three issues.
|
||||
|
||||
## Session arc
|
||||
|
||||
1. **Diagnosis (the key work).** Both reported bugs were reproduced *only* in the
|
||||
operator's real browser — neither appeared in headless Chromium **or** WebKit
|
||||
(installed `playwright install webkit` to test Safari's engine). Root causes:
|
||||
- **Video-off doesn't blank:** `#vid` (`<video>`) and `#paint` (WebGL `<canvas>`)
|
||||
are GPU-composited layers; the `#black` cover had *auto* z-index, so on a real
|
||||
GPU (Safari/Chrome) the video layer can composite *above* it despite DOM order.
|
||||
Headless software-raster hides this.
|
||||
- **No soundtrack audio:** real Safari/iOS unlocks an `<audio>` element only when
|
||||
`play()` is called *synchronously inside* the user gesture; the 0020 gesture
|
||||
played in an async continuation (after `await fetch`), which Chrome allows but
|
||||
Safari blocks. Headless WebKit relaxes autoplay, so it passed.
|
||||
This is the session's main lesson: **Playwright headless relaxes both GPU
|
||||
compositing and autoplay**, so the 0020 E2E couldn't have caught either — real
|
||||
device / by-eye review is required.
|
||||
2. **Fixes (branch `session-0021`):**
|
||||
- **Video** + **Audio** are now Dev-Mode-style on/off toggles (`.dev-switch`),
|
||||
wired on `change`. Audio on → soundtrack; **white-noise deferred** (removed from
|
||||
the live control; `AUDIO_SOURCES={off,soundtrack}`; the pure pink-noise
|
||||
machinery + tests stay for later).
|
||||
- Video-off: `.black` gets `z-index:50` + `translateZ(0)` (own layer) **and** the
|
||||
video layers are set `opacity:0` on video-off (belt-and-suspenders blanking).
|
||||
- Safari audio: the tap-to-start gesture now primes both A/B `<audio>` elements
|
||||
**synchronously in-gesture** (set src, vol 0, `play().then(pause)`), unlocking
|
||||
the later async crossfades.
|
||||
- Server: `player/audio.py` resolver + `controls.py`/`state.py` + `/api/alteration`
|
||||
drop white-noise; `build_audio_media.py` stops emitting the noise bed.
|
||||
3. **Verify + ship.** Updated unit/contract/E2E tests; rewrote the 3 E2E for the new
|
||||
toggles (click the visible `.dev-switch-track`; wait for the opacity transition).
|
||||
**288 passed, 2 skipped.** Re-verified in **both Chromium and WebKit**: Audio toggle
|
||||
plays `cosmos/pillars.loop.mp3`, video-off sets vid+paint opacity 0 with `#black`
|
||||
shown, zero JS errors; screenshotted the new toggle UI. Merged `session-0021` →
|
||||
`main` and pushed. (Also untracked a stray `docs/.threads/*.json` editor artifact
|
||||
swept in by `git add -A`, and gitignored `.threads/`.)
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
_No low-confidence calls. All three changes were explicit operator requests; the
|
||||
root-cause fixes (GPU z-index/layer-hide; Safari in-gesture unlock) are the standard
|
||||
remedies for these well-known browser behaviors._
|
||||
|
||||
## Cut state
|
||||
|
||||
- `main` @ `fddb6d6` (merge of session-0021), clean, pushed.
|
||||
- **288 tests + 3 Playwright E2E pass** (2 skipped: opt-in media integration).
|
||||
- White-noise is deferred, not deleted: `tools/pipeline/audio_ops.white_noise_args` +
|
||||
`audio_run.generate_white_noise` + their tests remain; only the live control + the
|
||||
build-script emission were removed.
|
||||
- **Unvalidated by this session:** real-device audio (Safari autoplay) and real-GPU
|
||||
video-off — headless can't exercise either; needs the operator's by-ear/by-eye pass.
|
||||
|
||||
## Operator plate
|
||||
|
||||
- Video + Audio are simple on/off toggles (Dev-Mode style). Audio on = the current
|
||||
altitude's soundtrack. The two real-browser bugs should now be fixed — but please
|
||||
confirm on a real device (especially iPad/Safari).
|
||||
- Deferred: white-noise (easy to re-add as a 3rd Audio position), the per-altitude
|
||||
music layer, the networked-control-surface build, Pi renderer.
|
||||
|
||||
## Next /goal
|
||||
|
||||
```
|
||||
/goal by-ear/by-eye review the audio + Video/Audio toggles on a REAL device (esp. iPad/Safari — headless can't validate Safari autoplay or GPU compositing), then source/compose the deferred per-altitude music layer per docs/audio-candidate-pool.md (or re-enable white-noise as a 3rd Audio position if wanted)
|
||||
```
|
||||
@@ -46,5 +46,20 @@
|
||||
},
|
||||
"0016": {
|
||||
"title": ""
|
||||
},
|
||||
"0017": {
|
||||
"title": ""
|
||||
},
|
||||
"0018": {
|
||||
"title": ""
|
||||
},
|
||||
"0019": {
|
||||
"title": ""
|
||||
},
|
||||
"0020": {
|
||||
"title": ""
|
||||
},
|
||||
"0021": {
|
||||
"title": ""
|
||||
}
|
||||
}
|
||||
|
||||
+176
-14
@@ -8,7 +8,9 @@ endpoints (/api/select, /api/catalog/meta) and the X-ray are retired.
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
@@ -24,9 +26,13 @@ from player.alteration import (
|
||||
plan_alteration,
|
||||
render_plan_to_dict,
|
||||
)
|
||||
from player.content import resolve_content
|
||||
from player.controls import CONTENT_POSITIONS
|
||||
from player.ring import DEFAULT_FAST_SPIN_THRESHOLD, advance_ring
|
||||
from player.audio import AUDIO_SOURCES, VISUAL_POSITIONS, resolve_audio, resolve_visual
|
||||
from player.ring import (
|
||||
DEFAULT_FAST_SPIN_THRESHOLD,
|
||||
advance_ring,
|
||||
pick_clip_id,
|
||||
scale_at,
|
||||
)
|
||||
from simulator.clips import load_manifest, load_ring, ring_move_to_dict, ring_to_dict
|
||||
|
||||
STATIC_DIR = Path(__file__).parent / "static"
|
||||
@@ -52,8 +58,44 @@ def _asset_version() -> str:
|
||||
return hashlib.sha1("|".join(parts).encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
# Content-hash tokens for served media, so the client can append `?v=<hash>` to a
|
||||
# /media URL. A clip re-baked under the SAME path (e.g. a re-sourced cosmos base)
|
||||
# changes its hash → its URL → a fresh fetch, busting any cached prior bytes
|
||||
# permanently (even an immutable-pinned entry a plain reload can't revalidate).
|
||||
# Cached by (mtime_ns, size) so the full-file hash is recomputed only when the
|
||||
# file actually changes — a re-bake is picked up without a server restart.
|
||||
_media_hash_cache: dict[str, tuple[int, int, str]] = {}
|
||||
|
||||
|
||||
def _media_version(rel: str) -> Optional[str]:
|
||||
"""Short content hash of the media file at `rel` under MEDIA_DIR, or None if
|
||||
it's absent. Cheap on repeat calls: re-hashes only when (mtime, size) change."""
|
||||
path = MEDIA_DIR / rel
|
||||
try:
|
||||
st = path.stat()
|
||||
except OSError:
|
||||
return None
|
||||
cached = _media_hash_cache.get(rel)
|
||||
if cached and cached[0] == st.st_mtime_ns and cached[1] == st.st_size:
|
||||
return cached[2]
|
||||
h = hashlib.sha1()
|
||||
with path.open("rb") as f:
|
||||
for chunk in iter(lambda: f.read(1 << 20), b""):
|
||||
h.update(chunk)
|
||||
token = h.hexdigest()[:12]
|
||||
_media_hash_cache[rel] = (st.st_mtime_ns, st.st_size, token)
|
||||
return token
|
||||
|
||||
|
||||
def _rev_file(file: str) -> str:
|
||||
"""The baked zoom-out companion path for a transition file (mirrors the
|
||||
client's `reverseFile`): `<edge>.mp4` -> `<edge>.rev.mp4`."""
|
||||
return file[:-4] + ".rev.mp4" if file.endswith(".mp4") else file
|
||||
|
||||
|
||||
class ControlsModel(BaseModel):
|
||||
content: str
|
||||
visual: str
|
||||
audio: str
|
||||
left: int = Field(ge=0, le=4)
|
||||
right: int = Field(ge=0, le=4)
|
||||
dark: int = Field(ge=0, le=4)
|
||||
@@ -70,6 +112,7 @@ class CalibrationModel(BaseModel):
|
||||
|
||||
class AlterationRequest(BaseModel):
|
||||
controls: ControlsModel
|
||||
altitude_index: int = 0
|
||||
calibration: Optional[CalibrationModel] = None
|
||||
|
||||
|
||||
@@ -78,6 +121,29 @@ class RingAdvanceRequest(BaseModel):
|
||||
delta: int
|
||||
|
||||
|
||||
class AuthorTrackRequest(BaseModel):
|
||||
"""Author mode: propagate a hand-seeded box on a clip into a keyframed track
|
||||
(content-pipeline §11.5). `seed_box` + `seed_t` are normalized."""
|
||||
|
||||
clip_id: str
|
||||
key: str
|
||||
seed_box: list[float] = Field(min_length=4, max_length=4)
|
||||
seed_t: float = Field(default=0.0, ge=0.0, le=1.0)
|
||||
salience: int = Field(default=4, ge=1, le=4)
|
||||
n_keyframes: int = Field(default=5, ge=2, le=20)
|
||||
max_frames: Optional[int] = None
|
||||
|
||||
|
||||
class AuthorClipRequest(BaseModel):
|
||||
"""Author mode: persist the authored labels/affect/strings for an existing
|
||||
pool clip into the manifest (geometry from the tracker, semantics by hand)."""
|
||||
|
||||
clip_id: str
|
||||
annotations: list = []
|
||||
affect: list = []
|
||||
strings: dict = {}
|
||||
|
||||
|
||||
def _load_clips(manifest_path: Optional[Path]):
|
||||
path = Path(manifest_path) if manifest_path else DEFAULT_MANIFEST
|
||||
if path.exists():
|
||||
@@ -94,14 +160,17 @@ def _load_ring(manifest_path: Optional[Path]):
|
||||
|
||||
def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
|
||||
app = FastAPI(title="HEF Alteration Simulator")
|
||||
app.state.manifest_path = Path(manifest_path) if manifest_path else DEFAULT_MANIFEST
|
||||
app.state.clips = _load_clips(manifest_path)
|
||||
app.state.ring = _load_ring(manifest_path)
|
||||
|
||||
@app.post("/api/alteration")
|
||||
def api_alteration(req: AlterationRequest):
|
||||
c = req.controls
|
||||
if c.content not in CONTENT_POSITIONS:
|
||||
raise HTTPException(status_code=422, detail=f"invalid content {c.content!r}")
|
||||
if c.visual not in VISUAL_POSITIONS:
|
||||
raise HTTPException(status_code=422, detail=f"invalid visual {c.visual!r}")
|
||||
if c.audio not in AUDIO_SOURCES:
|
||||
raise HTTPException(status_code=422, detail=f"invalid audio {c.audio!r}")
|
||||
coord = Coordinate(c.left, c.right, c.dark, c.light)
|
||||
cal = (
|
||||
Calibration(
|
||||
@@ -113,10 +182,22 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
|
||||
else DEFAULT_CALIBRATION
|
||||
)
|
||||
plan = plan_alteration(coord, cal)
|
||||
content = resolve_content(c.content)
|
||||
# Resolve the soundtrack url against the CURRENT altitude (server-side, audio
|
||||
# spec §6.1). White-noise/off ignore the scale; soundtrack follows it.
|
||||
scale_audio = ""
|
||||
if app.state.ring is not None:
|
||||
scale_audio = scale_at(app.state.ring, req.altitude_index).audio
|
||||
audio = resolve_audio(c.audio, scale_audio=scale_audio)
|
||||
return {
|
||||
"plan": render_plan_to_dict(plan),
|
||||
"content": {"audio_source": content.audio_source, "video": content.video},
|
||||
"render": {
|
||||
"video": {"shown": resolve_visual(c.visual)},
|
||||
"audio": {
|
||||
"source": audio.source,
|
||||
"url": audio.url,
|
||||
"altitude_coupled": audio.altitude_coupled,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
@app.get("/dev/version")
|
||||
@@ -129,6 +210,24 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
|
||||
def api_clips():
|
||||
return {"clips": [c.to_dict() for c in app.state.clips]}
|
||||
|
||||
@app.get("/api/media-versions")
|
||||
def api_media_versions():
|
||||
"""Per-file content-hash tokens the client appends to /media URLs as
|
||||
`?v=<hash>`. Covers every served file: each clip's base footage plus each
|
||||
ring transition and its baked reverse. A re-baked clip's hash changes, so
|
||||
its URL changes and the browser refetches — a permanent cache-bust."""
|
||||
files = {c.base_file for c in app.state.clips}
|
||||
if app.state.ring is not None:
|
||||
for t in app.state.ring.transitions:
|
||||
files.add(t.file)
|
||||
files.add(_rev_file(t.file))
|
||||
versions = {}
|
||||
for f in sorted(files):
|
||||
v = _media_version(f)
|
||||
if v:
|
||||
versions[f] = v
|
||||
return {"versions": versions}
|
||||
|
||||
@app.get("/api/ring")
|
||||
def api_ring():
|
||||
if app.state.ring is None:
|
||||
@@ -145,15 +244,78 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
|
||||
req.delta,
|
||||
fast_spin_threshold=DEFAULT_FAST_SPIN_THRESHOLD,
|
||||
)
|
||||
return ring_move_to_dict(move, app.state.ring)
|
||||
# Rotating pool: pick a random member of the LANDED scale (content-pipeline
|
||||
# §11.1). A delta=0 advance is the initial / re-roll pick (a no-op move that
|
||||
# still yields a fresh random clip). The pure pick takes injected randomness.
|
||||
landed = scale_at(app.state.ring, move.to_index)
|
||||
chosen = pick_clip_id(landed, random.random())
|
||||
return ring_move_to_dict(move, app.state.ring, chosen)
|
||||
|
||||
@app.post("/api/author/track")
|
||||
def api_author_track(req: AuthorTrackRequest):
|
||||
# Author mode (§11.5): run the classical optical-flow tracker on a clip's
|
||||
# base footage, returning the keyframed track geometry for the author to
|
||||
# accept/correct. Semantics (key/strings/tiers) stay the author's.
|
||||
clip = next((c for c in app.state.clips if c.id == req.clip_id), None)
|
||||
if clip is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown clip {req.clip_id!r}")
|
||||
base = MEDIA_DIR / clip.base_file
|
||||
if not base.exists():
|
||||
raise HTTPException(status_code=404, detail=f"media absent: {clip.base_file}")
|
||||
try:
|
||||
import cv2
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="opencv-python not installed")
|
||||
from tools.pipeline.track import track_seed
|
||||
|
||||
cap = cv2.VideoCapture(str(base))
|
||||
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
|
||||
cap.release()
|
||||
seed_frame = max(0, min(total - 1, round(req.seed_t * total))) if total else 0
|
||||
return track_seed(
|
||||
base, req.key, tuple(req.seed_box), seed_frame=seed_frame,
|
||||
salience=req.salience, n_keyframes=req.n_keyframes, max_frames=req.max_frames,
|
||||
)
|
||||
|
||||
@app.post("/api/author/clip")
|
||||
def api_author_clip(req: AuthorClipRequest):
|
||||
# Author mode (§11.5 / stage 6): persist authored labels/affect/strings for
|
||||
# an EXISTING pool clip — idempotent upsert that keeps the clip's media +
|
||||
# provenance and replaces only the authored content. Reloads in place so
|
||||
# the preview reflects the edit without a restart.
|
||||
from tools.pipeline.manifest import upsert_clip
|
||||
|
||||
path = app.state.manifest_path
|
||||
data = json.loads(path.read_text())
|
||||
existing = next((c for c in data.get("clips", []) if c["id"] == req.clip_id), None)
|
||||
if existing is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown clip {req.clip_id!r}")
|
||||
merged = dict(existing)
|
||||
merged["annotations"] = req.annotations
|
||||
merged["affect"] = req.affect
|
||||
if req.strings:
|
||||
merged["strings"] = req.strings
|
||||
data = upsert_clip(data, merged)
|
||||
path.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n")
|
||||
app.state.clips = load_manifest(path)
|
||||
app.state.ring = load_ring(path)
|
||||
return {"ok": True, "clip": merged}
|
||||
|
||||
@app.middleware("http")
|
||||
async def _no_cache(request, call_next):
|
||||
# Dev preview server: never let a browser serve a stale app.js/style.css.
|
||||
# The whole point is by-eye iteration, so a plain refresh must always get
|
||||
# the latest assets (the heuristic cache otherwise hides JS/CSS edits).
|
||||
async def _cache_policy(request, call_next):
|
||||
# Dev preview server: never let a browser serve a stale app.js/style.css —
|
||||
# by-eye iteration needs a plain refresh to always get the latest assets.
|
||||
# Media uses `no-cache` (store, but REVALIDATE every load): instant altitude
|
||||
# swaps already come from the in-memory blob preload (static/app.js), so the
|
||||
# HTTP cache only matters on (re)load — where a conditional request returns a
|
||||
# cheap 304 when unchanged, yet picks up a re-sourced/re-cropped clip
|
||||
# immediately. `immutable` was wrong here: it pinned stale footage for a year
|
||||
# so edited clips never showed without a manual cache clear.
|
||||
response = await call_next(request)
|
||||
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
|
||||
if request.url.path.startswith("/media/"):
|
||||
response.headers["Cache-Control"] = "no-cache"
|
||||
else:
|
||||
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
|
||||
return response
|
||||
|
||||
if MEDIA_DIR.exists():
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Production pass for the 5 per-altitude soundtracks (audio spec §5.3). The audio
|
||||
analogue of build_pool_manifest.py --media: read the sourced ambience clips
|
||||
(re-downloadable per docs/audio-candidate-pool.md), and make each a seamless
|
||||
loudness-normalized loop.
|
||||
|
||||
Media is gitignored; this script is the reproducible record. Outputs land under
|
||||
simulator/sample_media/audio/<scale>/<name>.loop.mp3, served at /media/audio/...
|
||||
Run: python simulator/build_audio_media.py
|
||||
|
||||
(White-noise is deferred — the live Audio control is a simple on/off soundtrack
|
||||
toggle. `tools.pipeline.audio_run.generate_white_noise` remains for when it's
|
||||
wanted, but no noise bed is built here.)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from tools.pipeline.audio_run import process_soundtrack
|
||||
|
||||
AUDIO = Path(__file__).parent / "sample_media" / "audio"
|
||||
|
||||
# scale -> (sourced raw file, production output file), both under audio/<scale>/.
|
||||
# Sources per docs/audio-candidate-pool.md (the "✓ KEEP" picks).
|
||||
SOURCES: dict[str, tuple[str, str]] = {
|
||||
"cosmos": ("pillars.mp3", "pillars.loop.mp3"),
|
||||
"orbit": ("spaceamb.mp3", "spaceamb.loop.mp3"),
|
||||
"coast": ("waves.mp3", "waves.loop.mp3"),
|
||||
"reef": ("soundscape.mp3", "soundscape.loop.mp3"),
|
||||
"abyss": ("whale.wav", "whale.loop.mp3"),
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
for scale, (raw, out) in SOURCES.items():
|
||||
src = AUDIO / scale / raw
|
||||
if not src.exists():
|
||||
print(f" SKIP {scale}: source missing ({src}) — re-source per audio-candidate-pool.md")
|
||||
continue
|
||||
dst = process_soundtrack(src, AUDIO / scale / out)
|
||||
print(f" produced {dst.relative_to(AUDIO.parent)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,445 @@
|
||||
"""Build the rotating-pool manifest for the 5 ring scales (content-pipeline §11).
|
||||
|
||||
This is the reproducible, hand-authored BASELINE for Increment 2's manifest:
|
||||
each ring scale carries a POOL of vetted clips (docs/content-candidate-pool.md);
|
||||
the player picks one member at random on landing (player.ring.pick_clip_id). The
|
||||
authored label/affect content lives here in readable Python and is emitted to
|
||||
`simulator/sample_media/manifest.json` — far less error-prone than hand-editing 19
|
||||
clips of JSON, and a clean place for the simulator author mode to regenerate from.
|
||||
|
||||
Authoring model (design §11.3 / §11.4):
|
||||
- LEFT detail tiers: each label carries `salience` (1-4; first shows at Left
|
||||
level `5 - salience`) and a tiered string list (general -> specific ->
|
||||
scientific -> +fact) that escalates with the Left knob.
|
||||
- RIGHT emotion tiers: affect words are a tiered list per feeling (basic ->
|
||||
compound) escalating with the Right knob; still gated on min(Left, Right).
|
||||
- TIME-WINDOWED + TRACKED labels: a label may carry `appear`/`disappear`
|
||||
(loop-normalized) and a keyframed `track`; it shows only while on screen and
|
||||
its box follows the subject. The abyss + reef pools are the test material.
|
||||
|
||||
Strings: a value may be a LIST (indexed by tier-1) or a plain string (static,
|
||||
back-compat). Affect appears only when both knobs are up.
|
||||
|
||||
Usage:
|
||||
python simulator/build_pool_manifest.py # write manifest.json
|
||||
python simulator/build_pool_manifest.py --media # + generate new transition placeholders
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
MEDIA = Path(__file__).parent / "sample_media"
|
||||
MANIFEST = MEDIA / "manifest.json"
|
||||
|
||||
# --- Pools: scale id -> ordered clip ids (primary first) (content-candidate-pool.md) ---
|
||||
POOLS: dict[str, list[str]] = {
|
||||
"cosmos": ["cosmos", "cosmos_miri", "cosmos_galaxies", "cosmos_hudf", "cosmos_xdf"],
|
||||
"orbit": ["orbit_planetearth", "orbit_crewobs", "orbit_bluemarble"],
|
||||
"coast": ["coast_birdrock", "coast_surfgrass", "coast_elkbeach", "coast_drakesbeach"],
|
||||
"reef": ["reef_lionfish", "reef_spawning", "reef_hawkfish", "reef_snapper", "reef_coralspacific"],
|
||||
"abyss": ["abyss_wow", "abyss_midwaterexp", "abyss_hiding"],
|
||||
}
|
||||
|
||||
# Ring order (large -> small, wraps): cosmos -> orbit -> coast -> reef -> abyss -> cosmos.
|
||||
RING_ORDER = ["cosmos", "orbit", "coast", "reef", "abyss"]
|
||||
|
||||
# Per-scale soundtrack (audio spec §5.1) — the production-pass output of
|
||||
# simulator/build_audio_media.py, served at /media/audio/<path>.
|
||||
SCALE_AUDIO: dict[str, str] = {
|
||||
"cosmos": "cosmos/pillars.loop.mp3",
|
||||
"orbit": "orbit/spaceamb.loop.mp3",
|
||||
"coast": "coast/waves.loop.mp3",
|
||||
"reef": "reef/soundscape.loop.mp3",
|
||||
"abyss": "abyss/whale.loop.mp3",
|
||||
}
|
||||
|
||||
PD = "public-domain (US Gov, 17 U.S.C. §105)"
|
||||
PD_NPS = "public-domain (NPS, no © — 17 U.S.C. §105)"
|
||||
CCBY_STSCI = "CC-BY-class — credit NASA, ESA, STScI"
|
||||
CCBY_WEBB = "CC-BY-class — credit NASA, ESA, CSA, STScI"
|
||||
|
||||
# --- Per-clip provenance: id -> (title, license, source) ---
|
||||
META: dict[str, tuple[str, str, str]] = {
|
||||
# cosmos
|
||||
"cosmos": ("Cosmic Cliffs — Carina Nebula flythrough (NASA/ESA/CSA/STScI)", CCBY_WEBB,
|
||||
"NASA SVS 31348 Clifs-3d-STScI (Exploring the Cosmic Cliffs in 3D, Webb NIRCam); "
|
||||
"4K source, trim 10–34s, crossfade-loop. Text-free — replaces the Orion SVS 30957 flythrough."),
|
||||
"cosmos_miri": ("Cosmic Cliffs — Carina Nebula, mid-infrared (NASA/ESA/CSA/STScI)", CCBY_WEBB,
|
||||
"ESA/Webb weic2205c (Pan of the Carina Nebula, combined NIRCam + MIRI); 4K source, "
|
||||
"trim 4–26s, crossfade-loop. The MIRI mid-infrared composite — a muted gray/teal/gold "
|
||||
"dust palette distinct from the NIRCam flythrough, same Cosmic Cliffs towers. Text-free."),
|
||||
"cosmos_galaxies": ("Flying Through Galaxies (NASA SVS)", PD,
|
||||
"NASA SVS a014950 14950_Galaxies_FlyThrough_4k; trim 8–32s, crossfade-loop"),
|
||||
"cosmos_hudf": ("Hubble Ultra Deep Field zoom (NASA/ESA/STScI)", CCBY_STSCI,
|
||||
"NASA SVS a030687 hudf-b-1920x1080p30; trim 20–44s, crossfade-loop"),
|
||||
"cosmos_xdf": ("eXtreme Deep Field flythrough (NASA/ESA/STScI)", CCBY_STSCI,
|
||||
"NASA SVS a030681 hxdf_fly-b-1920x1080p30; trim 2–26s, crossfade-loop"),
|
||||
# orbit
|
||||
"orbit_planetearth": ("ISS — View of Planet Earth (NASA SVS)", PD,
|
||||
"NASA SVS a030771 ISS_View_of_Planet_Earth_2160p; trim 10–34s, crossfade-loop"),
|
||||
"orbit_crewobs": ("ISS — Crew Earth Observations (NASA SVS)", PD,
|
||||
"NASA SVS a030771 ISS_Crew_Earth_Observations_2160p; trim 9–33s, crossfade-loop"),
|
||||
"orbit_bluemarble": ("NPP “Blue Marble” rotating globe (NASA SVS)", PD,
|
||||
"NASA SVS a004550 BlueMarble_starfield_4k_2160p30_v2; trim 10–34s, crossfade-loop"),
|
||||
# coast
|
||||
"coast_birdrock": ("Bird rock + ocean, aerial (NPS Channel Islands)", PD_NPS,
|
||||
"NPS Channel Islands AV 663a2f13; trim 3–27s, crossfade-loop"),
|
||||
"coast_surfgrass": ("Surfgrass / coralline tidepool (NPS Cabrillo)", PD_NPS,
|
||||
"NPS Cabrillo AV de7d1cf2; trim 2–26s, crossfade-loop"),
|
||||
"coast_elkbeach": ("Elk resting in coastal fog (NPS Redwood)", PD_NPS,
|
||||
"NPS Redwood redw-elkbeach_1280x720; trim 20–38s, crossfade-loop (720p source)"),
|
||||
"coast_drakesbeach": ("Elephant seals on dark sand (NPS Point Reyes)", PD_NPS,
|
||||
"NPS Point Reyes AV 1cfd8165; trim 20–35s, crossfade-loop"),
|
||||
# reef
|
||||
"reef_lionfish": ("Lionfish hovering over reef (NOAA Fisheries)", PD,
|
||||
"NOAA Fisheries b-roll VIDEO_ID 4088881464001; trim 12–36s, crossfade-loop"),
|
||||
"reef_spawning": ("Snapper school over sunlit reef (NOAA Fisheries)", PD,
|
||||
"NOAA Fisheries b-roll VIDEO_ID 1553921798001; trim 50–78s, crossfade-loop"),
|
||||
"reef_hawkfish": ("Humphead parrotfish over reef (NOAA Fisheries)", PD,
|
||||
"NOAA Fisheries b-roll VIDEO_ID 6039896460001; trim 250–274s, crossfade-loop"),
|
||||
"reef_snapper": ("Red snapper schooling (NOAA Fisheries)", PD,
|
||||
"NOAA Fisheries b-roll VIDEO_ID 5305427942001; trim 7–31s, crossfade-loop"),
|
||||
"reef_coralspacific": ("Pacific coral macro (NOAA Fisheries)", PD,
|
||||
"NOAA Fisheries b-roll VIDEO_ID 5231464663001; trim 30–54s, crossfade-loop"),
|
||||
# abyss
|
||||
"abyss_wow": ("“World of Water” (NOAA Ocean Exploration)", PD,
|
||||
"NOAA Ocean Exploration wow-1280x720-1; trim 10–34s, crossfade-loop"),
|
||||
"abyss_midwaterexp": ("“Midwater Exploration” (NOAA Ocean Exploration)", PD,
|
||||
"NOAA Ocean Exploration midwater-exploration-1920x1080-1; trim 16–40s, crossfade-loop"),
|
||||
"abyss_hiding": ("“Hiding in the Dark” (NOAA Ocean Exploration)", PD,
|
||||
"NOAA Ocean Exploration dark-1280x720-1; trim 20–44s, crossfade-loop"),
|
||||
}
|
||||
|
||||
# --- Affect vocabulary per SCALE (shared by all the scale's pool members) -------
|
||||
# Each: (key, [x, y], min_level, [tier1 basic .. tier4 compound]). The word shown
|
||||
# escalates with the RIGHT knob; appears only when both knobs are up (§11.4).
|
||||
AFFECT: dict[str, list[tuple]] = {
|
||||
"cosmos": [
|
||||
("feel.wonder", [0.50, 0.44], 1, ["wow", "wonder", "awe", "transcendent awe"]),
|
||||
("feel.vastness", [0.20, 0.70], 2, ["big", "vastness", "immensity", "a dizzying immensity"]),
|
||||
("feel.insignificance", [0.66, 0.60], 3, ["small", "smallness", "insignificance", "a humbling insignificance"]),
|
||||
("feel.longing", [0.42, 0.84], 4, ["want", "longing", "yearning", "an aching yearning"]),
|
||||
],
|
||||
"orbit": [
|
||||
("feel.serenity", [0.50, 0.44], 1, ["calm", "serenity", "peace", "a quiet, weightless peace"]),
|
||||
("feel.fragility", [0.22, 0.68], 2, ["thin", "fragility", "vulnerability", "a tender fragility"]),
|
||||
("feel.unity", [0.66, 0.58], 3, ["one", "unity", "belonging", "a borderless belonging"]),
|
||||
("feel.distance", [0.42, 0.82], 4, ["far", "distance", "remoteness", "an exquisite remoteness"]),
|
||||
],
|
||||
"coast": [
|
||||
("feel.ease", [0.50, 0.44], 1, ["nice", "ease", "calm", "an unhurried calm"]),
|
||||
("feel.nostalgia", [0.22, 0.68], 2, ["miss", "nostalgia", "wistfulness", "a salt-air wistfulness"]),
|
||||
("feel.belonging", [0.66, 0.58], 3, ["home", "belonging", "rootedness", "a tidal rootedness"]),
|
||||
("feel.melancholy", [0.42, 0.82], 4, ["sad", "melancholy", "longing", "a soft seaward longing"]),
|
||||
],
|
||||
"reef": [
|
||||
("feel.delight", [0.50, 0.44], 1, ["fun", "delight", "joy", "a darting, bright joy"]),
|
||||
("feel.abundance", [0.22, 0.68], 2, ["full", "abundance", "richness", "a teeming richness"]),
|
||||
("feel.curiosity", [0.66, 0.58], 3, ["look", "curiosity", "fascination", "an absorbed fascination"]),
|
||||
("feel.immersion", [0.42, 0.82], 4, ["in", "immersion", "absorption", "a held, breathless absorption"]),
|
||||
],
|
||||
"abyss": [
|
||||
("feel.unease", [0.50, 0.44], 1, ["uh", "unease", "disquiet", "a creeping disquiet"]),
|
||||
("feel.fascination", [0.22, 0.68], 2, ["ooh", "fascination", "wonder", "a forbidden wonder"]),
|
||||
("feel.isolation", [0.66, 0.58], 3, ["alone", "isolation", "solitude", "a crushing solitude"]),
|
||||
("feel.dread", [0.42, 0.82], 4, ["fear", "dread", "foreboding", "a slow, cold foreboding"]),
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def static_label(key, salience, tiers, box):
|
||||
"""A fixed-box tiered label (always on-screen, salience-gated by Left)."""
|
||||
return {"key": key, "salience": salience, "box": box, "_tiers": tiers}
|
||||
|
||||
|
||||
def tracked_label(key, salience, tiers, appear, disappear, track):
|
||||
"""A time-windowed tracked label: shows only in [appear, disappear] and its
|
||||
box follows the subject (§11.2). `track` = [(t, [x,y,w,h]), ...]."""
|
||||
return {
|
||||
"key": key, "salience": salience, "appear": appear, "disappear": disappear,
|
||||
"track": [{"t": t, "box": b} for t, b in track], "_tiers": tiers,
|
||||
}
|
||||
|
||||
|
||||
def measure(key, value, box, min_level):
|
||||
"""A single-tier measurement readout (not a tiered object), gated by Left."""
|
||||
return {"key": key, "box": box, "min_level": min_level, "_tiers": value}
|
||||
|
||||
|
||||
# --- Labels per clip. `_tiers` is lifted into the strings table by build(). ------
|
||||
# Showcase tracked/windowed labels live on the abyss + reef pools (the creatures
|
||||
# drift through frame). Other clips get a few salience-gated static tiered labels.
|
||||
LABELS: dict[str, list[dict]] = {
|
||||
# ---------- cosmos ----------
|
||||
"cosmos": [
|
||||
static_label("detected.nebula", 4, ["cloud", "nebula", "emission nebula", "emission nebula · the Carina star-forming region"], [0.20, 0.42, 0.5, 0.42]),
|
||||
static_label("detected.star", 2, ["star", "young star", "protostar", "protostar · a newborn star, <1 Myr old"], [0.54, 0.12, 0.12, 0.14]),
|
||||
measure("measure.distance", "≈7,500 ly", [0.06, 0.06, 0.2, 0.08], 3),
|
||||
],
|
||||
"cosmos_miri": [
|
||||
static_label("detected.nebula", 4, ["dust", "nebula", "dust ridge", "dust ridge · mid-infrared glow of the Carina cliffs"], [0.10, 0.45, 0.7, 0.45]),
|
||||
static_label("detected.star", 2, ["star", "young star", "protostar", "protostar · its dust laid bare in mid-infrared"], [0.48, 0.18, 0.14, 0.16]),
|
||||
measure("measure.distance", "≈7,500 ly", [0.06, 0.06, 0.2, 0.08], 3),
|
||||
],
|
||||
"cosmos_galaxies": [
|
||||
static_label("detected.galaxy", 4, ["galaxy", "spiral galaxy", "barred spiral", "barred spiral · ~10¹¹ stars"], [0.34, 0.30, 0.30, 0.34]),
|
||||
measure("measure.distance", "~Mly", [0.06, 0.06, 0.18, 0.08], 3),
|
||||
],
|
||||
"cosmos_hudf": [
|
||||
static_label("detected.galaxy", 4, ["smudge", "galaxy", "early galaxy", "early galaxy · z ≈ 1–6, light Gyr-old"], [0.40, 0.36, 0.20, 0.22]),
|
||||
measure("measure.field", "deep field", [0.06, 0.06, 0.2, 0.08], 3),
|
||||
],
|
||||
"cosmos_xdf": [
|
||||
static_label("detected.galaxy", 4, ["dot", "galaxy", "faint galaxy", "faint galaxy · among the earliest seen"], [0.42, 0.38, 0.18, 0.2]),
|
||||
],
|
||||
# ---------- orbit ----------
|
||||
"orbit_planetearth": [
|
||||
static_label("detected.cloud_band", 4, ["clouds", "cloud band", "cumulus field", "cumulus field · tropical convection"], [0.30, 0.30, 0.30, 0.20]),
|
||||
static_label("detected.limb", 2, ["edge", "Earth’s limb", "atmospheric limb", "atmospheric limb · ~100 km of air"], [0.05, 0.70, 0.9, 0.12]),
|
||||
measure("measure.altitude", "~408 km", [0.06, 0.06, 0.18, 0.08], 3),
|
||||
],
|
||||
"orbit_crewobs": [
|
||||
static_label("detected.coastline", 4, ["land", "coastline", "continental margin", "continental margin · land meets sea"], [0.30, 0.34, 0.34, 0.26]),
|
||||
measure("measure.altitude", "~408 km", [0.06, 0.06, 0.18, 0.08], 3),
|
||||
],
|
||||
"orbit_bluemarble": [
|
||||
static_label("detected.globe", 4, ["Earth", "the globe", "terrestrial planet", "terrestrial planet · 12,742 km across"], [0.28, 0.18, 0.44, 0.6]),
|
||||
],
|
||||
# ---------- coast ----------
|
||||
"coast_birdrock": [
|
||||
static_label("detected.surf", 4, ["waves", "surf", "breaking swell", "breaking swell · wind-driven, ~10 s period"], [0.20, 0.55, 0.6, 0.3]),
|
||||
static_label("detected.searock", 2, ["rock", "sea stack", "coastal sea stack", "sea stack · wave-cut residual rock"], [0.30, 0.25, 0.22, 0.3]),
|
||||
],
|
||||
"coast_surfgrass": [
|
||||
static_label("detected.surfgrass", 4, ["grass", "surfgrass", "Phyllospadix", "Phyllospadix · a marine seagrass, not algae"], [0.18, 0.40, 0.5, 0.4]),
|
||||
static_label("detected.coralline", 2, ["pink", "coralline algae", "crustose coralline", "crustose coralline · calcified red algae"], [0.6, 0.55, 0.2, 0.2]),
|
||||
],
|
||||
"coast_elkbeach": [
|
||||
static_label("detected.elk", 4, ["elk", "Roosevelt elk", "Cervus canadensis roosevelti", "C. c. roosevelti · largest elk subspecies"], [0.34, 0.42, 0.3, 0.3]),
|
||||
static_label("detected.fog", 2, ["fog", "coastal fog", "advection fog", "advection fog · warm air over cold upwelling"], [0.05, 0.05, 0.9, 0.25]),
|
||||
],
|
||||
"coast_drakesbeach": [
|
||||
static_label("detected.seal", 4, ["seals", "elephant seals", "Mirounga angustirostris", "M. angustirostris · males to 2,000 kg"], [0.25, 0.55, 0.5, 0.3]),
|
||||
static_label("detected.sand", 1, ["sand", "dark sand", "mineral-dark beach", "dark beach · eroded coastal sediment"], [0.05, 0.8, 0.9, 0.15]),
|
||||
],
|
||||
# ---------- reef (showcase: tracked + windowed) ----------
|
||||
"reef_lionfish": [
|
||||
tracked_label(
|
||||
"detected.lionfish", 4,
|
||||
["fish", "lionfish", "Pterois volitans", "Pterois volitans · venomous spines, invasive in the Atlantic"],
|
||||
0.05, 0.95,
|
||||
[(0.05, [0.15, 0.30, 0.16, 0.18]), (0.5, [0.50, 0.38, 0.16, 0.18]), (0.95, [0.72, 0.34, 0.16, 0.18])],
|
||||
),
|
||||
tracked_label(
|
||||
"detected.snapper", 2,
|
||||
["fish", "blue snapper", "Lutjanus", "Lutjanus · schools over reef structure"],
|
||||
0.0, 0.6,
|
||||
[(0.0, [0.05, 0.55, 0.12, 0.1]), (0.3, [0.22, 0.5, 0.12, 0.1]), (0.6, [0.40, 0.56, 0.12, 0.1])],
|
||||
),
|
||||
measure("measure.depth", "−18 m", [0.06, 0.06, 0.16, 0.08], 3),
|
||||
],
|
||||
"reef_spawning": [
|
||||
tracked_label(
|
||||
"detected.school", 4,
|
||||
["fish", "fish school", "snapper aggregation", "snapper aggregation · synchronized spawning run"],
|
||||
0.0, 1.0,
|
||||
[(0.0, [0.20, 0.30, 0.2, 0.18]), (0.5, [0.55, 0.34, 0.2, 0.18]), (1.0, [0.20, 0.30, 0.2, 0.18])],
|
||||
),
|
||||
static_label("detected.coral", 2, ["coral", "reef coral", "scleractinian", "scleractinian · reef-building stony coral"], [0.15, 0.6, 0.35, 0.3]),
|
||||
measure("measure.depth", "−22 m", [0.06, 0.06, 0.16, 0.08], 3),
|
||||
],
|
||||
"reef_hawkfish": [
|
||||
tracked_label(
|
||||
"detected.parrotfish", 4,
|
||||
["fish", "parrotfish", "Bolbometopon muricatum", "B. muricatum · humphead, grazes reef into sand"],
|
||||
0.1, 0.9,
|
||||
[(0.1, [0.55, 0.25, 0.2, 0.22]), (0.5, [0.35, 0.34, 0.2, 0.22]), (0.9, [0.12, 0.40, 0.2, 0.22])],
|
||||
),
|
||||
measure("measure.depth", "−12 m", [0.06, 0.06, 0.16, 0.08], 3),
|
||||
],
|
||||
"reef_snapper": [
|
||||
tracked_label(
|
||||
"detected.snapper", 4,
|
||||
["fish", "red snapper", "Lutjanus campechanus", "L. campechanus · long-lived, can reach 50+ yr"],
|
||||
0.0, 0.7,
|
||||
[(0.0, [0.10, 0.3, 0.16, 0.14]), (0.35, [0.40, 0.36, 0.16, 0.14]), (0.7, [0.68, 0.30, 0.16, 0.14])],
|
||||
),
|
||||
],
|
||||
"reef_coralspacific": [
|
||||
static_label("detected.coral", 4, ["coral", "coral colony", "Pacific scleractinian", "Pacific scleractinian · a symbiosis with algae"], [0.2, 0.4, 0.4, 0.4]),
|
||||
tracked_label(
|
||||
"detected.spotfish", 2,
|
||||
["fish", "spotted fish", "reef damselfish", "reef damselfish · territorial over its coral"],
|
||||
0.2, 0.8,
|
||||
[(0.2, [0.6, 0.25, 0.1, 0.09]), (0.5, [0.45, 0.3, 0.1, 0.09]), (0.8, [0.3, 0.26, 0.1, 0.09])],
|
||||
),
|
||||
],
|
||||
# ---------- abyss (showcase: tracked + windowed; creatures drift through) ----------
|
||||
"abyss_wow": [
|
||||
tracked_label(
|
||||
"detected.jelly", 4,
|
||||
["jelly", "comb jelly", "Ctenophora", "Ctenophora · swims by beating rows of cilia"],
|
||||
0.0, 0.55,
|
||||
[(0.0, [0.05, 0.25, 0.14, 0.18]), (0.3, [0.30, 0.32, 0.14, 0.18]), (0.55, [0.52, 0.28, 0.14, 0.18])],
|
||||
),
|
||||
tracked_label(
|
||||
"detected.siphonophore", 3,
|
||||
["chain", "siphonophore", "Siphonophorae", "Siphonophorae · a colony of clones, can exceed 40 m"],
|
||||
0.45, 1.0,
|
||||
[(0.45, [0.80, 0.6, 0.1, 0.3]), (0.7, [0.6, 0.45, 0.1, 0.34]), (1.0, [0.45, 0.3, 0.1, 0.38])],
|
||||
),
|
||||
measure("measure.depth", "−1,200 m", [0.06, 0.06, 0.16, 0.08], 2),
|
||||
],
|
||||
"abyss_midwaterexp": [
|
||||
tracked_label(
|
||||
"detected.medusa", 4,
|
||||
["jelly", "white medusa", "Scyphozoa", "Scyphozoa · the swimming medusa stage of a jellyfish"],
|
||||
0.0, 0.6,
|
||||
[(0.0, [0.55, 0.20, 0.16, 0.2]), (0.3, [0.40, 0.34, 0.16, 0.2]), (0.6, [0.22, 0.30, 0.16, 0.2])],
|
||||
),
|
||||
tracked_label(
|
||||
"detected.worm", 2,
|
||||
["worm", "red worm", "polychaete", "polychaete · a free-swimming bristle worm"],
|
||||
0.5, 1.0,
|
||||
[(0.5, [0.10, 0.7, 0.12, 0.1]), (0.75, [0.3, 0.6, 0.12, 0.1]), (1.0, [0.5, 0.66, 0.12, 0.1])],
|
||||
),
|
||||
measure("measure.depth", "−1,600 m", [0.06, 0.06, 0.16, 0.08], 2),
|
||||
],
|
||||
"abyss_hiding": [
|
||||
tracked_label(
|
||||
"detected.jelly", 4,
|
||||
["jelly", "crimson jelly", "Scyphozoa", "Scyphozoa · red is invisible in the lightless deep"],
|
||||
0.0, 0.7,
|
||||
[(0.0, [0.15, 0.28, 0.16, 0.2]), (0.35, [0.42, 0.36, 0.16, 0.2]), (0.7, [0.66, 0.30, 0.16, 0.2])],
|
||||
),
|
||||
measure("measure.depth", "−2,140 m", [0.06, 0.06, 0.16, 0.08], 2),
|
||||
],
|
||||
}
|
||||
|
||||
# Human-readable strings for affect/measurement keys are produced from the tiered
|
||||
# data above; this maps measurement keys to nothing extra (their value IS the string).
|
||||
|
||||
|
||||
def _affect_for_clip(scale: str) -> tuple[list, dict]:
|
||||
"""Build the affect list + its strings for a scale's pool member."""
|
||||
entries, strings = [], {}
|
||||
for key, at, min_level, tiers in AFFECT[scale]:
|
||||
entries.append({"key": key, "at": at, "min_level": min_level})
|
||||
strings[key] = tiers
|
||||
return entries, strings
|
||||
|
||||
|
||||
def _labels_for_clip(clip_id: str) -> tuple[list, dict]:
|
||||
"""Build the annotation list + its strings, lifting `_tiers` into the table."""
|
||||
anns, strings = [], {}
|
||||
for raw in LABELS.get(clip_id, []):
|
||||
a = {k: v for k, v in raw.items() if k != "_tiers"}
|
||||
anns.append(a)
|
||||
strings[a["key"]] = raw["_tiers"]
|
||||
return anns, strings
|
||||
|
||||
|
||||
def _clip_entry(scale: str, clip_id: str) -> dict:
|
||||
title, license_, source = META[clip_id]
|
||||
anns, lab_strings = _labels_for_clip(clip_id)
|
||||
affect, aff_strings = _affect_for_clip(scale)
|
||||
return {
|
||||
"id": clip_id,
|
||||
"title": title,
|
||||
"base_file": f"{clip_id}/base.mp4",
|
||||
"license": license_,
|
||||
"source": source,
|
||||
"right_variants": {},
|
||||
"annotations": anns,
|
||||
"affect": affect,
|
||||
"strings": {"en": {**lab_strings, **aff_strings}},
|
||||
}
|
||||
|
||||
|
||||
def build_manifest() -> dict:
|
||||
"""Assemble the full rotating-pool manifest dict."""
|
||||
clips = []
|
||||
for scale in RING_ORDER:
|
||||
for clip_id in POOLS[scale]:
|
||||
clips.append(_clip_entry(scale, clip_id))
|
||||
scales = [
|
||||
{"id": s, "clip_id": POOLS[s][0], "pool": POOLS[s], "audio": SCALE_AUDIO[s]}
|
||||
for s in RING_ORDER
|
||||
]
|
||||
transitions = []
|
||||
n = len(RING_ORDER)
|
||||
for i in range(n):
|
||||
a, b = RING_ORDER[i], RING_ORDER[(i + 1) % n]
|
||||
transitions.append({"file": f"transitions/{a}-{b}.mp4", "model": "placeholder-zoom"})
|
||||
return {"clips": clips, "ring": {"scales": scales, "transitions": transitions}}
|
||||
|
||||
|
||||
# --- Generate the per-edge transition clips (zoom morph + reversed companion) ---
|
||||
|
||||
def _ffmpeg() -> str:
|
||||
if shutil.which("ffmpeg"):
|
||||
return "ffmpeg"
|
||||
import imageio_ffmpeg
|
||||
return imageio_ffmpeg.get_ffmpeg_exe()
|
||||
|
||||
|
||||
def _make_transition(ff: str, scale_a: str, scale_b: str) -> Path:
|
||||
"""A zoom/warp morph between two scales, from their PRIMARY pool members'
|
||||
bases (mirrors setup_scales_media.py but keyed by scale->primary). This is the
|
||||
recipe behind the well-liked orbit-coast edge; generate_media() now applies it
|
||||
uniformly to every edge so none carry stale footage from an earlier ring."""
|
||||
a = MEDIA / POOLS[scale_a][0] / "base.mp4"
|
||||
b = MEDIA / POOLS[scale_b][0] / "base.mp4"
|
||||
out = MEDIA / "transitions" / f"{scale_a}-{scale_b}.mp4"
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
norm = "trim=0:3,setpts=PTS-STARTPTS,scale=1280:720,fps=25,setsar=1,format=yuv420p"
|
||||
subprocess.run([
|
||||
ff, "-y", "-i", str(a), "-i", str(b), "-filter_complex",
|
||||
f"[0:v]{norm}[a];[1:v]{norm}[b];"
|
||||
"[a][b]xfade=transition=zoomin:duration=1.5:offset=0.75,format=yuv420p[v]",
|
||||
"-map", "[v]", "-an", str(out),
|
||||
], check=True, capture_output=True)
|
||||
return out
|
||||
|
||||
|
||||
def _make_reverse(ff: str, forward: Path) -> Path:
|
||||
"""The zoom-OUT companion of a forward (zoom-in) edge: the same clip played
|
||||
backward, written alongside it as `<edge>.rev.mp4`. An ascending ring move
|
||||
crosses its edge `reversed`; the renderer then plays this file, so zooming out
|
||||
recedes from the current scale back to the higher one instead of zooming in."""
|
||||
out = forward.with_suffix(".rev.mp4")
|
||||
subprocess.run([
|
||||
ff, "-y", "-i", str(forward), "-vf", "reverse", "-an", str(out),
|
||||
], check=True, capture_output=True)
|
||||
return out
|
||||
|
||||
|
||||
def generate_media() -> None:
|
||||
"""(Re)build EVERY ring-edge transition from the current real primary bases —
|
||||
the orbit-coast recipe applied uniformly, overwriting any stale clips left from
|
||||
an earlier ring — plus a reversed companion per edge for zoom-out moves."""
|
||||
ff = _ffmpeg()
|
||||
n = len(RING_ORDER)
|
||||
for i in range(n):
|
||||
a, b = RING_ORDER[i], RING_ORDER[(i + 1) % n]
|
||||
fwd = _make_transition(ff, a, b)
|
||||
_make_reverse(ff, fwd)
|
||||
print(f"generated transitions/{a}-{b}.mp4 (+ .rev) from real bases")
|
||||
|
||||
|
||||
def main(argv: list[str]) -> None:
|
||||
manifest = build_manifest()
|
||||
MANIFEST.write_text(json.dumps(manifest, indent=2, ensure_ascii=False) + "\n")
|
||||
print(f"wrote {MANIFEST} — {len(manifest['clips'])} clips, "
|
||||
f"{len(manifest['ring']['scales'])} ring scales")
|
||||
if "--media" in argv:
|
||||
generate_media()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main(sys.argv[1:])
|
||||
+38
-8
@@ -73,6 +73,18 @@ def load_manifest(path: str | Path) -> list[Clip]:
|
||||
return [_clip_from_dict(c) for c in data["clips"]]
|
||||
|
||||
|
||||
def _scale_from_dict(s: dict[str, Any]) -> Scale:
|
||||
"""A ring scale node, with its rotating clip pool (content-pipeline §11.1).
|
||||
|
||||
Accepts a `pool` list (the rotating model) and/or a legacy single `clip_id`.
|
||||
The primary `clip_id` is the explicit one if given, else the first pool
|
||||
member; `pool` defaults to `(clip_id,)` for a pre-pool manifest.
|
||||
"""
|
||||
pool = tuple(s.get("pool", []))
|
||||
clip_id = s.get("clip_id") or (pool[0] if pool else "")
|
||||
return Scale(id=s["id"], clip_id=clip_id, pool=pool, audio=s.get("audio", ""))
|
||||
|
||||
|
||||
def load_ring(path: str | Path) -> ScaleRing | None:
|
||||
"""Load the optional `ring` section as a `ScaleRing`, or None if absent.
|
||||
|
||||
@@ -85,9 +97,7 @@ def load_ring(path: str | Path) -> ScaleRing | None:
|
||||
ring = data.get("ring")
|
||||
if not ring:
|
||||
return None
|
||||
scales = tuple(
|
||||
Scale(id=s["id"], clip_id=s["clip_id"]) for s in ring.get("scales", [])
|
||||
)
|
||||
scales = tuple(_scale_from_dict(s) for s in ring.get("scales", []))
|
||||
transitions = tuple(
|
||||
Transition(file=t["file"], model=t.get("model", ""))
|
||||
for t in ring.get("transitions", [])
|
||||
@@ -97,27 +107,47 @@ def load_ring(path: str | Path) -> ScaleRing | None:
|
||||
|
||||
def ring_to_dict(ring: ScaleRing, clips: list[Clip]) -> dict:
|
||||
"""JSON form of the ring for the API: ordered scales (with their clip title)
|
||||
and the per-edge transitions."""
|
||||
and the per-edge transitions.
|
||||
|
||||
Each scale carries its full rotating `pool` (clip id + title per member) plus
|
||||
the primary `clip_id`/`title` for back-compat. The client lands on a member
|
||||
chosen at the API boundary (`/api/ring/advance` → `target_clip_id`)."""
|
||||
titles = {c.id: c.title for c in clips}
|
||||
return {
|
||||
"scales": [
|
||||
{"id": s.id, "clip_id": s.clip_id, "title": titles.get(s.clip_id, s.id)}
|
||||
{
|
||||
"id": s.id,
|
||||
"clip_id": s.clip_id,
|
||||
"title": titles.get(s.clip_id, s.id),
|
||||
"audio": s.audio,
|
||||
"pool": [
|
||||
{"clip_id": cid, "title": titles.get(cid, cid)}
|
||||
for cid in s.members
|
||||
],
|
||||
}
|
||||
for s in ring.scales
|
||||
],
|
||||
"transitions": [{"file": t.file, "model": t.model} for t in ring.transitions],
|
||||
}
|
||||
|
||||
|
||||
def ring_move_to_dict(move: RingMove, ring: ScaleRing) -> dict:
|
||||
def ring_move_to_dict(
|
||||
move: RingMove, ring: ScaleRing, chosen_clip_id: str | None = None
|
||||
) -> dict:
|
||||
"""JSON form of an encoder move: the landing scale's clip and the ordered
|
||||
transition clips to play (with direction). `fast` flags a collapsed fast-spin
|
||||
pass; the single step then carries `blended` so the renderer plays it quick."""
|
||||
pass; the single step then carries `blended` so the renderer plays it quick.
|
||||
|
||||
`target_clip_id` is the pool member the player should load on landing. The
|
||||
caller passes the random pick (`pick_clip_id(landed_scale, random.random())`,
|
||||
content-pipeline §11.1); when omitted it falls back to the scale's primary
|
||||
`clip_id` (deterministic — keeps pre-pool callers/tests working)."""
|
||||
return {
|
||||
"from_index": move.from_index,
|
||||
"to_index": move.to_index,
|
||||
"wrapped": move.wrapped,
|
||||
"fast": move.fast,
|
||||
"target_clip_id": scale_at(ring, move.to_index).clip_id,
|
||||
"target_clip_id": chosen_clip_id or scale_at(ring, move.to_index).clip_id,
|
||||
"steps": [
|
||||
{
|
||||
"edge": st.edge,
|
||||
|
||||
+2112
-184
File diff suppressed because it is too large
Load Diff
@@ -1,5 +1,13 @@
|
||||
"""Generate cheap placeholder media for the full 5-scale RING (scales design §3).
|
||||
|
||||
SUPERSEDED (session 0016): the ring now uses real strict-PD footage in a ROTATING
|
||||
POOL per scale, regenerated by `simulator/build_pool_manifest.py` (which also makes
|
||||
the new coast-edge transition placeholders). This generator built the older
|
||||
one-base-per-scale forest ring (cosmos/orbit/forest/reef/abyss) and is kept only
|
||||
for reference / a clean-room placeholder rebuild; it does NOT match the current
|
||||
pool manifest or ring order (cosmos → orbit → coast → reef → abyss).
|
||||
|
||||
|
||||
Produces offline synthetic placeholders for all five ring scales — **cosmos**
|
||||
(NASA/Hubble), **orbit** (NASA/ISS), **forest** (NPS/USGS), **reef** (NOAA Ocean
|
||||
Exploration), and **abyss** (NOAA Ocean Exploration) — plus the short zoom/warp
|
||||
|
||||
+615
-37
@@ -31,8 +31,10 @@ window.addEventListener("error", (e) =>
|
||||
showError(`${e.message} @ ${(e.filename || "").split("/").pop()}:${e.lineno}`));
|
||||
|
||||
let clipsById = {}; // id -> clip manifest entry
|
||||
let ring = null; // {scales:[{id,clip_id,title}], transitions:[...]} or null
|
||||
let ring = null; // {scales:[{id,clip_id,title,pool:[...]}], transitions:[...]} or null
|
||||
let serverRing = false; // true when /api/ring served a real ring (vs the synthesized fallback)
|
||||
let ringIndex = 0; // current scale position on the ring
|
||||
let activeClipId = null; // the pool member chosen for the current scale landing
|
||||
let currentClipId = null; // base media currently loaded (reset to force a reload)
|
||||
let busy = false; // true while a ring transition is playing
|
||||
let lastOverlay = null; // {level, intensity} from the most-recent renderOverlay call; drives per-frame track animation
|
||||
@@ -40,21 +42,143 @@ let lastOverlay = null; // {level, intensity} from the most-recent renderOver
|
||||
async function loadData() {
|
||||
const clips = (await (await fetch("/api/clips")).json()).clips || [];
|
||||
clipsById = Object.fromEntries(clips.map((c) => [c.id, c]));
|
||||
// Per-file content-hash tokens → appended to /media URLs as ?v=<hash> so a
|
||||
// re-baked clip (new bytes, same path) gets a fresh URL the browser can't serve
|
||||
// stale. Best-effort: an empty map just yields un-versioned URLs.
|
||||
try { mediaVersions = (await (await fetch("/api/media-versions")).json()).versions || {}; }
|
||||
catch (_) { mediaVersions = {}; }
|
||||
const r = await fetch("/api/ring");
|
||||
serverRing = r.ok;
|
||||
ring = r.ok ? await r.json() : null;
|
||||
if (!ring && clips.length) {
|
||||
// No ring: single-clip mode — synthesize a 1-scale ring over clip 0.
|
||||
ring = { scales: [{ id: clips[0].id, clip_id: clips[0].id, title: clips[0].title }], transitions: [] };
|
||||
// No ring: single-clip mode — synthesize a 1-scale ring (pool of one) over clip 0.
|
||||
ring = { scales: [{ id: clips[0].id, clip_id: clips[0].id, title: clips[0].title, pool: [{ clip_id: clips[0].id, title: clips[0].title }] }], transitions: [] };
|
||||
}
|
||||
ringIndex = 0;
|
||||
}
|
||||
|
||||
function activeClip() {
|
||||
if (!ring) return null;
|
||||
return clipsById[ring.scales[ringIndex].clip_id] || null;
|
||||
// Server-canonical random pick of a pool member for the current scale: a delta=0
|
||||
// advance (content-pipeline §11.1, Python owns the randomness). The synthesized
|
||||
// fallback ring has a pool of one, so it just returns that member. Returns a
|
||||
// clip_id or null. Shared by the initial landing AND the Dev Mode re-roll button.
|
||||
async function pickRandomMember() {
|
||||
const scale = ring && ring.scales[ringIndex];
|
||||
if (!scale) return null;
|
||||
if (serverRing) {
|
||||
try {
|
||||
const resp = await fetch("/api/ring/advance", {
|
||||
method: "POST", headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ from_index: ringIndex, delta: 0 }),
|
||||
});
|
||||
if (resp.ok) return (await resp.json()).target_clip_id;
|
||||
} catch (_) { /* fall through to the primary member */ }
|
||||
}
|
||||
return scale.clip_id;
|
||||
}
|
||||
|
||||
function mediaUrl(file) { return "/media/" + file; }
|
||||
// Land on the current scale: pick a random pool member and force its media to load.
|
||||
async function landScale() {
|
||||
activeClipId = await pickRandomMember();
|
||||
currentClipId = null;
|
||||
}
|
||||
|
||||
function activeClip() {
|
||||
return clipsById[activeClipId] || null;
|
||||
}
|
||||
|
||||
// In-memory preload cache: media file path -> blob object URL. Once a clip's bytes
|
||||
// are cached, mediaUrl() hands the <video> an in-memory blob instead of a network
|
||||
// path, so swapping the altitude scale loads near-instantly (no fetch round-trip).
|
||||
// Falls back to the network path for anything not yet (or never) cached.
|
||||
const mediaBlobs = {};
|
||||
let mediaVersions = {}; // file -> content-hash token (from /api/media-versions)
|
||||
// Network path for a media file, content-hash-versioned so a re-baked clip's URL
|
||||
// changes with its bytes (permanent cache-bust). The blob cache, when present,
|
||||
// wins — those bytes are already the current ones for this session.
|
||||
function mediaNetUrl(file) { const v = mediaVersions[file]; return "/media/" + file + (v ? "?v=" + v : ""); }
|
||||
function mediaUrl(file) {
|
||||
if (file.startsWith("/media/")) return file; // already a resolved absolute url (audio layer)
|
||||
return mediaBlobs[file] || mediaNetUrl(file);
|
||||
}
|
||||
|
||||
// Every media file the ring can show: each scale-pool clip's base footage plus the
|
||||
// per-edge transition clips. This is the full preload set (~two dozen files).
|
||||
function preloadList() {
|
||||
const files = new Set();
|
||||
for (const c of Object.values(clipsById)) if (c && c.base_file) files.add(c.base_file);
|
||||
// Each ring edge ships both directions: the forward (zoom-in) clip and its baked
|
||||
// reverse (zoom-out) companion — cache both so up and down moves are instant.
|
||||
if (ring && ring.transitions) for (const t of ring.transitions) {
|
||||
if (t && t.file) { files.add(t.file); files.add(reverseFile(t.file)); }
|
||||
}
|
||||
return [...files];
|
||||
}
|
||||
|
||||
// Order the preload so the clips most likely to be shown next come first: the
|
||||
// current scale's pool, then outward to neighboring scales, then transitions.
|
||||
function preloadOrder() {
|
||||
if (!ring || !ring.scales) return preloadList();
|
||||
const n = ring.scales.length, seen = new Set(), ordered = [];
|
||||
const add = (f) => { if (f && !seen.has(f)) { seen.add(f); ordered.push(f); } };
|
||||
for (let d = 0; d < n; d++) {
|
||||
for (const sign of d === 0 ? [0] : [1, -1]) {
|
||||
const s = ring.scales[((ringIndex + sign * d) % n + n) % n];
|
||||
if (s && s.pool) for (const m of s.pool) add((clipsById[m.clip_id] || {}).base_file);
|
||||
else if (s) add((clipsById[s.clip_id] || {}).base_file);
|
||||
}
|
||||
}
|
||||
if (ring.transitions) for (const t of ring.transitions) { add(t.file); add(reverseFile(t.file)); }
|
||||
for (const f of preloadList()) add(f); // sweep up anything not on the ring
|
||||
return ordered;
|
||||
}
|
||||
|
||||
// Fetch every clip into the blob cache in the background — non-blocking, so the
|
||||
// first scale plays immediately while the rest stream in. Bounded concurrency keeps
|
||||
// the ~350 MB fetch from stampeding. A tiny progress chip self-removes when done.
|
||||
async function preloadAllMedia(concurrency = 4) {
|
||||
const files = preloadOrder();
|
||||
if (!files.length) return;
|
||||
let done = 0;
|
||||
const total = files.length;
|
||||
const chip = preloadChip();
|
||||
const tick = () => {
|
||||
if (chip) chip.textContent = `⬇ caching clips ${done}/${total}`;
|
||||
setLoadingProgress(done, total); // drive the "Loading Universe…" bar
|
||||
};
|
||||
tick();
|
||||
let i = 0;
|
||||
async function worker() {
|
||||
while (i < files.length) {
|
||||
const file = files[i++];
|
||||
if (!mediaBlobs[file]) {
|
||||
try {
|
||||
// The content-hash `?v=` already makes a re-baked clip's URL unique, but
|
||||
// `cache: "reload"` is belt-and-suspenders: it bypasses the HTTP cache for
|
||||
// this fetch (so even an un-versioned or immutable-pinned prior entry can't
|
||||
// serve stale) and refreshes the entry. The blob cache above still gives
|
||||
// instant in-session swaps; this only affects the one fetch per (re)load.
|
||||
const blob = await (await fetch(mediaNetUrl(file), { cache: "reload" })).blob();
|
||||
mediaBlobs[file] = URL.createObjectURL(blob);
|
||||
} catch (_) { /* leave it to the network path on demand */ }
|
||||
}
|
||||
done++; tick();
|
||||
}
|
||||
}
|
||||
await Promise.all(Array.from({ length: Math.min(concurrency, files.length) }, worker));
|
||||
if (chip) chip.remove();
|
||||
}
|
||||
|
||||
function preloadChip() {
|
||||
let c = document.getElementById("preload-chip");
|
||||
if (!c) {
|
||||
c = document.createElement("div");
|
||||
c.id = "preload-chip";
|
||||
c.style.cssText = "position:fixed;bottom:8px;right:10px;z-index:9998;background:rgba(0,0,0,.55);" +
|
||||
"color:#9fe;font:11px/1.4 monospace;padding:4px 8px;border-radius:4px;pointer-events:none;";
|
||||
document.body.appendChild(c);
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
// Load the active scale's BASE footage into the video (the painterly canvas reads
|
||||
// its frames live). The Right knob no longer swaps the source — it only drives the
|
||||
@@ -183,10 +307,12 @@ function paintLoop() {
|
||||
gl.drawArrays(gl.TRIANGLES, 0, 3);
|
||||
paint.style.filter = busy ? "none" : gradeFilter;
|
||||
}
|
||||
// Animate keyframed annotation tracks: re-place boxes against playback time.
|
||||
// Animate per-frame: re-place keyframed tracks AND re-evaluate time-windowed
|
||||
// labels (a label enters/leaves frame as the loop plays) against playback time.
|
||||
const c = activeClip();
|
||||
if (lastOverlay && lastOverlay.level > 0 && c &&
|
||||
c.annotations.some((a) => a.track && a.track.length)) {
|
||||
c.annotations.some((a) => (a.track && a.track.length) ||
|
||||
typeof a.appear === "number" || typeof a.disappear === "number")) {
|
||||
renderOverlay(lastOverlay.level, lastOverlay.intensity);
|
||||
}
|
||||
requestAnimationFrame(paintLoop);
|
||||
@@ -261,17 +387,50 @@ function loopT() {
|
||||
return vid && vid.duration ? (vid.currentTime % vid.duration) / vid.duration : 0;
|
||||
}
|
||||
|
||||
// --- Progressive labels & emotions (content-pipeline §11.3 / §11.4) ---
|
||||
const clamp = (v, lo, hi) => Math.max(lo, Math.min(hi, v));
|
||||
|
||||
// The Left level at which a label first appears. From `salience` (1..4): high
|
||||
// salience appears early (first_level = 5 - salience), so raising Left brings in
|
||||
// lower-salience objects. Falls back to a legacy flat `min_level`, else 1.
|
||||
function firstLevel(a) {
|
||||
if (typeof a.salience === "number") return clamp(5 - a.salience, 1, 4);
|
||||
if (typeof a.min_level === "number") return a.min_level;
|
||||
return 1;
|
||||
}
|
||||
|
||||
// A tiered string escalates with a knob: a LIST is indexed by tier-1 (clamped to
|
||||
// the available tiers); a plain string is static (tier ignored — back-compat).
|
||||
function pickTier(value, tier) {
|
||||
if (Array.isArray(value)) return value[clamp(tier - 1, 0, value.length - 1)];
|
||||
return value;
|
||||
}
|
||||
function tierCount(value) { return Array.isArray(value) ? value.length : 1; }
|
||||
|
||||
// Time-windowed presence (§11.2): a label with `appear`/`disappear` (loop-norm t)
|
||||
// shows only while on screen. A `disappear < appear` window wraps across the loop
|
||||
// seam (visible when t >= appear OR t <= disappear). No window => always present.
|
||||
function inWindow(a, t) {
|
||||
const hasA = typeof a.appear === "number", hasD = typeof a.disappear === "number";
|
||||
if (!hasA && !hasD) return true;
|
||||
const ap = hasA ? a.appear : 0, dis = hasD ? a.disappear : 1;
|
||||
return ap <= dis ? (t >= ap && t <= dis) : (t >= ap || t <= dis);
|
||||
}
|
||||
|
||||
function renderOverlay(level, intensity) {
|
||||
const clip = activeClip();
|
||||
overlay.innerHTML = "";
|
||||
if (!clip || level <= 0) { overlay.style.opacity = "0"; return; }
|
||||
overlay.style.opacity = String(intensity);
|
||||
const strings = (clip.strings && clip.strings.en) || {};
|
||||
const t = loopT();
|
||||
let shown = 0;
|
||||
for (const a of clip.annotations) {
|
||||
if (a.min_level > level) continue;
|
||||
const first = firstLevel(a);
|
||||
if (level < first) continue; // salience gating: low Left = fewer/general labels
|
||||
if (!inWindow(a, t)) continue; // time-windowed: only while the object is on screen
|
||||
shown++;
|
||||
const [x, y, w, h] = boxAt(a, loopT()).map((n) => n * 100);
|
||||
const [x, y, w, h] = boxAt(a, t).map((n) => n * 100);
|
||||
const measure = a.key.startsWith("measure.");
|
||||
const kind = measure ? "measure" : "detect";
|
||||
reticle(x, y, w, h, "hud-reticle " + kind, overlay);
|
||||
@@ -281,7 +440,12 @@ function renderOverlay(level, intensity) {
|
||||
svg("line", { x1: cx - 1, y1: cyc, x2: cx + 1, y2: cyc, class: "hud-reticle measure" }, overlay);
|
||||
svg("line", { x1: cx, y1: cyc - 1, x2: cx, y2: cyc + 1, class: "hud-reticle measure" }, overlay);
|
||||
}
|
||||
chip(x, y, strings[a.key] || a.key, measure ? null : confidence(a.key), kind, overlay);
|
||||
// LEFT detail tier: a detection escalates general -> scientific+fact as Left
|
||||
// rises above where it first appeared; measurements stay single-tier.
|
||||
const raw = strings[a.key];
|
||||
const tier = measure ? 1 : clamp(level - first + 1, 1, tierCount(raw));
|
||||
const label = pickTier(raw !== undefined ? raw : a.key, tier);
|
||||
chip(x, y, label, measure ? null : confidence(a.key), kind, overlay);
|
||||
}
|
||||
// Global status tag — the machine announcing it is analyzing, escalating with Left.
|
||||
// Right-anchored at the frame edge; the plate is sized from the text's real bbox
|
||||
@@ -298,11 +462,12 @@ function renderOverlay(level, intensity) {
|
||||
lastOverlay = { level, intensity };
|
||||
}
|
||||
|
||||
// Affect channel: soft, glowing emotion-words surfaced only when BOTH knobs are
|
||||
// up. `strength` = min(left, right); `intensity` is the layer opacity. Words are
|
||||
// placed at authored scene points (no boxes — feelings are scene-level) and read
|
||||
// softer than the clinical reticles — the dream leaking into the machine's read.
|
||||
function renderAffect(strength, intensity) {
|
||||
// Affect channel: soft, glowing emotion-words — a RIGHT-brain output, surfaced by
|
||||
// the Right (dreamlike) knob alone (`strength` = right; the Left analytical knob no
|
||||
// longer gates them). `intensity` is the layer opacity. Words are placed at authored
|
||||
// scene points (no boxes — feelings are scene-level) and read softer than the
|
||||
// clinical reticles — feeling, not measurement.
|
||||
function renderAffect(strength, intensity, right) {
|
||||
const clip = activeClip();
|
||||
if (!affectLayer) return; // tolerate a stale page missing the affect layer
|
||||
affectLayer.innerHTML = "";
|
||||
@@ -313,23 +478,34 @@ function renderAffect(strength, intensity) {
|
||||
if (f.min_level > strength) continue;
|
||||
const [x, y] = f.at.map((n) => n * 100);
|
||||
const t = svg("text", { x, y, "text-anchor": "middle", class: "hud-affect" }, affectLayer);
|
||||
t.textContent = strings[f.key] || f.key;
|
||||
// RIGHT emotion tier: the vocabulary escalates basic -> compound as the Right
|
||||
// knob rises (appearance is gated by the Right knob via `strength` above).
|
||||
const raw = strings[f.key];
|
||||
t.textContent = pickTier(raw !== undefined ? raw : f.key, clamp(right || 0, 1, tierCount(raw)));
|
||||
}
|
||||
}
|
||||
|
||||
function renderScaleReadout() {
|
||||
if (!ring) return;
|
||||
const s = ring.scales[ringIndex];
|
||||
$("scale-name").textContent = `${s.title} (${ringIndex + 1}/${ring.scales.length})`;
|
||||
const poolN = (s.pool && s.pool.length) || 1;
|
||||
const member = (activeClip() && activeClip().title) || s.title;
|
||||
// scale id · the chosen pool member · position on the ring (pool size if >1)
|
||||
const poolTag = poolN > 1 ? ` · pool ${poolN}` : "";
|
||||
$("scale-name").textContent = `${s.id} · ${member} (${ringIndex + 1}/${ring.scales.length}${poolTag})`;
|
||||
renderDial();
|
||||
refreshDevClip(); // keep the Dev Mode pool picker + clip data in sync with the landing
|
||||
}
|
||||
|
||||
function controls() {
|
||||
// One bipolar Mood dial (-4 dark .. 0 neutral .. +4 light) maps onto the engine's
|
||||
// two poles. The calibration gains are pinned to their locked 1.0 (full dial = full
|
||||
// effect), so the simulator sends no calibration — the server uses DEFAULT_CALIBRATION.
|
||||
// Video (on/off) and Audio (on/off) are orthogonal toggles (audio spec §2).
|
||||
// Audio on = the per-altitude soundtrack (white-noise is deferred). One bipolar
|
||||
// Mood dial (-4 dark .. +4 light) maps onto the engine's two poles. Calibration
|
||||
// gains are pinned to 1.0, so the simulator sends no calibration.
|
||||
const mood = +$("mood").value;
|
||||
return {
|
||||
content: $("content").value,
|
||||
visual: $("visual").checked ? "on" : "off",
|
||||
audio: $("audio").checked ? "soundtrack" : "off",
|
||||
left: +$("left").value, right: +$("right").value,
|
||||
dark: mood < 0 ? -mood : 0, light: mood > 0 ? mood : 0,
|
||||
volume: 2, brightness: 2,
|
||||
@@ -341,18 +517,27 @@ async function update() {
|
||||
if (busy) return;
|
||||
const resp = await fetch("/api/alteration", {
|
||||
method: "POST", headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ controls: controls() }),
|
||||
body: JSON.stringify({ controls: controls(), altitude_index: ringIndex }),
|
||||
});
|
||||
if (!resp.ok) { readout.textContent = "invalid: " + resp.status; return; }
|
||||
const data = await resp.json();
|
||||
readout.textContent = JSON.stringify(data, null, 2);
|
||||
if (!data.content.video) { black.classList.remove("hidden"); return; }
|
||||
if (!data.render.video.shown) {
|
||||
// Blank the screen. Cover with #black AND hide the video layers themselves —
|
||||
// the <video>/<canvas> are GPU-composited and can otherwise show through the
|
||||
// overlay on a real GPU (Safari/Chrome).
|
||||
black.style.opacity = "1"; black.classList.remove("hidden");
|
||||
vid.style.opacity = "0"; paint.style.opacity = "0";
|
||||
return;
|
||||
}
|
||||
black.classList.add("hidden");
|
||||
vid.style.opacity = ""; paint.style.opacity = ""; // restore the video layers
|
||||
|
||||
try {
|
||||
ensureClipMedia();
|
||||
applyVideoLook(data.plan.grade.tone, data.plan.dream.intensity);
|
||||
renderOverlay(data.plan.overlay.level, data.plan.overlay.intensity);
|
||||
renderAffect(data.plan.affect.strength, data.plan.affect.intensity);
|
||||
renderAffect(data.plan.affect.strength, data.plan.affect.intensity, data.plan.dream.strength);
|
||||
const b = document.getElementById("err-banner");
|
||||
if (b) b.remove(); // render succeeded — clear any prior error
|
||||
} catch (err) {
|
||||
@@ -367,11 +552,17 @@ function debounced() { clearTimeout(timer); timer = setTimeout(update, 80); }
|
||||
|
||||
const FAST_BLEND_RATE = 2.5; // playback speed for a collapsed fast-spin pass
|
||||
|
||||
function playTransition(file, blended) {
|
||||
// The zoom-OUT companion of an edge clip: the baked `<edge>.rev.mp4` reverse. A
|
||||
// `reversed` step (an ascending / zoom-out ring move) plays this instead of the
|
||||
// forward zoom-in clip, so going up recedes from the current scale.
|
||||
function reverseFile(file) { return file.replace(/\.mp4$/, ".rev.mp4"); }
|
||||
|
||||
function playTransition(file, blended, reversed) {
|
||||
// Play one zoom/warp transition clip start-to-finish, with the alteration
|
||||
// layers hidden. A `blended` (fast-spin) pass runs at FAST_BLEND_RATE so a
|
||||
// quick encoder spin lands fast instead of grinding through every morph.
|
||||
// Resolves on 'ended' (or a safety timeout that scales with playback rate).
|
||||
// Zoom-out moves (`reversed`) play the baked reverse clip. Resolves on 'ended'
|
||||
// (or a safety timeout that scales with playback rate).
|
||||
return new Promise((resolve) => {
|
||||
overlay.style.opacity = "0";
|
||||
affectLayer.style.opacity = "0";
|
||||
@@ -379,7 +570,7 @@ function playTransition(file, blended) {
|
||||
vid.style.filter = "none";
|
||||
vid.loop = false;
|
||||
vid.style.opacity = "1";
|
||||
vid.src = mediaUrl(file);
|
||||
vid.src = mediaUrl(reversed ? reverseFile(file) : file);
|
||||
vid.playbackRate = blended ? FAST_BLEND_RATE : 1;
|
||||
let done = false;
|
||||
const finish = () => {
|
||||
@@ -395,6 +586,29 @@ function playTransition(file, blended) {
|
||||
});
|
||||
}
|
||||
|
||||
// Briefly fade through the #black overlay while `swap` changes the displayed clip,
|
||||
// so a content swap reads as a deliberate cut, not a jarring jump. Restores #black
|
||||
// to its hidden/opaque resting state (the video-off use) afterward.
|
||||
function fadeThroughBlack(swap) {
|
||||
return new Promise((resolve) => {
|
||||
black.style.opacity = "0";
|
||||
black.classList.remove("hidden");
|
||||
void black.offsetWidth; // reflow so the 0 -> 1 transition runs
|
||||
black.style.opacity = "1";
|
||||
setTimeout(() => {
|
||||
try { swap(); } catch (_) {}
|
||||
setTimeout(() => {
|
||||
black.style.opacity = "0";
|
||||
setTimeout(() => {
|
||||
black.classList.add("hidden");
|
||||
black.style.opacity = ""; // back to CSS default for the video-off use
|
||||
resolve();
|
||||
}, 210);
|
||||
}, 160); // let the chosen clip decode under black
|
||||
}, 210);
|
||||
});
|
||||
}
|
||||
|
||||
async function advance(delta) {
|
||||
if (busy || !ring || ring.scales.length < 2) return;
|
||||
busy = true;
|
||||
@@ -408,14 +622,24 @@ async function advance(delta) {
|
||||
// A fast spin comes back collapsed to a single blended step (move.fast);
|
||||
// a slow spin chains one full transition per scale crossed.
|
||||
for (const step of move.steps) {
|
||||
await playTransition(step.file, step.blended);
|
||||
await playTransition(step.file, step.blended, step.reversed);
|
||||
}
|
||||
ringIndex = move.to_index;
|
||||
currentClipId = null; // force the target scale's base media to (re)load
|
||||
activeClipId = move.target_clip_id; // the randomly-picked pool member to load
|
||||
renderScaleReadout();
|
||||
// The baked transition lands on the scale PRIMARY; if the rotating pool picked a
|
||||
// DIFFERENT member, mask the swap behind a fade to black so it doesn't hard-cut
|
||||
// from the primary to the chosen clip (e.g. coast birdrock -> surfgrass).
|
||||
const landedPrimary = (ring.scales[move.to_index] || {}).clip_id;
|
||||
if (move.target_clip_id && move.target_clip_id !== landedPrimary) {
|
||||
await fadeThroughBlack(() => { currentClipId = null; ensureClipMedia(); });
|
||||
} else {
|
||||
currentClipId = null; // same clip the transition ended on — plain (re)load
|
||||
}
|
||||
} finally {
|
||||
busy = false;
|
||||
update();
|
||||
applyAudio(); // soundtrack follows the Altitude dial (crossfade to the new scale)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -431,6 +655,248 @@ function onWheel(e) {
|
||||
}, 90);
|
||||
}
|
||||
|
||||
// --- Altitude knob: the endless rotary encoder, drawn as a turnable dial ---
|
||||
// cosmos (highest) sits at the top; turning CLOCKWISE descends through the scales
|
||||
// (cosmos → orbit → coast → reef → abyss) and past the deepest WRAPS back to the
|
||||
// top — the same endless ring the server already models. The drag accumulates a
|
||||
// rotation and commits whole detents on release (so a big spin becomes one fast
|
||||
// blended pass, exactly like the wheel); a tap on a label jumps to that scale.
|
||||
const dial = $("dial");
|
||||
const DIAL_C = 50, DIAL_LABEL_R = 41, DIAL_BODY_R = 27;
|
||||
let needleDeg = 0;
|
||||
let dialDrag = null; // {lastAng, accum, moved} while turning
|
||||
|
||||
function dialStep() { return ring && ring.scales.length ? 360 / ring.scales.length : 360; }
|
||||
|
||||
function _xy(r, deg) {
|
||||
const a = (deg - 90) * Math.PI / 180; // -90° so 0° points UP
|
||||
return [DIAL_C + r * Math.cos(a), DIAL_C + r * Math.sin(a)];
|
||||
}
|
||||
|
||||
function buildDial() {
|
||||
if (!dial || !ring) return;
|
||||
dial.innerHTML = "";
|
||||
const n = ring.scales.length, step = 360 / n;
|
||||
svg("circle", { cx: DIAL_C, cy: DIAL_C, r: DIAL_LABEL_R + 6, class: "dial-rim" }, dial);
|
||||
svg("circle", { cx: DIAL_C, cy: DIAL_C, r: DIAL_BODY_R, class: "dial-body" }, dial);
|
||||
for (let i = 0; i < n; i++) {
|
||||
const deg = i * step;
|
||||
const [tx0, ty0] = _xy(DIAL_BODY_R - 1.5, deg);
|
||||
const [tx1, ty1] = _xy(DIAL_BODY_R + 2.5, deg);
|
||||
svg("line", { x1: tx0, y1: ty0, x2: tx1, y2: ty1, class: "dial-tick" }, dial);
|
||||
const [lx, ly] = _xy(DIAL_LABEL_R, deg);
|
||||
const t = svg("text", {
|
||||
x: lx, y: ly, "text-anchor": "middle", "dominant-baseline": "central",
|
||||
class: "dial-label", "data-index": String(i),
|
||||
}, dial);
|
||||
t.textContent = ring.scales[i].id;
|
||||
}
|
||||
svg("text", { x: DIAL_C, y: DIAL_C - 7, "text-anchor": "middle",
|
||||
"dominant-baseline": "central", class: "dial-caption" }, dial)
|
||||
.textContent = "ALTITUDE";
|
||||
const needle = svg("g", { id: "needle", class: "dial-needle" }, dial);
|
||||
const tip = DIAL_C - (DIAL_BODY_R - 4);
|
||||
svg("polygon", { points: `${DIAL_C - 2.4},${DIAL_C} ${DIAL_C + 2.4},${DIAL_C} ${DIAL_C},${tip}` }, needle);
|
||||
svg("circle", { cx: DIAL_C, cy: DIAL_C, r: 3, class: "dial-hub" }, dial);
|
||||
renderDial();
|
||||
}
|
||||
|
||||
function setNeedle(deg) {
|
||||
const needle = $("needle");
|
||||
if (needle) needle.setAttribute("transform", `rotate(${deg} ${DIAL_C} ${DIAL_C})`);
|
||||
}
|
||||
|
||||
function renderDial() {
|
||||
if (!dial || !ring) return;
|
||||
needleDeg = ringIndex * dialStep();
|
||||
setNeedle(needleDeg);
|
||||
for (const el of dial.querySelectorAll(".dial-label")) {
|
||||
el.classList.toggle("active", +el.getAttribute("data-index") === ringIndex);
|
||||
}
|
||||
}
|
||||
|
||||
function dialAngle(e) {
|
||||
const r = dial.getBoundingClientRect();
|
||||
const dx = e.clientX - (r.left + r.width / 2);
|
||||
const dy = e.clientY - (r.top + r.height / 2);
|
||||
return (Math.atan2(dx, -dy) * 180 / Math.PI + 360) % 360; // 0 at top, clockwise +
|
||||
}
|
||||
function angDelta(a, b) {
|
||||
let d = a - b;
|
||||
while (d > 180) d -= 360;
|
||||
while (d < -180) d += 360;
|
||||
return d;
|
||||
}
|
||||
|
||||
function onDialDown(e) {
|
||||
if (busy || !ring || ring.scales.length < 2) return;
|
||||
e.preventDefault();
|
||||
dialDrag = { lastAng: dialAngle(e), accum: 0, moved: 0, target: e.target };
|
||||
}
|
||||
function onDialMove(e) {
|
||||
if (!dialDrag) return;
|
||||
const a = dialAngle(e);
|
||||
const d = angDelta(a, dialDrag.lastAng);
|
||||
dialDrag.lastAng = a;
|
||||
dialDrag.accum += d;
|
||||
dialDrag.moved += Math.abs(d);
|
||||
setNeedle(ringIndex * dialStep() + dialDrag.accum); // live feedback while turning
|
||||
}
|
||||
function onDialUp(e) {
|
||||
if (!dialDrag) return;
|
||||
const { accum, moved, target } = dialDrag;
|
||||
dialDrag = null;
|
||||
if (moved < 6) { // a tap, not a turn
|
||||
if (target && target.classList && target.classList.contains("dial-label")) {
|
||||
jumpToScale(+target.getAttribute("data-index"));
|
||||
}
|
||||
renderDial();
|
||||
return;
|
||||
}
|
||||
const detents = Math.round(accum / dialStep());
|
||||
if (detents) advance(detents); else renderDial(); // snap back if it didn't cross a detent
|
||||
}
|
||||
|
||||
// Click a label → travel the SHORTEST signed way around the ring to that scale.
|
||||
function jumpToScale(idx) {
|
||||
if (!ring) return;
|
||||
const n = ring.scales.length;
|
||||
let d = (idx - ringIndex) % n;
|
||||
if (d > n / 2) d -= n;
|
||||
if (d < -n / 2) d += n;
|
||||
if (d) advance(d);
|
||||
}
|
||||
|
||||
// --- Dev Mode: pool picker + live analysis, all under one toggle ---
|
||||
// Off by default, state persisted in localStorage. Everything here is read from
|
||||
// data the renderer already has (clips, ring, the alteration response, the preload
|
||||
// cache) — no server endpoints. Editing labels stays in /author.html.
|
||||
const DEV_KEY = "hef.devMode";
|
||||
let devMode = false;
|
||||
let devStatsTimer = null;
|
||||
|
||||
const escapeHtml = (s) => String(s).replace(/[&<>"]/g,
|
||||
(c) => ({ "&": "&", "<": "<", ">": ">", '"': """ }[c]));
|
||||
|
||||
function applyDevVisibility() {
|
||||
const panel = $("dev-panel");
|
||||
if (panel) panel.classList.toggle("hidden", !devMode);
|
||||
if (devMode) { refreshDevClip(); renderDevStats(); }
|
||||
}
|
||||
|
||||
function initDev() {
|
||||
const toggle = $("dev-mode");
|
||||
if (!toggle) return;
|
||||
try { devMode = localStorage.getItem(DEV_KEY) === "1"; } catch (_) { devMode = false; }
|
||||
toggle.checked = devMode;
|
||||
toggle.addEventListener("change", () => {
|
||||
devMode = toggle.checked;
|
||||
try { localStorage.setItem(DEV_KEY, devMode ? "1" : "0"); } catch (_) {}
|
||||
applyDevVisibility();
|
||||
});
|
||||
$("pool-select").addEventListener("change", onPoolSelect);
|
||||
$("pool-reroll").addEventListener("click", reroll);
|
||||
// Cache + playback stats are live-ish; cheap to repaint twice a second when shown.
|
||||
devStatsTimer = setInterval(() => { if (devMode) renderDevStats(); }, 500);
|
||||
applyDevVisibility();
|
||||
}
|
||||
|
||||
// Rebuild the pool picker for the current scale and repaint the clip data. Cheap
|
||||
// (pools are <= a handful), so just rebuild on every landing/pick.
|
||||
function refreshDevClip() {
|
||||
if (!devMode) return;
|
||||
const sel = $("pool-select");
|
||||
const scale = ring && ring.scales[ringIndex];
|
||||
if (sel && scale) {
|
||||
const pool = (scale.pool && scale.pool.length) ? scale.pool : [{ clip_id: scale.clip_id, title: scale.title }];
|
||||
sel.innerHTML = "";
|
||||
for (const m of pool) {
|
||||
const o = document.createElement("option");
|
||||
o.value = m.clip_id;
|
||||
o.textContent = `${m.clip_id} · ${m.title || (clipsById[m.clip_id] || {}).title || ""}`;
|
||||
sel.appendChild(o);
|
||||
}
|
||||
if (activeClipId) sel.value = activeClipId;
|
||||
sel.disabled = pool.length < 2;
|
||||
}
|
||||
renderDevMeta();
|
||||
renderDevAnno();
|
||||
}
|
||||
|
||||
function renderDevMeta() {
|
||||
const dl = $("clip-meta");
|
||||
const c = activeClip();
|
||||
if (!dl) return;
|
||||
if (!c) { dl.innerHTML = "<dt>—</dt><dd>no clip</dd>"; return; }
|
||||
const rows = [["id", c.id], ["title", c.title], ["source", c.source],
|
||||
["license", c.license], ["base_file", c.base_file]];
|
||||
dl.innerHTML = rows.map(([k, v]) =>
|
||||
`<dt>${escapeHtml(k)}</dt><dd>${escapeHtml(v != null ? v : "—")}</dd>`).join("");
|
||||
}
|
||||
|
||||
function renderDevAnno() {
|
||||
const box = $("clip-anno");
|
||||
const c = activeClip();
|
||||
if (!box) return;
|
||||
const strings = (c && c.strings && c.strings.en) || {};
|
||||
const win = (a) => (typeof a.appear === "number" || typeof a.disappear === "number")
|
||||
? ` win ${(a.appear ?? 0).toFixed(2)}–${(a.disappear ?? 1).toFixed(2)}` : "";
|
||||
const rows = [];
|
||||
for (const a of (c && c.annotations) || []) {
|
||||
const measure = a.key.startsWith("measure.");
|
||||
const sal = typeof a.salience === "number" ? `sal ${a.salience}`
|
||||
: typeof a.min_level === "number" ? `min L${a.min_level}` : "sal 1";
|
||||
const tiers = tierCount(strings[a.key]);
|
||||
rows.push(`<div class="anno-row"><span class="anno-key ${measure ? "measure" : ""}">${escapeHtml(a.key)}</span>` +
|
||||
`<span class="anno-meta"> · ${sal} · ${tiers} tier${tiers > 1 ? "s" : ""}${win(a)}</span></div>`);
|
||||
}
|
||||
for (const f of (c && c.affect) || []) {
|
||||
const tiers = tierCount(strings[f.key]);
|
||||
rows.push(`<div class="anno-row"><span class="anno-key affect">${escapeHtml(f.key)}</span>` +
|
||||
`<span class="anno-meta"> · min L${f.min_level ?? 1} · ${tiers} tier${tiers > 1 ? "s" : ""}</span></div>`);
|
||||
}
|
||||
box.innerHTML = rows.length ? rows.join("") : `<div class="anno-empty">no annotations or affect</div>`;
|
||||
}
|
||||
|
||||
function renderDevStats() {
|
||||
const dl = $("dev-stats");
|
||||
if (!dl) return;
|
||||
const c = activeClip();
|
||||
const total = preloadList().length;
|
||||
const cached = Object.keys(mediaBlobs).length;
|
||||
const fromMem = !!(c && mediaBlobs[c.base_file]);
|
||||
const res = vid.videoWidth ? `${vid.videoWidth}×${vid.videoHeight}` : "—";
|
||||
const dur = vid.duration ? `${vid.duration.toFixed(1)}s` : "—";
|
||||
const loop = vid.duration ? `${Math.round(loopT() * 100)}%` : "—";
|
||||
dl.innerHTML =
|
||||
`<dt>cache</dt><dd>${cached}/${total} cached</dd>` +
|
||||
`<dt>active source</dt><dd class="${fromMem ? "mem" : "net"}">${fromMem ? "memory (blob)" : "network"}</dd>` +
|
||||
`<dt>resolution</dt><dd>${res}</dd>` +
|
||||
`<dt>duration</dt><dd>${dur}</dd>` +
|
||||
`<dt>loop pos</dt><dd>${loop}</dd>`;
|
||||
}
|
||||
|
||||
// Pick a specific pool member (Dev Mode override of the random landing pick).
|
||||
function onPoolSelect() {
|
||||
const id = $("pool-select").value;
|
||||
if (!id || id === activeClipId) return;
|
||||
activeClipId = id;
|
||||
currentClipId = null; // force the base media to reload
|
||||
renderScaleReadout(); // refresh scale name + dev clip data
|
||||
update(); // reload media + plan
|
||||
}
|
||||
|
||||
// Re-roll: a fresh server-canonical random pick for the current scale.
|
||||
async function reroll() {
|
||||
if (busy) return;
|
||||
const id = await pickRandomMember();
|
||||
if (!id) return;
|
||||
activeClipId = id;
|
||||
currentClipId = null;
|
||||
renderScaleReadout();
|
||||
update();
|
||||
}
|
||||
|
||||
// Dev live-reload: poll the asset version and reload when it changes, so an open
|
||||
// tab never keeps running a stale renderer while we iterate (the readout updates
|
||||
// from the live API and masks it otherwise). Reloads on my edits AND on a server
|
||||
@@ -446,17 +912,129 @@ function devLiveReload() {
|
||||
}, 1000);
|
||||
}
|
||||
|
||||
// --- Audio layer: ONE <audio> element, started in-gesture (Safari-safe) ---
|
||||
// Real Safari/iOS only "unlocks" an <audio> element when play() is called
|
||||
// SYNCHRONOUSLY inside a user gesture (Chrome is lenient; Safari is not). So we use
|
||||
// a single element and start it directly from the Video/Audio toggle's click — not
|
||||
// via the async server roundtrip; once unlocked it can be re-played programmatically
|
||||
// (e.g. on an altitude change). audio on = the current scale's soundtrack.
|
||||
const aud = $("aud");
|
||||
let audUrl = null; // soundtrack url currently loaded (null = silent)
|
||||
let audLastErr = ""; // last play() rejection / element error (for the readout)
|
||||
const FADE_MS = 500;
|
||||
|
||||
// --- Live audio status readout (diagnostic) — so a "no sound" report is legible ---
|
||||
const AUD_ERR = ["", "aborted", "network", "decode", "format-not-supported"];
|
||||
function audioStatusText() {
|
||||
if (!$("audio").checked) return "audio: off";
|
||||
if (aud.error) return `audio: MEDIA ERROR ${aud.error.code} (${AUD_ERR[aud.error.code] || "?"})`;
|
||||
if (audLastErr) return "audio: " + audLastErr;
|
||||
if (aud.paused) return `audio: on, PAUSED (readyState ${aud.readyState})`;
|
||||
return `audio: ▶ playing ${aud.currentTime.toFixed(1)}s · vol ${Math.round(aud.volume * 100)}% · rs ${aud.readyState}`;
|
||||
}
|
||||
function updateAudioStatus() {
|
||||
const el = $("audio-status");
|
||||
if (!el) return;
|
||||
el.textContent = audioStatusText();
|
||||
const playing = $("audio").checked && !aud.paused && !aud.error && !audLastErr;
|
||||
el.classList.toggle("playing", playing);
|
||||
el.classList.toggle("blocked", !!aud.error || !!audLastErr);
|
||||
}
|
||||
aud.addEventListener("error", () => { audLastErr = ""; updateAudioStatus(); });
|
||||
setInterval(updateAudioStatus, 400);
|
||||
|
||||
function fadeVolume(el, to, ms, done) {
|
||||
// setInterval (not requestAnimationFrame): rAF throttles/pauses when the tab is
|
||||
// backgrounded, which would stall the gain ramp; a timer fires regardless.
|
||||
const from = el.volume, steps = Math.max(1, Math.round(ms / 30));
|
||||
let i = 0;
|
||||
const iv = setInterval(() => {
|
||||
i += 1;
|
||||
const k = Math.min(1, i / steps);
|
||||
el.volume = Math.min(1, Math.max(0, from + (to - from) * k));
|
||||
if (k >= 1) { clearInterval(iv); if (done) done(); }
|
||||
}, 30);
|
||||
}
|
||||
|
||||
// The current altitude's soundtrack url (the ring carries each scale's `audio`).
|
||||
function soundtrackUrl() {
|
||||
const s = ring && ring.scales[ringIndex];
|
||||
return s && s.audio ? "/media/audio/" + s.audio : null;
|
||||
}
|
||||
|
||||
// Load `url` into the single element and fade it in. Call this SYNCHRONOUSLY from
|
||||
// the toggle's click the first time so play() is allowed; later calls reuse the
|
||||
// now-unlocked element.
|
||||
function playUrl(url) {
|
||||
aud.src = mediaUrl(url);
|
||||
aud.volume = 0;
|
||||
audLastErr = "";
|
||||
const pr = aud.play();
|
||||
if (pr) {
|
||||
pr.then(() => { audLastErr = ""; updateAudioStatus(); })
|
||||
.catch((e) => { audLastErr = "play BLOCKED: " + ((e && e.name) || e); updateAudioStatus(); });
|
||||
}
|
||||
fadeVolume(aud, 1, FADE_MS);
|
||||
}
|
||||
|
||||
// Reconcile audio to the Audio toggle + current altitude: on → the scale's
|
||||
// soundtrack, off → silence. Idempotent on an unchanged url (so a same-scale pool
|
||||
// re-roll doesn't restart it). MUST be called synchronously from the toggle gesture
|
||||
// the first time it plays.
|
||||
function applyAudio() {
|
||||
const url = $("audio").checked ? soundtrackUrl() : null;
|
||||
if (url === audUrl) return;
|
||||
audUrl = url;
|
||||
if (!url) { fadeVolume(aud, 0, FADE_MS, () => aud.pause()); return; }
|
||||
if (aud.paused || !aud.src) playUrl(url); // start fresh
|
||||
else fadeVolume(aud, 0, 250, () => playUrl(url)); // dip out, then swap + fade in
|
||||
}
|
||||
|
||||
// "Loading Universe…" splash — hidden once media is preloaded; reflects progress.
|
||||
function setLoadingProgress(done, total) {
|
||||
const fill = document.getElementById("loading-fill");
|
||||
if (fill && total) fill.style.width = Math.round((done / total) * 100) + "%";
|
||||
}
|
||||
function hideLoading() {
|
||||
const el = document.getElementById("loading");
|
||||
if (!el) return;
|
||||
el.classList.add("done");
|
||||
setTimeout(() => el.remove(), 700);
|
||||
}
|
||||
|
||||
let videoEverOn = false; // has Video ever been switched on this session?
|
||||
|
||||
async function main() {
|
||||
devLiveReload();
|
||||
try { initPaint(); } catch (e) { paintOK = false; paint.style.display = "none"; showError("WebGL init: " + e.message); }
|
||||
await loadData();
|
||||
await landScale(); // pick the initial scale's pool member before first render
|
||||
buildDial(); // draw the altitude knob from the ring's scales
|
||||
initDev(); // wire the Dev Mode toggle + pool picker (reads persisted state)
|
||||
renderScaleReadout();
|
||||
for (const id of ["content", "left", "right", "mood"]) {
|
||||
$(id).addEventListener("input", debounced);
|
||||
}
|
||||
$("zoom-in").addEventListener("click", () => advance(1));
|
||||
$("zoom-out").addEventListener("click", () => advance(-1));
|
||||
// Sliders stream on "input"; the toggles fire on "change".
|
||||
for (const id of ["left", "right", "mood"]) $(id).addEventListener("input", debounced);
|
||||
// Audio toggle: start/stop the soundtrack SYNCHRONOUSLY in this gesture so it
|
||||
// unlocks + plays on Safari (which blocks play() outside a user gesture).
|
||||
$("audio").addEventListener("change", () => { applyAudio(); debounced(); });
|
||||
// Video toggle: the FIRST time video turns on, bring audio with it (one flip = the
|
||||
// full experience). Audio toggled on its own first stays audio-only. The audio is
|
||||
// set + played in THIS gesture so it unlocks on Safari.
|
||||
$("visual").addEventListener("change", () => {
|
||||
if ($("visual").checked && !videoEverOn) {
|
||||
videoEverOn = true;
|
||||
if (!$("audio").checked) { $("audio").checked = true; applyAudio(); }
|
||||
}
|
||||
debounced();
|
||||
});
|
||||
// Altitude knob: drag to turn (commit detents on release), scroll to step, tap a label to jump.
|
||||
dial.addEventListener("pointerdown", onDialDown);
|
||||
window.addEventListener("pointermove", onDialMove);
|
||||
window.addEventListener("pointerup", onDialUp);
|
||||
dial.addEventListener("wheel", onWheel, { passive: false });
|
||||
$("stage").addEventListener("wheel", onWheel, { passive: false });
|
||||
update();
|
||||
update(); // render the initial state (both toggles off → black, silent)
|
||||
await preloadAllMedia(); // download all media, updating the loading bar
|
||||
hideLoading(); // experience is ready to run smoothly
|
||||
}
|
||||
main();
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
/* Author-mode-only styling (content-pipeline §11.5). Reuses style.css for the
|
||||
shared stage/panel chrome; this adds the box-draw layer + editor widgets. */
|
||||
|
||||
.author #draw {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
cursor: crosshair;
|
||||
}
|
||||
.author #draw .seed-box {
|
||||
fill: rgba(80, 200, 255, 0.12);
|
||||
stroke: #50c8ff;
|
||||
stroke-width: 0.4;
|
||||
vector-effect: non-scaling-stroke;
|
||||
}
|
||||
.author #draw .track-box {
|
||||
fill: rgba(120, 255, 160, 0.10);
|
||||
stroke: #78ffa0;
|
||||
stroke-width: 0.4;
|
||||
stroke-dasharray: 1.5 1;
|
||||
vector-effect: non-scaling-stroke;
|
||||
}
|
||||
.author #draw .affect-dot {
|
||||
fill: rgba(190, 150, 255, 0.85);
|
||||
stroke: #fff;
|
||||
stroke-width: 0.25;
|
||||
vector-effect: non-scaling-stroke;
|
||||
}
|
||||
|
||||
.author .scrub {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-top: 8px;
|
||||
}
|
||||
.author .scrub input[type="range"] { flex: 1; }
|
||||
.author .mono { font-family: ui-monospace, monospace; font-size: 12px; opacity: 0.85; }
|
||||
|
||||
.author .row { display: flex; gap: 8px; margin-top: 6px; flex-wrap: wrap; }
|
||||
.author fieldset label { display: block; margin: 4px 0; font-size: 13px; }
|
||||
.author fieldset input[type="text"],
|
||||
.author fieldset input[type="number"] { width: 100%; box-sizing: border-box; }
|
||||
.author fieldset input[type="number"] { width: 5em; }
|
||||
|
||||
.author ul.authored { list-style: none; padding: 0; margin: 6px 0; }
|
||||
.author ul.authored li {
|
||||
font-size: 12px;
|
||||
padding: 3px 6px;
|
||||
margin: 2px 0;
|
||||
background: rgba(255, 255, 255, 0.05);
|
||||
border-radius: 3px;
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
gap: 6px;
|
||||
}
|
||||
.author ul.authored .rm {
|
||||
background: none;
|
||||
border: none;
|
||||
color: #f88;
|
||||
cursor: pointer;
|
||||
font-size: 12px;
|
||||
}
|
||||
.author header .hint { font-size: 12px; opacity: 0.8; max-width: 70ch; }
|
||||
@@ -0,0 +1,79 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>HEF — Label Author Mode</title>
|
||||
<link rel="stylesheet" href="/style.css" />
|
||||
<link rel="stylesheet" href="/author.css" />
|
||||
</head>
|
||||
<body class="author">
|
||||
<header><h1>HEF — Label Author Mode</h1>
|
||||
<p class="hint">Scrub a pool clip · drag a box on a subject · run the tracker · author the tiers · save to the manifest. Geometry is propagated; <strong>semantics are hand-authored</strong> (content-pipeline §11.5). <a href="/">← back to preview</a></p>
|
||||
</header>
|
||||
<main>
|
||||
<section class="stage" id="stage">
|
||||
<div class="screen">
|
||||
<video id="vid" muted playsinline></video>
|
||||
<svg id="draw" viewBox="0 0 100 100" preserveAspectRatio="none"></svg>
|
||||
</div>
|
||||
<div class="scrub">
|
||||
<button type="button" id="play">▶/⏸</button>
|
||||
<input type="range" id="seek" min="0" max="1000" value="0" />
|
||||
<span id="time" class="mono">0.00</span>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="panel">
|
||||
<fieldset>
|
||||
<legend>Clip</legend>
|
||||
<select id="clip"></select>
|
||||
<p class="hint" id="clip-meta">—</p>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Seed box (drag on the video)</legend>
|
||||
<p class="mono" id="seed-readout">no box — drag on the stage</p>
|
||||
<label>label key <input type="text" id="ann-key" placeholder="detected.jelly" /></label>
|
||||
<label>salience (1–4; 4 = shows first/at low Left)
|
||||
<input type="number" id="ann-salience" min="1" max="4" value="4" /></label>
|
||||
<label>tier 1 — general <input type="text" id="t1" placeholder="jelly" /></label>
|
||||
<label>tier 2 — specific <input type="text" id="t2" placeholder="comb jelly" /></label>
|
||||
<label>tier 3 — scientific <input type="text" id="t3" placeholder="Ctenophora" /></label>
|
||||
<label>tier 4 — +fact <input type="text" id="t4" placeholder="Ctenophora · beats rows of cilia" /></label>
|
||||
<label>keyframes <input type="number" id="ann-keyframes" min="2" max="20" value="5" /></label>
|
||||
<div class="row">
|
||||
<button type="button" id="run-track">Run tracker → track</button>
|
||||
<button type="button" id="add-static">Add as static box</button>
|
||||
</div>
|
||||
<p class="mono" id="track-readout"></p>
|
||||
<button type="button" id="add-ann" disabled>+ Add label</button>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Affect anchor (click the stage to place)</legend>
|
||||
<p class="mono" id="affect-readout">no point — shift-click the stage</p>
|
||||
<label>feel key <input type="text" id="aff-key" placeholder="feel.unease" /></label>
|
||||
<label>min strength (1–4) <input type="number" id="aff-min" min="1" max="4" value="1" /></label>
|
||||
<label>tier 1 — basic <input type="text" id="a1" placeholder="uh" /></label>
|
||||
<label>tier 2 <input type="text" id="a2" placeholder="unease" /></label>
|
||||
<label>tier 3 <input type="text" id="a3" placeholder="disquiet" /></label>
|
||||
<label>tier 4 — compound <input type="text" id="a4" placeholder="a creeping disquiet" /></label>
|
||||
<button type="button" id="add-aff" disabled>+ Add emotion</button>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Authored for this clip</legend>
|
||||
<ul id="ann-list" class="authored"></ul>
|
||||
<ul id="aff-list" class="authored"></ul>
|
||||
<div class="row">
|
||||
<button type="button" id="load-existing">Load existing</button>
|
||||
<button type="button" id="save">Save to manifest</button>
|
||||
</div>
|
||||
<p class="mono" id="save-readout"></p>
|
||||
</fieldset>
|
||||
</section>
|
||||
</main>
|
||||
<script src="/author.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,254 @@
|
||||
// Label author mode (content-pipeline §11.5): scrub a pool clip, drag a seed box
|
||||
// on a subject, run the classical tracker to propagate a keyframed track, author
|
||||
// the LEFT detail tiers + RIGHT emotion tiers, and save the entry to the manifest.
|
||||
// Geometry comes from the tracker; SEMANTICS (keys, strings, tiers) are hand-typed.
|
||||
const $ = (id) => document.getElementById(id);
|
||||
const vid = $("vid"), draw = $("draw");
|
||||
const SVGNS = "http://www.w3.org/2000/svg";
|
||||
|
||||
let clips = [];
|
||||
let currentClip = null;
|
||||
let seedBox = null; // [x,y,w,h] normalized, from a drag
|
||||
let pendingTrack = null; // {track, appear, disappear} from the tracker (or static)
|
||||
let affectPoint = null; // [x,y] normalized, from a shift-click
|
||||
let annotations = []; // authored annotation dicts (carry _tiers)
|
||||
let affect = []; // authored affect dicts (carry _tiers)
|
||||
|
||||
function svg(tag, attrs, parent) {
|
||||
const el = document.createElementNS(SVGNS, tag);
|
||||
for (const k in attrs) el.setAttribute(k, attrs[k]);
|
||||
if (parent) parent.appendChild(el);
|
||||
return el;
|
||||
}
|
||||
const mono = (n) => Number(n).toFixed(3);
|
||||
|
||||
async function loadClips() {
|
||||
clips = (await (await fetch("/api/clips")).json()).clips || [];
|
||||
const sel = $("clip");
|
||||
sel.innerHTML = "";
|
||||
for (const c of clips) {
|
||||
const o = document.createElement("option");
|
||||
o.value = c.id; o.textContent = c.id;
|
||||
sel.appendChild(o);
|
||||
}
|
||||
if (clips.length) selectClip(clips[0].id);
|
||||
}
|
||||
|
||||
function selectClip(id) {
|
||||
currentClip = clips.find((c) => c.id === id) || null;
|
||||
if (!currentClip) return;
|
||||
$("clip-meta").textContent = `${currentClip.title} — ${currentClip.license}`;
|
||||
vid.src = "/media/" + currentClip.base_file;
|
||||
vid.currentTime = 0;
|
||||
annotations = []; affect = []; seedBox = null; pendingTrack = null; affectPoint = null;
|
||||
renderLists(); renderDrawLayer();
|
||||
$("seed-readout").textContent = "no box — drag on the stage";
|
||||
$("track-readout").textContent = "";
|
||||
$("affect-readout").textContent = "no point — shift-click the stage";
|
||||
}
|
||||
|
||||
// --- scrub ---
|
||||
function seekT() { return vid.duration ? vid.currentTime / vid.duration : 0; }
|
||||
$("play").addEventListener("click", () => { vid.paused ? vid.play() : vid.pause(); });
|
||||
$("seek").addEventListener("input", () => {
|
||||
if (vid.duration) vid.currentTime = (+$("seek").value / 1000) * vid.duration;
|
||||
});
|
||||
vid.addEventListener("timeupdate", () => {
|
||||
if (!vid.duration) return;
|
||||
$("seek").value = String(Math.round(seekT() * 1000));
|
||||
$("time").textContent = seekT().toFixed(3);
|
||||
});
|
||||
|
||||
// --- draw a seed box / place an affect point on the stage ---
|
||||
function evtNorm(e) {
|
||||
const r = draw.getBoundingClientRect();
|
||||
return [(e.clientX - r.left) / r.width, (e.clientY - r.top) / r.height];
|
||||
}
|
||||
let dragStart = null;
|
||||
draw.addEventListener("mousedown", (e) => {
|
||||
if (e.shiftKey) { // shift-click = affect anchor
|
||||
affectPoint = evtNorm(e).map((v) => Math.min(Math.max(v, 0), 1));
|
||||
$("affect-readout").textContent = `affect @ ${affectPoint.map(mono).join(", ")}`;
|
||||
$("add-aff").disabled = false;
|
||||
renderDrawLayer();
|
||||
return;
|
||||
}
|
||||
dragStart = evtNorm(e);
|
||||
});
|
||||
draw.addEventListener("mousemove", (e) => {
|
||||
if (!dragStart) return;
|
||||
seedBox = boxFrom(dragStart, evtNorm(e));
|
||||
renderDrawLayer();
|
||||
});
|
||||
window.addEventListener("mouseup", (e) => {
|
||||
if (!dragStart) return;
|
||||
seedBox = boxFrom(dragStart, evtNorm(e));
|
||||
dragStart = null;
|
||||
pendingTrack = null;
|
||||
$("seed-readout").textContent = `seed [${seedBox.map(mono).join(", ")}] @ t=${seekT().toFixed(3)}`;
|
||||
$("add-ann").disabled = false;
|
||||
renderDrawLayer();
|
||||
});
|
||||
function boxFrom(a, b) {
|
||||
const x = Math.min(a[0], b[0]), y = Math.min(a[1], b[1]);
|
||||
const w = Math.abs(b[0] - a[0]), h = Math.abs(b[1] - a[1]);
|
||||
const cl = (v) => Math.min(Math.max(v, 0), 1);
|
||||
return [cl(x), cl(y), Math.min(w, 1 - cl(x)), Math.min(h, 1 - cl(y))];
|
||||
}
|
||||
|
||||
function renderDrawLayer() {
|
||||
draw.innerHTML = "";
|
||||
if (seedBox) {
|
||||
const [x, y, w, h] = seedBox.map((n) => n * 100);
|
||||
svg("rect", { x, y, width: w, height: h, class: "seed-box" }, draw);
|
||||
}
|
||||
if (pendingTrack && pendingTrack.track && vid.duration) {
|
||||
const b = boxAt(pendingTrack.track, seekT());
|
||||
if (b) {
|
||||
const [x, y, w, h] = b.map((n) => n * 100);
|
||||
svg("rect", { x, y, width: w, height: h, class: "track-box" }, draw);
|
||||
}
|
||||
}
|
||||
if (affectPoint) {
|
||||
const [x, y] = affectPoint.map((n) => n * 100);
|
||||
svg("circle", { cx: x, cy: y, r: 1.2, class: "affect-dot" }, draw);
|
||||
}
|
||||
}
|
||||
// Interpolate a track box at loop-normalized t (mirrors the preview renderer).
|
||||
function boxAt(track, t) {
|
||||
if (!track || !track.length) return null;
|
||||
if (t <= track[0].t) return track[0].box;
|
||||
if (t >= track[track.length - 1].t) return track[track.length - 1].box;
|
||||
for (let i = 1; i < track.length; i++) {
|
||||
if (t <= track[i].t) {
|
||||
const a = track[i - 1], b = track[i], f = (t - a.t) / (b.t - a.t);
|
||||
return a.box.map((v, j) => v + (b.box[j] - v) * f);
|
||||
}
|
||||
}
|
||||
return track[track.length - 1].box;
|
||||
}
|
||||
// Animate the pending track preview while playing.
|
||||
function previewLoop() { renderDrawLayer(); requestAnimationFrame(previewLoop); }
|
||||
|
||||
// --- run the tracker on the seed box ---
|
||||
$("run-track").addEventListener("click", async () => {
|
||||
if (!seedBox || !currentClip) { $("track-readout").textContent = "draw a seed box first"; return; }
|
||||
const key = $("ann-key").value.trim() || "detected.object";
|
||||
$("track-readout").textContent = "tracking…";
|
||||
const resp = await fetch("/api/author/track", {
|
||||
method: "POST", headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
clip_id: currentClip.id, key, seed_box: seedBox, seed_t: seekT(),
|
||||
salience: +$("ann-salience").value, n_keyframes: +$("ann-keyframes").value,
|
||||
}),
|
||||
});
|
||||
if (!resp.ok) { $("track-readout").textContent = "tracker error " + resp.status; return; }
|
||||
const ann = await resp.json();
|
||||
pendingTrack = { track: ann.track, appear: ann.appear, disappear: ann.disappear };
|
||||
$("track-readout").textContent =
|
||||
`track: ${ann.track.length} keyframes · window ${mono(ann.appear)}–${mono(ann.disappear)}`;
|
||||
$("add-ann").disabled = false;
|
||||
});
|
||||
|
||||
// Add the current seed as a STATIC (untracked) box at the scrub position.
|
||||
$("add-static").addEventListener("click", () => {
|
||||
if (!seedBox) return;
|
||||
pendingTrack = { static: true, box: seedBox.slice() };
|
||||
$("track-readout").textContent = "static box (no track) at the drawn position";
|
||||
$("add-ann").disabled = false;
|
||||
});
|
||||
|
||||
function tiers(ids) {
|
||||
const vals = ids.map((id) => $(id).value.trim());
|
||||
while (vals.length && !vals[vals.length - 1]) vals.pop(); // drop empty trailing tiers
|
||||
return vals.length ? vals : null;
|
||||
}
|
||||
|
||||
$("add-ann").addEventListener("click", () => {
|
||||
const key = $("ann-key").value.trim();
|
||||
if (!key) { $("track-readout").textContent = "a label key is required"; return; }
|
||||
const t = tiers(["t1", "t2", "t3", "t4"]);
|
||||
const ann = { key, salience: +$("ann-salience").value, _tiers: t || key };
|
||||
if (pendingTrack && pendingTrack.track) {
|
||||
ann.track = pendingTrack.track;
|
||||
ann.appear = pendingTrack.appear; ann.disappear = pendingTrack.disappear;
|
||||
} else if (pendingTrack && pendingTrack.static) {
|
||||
ann.box = pendingTrack.box;
|
||||
} else if (seedBox) {
|
||||
ann.box = seedBox.slice();
|
||||
} else { $("track-readout").textContent = "draw + (optionally) track a box first"; return; }
|
||||
annotations.push(ann);
|
||||
seedBox = null; pendingTrack = null; $("add-ann").disabled = true;
|
||||
renderLists(); renderDrawLayer();
|
||||
});
|
||||
|
||||
$("add-aff").addEventListener("click", () => {
|
||||
const key = $("aff-key").value.trim();
|
||||
if (!key || !affectPoint) return;
|
||||
const t = tiers(["a1", "a2", "a3", "a4"]);
|
||||
affect.push({ key, at: affectPoint.slice(), min_level: +$("aff-min").value, _tiers: t || key });
|
||||
affectPoint = null; $("add-aff").disabled = true;
|
||||
renderLists(); renderDrawLayer();
|
||||
});
|
||||
|
||||
function renderLists() {
|
||||
const al = $("ann-list"); al.innerHTML = "";
|
||||
annotations.forEach((a, i) => {
|
||||
const li = document.createElement("li");
|
||||
const kind = a.track ? `track ${a.track.length}kf ${mono(a.appear)}–${mono(a.disappear)}` : "static";
|
||||
const tip = Array.isArray(a._tiers) ? a._tiers.join(" → ") : a._tiers;
|
||||
li.textContent = `${a.key} (s${a.salience}, ${kind}) — ${tip}`;
|
||||
li.appendChild(rm(() => { annotations.splice(i, 1); renderLists(); }));
|
||||
al.appendChild(li);
|
||||
});
|
||||
const fl = $("aff-list"); fl.innerHTML = "";
|
||||
affect.forEach((f, i) => {
|
||||
const li = document.createElement("li");
|
||||
const tip = Array.isArray(f._tiers) ? f._tiers.join(" → ") : f._tiers;
|
||||
li.textContent = `${f.key} (min ${f.min_level}) — ${tip}`;
|
||||
li.appendChild(rm(() => { affect.splice(i, 1); renderLists(); }));
|
||||
fl.appendChild(li);
|
||||
});
|
||||
}
|
||||
function rm(fn) {
|
||||
const b = document.createElement("button");
|
||||
b.textContent = "✕"; b.className = "rm"; b.onclick = fn;
|
||||
return b;
|
||||
}
|
||||
|
||||
// Load whatever is already authored for this clip into the editor.
|
||||
$("load-existing").addEventListener("click", () => {
|
||||
if (!currentClip) return;
|
||||
const strings = (currentClip.strings && currentClip.strings.en) || {};
|
||||
annotations = (currentClip.annotations || []).map((a) => ({ ...a, _tiers: strings[a.key] || a.key }));
|
||||
affect = (currentClip.affect || []).map((f) => ({ ...f, _tiers: strings[f.key] || f.key }));
|
||||
renderLists();
|
||||
});
|
||||
|
||||
// Assemble strings.en from each entry's _tiers and POST to the manifest.
|
||||
$("save").addEventListener("click", async () => {
|
||||
if (!currentClip) return;
|
||||
const en = {};
|
||||
const strip = (x) => { const { _tiers, ...rest } = x; return rest; };
|
||||
const anns = annotations.map((a) => { en[a.key] = a._tiers; return strip(a); });
|
||||
const affs = affect.map((f) => { en[f.key] = f._tiers; return strip(f); });
|
||||
const resp = await fetch("/api/author/clip", {
|
||||
method: "POST", headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ clip_id: currentClip.id, annotations: anns, affect: affs, strings: { en } }),
|
||||
});
|
||||
$("save-readout").textContent = resp.ok
|
||||
? `saved ${anns.length} labels + ${affs.length} emotions to the manifest ✓`
|
||||
: "save error " + resp.status;
|
||||
if (resp.ok) { // refresh the in-memory clip copy
|
||||
clips = (await (await fetch("/api/clips")).json()).clips || [];
|
||||
currentClip = clips.find((c) => c.id === currentClip.id) || currentClip;
|
||||
}
|
||||
});
|
||||
|
||||
$("clip").addEventListener("change", (e) => selectClip(e.target.value));
|
||||
|
||||
async function main() {
|
||||
await loadClips();
|
||||
requestAnimationFrame(previewLoop);
|
||||
}
|
||||
main();
|
||||
+71
-20
@@ -7,11 +7,18 @@
|
||||
<link rel="stylesheet" href="/style.css" />
|
||||
</head>
|
||||
<body>
|
||||
<div id="loading">
|
||||
<div class="loading-inner">
|
||||
<div class="loading-title">Loading Universe<span class="loading-dots"></span></div>
|
||||
<div class="loading-bar"><div id="loading-fill"></div></div>
|
||||
</div>
|
||||
</div>
|
||||
<header><h1>Human Experience Filter — Alteration Preview</h1></header>
|
||||
<main>
|
||||
<section class="stage" id="stage">
|
||||
<div class="screen">
|
||||
<video id="vid" loop muted playsinline></video>
|
||||
<audio id="aud" loop preload="auto"></audio>
|
||||
<canvas id="paint"></canvas>
|
||||
<div id="tint"></div>
|
||||
<svg id="overlay" viewBox="0 0 100 100" preserveAspectRatio="none"></svg>
|
||||
@@ -22,26 +29,35 @@
|
||||
|
||||
<section class="panel">
|
||||
<fieldset>
|
||||
<legend>Content dial</legend>
|
||||
<select id="content">
|
||||
<option value="video">video</option>
|
||||
<option value="audio_video">audio + video</option>
|
||||
<option value="music_video">music + video</option>
|
||||
<option value="off">off (black)</option>
|
||||
<option value="white_noise">white noise (no video)</option>
|
||||
<option value="music">music (no video)</option>
|
||||
<option value="audio_track">audio track (no video)</option>
|
||||
</select>
|
||||
<legend>Output</legend>
|
||||
<label class="dev-switch" for="visual">
|
||||
<input type="checkbox" id="visual" />
|
||||
<span class="dev-switch-track"><span class="dev-switch-thumb"></span></span>
|
||||
<span class="dev-switch-label">Video</span>
|
||||
</label>
|
||||
<label class="dev-switch" for="audio">
|
||||
<input type="checkbox" id="audio" />
|
||||
<span class="dev-switch-track"><span class="dev-switch-thumb"></span></span>
|
||||
<span class="dev-switch-label">Audio</span>
|
||||
</label>
|
||||
<div id="audio-status" class="audio-status">audio: off</div>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Scale ring (endless encoder)</legend>
|
||||
<div class="ring-control">
|
||||
<button type="button" id="zoom-out" title="zoom out (toward cosmos)">⊖ out</button>
|
||||
<span id="scale-name" class="scale-name">—</span>
|
||||
<button type="button" id="zoom-in" title="zoom in (toward microscopic)">in ⊕</button>
|
||||
<legend>Audio check (diagnostic)</legend>
|
||||
<p class="hint">Press play below to test the soundtrack file directly in the
|
||||
browser's own player — this bypasses all app code. If you hear it here but
|
||||
not from the Audio toggle, the file is fine and it's a code/autoplay issue.</p>
|
||||
<audio id="aud-test" controls preload="auto" src="/media/audio/cosmos/pillars.loop.mp3" style="width:100%"></audio>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Altitude</legend>
|
||||
<div class="dial-wrap">
|
||||
<svg id="dial" viewBox="0 0 100 100" aria-label="Altitude knob (turn to change scale)"></svg>
|
||||
</div>
|
||||
<p class="hint">Relative, endless — wraps past the smallest back to the largest. Scroll the stage too.</p>
|
||||
<span id="scale-name" class="scale-name">—</span>
|
||||
<p class="hint">Turn the knob (drag it, or scroll) to change altitude — endless: past the deepest it wraps back up to the highest. Click a label to jump there.</p>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
@@ -51,11 +67,46 @@
|
||||
<label>Mood — dark ◀ 0 ▶ light <input type="range" id="mood" min="-4" max="4" value="0" /></label>
|
||||
</fieldset>
|
||||
|
||||
<!-- Dev Mode: a single toggle; everything dev (pool picker + analysis) lives below it. -->
|
||||
<label class="dev-switch" for="dev-mode">
|
||||
<input type="checkbox" id="dev-mode" />
|
||||
<span class="dev-switch-track"><span class="dev-switch-thumb"></span></span>
|
||||
<span class="dev-switch-label">Dev Mode</span>
|
||||
</label>
|
||||
|
||||
<fieldset>
|
||||
<legend>RenderPlan readout</legend>
|
||||
<pre id="readout">—</pre>
|
||||
</fieldset>
|
||||
<div id="dev-panel" class="dev-panel hidden">
|
||||
<fieldset>
|
||||
<legend>Pool — pick a clip</legend>
|
||||
<select id="pool-select" aria-label="Choose a clip from this altitude's pool"></select>
|
||||
<button type="button" id="pool-reroll" class="dev-btn">🎲 Re-roll random</button>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Active clip</legend>
|
||||
<dl id="clip-meta" class="dev-dl"></dl>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Annotations & affect</legend>
|
||||
<div id="clip-anno" class="dev-anno"></div>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>RenderPlan</legend>
|
||||
<pre id="readout">—</pre>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Cache & playback</legend>
|
||||
<dl id="dev-stats" class="dev-dl"></dl>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Review galleries</legend>
|
||||
<a class="dev-link" href="/review.html" target="_blank" rel="noopener">🎞️ Clip pool review</a>
|
||||
<a class="dev-link" href="/review_audio.html" target="_blank" rel="noopener">🔊 Audio pool review</a>
|
||||
</fieldset>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
<script src="/app.js"></script>
|
||||
|
||||
+99
-10
@@ -2,8 +2,9 @@
|
||||
body { margin: 0; font: 14px/1.4 system-ui, sans-serif; background: #111; color: #eee; }
|
||||
header { padding: 0.6rem 1rem; background: #000; }
|
||||
h1 { font-size: 1rem; margin: 0; font-weight: 600; }
|
||||
main { display: flex; gap: 1rem; padding: 1rem; flex-wrap: wrap; }
|
||||
.stage { flex: 1 1 640px; }
|
||||
main { display: flex; gap: 1rem; padding: 1rem; flex-wrap: wrap; align-items: flex-start; }
|
||||
/* Stage stays pinned in view while the right pane scrolls on its own. */
|
||||
.stage { flex: 1 1 640px; position: sticky; top: 1rem; }
|
||||
.screen { position: relative; width: 100%; aspect-ratio: 16 / 9; background: #000;
|
||||
border-radius: 6px; overflow: hidden; }
|
||||
#vid { width: 100%; height: 100%; object-fit: cover; transition: opacity 0.15s ease; }
|
||||
@@ -39,19 +40,107 @@ main { display: flex; gap: 1rem; padding: 1rem; flex-wrap: wrap; }
|
||||
.hud-chip-text.measure { fill: #ffd79a; }
|
||||
.hud-conf { fill: #6cf; opacity: 0.8; }
|
||||
.hud-status { fill: #8fdcff; font: 2.4px monospace; opacity: 0.9; letter-spacing: 0.15px; }
|
||||
.black { position: absolute; inset: 0; background: #000; }
|
||||
/* z-index + own compositing layer: the <video> and WebGL <canvas> are GPU-
|
||||
composited layers that can paint ABOVE a plain auto-z sibling (Safari/Chrome
|
||||
with a real GPU) despite DOM order — so the black cover needs an explicit
|
||||
stacking order above them to actually blank the screen. */
|
||||
.black { position: absolute; inset: 0; background: #000; opacity: 1;
|
||||
z-index: 50; transform: translateZ(0); transition: opacity 200ms ease; }
|
||||
.hidden { display: none; }
|
||||
.panel { flex: 0 0 280px; display: flex; flex-direction: column; gap: 0.8rem; }
|
||||
/* Stack the two Output toggles (Video / Audio) with a little breathing room. */
|
||||
.dev-switch + .dev-switch { margin-top: 0.45rem; }
|
||||
/* Live audio status readout (diagnostic) — turns green when actually playing. */
|
||||
.audio-status { margin-top: 0.5rem; font: 11px/1.4 monospace; color: #9ab; }
|
||||
.audio-status.playing { color: #4e9; }
|
||||
.audio-status.blocked { color: #f97; }
|
||||
.panel { flex: 0 0 280px; display: flex; flex-direction: column; gap: 0.8rem;
|
||||
max-height: calc(100vh - 2rem); overflow-y: auto;
|
||||
position: sticky; top: 1rem; }
|
||||
fieldset { border: 1px solid #333; border-radius: 6px; }
|
||||
legend { color: #9af; padding: 0 0.4rem; }
|
||||
label { display: block; margin: 0.4rem 0; }
|
||||
input[type=range], select { width: 100%; }
|
||||
#readout { background: #000; padding: 0.5rem; border-radius: 4px; font-size: 12px;
|
||||
white-space: pre-wrap; max-height: 240px; overflow: auto; }
|
||||
.ring-control { display: flex; align-items: center; gap: 0.4rem; }
|
||||
.ring-control button { flex: 0 0 auto; background: #1a2436; color: #9af;
|
||||
border: 1px solid #345; border-radius: 4px; padding: 0.3rem 0.5rem;
|
||||
cursor: pointer; font-size: 13px; }
|
||||
.ring-control button:hover { background: #243352; }
|
||||
.scale-name { flex: 1 1 auto; text-align: center; font-size: 12px; color: #cde; }
|
||||
/* Altitude knob */
|
||||
.dial-wrap { display: flex; justify-content: center; padding: 0.3rem 0 0.1rem; }
|
||||
#dial { width: 190px; height: 190px; touch-action: none; cursor: grab; user-select: none; }
|
||||
#dial:active { cursor: grabbing; }
|
||||
.dial-rim { fill: #0d1320; stroke: #243352; stroke-width: 1.2; }
|
||||
.dial-body { fill: #16203200; stroke: #2c3c5c; stroke-width: 1; }
|
||||
.dial-tick { stroke: #3a4d70; stroke-width: 0.8; }
|
||||
.dial-label { fill: #789ac0; font-size: 6px; font-family: ui-monospace, monospace;
|
||||
letter-spacing: 0.2px; cursor: pointer; }
|
||||
.dial-label:hover { fill: #cde; }
|
||||
.dial-label.active { fill: #9cf; font-weight: 700; }
|
||||
.dial-caption { fill: #4d6184; font-size: 4.4px; letter-spacing: 1.2px;
|
||||
font-family: ui-monospace, monospace; }
|
||||
.dial-needle { fill: #9cf; }
|
||||
.dial-needle polygon { filter: drop-shadow(0 0 1px #9cf); }
|
||||
.dial-hub { fill: #2c3c5c; stroke: #9cf; stroke-width: 0.6; }
|
||||
.scale-name { display: block; text-align: center; font-size: 12px; color: #cde; }
|
||||
.hint { margin: 0.3rem 0 0; font-size: 11px; color: #789; }
|
||||
|
||||
/* --- Dev Mode --- a single switch; all dev controls/data live in #dev-panel below it. */
|
||||
.dev-switch { display: flex; align-items: center; gap: 0.5rem; cursor: pointer;
|
||||
padding: 0.5rem 0.2rem 0.2rem; border-top: 1px solid #222; user-select: none; }
|
||||
.dev-switch input { position: absolute; opacity: 0; width: 0; height: 0; }
|
||||
.dev-switch-track { position: relative; width: 34px; height: 18px; border-radius: 9px;
|
||||
background: #2a2a2a; border: 1px solid #444; transition: background 0.15s; }
|
||||
.dev-switch-thumb { position: absolute; top: 1px; left: 1px; width: 14px; height: 14px;
|
||||
border-radius: 50%; background: #888; transition: transform 0.15s, background 0.15s; }
|
||||
.dev-switch input:checked + .dev-switch-track { background: #1d3a2a; border-color: #2e6; }
|
||||
.dev-switch input:checked + .dev-switch-track .dev-switch-thumb { transform: translateX(16px); background: #4e9; }
|
||||
.dev-switch input:focus-visible + .dev-switch-track { outline: 2px solid #9af; outline-offset: 1px; }
|
||||
.dev-switch-label { color: #9af; font-weight: 600; letter-spacing: 0.3px; }
|
||||
.dev-panel { display: flex; flex-direction: column; gap: 0.8rem; }
|
||||
/* Beat the generic `.hidden` (defined earlier, equal specificity): two classes
|
||||
win, so the panel truly collapses when Dev Mode is off. */
|
||||
.dev-panel.hidden { display: none; }
|
||||
.dev-panel legend { color: #4e9; }
|
||||
.dev-btn { margin-top: 0.5rem; width: 100%; padding: 0.4rem; cursor: pointer;
|
||||
background: #182028; color: #cde; border: 1px solid #2c3c5c; border-radius: 4px; }
|
||||
.dev-btn:hover { background: #20303f; }
|
||||
.dev-link { display: block; margin-top: 0.4rem; padding: 0.4rem; text-align: center;
|
||||
background: #182028; color: #cde; border: 1px solid #2c3c5c; border-radius: 4px;
|
||||
text-decoration: none; }
|
||||
.dev-link:hover { background: #20303f; }
|
||||
.dev-dl { margin: 0; display: grid; grid-template-columns: auto 1fr; gap: 0.15rem 0.6rem; font-size: 12px; }
|
||||
.dev-dl dt { color: #789; }
|
||||
.dev-dl dd { margin: 0; color: #dde; word-break: break-word; }
|
||||
.dev-dl dd.mem { color: #4e9; }
|
||||
.dev-dl dd.net { color: #fc6; }
|
||||
.dev-anno { font-size: 11px; max-height: 220px; overflow: auto;
|
||||
font-family: ui-monospace, monospace; }
|
||||
.dev-anno .anno-row { padding: 0.2rem 0; border-bottom: 1px solid #1d1d1d; }
|
||||
.dev-anno .anno-key { color: #aee6ff; }
|
||||
.dev-anno .anno-key.affect { color: #e3cfff; }
|
||||
.dev-anno .anno-key.measure { color: #ffd79a; }
|
||||
.dev-anno .anno-meta { color: #678; }
|
||||
.dev-anno .anno-empty { color: #567; font-style: italic; }
|
||||
|
||||
/* "Loading Universe…" splash — shown until all media is preloaded and the
|
||||
experience is ready to run smoothly, then faded out. */
|
||||
#loading {
|
||||
position: fixed; inset: 0; z-index: 10000;
|
||||
display: flex; align-items: center; justify-content: center;
|
||||
background: radial-gradient(ellipse at center, #0a1230 0%, #02030a 70%);
|
||||
color: #cfe3ff; user-select: none;
|
||||
transition: opacity 0.6s ease;
|
||||
}
|
||||
#loading.done { opacity: 0; pointer-events: none; }
|
||||
.loading-inner { display: flex; flex-direction: column; align-items: center; gap: 1.1rem; }
|
||||
.loading-title { font: 600 30px/1.2 system-ui, sans-serif; letter-spacing: 0.04em; }
|
||||
.loading-dots::after {
|
||||
content: ""; animation: loading-dots 1.4s steps(4, end) infinite;
|
||||
}
|
||||
@keyframes loading-dots { 0% { content: ""; } 25% { content: "."; } 50% { content: ".."; } 75% { content: "..."; } 100% { content: ""; } }
|
||||
.loading-bar {
|
||||
width: 260px; height: 4px; border-radius: 2px;
|
||||
background: rgba(255, 255, 255, 0.12); overflow: hidden;
|
||||
}
|
||||
#loading-fill {
|
||||
height: 100%; width: 0%; border-radius: 2px;
|
||||
background: linear-gradient(90deg, #4e9cff, #9af);
|
||||
transition: width 0.25s ease;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
"""Playwright E2E for the audio + Video/Audio toggles (audio spec §9). Skips
|
||||
cleanly when Playwright or its browser binary is absent.
|
||||
|
||||
NOTE: headless engines relax BOTH autoplay and GPU compositing, so these assert
|
||||
the WIRING (toggle→play, the first-video-on→audio coupling, video-off blanking) —
|
||||
real Safari/iOS autoplay and real-GPU compositing still need a device by-ear/eye check.
|
||||
|
||||
Install: pip install -e '.[e2e]' && python -m playwright install chromium
|
||||
"""
|
||||
import socket
|
||||
import threading
|
||||
import time
|
||||
from contextlib import closing
|
||||
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("playwright.sync_api")
|
||||
|
||||
|
||||
def _free_port() -> int:
|
||||
with closing(socket.socket()) as s:
|
||||
s.bind(("127.0.0.1", 0))
|
||||
return s.getsockname()[1]
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url():
|
||||
import uvicorn
|
||||
|
||||
from simulator.app import app
|
||||
|
||||
port = _free_port()
|
||||
config = uvicorn.Config(app, host="127.0.0.1", port=port, log_level="error")
|
||||
server = uvicorn.Server(config)
|
||||
thread = threading.Thread(target=server.run, daemon=True)
|
||||
thread.start()
|
||||
for _ in range(50):
|
||||
if server.started:
|
||||
break
|
||||
time.sleep(0.1)
|
||||
if not server.started:
|
||||
pytest.skip("uvicorn did not start")
|
||||
yield f"http://127.0.0.1:{port}"
|
||||
server.should_exit = True
|
||||
thread.join(timeout=5)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def page(app_url):
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
with sync_playwright() as p:
|
||||
try:
|
||||
# headless blocks media playback without a device; this flag lets play()
|
||||
# actually start so `paused` is meaningful (spec §9 asserts src/paused)
|
||||
browser = p.chromium.launch(args=["--autoplay-policy=no-user-gesture-required"])
|
||||
except Exception as exc: # no browser binary installed
|
||||
pytest.skip(f"chromium not available: {exc}")
|
||||
pg = browser.new_page()
|
||||
pg.goto(app_url)
|
||||
# wait out the "Loading Universe…" splash (it overlays + blocks clicks)
|
||||
pg.wait_for_selector("#loading", state="detached", timeout=60000)
|
||||
yield pg
|
||||
browser.close()
|
||||
|
||||
|
||||
def _toggle(page, which):
|
||||
"""Click the visible toggle track for #visual / #audio (the input is hidden)."""
|
||||
page.click(f"label[for='{which}'] .dev-switch-track")
|
||||
|
||||
|
||||
def test_app_starts_blanked_and_silent(page):
|
||||
# both toggles default off → black screen, no audio (no tap-to-start wall)
|
||||
assert page.is_checked("#visual") is False
|
||||
assert page.is_checked("#audio") is False
|
||||
page.wait_for_selector("#black:not(.hidden)")
|
||||
assert page.evaluate("document.getElementById('aud').paused") is True
|
||||
|
||||
|
||||
def test_audio_toggle_plays_the_current_scale_soundtrack(page):
|
||||
_toggle(page, "audio")
|
||||
page.wait_for_function(
|
||||
"(() => { const a = document.getElementById('aud');"
|
||||
" return /\\/media\\/audio\\/.+\\.mp3/.test(a.src) && !a.paused; })()"
|
||||
)
|
||||
|
||||
|
||||
def test_first_video_on_also_enables_audio(page):
|
||||
_toggle(page, "visual") # first video-on couples audio
|
||||
page.wait_for_function("document.getElementById('audio').checked === true")
|
||||
page.wait_for_function(
|
||||
"(() => { const a = document.getElementById('aud'); return !!a.src && !a.paused; })()"
|
||||
)
|
||||
page.wait_for_selector("#black.hidden", state="attached") # video showing (black hidden=display:none)
|
||||
|
||||
|
||||
def test_audio_before_video_stays_audio_only(page):
|
||||
_toggle(page, "audio") # audio on first
|
||||
page.wait_for_function("!document.getElementById('aud').paused")
|
||||
assert page.is_checked("#visual") is False # video untouched
|
||||
page.wait_for_selector("#black:not(.hidden)") # still blanked
|
||||
|
||||
|
||||
def test_video_off_blanks_after_being_on(page):
|
||||
_toggle(page, "visual") # on
|
||||
page.wait_for_selector("#black.hidden", state="attached")
|
||||
_toggle(page, "visual") # off
|
||||
page.wait_for_selector("#black:not(.hidden)")
|
||||
# the GPU-composited video layers are hidden too (settle the opacity transition)
|
||||
page.wait_for_function(
|
||||
"['vid','paint'].every(id => getComputedStyle(document.getElementById(id)).opacity === '0')"
|
||||
)
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Unit tests for the pure audio ffmpeg arg builders (no ffmpeg run), plus an
|
||||
opt-in integration test that actually synthesizes a noise bed when ffmpeg is
|
||||
present."""
|
||||
|
||||
import shutil
|
||||
|
||||
import pytest
|
||||
|
||||
from tools.pipeline.audio_ops import audio_loop_args, loudnorm_args, white_noise_args
|
||||
|
||||
|
||||
def test_loop_args_mirror_the_video_crossfade_recipe():
|
||||
args = audio_loop_args("in.mp3", "out.mp3", duration=30.0, overlap=2.0, ff="FF")
|
||||
assert args[0] == "FF"
|
||||
joined = " ".join(args)
|
||||
# tail crossfades over head, then concats the middle — the audio analogue of
|
||||
# crossfade_loop_args. Output length = duration - overlap.
|
||||
assert "acrossfade=d=2.0" in joined
|
||||
assert "atrim=0:2.0" in joined # head
|
||||
assert "atrim=28.0:30.0" in joined # tail (d-overlap : d)
|
||||
assert "concat=n=2:v=0:a=1" in joined
|
||||
assert "-map" in args and "[out]" in args
|
||||
assert "-q:a" in args and "4" in args
|
||||
|
||||
|
||||
def test_loop_args_reject_overlap_past_half():
|
||||
with pytest.raises(ValueError):
|
||||
audio_loop_args("in.mp3", "out.mp3", duration=10.0, overlap=5.0)
|
||||
|
||||
|
||||
def test_loudnorm_args_carry_the_locked_targets():
|
||||
args = loudnorm_args("in.mp3", "out.mp3", ff="FF")
|
||||
joined = " ".join(args)
|
||||
assert "loudnorm=I=-18.0:TP=-1.5:LRA=11.0" in joined
|
||||
assert args[0] == "FF" and args[-1] == "out.mp3"
|
||||
|
||||
|
||||
def test_white_noise_args_are_pink_and_clean():
|
||||
args = white_noise_args("noise.mp3", duration=60.0, color="pink", ff="FF")
|
||||
joined = " ".join(args)
|
||||
assert "anoisesrc=color=pink" in joined
|
||||
assert "-t" in args and "60.0" in args
|
||||
assert args[-1] == "noise.mp3"
|
||||
|
||||
|
||||
def test_white_noise_rejects_unknown_color():
|
||||
with pytest.raises(ValueError):
|
||||
white_noise_args("noise.mp3", color="ultraviolet")
|
||||
|
||||
|
||||
def _have_ffmpeg() -> bool:
|
||||
if shutil.which("ffmpeg"):
|
||||
return True
|
||||
try:
|
||||
import imageio_ffmpeg # noqa: F401
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _have_ffmpeg(), reason="no ffmpeg available")
|
||||
def test_generate_white_noise_writes_a_playable_loop(tmp_path):
|
||||
from tools.pipeline.audio_run import generate_white_noise
|
||||
out = generate_white_noise(tmp_path / "pink.mp3", duration=2.0)
|
||||
assert out.exists() and out.stat().st_size > 0
|
||||
@@ -4,7 +4,9 @@ import pytest
|
||||
|
||||
from tools.pipeline.run import probe_duration, process_clip, resolve_ffmpeg
|
||||
|
||||
_FOREST = Path("simulator/sample_media/forest/base.mp4")
|
||||
# A real pool base (forest was retired with the rotating-pool rename, session 0016);
|
||||
# cosmos/base.mp4 is the cosmos pool primary and the most stable PD clip on disk.
|
||||
_BASE = Path("simulator/sample_media/cosmos/base.mp4")
|
||||
|
||||
|
||||
def _ffmpeg_available() -> bool:
|
||||
@@ -18,20 +20,18 @@ def _ffmpeg_available() -> bool:
|
||||
_HAS_FF = _ffmpeg_available()
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _FOREST.exists(),
|
||||
reason="forest POC base absent (run simulator/setup_sample_media.py)")
|
||||
def test_probe_duration_reads_forest_base():
|
||||
if not _FOREST.exists():
|
||||
pytest.skip("forest POC base absent")
|
||||
assert probe_duration(_FOREST) > 0
|
||||
@pytest.mark.skipif(not _BASE.exists(),
|
||||
reason="cosmos base absent (run simulator/build_pool_manifest.py)")
|
||||
def test_probe_duration_reads_real_base():
|
||||
assert probe_duration(_BASE) > 0
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _HAS_FF,
|
||||
reason="neither system ffmpeg nor imageio-ffmpeg available")
|
||||
@pytest.mark.skipif(not _FOREST.exists(),
|
||||
reason="forest POC base absent (run simulator/setup_sample_media.py)")
|
||||
@pytest.mark.skipif(not _BASE.exists(),
|
||||
reason="cosmos base absent (run simulator/build_pool_manifest.py)")
|
||||
def test_process_clip_emits_proxy_and_master(tmp_path):
|
||||
out = process_clip(_FOREST, tmp_path, overlap=0.5, start=0.0, duration=4.0)
|
||||
out = process_clip(_BASE, tmp_path, overlap=0.5, start=0.0, duration=4.0)
|
||||
assert out["proxy"].exists() and out["master"].exists()
|
||||
# Proxy is exactly 1920×1080 — verified with cv2, no ffprobe dependency.
|
||||
import cv2
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
"""tools/pipeline/track.py — the hybrid motion-track pass (content-pipeline §11.5).
|
||||
|
||||
Pure geometry helpers are unit-tested here; the cv2 optical-flow propagation is an
|
||||
opt-in integration test that runs on a real pool clip when present."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tools.pipeline.track import (
|
||||
clamp01_box,
|
||||
denormalize_box,
|
||||
infer_window,
|
||||
loop_t,
|
||||
normalize_box,
|
||||
sample_track,
|
||||
track_to_annotation,
|
||||
)
|
||||
|
||||
|
||||
# --- pure helpers -----------------------------------------------------------
|
||||
|
||||
|
||||
def test_clamp01_box_keeps_box_on_screen():
|
||||
assert clamp01_box((-0.2, 0.1, 0.5, 0.5)) == [0.0, 0.1, 0.5, 0.5]
|
||||
# size trimmed so the box never runs past the right/bottom edge
|
||||
assert clamp01_box((0.8, 0.8, 0.5, 0.5)) == [0.8, 0.8, pytest.approx(0.2), pytest.approx(0.2)]
|
||||
|
||||
|
||||
def test_normalize_denormalize_round_trip():
|
||||
px = (192, 108, 384, 216)
|
||||
norm = normalize_box(px, 1920, 1080)
|
||||
assert norm == [pytest.approx(0.1), pytest.approx(0.1), pytest.approx(0.2), pytest.approx(0.2)]
|
||||
assert denormalize_box(norm, 1920, 1080) == (192, 108, 384, 216)
|
||||
|
||||
|
||||
def test_loop_t_is_frame_over_total():
|
||||
assert loop_t(0, 100) == 0.0
|
||||
assert loop_t(50, 100) == 0.5
|
||||
assert loop_t(5, 0) == 0.0 # guard divide-by-zero
|
||||
|
||||
|
||||
def test_infer_window_spans_tracked_frames():
|
||||
appear, disappear = infer_window([10, 11, 30, 31], 100)
|
||||
assert appear == pytest.approx(0.1)
|
||||
assert disappear == pytest.approx(0.31)
|
||||
# no frames -> full clip
|
||||
assert infer_window([], 100) == (0.0, 1.0)
|
||||
|
||||
|
||||
def test_sample_track_downsamples_to_sparse_keyframes():
|
||||
# 21 dense frames -> at most n_keyframes, including the first and last
|
||||
per_frame = {i: [i / 100, 0.2, 0.1, 0.1] for i in range(0, 21)}
|
||||
track = sample_track(per_frame, total_frames=100, n_keyframes=5)
|
||||
assert len(track) <= 5
|
||||
ts = [k["t"] for k in track]
|
||||
assert ts == sorted(ts) # sorted by t
|
||||
assert ts[0] == pytest.approx(0.0) # first tracked frame
|
||||
assert ts[-1] == pytest.approx(0.2) # last tracked frame (frame 20 / 100)
|
||||
assert len(set(ts)) == len(ts) # de-duplicated on t
|
||||
|
||||
|
||||
def test_sample_track_keeps_all_when_already_sparse():
|
||||
per_frame = {0: [0.1, 0.1, 0.1, 0.1], 60: [0.5, 0.3, 0.1, 0.1]}
|
||||
track = sample_track(per_frame, total_frames=120, n_keyframes=5)
|
||||
assert [k["t"] for k in track] == [pytest.approx(0.0), pytest.approx(0.5)]
|
||||
|
||||
|
||||
def test_sample_track_empty_is_empty():
|
||||
assert sample_track({}, total_frames=100) == []
|
||||
|
||||
|
||||
def test_track_to_annotation_assembles_geometry_only():
|
||||
track = [{"t": 0.0, "box": [0.1, 0.2, 0.1, 0.1]}, {"t": 0.5, "box": [0.4, 0.2, 0.1, 0.1]}]
|
||||
ann = track_to_annotation("detected.jelly", track, salience=3, appear=0.0, disappear=0.55)
|
||||
assert ann["key"] == "detected.jelly"
|
||||
assert ann["salience"] == 3
|
||||
assert ann["track"] == track
|
||||
assert ann["appear"] == 0.0 and ann["disappear"] == 0.55
|
||||
# no semantics leaked in — strings/tiers are the author's, added elsewhere
|
||||
assert "strings" not in ann and "_tiers" not in ann
|
||||
|
||||
|
||||
def test_track_to_annotation_omits_window_when_absent():
|
||||
ann = track_to_annotation("detected.x", [], salience=4)
|
||||
assert "appear" not in ann and "disappear" not in ann
|
||||
|
||||
|
||||
# --- opt-in: classical optical-flow propagation on a real clip --------------
|
||||
|
||||
_CLIP = Path("simulator/sample_media/abyss_wow/base.mp4")
|
||||
|
||||
|
||||
def _cv2_available() -> bool:
|
||||
try:
|
||||
import cv2 # noqa: F401
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _CLIP.exists(),
|
||||
reason="abyss_wow base absent (run simulator/build_pool_manifest.py)")
|
||||
@pytest.mark.skipif(not _cv2_available(), reason="opencv-python not available")
|
||||
def test_track_seed_produces_a_sparse_annotation_on_real_footage():
|
||||
from tools.pipeline.track import track_box, track_seed
|
||||
|
||||
seed = (0.2, 0.3, 0.15, 0.18)
|
||||
per_frame, total = track_box(_CLIP, seed, seed_frame=0, max_frames=40)
|
||||
assert total > 0
|
||||
assert per_frame[0] == pytest.approx(list(seed), abs=0.02)
|
||||
assert len(per_frame) > 1 # propagated beyond the seed frame
|
||||
|
||||
ann = track_seed(_CLIP, "detected.jelly", seed, seed_frame=0, n_keyframes=5, max_frames=40)
|
||||
assert ann["key"] == "detected.jelly"
|
||||
assert 2 <= len(ann["track"]) <= 5 # sparse keyframes
|
||||
for kf in ann["track"]:
|
||||
assert 0.0 <= kf["t"] <= 1.0
|
||||
assert all(0.0 <= v <= 1.0 for v in kf["box"])
|
||||
assert 0.0 <= ann["appear"] <= ann["disappear"] <= 1.0
|
||||
@@ -55,42 +55,44 @@ def test_left_and_right_stack_not_cancel():
|
||||
assert plan.dream.intensity == 1.0
|
||||
|
||||
|
||||
def test_affect_needs_both_knobs_up():
|
||||
# affect-channel design (session 0013): emotion-words surface only when BOTH
|
||||
# the Left and Right knobs are up — either at 0 means no affect at all.
|
||||
def test_affect_is_driven_by_the_right_knob_alone():
|
||||
# Right-brain affect (session 0018): emotion-words are controlled by the Right
|
||||
# knob alone — Right at 0 means no affect regardless of Left, and Left no longer
|
||||
# gates the channel (full Right with Left at 0 still surfaces affect).
|
||||
assert plan_alteration(_coord(left=4, right=0)).affect.strength == 0
|
||||
assert plan_alteration(_coord(left=0, right=4)).affect.strength == 0
|
||||
assert plan_alteration(_coord(left=4, right=0)).affect.intensity == 0.0
|
||||
assert plan_alteration(_coord(left=0, right=4)).affect.intensity == 0.0
|
||||
assert plan_alteration(_coord(left=0, right=4)).affect.strength == 4
|
||||
assert plan_alteration(_coord(left=0, right=4)).affect.intensity == 1.0
|
||||
|
||||
|
||||
def test_affect_strength_is_the_smaller_knob():
|
||||
# strength = min(left, right) — the smaller knob gates the channel.
|
||||
assert plan_alteration(_coord(left=2, right=4)).affect.strength == 2
|
||||
def test_affect_strength_is_the_right_knob():
|
||||
# strength = right — Left is irrelevant to the affect channel.
|
||||
assert plan_alteration(_coord(left=2, right=4)).affect.strength == 4
|
||||
assert plan_alteration(_coord(left=4, right=2)).affect.strength == 2
|
||||
assert plan_alteration(_coord(left=4, right=4)).affect.strength == 4
|
||||
assert plan_alteration(_coord(left=3, right=1)).affect.strength == 1
|
||||
assert plan_alteration(_coord(left=0, right=3)).affect.strength == 3
|
||||
assert plan_alteration(_coord(left=4, right=1)).affect.strength == 1
|
||||
|
||||
|
||||
def test_affect_intensity_scales_with_strength_over_full_scale():
|
||||
for lo in range(5):
|
||||
plan = plan_alteration(_coord(left=lo, right=4))
|
||||
assert plan.affect.intensity == pytest.approx(lo / 4)
|
||||
assert plan_alteration(_coord(left=4, right=4)).affect.intensity == 1.0
|
||||
def test_affect_intensity_scales_with_right_over_full_scale():
|
||||
for r in range(5):
|
||||
plan = plan_alteration(_coord(left=0, right=r))
|
||||
assert plan.affect.intensity == pytest.approx(r / 4)
|
||||
assert plan_alteration(_coord(left=0, right=4)).affect.intensity == 1.0
|
||||
|
||||
|
||||
def test_affect_intensity_honors_overlay_gain():
|
||||
# affect rides the same overlay_gain as the analytical HUD it belongs to.
|
||||
# affect text rides the overlay-opacity gain, but is keyed off the Right knob.
|
||||
cal = Calibration(overlay_gain=0.5)
|
||||
assert plan_alteration(_coord(left=4, right=4), cal).affect.intensity == pytest.approx(0.5)
|
||||
assert plan_alteration(_coord(left=0, right=4), cal).affect.intensity == pytest.approx(0.5)
|
||||
clamp = Calibration(overlay_gain=10.0)
|
||||
assert plan_alteration(_coord(left=4, right=4), clamp).affect.intensity == 1.0
|
||||
assert plan_alteration(_coord(left=0, right=4), clamp).affect.intensity == 1.0
|
||||
|
||||
|
||||
def test_affect_active_implies_non_identity():
|
||||
# min(left,right) > 0 implies left > 0, so an active affect plan is never identity.
|
||||
plan = plan_alteration(_coord(left=2, right=3))
|
||||
assert plan.affect.strength == 2
|
||||
# affect.strength == right, and right > 0 makes dream.strength > 0, so an active
|
||||
# affect plan (even with Left at 0) is never identity.
|
||||
plan = plan_alteration(_coord(left=0, right=3))
|
||||
assert plan.affect.strength == 3
|
||||
assert not plan.is_identity
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Unit tests for the pure Visual×Audio resolver (audio spec §4/§6)."""
|
||||
import pytest
|
||||
|
||||
from player.audio import (
|
||||
AUDIO_SOURCES,
|
||||
VISUAL_POSITIONS,
|
||||
AudioResolution,
|
||||
resolve_audio,
|
||||
resolve_audio_url,
|
||||
resolve_visual,
|
||||
)
|
||||
|
||||
def test_visual_on_off():
|
||||
assert resolve_visual("on") is True
|
||||
assert resolve_visual("off") is False
|
||||
|
||||
|
||||
def test_visual_rejects_unknown():
|
||||
with pytest.raises(ValueError):
|
||||
resolve_visual("flicker")
|
||||
|
||||
|
||||
def test_visual_positions_are_on_off():
|
||||
assert VISUAL_POSITIONS == frozenset({"on", "off"})
|
||||
|
||||
|
||||
def test_audio_sources_are_off_and_soundtrack():
|
||||
assert AUDIO_SOURCES == frozenset({"off", "soundtrack"})
|
||||
|
||||
|
||||
def test_soundtrack_resolves_to_the_current_scales_asset():
|
||||
url = resolve_audio_url("soundtrack", scale_audio="coast/waves.loop.mp3")
|
||||
assert url == "/media/audio/coast/waves.loop.mp3"
|
||||
|
||||
|
||||
def test_off_resolves_to_silence():
|
||||
assert resolve_audio_url("off", scale_audio="coast/waves.loop.mp3") is None
|
||||
|
||||
|
||||
def test_soundtrack_with_no_scale_asset_is_silent():
|
||||
assert resolve_audio_url("soundtrack", scale_audio="") is None
|
||||
|
||||
|
||||
def test_resolve_audio_marks_only_soundtrack_altitude_coupled():
|
||||
r = resolve_audio("soundtrack", scale_audio="reef/soundscape.loop.mp3")
|
||||
assert isinstance(r, AudioResolution)
|
||||
assert r.source == "soundtrack" and r.altitude_coupled is True
|
||||
assert r.url == "/media/audio/reef/soundscape.loop.mp3"
|
||||
assert resolve_audio("off", scale_audio="x/y.mp3").altitude_coupled is False
|
||||
|
||||
|
||||
def test_resolve_audio_url_rejects_unknown_source():
|
||||
with pytest.raises(ValueError):
|
||||
resolve_audio_url("white_noise", scale_audio="") # deferred, not a v1 source
|
||||
|
||||
|
||||
def test_music_and_white_noise_are_not_v1_sources():
|
||||
assert "music" not in AUDIO_SOURCES
|
||||
assert "white_noise" not in AUDIO_SOURCES
|
||||
@@ -1,34 +0,0 @@
|
||||
import pytest
|
||||
|
||||
from player.content import AUDIO_SOURCES, ContentResolution, resolve_content
|
||||
|
||||
|
||||
def test_audio_sources_are_the_four_distinct_sources():
|
||||
assert AUDIO_SOURCES == frozenset({"none", "white_noise", "music", "audio_track"})
|
||||
|
||||
|
||||
# The §6 table, row by row: position -> (audio_source, video).
|
||||
@pytest.mark.parametrize(
|
||||
"position,audio_source,video",
|
||||
[
|
||||
("off", "none", False),
|
||||
("white_noise", "white_noise", False),
|
||||
("music", "music", False),
|
||||
("audio_track", "audio_track", False),
|
||||
("video", "none", True),
|
||||
("music_video", "music", True),
|
||||
("audio_video", "audio_track", True),
|
||||
],
|
||||
)
|
||||
def test_resolve_content_matches_spec_table(position, audio_source, video):
|
||||
assert resolve_content(position) == ContentResolution(audio_source=audio_source, video=video)
|
||||
|
||||
|
||||
def test_off_is_void_state_black_and_silent():
|
||||
r = resolve_content("off")
|
||||
assert r.video is False and r.audio_source == "none"
|
||||
|
||||
|
||||
def test_resolve_content_rejects_unknown_position():
|
||||
with pytest.raises(ValueError):
|
||||
resolve_content("bogus")
|
||||
@@ -1,27 +1,36 @@
|
||||
import pytest
|
||||
|
||||
from player.audio import AUDIO_SOURCES, VISUAL_POSITIONS
|
||||
from player.controls import (
|
||||
Controls,
|
||||
CONTENT_POSITIONS,
|
||||
ControlsError,
|
||||
validate_controls,
|
||||
parse_controls,
|
||||
)
|
||||
|
||||
|
||||
def test_content_positions_are_the_seven_from_the_spec():
|
||||
assert CONTENT_POSITIONS == frozenset(
|
||||
{"off", "white_noise", "music", "audio_track", "video", "music_video", "audio_video"}
|
||||
)
|
||||
def test_visual_and_audio_enums_are_the_v1_positions():
|
||||
assert VISUAL_POSITIONS == frozenset({"on", "off"})
|
||||
assert AUDIO_SOURCES == frozenset({"off", "soundtrack"})
|
||||
|
||||
|
||||
def test_valid_controls_pass_validation():
|
||||
c = Controls(content="video", left=0, right=4, dark=2, light=2, volume=3, brightness=4)
|
||||
c = Controls(visual="on", audio="soundtrack", left=0, right=4, dark=2, light=2,
|
||||
volume=3, brightness=4)
|
||||
validate_controls(c) # does not raise
|
||||
|
||||
|
||||
def test_invalid_content_position_rejected():
|
||||
c = Controls(content="bogus", left=0, right=0, dark=0, light=0, volume=0, brightness=0)
|
||||
def test_invalid_visual_position_rejected():
|
||||
c = Controls(visual="flicker", audio="off", left=0, right=0, dark=0, light=0,
|
||||
volume=0, brightness=0)
|
||||
with pytest.raises(ControlsError):
|
||||
validate_controls(c)
|
||||
|
||||
|
||||
def test_invalid_audio_source_rejected():
|
||||
# "music" is a reserved/deferred dial position, not a v1 source
|
||||
c = Controls(visual="on", audio="music", left=0, right=0, dark=0, light=0,
|
||||
volume=0, brightness=0)
|
||||
with pytest.raises(ControlsError):
|
||||
validate_controls(c)
|
||||
|
||||
@@ -29,7 +38,8 @@ def test_invalid_content_position_rejected():
|
||||
@pytest.mark.parametrize("field", ["left", "right", "dark", "light", "volume", "brightness"])
|
||||
@pytest.mark.parametrize("bad", [-1, 5, True])
|
||||
def test_out_of_range_or_non_int_knob_rejected(field, bad):
|
||||
kwargs = dict(content="off", left=0, right=0, dark=0, light=0, volume=0, brightness=0)
|
||||
kwargs = dict(visual="off", audio="off", left=0, right=0, dark=0, light=0,
|
||||
volume=0, brightness=0)
|
||||
kwargs[field] = bad
|
||||
with pytest.raises(ControlsError):
|
||||
validate_controls(Controls(**kwargs))
|
||||
@@ -37,17 +47,18 @@ def test_out_of_range_or_non_int_knob_rejected(field, bad):
|
||||
|
||||
def test_parse_controls_from_mapping():
|
||||
c = parse_controls(
|
||||
{"content": "music_video", "left": 1, "right": 2, "dark": 3, "light": 0, "volume": 2, "brightness": 1}
|
||||
{"visual": "on", "audio": "soundtrack", "left": 1, "right": 2, "dark": 3,
|
||||
"light": 0, "volume": 2, "brightness": 1}
|
||||
)
|
||||
assert c == Controls("music_video", 1, 2, 3, 0, 2, 1)
|
||||
assert c == Controls("on", "soundtrack", 1, 2, 3, 0, 2, 1)
|
||||
|
||||
|
||||
def test_parse_controls_rejects_unknown_keys():
|
||||
with pytest.raises(ControlsError):
|
||||
parse_controls({"content": "off", "left": 0, "right": 0, "dark": 0,
|
||||
parse_controls({"visual": "off", "audio": "off", "left": 0, "right": 0, "dark": 0,
|
||||
"light": 0, "volume": 0, "brightness": 0, "bogus": 1})
|
||||
|
||||
|
||||
def test_parse_controls_rejects_missing_keys():
|
||||
with pytest.raises(ControlsError):
|
||||
parse_controls({"content": "off", "left": 0})
|
||||
parse_controls({"visual": "off", "left": 0})
|
||||
|
||||
@@ -8,6 +8,7 @@ from player.ring import (
|
||||
Transition,
|
||||
TransitionStep,
|
||||
advance_ring,
|
||||
pick_clip_id,
|
||||
scale_at,
|
||||
)
|
||||
|
||||
@@ -206,3 +207,43 @@ def test_fast_spin_noop_on_degenerate_ring():
|
||||
move = advance_ring(r, from_index=0, delta=9, fast_spin_threshold=3)
|
||||
assert move.fast is False
|
||||
assert move.steps == ()
|
||||
|
||||
|
||||
# --- rotating clip pool (content-pipeline design §11.1) ---
|
||||
|
||||
|
||||
def test_scale_without_pool_is_a_pool_of_one():
|
||||
s = Scale(id="cosmos", clip_id="cosmos")
|
||||
assert s.members == ("cosmos",)
|
||||
# any r maps to the single member
|
||||
assert pick_clip_id(s, 0.0) == "cosmos"
|
||||
assert pick_clip_id(s, 0.99) == "cosmos"
|
||||
|
||||
|
||||
def test_scale_with_pool_exposes_all_members():
|
||||
s = Scale(id="coast", clip_id="coast_a", pool=("coast_a", "coast_b", "coast_c"))
|
||||
assert s.members == ("coast_a", "coast_b", "coast_c")
|
||||
|
||||
|
||||
def test_pick_clip_id_selects_member_by_injected_r():
|
||||
s = Scale(id="coast", clip_id="a", pool=("a", "b", "c", "d"))
|
||||
# r in [0,1) partitions the pool into len equal buckets
|
||||
assert pick_clip_id(s, 0.0) == "a"
|
||||
assert pick_clip_id(s, 0.24) == "a"
|
||||
assert pick_clip_id(s, 0.25) == "b"
|
||||
assert pick_clip_id(s, 0.5) == "c"
|
||||
assert pick_clip_id(s, 0.75) == "d"
|
||||
|
||||
|
||||
def test_pick_clip_id_clamps_out_of_range_r():
|
||||
s = Scale(id="coast", clip_id="a", pool=("a", "b", "c"))
|
||||
assert pick_clip_id(s, -5.0) == "a" # below 0 clamps to first
|
||||
assert pick_clip_id(s, 1.0) == "c" # 1.0 must not index past the end
|
||||
assert pick_clip_id(s, 999.0) == "c"
|
||||
|
||||
|
||||
def test_scale_carries_an_optional_audio_path():
|
||||
s = Scale(id="coast", clip_id="coast_birdrock", audio="coast/waves.loop.mp3")
|
||||
assert s.audio == "coast/waves.loop.mp3"
|
||||
# back-compat: audio defaults to empty
|
||||
assert Scale(id="x", clip_id="x").audio == ""
|
||||
|
||||
+35
-25
@@ -14,47 +14,57 @@ class FakeClip:
|
||||
LIB = [FakeClip("base-a"), FakeClip("base-b")]
|
||||
|
||||
|
||||
def _controls(content="video", left=0, right=0, dark=0, light=0, volume=2, brightness=2):
|
||||
return Controls(content, left, right, dark, light, volume, brightness)
|
||||
def _controls(visual="on", audio="off", left=0, right=0, dark=0, light=0,
|
||||
volume=2, brightness=2):
|
||||
return Controls(visual, audio, left, right, dark, light, volume, brightness)
|
||||
|
||||
|
||||
def test_first_update_to_video_fades_in_from_black():
|
||||
p = Player(LIB)
|
||||
t = p.update(_controls(content="video"))
|
||||
t = p.update(_controls(visual="on"))
|
||||
assert t.kind == TransitionKind.FADE_FROM_BLACK
|
||||
assert t.playback.clip_id == "base-a"
|
||||
assert t.playback.content.video is True
|
||||
assert t.playback.video is True
|
||||
|
||||
|
||||
def test_off_from_video_fades_to_black_and_silences():
|
||||
def test_visual_off_from_video_fades_to_black():
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="video"))
|
||||
t = p.update(_controls(content="off"))
|
||||
p.update(_controls(visual="on"))
|
||||
t = p.update(_controls(visual="off"))
|
||||
assert t.kind == TransitionKind.FADE_TO_BLACK
|
||||
assert t.playback.clip_id is None
|
||||
assert t.playback.content.audio_source == "none"
|
||||
assert t.playback.video is False
|
||||
assert t.playback.audio_source == "off"
|
||||
|
||||
|
||||
def test_audio_survives_visual_off():
|
||||
# soundtrack + black is reachable (a gap the bundled dial omitted)
|
||||
p = Player(LIB)
|
||||
t = p.update(_controls(visual="off", audio="soundtrack"))
|
||||
assert t.playback.video is False
|
||||
assert t.playback.audio_source == "soundtrack"
|
||||
|
||||
|
||||
def test_no_change_yields_none_transition():
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="video", left=1))
|
||||
t = p.update(_controls(content="video", left=1))
|
||||
p.update(_controls(visual="on", left=1))
|
||||
t = p.update(_controls(visual="on", left=1))
|
||||
assert t.kind == TransitionKind.NONE
|
||||
|
||||
|
||||
def test_grade_change_is_a_live_update_not_a_crossfade():
|
||||
# design §4.3: the Dark/Light grade is a continuous runtime op
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="video", dark=0, light=0))
|
||||
t = p.update(_controls(content="video", dark=4, light=0))
|
||||
p.update(_controls(visual="on", dark=0, light=0))
|
||||
t = p.update(_controls(visual="on", dark=4, light=0))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE
|
||||
assert t.playback.plan.grade.tone == -1.0
|
||||
|
||||
|
||||
def test_overlay_change_is_a_live_update():
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="video", left=0))
|
||||
t = p.update(_controls(content="video", left=4))
|
||||
p.update(_controls(visual="on", left=0))
|
||||
t = p.update(_controls(visual="on", left=4))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE
|
||||
assert t.playback.plan.overlay.level == 4
|
||||
|
||||
@@ -63,8 +73,8 @@ def test_dream_change_is_a_live_update_not_a_crossfade():
|
||||
# Right-axis dream reframe (session 0013): the Right dream is a deterministic
|
||||
# LIVE filter now, so changing it no longer crossfades — only a clip swap does.
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="video", right=0))
|
||||
t = p.update(_controls(content="video", right=4))
|
||||
p.update(_controls(visual="on", right=0))
|
||||
t = p.update(_controls(visual="on", right=4))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE
|
||||
assert t.playback.plan.dream.strength == 4
|
||||
assert t.playback.plan.dream.intensity == 1.0
|
||||
@@ -72,43 +82,43 @@ def test_dream_change_is_a_live_update_not_a_crossfade():
|
||||
|
||||
def test_volume_only_change_is_a_live_update():
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="video", volume=1))
|
||||
t = p.update(_controls(content="video", volume=4))
|
||||
p.update(_controls(visual="on", volume=1))
|
||||
t = p.update(_controls(visual="on", volume=4))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE
|
||||
assert t.playback.volume == 4
|
||||
|
||||
|
||||
def test_audio_source_change_while_black_is_a_live_update():
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="white_noise"))
|
||||
t = p.update(_controls(content="music"))
|
||||
p.update(_controls(visual="off", audio="off"))
|
||||
t = p.update(_controls(visual="off", audio="soundtrack"))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE
|
||||
assert t.playback.content.audio_source == "music"
|
||||
assert t.playback.audio_source == "soundtrack"
|
||||
|
||||
|
||||
def test_injected_base_chooser_is_used():
|
||||
p = Player(LIB, choose_base=lambda lib: lib[1])
|
||||
t = p.update(_controls(content="video"))
|
||||
t = p.update(_controls(visual="on"))
|
||||
assert t.playback.clip_id == "base-b"
|
||||
|
||||
|
||||
def test_empty_library_with_video_raises():
|
||||
p = Player([])
|
||||
with pytest.raises(ValueError):
|
||||
p.update(_controls(content="video"))
|
||||
p.update(_controls(visual="on"))
|
||||
|
||||
|
||||
def test_off_with_empty_library_is_fine():
|
||||
p = Player([])
|
||||
# levels at 0 match the initial black state, so this is a no-op transition
|
||||
t = p.update(_controls(content="off", volume=0, brightness=0))
|
||||
t = p.update(_controls(visual="off", volume=0, brightness=0))
|
||||
assert t.kind == TransitionKind.NONE # already black at init
|
||||
assert t.playback.clip_id is None
|
||||
|
||||
|
||||
def test_off_with_levels_from_black_is_a_live_update():
|
||||
p = Player([])
|
||||
t = p.update(_controls(content="off", volume=3, brightness=2))
|
||||
t = p.update(_controls(visual="off", volume=3, brightness=2))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE # black->black, levels set
|
||||
assert t.playback.clip_id is None
|
||||
assert t.playback.volume == 3
|
||||
|
||||
+243
-12
@@ -28,8 +28,9 @@ def client(manifest_path):
|
||||
return TestClient(create_app(manifest_path=manifest_path))
|
||||
|
||||
|
||||
def _controls(content="video", left=0, right=0, dark=0, light=0, volume=2, brightness=2):
|
||||
return dict(content=content, left=left, right=right, dark=dark,
|
||||
def _controls(visual="on", audio="off", left=0, right=0, dark=0, light=0,
|
||||
volume=2, brightness=2):
|
||||
return dict(visual=visual, audio=audio, left=left, right=right, dark=dark,
|
||||
light=light, volume=volume, brightness=brightness)
|
||||
|
||||
|
||||
@@ -41,13 +42,40 @@ def test_alteration_returns_the_engine_plan(client):
|
||||
assert data["plan"]["dream"]["strength"] == 2
|
||||
assert data["plan"]["dream"]["intensity"] == 0.5
|
||||
assert data["plan"]["grade"]["tone"] == -1.0
|
||||
assert data["content"]["video"] is True
|
||||
assert data["render"]["video"]["shown"] is True
|
||||
|
||||
|
||||
def test_alteration_honors_off_as_black(client):
|
||||
resp = client.post("/api/alteration", json={"controls": _controls(content="off")})
|
||||
data = resp.json()
|
||||
assert data["content"]["video"] is False
|
||||
def test_alteration_returns_render_video_and_audio(client):
|
||||
body = client.post("/api/alteration", json={"controls": _controls()}).json()
|
||||
assert body["render"]["video"]["shown"] is True
|
||||
assert body["render"]["audio"] == {"source": "off", "url": None, "altitude_coupled": False}
|
||||
|
||||
|
||||
def test_alteration_visual_off_hides_video(client):
|
||||
body = client.post("/api/alteration", json={"controls": _controls(visual="off")}).json()
|
||||
assert body["render"]["video"]["shown"] is False
|
||||
|
||||
|
||||
def test_alteration_off_is_silent(ring_client):
|
||||
body = ring_client.post("/api/alteration", json={
|
||||
"controls": _controls(audio="off"), "altitude_index": 0,
|
||||
}).json()
|
||||
assert body["render"]["audio"] == {"source": "off", "url": None, "altitude_coupled": False}
|
||||
|
||||
|
||||
def test_alteration_soundtrack_couples_to_the_given_altitude(ring_client):
|
||||
# ring fixture order: cosmos(0), forest(1), abyss(2) — each carries its audio
|
||||
body = ring_client.post("/api/alteration", json={
|
||||
"controls": _controls(audio="soundtrack"), "altitude_index": 0,
|
||||
}).json()
|
||||
a = body["render"]["audio"]
|
||||
assert a["source"] == "soundtrack" and a["altitude_coupled"] is True
|
||||
assert a["url"] == "/media/audio/cosmos/pillars.loop.mp3"
|
||||
# a different altitude resolves a different soundtrack url
|
||||
body2 = ring_client.post("/api/alteration", json={
|
||||
"controls": _controls(audio="soundtrack"), "altitude_index": 2,
|
||||
}).json()
|
||||
assert body2["render"]["audio"]["url"] == "/media/audio/abyss/whale.loop.mp3"
|
||||
|
||||
|
||||
def test_alteration_accepts_calibration(client):
|
||||
@@ -62,8 +90,13 @@ def test_alteration_rejects_out_of_range_knob(client):
|
||||
assert resp.status_code == 422
|
||||
|
||||
|
||||
def test_alteration_rejects_bad_content(client):
|
||||
resp = client.post("/api/alteration", json={"controls": _controls(content="banana")})
|
||||
def test_alteration_rejects_bad_audio(client):
|
||||
resp = client.post("/api/alteration", json={"controls": _controls(audio="music")})
|
||||
assert resp.status_code == 422
|
||||
|
||||
|
||||
def test_alteration_rejects_bad_visual(client):
|
||||
resp = client.post("/api/alteration", json={"controls": _controls(visual="dim")})
|
||||
assert resp.status_code == 422
|
||||
|
||||
|
||||
@@ -76,6 +109,30 @@ def test_clips_returns_the_manifest(client):
|
||||
assert data["clips"][0]["annotations"][0]["key"] == "detected.water"
|
||||
|
||||
|
||||
def test_media_versions_endpoint_shape(client):
|
||||
resp = client.get("/api/media-versions")
|
||||
assert resp.status_code == 200
|
||||
assert isinstance(resp.json()["versions"], dict)
|
||||
|
||||
|
||||
def test_media_version_is_a_content_hash(tmp_path, monkeypatch):
|
||||
import hashlib
|
||||
|
||||
import simulator.app as appmod
|
||||
|
||||
# Point the media root at a temp dir (the served dir is otherwise fixed).
|
||||
monkeypatch.setattr(appmod, "MEDIA_DIR", tmp_path)
|
||||
(tmp_path / "clip").mkdir()
|
||||
f = tmp_path / "clip" / "base.mp4"
|
||||
f.write_bytes(b"hello-bytes")
|
||||
assert appmod._media_version("clip/base.mp4") == hashlib.sha1(b"hello-bytes").hexdigest()[:12]
|
||||
# Absent file -> None (it's simply omitted from the versions map).
|
||||
assert appmod._media_version("clip/missing.mp4") is None
|
||||
# Re-baked under the same path (new bytes) -> a new token, not the cached one.
|
||||
f.write_bytes(b"different-bytes-entirely")
|
||||
assert appmod._media_version("clip/base.mp4") == hashlib.sha1(b"different-bytes-entirely").hexdigest()[:12]
|
||||
|
||||
|
||||
def test_retired_selection_endpoints_are_gone(client):
|
||||
# The route no longer exists; the static catch-all yields 404 on GET and
|
||||
# 405 on the (now-unrouted) POST. Either proves the endpoint is gone.
|
||||
@@ -91,6 +148,35 @@ def test_index_is_served():
|
||||
assert "Alteration" in resp.text
|
||||
|
||||
|
||||
def test_app_assets_are_not_cached(client):
|
||||
# Dev iteration relies on the browser never serving stale renderer code.
|
||||
resp = client.get("/")
|
||||
assert "no-store" in resp.headers.get("cache-control", "")
|
||||
|
||||
|
||||
def test_media_is_cacheable(tmp_path, monkeypatch):
|
||||
# Media is static: it must be browser-cacheable so the preloader's fetch (and
|
||||
# any fallback) doesn't re-hit the network on every altitude change.
|
||||
import simulator.app as appmod
|
||||
|
||||
media = tmp_path / "media"
|
||||
(media / "cosmos").mkdir(parents=True)
|
||||
(media / "cosmos" / "base.mp4").write_bytes(b"\x00\x00\x00\x18ftyp")
|
||||
monkeypatch.setattr(appmod, "MEDIA_DIR", media)
|
||||
manifest = tmp_path / "manifest.json"
|
||||
manifest.write_text(json.dumps({"clips": []}))
|
||||
|
||||
client = TestClient(create_app(manifest_path=manifest))
|
||||
resp = client.get("/media/cosmos/base.mp4")
|
||||
assert resp.status_code == 200
|
||||
cc = resp.headers.get("cache-control", "")
|
||||
# Stored but revalidated: not no-store (so the blob preload can cache it), and
|
||||
# not pinned immutable (so a re-sourced/re-cropped clip shows on a plain reload).
|
||||
assert "no-store" not in cc
|
||||
assert "immutable" not in cc
|
||||
assert "no-cache" in cc
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def ring_manifest_path(tmp_path):
|
||||
p = tmp_path / "manifest.json"
|
||||
@@ -109,9 +195,9 @@ def ring_manifest_path(tmp_path):
|
||||
],
|
||||
"ring": {
|
||||
"scales": [
|
||||
{"id": "cosmos", "clip_id": "cosmos"},
|
||||
{"id": "forest", "clip_id": "forest"},
|
||||
{"id": "abyss", "clip_id": "abyss"},
|
||||
{"id": "cosmos", "clip_id": "cosmos", "audio": "cosmos/pillars.loop.mp3"},
|
||||
{"id": "forest", "clip_id": "forest", "audio": "forest/amb.loop.mp3"},
|
||||
{"id": "abyss", "clip_id": "abyss", "audio": "abyss/whale.loop.mp3"},
|
||||
],
|
||||
"transitions": [
|
||||
{"file": "transitions/cosmos-forest.mp4", "model": "placeholder"},
|
||||
@@ -137,6 +223,15 @@ def test_ring_returns_scales_and_transitions(ring_client):
|
||||
assert len(data["transitions"]) == 3
|
||||
|
||||
|
||||
def test_ring_exposes_each_scales_audio(ring_client):
|
||||
body = ring_client.get("/api/ring").json()
|
||||
by_id = {s["id"]: s for s in body["scales"]}
|
||||
# every scale advertises its soundtrack path (audio spec §5.1)
|
||||
assert by_id["cosmos"]["audio"] == "cosmos/pillars.loop.mp3"
|
||||
assert by_id["abyss"]["audio"].endswith(".mp3")
|
||||
assert "audio" in by_id["forest"]
|
||||
|
||||
|
||||
def test_ring_advance_inward_one_step(ring_client):
|
||||
resp = ring_client.post("/api/ring/advance", json={"from_index": 0, "delta": 1})
|
||||
assert resp.status_code == 200
|
||||
@@ -189,3 +284,139 @@ def test_ring_404_when_no_ring_in_manifest(client):
|
||||
def test_ring_advance_rejects_bad_delta(ring_client):
|
||||
resp = ring_client.post("/api/ring/advance", json={"from_index": 0, "delta": "x"})
|
||||
assert resp.status_code == 422
|
||||
|
||||
|
||||
# --- rotating clip pool (content-pipeline §11.1) ---
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def pool_client(tmp_path):
|
||||
p = tmp_path / "manifest.json"
|
||||
members = ["coast_a", "coast_b", "coast_c", "coast_d"]
|
||||
clips = [{"id": "cosmos", "title": "Cosmos", "base_file": "cosmos/base.mp4",
|
||||
"license": "PD", "source": "NASA", "right_variants": {},
|
||||
"annotations": [], "affect": [], "strings": {}}]
|
||||
clips += [{"id": m, "title": m, "base_file": f"{m}/base.mp4", "license": "PD",
|
||||
"source": "NPS", "right_variants": {}, "annotations": [], "affect": [],
|
||||
"strings": {}} for m in members]
|
||||
p.write_text(json.dumps({
|
||||
"clips": clips,
|
||||
"ring": {
|
||||
"scales": [
|
||||
{"id": "cosmos", "clip_id": "cosmos"},
|
||||
{"id": "coast", "clip_id": members[0], "pool": members},
|
||||
],
|
||||
"transitions": [
|
||||
{"file": "transitions/cosmos-coast.mp4"},
|
||||
{"file": "transitions/coast-cosmos.mp4"},
|
||||
],
|
||||
},
|
||||
}))
|
||||
return TestClient(create_app(manifest_path=p)), members
|
||||
|
||||
|
||||
def test_ring_exposes_each_scales_pool(pool_client):
|
||||
client, members = pool_client
|
||||
data = client.get("/api/ring").json()
|
||||
coast = next(s for s in data["scales"] if s["id"] == "coast")
|
||||
assert [m["clip_id"] for m in coast["pool"]] == members
|
||||
# a single-clip scale still reports a pool of one
|
||||
cosmos = next(s for s in data["scales"] if s["id"] == "cosmos")
|
||||
assert [m["clip_id"] for m in cosmos["pool"]] == ["cosmos"]
|
||||
|
||||
|
||||
def test_advance_into_a_pool_lands_on_a_member(pool_client):
|
||||
client, members = pool_client
|
||||
# landing on the coast scale (index 1) must pick one of its pool members
|
||||
seen = set()
|
||||
for _ in range(40):
|
||||
data = client.post("/api/ring/advance", json={"from_index": 0, "delta": 1}).json()
|
||||
assert data["to_index"] == 1
|
||||
assert data["target_clip_id"] in members
|
||||
seen.add(data["target_clip_id"])
|
||||
# over many landings the random pick should cover more than one member
|
||||
assert len(seen) > 1
|
||||
|
||||
|
||||
def test_delta_zero_is_an_initial_pool_pick(pool_client):
|
||||
client, members = pool_client
|
||||
# a no-op (delta 0) advance on the pool scale is the initial / re-roll pick
|
||||
data = client.post("/api/ring/advance", json={"from_index": 1, "delta": 0}).json()
|
||||
assert data["to_index"] == 1
|
||||
assert data["steps"] == []
|
||||
assert data["target_clip_id"] in members
|
||||
|
||||
|
||||
# --- author mode (content-pipeline §11.5) ---
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def author_client(tmp_path):
|
||||
p = tmp_path / "manifest.json"
|
||||
p.write_text(json.dumps({
|
||||
"clips": [{
|
||||
"id": "abyss_wow", "title": "World of Water", "base_file": "abyss_wow/base.mp4",
|
||||
"license": "PD", "source": "NOAA", "right_variants": {},
|
||||
"annotations": [], "affect": [], "strings": {"en": {}},
|
||||
}],
|
||||
}))
|
||||
return TestClient(create_app(manifest_path=p)), p
|
||||
|
||||
|
||||
def test_author_clip_persists_labels_and_keeps_provenance(author_client):
|
||||
client, path = author_client
|
||||
body = {
|
||||
"clip_id": "abyss_wow",
|
||||
"annotations": [{"key": "detected.jelly", "salience": 4,
|
||||
"appear": 0.0, "disappear": 0.5,
|
||||
"track": [{"t": 0.0, "box": [0.1, 0.2, 0.1, 0.1]}]}],
|
||||
"affect": [{"key": "feel.unease", "at": [0.5, 0.5], "min_level": 1}],
|
||||
"strings": {"en": {"detected.jelly": ["jelly", "comb jelly", "Ctenophora", "Ctenophora · cilia rows"],
|
||||
"feel.unease": ["uh", "unease", "disquiet", "a creeping disquiet"]}},
|
||||
}
|
||||
resp = client.post("/api/author/clip", json=body)
|
||||
assert resp.status_code == 200 and resp.json()["ok"] is True
|
||||
# persisted to disk
|
||||
saved = json.loads(path.read_text())["clips"][0]
|
||||
assert saved["annotations"][0]["key"] == "detected.jelly"
|
||||
assert saved["affect"][0]["key"] == "feel.unease"
|
||||
assert saved["strings"]["en"]["detected.jelly"][2] == "Ctenophora"
|
||||
# provenance preserved (only authored content replaced)
|
||||
assert saved["base_file"] == "abyss_wow/base.mp4"
|
||||
assert saved["license"] == "PD"
|
||||
# reflected by /api/clips without a restart
|
||||
served = client.get("/api/clips").json()["clips"][0]
|
||||
assert served["annotations"][0]["key"] == "detected.jelly"
|
||||
|
||||
|
||||
def test_author_clip_unknown_clip_404(author_client):
|
||||
client, _ = author_client
|
||||
resp = client.post("/api/author/clip", json={"clip_id": "nope", "annotations": []})
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
def test_author_track_unknown_clip_404(author_client):
|
||||
client, _ = author_client
|
||||
resp = client.post("/api/author/track", json={
|
||||
"clip_id": "nope", "key": "detected.x", "seed_box": [0.1, 0.1, 0.1, 0.1]})
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
_REAL_CLIP = __import__("pathlib").Path("simulator/sample_media/abyss_wow/base.mp4")
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _REAL_CLIP.exists(),
|
||||
reason="abyss_wow base absent (run simulator/build_pool_manifest.py)")
|
||||
def test_author_track_on_real_clip_returns_keyframes():
|
||||
# Against the real default manifest + footage: the tracker yields a sparse track.
|
||||
client = TestClient(create_app())
|
||||
resp = client.post("/api/author/track", json={
|
||||
"clip_id": "abyss_wow", "key": "detected.jelly",
|
||||
"seed_box": [0.2, 0.3, 0.15, 0.18], "seed_t": 0.0,
|
||||
"salience": 4, "n_keyframes": 5, "max_frames": 30,
|
||||
})
|
||||
assert resp.status_code == 200
|
||||
ann = resp.json()
|
||||
assert ann["key"] == "detected.jelly"
|
||||
assert 2 <= len(ann["track"]) <= 5
|
||||
assert 0.0 <= ann["appear"] <= ann["disappear"] <= 1.0
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Pure ffmpeg argument builders (no I/O) for the audio production pass.
|
||||
|
||||
The audio analogue of tools/pipeline/ffmpeg_ops.py: each function returns a
|
||||
list[str] ready for ffmpeg, so command construction is unit-testable without
|
||||
running ffmpeg. tools/pipeline/audio_run.py executes them. The ffmpeg binary
|
||||
name is injected (default "ffmpeg")."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
_NOISE_COLORS = frozenset({"white", "pink", "brown", "blue", "violet"})
|
||||
|
||||
|
||||
def audio_loop_args(src, dst, *, duration: float, overlap: float,
|
||||
ff: str = "ffmpeg") -> list[str]:
|
||||
"""Make `src` loop seamlessly by crossfading its tail over its head — the
|
||||
audio mirror of ffmpeg_ops.crossfade_loop_args. Output length = duration -
|
||||
overlap. Requires overlap < duration/2 so a non-empty middle remains."""
|
||||
if overlap <= 0 or overlap >= duration / 2:
|
||||
raise ValueError(f"overlap {overlap} must be in (0, duration/2={duration / 2})")
|
||||
d, o = duration, overlap
|
||||
fc = (
|
||||
f"[0:a]atrim=0:{o},asetpts=N/SR/TB[head];"
|
||||
f"[0:a]atrim={o}:{d - o},asetpts=N/SR/TB[mid];"
|
||||
f"[0:a]atrim={d - o}:{d},asetpts=N/SR/TB[tail];"
|
||||
f"[tail][head]acrossfade=d={o}:c1=tri:c2=tri[xf];"
|
||||
f"[xf][mid]concat=n=2:v=0:a=1[out]"
|
||||
)
|
||||
return [
|
||||
ff, "-y", "-i", str(src), "-filter_complex", fc,
|
||||
"-map", "[out]", "-c:a", "libmp3lame", "-q:a", "4", str(dst),
|
||||
]
|
||||
|
||||
|
||||
def loudnorm_args(src, dst, *, i: float = -18.0, tp: float = -1.5,
|
||||
lra: float = 11.0, ff: str = "ffmpeg") -> list[str]:
|
||||
"""EBU R128 loudness normalize to the locked installation target so all five
|
||||
soundtracks (and the noise bed) sit at one comfortable level."""
|
||||
return [
|
||||
ff, "-y", "-i", str(src),
|
||||
"-af", f"loudnorm=I={i}:TP={tp}:LRA={lra}",
|
||||
"-c:a", "libmp3lame", "-q:a", "4", str(dst),
|
||||
]
|
||||
|
||||
|
||||
def white_noise_args(dst, *, duration: float = 60.0, color: str = "pink",
|
||||
ff: str = "ffmpeg") -> list[str]:
|
||||
"""Synthesize a calm colored-noise bed (default pink) of `duration` seconds —
|
||||
deterministic, zero licensing. Loops cleanly (steady-state noise has no seam)."""
|
||||
if color not in _NOISE_COLORS:
|
||||
raise ValueError(
|
||||
f"unknown noise color {color!r}; expected one of {sorted(_NOISE_COLORS)}"
|
||||
)
|
||||
return [
|
||||
ff, "-y", "-f", "lavfi",
|
||||
"-i", f"anoisesrc=color={color}:amplitude=0.5:duration={duration}",
|
||||
"-t", str(duration), "-c:a", "libmp3lame", "-q:a", "4", str(dst),
|
||||
]
|
||||
@@ -0,0 +1,61 @@
|
||||
"""Thin runner: probe duration, then run the audio production pass (seamless
|
||||
loop -> loudness normalize) and synthesize the white-noise bed. The pure arg
|
||||
builders (tools.pipeline.audio_ops) are unit-tested; this glue is covered by the
|
||||
opt-in integration test."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from .audio_ops import audio_loop_args, loudnorm_args, white_noise_args
|
||||
from .run import resolve_ffmpeg
|
||||
|
||||
|
||||
def _probe_audio_duration(src, ff: str) -> float:
|
||||
"""Audio length in seconds via ffmpeg (no system ffprobe): decode to null and
|
||||
read the last reported time off stderr."""
|
||||
proc = subprocess.run(
|
||||
[ff, "-i", str(src), "-f", "null", "-"],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
times = re.findall(r"time=(\d+):(\d+):(\d+\.\d+)", proc.stderr)
|
||||
if not times:
|
||||
raise ValueError(f"could not probe audio duration of {src}")
|
||||
h, m, s = times[-1]
|
||||
return int(h) * 3600 + int(m) * 60 + float(s)
|
||||
|
||||
|
||||
def _run(args: list[str]) -> None:
|
||||
subprocess.run(args, check=True, capture_output=True)
|
||||
|
||||
|
||||
def process_soundtrack(src, dst, *, overlap: float = 2.0, ff: str | None = None) -> Path:
|
||||
"""Produce a seamless, loudness-normalized loop from a sourced ambience clip.
|
||||
Clips too short to crossfade-loop (<= 2*overlap) are normalized only."""
|
||||
ff = ff or resolve_ffmpeg()
|
||||
src, dst = Path(src), Path(dst)
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
duration = _probe_audio_duration(src, ff)
|
||||
tmp = dst.with_suffix(".loop.tmp.mp3")
|
||||
if duration > 2 * overlap + 0.5:
|
||||
_run(audio_loop_args(src, tmp, duration=duration, overlap=overlap, ff=ff))
|
||||
_run(loudnorm_args(tmp, dst, ff=ff))
|
||||
tmp.unlink(missing_ok=True)
|
||||
else:
|
||||
_run(loudnorm_args(src, dst, ff=ff))
|
||||
return dst
|
||||
|
||||
|
||||
def generate_white_noise(dst, *, duration: float = 60.0, color: str = "pink",
|
||||
ff: str | None = None) -> Path:
|
||||
"""Synthesize + normalize the global white-noise bed."""
|
||||
ff = ff or resolve_ffmpeg()
|
||||
dst = Path(dst)
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = dst.with_suffix(".raw.tmp.mp3")
|
||||
_run(white_noise_args(tmp, duration=duration, color=color, ff=ff))
|
||||
_run(loudnorm_args(tmp, dst, ff=ff))
|
||||
tmp.unlink(missing_ok=True)
|
||||
return dst
|
||||
@@ -0,0 +1,297 @@
|
||||
"""Stage 5 geometry — the hybrid motion-track pass (content-pipeline §3.5 / §11.5).
|
||||
|
||||
Turns a hand-seeded box (or an ML detection) on one keyframe into the sparse,
|
||||
loop-normalized keyframed `track` the manifest stores (design §4) plus an
|
||||
appear/disappear window (§11.2). SEMANTICS are never produced here — the author
|
||||
owns the `key`, strings, `salience` and tiers; this module only propagates BOX
|
||||
GEOMETRY.
|
||||
|
||||
Two geometry paths, same artifact:
|
||||
- **classical** (always available): a hand-seeded box propagated by OpenCV
|
||||
Lucas-Kanade optical flow (a CSRT tracker is used instead when a contrib
|
||||
build provides one). Translation model — right for calm drifting subjects
|
||||
(the abyss/reef creatures). Re-seeds features when too few survive.
|
||||
- **ML detect+track** (optional, lazy-imported): a YOLO+tracker pass proposing
|
||||
boxes the author maps to a key. Imported only on demand so the base install
|
||||
needs no torch/ultralytics; a clear ImportError tells you what to `pip
|
||||
install` if you ask for it without it present.
|
||||
|
||||
The PURE helpers (normalization, keyframe sampling, window inference, annotation
|
||||
assembly) are unit-tested; the cv2/ML propagation is covered by the opt-in
|
||||
integration test on real footage.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Optional
|
||||
|
||||
Box = tuple[float, float, float, float] # (x, y, w, h)
|
||||
|
||||
|
||||
# --- pure geometry helpers --------------------------------------------------
|
||||
|
||||
def clamp01_box(box: Box) -> list[float]:
|
||||
"""Clamp a normalized box into the visible [0,1] frame, keeping it on-screen
|
||||
(origin clamped, then size trimmed so x+w<=1, y+h<=1)."""
|
||||
x, y, w, h = box
|
||||
x = min(max(x, 0.0), 1.0)
|
||||
y = min(max(y, 0.0), 1.0)
|
||||
w = min(max(w, 0.0), 1.0 - x)
|
||||
h = min(max(h, 0.0), 1.0 - y)
|
||||
return [x, y, w, h]
|
||||
|
||||
|
||||
def normalize_box(box_px: Box, width: int, height: int) -> list[float]:
|
||||
"""Pixel box -> normalized [0,1] box (the manifest's resolution-independent
|
||||
form). Clamped on-screen."""
|
||||
x, y, w, h = box_px
|
||||
return clamp01_box((x / width, y / height, w / width, h / height))
|
||||
|
||||
|
||||
def denormalize_box(box_norm: Box, width: int, height: int) -> tuple[int, int, int, int]:
|
||||
"""Normalized [0,1] box -> integer pixel box for the cv2 tracker."""
|
||||
x, y, w, h = box_norm
|
||||
return (round(x * width), round(y * height), round(w * width), round(h * height))
|
||||
|
||||
|
||||
def loop_t(frame_idx: int, total_frames: int) -> float:
|
||||
"""Loop-normalized time of a frame in [0,1), matching the client's
|
||||
`currentTime / duration` clock (design §4)."""
|
||||
if total_frames <= 0:
|
||||
return 0.0
|
||||
return frame_idx / total_frames
|
||||
|
||||
|
||||
def infer_window(tracked_frames: list[int], total_frames: int) -> tuple[float, float]:
|
||||
"""The appear/disappear window (§11.2) covering the frames where a box was
|
||||
tracked, as loop-normalized t. The first/last tracked frame bound it."""
|
||||
if not tracked_frames:
|
||||
return (0.0, 1.0)
|
||||
lo, hi = min(tracked_frames), max(tracked_frames)
|
||||
return (loop_t(lo, total_frames), loop_t(hi, total_frames))
|
||||
|
||||
|
||||
def sample_track(
|
||||
per_frame: dict[int, Box], total_frames: int, n_keyframes: int = 5
|
||||
) -> list[dict]:
|
||||
"""Down-sample a dense per-frame box map to a SPARSE keyframed track
|
||||
(design §4: sparse keyframes interpolated at runtime, never dense per-frame).
|
||||
|
||||
Picks `n_keyframes` frames evenly across the tracked span (always including
|
||||
the first and last tracked frame) and emits `[{t, box}, ...]` sorted by t,
|
||||
de-duplicated on t. Pure — takes already-normalized boxes."""
|
||||
if not per_frame:
|
||||
return []
|
||||
frames = sorted(per_frame)
|
||||
n = max(2, n_keyframes)
|
||||
if len(frames) <= n:
|
||||
picks = frames
|
||||
else:
|
||||
lo, hi = frames[0], frames[-1]
|
||||
step = (hi - lo) / (n - 1)
|
||||
targets = [lo + round(step * i) for i in range(n)]
|
||||
# snap each target to the nearest actually-tracked frame
|
||||
picks = sorted({min(frames, key=lambda f: abs(f - t)) for t in targets})
|
||||
out, seen = [], set()
|
||||
for f in picks:
|
||||
t = round(loop_t(f, total_frames), 4)
|
||||
if t in seen:
|
||||
continue
|
||||
seen.add(t)
|
||||
out.append({"t": t, "box": clamp01_box(per_frame[f])})
|
||||
return out
|
||||
|
||||
|
||||
def track_to_annotation(
|
||||
key: str,
|
||||
track: list[dict],
|
||||
*,
|
||||
salience: int = 4,
|
||||
appear: Optional[float] = None,
|
||||
disappear: Optional[float] = None,
|
||||
) -> dict:
|
||||
"""Assemble a manifest annotation from a propagated track + the author's
|
||||
semantics. Geometry (track + window) comes from this module; `key`,
|
||||
`salience` and the tiered strings are the author's (added separately)."""
|
||||
ann: dict = {"key": key, "salience": salience, "track": track}
|
||||
if appear is not None:
|
||||
ann["appear"] = round(appear, 4)
|
||||
if disappear is not None:
|
||||
ann["disappear"] = round(disappear, 4)
|
||||
return ann
|
||||
|
||||
|
||||
# --- classical propagation (cv2; covered by the opt-in integration test) -----
|
||||
|
||||
def _new_csrt():
|
||||
"""A CSRT tracker if this OpenCV build ships one (contrib), else None — then
|
||||
the optical-flow path is used. Kept tiny so the import stays lazy."""
|
||||
import cv2
|
||||
for factory in ("TrackerCSRT_create",):
|
||||
if hasattr(cv2, factory):
|
||||
return getattr(cv2, factory)()
|
||||
legacy = getattr(cv2, "legacy", None)
|
||||
if legacy is not None and hasattr(legacy, "TrackerCSRT_create"):
|
||||
return legacy.TrackerCSRT_create()
|
||||
return None
|
||||
|
||||
|
||||
def _features_in_box(gray, box_px, max_corners: int = 60):
|
||||
"""Good-feature points inside a pixel box, in full-frame coords (for LK)."""
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
x, y, w, h = (int(v) for v in box_px)
|
||||
h_img, w_img = gray.shape[:2]
|
||||
x0, y0 = max(x, 0), max(y, 0)
|
||||
x1, y1 = min(x + w, w_img), min(y + h, h_img)
|
||||
if x1 - x0 < 4 or y1 - y0 < 4:
|
||||
return None
|
||||
roi = gray[y0:y1, x0:x1]
|
||||
pts = cv2.goodFeaturesToTrack(roi, max_corners, 0.01, 5)
|
||||
if pts is None:
|
||||
return None
|
||||
pts = pts.reshape(-1, 2) + np.array([x0, y0], dtype="float32")
|
||||
return pts.reshape(-1, 1, 2)
|
||||
|
||||
|
||||
def track_box(
|
||||
video_path,
|
||||
seed_box_norm: Box,
|
||||
*,
|
||||
seed_frame: int = 0,
|
||||
max_frames: Optional[int] = None,
|
||||
min_features: int = 8,
|
||||
) -> tuple[dict[int, list[float]], int]:
|
||||
"""Propagate a hand-seeded normalized box FORWARD from `seed_frame` with
|
||||
optical flow (or CSRT when available). Returns `(per_frame_norm_boxes,
|
||||
total_frames)` — feed `per_frame` to `sample_track`. Translation model: the
|
||||
box follows the median displacement of features inside it, re-seeding when too
|
||||
few survive. Impure (reads the video) — opt-in integration test."""
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
cap = cv2.VideoCapture(str(video_path))
|
||||
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
|
||||
if seed_frame:
|
||||
cap.set(cv2.CAP_PROP_POS_FRAMES, seed_frame)
|
||||
ok, frame = cap.read()
|
||||
if not ok:
|
||||
cap.release()
|
||||
raise ValueError(f"could not read frame {seed_frame} of {video_path}")
|
||||
h_img, w_img = frame.shape[:2]
|
||||
box = list(denormalize_box(seed_box_norm, w_img, h_img)) # px [x,y,w,h]
|
||||
|
||||
csrt = _new_csrt()
|
||||
if csrt is not None:
|
||||
csrt.init(frame, tuple(int(v) for v in box))
|
||||
|
||||
prev_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
|
||||
pts = None if csrt is not None else _features_in_box(prev_gray, box)
|
||||
out: dict[int, list[float]] = {seed_frame: normalize_box(box, w_img, h_img)}
|
||||
|
||||
fidx = seed_frame
|
||||
while True:
|
||||
ok, frame = cap.read()
|
||||
if not ok:
|
||||
break
|
||||
fidx += 1
|
||||
if csrt is not None:
|
||||
ok2, b = csrt.update(frame)
|
||||
if ok2:
|
||||
box = [b[0], b[1], b[2], b[3]]
|
||||
else:
|
||||
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
|
||||
if pts is None or len(pts) < min_features:
|
||||
pts = _features_in_box(prev_gray, box)
|
||||
if pts is not None and len(pts):
|
||||
nxt, status, _ = cv2.calcOpticalFlowPyrLK(prev_gray, gray, pts, None)
|
||||
if nxt is not None:
|
||||
good_old = pts[status == 1]
|
||||
good_new = nxt[status == 1]
|
||||
if len(good_new) >= 3:
|
||||
dx = float(np.median(good_new[:, 0] - good_old[:, 0]))
|
||||
dy = float(np.median(good_new[:, 1] - good_old[:, 1]))
|
||||
box[0] += dx
|
||||
box[1] += dy
|
||||
pts = good_new.reshape(-1, 1, 2)
|
||||
else:
|
||||
pts = None
|
||||
prev_gray = gray
|
||||
out[fidx] = normalize_box(box, w_img, h_img)
|
||||
if max_frames is not None and fidx - seed_frame >= max_frames:
|
||||
break
|
||||
cap.release()
|
||||
return out, total
|
||||
|
||||
|
||||
def track_seed(
|
||||
video_path,
|
||||
key: str,
|
||||
seed_box_norm: Box,
|
||||
*,
|
||||
seed_frame: int = 0,
|
||||
salience: int = 4,
|
||||
n_keyframes: int = 5,
|
||||
max_frames: Optional[int] = None,
|
||||
) -> dict:
|
||||
"""End-to-end classical pass: seed -> propagate -> sample -> annotation
|
||||
(geometry only; the author adds strings/tiers). Impure (reads the video)."""
|
||||
per_frame, total = track_box(
|
||||
video_path, seed_box_norm, seed_frame=seed_frame, max_frames=max_frames
|
||||
)
|
||||
track = sample_track(per_frame, total, n_keyframes=n_keyframes)
|
||||
appear, disappear = infer_window(list(per_frame), total)
|
||||
return track_to_annotation(
|
||||
key, track, salience=salience, appear=appear, disappear=disappear
|
||||
)
|
||||
|
||||
|
||||
# --- optional ML detect+track (lazy-imported) --------------------------------
|
||||
|
||||
def detect_and_track(video_path, *, model: str = "yolov8n.pt", classes=None) -> list[dict]:
|
||||
"""Optional ML path (content-pipeline §3.5): run a detect+track model and
|
||||
return candidate tracks `[{track_id, label, salience, track:[{t,box}],
|
||||
appear, disappear}]` for the author to map to keys. Heavy deps are
|
||||
lazy-imported so the base install stays light.
|
||||
|
||||
Raises a clear ImportError naming the install if `ultralytics` is absent —
|
||||
this path is opt-in by design; the classical `track_seed` needs only cv2."""
|
||||
try:
|
||||
from ultralytics import YOLO # noqa: F401
|
||||
except ImportError as e: # pragma: no cover - exercised only without the dep
|
||||
raise ImportError(
|
||||
"the ML detect+track path needs ultralytics — "
|
||||
"`pip install ultralytics` (optional; the classical track_seed path "
|
||||
"uses only opencv-python)."
|
||||
) from e
|
||||
import cv2 # noqa: F401 (pragma: no cover below — needs the real model)
|
||||
|
||||
yolo = YOLO(model) # pragma: no cover
|
||||
cap = cv2.VideoCapture(str(video_path)) # pragma: no cover
|
||||
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0) # pragma: no cover
|
||||
cap.release() # pragma: no cover
|
||||
per_track: dict[int, dict[int, list[float]]] = {} # pragma: no cover
|
||||
labels: dict[int, str] = {} # pragma: no cover
|
||||
for fidx, res in enumerate( # pragma: no cover
|
||||
yolo.track(source=str(video_path), persist=True, classes=classes, stream=True)
|
||||
):
|
||||
if res.boxes is None or res.boxes.id is None: # pragma: no cover
|
||||
continue
|
||||
w_img, h_img = res.orig_shape[1], res.orig_shape[0] # pragma: no cover
|
||||
for box, tid, cls in zip( # pragma: no cover
|
||||
res.boxes.xywh.tolist(), res.boxes.id.tolist(), res.boxes.cls.tolist()
|
||||
):
|
||||
cx, cy, bw, bh = box # pragma: no cover
|
||||
nb = normalize_box((cx - bw / 2, cy - bh / 2, bw, bh), w_img, h_img)
|
||||
per_track.setdefault(int(tid), {})[fidx] = nb # pragma: no cover
|
||||
labels[int(tid)] = yolo.names.get(int(cls), str(cls))
|
||||
out = [] # pragma: no cover
|
||||
for tid, frames in per_track.items(): # pragma: no cover
|
||||
track = sample_track(frames, total) # pragma: no cover
|
||||
appear, disappear = infer_window(list(frames), total) # pragma: no cover
|
||||
out.append({ # pragma: no cover
|
||||
"track_id": tid, "label": labels.get(tid, ""), "salience": 4,
|
||||
"track": track, "appear": round(appear, 4), "disappear": round(disappear, 4),
|
||||
})
|
||||
return out # pragma: no cover
|
||||
Reference in New Issue
Block a user