diff --git a/.gitignore b/.gitignore index 11bc4ce..889577c 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ __pycache__/ *.pyc +.DS_Store .pytest_cache/ .venv/ media/ @@ -8,5 +9,11 @@ media/ # Simulator sample media (look-tuning only; populate via setup_sample_media.py # / build_pool_manifest.py). All scale + transition binaries are gitignored. simulator/sample_media/**/*.mp4 -# Local-only candidate review gallery (re-buildable; not shipped content). +# Audio candidate clips (re-sourceable via docs/audio-candidate-pool.md). +simulator/sample_media/**/*.wav +simulator/sample_media/**/*.mp3 +simulator/sample_media/**/*.ogg +simulator/sample_media/**/*.m4a +# Local-only candidate review galleries (re-buildable; not shipped content). simulator/static/review.html +simulator/static/review_audio.html diff --git a/docs/audio-candidate-pool.md b/docs/audio-candidate-pool.md new file mode 100644 index 0000000..77cd033 --- /dev/null +++ b/docs/audio-candidate-pool.md @@ -0,0 +1,93 @@ +# Audio candidate pool — rotating pools per altitude (2026-06-26) + +Candidate soundtrack/ambience for the **"audio + video" setting**. Following operator +feedback, each altitude now has a **rotating pool** (mirrors the video `Scale.pool` +model — random pick per landing, or pairable per-clip). + +## Two audio layers per altitude (operator, 2026-06-26) + +The "audio + video" setting layers **two** kinds of audio per altitude: +1. **Ambience / audio track** — the soundscape (waves, reef crackle, chorus, etc.). + **This is what we're sourcing now.** +2. **Music track** — a separate composed musical layer. **Deferred** until the + ambience tracks are settled across all altitudes. + +Pools below are the **ambience** layer unless marked `music`. **Distinction (operator):** +the ambience/soundtrack layer is an *environmental, real-world soundscape* (waves, reef, +whale, room-tone hum) — **not composed music**. A musical pad/track, however nice, belongs +in the music layer, even for "atmospheric" altitudes like cosmos/orbit. + +**Media is gitignored** (`simulator/sample_media/**/*.{wav,mp3,…}`); this file is the +re-sourceable record. Files: `simulator/sample_media/audio//.`, +served at `/media/audio//.`. Review gallery (gitignored): +`simulator/static/review_audio.html` → `/review_audio.html` with the sim running. + +## Round-2 feedback applied (operator, session 2026-06-26) + +- **abyss** ✓ kept (blue-whale moan) · **coast** ✓ kept (waves + terns) +- **cosmos** ✗ plasma static "hurt my ears" → replaced with 3 NASA **sonifications** +- **orbit** ✗ whistlers "sound like lasers" → replaced with gentle Earth **chorus** (+ wind) +- **reef** ✗ "shouldn't have whales (too deep for a reef)" → whale removed; now crackle + diver +- Operator: *"try all three… rotation of audio like the video, or per-video audio track."* + +## Design arc + +Descending the dial is one journey: **cosmos & abyss** = vast/dark/sparse siblings; +**coast** = warm human middle; **orbit & reef** = transitional membranes. The science +recordings (NASA/NOAA) keep the "machines reshaping perception" thesis literal. + +## Pools + +| altitude | id | clip | source | dur | license | +|---|---|---|---|---|---| +| 🌌 cosmos | `cosmos/pillars` | **✓ KEEP** — Pillars of Creation (M16), twinkly sweep | NASA/CXC/SAO — https://chandra.si.edu/sound/m16.html (`sounds/m16_all.mp4`) | 30s | PD (NASA, credit) | +| 🌌 cosmos | `cosmos/music` | 🎵 **music track — TBD** (deferred). Black-hole drone DROPPED (too creepy, not awe-inspiring); Galactic Center dropped earlier | — | — | — | +| 🛰 orbit | `orbit/spaceamb` | **✓ KEEP** — station room-tone hum ("inside the orbiting craft") | Sonicfreak — https://freesound.org/s/174450/ (`cdn.freesound.org/previews/174/174450_746632-hq.mp3`) | 2:31 | CC0 | +| 🛰 (music) | _deferred_ | 🎵 "Frozen Star" warm pad — moved to MUSIC layer (it's music, not ambience) | Kevin MacLeod / incompetech — https://incompetech.com/music/royalty-free/mp3-royaltyfree/Frozen%20Star.mp3 | 3:41 | CC-BY | +| 🐠 reef | `reef/soundscape` | **✓** richer reef (round 4) — 6-layer bake: submerged underwater bed (bubbles/sloosh) + shrimp crackle + fish chorus + toadfish + grouper + damselfish | NOAA SanctSound (FK+GR `*_snappingshrimp_*`, `FK02_01_fishchorus`, `GR01_01_toadfish`, `FK03_01_redgrouper`, `PM05_01_damselfish`) + DCSFX "Underwater [Loop] AMB" https://freesound.org/s/366159/ (`cdn.freesound.org/previews/366/366159_6725579-hq.mp3`); loop-to-60 + amix + dynaudnorm | 60s | PD (NOAA) + CC0 (underwater) | +| 🏖 coast | `coast/waves` | Sea waves + tern calls | BigSoundBank #0267 — https://bigsoundbank.com/sea-waves-and-seagulls-s0267.html (`/UPLOAD/mp3/0267.mp3`) | 57s | CC0 | +| 🕳 abyss | `abyss/whale` | NE Pacific blue-whale "AB call" | NOAA PMEL — https://www.pmel.noaa.gov/acoustics/multimedia/NETS_bluWhale.wav | 13s | PD (NOAA) | + +## Sourcing rule (learned 2026-06-26) + +**Source only from a browsable page the operator can open and whose license is stated.** +The Mixkit clips were pulled by raw CDN asset id (`assets.mixkit.co/.../-preview.mp3`) +with no findable catalog page — operator couldn't see them, so they were removed. Good +hubs: NASA SVS / Chandra, NOAA PMEL/SanctSound (PD), BigSoundBank (CC0), Pixabay +(visible license). Avoid asset-id grabs and ND/NC licenses. + +## Open items / next pass + +- **orbit** — ✅ SETTLED: `orbit/spaceamb` (Sonicfreak station room-tone hum, CC0). Rejected + en route: laser whistlers, Earth chorus, plain wind, "Frozen Star" pad (music). A real NASA + ISS recording (Hadfield "Space Station Noise", SoundCloud) remains the authentic upgrade if + ever wanted — needs a harder fetch (no yt-dlp; brew install denied). +- **ambience layer COMPLETE** — all 5 picked: cosmos/pillars · orbit/spaceamb · coast/waves · + reef/soundscape · abyss/whale. Next is the production pass + wiring (below) and the music layer. +- **reef** — ✅ round-4 6-layer bake (`reef/soundscape`): NOAA PD biological layers + a CC0 + submerged underwater "you're under" bed (DCSFX freesound #366159, hq preview). License clean. + SanctSound clips: `files/SanctSound_*.mp4` at `https://sanctsound.ioos.us/files/` (extract audio). +- **music layer** — separate composed music track per altitude, deferred (cosmos slot reserved). +- **Production (after picks):** loop seams, length normalization, cross-altitude crossfade + (mirror video); cosmos/abyss clips are short → loop. + +## Re-download / re-bake + +``` +A=simulator/sample_media/audio; FF=$(python -c "import imageio_ffmpeg;print(imageio_ffmpeg.get_ffmpeg_exe())") +mkdir -p $A/cosmos $A/orbit $A/reef $A/coast $A/abyss +# NASA sonifications (mp4 -> mp3) +for u in perseus_sonification:blackhole m16_all:pillars galactic_all:galaxy; do + f=${u%%:*}; o=${u##*:}; curl -fsSL "https://chandra.si.edu/sound/sounds/$f.mp4" -o /tmp/$o.mp4 + "$FF" -y -i /tmp/$o.mp4 -vn -q:a 4 $A/cosmos/$o.mp3; done +curl -fsSL "https://svs.gsfc.nasa.gov/vis/a010000/a011000/a011073/Earthsong-540-MASTER_high.mp4" -o /tmp/es.mp4 +"$FF" -y -i /tmp/es.mp4 -vn -q:a 4 $A/orbit/earthsong.mp3 +curl -fsSL "https://assets.mixkit.co/active_storage/sfx/1162/1162-preview.mp3" -o $A/orbit/wind.mp3 +curl -fsSL "https://www.nhm.ac.uk/content/dam/nhm-www/discover/audio/sound-of-coral-reef.mp3" -o $A/reef/crackle.mp3 +curl -fsSL "https://assets.mixkit.co/active_storage/sfx/1242/1242-preview.mp3" -o $A/reef/scuba.mp3 +curl -fsSL "https://bigsoundbank.com/UPLOAD/mp3/0267.mp3" -o $A/coast/waves.mp3 +curl -fsSL "https://www.pmel.noaa.gov/acoustics/multimedia/NETS_bluWhale.wav" -o $A/abyss/whale.wav +# layered reef +"$FF" -y -i $A/reef/crackle.mp3 -i $A/reef/scuba.mp3 \ + -filter_complex "[1:a]volume=0.55[s];[0:a][s]amix=inputs=2:duration=first,dynaudnorm" -q:a 4 $A/reef/layered.mp3 +``` diff --git a/docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md b/docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md new file mode 100644 index 0000000..2625c91 --- /dev/null +++ b/docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md @@ -0,0 +1,248 @@ +# Audio + Video — separated Visual & Audio controls — design + +**Status:** proposed +**Date:** 2026-06-26 +**Session:** 0020 (audio) +**Extends / reframes:** [`2026-06-26-networked-control-surface-design.md`](./2026-06-26-networked-control-surface-design.md) +§3 (control inventory) and §5 (server contract) — its bundled **Content** selector +is split into two orthogonal controls. Everything else in that spec (server-of-truth ++ SSE, the two pages, the Arduino seam, transition lifecycle) stands unchanged. +**Asset record:** [`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md). + +--- + +## 1. Goal + +Give the experience **sound**, and make audio and video **independently +controllable**. Today the control surface bundles audio+video into a single 7-way +**Content** selector (*video / audio+video / music+video / off / white noise / +music / audio track*). That bundle is both awkward to operate and **incomplete** — +it has no "white noise **with** video," for instance. + +Replace it with **two orthogonal dials** — **Visual** and **Audio** — so every +combination is reachable, including the ones the bundle omitted. Operator framing +(2026-06-26): *"separate these dials… it's ok to let people have e.g. white noise +with video."* + +## 2. Core idea: orthogonalize Content into Visual × Audio + +``` + BEFORE (bundled) AFTER (orthogonal) + Content: 7-way selector Visual: on / off + video Audio: off / soundtrack / white noise + audio+video (· music — deferred) + music+video ──▶ ─────────────────────────────────────── + off Any Visual × Any Audio is now reachable: + white noise video + soundtrack (was "audio+video") + music video + white noise (NEW — was missing) + audio track black + white noise (was "white noise") + black + soundtrack (was "audio track") +``` + +The 7 bundled modes were really points in a **2 × N grid** (visual on/off × +audio source). Making the grid explicit removes the awkwardness and fills the gap. +The **Altitude** dial is unchanged and remains the single scale selector. + +## 3. The controls (reframes control-surface §3) + +| Control | Positions (v1) | Drives | Class (§4 of control-surface spec) | +|---|---|---|---| +| **Altitude** | endless dial, cosmos→abyss (wraps) | the visual **scale** **and**, when Audio=soundtrack, the matching ambience | transitional | +| **Visual** | on / off | video shown vs. black | transitional (video fade) | +| **Audio** | off / soundtrack / white noise (· *music* deferred) | the audio **source** | live + short crossfade (§7) | +| **Left** / **Right** / **Mood** | unchanged | the alteration engine | live | + +So the panel grows from 5 controls to **6** (Content → Visual + Audio). The +physical form (later): Visual = a toggle/2-pos switch; Audio = a small rotary +selector (3-pos in v1, 4 with music). + +## 4. Audio model (the coupling decision) + +**Decision (operator):** the audio dial is a **source** selector; the per-altitude +soundtrack **follows the single Altitude dial**. There is **no** separate audio +altitude. + +- **`soundtrack`** → plays the ambience of the **current Altitude scale**. Turning + Altitude to coast shows coast video *and* plays coast waves; the soundtrack + **crossfades** as the altitude transition settles (mirrors the video ring + crossfade). One dial, two coupled outputs. +- **`white noise`** → a single global bed, **altitude-independent**. Changing + Altitude does not change it. This is what makes "white noise + any video" work. +- **`off`** → silence. Video (if Visual=on) plays mute. +- **`music`** *(deferred)* → the composed per-altitude music layer. Reserved as a + future Audio position; **not in v1** (no assets yet — see + [`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md) "two audio layers"). + +Independence summary: **Visual ⟂ Audio** (any pairing allowed); **Audio source ⟂ +Altitude** for white noise/music, **coupled to Altitude** for soundtrack. + +## 5. Assets & manifest + +### 5.1 Per-scale soundtrack + +The five ambiences are sourced and reviewed (PD/CC0; record in +[`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md), local-only review +gallery `simulator/static/review_audio.html`): + +| scale | soundtrack | license | +|---|---|---| +| cosmos | Pillars of Creation sonification | PD (NASA) | +| orbit | station room-tone hum | CC0 | +| coast | sea waves + terns | CC0 | +| reef | 6-layer reef soundscape | PD (NOAA) + CC0 | +| abyss | blue-whale moan | PD (NOAA) | + +The manifest's `Scale` gains an **`audio`** field (path to the scale's soundtrack), +parallel to how clips are referenced; media stays gitignored and is served at +`/media/audio//`. Soundtracks are a **per-scale single** (not a +rotating pool) in v1 — one bed per altitude. (A rotating audio pool, like the clip +pool, is a possible later extension; out of v1.) + +### 5.2 White noise + +Altitude-independent global asset. **Synthesized, not sourced** — a calm +**pink/brown** noise (gentler than pure white) generated deterministically with +`ffmpeg -f lavfi -i anoisesrc=color=pink` → a clean ~60 s loop. Zero licensing +concerns. (Colloquially "white noise"; spec a calm colored noise.) + +### 5.3 Production pass (assets) + +Before ship the source clips get a light, deterministic pass (the existing +`tools/pipeline` ffmpeg approach): **seamless loop** (crossfade the loop point), +**normalize loudness** across the five so altitudes match, and a consistent format +(mp3/ogg). Several are short (abyss 13 s, cosmos 30 s) and must loop cleanly. The +6-layer reef bake is already loop-length (60 s). + +## 6. Server contract (reframes control-surface §5) + +The control-surface spec's authoritative `SessionState` + four endpoints are +unchanged in shape; only the **controls** payload changes: `content` is replaced +by `visual` + `audio`, and the derived `render.content` is split. + +### 6.1 `GET /api/state` — updated `controls` + `render` + +```json +{ + "seq": 42, + "controls": { "visual": "on", "audio": "soundtrack", "left": 3, "right": 1, "mood": -2 }, + "altitude": { "index": 2, "scale": "coast", "clip_id": "coast_07" }, + "transitions": { "altitude": "idle", "visual": "idle" }, + "render": { + "plan": { }, + "video": { "shown": true }, + "audio": { "source": "soundtrack", "url": "/media/audio/coast/waves.mp3", "altitude_coupled": true } + } +} +``` + +- `audio.url` is resolved **server-side** from `(audio source, altitude)`: + `soundtrack` → the current scale's `audio`; `white_noise` → the global noise + asset; `off` → `null`. The renderer just plays `audio.url` (or nothing). +- `transitions` tracks `visual` instead of `content` (Audio is not transitional — + §7). + +### 6.2 `POST /api/control` + +```json +{ "set": { "audio": "white_noise" } } // audio source change +{ "set": { "visual": "off" } } // visual fade to black +{ "set": {}, "altitude_delta": -1 } // altitude tick (also re-resolves audio.url when source=soundtrack) +``` + +Server applies, bumps `seq`, recomputes `render.audio.url`, and broadcasts. +Absolute `set` values keep retries idempotent (control-surface §5.6). + +### 6.3 SSE events + +- `state` — carries the new controls/render; emitted on Audio change and on any + live change (Audio source change rides the `state` event — §7). +- `ring` — altitude move; now **also** carries the new `audio.url` so the renderer + can crossfade the soundtrack as it plays the visual transition (when + source=soundtrack). +- `mode` — repurposed to the **Visual** fade (on/off). (Same renderer fade machinery + the bundled Content used.) + +## 7. Live vs. transitional classification + +- **Visual on/off → transitional.** It's a visible video fade (to/from black); it + uses the renderer's existing fade + `settled` ack, exactly as the bundled + Content's fade did. `transitions.visual` flips `transitioning`→`idle`. +- **Audio source → live, with a short renderer-side crossfade.** Swapping + soundtrack/white-noise/off is a ~0.5–1 s **gain crossfade** between two `