feat(audio): per-altitude soundtracks + separated audio/visual controls spec

Sound for the experience, plus the design to control it. Soundtrack layer (ambience), all license-clean (PD/CC0), recorded in docs/audio-candidate-pool.md (media gitignored; re-sourceable): - cosmos: Pillars of Creation sonification (NASA, PD) - orbit: station room-tone hum (CC0) - coast: sea waves + terns (CC0) - reef: 6-layer NOAA soundscape + CC0 submerged bed (PD + CC0) - abyss: blue-whale moan (NOAA, PD) Solution Design (proposed): split the bundled Content selector into two orthogonal dials, Visual (on/off) x Audio (off/soundtrack/white-noise), unlocking white-noise+video. Soundtrack follows the single Altitude dial; white noise is altitude-independent. Reframes the networked-control-surface spec SS3/SS5 (still proposed/unbuilt -> clean pre-impl edit). Music layer deferred. Dev panel: links to the clip + audio review galleries; sticky stage/panel layout. gitignore: audio media, review_audio.html, .DS_Store. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 06:37:56 -07:00
parent f28a06cc7f
commit a3ec09af7e
5 changed files with 365 additions and 4 deletions
@@ -0,0 +1,248 @@
+# Audio + Video — separated Visual & Audio controls — design
+
+**Status:** proposed
+**Date:** 2026-06-26
+**Session:** 0020 (audio)
+**Extends / reframes:** [`2026-06-26-networked-control-surface-design.md`](./2026-06-26-networked-control-surface-design.md)
+§3 (control inventory) and §5 (server contract) — its bundled **Content** selector
+is split into two orthogonal controls. Everything else in that spec (server-of-truth
+ SSE, the two pages, the Arduino seam, transition lifecycle) stands unchanged.
+**Asset record:** [`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md).
+
+---
+
+## 1. Goal
+
+Give the experience **sound**, and make audio and video **independently
+controllable**. Today the control surface bundles audio+video into a single 7-way
+**Content** selector (*video / audio+video / music+video / off / white noise /
+music / audio track*). That bundle is both awkward to operate and **incomplete** —
+it has no "white noise **with** video," for instance.
+
+Replace it with **two orthogonal dials** — **Visual** and **Audio** — so every
+combination is reachable, including the ones the bundle omitted. Operator framing
+(2026-06-26): *"separate these dials… it's ok to let people have e.g. white noise
+with video."*
+
+## 2. Core idea: orthogonalize Content into Visual × Audio
+
+```
+        BEFORE (bundled)                      AFTER (orthogonal)
+  Content: 7-way selector            Visual: on / off
+   video                             Audio:  off / soundtrack / white noise
+   audio+video                                        (· music — deferred)
+   music+video              ──▶      ───────────────────────────────────────
+   off                               Any Visual × Any Audio is now reachable:
+   white noise                         video + soundtrack   (was "audio+video")
+   music                               video + white noise  (NEW — was missing)
+   audio track                         black + white noise  (was "white noise")
+                                       black + soundtrack   (was "audio track")
+```
+
+The 7 bundled modes were really points in a **2 × N grid** (visual on/off ×
+audio source). Making the grid explicit removes the awkwardness and fills the gap.
+The **Altitude** dial is unchanged and remains the single scale selector.
+
+## 3. The controls (reframes control-surface §3)
+
+| Control | Positions (v1) | Drives | Class (§4 of control-surface spec) |
+|---|---|---|---|
+| **Altitude** | endless dial, cosmos→abyss (wraps) | the visual **scale** **and**, when Audio=soundtrack, the matching ambience | transitional |
+| **Visual** | on / off | video shown vs. black | transitional (video fade) |
+| **Audio** | off / soundtrack / white noise (· *music* deferred) | the audio **source** | live + short crossfade (§7) |
+| **Left** / **Right** / **Mood** | unchanged | the alteration engine | live |
+
+So the panel grows from 5 controls to **6** (Content → Visual + Audio). The
+physical form (later): Visual = a toggle/2-pos switch; Audio = a small rotary
+selector (3-pos in v1, 4 with music).
+
+## 4. Audio model (the coupling decision)
+
+**Decision (operator):** the audio dial is a **source** selector; the per-altitude
+soundtrack **follows the single Altitude dial**. There is **no** separate audio
+altitude.
+
+- **`soundtrack`** → plays the ambience of the **current Altitude scale**. Turning
+  Altitude to coast shows coast video *and* plays coast waves; the soundtrack
+  **crossfades** as the altitude transition settles (mirrors the video ring
+  crossfade). One dial, two coupled outputs.
+- **`white noise`** → a single global bed, **altitude-independent**. Changing
+  Altitude does not change it. This is what makes "white noise + any video" work.
+- **`off`** → silence. Video (if Visual=on) plays mute.
+- **`music`** *(deferred)* → the composed per-altitude music layer. Reserved as a
+  future Audio position; **not in v1** (no assets yet — see
+  [`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md) "two audio layers").
+
+Independence summary: **Visual ⟂ Audio** (any pairing allowed); **Audio source ⟂
+Altitude** for white noise/music, **coupled to Altitude** for soundtrack.
+
+## 5. Assets & manifest
+
+### 5.1 Per-scale soundtrack
+
+The five ambiences are sourced and reviewed (PD/CC0; record in
+[`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md), local-only review
+gallery `simulator/static/review_audio.html`):
+
+| scale | soundtrack | license |
+|---|---|---|
+| cosmos | Pillars of Creation sonification | PD (NASA) |
+| orbit | station room-tone hum | CC0 |
+| coast | sea waves + terns | CC0 |
+| reef | 6-layer reef soundscape | PD (NOAA) + CC0 |
+| abyss | blue-whale moan | PD (NOAA) |
+
+The manifest's `Scale` gains an **`audio`** field (path to the scale's soundtrack),
+parallel to how clips are referenced; media stays gitignored and is served at
+`/media/audio/<scale>/<file>`. Soundtracks are a **per-scale single** (not a
+rotating pool) in v1 — one bed per altitude. (A rotating audio pool, like the clip
+pool, is a possible later extension; out of v1.)
+
+### 5.2 White noise
+
+Altitude-independent global asset. **Synthesized, not sourced** — a calm
+**pink/brown** noise (gentler than pure white) generated deterministically with
+`ffmpeg -f lavfi -i anoisesrc=color=pink` → a clean ~60 s loop. Zero licensing
+concerns. (Colloquially "white noise"; spec a calm colored noise.)
+
+### 5.3 Production pass (assets)
+
+Before ship the source clips get a light, deterministic pass (the existing
+`tools/pipeline` ffmpeg approach): **seamless loop** (crossfade the loop point),
+**normalize loudness** across the five so altitudes match, and a consistent format
+(mp3/ogg). Several are short (abyss 13 s, cosmos 30 s) and must loop cleanly. The
+6-layer reef bake is already loop-length (60 s).
+
+## 6. Server contract (reframes control-surface §5)
+
+The control-surface spec's authoritative `SessionState` + four endpoints are
+unchanged in shape; only the **controls** payload changes: `content` is replaced
+by `visual` + `audio`, and the derived `render.content` is split.
+
+### 6.1 `GET /api/state` — updated `controls` + `render`
+
+```json
+{
+  "seq": 42,
+  "controls": { "visual": "on", "audio": "soundtrack", "left": 3, "right": 1, "mood": -2 },
+  "altitude": { "index": 2, "scale": "coast", "clip_id": "coast_07" },
+  "transitions": { "altitude": "idle", "visual": "idle" },
+  "render": {
+    "plan": { },
+    "video":  { "shown": true },
+    "audio":  { "source": "soundtrack", "url": "/media/audio/coast/waves.mp3", "altitude_coupled": true }
+  }
+}
+```
+
+- `audio.url` is resolved **server-side** from `(audio source, altitude)`:
+  `soundtrack` → the current scale's `audio`; `white_noise` → the global noise
+  asset; `off` → `null`. The renderer just plays `audio.url` (or nothing).
+- `transitions` tracks `visual` instead of `content` (Audio is not transitional —
+  §7).
+
+### 6.2 `POST /api/control`
+
+```json
+{ "set": { "audio": "white_noise" } }   // audio source change
+{ "set": { "visual": "off" } }          // visual fade to black
+{ "set": {}, "altitude_delta": -1 }     // altitude tick (also re-resolves audio.url when source=soundtrack)
+```
+
+Server applies, bumps `seq`, recomputes `render.audio.url`, and broadcasts.
+Absolute `set` values keep retries idempotent (control-surface §5.6).
+
+### 6.3 SSE events
+
+- `state` — carries the new controls/render; emitted on Audio change and on any
+  live change (Audio source change rides the `state` event — §7).
+- `ring` — altitude move; now **also** carries the new `audio.url` so the renderer
+  can crossfade the soundtrack as it plays the visual transition (when
+  source=soundtrack).
+- `mode` — repurposed to the **Visual** fade (on/off). (Same renderer fade machinery
+  the bundled Content used.)
+
+## 7. Live vs. transitional classification
+
+- **Visual on/off → transitional.** It's a visible video fade (to/from black); it
+  uses the renderer's existing fade + `settled` ack, exactly as the bundled
+  Content's fade did. `transitions.visual` flips `transitioning`→`idle`.
+- **Audio source → live, with a short renderer-side crossfade.** Swapping
+  soundtrack/white-noise/off is a ~0.5–1 s **gain crossfade** between two `<audio>`
+  elements — perceptible but **non-blocking** and not a renderer animation with
+  unknown timing. So it's classified **live** (no busy spinner, no `settled` ack),
+  like Left/Right/Mood. *Flagged decision:* if the operator wants the Audio knob to
+  feel "busy" during its fade, it can be promoted to transitional later; v1 keeps it
+  live for simplicity.
+- **Altitude → transitional** (unchanged). When source=soundtrack, the renderer
+  crossfades the audio bed in step with the ring transition; the audio crossfade is
+  slaved to the visual transition and needs no separate ack.
+
+## 8. Renderer behavior (extends control-surface §6.1)
+
+The display page adds an **audio layer**:
+
+- Two `<audio>` elements (A/B) for gapless gain crossfades; a tiny crossfade helper.
+- On `state`: reconcile `audio.url` — if it changed, crossfade A↔B to the new url
+  (or fade to silence for `off`). White-noise and soundtrack are the same mechanism;
+  only the url differs.
+- On `ring` (altitude move) with source=soundtrack: start the soundtrack crossfade
+  to the event's new `audio.url` so it lands with the visual scale.
+- On `mode` (Visual): fade the **video** to/from black; audio is untouched (you can
+  hear soundtrack/white-noise over black).
+- Each soundtrack loops (the production-pass clean loop); white-noise loops.
+- Autoplay: the renderer is launched full-screen by the operator; a one-time user
+  gesture (or `--autoplay`/muted-then-unmute) satisfies browser autoplay policy.
+  *Flagged:* document the launch gesture so the projection starts audio reliably.
+
+## 9. Testing (mandatory pipeline tiers)
+
+- **Pure unit** — `render.audio.url` resolution from `(source, altitude)`:
+  soundtrack→scale asset, white_noise→global, off→null; altitude tick re-resolves
+  only when source=soundtrack; `seq` bump. No I/O.
+- **API/contract** — `POST /api/control {visual}` flips `transitions.visual`;
+  `{audio}` rides a `state` event with the new url and **no** transition flag; bad
+  enum → 422; `GET /api/state` shape.
+- **E2E (Playwright)** — renderer + `/remote`: set Audio=white_noise → assert the
+  renderer's audio element src is the noise asset; tick Altitude with
+  Audio=soundtrack → assert the soundtrack url crossfades to the new scale; set
+  Visual=off → assert the video fades and audio keeps playing. (Audio *playback*
+  asserted via element `src`/`paused`, not waveform.)
+
+## 10. Scope
+
+**In v1:**
+
+- Split **Content** → **Visual** (on/off) + **Audio** (off / soundtrack / white
+  noise) across `SessionState`, `/api/state`, `/api/control`, SSE, and both pages.
+- Per-scale `audio` in the manifest + the 5 sourced soundtracks (production-passed
+  loops); synthesized white-noise asset.
+- Server-side `audio.url` resolution; renderer audio layer with crossfades;
+  altitude-coupled soundtrack crossfade.
+
+**Explicitly NOT in v1 (deferred):**
+
+- **Music layer** (`audio=music`) — reserved dial position; no assets yet.
+- **Separate audio altitude** (hear reef while seeing coast) — rejected coupling
+  option; soundtrack follows the one Altitude dial.
+- **Rotating audio pools** per scale (one bed per altitude in v1).
+- Per-altitude white-noise variants (one global noise).
+- Physical Visual/Audio switches (Arduino) — contract only, per control-surface §7.
+
+## 11. Spec & roadmap impact
+
+- **Reframes** control-surface §3/§5: `content` → `visual` + `audio`;
+  `transitions.content` → `transitions.visual`; `render.content` → `render.video`
+  + `render.audio`. That spec is still *proposed* (unbuilt), so this is a clean
+  pre-implementation edit, not a migration. Both can be built together, or this
+  one folds into the control-surface build as its Content step.
+- The **music layer** becomes a tracked follow-on (compose + source per-altitude
+  music; flip on the reserved Audio position).
+- `ROADMAP.md` to note the audio capability at implementation time.
+
+## 12. Open questions / flagged
+
+- **Audio knob feel** (§7) — live vs. transitional; v1 = live. Revisit on the iPad.
+- **White-noise color** — pink vs. brown vs. true white; v1 = pink/brown (calmer).
+- **Autoplay gesture** (§8) — document the launch step so the projection starts
+  audio without manual unmuting.