# Audio + Video — separated Visual & Audio controls — design

**Status:** proposed → built (sessions 0020 build + 0021 revision)
**Date:** 2026-06-26
**Session:** 0020 (audio) · 0021 (live-UI revision)

> **v1 build revision (session 0021, operator-driven):** the live **Audio** control
> ships as a simple **off / soundtrack** toggle (styled like the Dev Mode switch),
> and **Video** is the matching on/off toggle. **White-noise is deferred** with music
> (the orthogonal Visual × Audio model and the synthesized-pink-noise machinery
> remain in the codebase for when it's wanted). Two real-browser bugs were fixed in
> this revision: video-off now blanks GPU-composited layers (z-index + explicit layer
> hide), and the soundtrack now plays on Safari/iOS (audio elements unlocked
> synchronously inside the start gesture). Sections below describe the fuller
> off/soundtrack/white-noise design; v1 live = off/soundtrack.
>
> **Startup revision (session 0022):** the tap-to-start wall is **removed** — the app
> opens behind a **"Loading Universe…"** splash (shown until all media is preloaded
> and the experience can run smoothly), then both toggles start **off** (black +
> silent). Autoplay is satisfied without a wall: a **single** `<audio>` element is
> played **synchronously inside the toggle's own click** (the reliable Safari unlock),
> replacing the A/B-crossfade-after-gesture approach that failed on real Safari. The
> first time **Video** is turned on, **Audio** turns on with it (one flip = the full
> experience); turning Audio on first stays audio-only. Altitude changes swap the
> single element's source with a brief fade.
**Extends / reframes:** [`2026-06-26-networked-control-surface-design.md`](./2026-06-26-networked-control-surface-design.md)
§3 (control inventory) and §5 (server contract) — its bundled **Content** selector
is split into two orthogonal controls. Everything else in that spec (server-of-truth
+ SSE, the two pages, the Arduino seam, transition lifecycle) stands unchanged.
**Asset record:** [`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md).

---

## 1. Goal

Give the experience **sound**, and make audio and video **independently
controllable**. Today the control surface bundles audio+video into a single 7-way
**Content** selector (*video / audio+video / music+video / off / white noise /
music / audio track*). That bundle is both awkward to operate and **incomplete** —
it has no "white noise **with** video," for instance.

Replace it with **two orthogonal dials** — **Visual** and **Audio** — so every
combination is reachable, including the ones the bundle omitted. Operator framing
(2026-06-26): *"separate these dials… it's ok to let people have e.g. white noise
with video."*

## 2. Core idea: orthogonalize Content into Visual × Audio

```
        BEFORE (bundled)                      AFTER (orthogonal)
  Content: 7-way selector            Visual: on / off
   video                             Audio:  off / soundtrack / white noise
   audio+video                                        (· music — deferred)
   music+video              ──▶      ───────────────────────────────────────
   off                               Any Visual × Any Audio is now reachable:
   white noise                         video + soundtrack   (was "audio+video")
   music                               video + white noise  (NEW — was missing)
   audio track                         black + white noise  (was "white noise")
                                       black + soundtrack   (was "audio track")
```

The 7 bundled modes were really points in a **2 × N grid** (visual on/off ×
audio source). Making the grid explicit removes the awkwardness and fills the gap.
The **Altitude** dial is unchanged and remains the single scale selector.

## 3. The controls (reframes control-surface §3)

| Control | Positions (v1) | Drives | Class (§4 of control-surface spec) |
|---|---|---|---|
| **Altitude** | endless dial, cosmos→abyss (wraps) | the visual **scale** **and**, when Audio=soundtrack, the matching ambience | transitional |
| **Visual** | on / off | video shown vs. black | transitional (video fade) |
| **Audio** | off / soundtrack / white noise (· *music* deferred) | the audio **source** | live + short crossfade (§7) |
| **Left** / **Right** / **Mood** | unchanged | the alteration engine | live |

So the panel grows from 5 controls to **6** (Content → Visual + Audio). The
physical form (later): Visual = a toggle/2-pos switch; Audio = a small rotary
selector (3-pos in v1, 4 with music).

## 4. Audio model (the coupling decision)

**Decision (operator):** the audio dial is a **source** selector; the per-altitude
soundtrack **follows the single Altitude dial**. There is **no** separate audio
altitude.

- **`soundtrack`** → plays the ambience of the **current Altitude scale**. Turning
  Altitude to coast shows coast video *and* plays coast waves; the soundtrack
  **crossfades** as the altitude transition settles (mirrors the video ring
  crossfade). One dial, two coupled outputs.
- **`white noise`** → a single global bed, **altitude-independent**. Changing
  Altitude does not change it. This is what makes "white noise + any video" work.
- **`off`** → silence. Video (if Visual=on) plays mute.
- **`music`** *(deferred)* → the composed per-altitude music layer. Reserved as a
  future Audio position; **not in v1** (no assets yet — see
  [`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md) "two audio layers").

Independence summary: **Visual ⟂ Audio** (any pairing allowed); **Audio source ⟂
Altitude** for white noise/music, **coupled to Altitude** for soundtrack.

## 5. Assets & manifest

### 5.1 Per-scale soundtrack

The five ambiences are sourced and reviewed (PD/CC0; record in
[`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md), local-only review
gallery `simulator/static/review_audio.html`):

| scale | soundtrack | license |
|---|---|---|
| cosmos | Pillars of Creation sonification | PD (NASA) |
| orbit | station room-tone hum | CC0 |
| coast | sea waves + terns | CC0 |
| reef | 6-layer reef soundscape | PD (NOAA) + CC0 |
| abyss | blue-whale moan | PD (NOAA) |

The manifest's `Scale` gains an **`audio`** field (path to the scale's soundtrack),
parallel to how clips are referenced; media stays gitignored and is served at
`/media/audio/<scale>/<file>`. Soundtracks are a **per-scale single** (not a
rotating pool) in v1 — one bed per altitude. (A rotating audio pool, like the clip
pool, is a possible later extension; out of v1.)

### 5.2 White noise

Altitude-independent global asset. **Synthesized, not sourced** — a calm
**pink/brown** noise (gentler than pure white) generated deterministically with
`ffmpeg -f lavfi -i anoisesrc=color=pink` → a clean ~60 s loop. Zero licensing
concerns. (Colloquially "white noise"; spec a calm colored noise.)

### 5.3 Production pass (assets)

Before ship the source clips get a light, deterministic pass (the existing
`tools/pipeline` ffmpeg approach): **seamless loop** (crossfade the loop point),
**normalize loudness** across the five so altitudes match, and a consistent format
(mp3/ogg). Several are short (abyss 13 s, cosmos 30 s) and must loop cleanly. The
6-layer reef bake is already loop-length (60 s).

## 6. Server contract (reframes control-surface §5)

The control-surface spec's authoritative `SessionState` + four endpoints are
unchanged in shape; only the **controls** payload changes: `content` is replaced
by `visual` + `audio`, and the derived `render.content` is split.

### 6.1 `GET /api/state` — updated `controls` + `render`

```json
{
  "seq": 42,
  "controls": { "visual": "on", "audio": "soundtrack", "left": 3, "right": 1, "mood": -2 },
  "altitude": { "index": 2, "scale": "coast", "clip_id": "coast_07" },
  "transitions": { "altitude": "idle", "visual": "idle" },
  "render": {
    "plan": { },
    "video":  { "shown": true },
    "audio":  { "source": "soundtrack", "url": "/media/audio/coast/waves.mp3", "altitude_coupled": true }
  }
}
```

- `audio.url` is resolved **server-side** from `(audio source, altitude)`:
  `soundtrack` → the current scale's `audio`; `white_noise` → the global noise
  asset; `off` → `null`. The renderer just plays `audio.url` (or nothing).
- `transitions` tracks `visual` instead of `content` (Audio is not transitional —
  §7).

### 6.2 `POST /api/control`

```json
{ "set": { "audio": "white_noise" } }   // audio source change
{ "set": { "visual": "off" } }          // visual fade to black
{ "set": {}, "altitude_delta": -1 }     // altitude tick (also re-resolves audio.url when source=soundtrack)
```

Server applies, bumps `seq`, recomputes `render.audio.url`, and broadcasts.
Absolute `set` values keep retries idempotent (control-surface §5.6).

### 6.3 SSE events

- `state` — carries the new controls/render; emitted on Audio change and on any
  live change (Audio source change rides the `state` event — §7).
- `ring` — altitude move; now **also** carries the new `audio.url` so the renderer
  can crossfade the soundtrack as it plays the visual transition (when
  source=soundtrack).
- `mode` — repurposed to the **Visual** fade (on/off). (Same renderer fade machinery
  the bundled Content used.)

## 7. Live vs. transitional classification

- **Visual on/off → transitional.** It's a visible video fade (to/from black); it
  uses the renderer's existing fade + `settled` ack, exactly as the bundled
  Content's fade did. `transitions.visual` flips `transitioning`→`idle`.
- **Audio source → live, with a short renderer-side crossfade.** Swapping
  soundtrack/white-noise/off is a ~0.5–1 s **gain crossfade** between two `<audio>`
  elements — perceptible but **non-blocking** and not a renderer animation with
  unknown timing. So it's classified **live** (no busy spinner, no `settled` ack),
  like Left/Right/Mood. *Flagged decision:* if the operator wants the Audio knob to
  feel "busy" during its fade, it can be promoted to transitional later; v1 keeps it
  live for simplicity.
- **Altitude → transitional** (unchanged). When source=soundtrack, the renderer
  crossfades the audio bed in step with the ring transition; the audio crossfade is
  slaved to the visual transition and needs no separate ack.

## 8. Renderer behavior (extends control-surface §6.1)

The display page adds an **audio layer**:

- Two `<audio>` elements (A/B) for gapless gain crossfades; a tiny crossfade helper.
- On `state`: reconcile `audio.url` — if it changed, crossfade A↔B to the new url
  (or fade to silence for `off`). White-noise and soundtrack are the same mechanism;
  only the url differs.
- On `ring` (altitude move) with source=soundtrack: start the soundtrack crossfade
  to the event's new `audio.url` so it lands with the visual scale.
- On `mode` (Visual): fade the **video** to/from black; audio is untouched (you can
  hear soundtrack/white-noise over black).
- Each soundtrack loops (the production-pass clean loop); white-noise loops.
- Autoplay: the renderer is launched full-screen by the operator; a one-time user
  gesture (or `--autoplay`/muted-then-unmute) satisfies browser autoplay policy.
  *Flagged:* document the launch gesture so the projection starts audio reliably.

## 9. Testing (mandatory pipeline tiers)

- **Pure unit** — `render.audio.url` resolution from `(source, altitude)`:
  soundtrack→scale asset, white_noise→global, off→null; altitude tick re-resolves
  only when source=soundtrack; `seq` bump. No I/O.
- **API/contract** — `POST /api/control {visual}` flips `transitions.visual`;
  `{audio}` rides a `state` event with the new url and **no** transition flag; bad
  enum → 422; `GET /api/state` shape.
- **E2E (Playwright)** — renderer + `/remote`: set Audio=white_noise → assert the
  renderer's audio element src is the noise asset; tick Altitude with
  Audio=soundtrack → assert the soundtrack url crossfades to the new scale; set
  Visual=off → assert the video fades and audio keeps playing. (Audio *playback*
  asserted via element `src`/`paused`, not waveform.)

## 10. Scope

**In v1:**

- Split **Content** → **Visual** (on/off) + **Audio** (off / soundtrack / white
  noise) across `SessionState`, `/api/state`, `/api/control`, SSE, and both pages.
- Per-scale `audio` in the manifest + the 5 sourced soundtracks (production-passed
  loops); synthesized white-noise asset.
- Server-side `audio.url` resolution; renderer audio layer with crossfades;
  altitude-coupled soundtrack crossfade.

**Explicitly NOT in v1 (deferred):**

- **Music layer** (`audio=music`) — reserved dial position; no assets yet.
- **Separate audio altitude** (hear reef while seeing coast) — rejected coupling
  option; soundtrack follows the one Altitude dial.
- **Rotating audio pools** per scale (one bed per altitude in v1).
- Per-altitude white-noise variants (one global noise).
- Physical Visual/Audio switches (Arduino) — contract only, per control-surface §7.

## 11. Spec & roadmap impact

- **Reframes** control-surface §3/§5: `content` → `visual` + `audio`;
  `transitions.content` → `transitions.visual`; `render.content` → `render.video`
  + `render.audio`. That spec is still *proposed* (unbuilt), so this is a clean
  pre-implementation edit, not a migration. Both can be built together, or this
  one folds into the control-surface build as its Content step.
- The **music layer** becomes a tracked follow-on (compose + source per-altitude
  music; flip on the reserved Audio position).
- `ROADMAP.md` to note the audio capability at implementation time.

## 12. Open questions / flagged — RESOLVED (session 0020 build)

- **Audio knob feel** (§7) — **RESOLVED: live**, with a ~0.6 s renderer-side gain
  crossfade (`XFADE_MS=600`), no `settled` ack. Promote to transitional later only
  if the iPad makes it feel like it needs a "busy" state.
- **White-noise color** — **RESOLVED: pink** (`anoisesrc=color=pink`), a balanced
  calm bed. Brown is a one-parameter alternative (`color=brown`) if a deeper rumble
  is wanted; the build is parameterized for it.
- **Autoplay gesture** (§8) — **RESOLVED: a one-time tap-to-start overlay**
  ("▶ tap to start sound", `#start-gesture`). The first click unlocks both `<audio>`
  elements within the user gesture; thereafter audio follows state. This is the
  documented launch step for the full-screen projection.
