From fd0aad66102b9a2a83bfff6d2bf1b8f3eec8b0ae Mon Sep 17 00:00:00 2001
From: BenStullsBets <ben@benbetson.us>
Date: Fri, 26 Jun 2026 07:50:34 -0700
Subject: [PATCH] =?UTF-8?q?docs(audio):=20reframe=20control-surface=20spec?=
 =?UTF-8?q?=20content->visual+audio;=20record=20=C2=A712=20decisions=20+?=
 =?UTF-8?q?=20roadmap?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

- networked-control-surface §3/§4/§5/§6: Content -> Visual + Audio; transitions.content
  -> transitions.visual; render.content -> render.video + render.audio; SSE notes
- audio spec §12: resolved (audio=live ~0.6s; noise=pink; tap-to-start overlay);
  status -> in implementation
- ROADMAP: session-0020 audio increment; music layer deferred

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 docs/ROADMAP.md                               |  17 ++-
 ...6-audio-video-separated-controls-design.md |  18 ++--
 ...-06-26-networked-control-surface-design.md | 102 +++++++++++-------
 3 files changed, 89 insertions(+), 48 deletions(-)

diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
index dea5dac..eb2ea6d 100644
--- a/docs/ROADMAP.md
+++ b/docs/ROADMAP.md
@@ -196,8 +196,23 @@ every transition, so fast navigation stays responsive on placeholder media.
     browser. Real strict-PD bases for all five scales sourced per
     [`docs/content-sourcing.md`](./content-sourcing.md).
 
+  - **Audio + separated Visual/Audio controls (session 0020) ✅ Done.** The
+    simulator now has **sound**: per-altitude soundtracks that follow the Altitude
+    dial + a synthesized pink **white-noise** bed, behind two orthogonal controls —
+    **Visual** (on/off) × **Audio** (off / soundtrack / white_noise) — replacing
+    the bundled 7-way Content dial. `tools/pipeline/audio_ops` + `audio_run` +
+    `simulator/build_audio_media.py` (ffmpeg loop/loudnorm/noise, gitignored
+    assets), `Scale.audio` in the manifest, a pure `player/audio.py` resolver
+    (retiring `player/content.py`), server-resolved `render.audio.url`, and a client
+    A/B `<audio>` crossfade layer. This is the "Content step" of the future control
+    surface (it reframes the proposed networked-control-surface spec §3/§5). Spec:
+    [`2026-06-26-audio-video-separated-controls-design.md`](./superpowers/specs/2026-06-26-audio-video-separated-controls-design.md);
+    plan:
+    [`2026-06-26-audio-video-separated-controls.md`](./superpowers/plans/2026-06-26-audio-video-separated-controls.md).
+
   - **Still deferred:** real **i2v ring transitions** (need a generative-video
-    model + adjacent real bases); Pi renderer + serial/firmware.
+    model + adjacent real bases); Pi renderer + serial/firmware; the **music**
+    audio layer (reserved dial position, no assets yet).
 - **Catalog model changes** — audio *source* + "neutral base" vs "altered
   variant" flag (sub-project 2 territory, design §13).
 
diff --git a/docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md b/docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md
index 2625c91..6c0a456 100644
--- a/docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md
+++ b/docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md
@@ -1,6 +1,6 @@
 # Audio + Video — separated Visual & Audio controls — design
 
-**Status:** proposed
+**Status:** proposed → in implementation (plan `docs/superpowers/plans/2026-06-26-audio-video-separated-controls.md`, session 0020)
 **Date:** 2026-06-26
 **Session:** 0020 (audio)
 **Extends / reframes:** [`2026-06-26-networked-control-surface-design.md`](./2026-06-26-networked-control-surface-design.md)
@@ -240,9 +240,15 @@ The display page adds an **audio layer**:
   music; flip on the reserved Audio position).
 - `ROADMAP.md` to note the audio capability at implementation time.
 
-## 12. Open questions / flagged
+## 12. Open questions / flagged — RESOLVED (session 0020 build)
 
-- **Audio knob feel** (§7) — live vs. transitional; v1 = live. Revisit on the iPad.
-- **White-noise color** — pink vs. brown vs. true white; v1 = pink/brown (calmer).
-- **Autoplay gesture** (§8) — document the launch step so the projection starts
-  audio without manual unmuting.
+- **Audio knob feel** (§7) — **RESOLVED: live**, with a ~0.6 s renderer-side gain
+  crossfade (`XFADE_MS=600`), no `settled` ack. Promote to transitional later only
+  if the iPad makes it feel like it needs a "busy" state.
+- **White-noise color** — **RESOLVED: pink** (`anoisesrc=color=pink`), a balanced
+  calm bed. Brown is a one-parameter alternative (`color=brown`) if a deeper rumble
+  is wanted; the build is parameterized for it.
+- **Autoplay gesture** (§8) — **RESOLVED: a one-time tap-to-start overlay**
+  ("▶ tap to start sound", `#start-gesture`). The first click unlocks both `<audio>`
+  elements within the user gesture; thereafter audio follows state. This is the
+  documented launch step for the full-screen projection.
diff --git a/docs/superpowers/specs/2026-06-26-networked-control-surface-design.md b/docs/superpowers/specs/2026-06-26-networked-control-surface-design.md
index b31d076..a3e2fea 100644
--- a/docs/superpowers/specs/2026-06-26-networked-control-surface-design.md
+++ b/docs/superpowers/specs/2026-06-26-networked-control-surface-design.md
@@ -66,7 +66,7 @@ needed. Alternatives considered and rejected: renderer polling — laggy/wastefu
 peer-to-peer WebRTC — most plumbing and it breaks the clean single-contract seam
 that makes the Arduino a drop-in.)
 
-## 3. Control surface: the five controls (the faithful twin)
+## 3. Control surface: the six controls (the faithful twin)
 
 The iPad remote is a **faithful on-screen twin** of the intended physical panel —
 what you like on the iPad is the spec for the Arduino panel. The control
@@ -74,29 +74,34 @@ inventory is exactly what the sim exposes today:
 
 | Control | Sim today | Physical form (later) |
 |---|---|---|
-| **Content** | 7-way `<select>` (video / audio+video / music+video / off / white noise / music / audio track) | rotary selector switch (7 pos) |
+| **Visual** | toggle (show video on/off) | 2-position switch |
+| **Audio** | 3-way `<select>` (off / soundtrack / white noise; ·music deferred) | rotary selector switch (3 pos) |
 | **Altitude** | endless circular dial — walks the 5 scales (cosmos→abyss), wraps | rotary encoder (endless, detented) |
 | **Left** (analytical) | slider 0–4 | knob/pot, 5 detents |
 | **Right** (dreamlike) | slider 0–4 | knob/pot, 5 detents |
 | **Mood** | slider −4…+4 (dark↔light) | center-detented knob |
 
 (This is the original sub-project-4 inventory, *updated*: Dark+Light collapsed
-into one **Mood** knob in session 0013, and the **Altitude** encoder added in
-session 0018.)
+into one **Mood** knob in session 0013, the **Altitude** encoder added in session
+0018, and the bundled 7-way **Content** dial split into orthogonal **Visual** ×
+**Audio** controls in session 0020 — see
+[`2026-06-26-audio-video-separated-controls-design.md`](./2026-06-26-audio-video-separated-controls-design.md).)
 
 ## 4. Live vs. transitional controls (per-knob transition state)
 
 The five controls split into two classes by how a **human** perceives the change:
 
-- **Live controls — Left, Right, Mood.** The alteration engine treats these as
-  `LIVE_UPDATE` (the WebGL Kuwahara dream + HUD overlay + color grade re-tune
-  within a frame). Effectively instantaneous → these are *always idle*; the
-  remote just shows the value.
-- **Transitional controls — Altitude, Content/mode.** These trigger a renderer
+- **Live controls — Left, Right, Mood, Audio.** The alteration engine treats
+  Left/Right/Mood as `LIVE_UPDATE` (the WebGL Kuwahara dream + HUD overlay + color
+  grade re-tune within a frame). **Audio** source changes are also live: a short
+  (~0.6 s) renderer-side gain crossfade between two `<audio>` elements —
+  perceptible but non-blocking, with no `settled` ack (audio spec §7). These are
+  *always idle*; the remote just shows the value.
+- **Transitional controls — Altitude, Visual.** These trigger a renderer
   animation that takes real, perceptible time: Altitude plays ring-transition
-  clips (zoom/warp between scales); Content/mode does fade-to-black / crossfade.
-  During that window the control is **busy**, and the control source must model
-  that so its feedback is honest.
+  clips (zoom/warp between scales); **Visual** on/off does fade-to-black /
+  fade-from-black. During that window the control is **busy**, and the control
+  source must model that so its feedback is honest.
 
 Per-transitional-control lifecycle:
 
@@ -116,8 +121,9 @@ Hardware payoff: a physical Altitude encoder can light an LED while
 `transitioning`; the iPad twin shows the matching spinner/disabled-detent.
 
 `SessionState` therefore carries
-`transitions: { altitude: idle|transitioning, content: idle|transitioning }`
-alongside the controls and `seq`. (Live controls need no entry.)
+`transitions: { altitude: idle|transitioning, visual: idle|transitioning }`
+alongside the controls and `seq`. (Live controls — Left/Right/Mood/Audio — need
+no entry.)
 
 ## 5. Server contract
 
@@ -130,13 +136,20 @@ control sources never hit them directly.
 ```json
 {
   "seq": 42,
-  "controls": { "content": "video", "left": 3, "right": 1, "mood": -2 },
-  "altitude": { "index": 2, "scale": "forest", "clip_id": "forest_07" },
-  "transitions": { "altitude": "idle", "content": "idle" },
-  "render": { "plan": { }, "content": { "audio_source": null, "video": true } }
+  "controls": { "visual": "on", "audio": "soundtrack", "left": 3, "right": 1, "mood": -2 },
+  "altitude": { "index": 2, "scale": "coast", "clip_id": "coast_07" },
+  "transitions": { "altitude": "idle", "visual": "idle" },
+  "render": {
+    "plan": { },
+    "video": { "shown": true },
+    "audio": { "source": "soundtrack", "url": "/media/audio/coast/waves.loop.mp3", "altitude_coupled": true }
+  }
 }
 ```
 
+(The `render.video` + `render.audio` split and server-side `audio.url` resolution
+are specified in the audio spec §6.1.)
+
 A control source GETs this on load to draw its knobs in the right positions; the
 renderer can use it to cold-start or resync after a reconnect.
 
@@ -145,17 +158,19 @@ renderer can use it to cold-start or resync after a reconnect.
 A partial patch — only the fields that changed:
 
 ```json
-{ "set": { "left": 4 },        "altitude_delta": 0 }   // live knob
-{ "set": { "content": "off" }, "altitude_delta": 0 }   // transitional (mode)
-{ "set": {},                   "altitude_delta": -1 }  // one encoder tick down
+{ "set": { "left": 4 },               "altitude_delta": 0 }   // live knob
+{ "set": { "audio": "white_noise" },  "altitude_delta": 0 }   // live (audio source)
+{ "set": { "visual": "off" },         "altitude_delta": 0 }   // transitional (video fade)
+{ "set": {},                          "altitude_delta": -1 }  // one encoder tick down
 ```
 
 Server applies it, bumps `seq`, recomputes, and:
 
-- **live field** → broadcasts a `state` event immediately;
-- **transitional field** (a `content` change, or a non-zero `altitude_delta`) →
-  computes the ring move / fade, marks that field `transitioning`, broadcasts a
-  `ring` (or `mode`) event for the renderer to animate.
+- **live field** (Left/Right/Mood, **or an `audio` source change**) → broadcasts
+  a `state` event immediately (the renderer crossfades audio on its own, §7);
+- **transitional field** (a `visual` change, or a non-zero `altitude_delta`) →
+  computes the ring move / video fade, marks that field `transitioning`,
+  broadcasts a `ring` (or `mode`) event for the renderer to animate.
 
 Returns the new snapshot (an immediate authoritative echo for the poster).
 Absolute `set` values make retries idempotent; `altitude_delta` is relative
@@ -166,10 +181,12 @@ Absolute `set` values make retries idempotent; `altitude_delta` is relative
 
 Event types:
 
-- `state` — full snapshot; sent on connect and on every live change.
+- `state` — full snapshot; sent on connect and on every live change (including an
+  **audio** source change, which the renderer crossfades — §7).
 - `ring` — an altitude move: the transition clip(s) to play + the landed
-  scale/clip. Renderer plays them, then settles.
-- `mode` — a content/mode change: the fade the renderer should run.
+  scale/clip, **plus the new `audio.url`** so the renderer can crossfade the
+  soundtrack in step (when `audio=soundtrack`). Renderer plays them, then settles.
+- `mode` — a **Visual** on/off change: the video fade the renderer should run.
 - `ping` — keepalive.
 
 The SSE event payloads reuse the existing serializers
@@ -207,14 +224,17 @@ server: transitions.altitude="idle" → SSE "state" to all → iPad clears spinn
 
 ### 6.1 Renderer — `/` (display), refactored from today's `index.html`
 
-- **Removes** the input controls (the `<select>`, the Altitude SVG, the three
-  sliders) and their handlers.
-- **Keeps** everything that draws the experience: the video element, the WebGL
-  Kuwahara dream shader, the HUD/annotation + affect overlay, the
-  ring-transition playback, the media preload pool, the crossfade-loop logic.
+- **Removes** the input controls (the Visual toggle, the Audio `<select>`, the
+  Altitude SVG, the three sliders) and their handlers.
+- **Keeps** everything that draws the experience: the video element, the **A/B
+  `<audio>` crossfade layer**, the WebGL Kuwahara dream shader, the
+  HUD/annotation + affect overlay, the ring-transition playback, the media
+  preload pool, the crossfade-loop logic.
 - **Adds** a thin `state` client: open `EventSource('/api/events')`; on `state`
-  apply controls→render live; on `ring` play the transition clip(s) then settle
-  and `POST /api/render/event {settled}`; on `mode` run the fade. On (re)connect,
+  apply controls→render live (incl. the audio crossfade); on `ring` play the
+  transition clip(s) — crossfading the soundtrack to the event's `audio.url` —
+  then settle and `POST /api/render/event {settled}`; on `mode` run the video
+  fade. On (re)connect,
   pull `GET /api/state` to resync. SSE auto-reconnects, so a flaky link
   self-heals.
 - Runs full-screen on the laptop driving the projector; no keyboard/mouse needed
@@ -222,16 +242,16 @@ server: transitions.altitude="idle" → SSE "state" to all → iPad clears spinn
 
 ### 6.2 Controller — `/remote`, a new touch-first page (the faithful twin)
 
-- The five controls laid out like the intended physical panel: **Content**
-  selector, **Altitude** encoder dial (reuse the existing SVG dial component),
-  **Left** / **Right** / **Mood** knobs.
+- The six controls laid out like the intended physical panel: **Visual** toggle +
+  **Audio** selector, **Altitude** encoder dial (reuse the existing SVG dial
+  component), **Left** / **Right** / **Mood** knobs.
 - On any input → `POST /api/control` with just the changed field. Live knobs post
-  on drag; Altitude posts encoder ticks; Content posts on selection.
+  on drag; Altitude posts encoder ticks; Visual/Audio post on change.
 - Per-knob transition state (§4): a transitional control goes **pending** on
   touch, reflects **transitioning** from the SSE feed, clears on **settled**.
   Altitude shows a settling indicator and its detents soft-lock (further ticks
-  queue as the next target rather than stacking); Content shows the selection as
-  pending until the fade completes.
+  queue as the next target rather than stacking); **Visual** shows the toggle as
+  pending until the video fade completes; **Audio** is live (no pending state).
 - It **also** subscribes to `/api/events` so it stays in sync with other
   sources (turn the future Arduino's knob and the iPad reflects it, and
   vice-versa). On load it GETs `/api/state` to draw current positions.