docs(audio): reframe control-surface spec content->visual+audio; record §12 decisions + roadmap
- networked-control-surface §3/§4/§5/§6: Content -> Visual + Audio; transitions.content -> transitions.visual; render.content -> render.video + render.audio; SSE notes - audio spec §12: resolved (audio=live ~0.6s; noise=pink; tap-to-start overlay); status -> in implementation - ROADMAP: session-0020 audio increment; music layer deferred Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Audio + Video — separated Visual & Audio controls — design
|
||||
|
||||
**Status:** proposed
|
||||
**Status:** proposed → in implementation (plan `docs/superpowers/plans/2026-06-26-audio-video-separated-controls.md`, session 0020)
|
||||
**Date:** 2026-06-26
|
||||
**Session:** 0020 (audio)
|
||||
**Extends / reframes:** [`2026-06-26-networked-control-surface-design.md`](./2026-06-26-networked-control-surface-design.md)
|
||||
@@ -240,9 +240,15 @@ The display page adds an **audio layer**:
|
||||
music; flip on the reserved Audio position).
|
||||
- `ROADMAP.md` to note the audio capability at implementation time.
|
||||
|
||||
## 12. Open questions / flagged
|
||||
## 12. Open questions / flagged — RESOLVED (session 0020 build)
|
||||
|
||||
- **Audio knob feel** (§7) — live vs. transitional; v1 = live. Revisit on the iPad.
|
||||
- **White-noise color** — pink vs. brown vs. true white; v1 = pink/brown (calmer).
|
||||
- **Autoplay gesture** (§8) — document the launch step so the projection starts
|
||||
audio without manual unmuting.
|
||||
- **Audio knob feel** (§7) — **RESOLVED: live**, with a ~0.6 s renderer-side gain
|
||||
crossfade (`XFADE_MS=600`), no `settled` ack. Promote to transitional later only
|
||||
if the iPad makes it feel like it needs a "busy" state.
|
||||
- **White-noise color** — **RESOLVED: pink** (`anoisesrc=color=pink`), a balanced
|
||||
calm bed. Brown is a one-parameter alternative (`color=brown`) if a deeper rumble
|
||||
is wanted; the build is parameterized for it.
|
||||
- **Autoplay gesture** (§8) — **RESOLVED: a one-time tap-to-start overlay**
|
||||
("▶ tap to start sound", `#start-gesture`). The first click unlocks both `<audio>`
|
||||
elements within the user gesture; thereafter audio follows state. This is the
|
||||
documented launch step for the full-screen projection.
|
||||
|
||||
@@ -66,7 +66,7 @@ needed. Alternatives considered and rejected: renderer polling — laggy/wastefu
|
||||
peer-to-peer WebRTC — most plumbing and it breaks the clean single-contract seam
|
||||
that makes the Arduino a drop-in.)
|
||||
|
||||
## 3. Control surface: the five controls (the faithful twin)
|
||||
## 3. Control surface: the six controls (the faithful twin)
|
||||
|
||||
The iPad remote is a **faithful on-screen twin** of the intended physical panel —
|
||||
what you like on the iPad is the spec for the Arduino panel. The control
|
||||
@@ -74,29 +74,34 @@ inventory is exactly what the sim exposes today:
|
||||
|
||||
| Control | Sim today | Physical form (later) |
|
||||
|---|---|---|
|
||||
| **Content** | 7-way `<select>` (video / audio+video / music+video / off / white noise / music / audio track) | rotary selector switch (7 pos) |
|
||||
| **Visual** | toggle (show video on/off) | 2-position switch |
|
||||
| **Audio** | 3-way `<select>` (off / soundtrack / white noise; ·music deferred) | rotary selector switch (3 pos) |
|
||||
| **Altitude** | endless circular dial — walks the 5 scales (cosmos→abyss), wraps | rotary encoder (endless, detented) |
|
||||
| **Left** (analytical) | slider 0–4 | knob/pot, 5 detents |
|
||||
| **Right** (dreamlike) | slider 0–4 | knob/pot, 5 detents |
|
||||
| **Mood** | slider −4…+4 (dark↔light) | center-detented knob |
|
||||
|
||||
(This is the original sub-project-4 inventory, *updated*: Dark+Light collapsed
|
||||
into one **Mood** knob in session 0013, and the **Altitude** encoder added in
|
||||
session 0018.)
|
||||
into one **Mood** knob in session 0013, the **Altitude** encoder added in session
|
||||
0018, and the bundled 7-way **Content** dial split into orthogonal **Visual** ×
|
||||
**Audio** controls in session 0020 — see
|
||||
[`2026-06-26-audio-video-separated-controls-design.md`](./2026-06-26-audio-video-separated-controls-design.md).)
|
||||
|
||||
## 4. Live vs. transitional controls (per-knob transition state)
|
||||
|
||||
The five controls split into two classes by how a **human** perceives the change:
|
||||
|
||||
- **Live controls — Left, Right, Mood.** The alteration engine treats these as
|
||||
`LIVE_UPDATE` (the WebGL Kuwahara dream + HUD overlay + color grade re-tune
|
||||
within a frame). Effectively instantaneous → these are *always idle*; the
|
||||
remote just shows the value.
|
||||
- **Transitional controls — Altitude, Content/mode.** These trigger a renderer
|
||||
- **Live controls — Left, Right, Mood, Audio.** The alteration engine treats
|
||||
Left/Right/Mood as `LIVE_UPDATE` (the WebGL Kuwahara dream + HUD overlay + color
|
||||
grade re-tune within a frame). **Audio** source changes are also live: a short
|
||||
(~0.6 s) renderer-side gain crossfade between two `<audio>` elements —
|
||||
perceptible but non-blocking, with no `settled` ack (audio spec §7). These are
|
||||
*always idle*; the remote just shows the value.
|
||||
- **Transitional controls — Altitude, Visual.** These trigger a renderer
|
||||
animation that takes real, perceptible time: Altitude plays ring-transition
|
||||
clips (zoom/warp between scales); Content/mode does fade-to-black / crossfade.
|
||||
During that window the control is **busy**, and the control source must model
|
||||
that so its feedback is honest.
|
||||
clips (zoom/warp between scales); **Visual** on/off does fade-to-black /
|
||||
fade-from-black. During that window the control is **busy**, and the control
|
||||
source must model that so its feedback is honest.
|
||||
|
||||
Per-transitional-control lifecycle:
|
||||
|
||||
@@ -116,8 +121,9 @@ Hardware payoff: a physical Altitude encoder can light an LED while
|
||||
`transitioning`; the iPad twin shows the matching spinner/disabled-detent.
|
||||
|
||||
`SessionState` therefore carries
|
||||
`transitions: { altitude: idle|transitioning, content: idle|transitioning }`
|
||||
alongside the controls and `seq`. (Live controls need no entry.)
|
||||
`transitions: { altitude: idle|transitioning, visual: idle|transitioning }`
|
||||
alongside the controls and `seq`. (Live controls — Left/Right/Mood/Audio — need
|
||||
no entry.)
|
||||
|
||||
## 5. Server contract
|
||||
|
||||
@@ -130,13 +136,20 @@ control sources never hit them directly.
|
||||
```json
|
||||
{
|
||||
"seq": 42,
|
||||
"controls": { "content": "video", "left": 3, "right": 1, "mood": -2 },
|
||||
"altitude": { "index": 2, "scale": "forest", "clip_id": "forest_07" },
|
||||
"transitions": { "altitude": "idle", "content": "idle" },
|
||||
"render": { "plan": { }, "content": { "audio_source": null, "video": true } }
|
||||
"controls": { "visual": "on", "audio": "soundtrack", "left": 3, "right": 1, "mood": -2 },
|
||||
"altitude": { "index": 2, "scale": "coast", "clip_id": "coast_07" },
|
||||
"transitions": { "altitude": "idle", "visual": "idle" },
|
||||
"render": {
|
||||
"plan": { },
|
||||
"video": { "shown": true },
|
||||
"audio": { "source": "soundtrack", "url": "/media/audio/coast/waves.loop.mp3", "altitude_coupled": true }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(The `render.video` + `render.audio` split and server-side `audio.url` resolution
|
||||
are specified in the audio spec §6.1.)
|
||||
|
||||
A control source GETs this on load to draw its knobs in the right positions; the
|
||||
renderer can use it to cold-start or resync after a reconnect.
|
||||
|
||||
@@ -145,17 +158,19 @@ renderer can use it to cold-start or resync after a reconnect.
|
||||
A partial patch — only the fields that changed:
|
||||
|
||||
```json
|
||||
{ "set": { "left": 4 }, "altitude_delta": 0 } // live knob
|
||||
{ "set": { "content": "off" }, "altitude_delta": 0 } // transitional (mode)
|
||||
{ "set": {}, "altitude_delta": -1 } // one encoder tick down
|
||||
{ "set": { "left": 4 }, "altitude_delta": 0 } // live knob
|
||||
{ "set": { "audio": "white_noise" }, "altitude_delta": 0 } // live (audio source)
|
||||
{ "set": { "visual": "off" }, "altitude_delta": 0 } // transitional (video fade)
|
||||
{ "set": {}, "altitude_delta": -1 } // one encoder tick down
|
||||
```
|
||||
|
||||
Server applies it, bumps `seq`, recomputes, and:
|
||||
|
||||
- **live field** → broadcasts a `state` event immediately;
|
||||
- **transitional field** (a `content` change, or a non-zero `altitude_delta`) →
|
||||
computes the ring move / fade, marks that field `transitioning`, broadcasts a
|
||||
`ring` (or `mode`) event for the renderer to animate.
|
||||
- **live field** (Left/Right/Mood, **or an `audio` source change**) → broadcasts
|
||||
a `state` event immediately (the renderer crossfades audio on its own, §7);
|
||||
- **transitional field** (a `visual` change, or a non-zero `altitude_delta`) →
|
||||
computes the ring move / video fade, marks that field `transitioning`,
|
||||
broadcasts a `ring` (or `mode`) event for the renderer to animate.
|
||||
|
||||
Returns the new snapshot (an immediate authoritative echo for the poster).
|
||||
Absolute `set` values make retries idempotent; `altitude_delta` is relative
|
||||
@@ -166,10 +181,12 @@ Absolute `set` values make retries idempotent; `altitude_delta` is relative
|
||||
|
||||
Event types:
|
||||
|
||||
- `state` — full snapshot; sent on connect and on every live change.
|
||||
- `state` — full snapshot; sent on connect and on every live change (including an
|
||||
**audio** source change, which the renderer crossfades — §7).
|
||||
- `ring` — an altitude move: the transition clip(s) to play + the landed
|
||||
scale/clip. Renderer plays them, then settles.
|
||||
- `mode` — a content/mode change: the fade the renderer should run.
|
||||
scale/clip, **plus the new `audio.url`** so the renderer can crossfade the
|
||||
soundtrack in step (when `audio=soundtrack`). Renderer plays them, then settles.
|
||||
- `mode` — a **Visual** on/off change: the video fade the renderer should run.
|
||||
- `ping` — keepalive.
|
||||
|
||||
The SSE event payloads reuse the existing serializers
|
||||
@@ -207,14 +224,17 @@ server: transitions.altitude="idle" → SSE "state" to all → iPad clears spinn
|
||||
|
||||
### 6.1 Renderer — `/` (display), refactored from today's `index.html`
|
||||
|
||||
- **Removes** the input controls (the `<select>`, the Altitude SVG, the three
|
||||
sliders) and their handlers.
|
||||
- **Keeps** everything that draws the experience: the video element, the WebGL
|
||||
Kuwahara dream shader, the HUD/annotation + affect overlay, the
|
||||
ring-transition playback, the media preload pool, the crossfade-loop logic.
|
||||
- **Removes** the input controls (the Visual toggle, the Audio `<select>`, the
|
||||
Altitude SVG, the three sliders) and their handlers.
|
||||
- **Keeps** everything that draws the experience: the video element, the **A/B
|
||||
`<audio>` crossfade layer**, the WebGL Kuwahara dream shader, the
|
||||
HUD/annotation + affect overlay, the ring-transition playback, the media
|
||||
preload pool, the crossfade-loop logic.
|
||||
- **Adds** a thin `state` client: open `EventSource('/api/events')`; on `state`
|
||||
apply controls→render live; on `ring` play the transition clip(s) then settle
|
||||
and `POST /api/render/event {settled}`; on `mode` run the fade. On (re)connect,
|
||||
apply controls→render live (incl. the audio crossfade); on `ring` play the
|
||||
transition clip(s) — crossfading the soundtrack to the event's `audio.url` —
|
||||
then settle and `POST /api/render/event {settled}`; on `mode` run the video
|
||||
fade. On (re)connect,
|
||||
pull `GET /api/state` to resync. SSE auto-reconnects, so a flaky link
|
||||
self-heals.
|
||||
- Runs full-screen on the laptop driving the projector; no keyboard/mouse needed
|
||||
@@ -222,16 +242,16 @@ server: transitions.altitude="idle" → SSE "state" to all → iPad clears spinn
|
||||
|
||||
### 6.2 Controller — `/remote`, a new touch-first page (the faithful twin)
|
||||
|
||||
- The five controls laid out like the intended physical panel: **Content**
|
||||
selector, **Altitude** encoder dial (reuse the existing SVG dial component),
|
||||
**Left** / **Right** / **Mood** knobs.
|
||||
- The six controls laid out like the intended physical panel: **Visual** toggle +
|
||||
**Audio** selector, **Altitude** encoder dial (reuse the existing SVG dial
|
||||
component), **Left** / **Right** / **Mood** knobs.
|
||||
- On any input → `POST /api/control` with just the changed field. Live knobs post
|
||||
on drag; Altitude posts encoder ticks; Content posts on selection.
|
||||
on drag; Altitude posts encoder ticks; Visual/Audio post on change.
|
||||
- Per-knob transition state (§4): a transitional control goes **pending** on
|
||||
touch, reflects **transitioning** from the SSE feed, clears on **settled**.
|
||||
Altitude shows a settling indicator and its detents soft-lock (further ticks
|
||||
queue as the next target rather than stacking); Content shows the selection as
|
||||
pending until the fade completes.
|
||||
queue as the next target rather than stacking); **Visual** shows the toggle as
|
||||
pending until the video fade completes; **Audio** is live (no pending state).
|
||||
- It **also** subscribes to `/api/events` so it stays in sync with other
|
||||
sources (turn the future Arduino's knob and the iPad reflects it, and
|
||||
vice-versa). On load it GETs `/api/state` to draw current positions.
|
||||
|
||||
Reference in New Issue
Block a user