Compare commits

..

7 Commits

Author SHA1 Message Date
BenStullsBets c07067492b test(audio): Playwright E2E (3 tests, skip-if-no-browser) + client robustness
E2E asserts in a real headless Chromium: white_noise loads the noise asset,
soundtrack loads the current scale's asset, and Visual=off fades video while
audio keeps playing. Driving it out surfaced + fixed three client issues:
- fadeVolume uses setInterval (rAF stalls when the tab is backgrounded)
- start gesture drops srcless priming (a click grants Chrome autoplay
  document-wide; priming empty <audio> only hung the handler / polluted state)
- applyAudio doesn't await play() (awaiting a srcless/buffering play can hang)
pyproject: [e2e] optional dep (pytest-playwright). Full suite 288 passed, 2 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 08:00:44 -07:00
BenStullsBets fd0aad6610 docs(audio): reframe control-surface spec content->visual+audio; record §12 decisions + roadmap
- networked-control-surface §3/§4/§5/§6: Content -> Visual + Audio; transitions.content
  -> transitions.visual; render.content -> render.video + render.audio; SSE notes
- audio spec §12: resolved (audio=live ~0.6s; noise=pink; tap-to-start overlay);
  status -> in implementation
- ROADMAP: session-0020 audio increment; music layer deferred

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 07:50:34 -07:00
BenStullsBets 68f9d7342a feat(audio): client — split Content into Visual+Audio controls + audio crossfade layer
- index.html: Content <select> -> Visual toggle + Audio source selector; A/B <audio>
- app.js: controls() emits visual+audio; update() posts altitude_index and reads
  render.video/render.audio; A/B gain-crossfade applyAudio(); one-time tap-to-start
  gesture (autoplay policy); mediaUrl() passes absolute /media urls through
- style.css: #start-gesture overlay
- smoke: page serves controls + audio elements; soundtrack url couples to altitude;
  noise + soundtrack media serve 200

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 07:48:03 -07:00
BenStullsBets 29c6d822bd feat(audio): Visual×Audio Python contract — resolver, controls, state, API
- player/audio.py: pure resolve_visual / resolve_audio_url / resolve_audio (§4/§6)
- retire player/content.py (7-way table) + its test
- player/controls.py: serial contract content -> visual + audio
- player/state.py: Playback carries video:bool + audio_source:str
- simulator/app.py: /api/alteration validates visual+audio, returns render.video
  + server-resolved render.audio.url from altitude_index; NOISE_URL constant
- tests reframed; full suite 285 passed, 2 skipped

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 07:46:22 -07:00
BenStullsBets 1f9b2a0bec feat(audio): Scale.audio field + per-scale soundtrack wiring through the manifest
Scale gains audio:str='' (back-compat). clips.py reads/serializes it; ring_to_dict
exposes it; build_pool_manifest emits SCALE_AUDIO per scale. Manifest regenerated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 07:42:40 -07:00
BenStullsBets 39fb9108db feat(audio): ffmpeg production pass — loop/loudnorm builders, runner, build script
Pure arg builders (audio_ops) + thin runner (audio_run) + build_audio_media.py.
Produces the 5 normalized soundtrack loops + a synthesized pink-noise bed under
the gitignored sample_media/audio tree. 6 tests pass (white-noise integration
runs against bundled imageio_ffmpeg).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 07:41:34 -07:00
BenStullsBets 062a283de4 docs(plan): audio+video separated Visual/Audio controls implementation plan
Session 0020. Anchored on the audio-video-separated-controls Solution Design.
Scope: build into the current single-page simulator (simulator-first directive);
reframe the unbuilt control-surface spec on paper. 10 tasks, all testing tiers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 07:01:06 -07:00
30 changed files with 2178 additions and 250 deletions
+16 -1
View File
@@ -196,8 +196,23 @@ every transition, so fast navigation stays responsive on placeholder media.
browser. Real strict-PD bases for all five scales sourced per
[`docs/content-sourcing.md`](./content-sourcing.md).
- **Audio + separated Visual/Audio controls (session 0020) ✅ Done.** The
simulator now has **sound**: per-altitude soundtracks that follow the Altitude
dial + a synthesized pink **white-noise** bed, behind two orthogonal controls —
**Visual** (on/off) × **Audio** (off / soundtrack / white_noise) — replacing
the bundled 7-way Content dial. `tools/pipeline/audio_ops` + `audio_run` +
`simulator/build_audio_media.py` (ffmpeg loop/loudnorm/noise, gitignored
assets), `Scale.audio` in the manifest, a pure `player/audio.py` resolver
(retiring `player/content.py`), server-resolved `render.audio.url`, and a client
A/B `<audio>` crossfade layer. This is the "Content step" of the future control
surface (it reframes the proposed networked-control-surface spec §3/§5). Spec:
[`2026-06-26-audio-video-separated-controls-design.md`](./superpowers/specs/2026-06-26-audio-video-separated-controls-design.md);
plan:
[`2026-06-26-audio-video-separated-controls.md`](./superpowers/plans/2026-06-26-audio-video-separated-controls.md).
- **Still deferred:** real **i2v ring transitions** (need a generative-video
model + adjacent real bases); Pi renderer + serial/firmware.
model + adjacent real bases); Pi renderer + serial/firmware; the **music**
audio layer (reserved dial position, no assets yet).
- **Catalog model changes** — audio *source* + "neutral base" vs "altered
variant" flag (sub-project 2 territory, design §13).
File diff suppressed because it is too large Load Diff
@@ -1,6 +1,6 @@
# Audio + Video — separated Visual & Audio controls — design
**Status:** proposed
**Status:** proposed → in implementation (plan `docs/superpowers/plans/2026-06-26-audio-video-separated-controls.md`, session 0020)
**Date:** 2026-06-26
**Session:** 0020 (audio)
**Extends / reframes:** [`2026-06-26-networked-control-surface-design.md`](./2026-06-26-networked-control-surface-design.md)
@@ -240,9 +240,15 @@ The display page adds an **audio layer**:
music; flip on the reserved Audio position).
- `ROADMAP.md` to note the audio capability at implementation time.
## 12. Open questions / flagged
## 12. Open questions / flagged — RESOLVED (session 0020 build)
- **Audio knob feel** (§7) — live vs. transitional; v1 = live. Revisit on the iPad.
- **White-noise color** — pink vs. brown vs. true white; v1 = pink/brown (calmer).
- **Autoplay gesture** (§8) — document the launch step so the projection starts
audio without manual unmuting.
- **Audio knob feel** (§7) — **RESOLVED: live**, with a ~0.6 s renderer-side gain
crossfade (`XFADE_MS=600`), no `settled` ack. Promote to transitional later only
if the iPad makes it feel like it needs a "busy" state.
- **White-noise color****RESOLVED: pink** (`anoisesrc=color=pink`), a balanced
calm bed. Brown is a one-parameter alternative (`color=brown`) if a deeper rumble
is wanted; the build is parameterized for it.
- **Autoplay gesture** (§8) — **RESOLVED: a one-time tap-to-start overlay**
("▶ tap to start sound", `#start-gesture`). The first click unlocks both `<audio>`
elements within the user gesture; thereafter audio follows state. This is the
documented launch step for the full-screen projection.
@@ -66,7 +66,7 @@ needed. Alternatives considered and rejected: renderer polling — laggy/wastefu
peer-to-peer WebRTC — most plumbing and it breaks the clean single-contract seam
that makes the Arduino a drop-in.)
## 3. Control surface: the five controls (the faithful twin)
## 3. Control surface: the six controls (the faithful twin)
The iPad remote is a **faithful on-screen twin** of the intended physical panel —
what you like on the iPad is the spec for the Arduino panel. The control
@@ -74,29 +74,34 @@ inventory is exactly what the sim exposes today:
| Control | Sim today | Physical form (later) |
|---|---|---|
| **Content** | 7-way `<select>` (video / audio+video / music+video / off / white noise / music / audio track) | rotary selector switch (7 pos) |
| **Visual** | toggle (show video on/off) | 2-position switch |
| **Audio** | 3-way `<select>` (off / soundtrack / white noise; ·music deferred) | rotary selector switch (3 pos) |
| **Altitude** | endless circular dial — walks the 5 scales (cosmos→abyss), wraps | rotary encoder (endless, detented) |
| **Left** (analytical) | slider 04 | knob/pot, 5 detents |
| **Right** (dreamlike) | slider 04 | knob/pot, 5 detents |
| **Mood** | slider 4…+4 (dark↔light) | center-detented knob |
(This is the original sub-project-4 inventory, *updated*: Dark+Light collapsed
into one **Mood** knob in session 0013, and the **Altitude** encoder added in
session 0018.)
into one **Mood** knob in session 0013, the **Altitude** encoder added in session
0018, and the bundled 7-way **Content** dial split into orthogonal **Visual** ×
**Audio** controls in session 0020 — see
[`2026-06-26-audio-video-separated-controls-design.md`](./2026-06-26-audio-video-separated-controls-design.md).)
## 4. Live vs. transitional controls (per-knob transition state)
The five controls split into two classes by how a **human** perceives the change:
- **Live controls — Left, Right, Mood.** The alteration engine treats these as
`LIVE_UPDATE` (the WebGL Kuwahara dream + HUD overlay + color grade re-tune
within a frame). Effectively instantaneous → these are *always idle*; the
remote just shows the value.
- **Transitional controls — Altitude, Content/mode.** These trigger a renderer
- **Live controls — Left, Right, Mood, Audio.** The alteration engine treats
Left/Right/Mood as `LIVE_UPDATE` (the WebGL Kuwahara dream + HUD overlay + color
grade re-tune within a frame). **Audio** source changes are also live: a short
(~0.6 s) renderer-side gain crossfade between two `<audio>` elements —
perceptible but non-blocking, with no `settled` ack (audio spec §7). These are
*always idle*; the remote just shows the value.
- **Transitional controls — Altitude, Visual.** These trigger a renderer
animation that takes real, perceptible time: Altitude plays ring-transition
clips (zoom/warp between scales); Content/mode does fade-to-black / crossfade.
During that window the control is **busy**, and the control source must model
that so its feedback is honest.
clips (zoom/warp between scales); **Visual** on/off does fade-to-black /
fade-from-black. During that window the control is **busy**, and the control
source must model that so its feedback is honest.
Per-transitional-control lifecycle:
@@ -116,8 +121,9 @@ Hardware payoff: a physical Altitude encoder can light an LED while
`transitioning`; the iPad twin shows the matching spinner/disabled-detent.
`SessionState` therefore carries
`transitions: { altitude: idle|transitioning, content: idle|transitioning }`
alongside the controls and `seq`. (Live controls need no entry.)
`transitions: { altitude: idle|transitioning, visual: idle|transitioning }`
alongside the controls and `seq`. (Live controls — Left/Right/Mood/Audio — need
no entry.)
## 5. Server contract
@@ -130,13 +136,20 @@ control sources never hit them directly.
```json
{
"seq": 42,
"controls": { "content": "video", "left": 3, "right": 1, "mood": -2 },
"altitude": { "index": 2, "scale": "forest", "clip_id": "forest_07" },
"transitions": { "altitude": "idle", "content": "idle" },
"render": { "plan": { }, "content": { "audio_source": null, "video": true } }
"controls": { "visual": "on", "audio": "soundtrack", "left": 3, "right": 1, "mood": -2 },
"altitude": { "index": 2, "scale": "coast", "clip_id": "coast_07" },
"transitions": { "altitude": "idle", "visual": "idle" },
"render": {
"plan": { },
"video": { "shown": true },
"audio": { "source": "soundtrack", "url": "/media/audio/coast/waves.loop.mp3", "altitude_coupled": true }
}
}
```
(The `render.video` + `render.audio` split and server-side `audio.url` resolution
are specified in the audio spec §6.1.)
A control source GETs this on load to draw its knobs in the right positions; the
renderer can use it to cold-start or resync after a reconnect.
@@ -145,17 +158,19 @@ renderer can use it to cold-start or resync after a reconnect.
A partial patch — only the fields that changed:
```json
{ "set": { "left": 4 }, "altitude_delta": 0 } // live knob
{ "set": { "content": "off" }, "altitude_delta": 0 } // transitional (mode)
{ "set": {}, "altitude_delta": -1 } // one encoder tick down
{ "set": { "left": 4 }, "altitude_delta": 0 } // live knob
{ "set": { "audio": "white_noise" }, "altitude_delta": 0 } // live (audio source)
{ "set": { "visual": "off" }, "altitude_delta": 0 } // transitional (video fade)
{ "set": {}, "altitude_delta": -1 } // one encoder tick down
```
Server applies it, bumps `seq`, recomputes, and:
- **live field** → broadcasts a `state` event immediately;
- **transitional field** (a `content` change, or a non-zero `altitude_delta`) →
computes the ring move / fade, marks that field `transitioning`, broadcasts a
`ring` (or `mode`) event for the renderer to animate.
- **live field** (Left/Right/Mood, **or an `audio` source change**) → broadcasts
a `state` event immediately (the renderer crossfades audio on its own, §7);
- **transitional field** (a `visual` change, or a non-zero `altitude_delta`) →
computes the ring move / video fade, marks that field `transitioning`,
broadcasts a `ring` (or `mode`) event for the renderer to animate.
Returns the new snapshot (an immediate authoritative echo for the poster).
Absolute `set` values make retries idempotent; `altitude_delta` is relative
@@ -166,10 +181,12 @@ Absolute `set` values make retries idempotent; `altitude_delta` is relative
Event types:
- `state` — full snapshot; sent on connect and on every live change.
- `state` — full snapshot; sent on connect and on every live change (including an
**audio** source change, which the renderer crossfades — §7).
- `ring` — an altitude move: the transition clip(s) to play + the landed
scale/clip. Renderer plays them, then settles.
- `mode` — a content/mode change: the fade the renderer should run.
scale/clip, **plus the new `audio.url`** so the renderer can crossfade the
soundtrack in step (when `audio=soundtrack`). Renderer plays them, then settles.
- `mode` — a **Visual** on/off change: the video fade the renderer should run.
- `ping` — keepalive.
The SSE event payloads reuse the existing serializers
@@ -207,14 +224,17 @@ server: transitions.altitude="idle" → SSE "state" to all → iPad clears spinn
### 6.1 Renderer — `/` (display), refactored from today's `index.html`
- **Removes** the input controls (the `<select>`, the Altitude SVG, the three
sliders) and their handlers.
- **Keeps** everything that draws the experience: the video element, the WebGL
Kuwahara dream shader, the HUD/annotation + affect overlay, the
ring-transition playback, the media preload pool, the crossfade-loop logic.
- **Removes** the input controls (the Visual toggle, the Audio `<select>`, the
Altitude SVG, the three sliders) and their handlers.
- **Keeps** everything that draws the experience: the video element, the **A/B
`<audio>` crossfade layer**, the WebGL Kuwahara dream shader, the
HUD/annotation + affect overlay, the ring-transition playback, the media
preload pool, the crossfade-loop logic.
- **Adds** a thin `state` client: open `EventSource('/api/events')`; on `state`
apply controls→render live; on `ring` play the transition clip(s) then settle
and `POST /api/render/event {settled}`; on `mode` run the fade. On (re)connect,
apply controls→render live (incl. the audio crossfade); on `ring` play the
transition clip(s) — crossfading the soundtrack to the event's `audio.url`
then settle and `POST /api/render/event {settled}`; on `mode` run the video
fade. On (re)connect,
pull `GET /api/state` to resync. SSE auto-reconnects, so a flaky link
self-heals.
- Runs full-screen on the laptop driving the projector; no keyboard/mouse needed
@@ -222,16 +242,16 @@ server: transitions.altitude="idle" → SSE "state" to all → iPad clears spinn
### 6.2 Controller — `/remote`, a new touch-first page (the faithful twin)
- The five controls laid out like the intended physical panel: **Content**
selector, **Altitude** encoder dial (reuse the existing SVG dial component),
**Left** / **Right** / **Mood** knobs.
- The six controls laid out like the intended physical panel: **Visual** toggle +
**Audio** selector, **Altitude** encoder dial (reuse the existing SVG dial
component), **Left** / **Right** / **Mood** knobs.
- On any input → `POST /api/control` with just the changed field. Live knobs post
on drag; Altitude posts encoder ticks; Content posts on selection.
on drag; Altitude posts encoder ticks; Visual/Audio post on change.
- Per-knob transition state (§4): a transitional control goes **pending** on
touch, reflects **transitioning** from the SSE feed, clears on **settled**.
Altitude shows a settling indicator and its detents soft-lock (further ticks
queue as the next target rather than stacking); Content shows the selection as
pending until the fade completes.
queue as the next target rather than stacking); **Visual** shows the toggle as
pending until the video fade completes; **Audio** is live (no pending state).
- It **also** subscribes to `/api/events` so it stays in sync with other
sources (turn the future Arduino's knob and the iPad reflects it, and
vice-versa). On load it GETs `/api/state` to draw current positions.
+54
View File
@@ -0,0 +1,54 @@
"""Resolve the orthogonal Visual × Audio controls into render outputs (audio spec
§4/§6). The single source of truth for: whether the projector shows video, and
which audio url plays for a given (audio source, current altitude).
Replaces player/content.py's 7-way bundled table. White-noise is a global,
altitude-independent bed; soundtrack follows the single Altitude dial; off is
silence. `music` is a reserved, deferred source not in v1."""
from __future__ import annotations
from dataclasses import dataclass
VISUAL_POSITIONS = frozenset({"on", "off"})
# v1 audio sources. "music" is reserved/deferred (no assets) and intentionally absent.
AUDIO_SOURCES = frozenset({"off", "soundtrack", "white_noise"})
@dataclass(frozen=True)
class AudioResolution:
source: str
url: str | None
altitude_coupled: bool
def resolve_visual(position: str) -> bool:
"""Visual on/off → whether the renderer shows video (vs fade-to-black)."""
if position not in VISUAL_POSITIONS:
raise ValueError(
f"unknown visual position {position!r}; expected one of {sorted(VISUAL_POSITIONS)}"
)
return position == "on"
def resolve_audio_url(source: str, *, scale_audio: str, noise_url: str) -> str | None:
"""The audio url for (source, current altitude): soundtrack → the current
scale's asset (or None if none authored); white_noise → the global bed; off →
None. `scale_audio` is the ring scale's `audio` path (relative to /media/audio/)."""
if source not in AUDIO_SOURCES:
raise ValueError(
f"unknown audio source {source!r}; expected one of {sorted(AUDIO_SOURCES)}"
)
if source == "off":
return None
if source == "white_noise":
return noise_url
# soundtrack — couple to the current altitude's asset
return f"/media/audio/{scale_audio}" if scale_audio else None
def resolve_audio(source: str, *, scale_audio: str, noise_url: str) -> AudioResolution:
"""The full render.audio view: source, resolved url, and whether it follows the
Altitude dial (true only for soundtrack)."""
url = resolve_audio_url(source, scale_audio=scale_audio, noise_url=noise_url)
return AudioResolution(source=source, url=url, altitude_coupled=(source == "soundtrack"))
-42
View File
@@ -1,42 +0,0 @@
"""Resolve the 7-way content dial into an audio source + video on/off (§6).
The single source of truth for design §6's table: which audio source plays and
whether the projector shows video, for each dial position.
"""
from __future__ import annotations
from dataclasses import dataclass
# The four distinct audio sources. "white_noise" is generated at runtime;
# "music" is the public-domain classical pool; "audio_track" is the clip's own
# field audio; "none" is silence.
AUDIO_SOURCES = frozenset({"none", "white_noise", "music", "audio_track"})
@dataclass(frozen=True)
class ContentResolution:
audio_source: str
video: bool
# Design §6, one row per dial position.
_TABLE = {
"off": ContentResolution("none", False),
"white_noise": ContentResolution("white_noise", False),
"music": ContentResolution("music", False),
"audio_track": ContentResolution("audio_track", False),
"video": ContentResolution("none", True),
"music_video": ContentResolution("music", True),
"audio_video": ContentResolution("audio_track", True),
}
def resolve_content(position: str) -> ContentResolution:
"""Map a content-dial position to its audio source and video flag (§6)."""
try:
return _TABLE[position]
except KeyError:
raise ValueError(
f"unknown content position {position!r}; expected one of {sorted(_TABLE)}"
) from None
+15 -12
View File
@@ -1,20 +1,17 @@
"""Control-panel state: the data shape read from the Arduino over serial.
This is the serial-contract payload shared with sub-project 4 (firmware). It
models the full panel from design §6/§7: the 7-way content dial, the four
experience knobs (0..4), and the two intensity levels (volume, brightness).
The wire framing itself is the separate 3<->4 serial contract; this module is
the *decoded* form.
models the full panel from the audio spec §3: a Visual toggle (on/off) + an
Audio source selector (off/soundtrack/white_noise), the four experience knobs
(0..4), and the two intensity levels (volume, brightness). The wire framing
itself is the separate 3<->4 serial contract; this module is the *decoded* form.
"""
from __future__ import annotations
from dataclasses import dataclass, fields
# The seven positions of the content dial (design §6).
CONTENT_POSITIONS = frozenset(
{"off", "white_noise", "music", "audio_track", "video", "music_video", "audio_video"}
)
from player.audio import AUDIO_SOURCES, VISUAL_POSITIONS
KNOB_FIELDS = ("left", "right", "dark", "light", "volume", "brightness")
KNOB_MIN = 0
@@ -27,7 +24,8 @@ class ControlsError(ValueError):
@dataclass(frozen=True)
class Controls:
content: str
visual: str
audio: str
left: int
right: int
dark: int
@@ -38,10 +36,15 @@ class Controls:
def validate_controls(c: Controls) -> None:
"""Raise ControlsError if the payload is structurally invalid."""
if c.content not in CONTENT_POSITIONS:
if c.visual not in VISUAL_POSITIONS:
raise ControlsError(
f"invalid content position {c.content!r}; "
f"expected one of {sorted(CONTENT_POSITIONS)}"
f"invalid visual position {c.visual!r}; "
f"expected one of {sorted(VISUAL_POSITIONS)}"
)
if c.audio not in AUDIO_SOURCES:
raise ControlsError(
f"invalid audio source {c.audio!r}; "
f"expected one of {sorted(AUDIO_SOURCES)}"
)
for name in KNOB_FIELDS:
value = getattr(c, name)
+1
View File
@@ -63,6 +63,7 @@ class Scale:
id: str
clip_id: str
pool: tuple[str, ...] = ()
audio: str = "" # per-scale soundtrack, relative to /media/audio/ (audio spec §5.1)
@property
def members(self) -> tuple[str, ...]:
+14 -8
View File
@@ -20,7 +20,7 @@ from typing import Callable, Optional
from hef.selection import Coordinate
from player.alteration import RenderPlan, plan_alteration
from player.content import ContentResolution, resolve_content
from player.audio import resolve_visual
from player.controls import Controls
@@ -34,11 +34,14 @@ class TransitionKind:
@dataclass(frozen=True)
class Playback:
"""What should currently be playing."""
"""What should currently be playing. Visual and audio are orthogonal (audio
spec §2): `video` is whether the projector shows footage, `audio_source` is
the independent audio dial (off / soundtrack / white_noise)."""
clip_id: Optional[str] # None = black walls
plan: Optional[RenderPlan] # None when black
content: ContentResolution
video: bool # whether footage is shown (Visual on/off)
audio_source: str # the Audio dial source (off/soundtrack/white_noise)
volume: int
brightness: int
@@ -58,7 +61,8 @@ def _first(library):
_BLACK = Playback(
clip_id=None,
plan=None,
content=ContentResolution("none", False),
video=False,
audio_source="off",
volume=0,
brightness=0,
)
@@ -78,12 +82,13 @@ class Player:
self._current = _BLACK
def _resolve(self, controls: Controls) -> Playback:
content = resolve_content(controls.content)
if not content.video:
video = resolve_visual(controls.visual)
if not video:
return Playback(
clip_id=None,
plan=None,
content=content,
video=False,
audio_source=controls.audio,
volume=controls.volume,
brightness=controls.brightness,
)
@@ -92,7 +97,8 @@ class Player:
return Playback(
clip_id=clip.id,
plan=plan_alteration(coord),
content=content,
video=True,
audio_source=controls.audio,
volume=controls.volume,
brightness=controls.brightness,
)
+6
View File
@@ -24,3 +24,9 @@ sim = [
"uvicorn[standard]>=0.29",
"httpx>=0.27",
]
# Playwright E2E browser tests (audio spec §9). Install with the browser binary:
# pip install -e '.[e2e]' && python -m playwright install chromium
# The E2E suite skips cleanly when Playwright/Chromium is absent (headless box).
e2e = [
"pytest-playwright>=0.4",
]
@@ -1,22 +0,0 @@
# Session 0020.0 — Transcript
> App: human-experience-filter-art
> Start: 2026-06-26T06-47 (PST)
> Type: writing-plans
> Posture: careful
> Claude-Session: e0cc053a-41fa-416d-a4f5-493f14984e75
> Status: **PLACEHOLDER — claimed at session start; finalized at session end.**
>
> This file reserves session ID 0020 for human-experience-filter-art. The driver replaces this
> body with the full transcript and renames the file to its final
> SESSION-0020.0-TRANSCRIPT-2026-06-26T06-47--<end>.md form at session end.
## Launch prompt
_(launch prompt not captured at claim time)_
## Deferred decisions
_Autonomous-mode low-confidence calls the driver made and would have
liked operator input on. Appended as the session runs; surfaced at
finalize. Empty if none._
-3
View File
@@ -55,8 +55,5 @@
},
"0019": {
"title": ""
},
"0020": {
"title": ""
}
}
+25 -7
View File
@@ -26,8 +26,7 @@ from player.alteration import (
plan_alteration,
render_plan_to_dict,
)
from player.content import resolve_content
from player.controls import CONTENT_POSITIONS
from player.audio import AUDIO_SOURCES, VISUAL_POSITIONS, resolve_audio, resolve_visual
from player.ring import (
DEFAULT_FAST_SPIN_THRESHOLD,
advance_ring,
@@ -40,6 +39,9 @@ STATIC_DIR = Path(__file__).parent / "static"
MEDIA_DIR = Path(__file__).parent / "sample_media"
DEFAULT_MANIFEST = MEDIA_DIR / "manifest.json"
# The global white-noise bed (audio spec §5.2): synthesized, altitude-independent.
NOISE_URL = "/media/audio/noise/pink.mp3"
# Per-process boot token: changes on every server restart so the dev live-reload
# (below) also fires when Python code changes (which needs a restart), not just
# when a static asset's mtime changes.
@@ -95,7 +97,8 @@ def _rev_file(file: str) -> str:
class ControlsModel(BaseModel):
content: str
visual: str
audio: str
left: int = Field(ge=0, le=4)
right: int = Field(ge=0, le=4)
dark: int = Field(ge=0, le=4)
@@ -112,6 +115,7 @@ class CalibrationModel(BaseModel):
class AlterationRequest(BaseModel):
controls: ControlsModel
altitude_index: int = 0
calibration: Optional[CalibrationModel] = None
@@ -166,8 +170,10 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
@app.post("/api/alteration")
def api_alteration(req: AlterationRequest):
c = req.controls
if c.content not in CONTENT_POSITIONS:
raise HTTPException(status_code=422, detail=f"invalid content {c.content!r}")
if c.visual not in VISUAL_POSITIONS:
raise HTTPException(status_code=422, detail=f"invalid visual {c.visual!r}")
if c.audio not in AUDIO_SOURCES:
raise HTTPException(status_code=422, detail=f"invalid audio {c.audio!r}")
coord = Coordinate(c.left, c.right, c.dark, c.light)
cal = (
Calibration(
@@ -179,10 +185,22 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
else DEFAULT_CALIBRATION
)
plan = plan_alteration(coord, cal)
content = resolve_content(c.content)
# Resolve the soundtrack url against the CURRENT altitude (server-side, audio
# spec §6.1). White-noise/off ignore the scale; soundtrack follows it.
scale_audio = ""
if app.state.ring is not None:
scale_audio = scale_at(app.state.ring, req.altitude_index).audio
audio = resolve_audio(c.audio, scale_audio=scale_audio, noise_url=NOISE_URL)
return {
"plan": render_plan_to_dict(plan),
"content": {"audio_source": content.audio_source, "video": content.video},
"render": {
"video": {"shown": resolve_visual(c.visual)},
"audio": {
"source": audio.source,
"url": audio.url,
"altitude_coupled": audio.altitude_coupled,
},
},
}
@app.get("/dev/version")
+43
View File
@@ -0,0 +1,43 @@
"""Production pass for the 5 per-altitude soundtracks + the white-noise bed
(audio spec §5.3). The audio analogue of build_pool_manifest.py --media: read the
sourced ambience clips (re-downloadable per docs/audio-candidate-pool.md), make
each a seamless loudness-normalized loop, and synthesize the pink-noise bed.
Media is gitignored; this script is the reproducible record. Outputs land under
simulator/sample_media/audio/<scale>/<name>.loop.mp3 and audio/noise/pink.mp3,
served at /media/audio/... Run: python simulator/build_audio_media.py
"""
from __future__ import annotations
from pathlib import Path
from tools.pipeline.audio_run import generate_white_noise, process_soundtrack
AUDIO = Path(__file__).parent / "sample_media" / "audio"
# scale -> (sourced raw file, production output file), both under audio/<scale>/.
# Sources per docs/audio-candidate-pool.md (the "✓ KEEP" picks).
SOURCES: dict[str, tuple[str, str]] = {
"cosmos": ("pillars.mp3", "pillars.loop.mp3"),
"orbit": ("spaceamb.mp3", "spaceamb.loop.mp3"),
"coast": ("waves.mp3", "waves.loop.mp3"),
"reef": ("soundscape.mp3", "soundscape.loop.mp3"),
"abyss": ("whale.wav", "whale.loop.mp3"),
}
def main() -> None:
for scale, (raw, out) in SOURCES.items():
src = AUDIO / scale / raw
if not src.exists():
print(f" SKIP {scale}: source missing ({src}) — re-source per audio-candidate-pool.md")
continue
dst = process_soundtrack(src, AUDIO / scale / out)
print(f" produced {dst.relative_to(AUDIO.parent)}")
noise = generate_white_noise(AUDIO / "noise" / "pink.mp3")
print(f" produced {noise.relative_to(AUDIO.parent)}")
if __name__ == "__main__":
main()
+14 -1
View File
@@ -48,6 +48,16 @@ POOLS: dict[str, list[str]] = {
# Ring order (large -> small, wraps): cosmos -> orbit -> coast -> reef -> abyss -> cosmos.
RING_ORDER = ["cosmos", "orbit", "coast", "reef", "abyss"]
# Per-scale soundtrack (audio spec §5.1) — the production-pass output of
# simulator/build_audio_media.py, served at /media/audio/<path>.
SCALE_AUDIO: dict[str, str] = {
"cosmos": "cosmos/pillars.loop.mp3",
"orbit": "orbit/spaceamb.loop.mp3",
"coast": "coast/waves.loop.mp3",
"reef": "reef/soundscape.loop.mp3",
"abyss": "abyss/whale.loop.mp3",
}
PD = "public-domain (US Gov, 17 U.S.C. §105)"
PD_NPS = "public-domain (NPS, no © — 17 U.S.C. §105)"
CCBY_STSCI = "CC-BY-class — credit NASA, ESA, STScI"
@@ -357,7 +367,10 @@ def build_manifest() -> dict:
for scale in RING_ORDER:
for clip_id in POOLS[scale]:
clips.append(_clip_entry(scale, clip_id))
scales = [{"id": s, "clip_id": POOLS[s][0], "pool": POOLS[s]} for s in RING_ORDER]
scales = [
{"id": s, "clip_id": POOLS[s][0], "pool": POOLS[s], "audio": SCALE_AUDIO[s]}
for s in RING_ORDER
]
transitions = []
n = len(RING_ORDER)
for i in range(n):
+2 -1
View File
@@ -82,7 +82,7 @@ def _scale_from_dict(s: dict[str, Any]) -> Scale:
"""
pool = tuple(s.get("pool", []))
clip_id = s.get("clip_id") or (pool[0] if pool else "")
return Scale(id=s["id"], clip_id=clip_id, pool=pool)
return Scale(id=s["id"], clip_id=clip_id, pool=pool, audio=s.get("audio", ""))
def load_ring(path: str | Path) -> ScaleRing | None:
@@ -119,6 +119,7 @@ def ring_to_dict(ring: ScaleRing, clips: list[Clip]) -> dict:
"id": s.id,
"clip_id": s.clip_id,
"title": titles.get(s.clip_id, s.id),
"audio": s.audio,
"pool": [
{"clip_id": cid, "title": titles.get(cid, cid)}
for cid in s.members
+10 -5
View File
@@ -2371,7 +2371,8 @@
"cosmos_galaxies",
"cosmos_hudf",
"cosmos_xdf"
]
],
"audio": "cosmos/pillars.loop.mp3"
},
{
"id": "orbit",
@@ -2380,7 +2381,8 @@
"orbit_planetearth",
"orbit_crewobs",
"orbit_bluemarble"
]
],
"audio": "orbit/spaceamb.loop.mp3"
},
{
"id": "coast",
@@ -2390,7 +2392,8 @@
"coast_surfgrass",
"coast_elkbeach",
"coast_drakesbeach"
]
],
"audio": "coast/waves.loop.mp3"
},
{
"id": "reef",
@@ -2401,7 +2404,8 @@
"reef_hawkfish",
"reef_snapper",
"reef_coralspacific"
]
],
"audio": "reef/soundscape.loop.mp3"
},
{
"id": "abyss",
@@ -2410,7 +2414,8 @@
"abyss_wow",
"abyss_midwaterexp",
"abyss_hiding"
]
],
"audio": "abyss/whale.loop.mp3"
}
],
"transitions": [
+79 -8
View File
@@ -96,7 +96,10 @@ let mediaVersions = {}; // file -> content-hash token (from /api/media-version
// changes with its bytes (permanent cache-bust). The blob cache, when present,
// wins — those bytes are already the current ones for this session.
function mediaNetUrl(file) { const v = mediaVersions[file]; return "/media/" + file + (v ? "?v=" + v : ""); }
function mediaUrl(file) { return mediaBlobs[file] || mediaNetUrl(file); }
function mediaUrl(file) {
if (file.startsWith("/media/")) return file; // already a resolved absolute url (audio layer)
return mediaBlobs[file] || mediaNetUrl(file);
}
// Every media file the ring can show: each scale-pool clip's base footage plus the
// per-edge transition clips. This is the full preload set (~two dozen files).
@@ -492,12 +495,14 @@ function renderScaleReadout() {
}
function controls() {
// One bipolar Mood dial (-4 dark .. 0 neutral .. +4 light) maps onto the engine's
// two poles. The calibration gains are pinned to their locked 1.0 (full dial = full
// effect), so the simulator sends no calibration — the server uses DEFAULT_CALIBRATION.
// Visual (on/off) and Audio (off/soundtrack/white_noise) are orthogonal dials
// (audio spec §2). One bipolar Mood dial (-4 dark .. +4 light) maps onto the
// engine's two poles. Calibration gains are pinned to 1.0, so the simulator
// sends no calibration — the server uses DEFAULT_CALIBRATION.
const mood = +$("mood").value;
return {
content: $("content").value,
visual: $("visual").checked ? "on" : "off",
audio: $("audio").value,
left: +$("left").value, right: +$("right").value,
dark: mood < 0 ? -mood : 0, light: mood > 0 ? mood : 0,
volume: 2, brightness: 2,
@@ -509,12 +514,13 @@ async function update() {
if (busy) return;
const resp = await fetch("/api/alteration", {
method: "POST", headers: { "content-type": "application/json" },
body: JSON.stringify({ controls: controls() }),
body: JSON.stringify({ controls: controls(), altitude_index: ringIndex }),
});
if (!resp.ok) { readout.textContent = "invalid: " + resp.status; return; }
const data = await resp.json();
readout.textContent = JSON.stringify(data, null, 2);
if (!data.content.video) { black.style.opacity = "1"; black.classList.remove("hidden"); return; }
applyAudio(data.render.audio); // reconcile the audio layer
if (!data.render.video.shown) { black.style.opacity = "1"; black.classList.remove("hidden"); return; }
black.classList.add("hidden");
try {
ensureClipMedia();
@@ -894,6 +900,70 @@ function devLiveReload() {
}, 1000);
}
// --- Audio layer: two <audio> elements, gain-crossfaded (audio spec §7/§8) ---
const audA = $("audA"), audB = $("audB");
let audActive = audA, audIdle = audB; // which element is currently audible
let audUrl = null; // the url currently playing (null = silence)
let audioReady = false; // unlocked after the first user gesture (autoplay policy)
const XFADE_MS = 600; // ~0.6s gain crossfade — "live", no busy ack (§7)
function fadeVolume(el, to, ms) {
// setInterval (not requestAnimationFrame): rAF throttles/pauses when the tab is
// backgrounded, which would stall the gain ramp; a timer fires regardless.
const from = el.volume, steps = Math.max(1, Math.round(ms / 30));
let i = 0;
return new Promise((resolve) => {
const iv = setInterval(() => {
i += 1;
const k = Math.min(1, i / steps);
el.volume = Math.min(1, Math.max(0, from + (to - from) * k));
if (k >= 1) { clearInterval(iv); resolve(); }
}, 30);
});
}
// Reconcile the audio layer to render.audio: crossfade the active element to the
// new url (or fade to silence for off). White-noise and soundtrack are the same
// mechanism — only the url differs; an unchanged url is a no-op so altitude
// re-rolls on the same scale don't restart the bed.
async function applyAudio(audio) {
const url = audio ? audio.url : null;
if (url === audUrl) return;
audUrl = url;
if (!audioReady) return; // deferred until the start gesture (§8)
if (!url) { // fade current out to silence
await fadeVolume(audActive, 0, XFADE_MS);
audActive.pause();
return;
}
audIdle.src = mediaUrl(url); // absolute /media/... passes through mediaUrl
audIdle.volume = 0;
audIdle.play().catch(() => {}); // start (fire-and-forget — awaiting can hang)
await Promise.all([fadeVolume(audIdle, 1, XFADE_MS), fadeVolume(audActive, 0, XFADE_MS)]);
audActive.pause();
[audActive, audIdle] = [audIdle, audActive]; // swap roles
}
// Browsers block autoplay until a user gesture. The renderer is launched
// full-screen by the operator; a one-time tap unlocks audio, then it follows
// state (audio spec §8 — the documented launch step).
function showStartGesture() {
if (audioReady || document.getElementById("start-gesture")) return;
const ov = document.createElement("div");
ov.id = "start-gesture";
ov.textContent = "▶ tap to start sound";
ov.addEventListener("click", () => {
// A click anywhere on the page grants Chrome's autoplay permission
// document-wide for the session, so later programmatic play() is allowed —
// no per-element priming needed (priming srcless elements only pollutes state).
audioReady = true;
ov.remove();
audUrl = null; // force the next applyAudio to (re)load
update(); // re-apply current state, now with audio
}, { once: true });
document.body.appendChild(ov);
}
async function main() {
devLiveReload();
try { initPaint(); } catch (e) { paintOK = false; paint.style.display = "none"; showError("WebGL init: " + e.message); }
@@ -903,9 +973,10 @@ async function main() {
buildDial(); // draw the altitude knob from the ring's scales
initDev(); // wire the Dev Mode toggle + pool picker (reads persisted state)
renderScaleReadout();
for (const id of ["content", "left", "right", "mood"]) {
for (const id of ["visual", "audio", "left", "right", "mood"]) {
$(id).addEventListener("input", debounced);
}
showStartGesture(); // one-time tap unlocks audio (browser autoplay policy, §8)
// Altitude knob: drag to turn (commit detents on release), scroll to step, tap a label to jump.
dial.addEventListener("pointerdown", onDialDown);
window.addEventListener("pointermove", onDialMove);
+12 -9
View File
@@ -12,6 +12,8 @@
<section class="stage" id="stage">
<div class="screen">
<video id="vid" loop muted playsinline></video>
<audio id="audA" loop preload="auto"></audio>
<audio id="audB" loop preload="auto"></audio>
<canvas id="paint"></canvas>
<div id="tint"></div>
<svg id="overlay" viewBox="0 0 100 100" preserveAspectRatio="none"></svg>
@@ -22,15 +24,16 @@
<section class="panel">
<fieldset>
<legend>Content dial</legend>
<select id="content">
<option value="video">video</option>
<option value="audio_video">audio + video</option>
<option value="music_video">music + video</option>
<option value="off">off (black)</option>
<option value="white_noise">white noise (no video)</option>
<option value="music">music (no video)</option>
<option value="audio_track">audio track (no video)</option>
<legend>Visual</legend>
<label class="toggle"><input type="checkbox" id="visual" checked /> show video</label>
</fieldset>
<fieldset>
<legend>Audio</legend>
<select id="audio">
<option value="off">off (silence)</option>
<option value="soundtrack">soundtrack (follows altitude)</option>
<option value="white_noise">white noise</option>
</select>
</fieldset>
+10
View File
@@ -107,3 +107,13 @@ input[type=range], select { width: 100%; }
.dev-anno .anno-key.measure { color: #ffd79a; }
.dev-anno .anno-meta { color: #678; }
.dev-anno .anno-empty { color: #567; font-style: italic; }
/* One-time tap-to-start overlay unlocks audio under the browser autoplay
policy (audio spec §8). Removed after the first gesture. */
#start-gesture {
position: fixed; inset: 0; z-index: 10000;
display: flex; align-items: center; justify-content: center;
background: rgba(0, 0, 0, 0.82); color: #fff;
font: 600 28px/1.2 system-ui, sans-serif; letter-spacing: 0.02em;
cursor: pointer; user-select: none;
}
+98
View File
@@ -0,0 +1,98 @@
"""Playwright E2E for the audio layer (audio spec §9). Skips cleanly when
Playwright or its browser binary is absent (the headless dev box) the unit +
contract tiers cover the logic; this asserts the rendered <audio> element src and
playback survives a Visual fade. Runs at the PPE stage where a browser exists.
Install: pip install -e '.[e2e]' && python -m playwright install chromium
"""
import socket
import threading
import time
from contextlib import closing
import pytest
# Skip the whole module unless Playwright is importable.
pytest.importorskip("playwright.sync_api")
def _free_port() -> int:
with closing(socket.socket()) as s:
s.bind(("127.0.0.1", 0))
return s.getsockname()[1]
@pytest.fixture(scope="module")
def app_url():
import uvicorn
from simulator.app import app
port = _free_port()
config = uvicorn.Config(app, host="127.0.0.1", port=port, log_level="error")
server = uvicorn.Server(config)
thread = threading.Thread(target=server.run, daemon=True)
thread.start()
for _ in range(50):
if server.started:
break
time.sleep(0.1)
if not server.started:
pytest.skip("uvicorn did not start")
yield f"http://127.0.0.1:{port}"
server.should_exit = True
thread.join(timeout=5)
@pytest.fixture
def page(app_url):
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
try:
# Headless Chromium blocks media playback without a real audio device;
# this flag lets play() actually start so `paused` is meaningful (the
# spec §9 asserts playback via element src/paused, not a waveform).
browser = p.chromium.launch(args=["--autoplay-policy=no-user-gesture-required"])
except Exception as exc: # no browser binary installed
pytest.skip(f"chromium not available: {exc}")
pg = browser.new_page()
pg.goto(app_url)
pg.click("#start-gesture") # unlock autoplay
yield pg
browser.close()
def test_white_noise_loads_the_noise_asset(page):
page.select_option("#audio", "white_noise")
page.wait_for_function(
"['audA','audB'].some(id => "
"document.getElementById(id).src.includes('/media/audio/noise/pink.mp3'))"
)
def test_soundtrack_loads_the_current_scale_asset(page):
# initial altitude is cosmos (index 0) → soundtrack resolves its scale asset,
# not the noise bed (the altitude-coupling itself is covered at the API tier)
page.select_option("#audio", "soundtrack")
page.wait_for_function(
"['audA','audB'].some(id => {"
" const s = document.getElementById(id).src;"
" return /\\/media\\/audio\\/.+\\.mp3/.test(s) && !s.includes('noise');"
"})"
)
def test_visual_off_keeps_audio_and_fades_video(page):
page.select_option("#audio", "white_noise")
page.wait_for_function(
"['audA','audB'].some(id => "
"document.getElementById(id).src.includes('/media/audio/noise/pink.mp3'))"
)
page.uncheck("#visual")
page.wait_for_selector("#black:not(.hidden)") # video faded to black
playing = page.evaluate(
"['audA','audB'].some(id => "
"{const a = document.getElementById(id); return !a.paused && !!a.src;})"
)
assert playing is True # audio survives the video fade
+65
View File
@@ -0,0 +1,65 @@
"""Unit tests for the pure audio ffmpeg arg builders (no ffmpeg run), plus an
opt-in integration test that actually synthesizes a noise bed when ffmpeg is
present."""
import shutil
import pytest
from tools.pipeline.audio_ops import audio_loop_args, loudnorm_args, white_noise_args
def test_loop_args_mirror_the_video_crossfade_recipe():
args = audio_loop_args("in.mp3", "out.mp3", duration=30.0, overlap=2.0, ff="FF")
assert args[0] == "FF"
joined = " ".join(args)
# tail crossfades over head, then concats the middle — the audio analogue of
# crossfade_loop_args. Output length = duration - overlap.
assert "acrossfade=d=2.0" in joined
assert "atrim=0:2.0" in joined # head
assert "atrim=28.0:30.0" in joined # tail (d-overlap : d)
assert "concat=n=2:v=0:a=1" in joined
assert "-map" in args and "[out]" in args
assert "-q:a" in args and "4" in args
def test_loop_args_reject_overlap_past_half():
with pytest.raises(ValueError):
audio_loop_args("in.mp3", "out.mp3", duration=10.0, overlap=5.0)
def test_loudnorm_args_carry_the_locked_targets():
args = loudnorm_args("in.mp3", "out.mp3", ff="FF")
joined = " ".join(args)
assert "loudnorm=I=-18.0:TP=-1.5:LRA=11.0" in joined
assert args[0] == "FF" and args[-1] == "out.mp3"
def test_white_noise_args_are_pink_and_clean():
args = white_noise_args("noise.mp3", duration=60.0, color="pink", ff="FF")
joined = " ".join(args)
assert "anoisesrc=color=pink" in joined
assert "-t" in args and "60.0" in args
assert args[-1] == "noise.mp3"
def test_white_noise_rejects_unknown_color():
with pytest.raises(ValueError):
white_noise_args("noise.mp3", color="ultraviolet")
def _have_ffmpeg() -> bool:
if shutil.which("ffmpeg"):
return True
try:
import imageio_ffmpeg # noqa: F401
return True
except Exception:
return False
@pytest.mark.skipif(not _have_ffmpeg(), reason="no ffmpeg available")
def test_generate_white_noise_writes_a_playable_loop(tmp_path):
from tools.pipeline.audio_run import generate_white_noise
out = generate_white_noise(tmp_path / "pink.mp3", duration=2.0)
assert out.exists() and out.stat().st_size > 0
+63
View File
@@ -0,0 +1,63 @@
"""Unit tests for the pure Visual×Audio resolver (audio spec §4/§6)."""
import pytest
from player.audio import (
AUDIO_SOURCES,
VISUAL_POSITIONS,
AudioResolution,
resolve_audio,
resolve_audio_url,
resolve_visual,
)
NOISE = "/media/audio/noise/pink.mp3"
def test_visual_on_off():
assert resolve_visual("on") is True
assert resolve_visual("off") is False
def test_visual_rejects_unknown():
with pytest.raises(ValueError):
resolve_visual("flicker")
def test_visual_positions_are_on_off():
assert VISUAL_POSITIONS == frozenset({"on", "off"})
def test_soundtrack_resolves_to_the_current_scales_asset():
url = resolve_audio_url("soundtrack", scale_audio="coast/waves.loop.mp3", noise_url=NOISE)
assert url == "/media/audio/coast/waves.loop.mp3"
def test_white_noise_resolves_to_the_global_bed_independent_of_altitude():
url = resolve_audio_url("white_noise", scale_audio="coast/waves.loop.mp3", noise_url=NOISE)
assert url == NOISE
def test_off_resolves_to_silence():
assert resolve_audio_url("off", scale_audio="coast/waves.loop.mp3", noise_url=NOISE) is None
def test_soundtrack_with_no_scale_asset_is_silent():
assert resolve_audio_url("soundtrack", scale_audio="", noise_url=NOISE) is None
def test_resolve_audio_marks_only_soundtrack_altitude_coupled():
r = resolve_audio("soundtrack", scale_audio="reef/soundscape.loop.mp3", noise_url=NOISE)
assert isinstance(r, AudioResolution)
assert r.source == "soundtrack" and r.altitude_coupled is True
assert r.url == "/media/audio/reef/soundscape.loop.mp3"
assert resolve_audio("white_noise", scale_audio="x/y.mp3", noise_url=NOISE).altitude_coupled is False
assert resolve_audio("off", scale_audio="x/y.mp3", noise_url=NOISE).altitude_coupled is False
def test_resolve_audio_url_rejects_unknown_source():
with pytest.raises(ValueError):
resolve_audio_url("podcast", scale_audio="", noise_url=NOISE)
def test_music_is_not_a_v1_source():
assert "music" not in AUDIO_SOURCES
-34
View File
@@ -1,34 +0,0 @@
import pytest
from player.content import AUDIO_SOURCES, ContentResolution, resolve_content
def test_audio_sources_are_the_four_distinct_sources():
assert AUDIO_SOURCES == frozenset({"none", "white_noise", "music", "audio_track"})
# The §6 table, row by row: position -> (audio_source, video).
@pytest.mark.parametrize(
"position,audio_source,video",
[
("off", "none", False),
("white_noise", "white_noise", False),
("music", "music", False),
("audio_track", "audio_track", False),
("video", "none", True),
("music_video", "music", True),
("audio_video", "audio_track", True),
],
)
def test_resolve_content_matches_spec_table(position, audio_source, video):
assert resolve_content(position) == ContentResolution(audio_source=audio_source, video=video)
def test_off_is_void_state_black_and_silent():
r = resolve_content("off")
assert r.video is False and r.audio_source == "none"
def test_resolve_content_rejects_unknown_position():
with pytest.raises(ValueError):
resolve_content("bogus")
+24 -13
View File
@@ -1,27 +1,36 @@
import pytest
from player.audio import AUDIO_SOURCES, VISUAL_POSITIONS
from player.controls import (
Controls,
CONTENT_POSITIONS,
ControlsError,
validate_controls,
parse_controls,
)
def test_content_positions_are_the_seven_from_the_spec():
assert CONTENT_POSITIONS == frozenset(
{"off", "white_noise", "music", "audio_track", "video", "music_video", "audio_video"}
)
def test_visual_and_audio_enums_are_the_v1_positions():
assert VISUAL_POSITIONS == frozenset({"on", "off"})
assert AUDIO_SOURCES == frozenset({"off", "soundtrack", "white_noise"})
def test_valid_controls_pass_validation():
c = Controls(content="video", left=0, right=4, dark=2, light=2, volume=3, brightness=4)
c = Controls(visual="on", audio="soundtrack", left=0, right=4, dark=2, light=2,
volume=3, brightness=4)
validate_controls(c) # does not raise
def test_invalid_content_position_rejected():
c = Controls(content="bogus", left=0, right=0, dark=0, light=0, volume=0, brightness=0)
def test_invalid_visual_position_rejected():
c = Controls(visual="flicker", audio="off", left=0, right=0, dark=0, light=0,
volume=0, brightness=0)
with pytest.raises(ControlsError):
validate_controls(c)
def test_invalid_audio_source_rejected():
# "music" is a reserved/deferred dial position, not a v1 source
c = Controls(visual="on", audio="music", left=0, right=0, dark=0, light=0,
volume=0, brightness=0)
with pytest.raises(ControlsError):
validate_controls(c)
@@ -29,7 +38,8 @@ def test_invalid_content_position_rejected():
@pytest.mark.parametrize("field", ["left", "right", "dark", "light", "volume", "brightness"])
@pytest.mark.parametrize("bad", [-1, 5, True])
def test_out_of_range_or_non_int_knob_rejected(field, bad):
kwargs = dict(content="off", left=0, right=0, dark=0, light=0, volume=0, brightness=0)
kwargs = dict(visual="off", audio="off", left=0, right=0, dark=0, light=0,
volume=0, brightness=0)
kwargs[field] = bad
with pytest.raises(ControlsError):
validate_controls(Controls(**kwargs))
@@ -37,17 +47,18 @@ def test_out_of_range_or_non_int_knob_rejected(field, bad):
def test_parse_controls_from_mapping():
c = parse_controls(
{"content": "music_video", "left": 1, "right": 2, "dark": 3, "light": 0, "volume": 2, "brightness": 1}
{"visual": "on", "audio": "white_noise", "left": 1, "right": 2, "dark": 3,
"light": 0, "volume": 2, "brightness": 1}
)
assert c == Controls("music_video", 1, 2, 3, 0, 2, 1)
assert c == Controls("on", "white_noise", 1, 2, 3, 0, 2, 1)
def test_parse_controls_rejects_unknown_keys():
with pytest.raises(ControlsError):
parse_controls({"content": "off", "left": 0, "right": 0, "dark": 0,
parse_controls({"visual": "off", "audio": "off", "left": 0, "right": 0, "dark": 0,
"light": 0, "volume": 0, "brightness": 0, "bogus": 1})
def test_parse_controls_rejects_missing_keys():
with pytest.raises(ControlsError):
parse_controls({"content": "off", "left": 0})
parse_controls({"visual": "off", "left": 0})
+7
View File
@@ -240,3 +240,10 @@ def test_pick_clip_id_clamps_out_of_range_r():
assert pick_clip_id(s, -5.0) == "a" # below 0 clamps to first
assert pick_clip_id(s, 1.0) == "c" # 1.0 must not index past the end
assert pick_clip_id(s, 999.0) == "c"
def test_scale_carries_an_optional_audio_path():
s = Scale(id="coast", clip_id="coast_birdrock", audio="coast/waves.loop.mp3")
assert s.audio == "coast/waves.loop.mp3"
# back-compat: audio defaults to empty
assert Scale(id="x", clip_id="x").audio == ""
+35 -25
View File
@@ -14,47 +14,57 @@ class FakeClip:
LIB = [FakeClip("base-a"), FakeClip("base-b")]
def _controls(content="video", left=0, right=0, dark=0, light=0, volume=2, brightness=2):
return Controls(content, left, right, dark, light, volume, brightness)
def _controls(visual="on", audio="off", left=0, right=0, dark=0, light=0,
volume=2, brightness=2):
return Controls(visual, audio, left, right, dark, light, volume, brightness)
def test_first_update_to_video_fades_in_from_black():
p = Player(LIB)
t = p.update(_controls(content="video"))
t = p.update(_controls(visual="on"))
assert t.kind == TransitionKind.FADE_FROM_BLACK
assert t.playback.clip_id == "base-a"
assert t.playback.content.video is True
assert t.playback.video is True
def test_off_from_video_fades_to_black_and_silences():
def test_visual_off_from_video_fades_to_black():
p = Player(LIB)
p.update(_controls(content="video"))
t = p.update(_controls(content="off"))
p.update(_controls(visual="on"))
t = p.update(_controls(visual="off"))
assert t.kind == TransitionKind.FADE_TO_BLACK
assert t.playback.clip_id is None
assert t.playback.content.audio_source == "none"
assert t.playback.video is False
assert t.playback.audio_source == "off"
def test_audio_survives_visual_off():
# white noise + black is now reachable (the gap the bundled dial omitted)
p = Player(LIB)
t = p.update(_controls(visual="off", audio="white_noise"))
assert t.playback.video is False
assert t.playback.audio_source == "white_noise"
def test_no_change_yields_none_transition():
p = Player(LIB)
p.update(_controls(content="video", left=1))
t = p.update(_controls(content="video", left=1))
p.update(_controls(visual="on", left=1))
t = p.update(_controls(visual="on", left=1))
assert t.kind == TransitionKind.NONE
def test_grade_change_is_a_live_update_not_a_crossfade():
# design §4.3: the Dark/Light grade is a continuous runtime op
p = Player(LIB)
p.update(_controls(content="video", dark=0, light=0))
t = p.update(_controls(content="video", dark=4, light=0))
p.update(_controls(visual="on", dark=0, light=0))
t = p.update(_controls(visual="on", dark=4, light=0))
assert t.kind == TransitionKind.LIVE_UPDATE
assert t.playback.plan.grade.tone == -1.0
def test_overlay_change_is_a_live_update():
p = Player(LIB)
p.update(_controls(content="video", left=0))
t = p.update(_controls(content="video", left=4))
p.update(_controls(visual="on", left=0))
t = p.update(_controls(visual="on", left=4))
assert t.kind == TransitionKind.LIVE_UPDATE
assert t.playback.plan.overlay.level == 4
@@ -63,8 +73,8 @@ def test_dream_change_is_a_live_update_not_a_crossfade():
# Right-axis dream reframe (session 0013): the Right dream is a deterministic
# LIVE filter now, so changing it no longer crossfades — only a clip swap does.
p = Player(LIB)
p.update(_controls(content="video", right=0))
t = p.update(_controls(content="video", right=4))
p.update(_controls(visual="on", right=0))
t = p.update(_controls(visual="on", right=4))
assert t.kind == TransitionKind.LIVE_UPDATE
assert t.playback.plan.dream.strength == 4
assert t.playback.plan.dream.intensity == 1.0
@@ -72,43 +82,43 @@ def test_dream_change_is_a_live_update_not_a_crossfade():
def test_volume_only_change_is_a_live_update():
p = Player(LIB)
p.update(_controls(content="video", volume=1))
t = p.update(_controls(content="video", volume=4))
p.update(_controls(visual="on", volume=1))
t = p.update(_controls(visual="on", volume=4))
assert t.kind == TransitionKind.LIVE_UPDATE
assert t.playback.volume == 4
def test_audio_source_change_while_black_is_a_live_update():
p = Player(LIB)
p.update(_controls(content="white_noise"))
t = p.update(_controls(content="music"))
p.update(_controls(visual="off", audio="white_noise"))
t = p.update(_controls(visual="off", audio="soundtrack"))
assert t.kind == TransitionKind.LIVE_UPDATE
assert t.playback.content.audio_source == "music"
assert t.playback.audio_source == "soundtrack"
def test_injected_base_chooser_is_used():
p = Player(LIB, choose_base=lambda lib: lib[1])
t = p.update(_controls(content="video"))
t = p.update(_controls(visual="on"))
assert t.playback.clip_id == "base-b"
def test_empty_library_with_video_raises():
p = Player([])
with pytest.raises(ValueError):
p.update(_controls(content="video"))
p.update(_controls(visual="on"))
def test_off_with_empty_library_is_fine():
p = Player([])
# levels at 0 match the initial black state, so this is a no-op transition
t = p.update(_controls(content="off", volume=0, brightness=0))
t = p.update(_controls(visual="off", volume=0, brightness=0))
assert t.kind == TransitionKind.NONE # already black at init
assert t.playback.clip_id is None
def test_off_with_levels_from_black_is_a_live_update():
p = Player([])
t = p.update(_controls(content="off", volume=3, brightness=2))
t = p.update(_controls(visual="off", volume=3, brightness=2))
assert t.kind == TransitionKind.LIVE_UPDATE # black->black, levels set
assert t.playback.clip_id is None
assert t.playback.volume == 3
+56 -12
View File
@@ -28,8 +28,9 @@ def client(manifest_path):
return TestClient(create_app(manifest_path=manifest_path))
def _controls(content="video", left=0, right=0, dark=0, light=0, volume=2, brightness=2):
return dict(content=content, left=left, right=right, dark=dark,
def _controls(visual="on", audio="off", left=0, right=0, dark=0, light=0,
volume=2, brightness=2):
return dict(visual=visual, audio=audio, left=left, right=right, dark=dark,
light=light, volume=volume, brightness=brightness)
@@ -41,13 +42,42 @@ def test_alteration_returns_the_engine_plan(client):
assert data["plan"]["dream"]["strength"] == 2
assert data["plan"]["dream"]["intensity"] == 0.5
assert data["plan"]["grade"]["tone"] == -1.0
assert data["content"]["video"] is True
assert data["render"]["video"]["shown"] is True
def test_alteration_honors_off_as_black(client):
resp = client.post("/api/alteration", json={"controls": _controls(content="off")})
data = resp.json()
assert data["content"]["video"] is False
def test_alteration_returns_render_video_and_audio(client):
body = client.post("/api/alteration", json={"controls": _controls()}).json()
assert body["render"]["video"]["shown"] is True
assert body["render"]["audio"] == {"source": "off", "url": None, "altitude_coupled": False}
def test_alteration_visual_off_hides_video(client):
body = client.post("/api/alteration", json={"controls": _controls(visual="off")}).json()
assert body["render"]["video"]["shown"] is False
def test_alteration_white_noise_is_altitude_independent(ring_client):
for idx in (0, 1, 2):
body = ring_client.post("/api/alteration", json={
"controls": _controls(audio="white_noise"), "altitude_index": idx,
}).json()
assert body["render"]["audio"]["url"] == "/media/audio/noise/pink.mp3"
assert body["render"]["audio"]["altitude_coupled"] is False
def test_alteration_soundtrack_couples_to_the_given_altitude(ring_client):
# ring fixture order: cosmos(0), forest(1), abyss(2) — each carries its audio
body = ring_client.post("/api/alteration", json={
"controls": _controls(audio="soundtrack"), "altitude_index": 0,
}).json()
a = body["render"]["audio"]
assert a["source"] == "soundtrack" and a["altitude_coupled"] is True
assert a["url"] == "/media/audio/cosmos/pillars.loop.mp3"
# a different altitude resolves a different soundtrack url
body2 = ring_client.post("/api/alteration", json={
"controls": _controls(audio="soundtrack"), "altitude_index": 2,
}).json()
assert body2["render"]["audio"]["url"] == "/media/audio/abyss/whale.loop.mp3"
def test_alteration_accepts_calibration(client):
@@ -62,8 +92,13 @@ def test_alteration_rejects_out_of_range_knob(client):
assert resp.status_code == 422
def test_alteration_rejects_bad_content(client):
resp = client.post("/api/alteration", json={"controls": _controls(content="banana")})
def test_alteration_rejects_bad_audio(client):
resp = client.post("/api/alteration", json={"controls": _controls(audio="music")})
assert resp.status_code == 422
def test_alteration_rejects_bad_visual(client):
resp = client.post("/api/alteration", json={"controls": _controls(visual="dim")})
assert resp.status_code == 422
@@ -162,9 +197,9 @@ def ring_manifest_path(tmp_path):
],
"ring": {
"scales": [
{"id": "cosmos", "clip_id": "cosmos"},
{"id": "forest", "clip_id": "forest"},
{"id": "abyss", "clip_id": "abyss"},
{"id": "cosmos", "clip_id": "cosmos", "audio": "cosmos/pillars.loop.mp3"},
{"id": "forest", "clip_id": "forest", "audio": "forest/amb.loop.mp3"},
{"id": "abyss", "clip_id": "abyss", "audio": "abyss/whale.loop.mp3"},
],
"transitions": [
{"file": "transitions/cosmos-forest.mp4", "model": "placeholder"},
@@ -190,6 +225,15 @@ def test_ring_returns_scales_and_transitions(ring_client):
assert len(data["transitions"]) == 3
def test_ring_exposes_each_scales_audio(ring_client):
body = ring_client.get("/api/ring").json()
by_id = {s["id"]: s for s in body["scales"]}
# every scale advertises its soundtrack path (audio spec §5.1)
assert by_id["cosmos"]["audio"] == "cosmos/pillars.loop.mp3"
assert by_id["abyss"]["audio"].endswith(".mp3")
assert "audio" in by_id["forest"]
def test_ring_advance_inward_one_step(ring_client):
resp = ring_client.post("/api/ring/advance", json={"from_index": 0, "delta": 1})
assert resp.status_code == 200
+57
View File
@@ -0,0 +1,57 @@
"""Pure ffmpeg argument builders (no I/O) for the audio production pass.
The audio analogue of tools/pipeline/ffmpeg_ops.py: each function returns a
list[str] ready for ffmpeg, so command construction is unit-testable without
running ffmpeg. tools/pipeline/audio_run.py executes them. The ffmpeg binary
name is injected (default "ffmpeg")."""
from __future__ import annotations
_NOISE_COLORS = frozenset({"white", "pink", "brown", "blue", "violet"})
def audio_loop_args(src, dst, *, duration: float, overlap: float,
ff: str = "ffmpeg") -> list[str]:
"""Make `src` loop seamlessly by crossfading its tail over its head — the
audio mirror of ffmpeg_ops.crossfade_loop_args. Output length = duration -
overlap. Requires overlap < duration/2 so a non-empty middle remains."""
if overlap <= 0 or overlap >= duration / 2:
raise ValueError(f"overlap {overlap} must be in (0, duration/2={duration / 2})")
d, o = duration, overlap
fc = (
f"[0:a]atrim=0:{o},asetpts=N/SR/TB[head];"
f"[0:a]atrim={o}:{d - o},asetpts=N/SR/TB[mid];"
f"[0:a]atrim={d - o}:{d},asetpts=N/SR/TB[tail];"
f"[tail][head]acrossfade=d={o}:c1=tri:c2=tri[xf];"
f"[xf][mid]concat=n=2:v=0:a=1[out]"
)
return [
ff, "-y", "-i", str(src), "-filter_complex", fc,
"-map", "[out]", "-c:a", "libmp3lame", "-q:a", "4", str(dst),
]
def loudnorm_args(src, dst, *, i: float = -18.0, tp: float = -1.5,
lra: float = 11.0, ff: str = "ffmpeg") -> list[str]:
"""EBU R128 loudness normalize to the locked installation target so all five
soundtracks (and the noise bed) sit at one comfortable level."""
return [
ff, "-y", "-i", str(src),
"-af", f"loudnorm=I={i}:TP={tp}:LRA={lra}",
"-c:a", "libmp3lame", "-q:a", "4", str(dst),
]
def white_noise_args(dst, *, duration: float = 60.0, color: str = "pink",
ff: str = "ffmpeg") -> list[str]:
"""Synthesize a calm colored-noise bed (default pink) of `duration` seconds —
deterministic, zero licensing. Loops cleanly (steady-state noise has no seam)."""
if color not in _NOISE_COLORS:
raise ValueError(
f"unknown noise color {color!r}; expected one of {sorted(_NOISE_COLORS)}"
)
return [
ff, "-y", "-f", "lavfi",
"-i", f"anoisesrc=color={color}:amplitude=0.5:duration={duration}",
"-t", str(duration), "-c:a", "libmp3lame", "-q:a", "4", str(dst),
]
+61
View File
@@ -0,0 +1,61 @@
"""Thin runner: probe duration, then run the audio production pass (seamless
loop -> loudness normalize) and synthesize the white-noise bed. The pure arg
builders (tools.pipeline.audio_ops) are unit-tested; this glue is covered by the
opt-in integration test."""
from __future__ import annotations
import re
import subprocess
from pathlib import Path
from .audio_ops import audio_loop_args, loudnorm_args, white_noise_args
from .run import resolve_ffmpeg
def _probe_audio_duration(src, ff: str) -> float:
"""Audio length in seconds via ffmpeg (no system ffprobe): decode to null and
read the last reported time off stderr."""
proc = subprocess.run(
[ff, "-i", str(src), "-f", "null", "-"],
capture_output=True, text=True,
)
times = re.findall(r"time=(\d+):(\d+):(\d+\.\d+)", proc.stderr)
if not times:
raise ValueError(f"could not probe audio duration of {src}")
h, m, s = times[-1]
return int(h) * 3600 + int(m) * 60 + float(s)
def _run(args: list[str]) -> None:
subprocess.run(args, check=True, capture_output=True)
def process_soundtrack(src, dst, *, overlap: float = 2.0, ff: str | None = None) -> Path:
"""Produce a seamless, loudness-normalized loop from a sourced ambience clip.
Clips too short to crossfade-loop (<= 2*overlap) are normalized only."""
ff = ff or resolve_ffmpeg()
src, dst = Path(src), Path(dst)
dst.parent.mkdir(parents=True, exist_ok=True)
duration = _probe_audio_duration(src, ff)
tmp = dst.with_suffix(".loop.tmp.mp3")
if duration > 2 * overlap + 0.5:
_run(audio_loop_args(src, tmp, duration=duration, overlap=overlap, ff=ff))
_run(loudnorm_args(tmp, dst, ff=ff))
tmp.unlink(missing_ok=True)
else:
_run(loudnorm_args(src, dst, ff=ff))
return dst
def generate_white_noise(dst, *, duration: float = 60.0, color: str = "pink",
ff: str | None = None) -> Path:
"""Synthesize + normalize the global white-noise bed."""
ff = ff or resolve_ffmpeg()
dst = Path(dst)
dst.parent.mkdir(parents=True, exist_ok=True)
tmp = dst.with_suffix(".raw.tmp.mp3")
_run(white_noise_args(tmp, duration=duration, color=color, ff=ff))
_run(loudnorm_args(tmp, dst, ff=ff))
tmp.unlink(missing_ok=True)
return dst