Compare commits
17 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 30ec0c9c26 | |||
| fddb6d65d4 | |||
| 2722949805 | |||
| d446238b79 | |||
| c51fdc422a | |||
| aa76cd1c9f | |||
| c07067492b | |||
| fd0aad6610 | |||
| 68f9d7342a | |||
| 29c6d822bd | |||
| 1f9b2a0bec | |||
| 39fb9108db | |||
| 062a283de4 | |||
| adc39fe648 | |||
| 99a751cb46 | |||
| 6f2d1c4f7b | |||
| a3ec09af7e |
+11
-1
@@ -1,5 +1,6 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.DS_Store
|
||||
.pytest_cache/
|
||||
.venv/
|
||||
media/
|
||||
@@ -8,5 +9,14 @@ media/
|
||||
# Simulator sample media (look-tuning only; populate via setup_sample_media.py
|
||||
# / build_pool_manifest.py). All scale + transition binaries are gitignored.
|
||||
simulator/sample_media/**/*.mp4
|
||||
# Local-only candidate review gallery (re-buildable; not shipped content).
|
||||
# Audio candidate clips (re-sourceable via docs/audio-candidate-pool.md).
|
||||
simulator/sample_media/**/*.wav
|
||||
simulator/sample_media/**/*.mp3
|
||||
simulator/sample_media/**/*.ogg
|
||||
simulator/sample_media/**/*.m4a
|
||||
# Local-only candidate review galleries (re-buildable; not shipped content).
|
||||
simulator/static/review.html
|
||||
simulator/static/review_audio.html
|
||||
|
||||
# Editor annotation/comment-thread metadata (not project content)
|
||||
.threads/
|
||||
|
||||
+16
-1
@@ -196,8 +196,23 @@ every transition, so fast navigation stays responsive on placeholder media.
|
||||
browser. Real strict-PD bases for all five scales sourced per
|
||||
[`docs/content-sourcing.md`](./content-sourcing.md).
|
||||
|
||||
- **Audio + separated Visual/Audio controls (session 0020) ✅ Done.** The
|
||||
simulator now has **sound**: per-altitude soundtracks that follow the Altitude
|
||||
dial + a synthesized pink **white-noise** bed, behind two orthogonal controls —
|
||||
**Visual** (on/off) × **Audio** (off / soundtrack / white_noise) — replacing
|
||||
the bundled 7-way Content dial. `tools/pipeline/audio_ops` + `audio_run` +
|
||||
`simulator/build_audio_media.py` (ffmpeg loop/loudnorm/noise, gitignored
|
||||
assets), `Scale.audio` in the manifest, a pure `player/audio.py` resolver
|
||||
(retiring `player/content.py`), server-resolved `render.audio.url`, and a client
|
||||
A/B `<audio>` crossfade layer. This is the "Content step" of the future control
|
||||
surface (it reframes the proposed networked-control-surface spec §3/§5). Spec:
|
||||
[`2026-06-26-audio-video-separated-controls-design.md`](./superpowers/specs/2026-06-26-audio-video-separated-controls-design.md);
|
||||
plan:
|
||||
[`2026-06-26-audio-video-separated-controls.md`](./superpowers/plans/2026-06-26-audio-video-separated-controls.md).
|
||||
|
||||
- **Still deferred:** real **i2v ring transitions** (need a generative-video
|
||||
model + adjacent real bases); Pi renderer + serial/firmware.
|
||||
model + adjacent real bases); Pi renderer + serial/firmware; the **music**
|
||||
audio layer (reserved dial position, no assets yet).
|
||||
- **Catalog model changes** — audio *source* + "neutral base" vs "altered
|
||||
variant" flag (sub-project 2 territory, design §13).
|
||||
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
# Audio candidate pool — rotating pools per altitude (2026-06-26)
|
||||
|
||||
Candidate soundtrack/ambience for the **"audio + video" setting**. Following operator
|
||||
feedback, each altitude now has a **rotating pool** (mirrors the video `Scale.pool`
|
||||
model — random pick per landing, or pairable per-clip).
|
||||
|
||||
## Two audio layers per altitude (operator, 2026-06-26)
|
||||
|
||||
The "audio + video" setting layers **two** kinds of audio per altitude:
|
||||
1. **Ambience / audio track** — the soundscape (waves, reef crackle, chorus, etc.).
|
||||
**This is what we're sourcing now.**
|
||||
2. **Music track** — a separate composed musical layer. **Deferred** until the
|
||||
ambience tracks are settled across all altitudes.
|
||||
|
||||
Pools below are the **ambience** layer unless marked `music`. **Distinction (operator):**
|
||||
the ambience/soundtrack layer is an *environmental, real-world soundscape* (waves, reef,
|
||||
whale, room-tone hum) — **not composed music**. A musical pad/track, however nice, belongs
|
||||
in the music layer, even for "atmospheric" altitudes like cosmos/orbit.
|
||||
|
||||
**Media is gitignored** (`simulator/sample_media/**/*.{wav,mp3,…}`); this file is the
|
||||
re-sourceable record. Files: `simulator/sample_media/audio/<altitude>/<name>.<ext>`,
|
||||
served at `/media/audio/<altitude>/<name>.<ext>`. Review gallery (gitignored):
|
||||
`simulator/static/review_audio.html` → `/review_audio.html` with the sim running.
|
||||
|
||||
## Round-2 feedback applied (operator, session 2026-06-26)
|
||||
|
||||
- **abyss** ✓ kept (blue-whale moan) · **coast** ✓ kept (waves + terns)
|
||||
- **cosmos** ✗ plasma static "hurt my ears" → replaced with 3 NASA **sonifications**
|
||||
- **orbit** ✗ whistlers "sound like lasers" → replaced with gentle Earth **chorus** (+ wind)
|
||||
- **reef** ✗ "shouldn't have whales (too deep for a reef)" → whale removed; now crackle + diver
|
||||
- Operator: *"try all three… rotation of audio like the video, or per-video audio track."*
|
||||
|
||||
## Design arc
|
||||
|
||||
Descending the dial is one journey: **cosmos & abyss** = vast/dark/sparse siblings;
|
||||
**coast** = warm human middle; **orbit & reef** = transitional membranes. The science
|
||||
recordings (NASA/NOAA) keep the "machines reshaping perception" thesis literal.
|
||||
|
||||
## Pools
|
||||
|
||||
| altitude | id | clip | source | dur | license |
|
||||
|---|---|---|---|---|---|
|
||||
| 🌌 cosmos | `cosmos/pillars` | **✓ KEEP** — Pillars of Creation (M16), twinkly sweep | NASA/CXC/SAO — https://chandra.si.edu/sound/m16.html (`sounds/m16_all.mp4`) | 30s | PD (NASA, credit) |
|
||||
| 🌌 cosmos | `cosmos/music` | 🎵 **music track — TBD** (deferred). Black-hole drone DROPPED (too creepy, not awe-inspiring); Galactic Center dropped earlier | — | — | — |
|
||||
| 🛰 orbit | `orbit/spaceamb` | **✓ KEEP** — station room-tone hum ("inside the orbiting craft") | Sonicfreak — https://freesound.org/s/174450/ (`cdn.freesound.org/previews/174/174450_746632-hq.mp3`) | 2:31 | CC0 |
|
||||
| 🛰 (music) | _deferred_ | 🎵 "Frozen Star" warm pad — moved to MUSIC layer (it's music, not ambience) | Kevin MacLeod / incompetech — https://incompetech.com/music/royalty-free/mp3-royaltyfree/Frozen%20Star.mp3 | 3:41 | CC-BY |
|
||||
| 🐠 reef | `reef/soundscape` | **✓** richer reef (round 4) — 6-layer bake: submerged underwater bed (bubbles/sloosh) + shrimp crackle + fish chorus + toadfish + grouper + damselfish | NOAA SanctSound (FK+GR `*_snappingshrimp_*`, `FK02_01_fishchorus`, `GR01_01_toadfish`, `FK03_01_redgrouper`, `PM05_01_damselfish`) + DCSFX "Underwater [Loop] AMB" https://freesound.org/s/366159/ (`cdn.freesound.org/previews/366/366159_6725579-hq.mp3`); loop-to-60 + amix + dynaudnorm | 60s | PD (NOAA) + CC0 (underwater) |
|
||||
| 🏖 coast | `coast/waves` | Sea waves + tern calls | BigSoundBank #0267 — https://bigsoundbank.com/sea-waves-and-seagulls-s0267.html (`/UPLOAD/mp3/0267.mp3`) | 57s | CC0 |
|
||||
| 🕳 abyss | `abyss/whale` | NE Pacific blue-whale "AB call" | NOAA PMEL — https://www.pmel.noaa.gov/acoustics/multimedia/NETS_bluWhale.wav | 13s | PD (NOAA) |
|
||||
|
||||
## Sourcing rule (learned 2026-06-26)
|
||||
|
||||
**Source only from a browsable page the operator can open and whose license is stated.**
|
||||
The Mixkit clips were pulled by raw CDN asset id (`assets.mixkit.co/.../<id>-preview.mp3`)
|
||||
with no findable catalog page — operator couldn't see them, so they were removed. Good
|
||||
hubs: NASA SVS / Chandra, NOAA PMEL/SanctSound (PD), BigSoundBank (CC0), Pixabay
|
||||
(visible license). Avoid asset-id grabs and ND/NC licenses.
|
||||
|
||||
## Open items / next pass
|
||||
|
||||
- **orbit** — ✅ SETTLED: `orbit/spaceamb` (Sonicfreak station room-tone hum, CC0). Rejected
|
||||
en route: laser whistlers, Earth chorus, plain wind, "Frozen Star" pad (music). A real NASA
|
||||
ISS recording (Hadfield "Space Station Noise", SoundCloud) remains the authentic upgrade if
|
||||
ever wanted — needs a harder fetch (no yt-dlp; brew install denied).
|
||||
- **ambience layer COMPLETE** — all 5 picked: cosmos/pillars · orbit/spaceamb · coast/waves ·
|
||||
reef/soundscape · abyss/whale. Next is the production pass + wiring (below) and the music layer.
|
||||
- **reef** — ✅ round-4 6-layer bake (`reef/soundscape`): NOAA PD biological layers + a CC0
|
||||
submerged underwater "you're under" bed (DCSFX freesound #366159, hq preview). License clean.
|
||||
SanctSound clips: `files/SanctSound_*.mp4` at `https://sanctsound.ioos.us/files/` (extract audio).
|
||||
- **music layer** — separate composed music track per altitude, deferred (cosmos slot reserved).
|
||||
- **Production (after picks):** loop seams, length normalization, cross-altitude crossfade
|
||||
(mirror video); cosmos/abyss clips are short → loop.
|
||||
|
||||
## Re-download / re-bake
|
||||
|
||||
```
|
||||
A=simulator/sample_media/audio; FF=$(python -c "import imageio_ffmpeg;print(imageio_ffmpeg.get_ffmpeg_exe())")
|
||||
mkdir -p $A/cosmos $A/orbit $A/reef $A/coast $A/abyss
|
||||
# NASA sonifications (mp4 -> mp3)
|
||||
for u in perseus_sonification:blackhole m16_all:pillars galactic_all:galaxy; do
|
||||
f=${u%%:*}; o=${u##*:}; curl -fsSL "https://chandra.si.edu/sound/sounds/$f.mp4" -o /tmp/$o.mp4
|
||||
"$FF" -y -i /tmp/$o.mp4 -vn -q:a 4 $A/cosmos/$o.mp3; done
|
||||
curl -fsSL "https://svs.gsfc.nasa.gov/vis/a010000/a011000/a011073/Earthsong-540-MASTER_high.mp4" -o /tmp/es.mp4
|
||||
"$FF" -y -i /tmp/es.mp4 -vn -q:a 4 $A/orbit/earthsong.mp3
|
||||
curl -fsSL "https://assets.mixkit.co/active_storage/sfx/1162/1162-preview.mp3" -o $A/orbit/wind.mp3
|
||||
curl -fsSL "https://www.nhm.ac.uk/content/dam/nhm-www/discover/audio/sound-of-coral-reef.mp3" -o $A/reef/crackle.mp3
|
||||
curl -fsSL "https://assets.mixkit.co/active_storage/sfx/1242/1242-preview.mp3" -o $A/reef/scuba.mp3
|
||||
curl -fsSL "https://bigsoundbank.com/UPLOAD/mp3/0267.mp3" -o $A/coast/waves.mp3
|
||||
curl -fsSL "https://www.pmel.noaa.gov/acoustics/multimedia/NETS_bluWhale.wav" -o $A/abyss/whale.wav
|
||||
# layered reef
|
||||
"$FF" -y -i $A/reef/crackle.mp3 -i $A/reef/scuba.mp3 \
|
||||
-filter_complex "[1:a]volume=0.55[s];[0:a][s]amix=inputs=2:duration=first,dynaudnorm" -q:a 4 $A/reef/layered.mp3
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,274 @@
|
||||
# Audio + Video — separated Visual & Audio controls — design
|
||||
|
||||
**Status:** proposed → built (sessions 0020 build + 0021 revision)
|
||||
**Date:** 2026-06-26
|
||||
**Session:** 0020 (audio) · 0021 (live-UI revision)
|
||||
|
||||
> **v1 build revision (session 0021, operator-driven):** the live **Audio** control
|
||||
> ships as a simple **off / soundtrack** toggle (styled like the Dev Mode switch),
|
||||
> and **Video** is the matching on/off toggle. **White-noise is deferred** with music
|
||||
> (the orthogonal Visual × Audio model and the synthesized-pink-noise machinery
|
||||
> remain in the codebase for when it's wanted). Two real-browser bugs were fixed in
|
||||
> this revision: video-off now blanks GPU-composited layers (z-index + explicit layer
|
||||
> hide), and the soundtrack now plays on Safari/iOS (audio elements unlocked
|
||||
> synchronously inside the start gesture). Sections below describe the fuller
|
||||
> off/soundtrack/white-noise design; v1 live = off/soundtrack.
|
||||
>
|
||||
> **Startup revision (session 0022):** the tap-to-start wall is **removed** — the app
|
||||
> opens behind a **"Loading Universe…"** splash (shown until all media is preloaded
|
||||
> and the experience can run smoothly), then both toggles start **off** (black +
|
||||
> silent). Autoplay is satisfied without a wall: a **single** `<audio>` element is
|
||||
> played **synchronously inside the toggle's own click** (the reliable Safari unlock),
|
||||
> replacing the A/B-crossfade-after-gesture approach that failed on real Safari. The
|
||||
> first time **Video** is turned on, **Audio** turns on with it (one flip = the full
|
||||
> experience); turning Audio on first stays audio-only. Altitude changes swap the
|
||||
> single element's source with a brief fade.
|
||||
**Extends / reframes:** [`2026-06-26-networked-control-surface-design.md`](./2026-06-26-networked-control-surface-design.md)
|
||||
§3 (control inventory) and §5 (server contract) — its bundled **Content** selector
|
||||
is split into two orthogonal controls. Everything else in that spec (server-of-truth
|
||||
+ SSE, the two pages, the Arduino seam, transition lifecycle) stands unchanged.
|
||||
**Asset record:** [`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md).
|
||||
|
||||
---
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Give the experience **sound**, and make audio and video **independently
|
||||
controllable**. Today the control surface bundles audio+video into a single 7-way
|
||||
**Content** selector (*video / audio+video / music+video / off / white noise /
|
||||
music / audio track*). That bundle is both awkward to operate and **incomplete** —
|
||||
it has no "white noise **with** video," for instance.
|
||||
|
||||
Replace it with **two orthogonal dials** — **Visual** and **Audio** — so every
|
||||
combination is reachable, including the ones the bundle omitted. Operator framing
|
||||
(2026-06-26): *"separate these dials… it's ok to let people have e.g. white noise
|
||||
with video."*
|
||||
|
||||
## 2. Core idea: orthogonalize Content into Visual × Audio
|
||||
|
||||
```
|
||||
BEFORE (bundled) AFTER (orthogonal)
|
||||
Content: 7-way selector Visual: on / off
|
||||
video Audio: off / soundtrack / white noise
|
||||
audio+video (· music — deferred)
|
||||
music+video ──▶ ───────────────────────────────────────
|
||||
off Any Visual × Any Audio is now reachable:
|
||||
white noise video + soundtrack (was "audio+video")
|
||||
music video + white noise (NEW — was missing)
|
||||
audio track black + white noise (was "white noise")
|
||||
black + soundtrack (was "audio track")
|
||||
```
|
||||
|
||||
The 7 bundled modes were really points in a **2 × N grid** (visual on/off ×
|
||||
audio source). Making the grid explicit removes the awkwardness and fills the gap.
|
||||
The **Altitude** dial is unchanged and remains the single scale selector.
|
||||
|
||||
## 3. The controls (reframes control-surface §3)
|
||||
|
||||
| Control | Positions (v1) | Drives | Class (§4 of control-surface spec) |
|
||||
|---|---|---|---|
|
||||
| **Altitude** | endless dial, cosmos→abyss (wraps) | the visual **scale** **and**, when Audio=soundtrack, the matching ambience | transitional |
|
||||
| **Visual** | on / off | video shown vs. black | transitional (video fade) |
|
||||
| **Audio** | off / soundtrack / white noise (· *music* deferred) | the audio **source** | live + short crossfade (§7) |
|
||||
| **Left** / **Right** / **Mood** | unchanged | the alteration engine | live |
|
||||
|
||||
So the panel grows from 5 controls to **6** (Content → Visual + Audio). The
|
||||
physical form (later): Visual = a toggle/2-pos switch; Audio = a small rotary
|
||||
selector (3-pos in v1, 4 with music).
|
||||
|
||||
## 4. Audio model (the coupling decision)
|
||||
|
||||
**Decision (operator):** the audio dial is a **source** selector; the per-altitude
|
||||
soundtrack **follows the single Altitude dial**. There is **no** separate audio
|
||||
altitude.
|
||||
|
||||
- **`soundtrack`** → plays the ambience of the **current Altitude scale**. Turning
|
||||
Altitude to coast shows coast video *and* plays coast waves; the soundtrack
|
||||
**crossfades** as the altitude transition settles (mirrors the video ring
|
||||
crossfade). One dial, two coupled outputs.
|
||||
- **`white noise`** → a single global bed, **altitude-independent**. Changing
|
||||
Altitude does not change it. This is what makes "white noise + any video" work.
|
||||
- **`off`** → silence. Video (if Visual=on) plays mute.
|
||||
- **`music`** *(deferred)* → the composed per-altitude music layer. Reserved as a
|
||||
future Audio position; **not in v1** (no assets yet — see
|
||||
[`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md) "two audio layers").
|
||||
|
||||
Independence summary: **Visual ⟂ Audio** (any pairing allowed); **Audio source ⟂
|
||||
Altitude** for white noise/music, **coupled to Altitude** for soundtrack.
|
||||
|
||||
## 5. Assets & manifest
|
||||
|
||||
### 5.1 Per-scale soundtrack
|
||||
|
||||
The five ambiences are sourced and reviewed (PD/CC0; record in
|
||||
[`docs/audio-candidate-pool.md`](../../audio-candidate-pool.md), local-only review
|
||||
gallery `simulator/static/review_audio.html`):
|
||||
|
||||
| scale | soundtrack | license |
|
||||
|---|---|---|
|
||||
| cosmos | Pillars of Creation sonification | PD (NASA) |
|
||||
| orbit | station room-tone hum | CC0 |
|
||||
| coast | sea waves + terns | CC0 |
|
||||
| reef | 6-layer reef soundscape | PD (NOAA) + CC0 |
|
||||
| abyss | blue-whale moan | PD (NOAA) |
|
||||
|
||||
The manifest's `Scale` gains an **`audio`** field (path to the scale's soundtrack),
|
||||
parallel to how clips are referenced; media stays gitignored and is served at
|
||||
`/media/audio/<scale>/<file>`. Soundtracks are a **per-scale single** (not a
|
||||
rotating pool) in v1 — one bed per altitude. (A rotating audio pool, like the clip
|
||||
pool, is a possible later extension; out of v1.)
|
||||
|
||||
### 5.2 White noise
|
||||
|
||||
Altitude-independent global asset. **Synthesized, not sourced** — a calm
|
||||
**pink/brown** noise (gentler than pure white) generated deterministically with
|
||||
`ffmpeg -f lavfi -i anoisesrc=color=pink` → a clean ~60 s loop. Zero licensing
|
||||
concerns. (Colloquially "white noise"; spec a calm colored noise.)
|
||||
|
||||
### 5.3 Production pass (assets)
|
||||
|
||||
Before ship the source clips get a light, deterministic pass (the existing
|
||||
`tools/pipeline` ffmpeg approach): **seamless loop** (crossfade the loop point),
|
||||
**normalize loudness** across the five so altitudes match, and a consistent format
|
||||
(mp3/ogg). Several are short (abyss 13 s, cosmos 30 s) and must loop cleanly. The
|
||||
6-layer reef bake is already loop-length (60 s).
|
||||
|
||||
## 6. Server contract (reframes control-surface §5)
|
||||
|
||||
The control-surface spec's authoritative `SessionState` + four endpoints are
|
||||
unchanged in shape; only the **controls** payload changes: `content` is replaced
|
||||
by `visual` + `audio`, and the derived `render.content` is split.
|
||||
|
||||
### 6.1 `GET /api/state` — updated `controls` + `render`
|
||||
|
||||
```json
|
||||
{
|
||||
"seq": 42,
|
||||
"controls": { "visual": "on", "audio": "soundtrack", "left": 3, "right": 1, "mood": -2 },
|
||||
"altitude": { "index": 2, "scale": "coast", "clip_id": "coast_07" },
|
||||
"transitions": { "altitude": "idle", "visual": "idle" },
|
||||
"render": {
|
||||
"plan": { },
|
||||
"video": { "shown": true },
|
||||
"audio": { "source": "soundtrack", "url": "/media/audio/coast/waves.mp3", "altitude_coupled": true }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `audio.url` is resolved **server-side** from `(audio source, altitude)`:
|
||||
`soundtrack` → the current scale's `audio`; `white_noise` → the global noise
|
||||
asset; `off` → `null`. The renderer just plays `audio.url` (or nothing).
|
||||
- `transitions` tracks `visual` instead of `content` (Audio is not transitional —
|
||||
§7).
|
||||
|
||||
### 6.2 `POST /api/control`
|
||||
|
||||
```json
|
||||
{ "set": { "audio": "white_noise" } } // audio source change
|
||||
{ "set": { "visual": "off" } } // visual fade to black
|
||||
{ "set": {}, "altitude_delta": -1 } // altitude tick (also re-resolves audio.url when source=soundtrack)
|
||||
```
|
||||
|
||||
Server applies, bumps `seq`, recomputes `render.audio.url`, and broadcasts.
|
||||
Absolute `set` values keep retries idempotent (control-surface §5.6).
|
||||
|
||||
### 6.3 SSE events
|
||||
|
||||
- `state` — carries the new controls/render; emitted on Audio change and on any
|
||||
live change (Audio source change rides the `state` event — §7).
|
||||
- `ring` — altitude move; now **also** carries the new `audio.url` so the renderer
|
||||
can crossfade the soundtrack as it plays the visual transition (when
|
||||
source=soundtrack).
|
||||
- `mode` — repurposed to the **Visual** fade (on/off). (Same renderer fade machinery
|
||||
the bundled Content used.)
|
||||
|
||||
## 7. Live vs. transitional classification
|
||||
|
||||
- **Visual on/off → transitional.** It's a visible video fade (to/from black); it
|
||||
uses the renderer's existing fade + `settled` ack, exactly as the bundled
|
||||
Content's fade did. `transitions.visual` flips `transitioning`→`idle`.
|
||||
- **Audio source → live, with a short renderer-side crossfade.** Swapping
|
||||
soundtrack/white-noise/off is a ~0.5–1 s **gain crossfade** between two `<audio>`
|
||||
elements — perceptible but **non-blocking** and not a renderer animation with
|
||||
unknown timing. So it's classified **live** (no busy spinner, no `settled` ack),
|
||||
like Left/Right/Mood. *Flagged decision:* if the operator wants the Audio knob to
|
||||
feel "busy" during its fade, it can be promoted to transitional later; v1 keeps it
|
||||
live for simplicity.
|
||||
- **Altitude → transitional** (unchanged). When source=soundtrack, the renderer
|
||||
crossfades the audio bed in step with the ring transition; the audio crossfade is
|
||||
slaved to the visual transition and needs no separate ack.
|
||||
|
||||
## 8. Renderer behavior (extends control-surface §6.1)
|
||||
|
||||
The display page adds an **audio layer**:
|
||||
|
||||
- Two `<audio>` elements (A/B) for gapless gain crossfades; a tiny crossfade helper.
|
||||
- On `state`: reconcile `audio.url` — if it changed, crossfade A↔B to the new url
|
||||
(or fade to silence for `off`). White-noise and soundtrack are the same mechanism;
|
||||
only the url differs.
|
||||
- On `ring` (altitude move) with source=soundtrack: start the soundtrack crossfade
|
||||
to the event's new `audio.url` so it lands with the visual scale.
|
||||
- On `mode` (Visual): fade the **video** to/from black; audio is untouched (you can
|
||||
hear soundtrack/white-noise over black).
|
||||
- Each soundtrack loops (the production-pass clean loop); white-noise loops.
|
||||
- Autoplay: the renderer is launched full-screen by the operator; a one-time user
|
||||
gesture (or `--autoplay`/muted-then-unmute) satisfies browser autoplay policy.
|
||||
*Flagged:* document the launch gesture so the projection starts audio reliably.
|
||||
|
||||
## 9. Testing (mandatory pipeline tiers)
|
||||
|
||||
- **Pure unit** — `render.audio.url` resolution from `(source, altitude)`:
|
||||
soundtrack→scale asset, white_noise→global, off→null; altitude tick re-resolves
|
||||
only when source=soundtrack; `seq` bump. No I/O.
|
||||
- **API/contract** — `POST /api/control {visual}` flips `transitions.visual`;
|
||||
`{audio}` rides a `state` event with the new url and **no** transition flag; bad
|
||||
enum → 422; `GET /api/state` shape.
|
||||
- **E2E (Playwright)** — renderer + `/remote`: set Audio=white_noise → assert the
|
||||
renderer's audio element src is the noise asset; tick Altitude with
|
||||
Audio=soundtrack → assert the soundtrack url crossfades to the new scale; set
|
||||
Visual=off → assert the video fades and audio keeps playing. (Audio *playback*
|
||||
asserted via element `src`/`paused`, not waveform.)
|
||||
|
||||
## 10. Scope
|
||||
|
||||
**In v1:**
|
||||
|
||||
- Split **Content** → **Visual** (on/off) + **Audio** (off / soundtrack / white
|
||||
noise) across `SessionState`, `/api/state`, `/api/control`, SSE, and both pages.
|
||||
- Per-scale `audio` in the manifest + the 5 sourced soundtracks (production-passed
|
||||
loops); synthesized white-noise asset.
|
||||
- Server-side `audio.url` resolution; renderer audio layer with crossfades;
|
||||
altitude-coupled soundtrack crossfade.
|
||||
|
||||
**Explicitly NOT in v1 (deferred):**
|
||||
|
||||
- **Music layer** (`audio=music`) — reserved dial position; no assets yet.
|
||||
- **Separate audio altitude** (hear reef while seeing coast) — rejected coupling
|
||||
option; soundtrack follows the one Altitude dial.
|
||||
- **Rotating audio pools** per scale (one bed per altitude in v1).
|
||||
- Per-altitude white-noise variants (one global noise).
|
||||
- Physical Visual/Audio switches (Arduino) — contract only, per control-surface §7.
|
||||
|
||||
## 11. Spec & roadmap impact
|
||||
|
||||
- **Reframes** control-surface §3/§5: `content` → `visual` + `audio`;
|
||||
`transitions.content` → `transitions.visual`; `render.content` → `render.video`
|
||||
+ `render.audio`. That spec is still *proposed* (unbuilt), so this is a clean
|
||||
pre-implementation edit, not a migration. Both can be built together, or this
|
||||
one folds into the control-surface build as its Content step.
|
||||
- The **music layer** becomes a tracked follow-on (compose + source per-altitude
|
||||
music; flip on the reserved Audio position).
|
||||
- `ROADMAP.md` to note the audio capability at implementation time.
|
||||
|
||||
## 12. Open questions / flagged — RESOLVED (session 0020 build)
|
||||
|
||||
- **Audio knob feel** (§7) — **RESOLVED: live**, with a ~0.6 s renderer-side gain
|
||||
crossfade (`XFADE_MS=600`), no `settled` ack. Promote to transitional later only
|
||||
if the iPad makes it feel like it needs a "busy" state.
|
||||
- **White-noise color** — **RESOLVED: pink** (`anoisesrc=color=pink`), a balanced
|
||||
calm bed. Brown is a one-parameter alternative (`color=brown`) if a deeper rumble
|
||||
is wanted; the build is parameterized for it.
|
||||
- **Autoplay gesture** (§8) — **RESOLVED: a one-time tap-to-start overlay**
|
||||
("▶ tap to start sound", `#start-gesture`). The first click unlocks both `<audio>`
|
||||
elements within the user gesture; thereafter audio follows state. This is the
|
||||
documented launch step for the full-screen projection.
|
||||
@@ -66,7 +66,7 @@ needed. Alternatives considered and rejected: renderer polling — laggy/wastefu
|
||||
peer-to-peer WebRTC — most plumbing and it breaks the clean single-contract seam
|
||||
that makes the Arduino a drop-in.)
|
||||
|
||||
## 3. Control surface: the five controls (the faithful twin)
|
||||
## 3. Control surface: the six controls (the faithful twin)
|
||||
|
||||
The iPad remote is a **faithful on-screen twin** of the intended physical panel —
|
||||
what you like on the iPad is the spec for the Arduino panel. The control
|
||||
@@ -74,29 +74,34 @@ inventory is exactly what the sim exposes today:
|
||||
|
||||
| Control | Sim today | Physical form (later) |
|
||||
|---|---|---|
|
||||
| **Content** | 7-way `<select>` (video / audio+video / music+video / off / white noise / music / audio track) | rotary selector switch (7 pos) |
|
||||
| **Visual** | toggle (show video on/off) | 2-position switch |
|
||||
| **Audio** | on/off toggle (on = per-altitude soundtrack; ·white-noise & music deferred) | 2-position switch (rotary selector when more sources land) |
|
||||
| **Altitude** | endless circular dial — walks the 5 scales (cosmos→abyss), wraps | rotary encoder (endless, detented) |
|
||||
| **Left** (analytical) | slider 0–4 | knob/pot, 5 detents |
|
||||
| **Right** (dreamlike) | slider 0–4 | knob/pot, 5 detents |
|
||||
| **Mood** | slider −4…+4 (dark↔light) | center-detented knob |
|
||||
|
||||
(This is the original sub-project-4 inventory, *updated*: Dark+Light collapsed
|
||||
into one **Mood** knob in session 0013, and the **Altitude** encoder added in
|
||||
session 0018.)
|
||||
into one **Mood** knob in session 0013, the **Altitude** encoder added in session
|
||||
0018, and the bundled 7-way **Content** dial split into orthogonal **Visual** ×
|
||||
**Audio** controls in session 0020 — see
|
||||
[`2026-06-26-audio-video-separated-controls-design.md`](./2026-06-26-audio-video-separated-controls-design.md).)
|
||||
|
||||
## 4. Live vs. transitional controls (per-knob transition state)
|
||||
|
||||
The five controls split into two classes by how a **human** perceives the change:
|
||||
|
||||
- **Live controls — Left, Right, Mood.** The alteration engine treats these as
|
||||
`LIVE_UPDATE` (the WebGL Kuwahara dream + HUD overlay + color grade re-tune
|
||||
within a frame). Effectively instantaneous → these are *always idle*; the
|
||||
remote just shows the value.
|
||||
- **Transitional controls — Altitude, Content/mode.** These trigger a renderer
|
||||
- **Live controls — Left, Right, Mood, Audio.** The alteration engine treats
|
||||
Left/Right/Mood as `LIVE_UPDATE` (the WebGL Kuwahara dream + HUD overlay + color
|
||||
grade re-tune within a frame). **Audio** source changes are also live: a short
|
||||
(~0.6 s) renderer-side gain crossfade between two `<audio>` elements —
|
||||
perceptible but non-blocking, with no `settled` ack (audio spec §7). These are
|
||||
*always idle*; the remote just shows the value.
|
||||
- **Transitional controls — Altitude, Visual.** These trigger a renderer
|
||||
animation that takes real, perceptible time: Altitude plays ring-transition
|
||||
clips (zoom/warp between scales); Content/mode does fade-to-black / crossfade.
|
||||
During that window the control is **busy**, and the control source must model
|
||||
that so its feedback is honest.
|
||||
clips (zoom/warp between scales); **Visual** on/off does fade-to-black /
|
||||
fade-from-black. During that window the control is **busy**, and the control
|
||||
source must model that so its feedback is honest.
|
||||
|
||||
Per-transitional-control lifecycle:
|
||||
|
||||
@@ -116,8 +121,9 @@ Hardware payoff: a physical Altitude encoder can light an LED while
|
||||
`transitioning`; the iPad twin shows the matching spinner/disabled-detent.
|
||||
|
||||
`SessionState` therefore carries
|
||||
`transitions: { altitude: idle|transitioning, content: idle|transitioning }`
|
||||
alongside the controls and `seq`. (Live controls need no entry.)
|
||||
`transitions: { altitude: idle|transitioning, visual: idle|transitioning }`
|
||||
alongside the controls and `seq`. (Live controls — Left/Right/Mood/Audio — need
|
||||
no entry.)
|
||||
|
||||
## 5. Server contract
|
||||
|
||||
@@ -130,13 +136,20 @@ control sources never hit them directly.
|
||||
```json
|
||||
{
|
||||
"seq": 42,
|
||||
"controls": { "content": "video", "left": 3, "right": 1, "mood": -2 },
|
||||
"altitude": { "index": 2, "scale": "forest", "clip_id": "forest_07" },
|
||||
"transitions": { "altitude": "idle", "content": "idle" },
|
||||
"render": { "plan": { }, "content": { "audio_source": null, "video": true } }
|
||||
"controls": { "visual": "on", "audio": "soundtrack", "left": 3, "right": 1, "mood": -2 },
|
||||
"altitude": { "index": 2, "scale": "coast", "clip_id": "coast_07" },
|
||||
"transitions": { "altitude": "idle", "visual": "idle" },
|
||||
"render": {
|
||||
"plan": { },
|
||||
"video": { "shown": true },
|
||||
"audio": { "source": "soundtrack", "url": "/media/audio/coast/waves.loop.mp3", "altitude_coupled": true }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(The `render.video` + `render.audio` split and server-side `audio.url` resolution
|
||||
are specified in the audio spec §6.1.)
|
||||
|
||||
A control source GETs this on load to draw its knobs in the right positions; the
|
||||
renderer can use it to cold-start or resync after a reconnect.
|
||||
|
||||
@@ -145,17 +158,19 @@ renderer can use it to cold-start or resync after a reconnect.
|
||||
A partial patch — only the fields that changed:
|
||||
|
||||
```json
|
||||
{ "set": { "left": 4 }, "altitude_delta": 0 } // live knob
|
||||
{ "set": { "content": "off" }, "altitude_delta": 0 } // transitional (mode)
|
||||
{ "set": {}, "altitude_delta": -1 } // one encoder tick down
|
||||
{ "set": { "left": 4 }, "altitude_delta": 0 } // live knob
|
||||
{ "set": { "audio": "white_noise" }, "altitude_delta": 0 } // live (audio source)
|
||||
{ "set": { "visual": "off" }, "altitude_delta": 0 } // transitional (video fade)
|
||||
{ "set": {}, "altitude_delta": -1 } // one encoder tick down
|
||||
```
|
||||
|
||||
Server applies it, bumps `seq`, recomputes, and:
|
||||
|
||||
- **live field** → broadcasts a `state` event immediately;
|
||||
- **transitional field** (a `content` change, or a non-zero `altitude_delta`) →
|
||||
computes the ring move / fade, marks that field `transitioning`, broadcasts a
|
||||
`ring` (or `mode`) event for the renderer to animate.
|
||||
- **live field** (Left/Right/Mood, **or an `audio` source change**) → broadcasts
|
||||
a `state` event immediately (the renderer crossfades audio on its own, §7);
|
||||
- **transitional field** (a `visual` change, or a non-zero `altitude_delta`) →
|
||||
computes the ring move / video fade, marks that field `transitioning`,
|
||||
broadcasts a `ring` (or `mode`) event for the renderer to animate.
|
||||
|
||||
Returns the new snapshot (an immediate authoritative echo for the poster).
|
||||
Absolute `set` values make retries idempotent; `altitude_delta` is relative
|
||||
@@ -166,10 +181,12 @@ Absolute `set` values make retries idempotent; `altitude_delta` is relative
|
||||
|
||||
Event types:
|
||||
|
||||
- `state` — full snapshot; sent on connect and on every live change.
|
||||
- `state` — full snapshot; sent on connect and on every live change (including an
|
||||
**audio** source change, which the renderer crossfades — §7).
|
||||
- `ring` — an altitude move: the transition clip(s) to play + the landed
|
||||
scale/clip. Renderer plays them, then settles.
|
||||
- `mode` — a content/mode change: the fade the renderer should run.
|
||||
scale/clip, **plus the new `audio.url`** so the renderer can crossfade the
|
||||
soundtrack in step (when `audio=soundtrack`). Renderer plays them, then settles.
|
||||
- `mode` — a **Visual** on/off change: the video fade the renderer should run.
|
||||
- `ping` — keepalive.
|
||||
|
||||
The SSE event payloads reuse the existing serializers
|
||||
@@ -207,14 +224,17 @@ server: transitions.altitude="idle" → SSE "state" to all → iPad clears spinn
|
||||
|
||||
### 6.1 Renderer — `/` (display), refactored from today's `index.html`
|
||||
|
||||
- **Removes** the input controls (the `<select>`, the Altitude SVG, the three
|
||||
sliders) and their handlers.
|
||||
- **Keeps** everything that draws the experience: the video element, the WebGL
|
||||
Kuwahara dream shader, the HUD/annotation + affect overlay, the
|
||||
ring-transition playback, the media preload pool, the crossfade-loop logic.
|
||||
- **Removes** the input controls (the Visual toggle, the Audio `<select>`, the
|
||||
Altitude SVG, the three sliders) and their handlers.
|
||||
- **Keeps** everything that draws the experience: the video element, the **A/B
|
||||
`<audio>` crossfade layer**, the WebGL Kuwahara dream shader, the
|
||||
HUD/annotation + affect overlay, the ring-transition playback, the media
|
||||
preload pool, the crossfade-loop logic.
|
||||
- **Adds** a thin `state` client: open `EventSource('/api/events')`; on `state`
|
||||
apply controls→render live; on `ring` play the transition clip(s) then settle
|
||||
and `POST /api/render/event {settled}`; on `mode` run the fade. On (re)connect,
|
||||
apply controls→render live (incl. the audio crossfade); on `ring` play the
|
||||
transition clip(s) — crossfading the soundtrack to the event's `audio.url` —
|
||||
then settle and `POST /api/render/event {settled}`; on `mode` run the video
|
||||
fade. On (re)connect,
|
||||
pull `GET /api/state` to resync. SSE auto-reconnects, so a flaky link
|
||||
self-heals.
|
||||
- Runs full-screen on the laptop driving the projector; no keyboard/mouse needed
|
||||
@@ -222,16 +242,16 @@ server: transitions.altitude="idle" → SSE "state" to all → iPad clears spinn
|
||||
|
||||
### 6.2 Controller — `/remote`, a new touch-first page (the faithful twin)
|
||||
|
||||
- The five controls laid out like the intended physical panel: **Content**
|
||||
selector, **Altitude** encoder dial (reuse the existing SVG dial component),
|
||||
**Left** / **Right** / **Mood** knobs.
|
||||
- The six controls laid out like the intended physical panel: **Visual** toggle +
|
||||
**Audio** selector, **Altitude** encoder dial (reuse the existing SVG dial
|
||||
component), **Left** / **Right** / **Mood** knobs.
|
||||
- On any input → `POST /api/control` with just the changed field. Live knobs post
|
||||
on drag; Altitude posts encoder ticks; Content posts on selection.
|
||||
on drag; Altitude posts encoder ticks; Visual/Audio post on change.
|
||||
- Per-knob transition state (§4): a transitional control goes **pending** on
|
||||
touch, reflects **transitioning** from the SSE feed, clears on **settled**.
|
||||
Altitude shows a settling indicator and its detents soft-lock (further ticks
|
||||
queue as the next target rather than stacking); Content shows the selection as
|
||||
pending until the fade completes.
|
||||
queue as the next target rather than stacking); **Visual** shows the toggle as
|
||||
pending until the video fade completes; **Audio** is live (no pending state).
|
||||
- It **also** subscribes to `/api/events` so it stays in sync with other
|
||||
sources (turn the future Arduino's knob and the iPad reflects it, and
|
||||
vice-versa). On load it GETs `/api/state` to draw current positions.
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Resolve the orthogonal Visual × Audio controls into render outputs (audio spec
|
||||
§4/§6). The single source of truth for: whether the projector shows video, and
|
||||
which audio url plays for a given (audio source, current altitude).
|
||||
|
||||
Replaces player/content.py's 7-way bundled table. Soundtrack follows the single
|
||||
Altitude dial; off is silence. `white_noise` and `music` are deferred sources —
|
||||
not in v1 (the live control is a simple Audio on/off toggle where on=soundtrack)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
VISUAL_POSITIONS = frozenset({"on", "off"})
|
||||
# v1 audio sources. The live UI is an Audio on/off toggle (on=soundtrack).
|
||||
# "white_noise" and "music" are deferred (no live control) and intentionally absent.
|
||||
AUDIO_SOURCES = frozenset({"off", "soundtrack"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AudioResolution:
|
||||
source: str
|
||||
url: str | None
|
||||
altitude_coupled: bool
|
||||
|
||||
|
||||
def resolve_visual(position: str) -> bool:
|
||||
"""Visual on/off → whether the renderer shows video (vs fade-to-black)."""
|
||||
if position not in VISUAL_POSITIONS:
|
||||
raise ValueError(
|
||||
f"unknown visual position {position!r}; expected one of {sorted(VISUAL_POSITIONS)}"
|
||||
)
|
||||
return position == "on"
|
||||
|
||||
|
||||
def resolve_audio_url(source: str, *, scale_audio: str) -> str | None:
|
||||
"""The audio url for (source, current altitude): soundtrack → the current
|
||||
scale's asset (or None if none authored); off → None. `scale_audio` is the ring
|
||||
scale's `audio` path (relative to /media/audio/)."""
|
||||
if source not in AUDIO_SOURCES:
|
||||
raise ValueError(
|
||||
f"unknown audio source {source!r}; expected one of {sorted(AUDIO_SOURCES)}"
|
||||
)
|
||||
if source == "off":
|
||||
return None
|
||||
# soundtrack — couple to the current altitude's asset
|
||||
return f"/media/audio/{scale_audio}" if scale_audio else None
|
||||
|
||||
|
||||
def resolve_audio(source: str, *, scale_audio: str) -> AudioResolution:
|
||||
"""The full render.audio view: source, resolved url, and whether it follows the
|
||||
Altitude dial (true only for soundtrack)."""
|
||||
url = resolve_audio_url(source, scale_audio=scale_audio)
|
||||
return AudioResolution(source=source, url=url, altitude_coupled=(source == "soundtrack"))
|
||||
@@ -1,42 +0,0 @@
|
||||
"""Resolve the 7-way content dial into an audio source + video on/off (§6).
|
||||
|
||||
The single source of truth for design §6's table: which audio source plays and
|
||||
whether the projector shows video, for each dial position.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
# The four distinct audio sources. "white_noise" is generated at runtime;
|
||||
# "music" is the public-domain classical pool; "audio_track" is the clip's own
|
||||
# field audio; "none" is silence.
|
||||
AUDIO_SOURCES = frozenset({"none", "white_noise", "music", "audio_track"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ContentResolution:
|
||||
audio_source: str
|
||||
video: bool
|
||||
|
||||
|
||||
# Design §6, one row per dial position.
|
||||
_TABLE = {
|
||||
"off": ContentResolution("none", False),
|
||||
"white_noise": ContentResolution("white_noise", False),
|
||||
"music": ContentResolution("music", False),
|
||||
"audio_track": ContentResolution("audio_track", False),
|
||||
"video": ContentResolution("none", True),
|
||||
"music_video": ContentResolution("music", True),
|
||||
"audio_video": ContentResolution("audio_track", True),
|
||||
}
|
||||
|
||||
|
||||
def resolve_content(position: str) -> ContentResolution:
|
||||
"""Map a content-dial position to its audio source and video flag (§6)."""
|
||||
try:
|
||||
return _TABLE[position]
|
||||
except KeyError:
|
||||
raise ValueError(
|
||||
f"unknown content position {position!r}; expected one of {sorted(_TABLE)}"
|
||||
) from None
|
||||
+15
-12
@@ -1,20 +1,17 @@
|
||||
"""Control-panel state: the data shape read from the Arduino over serial.
|
||||
|
||||
This is the serial-contract payload shared with sub-project 4 (firmware). It
|
||||
models the full panel from design §6/§7: the 7-way content dial, the four
|
||||
experience knobs (0..4), and the two intensity levels (volume, brightness).
|
||||
The wire framing itself is the separate 3<->4 serial contract; this module is
|
||||
the *decoded* form.
|
||||
models the full panel from the audio spec §3: a Visual toggle (on/off) + an
|
||||
Audio source selector (off/soundtrack; white-noise/music deferred), the four experience knobs
|
||||
(0..4), and the two intensity levels (volume, brightness). The wire framing
|
||||
itself is the separate 3<->4 serial contract; this module is the *decoded* form.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, fields
|
||||
|
||||
# The seven positions of the content dial (design §6).
|
||||
CONTENT_POSITIONS = frozenset(
|
||||
{"off", "white_noise", "music", "audio_track", "video", "music_video", "audio_video"}
|
||||
)
|
||||
from player.audio import AUDIO_SOURCES, VISUAL_POSITIONS
|
||||
|
||||
KNOB_FIELDS = ("left", "right", "dark", "light", "volume", "brightness")
|
||||
KNOB_MIN = 0
|
||||
@@ -27,7 +24,8 @@ class ControlsError(ValueError):
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Controls:
|
||||
content: str
|
||||
visual: str
|
||||
audio: str
|
||||
left: int
|
||||
right: int
|
||||
dark: int
|
||||
@@ -38,10 +36,15 @@ class Controls:
|
||||
|
||||
def validate_controls(c: Controls) -> None:
|
||||
"""Raise ControlsError if the payload is structurally invalid."""
|
||||
if c.content not in CONTENT_POSITIONS:
|
||||
if c.visual not in VISUAL_POSITIONS:
|
||||
raise ControlsError(
|
||||
f"invalid content position {c.content!r}; "
|
||||
f"expected one of {sorted(CONTENT_POSITIONS)}"
|
||||
f"invalid visual position {c.visual!r}; "
|
||||
f"expected one of {sorted(VISUAL_POSITIONS)}"
|
||||
)
|
||||
if c.audio not in AUDIO_SOURCES:
|
||||
raise ControlsError(
|
||||
f"invalid audio source {c.audio!r}; "
|
||||
f"expected one of {sorted(AUDIO_SOURCES)}"
|
||||
)
|
||||
for name in KNOB_FIELDS:
|
||||
value = getattr(c, name)
|
||||
|
||||
@@ -63,6 +63,7 @@ class Scale:
|
||||
id: str
|
||||
clip_id: str
|
||||
pool: tuple[str, ...] = ()
|
||||
audio: str = "" # per-scale soundtrack, relative to /media/audio/ (audio spec §5.1)
|
||||
|
||||
@property
|
||||
def members(self) -> tuple[str, ...]:
|
||||
|
||||
+14
-8
@@ -20,7 +20,7 @@ from typing import Callable, Optional
|
||||
|
||||
from hef.selection import Coordinate
|
||||
from player.alteration import RenderPlan, plan_alteration
|
||||
from player.content import ContentResolution, resolve_content
|
||||
from player.audio import resolve_visual
|
||||
from player.controls import Controls
|
||||
|
||||
|
||||
@@ -34,11 +34,14 @@ class TransitionKind:
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Playback:
|
||||
"""What should currently be playing."""
|
||||
"""What should currently be playing. Visual and audio are orthogonal (audio
|
||||
spec §2): `video` is whether the projector shows footage, `audio_source` is
|
||||
the independent audio dial (off / soundtrack)."""
|
||||
|
||||
clip_id: Optional[str] # None = black walls
|
||||
plan: Optional[RenderPlan] # None when black
|
||||
content: ContentResolution
|
||||
video: bool # whether footage is shown (Visual on/off)
|
||||
audio_source: str # the Audio dial source (off/soundtrack)
|
||||
volume: int
|
||||
brightness: int
|
||||
|
||||
@@ -58,7 +61,8 @@ def _first(library):
|
||||
_BLACK = Playback(
|
||||
clip_id=None,
|
||||
plan=None,
|
||||
content=ContentResolution("none", False),
|
||||
video=False,
|
||||
audio_source="off",
|
||||
volume=0,
|
||||
brightness=0,
|
||||
)
|
||||
@@ -78,12 +82,13 @@ class Player:
|
||||
self._current = _BLACK
|
||||
|
||||
def _resolve(self, controls: Controls) -> Playback:
|
||||
content = resolve_content(controls.content)
|
||||
if not content.video:
|
||||
video = resolve_visual(controls.visual)
|
||||
if not video:
|
||||
return Playback(
|
||||
clip_id=None,
|
||||
plan=None,
|
||||
content=content,
|
||||
video=False,
|
||||
audio_source=controls.audio,
|
||||
volume=controls.volume,
|
||||
brightness=controls.brightness,
|
||||
)
|
||||
@@ -92,7 +97,8 @@ class Player:
|
||||
return Playback(
|
||||
clip_id=clip.id,
|
||||
plan=plan_alteration(coord),
|
||||
content=content,
|
||||
video=True,
|
||||
audio_source=controls.audio,
|
||||
volume=controls.volume,
|
||||
brightness=controls.brightness,
|
||||
)
|
||||
|
||||
@@ -24,3 +24,9 @@ sim = [
|
||||
"uvicorn[standard]>=0.29",
|
||||
"httpx>=0.27",
|
||||
]
|
||||
# Playwright E2E browser tests (audio spec §9). Install with the browser binary:
|
||||
# pip install -e '.[e2e]' && python -m playwright install chromium
|
||||
# The E2E suite skips cleanly when Playwright/Chromium is absent (headless box).
|
||||
e2e = [
|
||||
"pytest-playwright>=0.4",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
# Session 0020.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-26T06-47 (PST)
|
||||
> End: 2026-06-26T07-58 (PST)
|
||||
> Type: writing-plans (extended into execution — operator approved "then build it" at the plan gate)
|
||||
> Posture: careful (execution proceeded under explicit operator approval)
|
||||
> Claude-Session: e0cc053a-41fa-416d-a4f5-493f14984e75
|
||||
> Anchor: design docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md (R2a)
|
||||
> Status: FINALIZED
|
||||
|
||||
## Launch prompt
|
||||
|
||||
> Write the implementation plan for the "Audio + Video — separated Visual & Audio
|
||||
> controls" feature, then build it. (Spec:
|
||||
> docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md.)
|
||||
> Split the bundled 7-way Content selector into orthogonal Visual (on/off) ×
|
||||
> Audio (off / soundtrack / white noise). Audio follows the single Altitude dial;
|
||||
> white noise synthesized (ffmpeg anoisesrc). Reframe the (unbuilt) networked-
|
||||
> control-surface spec §3/§5 as a clean pre-implementation edit. Honor the testing
|
||||
> tiers; resolve the three §12 flagged decisions.
|
||||
|
||||
## Pre-state
|
||||
|
||||
- `main` clean at 99a751c. Two stale `--INPROGRESS` placeholders (0015, 0017) from
|
||||
2026-06-24/25 — prior dead sessions, not live; left untouched.
|
||||
- The networked-control-surface architecture (SSE / `/remote` / renderer split) was
|
||||
**unbuilt** — the sim is a single fused page (`simulator/static/index.html`) talking
|
||||
to FastAPI via `POST /api/alteration` (knobs) + `POST /api/ring/advance` (altitude).
|
||||
- Audio assets (5 sourced ambiences) sat gitignored under `simulator/sample_media/audio/<scale>/`,
|
||||
unreferenced by the manifest, unserved; no `<audio>` element anywhere; video muted.
|
||||
|
||||
## Session arc (turn by turn)
|
||||
|
||||
1. **Session gate → writing-plans.** Classified the two-part "write the plan, then
|
||||
build it" prompt as the build track's entry; claimed session **0020** (`--type
|
||||
writing-plans`, careful posture). Baseline clean; created branch `session-0020`.
|
||||
2. **Scoping.** Read both specs + the audio candidate pool; dispatched an Explore
|
||||
agent to map the current simulator. Key finding: the control-surface architecture
|
||||
is unbuilt, so the audio spec's §6 server contract had no host. **Resolved by the
|
||||
standing simulator-first directive:** build the audio feature into the current
|
||||
single-page sim (ships sound now); reframe the control-surface spec on paper only;
|
||||
do NOT build the SSE/`/remote`/renderer split (it's "v1 hardware," deferred). This
|
||||
build is the "Content step" the audio spec §11 anticipates.
|
||||
3. **Plan.** Wrote `docs/superpowers/plans/2026-06-26-audio-video-separated-controls.md`
|
||||
(10 tasks, all testing tiers). Committed + pushed; presented at the careful-posture
|
||||
plan gate. Operator answered **"Approve — build it."** §12 decisions resolved:
|
||||
audio=live (~0.6 s crossfade), noise=pink, autoplay=tap-to-start overlay; asset
|
||||
params −18 LUFS / mp3 / loop mirrors the proven video crossfade recipe.
|
||||
4. **Build (TDD, commit per task):**
|
||||
- T1–2: pure ffmpeg audio builders (`tools/pipeline/audio_ops.py`) + runner
|
||||
(`audio_run.py`) + `simulator/build_audio_media.py`; produced the 5 normalized
|
||||
loops + pink-noise bed (gitignored).
|
||||
- T3: `Scale.audio` manifest field + read/emit/serialize + `SCALE_AUDIO`; manifest
|
||||
regenerated (clean diff — only 5 audio lines).
|
||||
- T4–6: pure `player/audio.py` resolver; retired `player/content.py` (+ test);
|
||||
reframed `player/controls.py` + `player/state.py`; `/api/alteration` validates
|
||||
visual+audio and returns `render.video` + server-resolved `render.audio.url` from
|
||||
`altitude_index`. Full Python suite green.
|
||||
- T7–8: client — Content `<select>` → Visual toggle + Audio selector; A/B `<audio>`
|
||||
gain-crossfade layer; tap-to-start gesture. Smoke-tested page + media (200s).
|
||||
- T9: reframed the networked-control-surface spec §3/§4/§5/§6; recorded §12
|
||||
resolutions; ROADMAP note.
|
||||
- T10: Playwright E2E (`[e2e]` extra). Installed Chromium and ran it for real —
|
||||
surfaced + fixed three client issues (see Cut state). **3 E2E pass**, full suite
|
||||
**288 passed, 2 skipped**.
|
||||
5. **Verify + ship.** Screenshotted the running app (cosmos plays, soundtrack audibly
|
||||
playing `cosmos/pillars.loop.mp3`, split controls visible, zero JS errors). Merged
|
||||
`session-0020` → `main` (`--no-ff`) and pushed. (`gh` PR create denied — Gitea host;
|
||||
merged locally with branch→merge discipline.)
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
_No low-confidence autonomous calls. The scope decision (current-sim vs full control
|
||||
surface) was approved at the plan gate; the three §12 resolutions were operator-
|
||||
delegated by the launch prompt ("resolve the three flagged decisions"); asset params
|
||||
(−18 LUFS, mp3, pink) follow the spec's stated direction._
|
||||
|
||||
## Cut state
|
||||
|
||||
- `main` @ `aa76cd1` (merge of session-0020), clean, synced with origin.
|
||||
- **288 tests + 3 Playwright E2E pass; 2 skipped** (opt-in media integration).
|
||||
- Three client robustness fixes the E2E drove out (worth remembering):
|
||||
- `fadeVolume` uses `setInterval`, not `requestAnimationFrame` (rAF stalls when the
|
||||
tab is backgrounded → the gain ramp hung in headless).
|
||||
- The start gesture drops per-element "priming": a single page click grants Chrome
|
||||
autoplay document-wide; priming srcless `<audio>` only hung the handler / polluted
|
||||
the paused-state.
|
||||
- `applyAudio` does not `await` `play()` (awaiting a srcless/buffering play can hang).
|
||||
- Plan archival: this app has **no content repo** (`CONTENT_REMOTE` empty), so the plan
|
||||
was not archived to a `plans/` collection — it lives in-repo at
|
||||
`docs/superpowers/plans/` and is already on `main`.
|
||||
- §9 pipeline: localhost + E2E stage is **green**; there is no PPE/prod infra for this
|
||||
simulator-first art app (deferred with the Pi renderer), so no further ship stage ran.
|
||||
|
||||
## Operator plate (what's true now / next)
|
||||
|
||||
- The simulator has **sound**: per-altitude soundtracks that follow the Altitude dial +
|
||||
a pink white-noise bed, behind orthogonal **Visual** × **Audio** controls. The unheard
|
||||
audio wants a by-ear review/tune.
|
||||
- Deferred (already roadmap-tracked, not re-parked in memory): the **music** layer
|
||||
(reserved Audio position, no assets), the networked-control-surface SSE/`/remote`
|
||||
build, i2v ring transitions, the Pi renderer.
|
||||
|
||||
## Next /goal
|
||||
|
||||
```
|
||||
/goal review/tune the new audio by ear in the sim (per-altitude soundtracks + pink white-noise + the ~0.6s A/B crossfade; tap-to-start), then source/compose the deferred per-altitude music layer (the reserved Audio position), per docs/audio-candidate-pool.md
|
||||
```
|
||||
@@ -55,5 +55,8 @@
|
||||
},
|
||||
"0019": {
|
||||
"title": ""
|
||||
},
|
||||
"0020": {
|
||||
"title": ""
|
||||
}
|
||||
}
|
||||
|
||||
+22
-7
@@ -26,8 +26,7 @@ from player.alteration import (
|
||||
plan_alteration,
|
||||
render_plan_to_dict,
|
||||
)
|
||||
from player.content import resolve_content
|
||||
from player.controls import CONTENT_POSITIONS
|
||||
from player.audio import AUDIO_SOURCES, VISUAL_POSITIONS, resolve_audio, resolve_visual
|
||||
from player.ring import (
|
||||
DEFAULT_FAST_SPIN_THRESHOLD,
|
||||
advance_ring,
|
||||
@@ -95,7 +94,8 @@ def _rev_file(file: str) -> str:
|
||||
|
||||
|
||||
class ControlsModel(BaseModel):
|
||||
content: str
|
||||
visual: str
|
||||
audio: str
|
||||
left: int = Field(ge=0, le=4)
|
||||
right: int = Field(ge=0, le=4)
|
||||
dark: int = Field(ge=0, le=4)
|
||||
@@ -112,6 +112,7 @@ class CalibrationModel(BaseModel):
|
||||
|
||||
class AlterationRequest(BaseModel):
|
||||
controls: ControlsModel
|
||||
altitude_index: int = 0
|
||||
calibration: Optional[CalibrationModel] = None
|
||||
|
||||
|
||||
@@ -166,8 +167,10 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
|
||||
@app.post("/api/alteration")
|
||||
def api_alteration(req: AlterationRequest):
|
||||
c = req.controls
|
||||
if c.content not in CONTENT_POSITIONS:
|
||||
raise HTTPException(status_code=422, detail=f"invalid content {c.content!r}")
|
||||
if c.visual not in VISUAL_POSITIONS:
|
||||
raise HTTPException(status_code=422, detail=f"invalid visual {c.visual!r}")
|
||||
if c.audio not in AUDIO_SOURCES:
|
||||
raise HTTPException(status_code=422, detail=f"invalid audio {c.audio!r}")
|
||||
coord = Coordinate(c.left, c.right, c.dark, c.light)
|
||||
cal = (
|
||||
Calibration(
|
||||
@@ -179,10 +182,22 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
|
||||
else DEFAULT_CALIBRATION
|
||||
)
|
||||
plan = plan_alteration(coord, cal)
|
||||
content = resolve_content(c.content)
|
||||
# Resolve the soundtrack url against the CURRENT altitude (server-side, audio
|
||||
# spec §6.1). White-noise/off ignore the scale; soundtrack follows it.
|
||||
scale_audio = ""
|
||||
if app.state.ring is not None:
|
||||
scale_audio = scale_at(app.state.ring, req.altitude_index).audio
|
||||
audio = resolve_audio(c.audio, scale_audio=scale_audio)
|
||||
return {
|
||||
"plan": render_plan_to_dict(plan),
|
||||
"content": {"audio_source": content.audio_source, "video": content.video},
|
||||
"render": {
|
||||
"video": {"shown": resolve_visual(c.visual)},
|
||||
"audio": {
|
||||
"source": audio.source,
|
||||
"url": audio.url,
|
||||
"altitude_coupled": audio.altitude_coupled,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
@app.get("/dev/version")
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Production pass for the 5 per-altitude soundtracks (audio spec §5.3). The audio
|
||||
analogue of build_pool_manifest.py --media: read the sourced ambience clips
|
||||
(re-downloadable per docs/audio-candidate-pool.md), and make each a seamless
|
||||
loudness-normalized loop.
|
||||
|
||||
Media is gitignored; this script is the reproducible record. Outputs land under
|
||||
simulator/sample_media/audio/<scale>/<name>.loop.mp3, served at /media/audio/...
|
||||
Run: python simulator/build_audio_media.py
|
||||
|
||||
(White-noise is deferred — the live Audio control is a simple on/off soundtrack
|
||||
toggle. `tools.pipeline.audio_run.generate_white_noise` remains for when it's
|
||||
wanted, but no noise bed is built here.)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from tools.pipeline.audio_run import process_soundtrack
|
||||
|
||||
AUDIO = Path(__file__).parent / "sample_media" / "audio"
|
||||
|
||||
# scale -> (sourced raw file, production output file), both under audio/<scale>/.
|
||||
# Sources per docs/audio-candidate-pool.md (the "✓ KEEP" picks).
|
||||
SOURCES: dict[str, tuple[str, str]] = {
|
||||
"cosmos": ("pillars.mp3", "pillars.loop.mp3"),
|
||||
"orbit": ("spaceamb.mp3", "spaceamb.loop.mp3"),
|
||||
"coast": ("waves.mp3", "waves.loop.mp3"),
|
||||
"reef": ("soundscape.mp3", "soundscape.loop.mp3"),
|
||||
"abyss": ("whale.wav", "whale.loop.mp3"),
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
for scale, (raw, out) in SOURCES.items():
|
||||
src = AUDIO / scale / raw
|
||||
if not src.exists():
|
||||
print(f" SKIP {scale}: source missing ({src}) — re-source per audio-candidate-pool.md")
|
||||
continue
|
||||
dst = process_soundtrack(src, AUDIO / scale / out)
|
||||
print(f" produced {dst.relative_to(AUDIO.parent)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -48,6 +48,16 @@ POOLS: dict[str, list[str]] = {
|
||||
# Ring order (large -> small, wraps): cosmos -> orbit -> coast -> reef -> abyss -> cosmos.
|
||||
RING_ORDER = ["cosmos", "orbit", "coast", "reef", "abyss"]
|
||||
|
||||
# Per-scale soundtrack (audio spec §5.1) — the production-pass output of
|
||||
# simulator/build_audio_media.py, served at /media/audio/<path>.
|
||||
SCALE_AUDIO: dict[str, str] = {
|
||||
"cosmos": "cosmos/pillars.loop.mp3",
|
||||
"orbit": "orbit/spaceamb.loop.mp3",
|
||||
"coast": "coast/waves.loop.mp3",
|
||||
"reef": "reef/soundscape.loop.mp3",
|
||||
"abyss": "abyss/whale.loop.mp3",
|
||||
}
|
||||
|
||||
PD = "public-domain (US Gov, 17 U.S.C. §105)"
|
||||
PD_NPS = "public-domain (NPS, no © — 17 U.S.C. §105)"
|
||||
CCBY_STSCI = "CC-BY-class — credit NASA, ESA, STScI"
|
||||
@@ -357,7 +367,10 @@ def build_manifest() -> dict:
|
||||
for scale in RING_ORDER:
|
||||
for clip_id in POOLS[scale]:
|
||||
clips.append(_clip_entry(scale, clip_id))
|
||||
scales = [{"id": s, "clip_id": POOLS[s][0], "pool": POOLS[s]} for s in RING_ORDER]
|
||||
scales = [
|
||||
{"id": s, "clip_id": POOLS[s][0], "pool": POOLS[s], "audio": SCALE_AUDIO[s]}
|
||||
for s in RING_ORDER
|
||||
]
|
||||
transitions = []
|
||||
n = len(RING_ORDER)
|
||||
for i in range(n):
|
||||
|
||||
+2
-1
@@ -82,7 +82,7 @@ def _scale_from_dict(s: dict[str, Any]) -> Scale:
|
||||
"""
|
||||
pool = tuple(s.get("pool", []))
|
||||
clip_id = s.get("clip_id") or (pool[0] if pool else "")
|
||||
return Scale(id=s["id"], clip_id=clip_id, pool=pool)
|
||||
return Scale(id=s["id"], clip_id=clip_id, pool=pool, audio=s.get("audio", ""))
|
||||
|
||||
|
||||
def load_ring(path: str | Path) -> ScaleRing | None:
|
||||
@@ -119,6 +119,7 @@ def ring_to_dict(ring: ScaleRing, clips: list[Clip]) -> dict:
|
||||
"id": s.id,
|
||||
"clip_id": s.clip_id,
|
||||
"title": titles.get(s.clip_id, s.id),
|
||||
"audio": s.audio,
|
||||
"pool": [
|
||||
{"clip_id": cid, "title": titles.get(cid, cid)}
|
||||
for cid in s.members
|
||||
|
||||
@@ -2371,7 +2371,8 @@
|
||||
"cosmos_galaxies",
|
||||
"cosmos_hudf",
|
||||
"cosmos_xdf"
|
||||
]
|
||||
],
|
||||
"audio": "cosmos/pillars.loop.mp3"
|
||||
},
|
||||
{
|
||||
"id": "orbit",
|
||||
@@ -2380,7 +2381,8 @@
|
||||
"orbit_planetearth",
|
||||
"orbit_crewobs",
|
||||
"orbit_bluemarble"
|
||||
]
|
||||
],
|
||||
"audio": "orbit/spaceamb.loop.mp3"
|
||||
},
|
||||
{
|
||||
"id": "coast",
|
||||
@@ -2390,7 +2392,8 @@
|
||||
"coast_surfgrass",
|
||||
"coast_elkbeach",
|
||||
"coast_drakesbeach"
|
||||
]
|
||||
],
|
||||
"audio": "coast/waves.loop.mp3"
|
||||
},
|
||||
{
|
||||
"id": "reef",
|
||||
@@ -2401,7 +2404,8 @@
|
||||
"reef_hawkfish",
|
||||
"reef_snapper",
|
||||
"reef_coralspacific"
|
||||
]
|
||||
],
|
||||
"audio": "reef/soundscape.loop.mp3"
|
||||
},
|
||||
{
|
||||
"id": "abyss",
|
||||
@@ -2410,7 +2414,8 @@
|
||||
"abyss_wow",
|
||||
"abyss_midwaterexp",
|
||||
"abyss_hiding"
|
||||
]
|
||||
],
|
||||
"audio": "abyss/whale.loop.mp3"
|
||||
}
|
||||
],
|
||||
"transitions": [
|
||||
|
||||
+111
-13
@@ -96,7 +96,10 @@ let mediaVersions = {}; // file -> content-hash token (from /api/media-version
|
||||
// changes with its bytes (permanent cache-bust). The blob cache, when present,
|
||||
// wins — those bytes are already the current ones for this session.
|
||||
function mediaNetUrl(file) { const v = mediaVersions[file]; return "/media/" + file + (v ? "?v=" + v : ""); }
|
||||
function mediaUrl(file) { return mediaBlobs[file] || mediaNetUrl(file); }
|
||||
function mediaUrl(file) {
|
||||
if (file.startsWith("/media/")) return file; // already a resolved absolute url (audio layer)
|
||||
return mediaBlobs[file] || mediaNetUrl(file);
|
||||
}
|
||||
|
||||
// Every media file the ring can show: each scale-pool clip's base footage plus the
|
||||
// per-edge transition clips. This is the full preload set (~two dozen files).
|
||||
@@ -138,7 +141,10 @@ async function preloadAllMedia(concurrency = 4) {
|
||||
let done = 0;
|
||||
const total = files.length;
|
||||
const chip = preloadChip();
|
||||
const tick = () => { if (chip) chip.textContent = `⬇ caching clips ${done}/${total}`; };
|
||||
const tick = () => {
|
||||
if (chip) chip.textContent = `⬇ caching clips ${done}/${total}`;
|
||||
setLoadingProgress(done, total); // drive the "Loading Universe…" bar
|
||||
};
|
||||
tick();
|
||||
let i = 0;
|
||||
async function worker() {
|
||||
@@ -492,12 +498,14 @@ function renderScaleReadout() {
|
||||
}
|
||||
|
||||
function controls() {
|
||||
// One bipolar Mood dial (-4 dark .. 0 neutral .. +4 light) maps onto the engine's
|
||||
// two poles. The calibration gains are pinned to their locked 1.0 (full dial = full
|
||||
// effect), so the simulator sends no calibration — the server uses DEFAULT_CALIBRATION.
|
||||
// Video (on/off) and Audio (on/off) are orthogonal toggles (audio spec §2).
|
||||
// Audio on = the per-altitude soundtrack (white-noise is deferred). One bipolar
|
||||
// Mood dial (-4 dark .. +4 light) maps onto the engine's two poles. Calibration
|
||||
// gains are pinned to 1.0, so the simulator sends no calibration.
|
||||
const mood = +$("mood").value;
|
||||
return {
|
||||
content: $("content").value,
|
||||
visual: $("visual").checked ? "on" : "off",
|
||||
audio: $("audio").checked ? "soundtrack" : "off",
|
||||
left: +$("left").value, right: +$("right").value,
|
||||
dark: mood < 0 ? -mood : 0, light: mood > 0 ? mood : 0,
|
||||
volume: 2, brightness: 2,
|
||||
@@ -509,13 +517,22 @@ async function update() {
|
||||
if (busy) return;
|
||||
const resp = await fetch("/api/alteration", {
|
||||
method: "POST", headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ controls: controls() }),
|
||||
body: JSON.stringify({ controls: controls(), altitude_index: ringIndex }),
|
||||
});
|
||||
if (!resp.ok) { readout.textContent = "invalid: " + resp.status; return; }
|
||||
const data = await resp.json();
|
||||
readout.textContent = JSON.stringify(data, null, 2);
|
||||
if (!data.content.video) { black.style.opacity = "1"; black.classList.remove("hidden"); return; }
|
||||
if (!data.render.video.shown) {
|
||||
// Blank the screen. Cover with #black AND hide the video layers themselves —
|
||||
// the <video>/<canvas> are GPU-composited and can otherwise show through the
|
||||
// overlay on a real GPU (Safari/Chrome).
|
||||
black.style.opacity = "1"; black.classList.remove("hidden");
|
||||
vid.style.opacity = "0"; paint.style.opacity = "0";
|
||||
return;
|
||||
}
|
||||
black.classList.add("hidden");
|
||||
vid.style.opacity = ""; paint.style.opacity = ""; // restore the video layers
|
||||
|
||||
try {
|
||||
ensureClipMedia();
|
||||
applyVideoLook(data.plan.grade.tone, data.plan.dream.intensity);
|
||||
@@ -622,6 +639,7 @@ async function advance(delta) {
|
||||
} finally {
|
||||
busy = false;
|
||||
update();
|
||||
applyAudio(); // soundtrack follows the Altitude dial (crossfade to the new scale)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -894,24 +912,104 @@ function devLiveReload() {
|
||||
}, 1000);
|
||||
}
|
||||
|
||||
// --- Audio layer: ONE <audio> element, started in-gesture (Safari-safe) ---
|
||||
// Real Safari/iOS only "unlocks" an <audio> element when play() is called
|
||||
// SYNCHRONOUSLY inside a user gesture (Chrome is lenient; Safari is not). So we use
|
||||
// a single element and start it directly from the Video/Audio toggle's click — not
|
||||
// via the async server roundtrip; once unlocked it can be re-played programmatically
|
||||
// (e.g. on an altitude change). audio on = the current scale's soundtrack.
|
||||
const aud = $("aud");
|
||||
let audUrl = null; // soundtrack url currently loaded (null = silent)
|
||||
const FADE_MS = 500;
|
||||
|
||||
function fadeVolume(el, to, ms, done) {
|
||||
// setInterval (not requestAnimationFrame): rAF throttles/pauses when the tab is
|
||||
// backgrounded, which would stall the gain ramp; a timer fires regardless.
|
||||
const from = el.volume, steps = Math.max(1, Math.round(ms / 30));
|
||||
let i = 0;
|
||||
const iv = setInterval(() => {
|
||||
i += 1;
|
||||
const k = Math.min(1, i / steps);
|
||||
el.volume = Math.min(1, Math.max(0, from + (to - from) * k));
|
||||
if (k >= 1) { clearInterval(iv); if (done) done(); }
|
||||
}, 30);
|
||||
}
|
||||
|
||||
// The current altitude's soundtrack url (the ring carries each scale's `audio`).
|
||||
function soundtrackUrl() {
|
||||
const s = ring && ring.scales[ringIndex];
|
||||
return s && s.audio ? "/media/audio/" + s.audio : null;
|
||||
}
|
||||
|
||||
// Load `url` into the single element and fade it in. Call this SYNCHRONOUSLY from
|
||||
// the toggle's click the first time so play() is allowed; later calls reuse the
|
||||
// now-unlocked element.
|
||||
function playUrl(url) {
|
||||
aud.src = mediaUrl(url);
|
||||
aud.volume = 0;
|
||||
const pr = aud.play();
|
||||
if (pr) pr.catch(() => {});
|
||||
fadeVolume(aud, 1, FADE_MS);
|
||||
}
|
||||
|
||||
// Reconcile audio to the Audio toggle + current altitude: on → the scale's
|
||||
// soundtrack, off → silence. Idempotent on an unchanged url (so a same-scale pool
|
||||
// re-roll doesn't restart it). MUST be called synchronously from the toggle gesture
|
||||
// the first time it plays.
|
||||
function applyAudio() {
|
||||
const url = $("audio").checked ? soundtrackUrl() : null;
|
||||
if (url === audUrl) return;
|
||||
audUrl = url;
|
||||
if (!url) { fadeVolume(aud, 0, FADE_MS, () => aud.pause()); return; }
|
||||
if (aud.paused || !aud.src) playUrl(url); // start fresh
|
||||
else fadeVolume(aud, 0, 250, () => playUrl(url)); // dip out, then swap + fade in
|
||||
}
|
||||
|
||||
// "Loading Universe…" splash — hidden once media is preloaded; reflects progress.
|
||||
function setLoadingProgress(done, total) {
|
||||
const fill = document.getElementById("loading-fill");
|
||||
if (fill && total) fill.style.width = Math.round((done / total) * 100) + "%";
|
||||
}
|
||||
function hideLoading() {
|
||||
const el = document.getElementById("loading");
|
||||
if (!el) return;
|
||||
el.classList.add("done");
|
||||
setTimeout(() => el.remove(), 700);
|
||||
}
|
||||
|
||||
let videoEverOn = false; // has Video ever been switched on this session?
|
||||
|
||||
async function main() {
|
||||
devLiveReload();
|
||||
try { initPaint(); } catch (e) { paintOK = false; paint.style.display = "none"; showError("WebGL init: " + e.message); }
|
||||
await loadData();
|
||||
await landScale(); // pick the initial scale's pool member before first render
|
||||
preloadAllMedia(); // background: cache every clip in memory for instant altitude swaps
|
||||
buildDial(); // draw the altitude knob from the ring's scales
|
||||
initDev(); // wire the Dev Mode toggle + pool picker (reads persisted state)
|
||||
renderScaleReadout();
|
||||
for (const id of ["content", "left", "right", "mood"]) {
|
||||
$(id).addEventListener("input", debounced);
|
||||
}
|
||||
// Sliders stream on "input"; the toggles fire on "change".
|
||||
for (const id of ["left", "right", "mood"]) $(id).addEventListener("input", debounced);
|
||||
// Audio toggle: start/stop the soundtrack SYNCHRONOUSLY in this gesture so it
|
||||
// unlocks + plays on Safari (which blocks play() outside a user gesture).
|
||||
$("audio").addEventListener("change", () => { applyAudio(); debounced(); });
|
||||
// Video toggle: the FIRST time video turns on, bring audio with it (one flip = the
|
||||
// full experience). Audio toggled on its own first stays audio-only. The audio is
|
||||
// set + played in THIS gesture so it unlocks on Safari.
|
||||
$("visual").addEventListener("change", () => {
|
||||
if ($("visual").checked && !videoEverOn) {
|
||||
videoEverOn = true;
|
||||
if (!$("audio").checked) { $("audio").checked = true; applyAudio(); }
|
||||
}
|
||||
debounced();
|
||||
});
|
||||
// Altitude knob: drag to turn (commit detents on release), scroll to step, tap a label to jump.
|
||||
dial.addEventListener("pointerdown", onDialDown);
|
||||
window.addEventListener("pointermove", onDialMove);
|
||||
window.addEventListener("pointerup", onDialUp);
|
||||
dial.addEventListener("wheel", onWheel, { passive: false });
|
||||
$("stage").addEventListener("wheel", onWheel, { passive: false });
|
||||
update();
|
||||
update(); // render the initial state (both toggles off → black, silent)
|
||||
await preloadAllMedia(); // download all media, updating the loading bar
|
||||
hideLoading(); // experience is ready to run smoothly
|
||||
}
|
||||
main();
|
||||
|
||||
+24
-10
@@ -7,11 +7,18 @@
|
||||
<link rel="stylesheet" href="/style.css" />
|
||||
</head>
|
||||
<body>
|
||||
<div id="loading">
|
||||
<div class="loading-inner">
|
||||
<div class="loading-title">Loading Universe<span class="loading-dots"></span></div>
|
||||
<div class="loading-bar"><div id="loading-fill"></div></div>
|
||||
</div>
|
||||
</div>
|
||||
<header><h1>Human Experience Filter — Alteration Preview</h1></header>
|
||||
<main>
|
||||
<section class="stage" id="stage">
|
||||
<div class="screen">
|
||||
<video id="vid" loop muted playsinline></video>
|
||||
<audio id="aud" loop preload="auto"></audio>
|
||||
<canvas id="paint"></canvas>
|
||||
<div id="tint"></div>
|
||||
<svg id="overlay" viewBox="0 0 100 100" preserveAspectRatio="none"></svg>
|
||||
@@ -22,16 +29,17 @@
|
||||
|
||||
<section class="panel">
|
||||
<fieldset>
|
||||
<legend>Content dial</legend>
|
||||
<select id="content">
|
||||
<option value="video">video</option>
|
||||
<option value="audio_video">audio + video</option>
|
||||
<option value="music_video">music + video</option>
|
||||
<option value="off">off (black)</option>
|
||||
<option value="white_noise">white noise (no video)</option>
|
||||
<option value="music">music (no video)</option>
|
||||
<option value="audio_track">audio track (no video)</option>
|
||||
</select>
|
||||
<legend>Output</legend>
|
||||
<label class="dev-switch" for="visual">
|
||||
<input type="checkbox" id="visual" />
|
||||
<span class="dev-switch-track"><span class="dev-switch-thumb"></span></span>
|
||||
<span class="dev-switch-label">Video</span>
|
||||
</label>
|
||||
<label class="dev-switch" for="audio">
|
||||
<input type="checkbox" id="audio" />
|
||||
<span class="dev-switch-track"><span class="dev-switch-thumb"></span></span>
|
||||
<span class="dev-switch-label">Audio</span>
|
||||
</label>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
@@ -83,6 +91,12 @@
|
||||
<legend>Cache & playback</legend>
|
||||
<dl id="dev-stats" class="dev-dl"></dl>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Review galleries</legend>
|
||||
<a class="dev-link" href="/review.html" target="_blank" rel="noopener">🎞️ Clip pool review</a>
|
||||
<a class="dev-link" href="/review_audio.html" target="_blank" rel="noopener">🔊 Audio pool review</a>
|
||||
</fieldset>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
body { margin: 0; font: 14px/1.4 system-ui, sans-serif; background: #111; color: #eee; }
|
||||
header { padding: 0.6rem 1rem; background: #000; }
|
||||
h1 { font-size: 1rem; margin: 0; font-weight: 600; }
|
||||
main { display: flex; gap: 1rem; padding: 1rem; flex-wrap: wrap; }
|
||||
.stage { flex: 1 1 640px; }
|
||||
main { display: flex; gap: 1rem; padding: 1rem; flex-wrap: wrap; align-items: flex-start; }
|
||||
/* Stage stays pinned in view while the right pane scrolls on its own. */
|
||||
.stage { flex: 1 1 640px; position: sticky; top: 1rem; }
|
||||
.screen { position: relative; width: 100%; aspect-ratio: 16 / 9; background: #000;
|
||||
border-radius: 6px; overflow: hidden; }
|
||||
#vid { width: 100%; height: 100%; object-fit: cover; transition: opacity 0.15s ease; }
|
||||
@@ -39,9 +40,18 @@ main { display: flex; gap: 1rem; padding: 1rem; flex-wrap: wrap; }
|
||||
.hud-chip-text.measure { fill: #ffd79a; }
|
||||
.hud-conf { fill: #6cf; opacity: 0.8; }
|
||||
.hud-status { fill: #8fdcff; font: 2.4px monospace; opacity: 0.9; letter-spacing: 0.15px; }
|
||||
.black { position: absolute; inset: 0; background: #000; opacity: 1; transition: opacity 200ms ease; }
|
||||
/* z-index + own compositing layer: the <video> and WebGL <canvas> are GPU-
|
||||
composited layers that can paint ABOVE a plain auto-z sibling (Safari/Chrome
|
||||
with a real GPU) despite DOM order — so the black cover needs an explicit
|
||||
stacking order above them to actually blank the screen. */
|
||||
.black { position: absolute; inset: 0; background: #000; opacity: 1;
|
||||
z-index: 50; transform: translateZ(0); transition: opacity 200ms ease; }
|
||||
.hidden { display: none; }
|
||||
.panel { flex: 0 0 280px; display: flex; flex-direction: column; gap: 0.8rem; }
|
||||
/* Stack the two Output toggles (Video / Audio) with a little breathing room. */
|
||||
.dev-switch + .dev-switch { margin-top: 0.45rem; }
|
||||
.panel { flex: 0 0 280px; display: flex; flex-direction: column; gap: 0.8rem;
|
||||
max-height: calc(100vh - 2rem); overflow-y: auto;
|
||||
position: sticky; top: 1rem; }
|
||||
fieldset { border: 1px solid #333; border-radius: 6px; }
|
||||
legend { color: #9af; padding: 0 0.4rem; }
|
||||
label { display: block; margin: 0.4rem 0; }
|
||||
@@ -87,6 +97,10 @@ input[type=range], select { width: 100%; }
|
||||
.dev-btn { margin-top: 0.5rem; width: 100%; padding: 0.4rem; cursor: pointer;
|
||||
background: #182028; color: #cde; border: 1px solid #2c3c5c; border-radius: 4px; }
|
||||
.dev-btn:hover { background: #20303f; }
|
||||
.dev-link { display: block; margin-top: 0.4rem; padding: 0.4rem; text-align: center;
|
||||
background: #182028; color: #cde; border: 1px solid #2c3c5c; border-radius: 4px;
|
||||
text-decoration: none; }
|
||||
.dev-link:hover { background: #20303f; }
|
||||
.dev-dl { margin: 0; display: grid; grid-template-columns: auto 1fr; gap: 0.15rem 0.6rem; font-size: 12px; }
|
||||
.dev-dl dt { color: #789; }
|
||||
.dev-dl dd { margin: 0; color: #dde; word-break: break-word; }
|
||||
@@ -100,3 +114,29 @@ input[type=range], select { width: 100%; }
|
||||
.dev-anno .anno-key.measure { color: #ffd79a; }
|
||||
.dev-anno .anno-meta { color: #678; }
|
||||
.dev-anno .anno-empty { color: #567; font-style: italic; }
|
||||
|
||||
/* "Loading Universe…" splash — shown until all media is preloaded and the
|
||||
experience is ready to run smoothly, then faded out. */
|
||||
#loading {
|
||||
position: fixed; inset: 0; z-index: 10000;
|
||||
display: flex; align-items: center; justify-content: center;
|
||||
background: radial-gradient(ellipse at center, #0a1230 0%, #02030a 70%);
|
||||
color: #cfe3ff; user-select: none;
|
||||
transition: opacity 0.6s ease;
|
||||
}
|
||||
#loading.done { opacity: 0; pointer-events: none; }
|
||||
.loading-inner { display: flex; flex-direction: column; align-items: center; gap: 1.1rem; }
|
||||
.loading-title { font: 600 30px/1.2 system-ui, sans-serif; letter-spacing: 0.04em; }
|
||||
.loading-dots::after {
|
||||
content: ""; animation: loading-dots 1.4s steps(4, end) infinite;
|
||||
}
|
||||
@keyframes loading-dots { 0% { content: ""; } 25% { content: "."; } 50% { content: ".."; } 75% { content: "..."; } 100% { content: ""; } }
|
||||
.loading-bar {
|
||||
width: 260px; height: 4px; border-radius: 2px;
|
||||
background: rgba(255, 255, 255, 0.12); overflow: hidden;
|
||||
}
|
||||
#loading-fill {
|
||||
height: 100%; width: 0%; border-radius: 2px;
|
||||
background: linear-gradient(90deg, #4e9cff, #9af);
|
||||
transition: width 0.25s ease;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
"""Playwright E2E for the audio + Video/Audio toggles (audio spec §9). Skips
|
||||
cleanly when Playwright or its browser binary is absent.
|
||||
|
||||
NOTE: headless engines relax BOTH autoplay and GPU compositing, so these assert
|
||||
the WIRING (toggle→play, the first-video-on→audio coupling, video-off blanking) —
|
||||
real Safari/iOS autoplay and real-GPU compositing still need a device by-ear/eye check.
|
||||
|
||||
Install: pip install -e '.[e2e]' && python -m playwright install chromium
|
||||
"""
|
||||
import socket
|
||||
import threading
|
||||
import time
|
||||
from contextlib import closing
|
||||
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("playwright.sync_api")
|
||||
|
||||
|
||||
def _free_port() -> int:
|
||||
with closing(socket.socket()) as s:
|
||||
s.bind(("127.0.0.1", 0))
|
||||
return s.getsockname()[1]
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url():
|
||||
import uvicorn
|
||||
|
||||
from simulator.app import app
|
||||
|
||||
port = _free_port()
|
||||
config = uvicorn.Config(app, host="127.0.0.1", port=port, log_level="error")
|
||||
server = uvicorn.Server(config)
|
||||
thread = threading.Thread(target=server.run, daemon=True)
|
||||
thread.start()
|
||||
for _ in range(50):
|
||||
if server.started:
|
||||
break
|
||||
time.sleep(0.1)
|
||||
if not server.started:
|
||||
pytest.skip("uvicorn did not start")
|
||||
yield f"http://127.0.0.1:{port}"
|
||||
server.should_exit = True
|
||||
thread.join(timeout=5)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def page(app_url):
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
with sync_playwright() as p:
|
||||
try:
|
||||
# headless blocks media playback without a device; this flag lets play()
|
||||
# actually start so `paused` is meaningful (spec §9 asserts src/paused)
|
||||
browser = p.chromium.launch(args=["--autoplay-policy=no-user-gesture-required"])
|
||||
except Exception as exc: # no browser binary installed
|
||||
pytest.skip(f"chromium not available: {exc}")
|
||||
pg = browser.new_page()
|
||||
pg.goto(app_url)
|
||||
# wait out the "Loading Universe…" splash (it overlays + blocks clicks)
|
||||
pg.wait_for_selector("#loading", state="detached", timeout=60000)
|
||||
yield pg
|
||||
browser.close()
|
||||
|
||||
|
||||
def _toggle(page, which):
|
||||
"""Click the visible toggle track for #visual / #audio (the input is hidden)."""
|
||||
page.click(f"label[for='{which}'] .dev-switch-track")
|
||||
|
||||
|
||||
def test_app_starts_blanked_and_silent(page):
|
||||
# both toggles default off → black screen, no audio (no tap-to-start wall)
|
||||
assert page.is_checked("#visual") is False
|
||||
assert page.is_checked("#audio") is False
|
||||
page.wait_for_selector("#black:not(.hidden)")
|
||||
assert page.evaluate("document.getElementById('aud').paused") is True
|
||||
|
||||
|
||||
def test_audio_toggle_plays_the_current_scale_soundtrack(page):
|
||||
_toggle(page, "audio")
|
||||
page.wait_for_function(
|
||||
"(() => { const a = document.getElementById('aud');"
|
||||
" return /\\/media\\/audio\\/.+\\.mp3/.test(a.src) && !a.paused; })()"
|
||||
)
|
||||
|
||||
|
||||
def test_first_video_on_also_enables_audio(page):
|
||||
_toggle(page, "visual") # first video-on couples audio
|
||||
page.wait_for_function("document.getElementById('audio').checked === true")
|
||||
page.wait_for_function(
|
||||
"(() => { const a = document.getElementById('aud'); return !!a.src && !a.paused; })()"
|
||||
)
|
||||
page.wait_for_selector("#black.hidden", state="attached") # video showing (black hidden=display:none)
|
||||
|
||||
|
||||
def test_audio_before_video_stays_audio_only(page):
|
||||
_toggle(page, "audio") # audio on first
|
||||
page.wait_for_function("!document.getElementById('aud').paused")
|
||||
assert page.is_checked("#visual") is False # video untouched
|
||||
page.wait_for_selector("#black:not(.hidden)") # still blanked
|
||||
|
||||
|
||||
def test_video_off_blanks_after_being_on(page):
|
||||
_toggle(page, "visual") # on
|
||||
page.wait_for_selector("#black.hidden", state="attached")
|
||||
_toggle(page, "visual") # off
|
||||
page.wait_for_selector("#black:not(.hidden)")
|
||||
# the GPU-composited video layers are hidden too (settle the opacity transition)
|
||||
page.wait_for_function(
|
||||
"['vid','paint'].every(id => getComputedStyle(document.getElementById(id)).opacity === '0')"
|
||||
)
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Unit tests for the pure audio ffmpeg arg builders (no ffmpeg run), plus an
|
||||
opt-in integration test that actually synthesizes a noise bed when ffmpeg is
|
||||
present."""
|
||||
|
||||
import shutil
|
||||
|
||||
import pytest
|
||||
|
||||
from tools.pipeline.audio_ops import audio_loop_args, loudnorm_args, white_noise_args
|
||||
|
||||
|
||||
def test_loop_args_mirror_the_video_crossfade_recipe():
|
||||
args = audio_loop_args("in.mp3", "out.mp3", duration=30.0, overlap=2.0, ff="FF")
|
||||
assert args[0] == "FF"
|
||||
joined = " ".join(args)
|
||||
# tail crossfades over head, then concats the middle — the audio analogue of
|
||||
# crossfade_loop_args. Output length = duration - overlap.
|
||||
assert "acrossfade=d=2.0" in joined
|
||||
assert "atrim=0:2.0" in joined # head
|
||||
assert "atrim=28.0:30.0" in joined # tail (d-overlap : d)
|
||||
assert "concat=n=2:v=0:a=1" in joined
|
||||
assert "-map" in args and "[out]" in args
|
||||
assert "-q:a" in args and "4" in args
|
||||
|
||||
|
||||
def test_loop_args_reject_overlap_past_half():
|
||||
with pytest.raises(ValueError):
|
||||
audio_loop_args("in.mp3", "out.mp3", duration=10.0, overlap=5.0)
|
||||
|
||||
|
||||
def test_loudnorm_args_carry_the_locked_targets():
|
||||
args = loudnorm_args("in.mp3", "out.mp3", ff="FF")
|
||||
joined = " ".join(args)
|
||||
assert "loudnorm=I=-18.0:TP=-1.5:LRA=11.0" in joined
|
||||
assert args[0] == "FF" and args[-1] == "out.mp3"
|
||||
|
||||
|
||||
def test_white_noise_args_are_pink_and_clean():
|
||||
args = white_noise_args("noise.mp3", duration=60.0, color="pink", ff="FF")
|
||||
joined = " ".join(args)
|
||||
assert "anoisesrc=color=pink" in joined
|
||||
assert "-t" in args and "60.0" in args
|
||||
assert args[-1] == "noise.mp3"
|
||||
|
||||
|
||||
def test_white_noise_rejects_unknown_color():
|
||||
with pytest.raises(ValueError):
|
||||
white_noise_args("noise.mp3", color="ultraviolet")
|
||||
|
||||
|
||||
def _have_ffmpeg() -> bool:
|
||||
if shutil.which("ffmpeg"):
|
||||
return True
|
||||
try:
|
||||
import imageio_ffmpeg # noqa: F401
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _have_ffmpeg(), reason="no ffmpeg available")
|
||||
def test_generate_white_noise_writes_a_playable_loop(tmp_path):
|
||||
from tools.pipeline.audio_run import generate_white_noise
|
||||
out = generate_white_noise(tmp_path / "pink.mp3", duration=2.0)
|
||||
assert out.exists() and out.stat().st_size > 0
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Unit tests for the pure Visual×Audio resolver (audio spec §4/§6)."""
|
||||
import pytest
|
||||
|
||||
from player.audio import (
|
||||
AUDIO_SOURCES,
|
||||
VISUAL_POSITIONS,
|
||||
AudioResolution,
|
||||
resolve_audio,
|
||||
resolve_audio_url,
|
||||
resolve_visual,
|
||||
)
|
||||
|
||||
def test_visual_on_off():
|
||||
assert resolve_visual("on") is True
|
||||
assert resolve_visual("off") is False
|
||||
|
||||
|
||||
def test_visual_rejects_unknown():
|
||||
with pytest.raises(ValueError):
|
||||
resolve_visual("flicker")
|
||||
|
||||
|
||||
def test_visual_positions_are_on_off():
|
||||
assert VISUAL_POSITIONS == frozenset({"on", "off"})
|
||||
|
||||
|
||||
def test_audio_sources_are_off_and_soundtrack():
|
||||
assert AUDIO_SOURCES == frozenset({"off", "soundtrack"})
|
||||
|
||||
|
||||
def test_soundtrack_resolves_to_the_current_scales_asset():
|
||||
url = resolve_audio_url("soundtrack", scale_audio="coast/waves.loop.mp3")
|
||||
assert url == "/media/audio/coast/waves.loop.mp3"
|
||||
|
||||
|
||||
def test_off_resolves_to_silence():
|
||||
assert resolve_audio_url("off", scale_audio="coast/waves.loop.mp3") is None
|
||||
|
||||
|
||||
def test_soundtrack_with_no_scale_asset_is_silent():
|
||||
assert resolve_audio_url("soundtrack", scale_audio="") is None
|
||||
|
||||
|
||||
def test_resolve_audio_marks_only_soundtrack_altitude_coupled():
|
||||
r = resolve_audio("soundtrack", scale_audio="reef/soundscape.loop.mp3")
|
||||
assert isinstance(r, AudioResolution)
|
||||
assert r.source == "soundtrack" and r.altitude_coupled is True
|
||||
assert r.url == "/media/audio/reef/soundscape.loop.mp3"
|
||||
assert resolve_audio("off", scale_audio="x/y.mp3").altitude_coupled is False
|
||||
|
||||
|
||||
def test_resolve_audio_url_rejects_unknown_source():
|
||||
with pytest.raises(ValueError):
|
||||
resolve_audio_url("white_noise", scale_audio="") # deferred, not a v1 source
|
||||
|
||||
|
||||
def test_music_and_white_noise_are_not_v1_sources():
|
||||
assert "music" not in AUDIO_SOURCES
|
||||
assert "white_noise" not in AUDIO_SOURCES
|
||||
@@ -1,34 +0,0 @@
|
||||
import pytest
|
||||
|
||||
from player.content import AUDIO_SOURCES, ContentResolution, resolve_content
|
||||
|
||||
|
||||
def test_audio_sources_are_the_four_distinct_sources():
|
||||
assert AUDIO_SOURCES == frozenset({"none", "white_noise", "music", "audio_track"})
|
||||
|
||||
|
||||
# The §6 table, row by row: position -> (audio_source, video).
|
||||
@pytest.mark.parametrize(
|
||||
"position,audio_source,video",
|
||||
[
|
||||
("off", "none", False),
|
||||
("white_noise", "white_noise", False),
|
||||
("music", "music", False),
|
||||
("audio_track", "audio_track", False),
|
||||
("video", "none", True),
|
||||
("music_video", "music", True),
|
||||
("audio_video", "audio_track", True),
|
||||
],
|
||||
)
|
||||
def test_resolve_content_matches_spec_table(position, audio_source, video):
|
||||
assert resolve_content(position) == ContentResolution(audio_source=audio_source, video=video)
|
||||
|
||||
|
||||
def test_off_is_void_state_black_and_silent():
|
||||
r = resolve_content("off")
|
||||
assert r.video is False and r.audio_source == "none"
|
||||
|
||||
|
||||
def test_resolve_content_rejects_unknown_position():
|
||||
with pytest.raises(ValueError):
|
||||
resolve_content("bogus")
|
||||
@@ -1,27 +1,36 @@
|
||||
import pytest
|
||||
|
||||
from player.audio import AUDIO_SOURCES, VISUAL_POSITIONS
|
||||
from player.controls import (
|
||||
Controls,
|
||||
CONTENT_POSITIONS,
|
||||
ControlsError,
|
||||
validate_controls,
|
||||
parse_controls,
|
||||
)
|
||||
|
||||
|
||||
def test_content_positions_are_the_seven_from_the_spec():
|
||||
assert CONTENT_POSITIONS == frozenset(
|
||||
{"off", "white_noise", "music", "audio_track", "video", "music_video", "audio_video"}
|
||||
)
|
||||
def test_visual_and_audio_enums_are_the_v1_positions():
|
||||
assert VISUAL_POSITIONS == frozenset({"on", "off"})
|
||||
assert AUDIO_SOURCES == frozenset({"off", "soundtrack"})
|
||||
|
||||
|
||||
def test_valid_controls_pass_validation():
|
||||
c = Controls(content="video", left=0, right=4, dark=2, light=2, volume=3, brightness=4)
|
||||
c = Controls(visual="on", audio="soundtrack", left=0, right=4, dark=2, light=2,
|
||||
volume=3, brightness=4)
|
||||
validate_controls(c) # does not raise
|
||||
|
||||
|
||||
def test_invalid_content_position_rejected():
|
||||
c = Controls(content="bogus", left=0, right=0, dark=0, light=0, volume=0, brightness=0)
|
||||
def test_invalid_visual_position_rejected():
|
||||
c = Controls(visual="flicker", audio="off", left=0, right=0, dark=0, light=0,
|
||||
volume=0, brightness=0)
|
||||
with pytest.raises(ControlsError):
|
||||
validate_controls(c)
|
||||
|
||||
|
||||
def test_invalid_audio_source_rejected():
|
||||
# "music" is a reserved/deferred dial position, not a v1 source
|
||||
c = Controls(visual="on", audio="music", left=0, right=0, dark=0, light=0,
|
||||
volume=0, brightness=0)
|
||||
with pytest.raises(ControlsError):
|
||||
validate_controls(c)
|
||||
|
||||
@@ -29,7 +38,8 @@ def test_invalid_content_position_rejected():
|
||||
@pytest.mark.parametrize("field", ["left", "right", "dark", "light", "volume", "brightness"])
|
||||
@pytest.mark.parametrize("bad", [-1, 5, True])
|
||||
def test_out_of_range_or_non_int_knob_rejected(field, bad):
|
||||
kwargs = dict(content="off", left=0, right=0, dark=0, light=0, volume=0, brightness=0)
|
||||
kwargs = dict(visual="off", audio="off", left=0, right=0, dark=0, light=0,
|
||||
volume=0, brightness=0)
|
||||
kwargs[field] = bad
|
||||
with pytest.raises(ControlsError):
|
||||
validate_controls(Controls(**kwargs))
|
||||
@@ -37,17 +47,18 @@ def test_out_of_range_or_non_int_knob_rejected(field, bad):
|
||||
|
||||
def test_parse_controls_from_mapping():
|
||||
c = parse_controls(
|
||||
{"content": "music_video", "left": 1, "right": 2, "dark": 3, "light": 0, "volume": 2, "brightness": 1}
|
||||
{"visual": "on", "audio": "soundtrack", "left": 1, "right": 2, "dark": 3,
|
||||
"light": 0, "volume": 2, "brightness": 1}
|
||||
)
|
||||
assert c == Controls("music_video", 1, 2, 3, 0, 2, 1)
|
||||
assert c == Controls("on", "soundtrack", 1, 2, 3, 0, 2, 1)
|
||||
|
||||
|
||||
def test_parse_controls_rejects_unknown_keys():
|
||||
with pytest.raises(ControlsError):
|
||||
parse_controls({"content": "off", "left": 0, "right": 0, "dark": 0,
|
||||
parse_controls({"visual": "off", "audio": "off", "left": 0, "right": 0, "dark": 0,
|
||||
"light": 0, "volume": 0, "brightness": 0, "bogus": 1})
|
||||
|
||||
|
||||
def test_parse_controls_rejects_missing_keys():
|
||||
with pytest.raises(ControlsError):
|
||||
parse_controls({"content": "off", "left": 0})
|
||||
parse_controls({"visual": "off", "left": 0})
|
||||
|
||||
@@ -240,3 +240,10 @@ def test_pick_clip_id_clamps_out_of_range_r():
|
||||
assert pick_clip_id(s, -5.0) == "a" # below 0 clamps to first
|
||||
assert pick_clip_id(s, 1.0) == "c" # 1.0 must not index past the end
|
||||
assert pick_clip_id(s, 999.0) == "c"
|
||||
|
||||
|
||||
def test_scale_carries_an_optional_audio_path():
|
||||
s = Scale(id="coast", clip_id="coast_birdrock", audio="coast/waves.loop.mp3")
|
||||
assert s.audio == "coast/waves.loop.mp3"
|
||||
# back-compat: audio defaults to empty
|
||||
assert Scale(id="x", clip_id="x").audio == ""
|
||||
|
||||
+35
-25
@@ -14,47 +14,57 @@ class FakeClip:
|
||||
LIB = [FakeClip("base-a"), FakeClip("base-b")]
|
||||
|
||||
|
||||
def _controls(content="video", left=0, right=0, dark=0, light=0, volume=2, brightness=2):
|
||||
return Controls(content, left, right, dark, light, volume, brightness)
|
||||
def _controls(visual="on", audio="off", left=0, right=0, dark=0, light=0,
|
||||
volume=2, brightness=2):
|
||||
return Controls(visual, audio, left, right, dark, light, volume, brightness)
|
||||
|
||||
|
||||
def test_first_update_to_video_fades_in_from_black():
|
||||
p = Player(LIB)
|
||||
t = p.update(_controls(content="video"))
|
||||
t = p.update(_controls(visual="on"))
|
||||
assert t.kind == TransitionKind.FADE_FROM_BLACK
|
||||
assert t.playback.clip_id == "base-a"
|
||||
assert t.playback.content.video is True
|
||||
assert t.playback.video is True
|
||||
|
||||
|
||||
def test_off_from_video_fades_to_black_and_silences():
|
||||
def test_visual_off_from_video_fades_to_black():
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="video"))
|
||||
t = p.update(_controls(content="off"))
|
||||
p.update(_controls(visual="on"))
|
||||
t = p.update(_controls(visual="off"))
|
||||
assert t.kind == TransitionKind.FADE_TO_BLACK
|
||||
assert t.playback.clip_id is None
|
||||
assert t.playback.content.audio_source == "none"
|
||||
assert t.playback.video is False
|
||||
assert t.playback.audio_source == "off"
|
||||
|
||||
|
||||
def test_audio_survives_visual_off():
|
||||
# soundtrack + black is reachable (a gap the bundled dial omitted)
|
||||
p = Player(LIB)
|
||||
t = p.update(_controls(visual="off", audio="soundtrack"))
|
||||
assert t.playback.video is False
|
||||
assert t.playback.audio_source == "soundtrack"
|
||||
|
||||
|
||||
def test_no_change_yields_none_transition():
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="video", left=1))
|
||||
t = p.update(_controls(content="video", left=1))
|
||||
p.update(_controls(visual="on", left=1))
|
||||
t = p.update(_controls(visual="on", left=1))
|
||||
assert t.kind == TransitionKind.NONE
|
||||
|
||||
|
||||
def test_grade_change_is_a_live_update_not_a_crossfade():
|
||||
# design §4.3: the Dark/Light grade is a continuous runtime op
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="video", dark=0, light=0))
|
||||
t = p.update(_controls(content="video", dark=4, light=0))
|
||||
p.update(_controls(visual="on", dark=0, light=0))
|
||||
t = p.update(_controls(visual="on", dark=4, light=0))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE
|
||||
assert t.playback.plan.grade.tone == -1.0
|
||||
|
||||
|
||||
def test_overlay_change_is_a_live_update():
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="video", left=0))
|
||||
t = p.update(_controls(content="video", left=4))
|
||||
p.update(_controls(visual="on", left=0))
|
||||
t = p.update(_controls(visual="on", left=4))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE
|
||||
assert t.playback.plan.overlay.level == 4
|
||||
|
||||
@@ -63,8 +73,8 @@ def test_dream_change_is_a_live_update_not_a_crossfade():
|
||||
# Right-axis dream reframe (session 0013): the Right dream is a deterministic
|
||||
# LIVE filter now, so changing it no longer crossfades — only a clip swap does.
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="video", right=0))
|
||||
t = p.update(_controls(content="video", right=4))
|
||||
p.update(_controls(visual="on", right=0))
|
||||
t = p.update(_controls(visual="on", right=4))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE
|
||||
assert t.playback.plan.dream.strength == 4
|
||||
assert t.playback.plan.dream.intensity == 1.0
|
||||
@@ -72,43 +82,43 @@ def test_dream_change_is_a_live_update_not_a_crossfade():
|
||||
|
||||
def test_volume_only_change_is_a_live_update():
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="video", volume=1))
|
||||
t = p.update(_controls(content="video", volume=4))
|
||||
p.update(_controls(visual="on", volume=1))
|
||||
t = p.update(_controls(visual="on", volume=4))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE
|
||||
assert t.playback.volume == 4
|
||||
|
||||
|
||||
def test_audio_source_change_while_black_is_a_live_update():
|
||||
p = Player(LIB)
|
||||
p.update(_controls(content="white_noise"))
|
||||
t = p.update(_controls(content="music"))
|
||||
p.update(_controls(visual="off", audio="off"))
|
||||
t = p.update(_controls(visual="off", audio="soundtrack"))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE
|
||||
assert t.playback.content.audio_source == "music"
|
||||
assert t.playback.audio_source == "soundtrack"
|
||||
|
||||
|
||||
def test_injected_base_chooser_is_used():
|
||||
p = Player(LIB, choose_base=lambda lib: lib[1])
|
||||
t = p.update(_controls(content="video"))
|
||||
t = p.update(_controls(visual="on"))
|
||||
assert t.playback.clip_id == "base-b"
|
||||
|
||||
|
||||
def test_empty_library_with_video_raises():
|
||||
p = Player([])
|
||||
with pytest.raises(ValueError):
|
||||
p.update(_controls(content="video"))
|
||||
p.update(_controls(visual="on"))
|
||||
|
||||
|
||||
def test_off_with_empty_library_is_fine():
|
||||
p = Player([])
|
||||
# levels at 0 match the initial black state, so this is a no-op transition
|
||||
t = p.update(_controls(content="off", volume=0, brightness=0))
|
||||
t = p.update(_controls(visual="off", volume=0, brightness=0))
|
||||
assert t.kind == TransitionKind.NONE # already black at init
|
||||
assert t.playback.clip_id is None
|
||||
|
||||
|
||||
def test_off_with_levels_from_black_is_a_live_update():
|
||||
p = Player([])
|
||||
t = p.update(_controls(content="off", volume=3, brightness=2))
|
||||
t = p.update(_controls(visual="off", volume=3, brightness=2))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE # black->black, levels set
|
||||
assert t.playback.clip_id is None
|
||||
assert t.playback.volume == 3
|
||||
|
||||
+54
-12
@@ -28,8 +28,9 @@ def client(manifest_path):
|
||||
return TestClient(create_app(manifest_path=manifest_path))
|
||||
|
||||
|
||||
def _controls(content="video", left=0, right=0, dark=0, light=0, volume=2, brightness=2):
|
||||
return dict(content=content, left=left, right=right, dark=dark,
|
||||
def _controls(visual="on", audio="off", left=0, right=0, dark=0, light=0,
|
||||
volume=2, brightness=2):
|
||||
return dict(visual=visual, audio=audio, left=left, right=right, dark=dark,
|
||||
light=light, volume=volume, brightness=brightness)
|
||||
|
||||
|
||||
@@ -41,13 +42,40 @@ def test_alteration_returns_the_engine_plan(client):
|
||||
assert data["plan"]["dream"]["strength"] == 2
|
||||
assert data["plan"]["dream"]["intensity"] == 0.5
|
||||
assert data["plan"]["grade"]["tone"] == -1.0
|
||||
assert data["content"]["video"] is True
|
||||
assert data["render"]["video"]["shown"] is True
|
||||
|
||||
|
||||
def test_alteration_honors_off_as_black(client):
|
||||
resp = client.post("/api/alteration", json={"controls": _controls(content="off")})
|
||||
data = resp.json()
|
||||
assert data["content"]["video"] is False
|
||||
def test_alteration_returns_render_video_and_audio(client):
|
||||
body = client.post("/api/alteration", json={"controls": _controls()}).json()
|
||||
assert body["render"]["video"]["shown"] is True
|
||||
assert body["render"]["audio"] == {"source": "off", "url": None, "altitude_coupled": False}
|
||||
|
||||
|
||||
def test_alteration_visual_off_hides_video(client):
|
||||
body = client.post("/api/alteration", json={"controls": _controls(visual="off")}).json()
|
||||
assert body["render"]["video"]["shown"] is False
|
||||
|
||||
|
||||
def test_alteration_off_is_silent(ring_client):
|
||||
body = ring_client.post("/api/alteration", json={
|
||||
"controls": _controls(audio="off"), "altitude_index": 0,
|
||||
}).json()
|
||||
assert body["render"]["audio"] == {"source": "off", "url": None, "altitude_coupled": False}
|
||||
|
||||
|
||||
def test_alteration_soundtrack_couples_to_the_given_altitude(ring_client):
|
||||
# ring fixture order: cosmos(0), forest(1), abyss(2) — each carries its audio
|
||||
body = ring_client.post("/api/alteration", json={
|
||||
"controls": _controls(audio="soundtrack"), "altitude_index": 0,
|
||||
}).json()
|
||||
a = body["render"]["audio"]
|
||||
assert a["source"] == "soundtrack" and a["altitude_coupled"] is True
|
||||
assert a["url"] == "/media/audio/cosmos/pillars.loop.mp3"
|
||||
# a different altitude resolves a different soundtrack url
|
||||
body2 = ring_client.post("/api/alteration", json={
|
||||
"controls": _controls(audio="soundtrack"), "altitude_index": 2,
|
||||
}).json()
|
||||
assert body2["render"]["audio"]["url"] == "/media/audio/abyss/whale.loop.mp3"
|
||||
|
||||
|
||||
def test_alteration_accepts_calibration(client):
|
||||
@@ -62,8 +90,13 @@ def test_alteration_rejects_out_of_range_knob(client):
|
||||
assert resp.status_code == 422
|
||||
|
||||
|
||||
def test_alteration_rejects_bad_content(client):
|
||||
resp = client.post("/api/alteration", json={"controls": _controls(content="banana")})
|
||||
def test_alteration_rejects_bad_audio(client):
|
||||
resp = client.post("/api/alteration", json={"controls": _controls(audio="music")})
|
||||
assert resp.status_code == 422
|
||||
|
||||
|
||||
def test_alteration_rejects_bad_visual(client):
|
||||
resp = client.post("/api/alteration", json={"controls": _controls(visual="dim")})
|
||||
assert resp.status_code == 422
|
||||
|
||||
|
||||
@@ -162,9 +195,9 @@ def ring_manifest_path(tmp_path):
|
||||
],
|
||||
"ring": {
|
||||
"scales": [
|
||||
{"id": "cosmos", "clip_id": "cosmos"},
|
||||
{"id": "forest", "clip_id": "forest"},
|
||||
{"id": "abyss", "clip_id": "abyss"},
|
||||
{"id": "cosmos", "clip_id": "cosmos", "audio": "cosmos/pillars.loop.mp3"},
|
||||
{"id": "forest", "clip_id": "forest", "audio": "forest/amb.loop.mp3"},
|
||||
{"id": "abyss", "clip_id": "abyss", "audio": "abyss/whale.loop.mp3"},
|
||||
],
|
||||
"transitions": [
|
||||
{"file": "transitions/cosmos-forest.mp4", "model": "placeholder"},
|
||||
@@ -190,6 +223,15 @@ def test_ring_returns_scales_and_transitions(ring_client):
|
||||
assert len(data["transitions"]) == 3
|
||||
|
||||
|
||||
def test_ring_exposes_each_scales_audio(ring_client):
|
||||
body = ring_client.get("/api/ring").json()
|
||||
by_id = {s["id"]: s for s in body["scales"]}
|
||||
# every scale advertises its soundtrack path (audio spec §5.1)
|
||||
assert by_id["cosmos"]["audio"] == "cosmos/pillars.loop.mp3"
|
||||
assert by_id["abyss"]["audio"].endswith(".mp3")
|
||||
assert "audio" in by_id["forest"]
|
||||
|
||||
|
||||
def test_ring_advance_inward_one_step(ring_client):
|
||||
resp = ring_client.post("/api/ring/advance", json={"from_index": 0, "delta": 1})
|
||||
assert resp.status_code == 200
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Pure ffmpeg argument builders (no I/O) for the audio production pass.
|
||||
|
||||
The audio analogue of tools/pipeline/ffmpeg_ops.py: each function returns a
|
||||
list[str] ready for ffmpeg, so command construction is unit-testable without
|
||||
running ffmpeg. tools/pipeline/audio_run.py executes them. The ffmpeg binary
|
||||
name is injected (default "ffmpeg")."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
_NOISE_COLORS = frozenset({"white", "pink", "brown", "blue", "violet"})
|
||||
|
||||
|
||||
def audio_loop_args(src, dst, *, duration: float, overlap: float,
|
||||
ff: str = "ffmpeg") -> list[str]:
|
||||
"""Make `src` loop seamlessly by crossfading its tail over its head — the
|
||||
audio mirror of ffmpeg_ops.crossfade_loop_args. Output length = duration -
|
||||
overlap. Requires overlap < duration/2 so a non-empty middle remains."""
|
||||
if overlap <= 0 or overlap >= duration / 2:
|
||||
raise ValueError(f"overlap {overlap} must be in (0, duration/2={duration / 2})")
|
||||
d, o = duration, overlap
|
||||
fc = (
|
||||
f"[0:a]atrim=0:{o},asetpts=N/SR/TB[head];"
|
||||
f"[0:a]atrim={o}:{d - o},asetpts=N/SR/TB[mid];"
|
||||
f"[0:a]atrim={d - o}:{d},asetpts=N/SR/TB[tail];"
|
||||
f"[tail][head]acrossfade=d={o}:c1=tri:c2=tri[xf];"
|
||||
f"[xf][mid]concat=n=2:v=0:a=1[out]"
|
||||
)
|
||||
return [
|
||||
ff, "-y", "-i", str(src), "-filter_complex", fc,
|
||||
"-map", "[out]", "-c:a", "libmp3lame", "-q:a", "4", str(dst),
|
||||
]
|
||||
|
||||
|
||||
def loudnorm_args(src, dst, *, i: float = -18.0, tp: float = -1.5,
|
||||
lra: float = 11.0, ff: str = "ffmpeg") -> list[str]:
|
||||
"""EBU R128 loudness normalize to the locked installation target so all five
|
||||
soundtracks (and the noise bed) sit at one comfortable level."""
|
||||
return [
|
||||
ff, "-y", "-i", str(src),
|
||||
"-af", f"loudnorm=I={i}:TP={tp}:LRA={lra}",
|
||||
"-c:a", "libmp3lame", "-q:a", "4", str(dst),
|
||||
]
|
||||
|
||||
|
||||
def white_noise_args(dst, *, duration: float = 60.0, color: str = "pink",
|
||||
ff: str = "ffmpeg") -> list[str]:
|
||||
"""Synthesize a calm colored-noise bed (default pink) of `duration` seconds —
|
||||
deterministic, zero licensing. Loops cleanly (steady-state noise has no seam)."""
|
||||
if color not in _NOISE_COLORS:
|
||||
raise ValueError(
|
||||
f"unknown noise color {color!r}; expected one of {sorted(_NOISE_COLORS)}"
|
||||
)
|
||||
return [
|
||||
ff, "-y", "-f", "lavfi",
|
||||
"-i", f"anoisesrc=color={color}:amplitude=0.5:duration={duration}",
|
||||
"-t", str(duration), "-c:a", "libmp3lame", "-q:a", "4", str(dst),
|
||||
]
|
||||
@@ -0,0 +1,61 @@
|
||||
"""Thin runner: probe duration, then run the audio production pass (seamless
|
||||
loop -> loudness normalize) and synthesize the white-noise bed. The pure arg
|
||||
builders (tools.pipeline.audio_ops) are unit-tested; this glue is covered by the
|
||||
opt-in integration test."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from .audio_ops import audio_loop_args, loudnorm_args, white_noise_args
|
||||
from .run import resolve_ffmpeg
|
||||
|
||||
|
||||
def _probe_audio_duration(src, ff: str) -> float:
|
||||
"""Audio length in seconds via ffmpeg (no system ffprobe): decode to null and
|
||||
read the last reported time off stderr."""
|
||||
proc = subprocess.run(
|
||||
[ff, "-i", str(src), "-f", "null", "-"],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
times = re.findall(r"time=(\d+):(\d+):(\d+\.\d+)", proc.stderr)
|
||||
if not times:
|
||||
raise ValueError(f"could not probe audio duration of {src}")
|
||||
h, m, s = times[-1]
|
||||
return int(h) * 3600 + int(m) * 60 + float(s)
|
||||
|
||||
|
||||
def _run(args: list[str]) -> None:
|
||||
subprocess.run(args, check=True, capture_output=True)
|
||||
|
||||
|
||||
def process_soundtrack(src, dst, *, overlap: float = 2.0, ff: str | None = None) -> Path:
|
||||
"""Produce a seamless, loudness-normalized loop from a sourced ambience clip.
|
||||
Clips too short to crossfade-loop (<= 2*overlap) are normalized only."""
|
||||
ff = ff or resolve_ffmpeg()
|
||||
src, dst = Path(src), Path(dst)
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
duration = _probe_audio_duration(src, ff)
|
||||
tmp = dst.with_suffix(".loop.tmp.mp3")
|
||||
if duration > 2 * overlap + 0.5:
|
||||
_run(audio_loop_args(src, tmp, duration=duration, overlap=overlap, ff=ff))
|
||||
_run(loudnorm_args(tmp, dst, ff=ff))
|
||||
tmp.unlink(missing_ok=True)
|
||||
else:
|
||||
_run(loudnorm_args(src, dst, ff=ff))
|
||||
return dst
|
||||
|
||||
|
||||
def generate_white_noise(dst, *, duration: float = 60.0, color: str = "pink",
|
||||
ff: str | None = None) -> Path:
|
||||
"""Synthesize + normalize the global white-noise bed."""
|
||||
ff = ff or resolve_ffmpeg()
|
||||
dst = Path(dst)
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = dst.with_suffix(".raw.tmp.mp3")
|
||||
_run(white_noise_args(tmp, duration=duration, color=color, ff=ff))
|
||||
_run(loudnorm_args(tmp, dst, ff=ff))
|
||||
tmp.unlink(missing_ok=True)
|
||||
return dst
|
||||
Reference in New Issue
Block a user