Compare commits
21 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 3b3d30e8e0 | |||
| 441645ae7a | |||
| 6427ab4a49 | |||
| 56d08d5c60 | |||
| 97d2ddd573 | |||
| 56e352fe65 | |||
| 2564168ac3 | |||
| deb7c34df9 | |||
| 625d32c0f3 | |||
| 58a6ee4644 | |||
| fd1da791bf | |||
| 4ac39c9034 | |||
| 30ec0c9c26 | |||
| 3d46b96ecb | |||
| 3b19d3eb92 | |||
| fddb6d65d4 | |||
| 2722949805 | |||
| d446238b79 | |||
| c51fdc422a | |||
| aa76cd1c9f | |||
| adc39fe648 |
@@ -17,3 +17,6 @@ simulator/sample_media/**/*.m4a
|
||||
# Local-only candidate review galleries (re-buildable; not shipped content).
|
||||
simulator/static/review.html
|
||||
simulator/static/review_audio.html
|
||||
|
||||
# Editor annotation/comment-thread metadata (not project content)
|
||||
.threads/
|
||||
|
||||
@@ -0,0 +1,931 @@
|
||||
# Altitude Clip-Pair Morph + Lock Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** When the viewer changes altitude, choose the destination clip *first*, play the morph footage that goes from the currently-shown clip to that chosen clip, then lock the clip in place so it never changes again until the altitude changes.
|
||||
|
||||
**Architecture:** Today the ring carries **5** per-edge morphs (each `scale_a → scale_b`, primary→primary), the server picks a random pool member only for the *landed* scale, and `advance()` plays the edge morph (which ends on the scale's *primary*) then does a *second*, jarring `fadeThroughBlack` swap to the randomly-chosen member. We replace the per-edge model with a **per-clip-pair** model: every pair of clips in adjacent altitudes gets its own morph, in both directions (**154** clips total). The server picks the destination member(s) *before* selecting the morph, threads the currently-shown clip in as the source, and chains morphs through each crossed altitude on a multi-detent jump. The renderer plays those exact morphs and locks on the final clip — no secondary swap, no re-roll while parked.
|
||||
|
||||
**Tech Stack:** Python 3.13 / FastAPI (`simulator/app.py`), pure ring logic (`player/ring.py`), pure manifest/schema (`simulator/clips.py`), media baking via ffmpeg `xfade=zoomin` (`simulator/build_pool_manifest.py`), vanilla JS renderer (`simulator/static/app.js`), pytest, and a new Playwright E2E tier.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **Morph file naming:** `transitions/<src_clip_id>__<dst_clip_id>.mp4` (double-underscore separator — clip ids themselves use single `_`, e.g. `cosmos_miri`). The zoom-out companion is `<src>__<dst>.rev.mp4` and represents the **reverse** direction (`dst → src`).
|
||||
- **Direction convention:** a morph is *baked* descending (higher altitude → lower altitude = zoom IN). The `.rev.mp4` companion is the ascending (zoom OUT) playback. The manifest lists **both** directions as explicit, separately-keyed entries so the renderer needs no reverse logic at play time.
|
||||
- **Ring order (descending, wraps):** `["cosmos", "orbit", "coast", "reef", "abyss"]`; `abyss → cosmos` is the wrap edge.
|
||||
- **Pool sizes (current):** cosmos 5, orbit 3, coast 4, reef 5, abyss 3 → **77** adjacent ordered-pair morphs descending + **77** reverse = **154** total transition clips.
|
||||
- **Lock invariant:** the displayed base clip changes **only** inside `advance()` (a deliberate altitude change) and the one initial `landScale()`. Nothing else may reload base media. The Dev-Mode re-roll button stays as an explicit dev affordance.
|
||||
- **Randomness stays injected:** `player/ring.py` is a pure module; the impure `random.random()` draw happens once at the API boundary (`simulator/app.py`). Pure functions take injected `float` picks in `[0,1)`.
|
||||
- **Back-compat:** a manifest with no `pool` (pool of one) and the synthesized single-clip fallback ring must still work. A pool-of-one scale needs no morphs to itself.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Manifest emits clip-pair transition entries
|
||||
|
||||
Re-key the manifest builder from 5 per-edge entries to 154 per-clip-pair entries (both directions), each carrying `from`/`to` clip ids. Pure dict assembly — no ffmpeg.
|
||||
|
||||
**Files:**
|
||||
- Modify: `simulator/build_pool_manifest.py:374-379` (`build_manifest()` transitions block)
|
||||
- Test: `tests/test_pipeline_manifest.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: module constants `RING_ORDER: list[str]`, `POOLS: dict[str, list[str]]`.
|
||||
- Produces: each transition dict shaped `{"from": <src_id>, "to": <dst_id>, "file": "transitions/<src>__<dst>.mp4", "model": "placeholder-zoom"}`. Forward entries are descending (`H→L`, `transitions/H__L.mp4`); reverse entries are ascending (`L→H`, `transitions/H__L.rev.mp4`).
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
# tests/test_pipeline_manifest.py (add)
|
||||
def test_transitions_are_per_clip_pair_both_directions():
|
||||
import simulator.build_pool_manifest as b
|
||||
m = b.build_manifest()
|
||||
trans = m["ring"]["transitions"]
|
||||
order = b.RING_ORDER
|
||||
n = len(order)
|
||||
# Every adjacent (higher H, lower L) edge, every member pair, BOTH directions.
|
||||
expected = set()
|
||||
for i in range(n):
|
||||
H, L = order[i], order[(i + 1) % n]
|
||||
for h in b.POOLS[H]:
|
||||
for l in b.POOLS[L]:
|
||||
expected.add((h, l, f"transitions/{h}__{l}.mp4")) # zoom in
|
||||
expected.add((l, h, f"transitions/{h}__{l}.rev.mp4")) # zoom out
|
||||
got = {(t["from"], t["to"], t["file"]) for t in trans}
|
||||
assert got == expected
|
||||
assert len(trans) == len(expected) == 154
|
||||
assert all(t["model"] == "placeholder-zoom" for t in trans)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pytest tests/test_pipeline_manifest.py::test_transitions_are_per_clip_pair_both_directions -v`
|
||||
Expected: FAIL — current builder emits 5 `scale-scale` entries with no `from`/`to`.
|
||||
|
||||
- [ ] **Step 3: Implement the per-clip-pair transitions block**
|
||||
|
||||
```python
|
||||
# simulator/build_pool_manifest.py — replace the transitions loop in build_manifest()
|
||||
transitions = []
|
||||
n = len(RING_ORDER)
|
||||
for i in range(n):
|
||||
H, L = RING_ORDER[i], RING_ORDER[(i + 1) % n] # H higher altitude, L lower
|
||||
for h in POOLS[H]:
|
||||
for l in POOLS[L]:
|
||||
fwd = f"transitions/{h}__{l}.mp4" # zoom IN (descend H->L)
|
||||
transitions.append({"from": h, "to": l, "file": fwd,
|
||||
"model": "placeholder-zoom"})
|
||||
transitions.append({"from": l, "to": h, "file": f"transitions/{h}__{l}.rev.mp4",
|
||||
"model": "placeholder-zoom"}) # zoom OUT (ascend L->H)
|
||||
return {"clips": clips, "ring": {"scales": scales, "transitions": transitions}}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `pytest tests/test_pipeline_manifest.py::test_transitions_are_per_clip_pair_both_directions -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add simulator/build_pool_manifest.py tests/test_pipeline_manifest.py
|
||||
git commit -m "feat(pipeline): manifest emits per-clip-pair morph transitions (both directions)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Bake the member-pair morph clips
|
||||
|
||||
Generalize the ffmpeg morph recipe from scale-primary→scale-primary to any member→member, and have `generate_media()` bake every adjacent member pair forward + reverse. Tested with a stubbed ffmpeg runner so the test is fast and deterministic (the real bake runs as a media step at execution time).
|
||||
|
||||
**Files:**
|
||||
- Modify: `simulator/build_pool_manifest.py:391-431` (`_make_transition`, `generate_media`)
|
||||
- Test: `tests/test_pipeline_manifest.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `RING_ORDER`, `POOLS`, `MEDIA` (Path), member base footage at `MEDIA/<clip_id>/base.mp4`.
|
||||
- Produces: `_make_transition(ff, src_clip, dst_clip) -> Path` writing `transitions/<src>__<dst>.mp4` from the two members' `base.mp4`; `_make_reverse(ff, forward)` unchanged (writes `.rev.mp4`). `generate_media()` loops every adjacent ordered member pair.
|
||||
|
||||
- [ ] **Step 1: Write the failing test (stubbed ffmpeg)**
|
||||
|
||||
```python
|
||||
# tests/test_pipeline_manifest.py (add)
|
||||
def test_generate_media_bakes_every_member_pair(monkeypatch, tmp_path):
|
||||
import simulator.build_pool_manifest as b
|
||||
calls = []
|
||||
def fake_run(args, **kw):
|
||||
# record the output path (last positional arg of each ffmpeg call)
|
||||
calls.append(args[-1])
|
||||
from pathlib import Path
|
||||
Path(args[-1]).parent.mkdir(parents=True, exist_ok=True)
|
||||
Path(args[-1]).write_bytes(b"x")
|
||||
class R: returncode = 0
|
||||
return R()
|
||||
monkeypatch.setattr(b, "MEDIA", tmp_path)
|
||||
monkeypatch.setattr(b.subprocess, "run", fake_run)
|
||||
monkeypatch.setattr(b, "_ffmpeg", lambda: "ffmpeg")
|
||||
b.generate_media()
|
||||
outs = {str(p).split("transitions/")[-1] for p in calls}
|
||||
# one forward + one reverse per adjacent member pair
|
||||
fwd = {f"{h}__{l}.mp4" for H, L in b._adjacent_edges()
|
||||
for h in b.POOLS[H] for l in b.POOLS[L]}
|
||||
rev = {f"{h}__{l}.rev.mp4" for H, L in b._adjacent_edges()
|
||||
for h in b.POOLS[H] for l in b.POOLS[L]}
|
||||
assert fwd <= outs and rev <= outs
|
||||
assert len(fwd) + len(rev) == 154
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pytest tests/test_pipeline_manifest.py::test_generate_media_bakes_every_member_pair -v`
|
||||
Expected: FAIL — `_adjacent_edges` undefined and `generate_media` loops scales, not members.
|
||||
|
||||
- [ ] **Step 3: Implement member-pair baking**
|
||||
|
||||
```python
|
||||
# simulator/build_pool_manifest.py
|
||||
|
||||
def _adjacent_edges() -> list[tuple[str, str]]:
|
||||
"""Ordered (higher, lower) altitude edges around the ring, including the wrap."""
|
||||
n = len(RING_ORDER)
|
||||
return [(RING_ORDER[i], RING_ORDER[(i + 1) % n]) for i in range(n)]
|
||||
|
||||
|
||||
def _make_transition(ff: str, src_clip: str, dst_clip: str) -> Path:
|
||||
"""A zoom/warp morph between two CLIP MEMBERS' base footage (the well-liked
|
||||
orbit-coast recipe), keyed by clip id so every adjacent member pair gets its
|
||||
own morph. Forward = zoom IN (descend src->dst)."""
|
||||
a = MEDIA / src_clip / "base.mp4"
|
||||
b = MEDIA / dst_clip / "base.mp4"
|
||||
out = MEDIA / "transitions" / f"{src_clip}__{dst_clip}.mp4"
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
norm = "trim=0:3,setpts=PTS-STARTPTS,scale=1280:720,fps=25,setsar=1,format=yuv420p"
|
||||
subprocess.run([
|
||||
ff, "-y", "-i", str(a), "-i", str(b), "-filter_complex",
|
||||
f"[0:v]{norm}[a];[1:v]{norm}[b];"
|
||||
"[a][b]xfade=transition=zoomin:duration=1.5:offset=0.75,format=yuv420p[v]",
|
||||
"-map", "[v]", "-an", str(out),
|
||||
], check=True, capture_output=True)
|
||||
return out
|
||||
|
||||
|
||||
def generate_media() -> None:
|
||||
"""Bake EVERY adjacent member-pair morph from real bases — forward (zoom in,
|
||||
descend) plus its reversed companion (zoom out, ascend)."""
|
||||
ff = _ffmpeg()
|
||||
for H, L in _adjacent_edges():
|
||||
for h in POOLS[H]:
|
||||
for l in POOLS[L]:
|
||||
fwd = _make_transition(ff, h, l)
|
||||
_make_reverse(ff, fwd)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `pytest tests/test_pipeline_manifest.py::test_generate_media_bakes_every_member_pair -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add simulator/build_pool_manifest.py tests/test_pipeline_manifest.py
|
||||
git commit -m "feat(pipeline): bake member-pair zoom morphs for every adjacent video combination"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Clip-pair transition schema + morph lookup
|
||||
|
||||
Change the `Transition` dataclass and ring loader to carry `from_clip`/`to_clip`, build a `(from, to) → file` lookup on the ring, and update the structural invariant. Touches both `player/ring.py` (dataclass + ring) and `simulator/clips.py` (parse + serialize).
|
||||
|
||||
**Files:**
|
||||
- Modify: `player/ring.py:75-113` (`Transition`, `ScaleRing.__post_init__`, add `morph_for`)
|
||||
- Modify: `simulator/clips.py:88-131` (`load_ring` transition parse, `ring_to_dict` transitions block)
|
||||
- Test: `tests/test_player_ring.py`, `tests/test_clips.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `Transition(from_clip: str, to_clip: str, file: str, model: str = "")`.
|
||||
- Produces: `ScaleRing.morph_for(from_clip: str, to_clip: str) -> str | None` — the morph file for that directed pair, or `None` if absent.
|
||||
- `ScaleRing.__post_init__`: drop the "N edges == N transitions" rule (no longer per-edge); keep "≥1 scale, single-scale ring has no transitions". Build the lookup dict in `__post_init__` (store on a private attr via `object.__setattr__`, since the dataclass is frozen).
|
||||
- `ring_to_dict` transitions become `[{"from","to","file","model"}, ...]`.
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
```python
|
||||
# tests/test_player_ring.py (add)
|
||||
from player.ring import Scale, Transition, ScaleRing
|
||||
|
||||
def _two_member_ring():
|
||||
scales = (Scale(id="cosmos", clip_id="c1", pool=("c1", "c2")),
|
||||
Scale(id="abyss", clip_id="a1", pool=("a1",)))
|
||||
# adjacency wraps: cosmos<->abyss; members c1,c2 x a1 -> 2 pairs x2 dirs = 4
|
||||
trans = (Transition("c1", "a1", "transitions/c1__a1.mp4"),
|
||||
Transition("a1", "c1", "transitions/c1__a1.rev.mp4"),
|
||||
Transition("c2", "a1", "transitions/c2__a1.mp4"),
|
||||
Transition("a1", "c2", "transitions/c2__a1.rev.mp4"))
|
||||
return ScaleRing(scales=scales, transitions=trans)
|
||||
|
||||
def test_morph_for_returns_directed_file():
|
||||
r = _two_member_ring()
|
||||
assert r.morph_for("c1", "a1") == "transitions/c1__a1.mp4"
|
||||
assert r.morph_for("a1", "c2") == "transitions/c2__a1.rev.mp4"
|
||||
assert r.morph_for("c1", "c2") is None # not an adjacent-altitude pair
|
||||
```
|
||||
|
||||
```python
|
||||
# tests/test_clips.py (add)
|
||||
def test_load_ring_parses_clip_pair_transitions(tmp_path):
|
||||
import json
|
||||
from simulator.clips import load_ring
|
||||
m = {"ring": {"scales": [{"id": "x", "clip_id": "x1", "pool": ["x1"]},
|
||||
{"id": "y", "clip_id": "y1", "pool": ["y1"]}],
|
||||
"transitions": [{"from": "x1", "to": "y1", "file": "transitions/x1__y1.mp4"},
|
||||
{"from": "y1", "to": "x1", "file": "transitions/x1__y1.rev.mp4"}]}}
|
||||
p = tmp_path / "manifest.json"; p.write_text(json.dumps(m))
|
||||
ring = load_ring(p)
|
||||
assert ring.morph_for("x1", "y1") == "transitions/x1__y1.mp4"
|
||||
assert ring.morph_for("y1", "x1") == "transitions/x1__y1.rev.mp4"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `pytest tests/test_player_ring.py::test_morph_for_returns_directed_file tests/test_clips.py::test_load_ring_parses_clip_pair_transitions -v`
|
||||
Expected: FAIL — `Transition` has no `from_clip`/`to_clip`; no `morph_for`.
|
||||
|
||||
- [ ] **Step 3: Implement the schema + lookup**
|
||||
|
||||
```python
|
||||
# player/ring.py — replace the Transition dataclass
|
||||
@dataclass(frozen=True)
|
||||
class Transition:
|
||||
"""A pre-baked zoom/warp morph between TWO specific clips in adjacent
|
||||
altitudes. `file` plays as-is (the manifest holds both directions as
|
||||
separate, directed entries — forward zoom-in and `.rev` zoom-out)."""
|
||||
from_clip: str
|
||||
to_clip: str
|
||||
file: str
|
||||
model: str = ""
|
||||
```
|
||||
|
||||
```python
|
||||
# player/ring.py — ScaleRing: relax the invariant, build the lookup
|
||||
@dataclass(frozen=True)
|
||||
class ScaleRing:
|
||||
scales: tuple[Scale, ...]
|
||||
transitions: tuple[Transition, ...]
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if len(self.scales) == 0:
|
||||
raise RingError("a ScaleRing needs at least one scale")
|
||||
lookup = {(t.from_clip, t.to_clip): t.file for t in self.transitions}
|
||||
object.__setattr__(self, "_morphs", lookup)
|
||||
|
||||
def __len__(self) -> int:
|
||||
return len(self.scales)
|
||||
|
||||
def morph_for(self, from_clip: str, to_clip: str) -> str | None:
|
||||
"""The morph file for the directed clip pair, or None if none baked."""
|
||||
return self._morphs.get((from_clip, to_clip))
|
||||
```
|
||||
|
||||
```python
|
||||
# simulator/clips.py — load_ring transition parse (replace lines 101-104)
|
||||
transitions = tuple(
|
||||
Transition(
|
||||
from_clip=t.get("from", ""),
|
||||
to_clip=t.get("to", ""),
|
||||
file=t["file"],
|
||||
model=t.get("model", ""),
|
||||
)
|
||||
for t in ring.get("transitions", [])
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
# simulator/clips.py — ring_to_dict transitions block (replace line 130)
|
||||
"transitions": [
|
||||
{"from": t.from_clip, "to": t.to_clip, "file": t.file, "model": t.model}
|
||||
for t in ring.transitions
|
||||
],
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the tests + the existing ring/clips suites**
|
||||
|
||||
Run: `pytest tests/test_player_ring.py tests/test_clips.py -v`
|
||||
Expected: PASS — fix any existing test that constructed `Transition(file=..., model=...)` positionally to the new 3-arg form (`Transition(from_clip, to_clip, file)`), and any test asserting the old `N == N` invariant.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add player/ring.py simulator/clips.py tests/test_player_ring.py tests/test_clips.py
|
||||
git commit -m "feat(ring): clip-pair Transition schema + morph_for lookup"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Resolve chained member→member morphs (pick before transition)
|
||||
|
||||
Add a pure function that, given a structural `RingMove`, the currently-shown clip, and one injected pick per step, walks the steps — choosing a destination member for each crossed altitude and resolving the morph from the *previous* clip to it — and reports the final locked clip. This is the heart of "choose the clip first, then play the matching morph," chained through each altitude.
|
||||
|
||||
**Files:**
|
||||
- Modify: `player/ring.py` (add `MorphStep`, `ResolvedMove`, `resolve_move`)
|
||||
- Test: `tests/test_player_ring.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `RingMove` (from `advance_ring`), `pick_clip_id`, `scale_at`, `ScaleRing.morph_for`.
|
||||
- Produces:
|
||||
- `MorphStep(from_clip: str, to_clip: str, file: str | None, to_index: int, blended: bool)`
|
||||
- `ResolvedMove(steps: tuple[MorphStep, ...], target_clip_id: str)`
|
||||
- `resolve_move(ring: ScaleRing, move: RingMove, from_clip_id: str, picks: tuple[float, ...]) -> ResolvedMove` — `len(picks)` must equal `len(move.steps)`; each `picks[k]` chooses the member of the scale that step `k` lands on. `from_clip` of step 0 is `from_clip_id`; each subsequent step's `from_clip` is the prior step's `to_clip`. `target_clip_id` is the last step's `to_clip` (or `from_clip_id` for an empty move).
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
```python
|
||||
# tests/test_player_ring.py (add)
|
||||
from player.ring import advance_ring, resolve_move
|
||||
|
||||
def _chain_ring():
|
||||
# 3 scales, pools of 2/1/2; descending edges cosmos->coast->abyss + wrap abyss->cosmos
|
||||
scales = (Scale("cosmos", "c1", ("c1", "c2")),
|
||||
Scale("coast", "o1", ("o1",)),
|
||||
Scale("abyss", "a1", ("a1", "a2")))
|
||||
t = []
|
||||
def pair(h, l):
|
||||
t.append(Transition(h, l, f"transitions/{h}__{l}.mp4"))
|
||||
t.append(Transition(l, h, f"transitions/{h}__{l}.rev.mp4"))
|
||||
for h in ("c1", "c2"):
|
||||
pair(h, "o1") # cosmos<->coast
|
||||
for h in ("o1",):
|
||||
pair(h, "a1"); pair(h, "a2") # coast<->abyss
|
||||
for h in ("a1", "a2"):
|
||||
pair(h, "c1"); pair(h, "c2") # abyss<->cosmos (wrap)
|
||||
return ScaleRing(scales=scales, transitions=tuple(t))
|
||||
|
||||
def test_resolve_single_step_zoom_in():
|
||||
r = _chain_ring()
|
||||
move = advance_ring(r, from_index=0, delta=1) # cosmos -> coast
|
||||
res = resolve_move(r, move, from_clip_id="c2", picks=(0.0,))
|
||||
assert len(res.steps) == 1
|
||||
s = res.steps[0]
|
||||
assert (s.from_clip, s.to_clip) == ("c2", "o1")
|
||||
assert s.file == "transitions/c2__o1.mp4"
|
||||
assert res.target_clip_id == "o1"
|
||||
|
||||
def test_resolve_chains_through_intermediate_altitude():
|
||||
r = _chain_ring()
|
||||
move = advance_ring(r, from_index=0, delta=2) # cosmos -> coast -> abyss
|
||||
# pick coast member (only o1) then abyss member: 0.6*2 -> index 1 -> a2
|
||||
res = resolve_move(r, move, from_clip_id="c1", picks=(0.0, 0.6))
|
||||
files = [s.file for s in res.steps]
|
||||
assert files == ["transitions/c1__o1.mp4", "transitions/o1__a2.mp4"]
|
||||
assert res.target_clip_id == "a2"
|
||||
|
||||
def test_resolve_zoom_out_uses_reverse_file():
|
||||
r = _chain_ring()
|
||||
move = advance_ring(r, from_index=2, delta=-1) # abyss -> coast (ascend)
|
||||
res = resolve_move(r, move, from_clip_id="a1", picks=(0.0,))
|
||||
assert res.steps[0].file == "transitions/o1__a1.rev.mp4"
|
||||
assert res.target_clip_id == "o1"
|
||||
|
||||
def test_resolve_empty_move_keeps_source():
|
||||
r = _chain_ring()
|
||||
move = advance_ring(r, from_index=0, delta=0)
|
||||
res = resolve_move(r, move, from_clip_id="c2", picks=())
|
||||
assert res.steps == () and res.target_clip_id == "c2"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `pytest tests/test_player_ring.py -k resolve -v`
|
||||
Expected: FAIL — `resolve_move` undefined.
|
||||
|
||||
- [ ] **Step 3: Implement `resolve_move`**
|
||||
|
||||
```python
|
||||
# player/ring.py (add near the bottom)
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MorphStep:
|
||||
"""One morph to play while advancing: from the clip currently shown to the
|
||||
chosen clip of the next altitude, the morph file, the index it lands on, and
|
||||
whether it plays fast (a fast-spin pass)."""
|
||||
from_clip: str
|
||||
to_clip: str
|
||||
file: str | None
|
||||
to_index: int
|
||||
blended: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResolvedMove:
|
||||
steps: tuple[MorphStep, ...]
|
||||
target_clip_id: str
|
||||
|
||||
|
||||
def resolve_move(
|
||||
ring: ScaleRing,
|
||||
move: RingMove,
|
||||
from_clip_id: str,
|
||||
picks: tuple[float, ...],
|
||||
) -> ResolvedMove:
|
||||
"""Choose the destination clip for each crossed altitude (injected `picks`,
|
||||
one per step) and resolve the morph from the previous clip to it. The morph
|
||||
is chosen AFTER the destination clip, so the footage matches what we land on.
|
||||
`file` is None if no morph was baked for a pair (renderer falls back to a
|
||||
plain cut); `target_clip_id` is the final locked clip."""
|
||||
if len(picks) != len(move.steps):
|
||||
raise ValueError(f"need one pick per step: {len(picks)} != {len(move.steps)}")
|
||||
steps: list[MorphStep] = []
|
||||
src = from_clip_id
|
||||
for st, r in zip(move.steps, picks):
|
||||
dst_scale = scale_at(ring, st.to_index)
|
||||
dst = pick_clip_id(dst_scale, r)
|
||||
steps.append(MorphStep(
|
||||
from_clip=src,
|
||||
to_clip=dst,
|
||||
file=ring.morph_for(src, dst),
|
||||
to_index=st.to_index,
|
||||
blended=st.blended,
|
||||
))
|
||||
src = dst
|
||||
return ResolvedMove(steps=tuple(steps), target_clip_id=src)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `pytest tests/test_player_ring.py -k resolve -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add player/ring.py tests/test_player_ring.py
|
||||
git commit -m "feat(ring): resolve_move picks destination clip then matching morph, chained"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Fast spin plays the whole chain quickly (no collapse)
|
||||
|
||||
The operator chose "chain through each altitude," with fast spins playing that chain *quickly* rather than collapsing to a single arrival pass. Change `advance_ring`'s fast-spin branch to keep every step but mark each `blended=True` (so the renderer plays each morph at `FAST_BLEND_RATE`) and still set `RingMove.fast=True`.
|
||||
|
||||
**Files:**
|
||||
- Modify: `player/ring.py:219-241` (fast-spin branch in `advance_ring`)
|
||||
- Test: `tests/test_player_ring.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Unchanged signature. Behavior change: a fast spin returns **all** steps (each `blended=True`), not one collapsed step.
|
||||
|
||||
- [ ] **Step 1: Write the failing test (and update any existing fast-spin assertions)**
|
||||
|
||||
```python
|
||||
# tests/test_player_ring.py (add)
|
||||
def test_fast_spin_keeps_all_steps_blended():
|
||||
r = _chain_ring() # 3-scale ring from Task 4
|
||||
move = advance_ring(r, from_index=0, delta=3, fast_spin_threshold=3)
|
||||
assert move.fast is True
|
||||
assert len(move.steps) == 3 # NOT collapsed to 1
|
||||
assert all(s.blended for s in move.steps)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pytest tests/test_player_ring.py::test_fast_spin_keeps_all_steps_blended -v`
|
||||
Expected: FAIL — current branch collapses to a single `(blended,)` step.
|
||||
|
||||
- [ ] **Step 3: Implement the chain-fast branch**
|
||||
|
||||
```python
|
||||
# player/ring.py — replace the fast-spin collapse block (lines ~219-237)
|
||||
if fast_spin_threshold >= 2 and abs(delta) >= fast_spin_threshold:
|
||||
fast_steps = tuple(
|
||||
TransitionStep(edge=s.edge, reversed=s.reversed, file=s.file,
|
||||
to_index=s.to_index, blended=True)
|
||||
for s in steps
|
||||
)
|
||||
return RingMove(from_index=start, to_index=index, steps=fast_steps,
|
||||
wrapped=wrapped, fast=True)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the full ring suite (catch the old collapse test)**
|
||||
|
||||
Run: `pytest tests/test_player_ring.py -v`
|
||||
Expected: PASS — update/replace any prior test that asserted fast spin yields exactly one step.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add player/ring.py tests/test_player_ring.py
|
||||
git commit -m "feat(ring): fast spin plays the full morph chain quickly instead of collapsing"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: API picks before transitioning and serializes the resolved chain
|
||||
|
||||
`/api/ring/advance` accepts the currently-shown clip (`from_clip_id`), draws one random pick per step, resolves the chained morphs, and returns the resolved steps + locked `target_clip_id`. Also extend `media-versions` to cover every transition file.
|
||||
|
||||
**Files:**
|
||||
- Modify: `simulator/app.py:119-121` (`RingAdvanceRequest`), `:237-252` (`api_ring_advance`), `:213-229` (`api_media_versions`)
|
||||
- Modify: `simulator/clips.py:134-160` (`ring_move_to_dict` → resolved-move serializer)
|
||||
- Test: `tests/test_simulator_api.py`
|
||||
|
||||
**Interfaces:**
|
||||
- `RingAdvanceRequest`: add `from_clip_id: str = ""`.
|
||||
- New serializer `resolved_move_to_dict(move: RingMove, resolved: ResolvedMove) -> dict`:
|
||||
```json
|
||||
{"from_index": int, "to_index": int, "wrapped": bool, "fast": bool,
|
||||
"target_clip_id": str,
|
||||
"steps": [{"from_clip": str, "to_clip": str, "file": str|null,
|
||||
"to_index": int, "blended": bool}, ...]}
|
||||
```
|
||||
- `media-versions`: iterate `t.file` for every transition (no `.rev` derivation — both directions are explicit entries now).
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
# tests/test_simulator_api.py (add)
|
||||
def test_advance_picks_then_returns_matching_morph(client):
|
||||
# client fixture serves the sample_media manifest
|
||||
r = client.post("/api/ring/advance",
|
||||
json={"from_index": 0, "delta": 1, "from_clip_id": "cosmos"})
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["to_index"] == 1
|
||||
target = body["target_clip_id"]
|
||||
step = body["steps"][-1]
|
||||
assert step["from_clip"] == "cosmos" # threaded current clip
|
||||
assert step["to_clip"] == target # morph lands on the locked clip
|
||||
# file matches the directed pair (zoom-in descend, forward file)
|
||||
assert step["file"] == f"transitions/cosmos__{target}.mp4"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `pytest tests/test_simulator_api.py::test_advance_picks_then_returns_matching_morph -v`
|
||||
Expected: FAIL — response has no `from_clip`/`to_clip`, file is the old per-edge name.
|
||||
|
||||
- [ ] **Step 3: Implement request field, endpoint, serializer, media-versions**
|
||||
|
||||
```python
|
||||
# simulator/app.py
|
||||
class RingAdvanceRequest(BaseModel):
|
||||
from_index: int = 0
|
||||
delta: int
|
||||
from_clip_id: str = ""
|
||||
```
|
||||
|
||||
```python
|
||||
# simulator/app.py — api_ring_advance
|
||||
@app.post("/api/ring/advance")
|
||||
def api_ring_advance(req: RingAdvanceRequest):
|
||||
if app.state.ring is None:
|
||||
raise HTTPException(status_code=404, detail="no scale ring in manifest")
|
||||
move = advance_ring(app.state.ring, req.from_index, req.delta,
|
||||
fast_spin_threshold=DEFAULT_FAST_SPIN_THRESHOLD)
|
||||
# Choose the destination member of each crossed altitude FIRST (one random
|
||||
# draw per step), then resolve the morph from the prior clip to it.
|
||||
src = req.from_clip_id or scale_at(app.state.ring, move.from_index).clip_id
|
||||
picks = tuple(random.random() for _ in move.steps)
|
||||
resolved = resolve_move(app.state.ring, move, src, picks)
|
||||
return resolved_move_to_dict(move, resolved)
|
||||
```
|
||||
|
||||
```python
|
||||
# simulator/clips.py — replace ring_move_to_dict with resolved_move_to_dict
|
||||
def resolved_move_to_dict(move, resolved) -> dict:
|
||||
"""JSON form of a resolved encoder move: where it lands, the locked clip, and
|
||||
the ordered chained morphs (each from the prior clip to its chosen
|
||||
destination, with the matching morph file)."""
|
||||
return {
|
||||
"from_index": move.from_index,
|
||||
"to_index": move.to_index,
|
||||
"wrapped": move.wrapped,
|
||||
"fast": move.fast,
|
||||
"target_clip_id": resolved.target_clip_id,
|
||||
"steps": [
|
||||
{"from_clip": s.from_clip, "to_clip": s.to_clip, "file": s.file,
|
||||
"to_index": s.to_index, "blended": s.blended}
|
||||
for s in resolved.steps
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
# simulator/app.py — api_media_versions: cover every transition file explicitly
|
||||
files = {c.base_file for c in app.state.clips}
|
||||
if app.state.ring is not None:
|
||||
for t in app.state.ring.transitions:
|
||||
files.add(t.file)
|
||||
```
|
||||
|
||||
Update imports in `simulator/app.py` (`resolve_move`, `scale_at` already imported; add `resolve_move`) and `simulator/clips.py` (drop `ring_move_to_dict`, add `resolved_move_to_dict`; remove the now-unused `_rev_file` if nothing else uses it — grep first).
|
||||
|
||||
- [ ] **Step 4: Run the API suite**
|
||||
|
||||
Run: `pytest tests/test_simulator_api.py -v`
|
||||
Expected: PASS — fix any test referencing the removed `ring_move_to_dict` / old response shape.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add simulator/app.py simulator/clips.py tests/test_simulator_api.py
|
||||
git commit -m "feat(api): /api/ring/advance picks clip then returns matching chained morphs"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Renderer plays the chosen morph and locks the clip (no secondary swap)
|
||||
|
||||
Rewire the frontend: send the currently-shown clip, play each resolved morph straight (the manifest already encodes direction, so drop the `reversed`/`reverseFile` logic), land + lock on `target_clip_id` with a plain (re)load, and **delete the secondary `fadeThroughBlack` swap**. Build a `(from,to)→file` lookup from the ring and preload only the morphs reachable from the current locked clip, refreshed on each landing (154 clips is too many to preload eagerly).
|
||||
|
||||
**Files:**
|
||||
- Modify: `simulator/static/app.js` — `advance` (620-652), `playTransition` (568-595), `onload`/ring load (33-57), `preloadOrder`/`preloadList`/`reverseFile` (104-131, 566), remove `fadeThroughBlack` (600-618) call.
|
||||
- Verified by: Task 8 (E2E) + manual run.
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes the Task 6 response: `move.steps[k] = {from_clip, to_clip, file, to_index, blended}`, `move.target_clip_id`.
|
||||
- New module state: `morphByPair = {}` — `"from→to"` → file string, built from `ring.transitions`.
|
||||
|
||||
- [ ] **Step 1: Build the morph lookup when the ring loads**
|
||||
|
||||
```javascript
|
||||
// simulator/static/app.js — in loadRing(), after `ring = ...` is set:
|
||||
morphByPair = {};
|
||||
if (ring && ring.transitions)
|
||||
for (const t of ring.transitions) morphByPair[`${t.from}→${t.to}`] = t.file;
|
||||
```
|
||||
|
||||
Add `let morphByPair = {};` near the other module state (around line 38). The synthesized single-clip fallback ring has no transitions → empty lookup → advance is a no-op (pool of one), which is correct.
|
||||
|
||||
- [ ] **Step 2: Simplify `playTransition` (no reverse logic)**
|
||||
|
||||
```javascript
|
||||
// simulator/static/app.js — replace playTransition
|
||||
function playTransition(file, blended) {
|
||||
// Play one resolved morph start-to-finish with alteration layers hidden. The
|
||||
// manifest already encodes direction (forward zoom-in vs `.rev` zoom-out), so
|
||||
// the file is played as-is. A fast-spin step runs at FAST_BLEND_RATE. If no
|
||||
// morph exists for the pair (file null), resolve immediately (plain cut).
|
||||
return new Promise((resolve) => {
|
||||
if (!file) return resolve();
|
||||
overlay.style.opacity = "0"; affectLayer.style.opacity = "0"; tint.style.opacity = "0";
|
||||
vid.style.filter = "none"; vid.loop = false; vid.style.opacity = "1";
|
||||
vid.src = mediaUrl(file);
|
||||
vid.playbackRate = blended ? FAST_BLEND_RATE : 1;
|
||||
let done = false;
|
||||
const finish = () => {
|
||||
if (done) return; done = true;
|
||||
vid.removeEventListener("ended", finish); vid.playbackRate = 1; resolve();
|
||||
};
|
||||
vid.addEventListener("ended", finish);
|
||||
vid.play().catch(finish);
|
||||
setTimeout(finish, blended ? 3000 : 6000);
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Rewrite `advance` — pick first (via server), play morphs, lock, no swap**
|
||||
|
||||
```javascript
|
||||
// simulator/static/app.js — replace advance
|
||||
async function advance(delta) {
|
||||
if (busy || !ring || ring.scales.length < 2) return;
|
||||
busy = true;
|
||||
try {
|
||||
const resp = await fetch("/api/ring/advance", {
|
||||
method: "POST", headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ from_index: ringIndex, delta, from_clip_id: activeClipId }),
|
||||
});
|
||||
if (!resp.ok) return;
|
||||
const move = await resp.json();
|
||||
for (const step of move.steps) {
|
||||
await playTransition(step.file, step.blended);
|
||||
}
|
||||
ringIndex = move.to_index;
|
||||
activeClipId = move.target_clip_id; // the locked clip — the morph ended on it
|
||||
currentClipId = null; // force ensureClipMedia to (re)load + loop it
|
||||
renderScaleReadout();
|
||||
} finally {
|
||||
busy = false;
|
||||
update(); // ensureClipMedia loads the locked clip; nothing re-rolls
|
||||
applyAudio();
|
||||
refreshReachablePreload();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then **delete** `fadeThroughBlack` (now unused) and `reverseFile` (the `.rev` companion is an explicit manifest entry, no longer derived). Grep to confirm no other callers before deleting.
|
||||
|
||||
- [ ] **Step 4: Preload only the reachable morphs, refreshed per landing**
|
||||
|
||||
```javascript
|
||||
// simulator/static/app.js — replace the eager transition preload in preloadOrder()/preloadList()
|
||||
// Reachable morphs from the CURRENT locked clip: to/from every member of the
|
||||
// neighboring altitudes (the only clips one detent can reach). 154 total is too
|
||||
// many to preload up front; we focus on what the next move can need.
|
||||
function reachableMorphFiles() {
|
||||
const files = new Set();
|
||||
if (!ring || ring.scales.length < 2 || !activeClipId) return files;
|
||||
const n = ring.scales.length;
|
||||
for (const sign of [1, -1]) {
|
||||
const nb = ring.scales[((ringIndex + sign) % n + n) % n];
|
||||
const members = (nb.pool || [{ clip_id: nb.clip_id }]).map((m) => m.clip_id);
|
||||
for (const m of members) {
|
||||
const a = morphByPair[`${activeClipId}→${m}`];
|
||||
const b = morphByPair[`${m}→${activeClipId}`];
|
||||
if (a) files.add(a);
|
||||
if (b) files.add(b);
|
||||
}
|
||||
}
|
||||
return files;
|
||||
}
|
||||
function refreshReachablePreload() {
|
||||
// Kick a background cache fill of the reachable morphs (bounded concurrency,
|
||||
// same cacheClip helper used for bases). Idempotent — cached files are skipped.
|
||||
for (const f of reachableMorphFiles()) cacheClip(f);
|
||||
}
|
||||
```
|
||||
|
||||
In `preloadOrder()` replace the `if (ring.transitions) for (const t ...) { add(t.file); add(reverseFile(t.file)); }` block with `for (const f of reachableMorphFiles()) add(f);`. Keep base-clip preloading (current + neighbor pools) as-is. (`cacheClip` = factor out the per-file fetch→blob step already inside the preload loop so `refreshReachablePreload` can reuse it.)
|
||||
|
||||
- [ ] **Step 5: Manual smoke + commit**
|
||||
|
||||
Run: `cd simulator && python -m uvicorn app:app --port 8011` then open `http://localhost:8011/`, turn the Altitude dial one detent each way and several detents; confirm: a zoom morph plays, it lands on a clip, and that clip **does not change** while parked (watch the Dev-Mode clip readout). No black-flash second swap.
|
||||
|
||||
```bash
|
||||
git add simulator/static/app.js
|
||||
git commit -m "feat(sim): play chosen morph then lock clip per altitude; drop secondary swap"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Playwright E2E — lock + matching morph (new tier)
|
||||
|
||||
Stand up a minimal Playwright tier (none exists) and add one spec that proves the two behaviors the operator asked for: after zooming, the displayed clip is stable (locked) until the next altitude change, and the morph that plays matches `from→to`. §9 requires an E2E browser tier for this UI change.
|
||||
|
||||
**Files:**
|
||||
- Create: `simulator/e2e/package.json`, `simulator/e2e/playwright.config.ts`, `simulator/e2e/tests/altitude-lock.spec.ts`
|
||||
- Create: `simulator/e2e/README.md` (how to run; how it boots uvicorn)
|
||||
|
||||
**Interfaces:**
|
||||
- The spec drives the real app (uvicorn on a fixed port via Playwright `webServer`), reads the Dev-Mode clip readout (enable Dev Mode via localStorage before load), and intercepts `/media/transitions/*` requests to assert the morph file played.
|
||||
|
||||
- [ ] **Step 1: Scaffold the tier (failing because no test yet)**
|
||||
|
||||
```jsonc
|
||||
// simulator/e2e/package.json
|
||||
{ "name": "hef-e2e", "private": true,
|
||||
"scripts": { "test": "playwright test" },
|
||||
"devDependencies": { "@playwright/test": "^1.45.0" } }
|
||||
```
|
||||
|
||||
```ts
|
||||
// simulator/e2e/playwright.config.ts
|
||||
import { defineConfig } from "@playwright/test";
|
||||
export default defineConfig({
|
||||
testDir: "./tests",
|
||||
webServer: {
|
||||
command: "cd .. && python -m uvicorn app:app --port 8099",
|
||||
url: "http://localhost:8099/api/clips",
|
||||
reuseExistingServer: !process.env.CI,
|
||||
timeout: 60_000,
|
||||
},
|
||||
use: { baseURL: "http://localhost:8099" },
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write the spec**
|
||||
|
||||
```ts
|
||||
// simulator/e2e/tests/altitude-lock.spec.ts
|
||||
import { test, expect } from "@playwright/test";
|
||||
|
||||
test("zoom plays a morph, lands locked, and does not change while parked", async ({ page }) => {
|
||||
const morphs: string[] = [];
|
||||
page.on("request", (r) => {
|
||||
const m = r.url().match(/\/media\/(transitions\/[^?]+)/);
|
||||
if (m) morphs.push(m[1]);
|
||||
});
|
||||
await page.addInitScript(() => localStorage.setItem("hef.devMode", "1"));
|
||||
await page.goto("/");
|
||||
const readout = page.locator("#scale-name");
|
||||
await expect(readout).toContainText("cosmos");
|
||||
|
||||
// Zoom inward one altitude (wheel down = +). A morph must play.
|
||||
await page.locator("#stage").dispatchEvent("wheel", { deltaY: 120 });
|
||||
await page.waitForTimeout(2500);
|
||||
expect(morphs.some((f) => f.startsWith("transitions/cosmos__"))).toBeTruthy();
|
||||
|
||||
// The landed clip is now locked: capture it, wait through several alteration
|
||||
// ticks, and assert it never changes.
|
||||
const landed = (await readout.textContent())!;
|
||||
await page.waitForTimeout(2000);
|
||||
expect(await readout.textContent()).toBe(landed);
|
||||
});
|
||||
|
||||
test("zooming back OUT plays a reverse morph and lands locked", async ({ page }) => {
|
||||
const morphs: string[] = [];
|
||||
page.on("request", (r) => {
|
||||
const m = r.url().match(/\/media\/(transitions\/[^?]+)/);
|
||||
if (m) morphs.push(m[1]);
|
||||
});
|
||||
await page.addInitScript(() => localStorage.setItem("hef.devMode", "1"));
|
||||
await page.goto("/");
|
||||
const readout = page.locator("#scale-name");
|
||||
await expect(readout).toContainText("cosmos");
|
||||
|
||||
// Zoom inward one altitude, then back OUT (wheel up = -). The outward move must
|
||||
// play a `.rev` morph and land back on cosmos, locked.
|
||||
await page.locator("#stage").dispatchEvent("wheel", { deltaY: 120 });
|
||||
await page.waitForTimeout(2500);
|
||||
morphs.length = 0; // only watch the zoom-out move
|
||||
await page.locator("#stage").dispatchEvent("wheel", { deltaY: -120 });
|
||||
await page.waitForTimeout(2500);
|
||||
expect(morphs.some((f) => /transitions\/cosmos__.*\.rev\.mp4/.test(f))).toBeTruthy();
|
||||
await expect(readout).toContainText("cosmos");
|
||||
|
||||
// Locked after the outward move too.
|
||||
const landed = (await readout.textContent())!;
|
||||
await page.waitForTimeout(2000);
|
||||
expect(await readout.textContent()).toBe(landed);
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Install + run**
|
||||
|
||||
Run: `cd simulator/e2e && npm install && npx playwright install chromium && npm test`
|
||||
Expected: PASS (1 test). If the wheel handler needs a different target, adjust the selector to the element bound in `onWheel` setup.
|
||||
|
||||
- [ ] **Step 4: Document the tier**
|
||||
|
||||
`simulator/e2e/README.md`: prerequisites (`npm install`, `npx playwright install chromium`), how `webServer` boots uvicorn, and `npm test`.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add simulator/e2e
|
||||
git commit -m "test(e2e): Playwright tier asserts altitude lock + matching morph"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Regenerate sample media + manifest, full suite green
|
||||
|
||||
Bake the 154 morphs into `simulator/sample_media`, rebuild the manifest, and run the entire pytest suite + the E2E spec so the shipped sample media matches the new schema.
|
||||
|
||||
**Files:**
|
||||
- Modify (generated): `simulator/sample_media/manifest.json`, `simulator/sample_media/transitions/*.mp4`
|
||||
|
||||
- [ ] **Step 1: Rebuild media + manifest**
|
||||
|
||||
Run: `cd simulator && python build_pool_manifest.py --media` (or the script's generate+manifest entrypoint — confirm its CLI flags) to bake morphs and rewrite `manifest.json`.
|
||||
Expected: `sample_media/transitions/` contains 154 files; `manifest.json` ring has 154 transition entries.
|
||||
|
||||
- [ ] **Step 2: Verify every manifest transition file exists**
|
||||
|
||||
```python
|
||||
# one-off check (or add as tests/test_catalog_integrity.py case)
|
||||
import json, pathlib
|
||||
m = json.load(open("simulator/sample_media/manifest.json"))
|
||||
base = pathlib.Path("simulator/sample_media")
|
||||
missing = [t["file"] for t in m["ring"]["transitions"] if not (base / t["file"]).exists()]
|
||||
assert not missing, missing
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run the full suite**
|
||||
|
||||
Run: `pytest -q`
|
||||
Expected: PASS (all existing + new tests; ~270+).
|
||||
|
||||
- [ ] **Step 4: Run the E2E spec against the regenerated media**
|
||||
|
||||
Run: `cd simulator/e2e && npm test`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add simulator/sample_media
|
||||
git commit -m "chore(media): regenerate 154 clip-pair morphs + manifest"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
**Spec coverage (operator directives):**
|
||||
- "Choose which clip plays in the next altitude *before* transitioning" → Task 4 (`resolve_move` picks dst then morph) + Task 6 (server draws picks before resolving).
|
||||
- "Play the appropriate morph footage" → Task 7 (renderer plays the resolved `step.file`).
|
||||
- "Morph footage for all transitions between all videos in adjacent altitudes" → Tasks 1–2 (manifest + bake 154, both directions) + Task 9 (regenerate).
|
||||
- "Video must not change after we've zoomed to a new altitude" / "lock per altitude until the altitude is changed" → Task 7 (delete secondary swap; lock `activeClipId`; reload only on altitude change) + Task 8 (E2E asserts stability while parked).
|
||||
- "Good transition for every video combination, in and out" → both directions baked (Task 1/2), reverse via explicit `.rev` entries (Task 3/4), E2E covers in + out (extend Task 8 spec with a zoom-out case if desired).
|
||||
- Multi-altitude jump chains through each altitude → Task 4 (`resolve_move` threads src→dst per step) + Task 5 (fast spin plays the chain quickly).
|
||||
|
||||
**Placeholder scan:** No TBD/TODO/"handle edge cases" — every code step shows real code; tests show real assertions.
|
||||
|
||||
**Type consistency:** `Transition(from_clip, to_clip, file, model)` (T3) used by `morph_for` (T3), `load_ring` (T3), `resolve_move` (T4), `resolved_move_to_dict` (T6). `MorphStep`/`ResolvedMove` fields (T4) match the serializer (T6) and the renderer's `step.{from_clip,to_clip,file,to_index,blended}` + `move.target_clip_id` (T7). `morphByPair` key `"from→to"` consistent in build (T7 S1) and lookup (T7 S4).
|
||||
|
||||
**Open items (resolved at the review gate by the operator):**
|
||||
1. **Storage/repo weight:** ✅ Commit the 154 short mp4s into `sample_media` (extends the existing precedent of committing the 5 + bases).
|
||||
2. **`build_pool_manifest.py` CLI:** ✅ Approach approved; confirm the actual entrypoint flag during execution (Task 9 Step 1).
|
||||
3. **Zoom-out E2E:** ✅ Added — Task 8 now covers zoom-in + lock **and** a zoom-out (`.rev` morph) + lock case.
|
||||
@@ -1,8 +1,28 @@
|
||||
# Audio + Video — separated Visual & Audio controls — design
|
||||
|
||||
**Status:** proposed → in implementation (plan `docs/superpowers/plans/2026-06-26-audio-video-separated-controls.md`, session 0020)
|
||||
**Status:** proposed → built (sessions 0020 build + 0021 revision)
|
||||
**Date:** 2026-06-26
|
||||
**Session:** 0020 (audio)
|
||||
**Session:** 0020 (audio) · 0021 (live-UI revision)
|
||||
|
||||
> **v1 build revision (session 0021, operator-driven):** the live **Audio** control
|
||||
> ships as a simple **off / soundtrack** toggle (styled like the Dev Mode switch),
|
||||
> and **Video** is the matching on/off toggle. **White-noise is deferred** with music
|
||||
> (the orthogonal Visual × Audio model and the synthesized-pink-noise machinery
|
||||
> remain in the codebase for when it's wanted). Two real-browser bugs were fixed in
|
||||
> this revision: video-off now blanks GPU-composited layers (z-index + explicit layer
|
||||
> hide), and the soundtrack now plays on Safari/iOS (audio elements unlocked
|
||||
> synchronously inside the start gesture). Sections below describe the fuller
|
||||
> off/soundtrack/white-noise design; v1 live = off/soundtrack.
|
||||
>
|
||||
> **Startup revision (session 0022):** the tap-to-start wall is **removed** — the app
|
||||
> opens behind a **"Loading Universe…"** splash (shown until all media is preloaded
|
||||
> and the experience can run smoothly), then both toggles start **off** (black +
|
||||
> silent). Autoplay is satisfied without a wall: a **single** `<audio>` element is
|
||||
> played **synchronously inside the toggle's own click** (the reliable Safari unlock),
|
||||
> replacing the A/B-crossfade-after-gesture approach that failed on real Safari. The
|
||||
> first time **Video** is turned on, **Audio** turns on with it (one flip = the full
|
||||
> experience); turning Audio on first stays audio-only. Altitude changes swap the
|
||||
> single element's source with a brief fade.
|
||||
**Extends / reframes:** [`2026-06-26-networked-control-surface-design.md`](./2026-06-26-networked-control-surface-design.md)
|
||||
§3 (control inventory) and §5 (server contract) — its bundled **Content** selector
|
||||
is split into two orthogonal controls. Everything else in that spec (server-of-truth
|
||||
|
||||
@@ -75,7 +75,7 @@ inventory is exactly what the sim exposes today:
|
||||
| Control | Sim today | Physical form (later) |
|
||||
|---|---|---|
|
||||
| **Visual** | toggle (show video on/off) | 2-position switch |
|
||||
| **Audio** | 3-way `<select>` (off / soundtrack / white noise; ·music deferred) | rotary selector switch (3 pos) |
|
||||
| **Audio** | on/off toggle (on = per-altitude soundtrack; ·white-noise & music deferred) | 2-position switch (rotary selector when more sources land) |
|
||||
| **Altitude** | endless circular dial — walks the 5 scales (cosmos→abyss), wraps | rotary encoder (endless, detented) |
|
||||
| **Left** (analytical) | slider 0–4 | knob/pot, 5 detents |
|
||||
| **Right** (dreamlike) | slider 0–4 | knob/pot, 5 detents |
|
||||
|
||||
+11
-12
@@ -2,17 +2,18 @@
|
||||
§4/§6). The single source of truth for: whether the projector shows video, and
|
||||
which audio url plays for a given (audio source, current altitude).
|
||||
|
||||
Replaces player/content.py's 7-way bundled table. White-noise is a global,
|
||||
altitude-independent bed; soundtrack follows the single Altitude dial; off is
|
||||
silence. `music` is a reserved, deferred source — not in v1."""
|
||||
Replaces player/content.py's 7-way bundled table. Soundtrack follows the single
|
||||
Altitude dial; off is silence. `white_noise` and `music` are deferred sources —
|
||||
not in v1 (the live control is a simple Audio on/off toggle where on=soundtrack)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
VISUAL_POSITIONS = frozenset({"on", "off"})
|
||||
# v1 audio sources. "music" is reserved/deferred (no assets) and intentionally absent.
|
||||
AUDIO_SOURCES = frozenset({"off", "soundtrack", "white_noise"})
|
||||
# v1 audio sources. The live UI is an Audio on/off toggle (on=soundtrack).
|
||||
# "white_noise" and "music" are deferred (no live control) and intentionally absent.
|
||||
AUDIO_SOURCES = frozenset({"off", "soundtrack"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -31,24 +32,22 @@ def resolve_visual(position: str) -> bool:
|
||||
return position == "on"
|
||||
|
||||
|
||||
def resolve_audio_url(source: str, *, scale_audio: str, noise_url: str) -> str | None:
|
||||
def resolve_audio_url(source: str, *, scale_audio: str) -> str | None:
|
||||
"""The audio url for (source, current altitude): soundtrack → the current
|
||||
scale's asset (or None if none authored); white_noise → the global bed; off →
|
||||
None. `scale_audio` is the ring scale's `audio` path (relative to /media/audio/)."""
|
||||
scale's asset (or None if none authored); off → None. `scale_audio` is the ring
|
||||
scale's `audio` path (relative to /media/audio/)."""
|
||||
if source not in AUDIO_SOURCES:
|
||||
raise ValueError(
|
||||
f"unknown audio source {source!r}; expected one of {sorted(AUDIO_SOURCES)}"
|
||||
)
|
||||
if source == "off":
|
||||
return None
|
||||
if source == "white_noise":
|
||||
return noise_url
|
||||
# soundtrack — couple to the current altitude's asset
|
||||
return f"/media/audio/{scale_audio}" if scale_audio else None
|
||||
|
||||
|
||||
def resolve_audio(source: str, *, scale_audio: str, noise_url: str) -> AudioResolution:
|
||||
def resolve_audio(source: str, *, scale_audio: str) -> AudioResolution:
|
||||
"""The full render.audio view: source, resolved url, and whether it follows the
|
||||
Altitude dial (true only for soundtrack)."""
|
||||
url = resolve_audio_url(source, scale_audio=scale_audio, noise_url=noise_url)
|
||||
url = resolve_audio_url(source, scale_audio=scale_audio)
|
||||
return AudioResolution(source=source, url=url, altitude_coupled=(source == "soundtrack"))
|
||||
|
||||
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
This is the serial-contract payload shared with sub-project 4 (firmware). It
|
||||
models the full panel from the audio spec §3: a Visual toggle (on/off) + an
|
||||
Audio source selector (off/soundtrack/white_noise), the four experience knobs
|
||||
Audio source selector (off/soundtrack; white-noise/music deferred), the four experience knobs
|
||||
(0..4), and the two intensity levels (volume, brightness). The wire framing
|
||||
itself is the separate 3<->4 serial contract; this module is the *decoded* form.
|
||||
"""
|
||||
|
||||
+2
-2
@@ -36,12 +36,12 @@ class TransitionKind:
|
||||
class Playback:
|
||||
"""What should currently be playing. Visual and audio are orthogonal (audio
|
||||
spec §2): `video` is whether the projector shows footage, `audio_source` is
|
||||
the independent audio dial (off / soundtrack / white_noise)."""
|
||||
the independent audio dial (off / soundtrack)."""
|
||||
|
||||
clip_id: Optional[str] # None = black walls
|
||||
plan: Optional[RenderPlan] # None when black
|
||||
video: bool # whether footage is shown (Visual on/off)
|
||||
audio_source: str # the Audio dial source (off/soundtrack/white_noise)
|
||||
audio_source: str # the Audio dial source (off/soundtrack)
|
||||
volume: int
|
||||
brightness: int
|
||||
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
# Session 0020.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-26T06-47 (PST)
|
||||
> End: 2026-06-26T07-58 (PST)
|
||||
> Type: writing-plans (extended into execution — operator approved "then build it" at the plan gate)
|
||||
> Posture: careful (execution proceeded under explicit operator approval)
|
||||
> Claude-Session: e0cc053a-41fa-416d-a4f5-493f14984e75
|
||||
> Anchor: design docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md (R2a)
|
||||
> Status: FINALIZED
|
||||
|
||||
## Launch prompt
|
||||
|
||||
> Write the implementation plan for the "Audio + Video — separated Visual & Audio
|
||||
> controls" feature, then build it. (Spec:
|
||||
> docs/superpowers/specs/2026-06-26-audio-video-separated-controls-design.md.)
|
||||
> Split the bundled 7-way Content selector into orthogonal Visual (on/off) ×
|
||||
> Audio (off / soundtrack / white noise). Audio follows the single Altitude dial;
|
||||
> white noise synthesized (ffmpeg anoisesrc). Reframe the (unbuilt) networked-
|
||||
> control-surface spec §3/§5 as a clean pre-implementation edit. Honor the testing
|
||||
> tiers; resolve the three §12 flagged decisions.
|
||||
|
||||
## Pre-state
|
||||
|
||||
- `main` clean at 99a751c. Two stale `--INPROGRESS` placeholders (0015, 0017) from
|
||||
2026-06-24/25 — prior dead sessions, not live; left untouched.
|
||||
- The networked-control-surface architecture (SSE / `/remote` / renderer split) was
|
||||
**unbuilt** — the sim is a single fused page (`simulator/static/index.html`) talking
|
||||
to FastAPI via `POST /api/alteration` (knobs) + `POST /api/ring/advance` (altitude).
|
||||
- Audio assets (5 sourced ambiences) sat gitignored under `simulator/sample_media/audio/<scale>/`,
|
||||
unreferenced by the manifest, unserved; no `<audio>` element anywhere; video muted.
|
||||
|
||||
## Session arc (turn by turn)
|
||||
|
||||
1. **Session gate → writing-plans.** Classified the two-part "write the plan, then
|
||||
build it" prompt as the build track's entry; claimed session **0020** (`--type
|
||||
writing-plans`, careful posture). Baseline clean; created branch `session-0020`.
|
||||
2. **Scoping.** Read both specs + the audio candidate pool; dispatched an Explore
|
||||
agent to map the current simulator. Key finding: the control-surface architecture
|
||||
is unbuilt, so the audio spec's §6 server contract had no host. **Resolved by the
|
||||
standing simulator-first directive:** build the audio feature into the current
|
||||
single-page sim (ships sound now); reframe the control-surface spec on paper only;
|
||||
do NOT build the SSE/`/remote`/renderer split (it's "v1 hardware," deferred). This
|
||||
build is the "Content step" the audio spec §11 anticipates.
|
||||
3. **Plan.** Wrote `docs/superpowers/plans/2026-06-26-audio-video-separated-controls.md`
|
||||
(10 tasks, all testing tiers). Committed + pushed; presented at the careful-posture
|
||||
plan gate. Operator answered **"Approve — build it."** §12 decisions resolved:
|
||||
audio=live (~0.6 s crossfade), noise=pink, autoplay=tap-to-start overlay; asset
|
||||
params −18 LUFS / mp3 / loop mirrors the proven video crossfade recipe.
|
||||
4. **Build (TDD, commit per task):**
|
||||
- T1–2: pure ffmpeg audio builders (`tools/pipeline/audio_ops.py`) + runner
|
||||
(`audio_run.py`) + `simulator/build_audio_media.py`; produced the 5 normalized
|
||||
loops + pink-noise bed (gitignored).
|
||||
- T3: `Scale.audio` manifest field + read/emit/serialize + `SCALE_AUDIO`; manifest
|
||||
regenerated (clean diff — only 5 audio lines).
|
||||
- T4–6: pure `player/audio.py` resolver; retired `player/content.py` (+ test);
|
||||
reframed `player/controls.py` + `player/state.py`; `/api/alteration` validates
|
||||
visual+audio and returns `render.video` + server-resolved `render.audio.url` from
|
||||
`altitude_index`. Full Python suite green.
|
||||
- T7–8: client — Content `<select>` → Visual toggle + Audio selector; A/B `<audio>`
|
||||
gain-crossfade layer; tap-to-start gesture. Smoke-tested page + media (200s).
|
||||
- T9: reframed the networked-control-surface spec §3/§4/§5/§6; recorded §12
|
||||
resolutions; ROADMAP note.
|
||||
- T10: Playwright E2E (`[e2e]` extra). Installed Chromium and ran it for real —
|
||||
surfaced + fixed three client issues (see Cut state). **3 E2E pass**, full suite
|
||||
**288 passed, 2 skipped**.
|
||||
5. **Verify + ship.** Screenshotted the running app (cosmos plays, soundtrack audibly
|
||||
playing `cosmos/pillars.loop.mp3`, split controls visible, zero JS errors). Merged
|
||||
`session-0020` → `main` (`--no-ff`) and pushed. (`gh` PR create denied — Gitea host;
|
||||
merged locally with branch→merge discipline.)
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
_No low-confidence autonomous calls. The scope decision (current-sim vs full control
|
||||
surface) was approved at the plan gate; the three §12 resolutions were operator-
|
||||
delegated by the launch prompt ("resolve the three flagged decisions"); asset params
|
||||
(−18 LUFS, mp3, pink) follow the spec's stated direction._
|
||||
|
||||
## Cut state
|
||||
|
||||
- `main` @ `aa76cd1` (merge of session-0020), clean, synced with origin.
|
||||
- **288 tests + 3 Playwright E2E pass; 2 skipped** (opt-in media integration).
|
||||
- Three client robustness fixes the E2E drove out (worth remembering):
|
||||
- `fadeVolume` uses `setInterval`, not `requestAnimationFrame` (rAF stalls when the
|
||||
tab is backgrounded → the gain ramp hung in headless).
|
||||
- The start gesture drops per-element "priming": a single page click grants Chrome
|
||||
autoplay document-wide; priming srcless `<audio>` only hung the handler / polluted
|
||||
the paused-state.
|
||||
- `applyAudio` does not `await` `play()` (awaiting a srcless/buffering play can hang).
|
||||
- Plan archival: this app has **no content repo** (`CONTENT_REMOTE` empty), so the plan
|
||||
was not archived to a `plans/` collection — it lives in-repo at
|
||||
`docs/superpowers/plans/` and is already on `main`.
|
||||
- §9 pipeline: localhost + E2E stage is **green**; there is no PPE/prod infra for this
|
||||
simulator-first art app (deferred with the Pi renderer), so no further ship stage ran.
|
||||
|
||||
## Operator plate (what's true now / next)
|
||||
|
||||
- The simulator has **sound**: per-altitude soundtracks that follow the Altitude dial +
|
||||
a pink white-noise bed, behind orthogonal **Visual** × **Audio** controls. The unheard
|
||||
audio wants a by-ear review/tune.
|
||||
- Deferred (already roadmap-tracked, not re-parked in memory): the **music** layer
|
||||
(reserved Audio position, no assets), the networked-control-surface SSE/`/remote`
|
||||
build, i2v ring transitions, the Pi renderer.
|
||||
|
||||
## Next /goal
|
||||
|
||||
```
|
||||
/goal review/tune the new audio by ear in the sim (per-altitude soundtracks + pink white-noise + the ~0.6s A/B crossfade; tap-to-start), then source/compose the deferred per-altitude music layer (the reserved Audio position), per docs/audio-candidate-pool.md
|
||||
```
|
||||
@@ -0,0 +1,88 @@
|
||||
# Session 0021.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-26T08-12 (PST)
|
||||
> End: 2026-06-26T08-20 (PST)
|
||||
> Type: executing-plans (operator-driven hotfix/revision of session 0020's audio feature)
|
||||
> Posture: yolo
|
||||
> Claude-Session: e0cc053a-41fa-416d-a4f5-493f14984e75
|
||||
> Status: FINALIZED
|
||||
|
||||
## Launch prompt
|
||||
|
||||
> Show video unchecked doesn't turn off the video. Make "show video" just "Video"
|
||||
> and have a toggle like "Dev Mode" has. I can't hear any audio when "soundtrack
|
||||
> (follows altitude)" is selected. Let's just have Audio as off/on like Video for
|
||||
> now and get rid of white noise.
|
||||
|
||||
## Pre-state
|
||||
|
||||
- Session 0020 had just merged the audio feature to `main` (Visual×Audio split,
|
||||
per-altitude soundtracks, pink white-noise, A/B crossfade). The operator tested it
|
||||
live and reported three issues.
|
||||
|
||||
## Session arc
|
||||
|
||||
1. **Diagnosis (the key work).** Both reported bugs were reproduced *only* in the
|
||||
operator's real browser — neither appeared in headless Chromium **or** WebKit
|
||||
(installed `playwright install webkit` to test Safari's engine). Root causes:
|
||||
- **Video-off doesn't blank:** `#vid` (`<video>`) and `#paint` (WebGL `<canvas>`)
|
||||
are GPU-composited layers; the `#black` cover had *auto* z-index, so on a real
|
||||
GPU (Safari/Chrome) the video layer can composite *above* it despite DOM order.
|
||||
Headless software-raster hides this.
|
||||
- **No soundtrack audio:** real Safari/iOS unlocks an `<audio>` element only when
|
||||
`play()` is called *synchronously inside* the user gesture; the 0020 gesture
|
||||
played in an async continuation (after `await fetch`), which Chrome allows but
|
||||
Safari blocks. Headless WebKit relaxes autoplay, so it passed.
|
||||
This is the session's main lesson: **Playwright headless relaxes both GPU
|
||||
compositing and autoplay**, so the 0020 E2E couldn't have caught either — real
|
||||
device / by-eye review is required.
|
||||
2. **Fixes (branch `session-0021`):**
|
||||
- **Video** + **Audio** are now Dev-Mode-style on/off toggles (`.dev-switch`),
|
||||
wired on `change`. Audio on → soundtrack; **white-noise deferred** (removed from
|
||||
the live control; `AUDIO_SOURCES={off,soundtrack}`; the pure pink-noise
|
||||
machinery + tests stay for later).
|
||||
- Video-off: `.black` gets `z-index:50` + `translateZ(0)` (own layer) **and** the
|
||||
video layers are set `opacity:0` on video-off (belt-and-suspenders blanking).
|
||||
- Safari audio: the tap-to-start gesture now primes both A/B `<audio>` elements
|
||||
**synchronously in-gesture** (set src, vol 0, `play().then(pause)`), unlocking
|
||||
the later async crossfades.
|
||||
- Server: `player/audio.py` resolver + `controls.py`/`state.py` + `/api/alteration`
|
||||
drop white-noise; `build_audio_media.py` stops emitting the noise bed.
|
||||
3. **Verify + ship.** Updated unit/contract/E2E tests; rewrote the 3 E2E for the new
|
||||
toggles (click the visible `.dev-switch-track`; wait for the opacity transition).
|
||||
**288 passed, 2 skipped.** Re-verified in **both Chromium and WebKit**: Audio toggle
|
||||
plays `cosmos/pillars.loop.mp3`, video-off sets vid+paint opacity 0 with `#black`
|
||||
shown, zero JS errors; screenshotted the new toggle UI. Merged `session-0021` →
|
||||
`main` and pushed. (Also untracked a stray `docs/.threads/*.json` editor artifact
|
||||
swept in by `git add -A`, and gitignored `.threads/`.)
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
_No low-confidence calls. All three changes were explicit operator requests; the
|
||||
root-cause fixes (GPU z-index/layer-hide; Safari in-gesture unlock) are the standard
|
||||
remedies for these well-known browser behaviors._
|
||||
|
||||
## Cut state
|
||||
|
||||
- `main` @ `fddb6d6` (merge of session-0021), clean, pushed.
|
||||
- **288 tests + 3 Playwright E2E pass** (2 skipped: opt-in media integration).
|
||||
- White-noise is deferred, not deleted: `tools/pipeline/audio_ops.white_noise_args` +
|
||||
`audio_run.generate_white_noise` + their tests remain; only the live control + the
|
||||
build-script emission were removed.
|
||||
- **Unvalidated by this session:** real-device audio (Safari autoplay) and real-GPU
|
||||
video-off — headless can't exercise either; needs the operator's by-ear/by-eye pass.
|
||||
|
||||
## Operator plate
|
||||
|
||||
- Video + Audio are simple on/off toggles (Dev-Mode style). Audio on = the current
|
||||
altitude's soundtrack. The two real-browser bugs should now be fixed — but please
|
||||
confirm on a real device (especially iPad/Safari).
|
||||
- Deferred: white-noise (easy to re-add as a 3rd Audio position), the per-altitude
|
||||
music layer, the networked-control-surface build, Pi renderer.
|
||||
|
||||
## Next /goal
|
||||
|
||||
```
|
||||
/goal by-ear/by-eye review the audio + Video/Audio toggles on a REAL device (esp. iPad/Safari — headless can't validate Safari autoplay or GPU compositing), then source/compose the deferred per-altitude music layer per docs/audio-candidate-pool.md (or re-enable white-noise as a 3rd Audio position if wanted)
|
||||
```
|
||||
@@ -0,0 +1,79 @@
|
||||
# Session 0022.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-26T08-40 (PST)
|
||||
> End: 2026-06-26T08-50 (PST)
|
||||
> Type: executing-plans (operator-driven follow-up to the audio feature)
|
||||
> Posture: yolo
|
||||
> Claude-Session: e0cc053a-41fa-416d-a4f5-493f14984e75
|
||||
> Status: FINALIZED
|
||||
|
||||
## Launch prompt
|
||||
|
||||
> (1) "It shouldn't say 'Tap to Start Sound'… it can start with both sound and video
|
||||
> off but immediately go to the app. We should have a 'Loading Universe…' loading
|
||||
> screen until all media has been downloaded and the experience is ready to run."
|
||||
> (2) "I still can't hear the sound, even in private mode."
|
||||
> (3) "The first time video is turned on, also turn on audio. If audio is turned on
|
||||
> before video after the app loads, just audio should turn on."
|
||||
|
||||
## Pre-state
|
||||
|
||||
- Sessions 0020/0021 had shipped the audio feature + a 0021 "Safari fix" (tap-to-start
|
||||
overlay that primed A/B `<audio>` elements). The operator reported audio STILL
|
||||
silent on their real browser (even in private mode), plus the UX requests above.
|
||||
|
||||
## Session arc
|
||||
|
||||
1. **Why the 0021 fix failed.** It primed *srcless* A/B elements in the tap gesture
|
||||
and played the real audio in an *async continuation* (after `await fetch`). Safari
|
||||
unlocks an element only when `play()` runs **synchronously inside** the gesture and
|
||||
on an element **with a src** — so neither condition was met. Headless WebKit relaxes
|
||||
autoplay, which is why automated tests never caught it.
|
||||
2. **Redesign (client-only, branch `session-0022`):**
|
||||
- **Single `<audio id="aud">`** element (dropped A/B). It's played **synchronously
|
||||
inside the toggle's own `change` handler** — the canonical Safari unlock. Altitude
|
||||
changes reuse the now-unlocked element with a brief fade-swap.
|
||||
- **"Loading Universe…" splash** (`#loading`, in the HTML so it shows pre-JS): `main()`
|
||||
now `await`s `preloadAllMedia()` (driving a progress bar), then `hideLoading()`.
|
||||
Replaces the tap-to-start wall.
|
||||
- **Both toggles default OFF** (black + silent at start) — no autoplay needed until a
|
||||
toggle is flipped.
|
||||
- **First Video-on couples Audio:** the Video `change` handler, on the first on,
|
||||
sets `#audio` checked and plays — in the same gesture (so it unlocks on Safari).
|
||||
Audio toggled on first stays audio-only.
|
||||
- Server contract unchanged (`{off,soundtrack}`); the client now drives playback
|
||||
directly from the ring's `scale.audio`, ignoring `render.audio`.
|
||||
3. **Verify + ship.** Rewrote the 5 E2E for the new flow (single `#aud`, no gesture,
|
||||
loading wait, the coupling, video-off blanking). Caught a test bug — `#black.hidden`
|
||||
is `display:none`, so `wait_for_selector` default (visible) timed out; switched to
|
||||
`state="attached"`. **290 passed, 2 skipped; 5 E2E pass.** Verified in headless
|
||||
**Chromium AND WebKit**: first-Video-on couples audio, soundtrack plays, video shows,
|
||||
zero JS errors; screenshotted the splash + running app. Merged to `main`.
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
_No low-confidence calls — all three were explicit operator requests. The
|
||||
single-element synchronous-in-gesture play is the standard Safari remedy._
|
||||
|
||||
## Cut state
|
||||
|
||||
- `main` @ `4ac39c9`, clean, pushed. 290 tests + 5 Playwright E2E pass.
|
||||
- **Honest caveat — still unverifiable headlessly:** every engine Playwright ships
|
||||
(Chromium, WebKit) relaxes autoplay, so I cannot *prove* real-Safari/iOS now plays.
|
||||
The fix is the canonical pattern (sync `play()` in the gesture, real src) and is a
|
||||
genuine change from 0021's broken approach — but it needs the operator's real-device ear.
|
||||
|
||||
## Operator plate
|
||||
|
||||
- No tap wall: "Loading Universe…" → app with both toggles off (black/silent). Flip
|
||||
**Video** → video + audio together; flip **Audio** alone → audio only.
|
||||
- If sound is STILL silent on the real device, that points to something beyond autoplay
|
||||
(codec/MIME, file, or element error) — next session should capture `aud.error` and the
|
||||
exact browser.
|
||||
|
||||
## Next /goal
|
||||
|
||||
```
|
||||
/goal confirm on a REAL device (esp. iPad/Safari) that audio is now audible when Video/Audio is toggled on and the "Loading Universe…" → app flow feels right; if still silent, capture aud.error + the exact browser. Then the deferred per-altitude music layer per docs/audio-candidate-pool.md
|
||||
```
|
||||
@@ -55,5 +55,14 @@
|
||||
},
|
||||
"0019": {
|
||||
"title": ""
|
||||
},
|
||||
"0020": {
|
||||
"title": ""
|
||||
},
|
||||
"0021": {
|
||||
"title": ""
|
||||
},
|
||||
"0022": {
|
||||
"title": ""
|
||||
}
|
||||
}
|
||||
|
||||
+1
-4
@@ -39,9 +39,6 @@ STATIC_DIR = Path(__file__).parent / "static"
|
||||
MEDIA_DIR = Path(__file__).parent / "sample_media"
|
||||
DEFAULT_MANIFEST = MEDIA_DIR / "manifest.json"
|
||||
|
||||
# The global white-noise bed (audio spec §5.2): synthesized, altitude-independent.
|
||||
NOISE_URL = "/media/audio/noise/pink.mp3"
|
||||
|
||||
# Per-process boot token: changes on every server restart so the dev live-reload
|
||||
# (below) also fires when Python code changes (which needs a restart), not just
|
||||
# when a static asset's mtime changes.
|
||||
@@ -190,7 +187,7 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
|
||||
scale_audio = ""
|
||||
if app.state.ring is not None:
|
||||
scale_audio = scale_at(app.state.ring, req.altitude_index).audio
|
||||
audio = resolve_audio(c.audio, scale_audio=scale_audio, noise_url=NOISE_URL)
|
||||
audio = resolve_audio(c.audio, scale_audio=scale_audio)
|
||||
return {
|
||||
"plan": render_plan_to_dict(plan),
|
||||
"render": {
|
||||
|
||||
@@ -1,18 +1,22 @@
|
||||
"""Production pass for the 5 per-altitude soundtracks + the white-noise bed
|
||||
(audio spec §5.3). The audio analogue of build_pool_manifest.py --media: read the
|
||||
sourced ambience clips (re-downloadable per docs/audio-candidate-pool.md), make
|
||||
each a seamless loudness-normalized loop, and synthesize the pink-noise bed.
|
||||
"""Production pass for the 5 per-altitude soundtracks (audio spec §5.3). The audio
|
||||
analogue of build_pool_manifest.py --media: read the sourced ambience clips
|
||||
(re-downloadable per docs/audio-candidate-pool.md), and make each a seamless
|
||||
loudness-normalized loop.
|
||||
|
||||
Media is gitignored; this script is the reproducible record. Outputs land under
|
||||
simulator/sample_media/audio/<scale>/<name>.loop.mp3 and audio/noise/pink.mp3,
|
||||
served at /media/audio/... Run: python simulator/build_audio_media.py
|
||||
simulator/sample_media/audio/<scale>/<name>.loop.mp3, served at /media/audio/...
|
||||
Run: python simulator/build_audio_media.py
|
||||
|
||||
(White-noise is deferred — the live Audio control is a simple on/off soundtrack
|
||||
toggle. `tools.pipeline.audio_run.generate_white_noise` remains for when it's
|
||||
wanted, but no noise bed is built here.)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from tools.pipeline.audio_run import generate_white_noise, process_soundtrack
|
||||
from tools.pipeline.audio_run import process_soundtrack
|
||||
|
||||
AUDIO = Path(__file__).parent / "sample_media" / "audio"
|
||||
|
||||
@@ -35,8 +39,6 @@ def main() -> None:
|
||||
continue
|
||||
dst = process_soundtrack(src, AUDIO / scale / out)
|
||||
print(f" produced {dst.relative_to(AUDIO.parent)}")
|
||||
noise = generate_white_noise(AUDIO / "noise" / "pink.mp3")
|
||||
print(f" produced {noise.relative_to(AUDIO.parent)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+144
-65
@@ -141,7 +141,10 @@ async function preloadAllMedia(concurrency = 4) {
|
||||
let done = 0;
|
||||
const total = files.length;
|
||||
const chip = preloadChip();
|
||||
const tick = () => { if (chip) chip.textContent = `⬇ caching clips ${done}/${total}`; };
|
||||
const tick = () => {
|
||||
if (chip) chip.textContent = `⬇ caching clips ${done}/${total}`;
|
||||
setLoadingProgress(done, total); // drive the "Loading Universe…" bar
|
||||
};
|
||||
tick();
|
||||
let i = 0;
|
||||
async function worker() {
|
||||
@@ -495,14 +498,18 @@ function renderScaleReadout() {
|
||||
}
|
||||
|
||||
function controls() {
|
||||
// Visual (on/off) and Audio (off/soundtrack/white_noise) are orthogonal dials
|
||||
// (audio spec §2). One bipolar Mood dial (-4 dark .. +4 light) maps onto the
|
||||
// engine's two poles. Calibration gains are pinned to 1.0, so the simulator
|
||||
// sends no calibration — the server uses DEFAULT_CALIBRATION.
|
||||
// Video (on/off) and Audio (on/off) are orthogonal toggles (audio spec §2).
|
||||
// Audio on = the per-altitude soundtrack (white-noise is deferred). One bipolar
|
||||
// Mood dial (-4 dark .. +4 light) maps onto the engine's two poles. Calibration
|
||||
// gains are pinned to 1.0, so the simulator sends no calibration.
|
||||
const mood = +$("mood").value;
|
||||
return {
|
||||
visual: $("visual").checked ? "on" : "off",
|
||||
audio: $("audio").value,
|
||||
audio: $("audio").checked ? "soundtrack" : "off",
|
||||
// Back-compat: a server process predating the visual/audio split still requires
|
||||
// a 7-way `content` and would 422 the new payload. Send both — the current
|
||||
// server ignores `content`, an old one ignores visual/audio.
|
||||
content: $("visual").checked ? "video" : "off",
|
||||
left: +$("left").value, right: +$("right").value,
|
||||
dark: mood < 0 ? -mood : 0, light: mood > 0 ? mood : 0,
|
||||
volume: 2, brightness: 2,
|
||||
@@ -519,9 +526,21 @@ async function update() {
|
||||
if (!resp.ok) { readout.textContent = "invalid: " + resp.status; return; }
|
||||
const data = await resp.json();
|
||||
readout.textContent = JSON.stringify(data, null, 2);
|
||||
applyAudio(data.render.audio); // reconcile the audio layer
|
||||
if (!data.render.video.shown) { black.style.opacity = "1"; black.classList.remove("hidden"); return; }
|
||||
// Tolerate a stale server that still returns {content:{video}} instead of the
|
||||
// {render:{video:{shown}}} (visual/audio-split) shape, so video works either way.
|
||||
const videoShown = data.render ? !!(data.render.video && data.render.video.shown)
|
||||
: !!(data.content && data.content.video);
|
||||
if (!videoShown) {
|
||||
// Blank the screen. Cover with #black AND hide the video layers themselves —
|
||||
// the <video>/<canvas> are GPU-composited and can otherwise show through the
|
||||
// overlay on a real GPU (Safari/Chrome).
|
||||
black.style.opacity = "1"; black.classList.remove("hidden");
|
||||
vid.style.opacity = "0"; paint.style.opacity = "0";
|
||||
return;
|
||||
}
|
||||
black.classList.add("hidden");
|
||||
vid.style.opacity = ""; paint.style.opacity = ""; // restore the video layers
|
||||
|
||||
try {
|
||||
ensureClipMedia();
|
||||
applyVideoLook(data.plan.grade.tone, data.plan.dream.intensity);
|
||||
@@ -628,6 +647,7 @@ async function advance(delta) {
|
||||
} finally {
|
||||
busy = false;
|
||||
update();
|
||||
applyAudio(); // soundtrack follows the Altitude dial (crossfade to the new scale)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -900,89 +920,148 @@ function devLiveReload() {
|
||||
}, 1000);
|
||||
}
|
||||
|
||||
// --- Audio layer: two <audio> elements, gain-crossfaded (audio spec §7/§8) ---
|
||||
const audA = $("audA"), audB = $("audB");
|
||||
let audActive = audA, audIdle = audB; // which element is currently audible
|
||||
let audUrl = null; // the url currently playing (null = silence)
|
||||
let audioReady = false; // unlocked after the first user gesture (autoplay policy)
|
||||
const XFADE_MS = 600; // ~0.6s gain crossfade — "live", no busy ack (§7)
|
||||
// --- Audio layer: ONE <audio> element, started in-gesture (Safari-safe) ---
|
||||
// Real Safari/iOS only "unlocks" an <audio> element when play() is called
|
||||
// SYNCHRONOUSLY inside a user gesture (Chrome is lenient; Safari is not). So we use
|
||||
// a single element and start it directly from the Video/Audio toggle's click — not
|
||||
// via the async server roundtrip; once unlocked it can be re-played programmatically
|
||||
// (e.g. on an altitude change). audio on = the current scale's soundtrack.
|
||||
const aud = $("aud");
|
||||
let audUrl = null; // soundtrack url currently loaded (null = silent)
|
||||
let audLastErr = ""; // last play() rejection / element error (for the readout)
|
||||
const FADE_MS = 500;
|
||||
|
||||
function fadeVolume(el, to, ms) {
|
||||
// --- Live audio status readout (diagnostic) — so a "no sound" report is legible ---
|
||||
const AUD_ERR = ["", "aborted", "network", "decode", "format-not-supported"];
|
||||
function audioStatusText() {
|
||||
if (!$("audio").checked) return "audio: off";
|
||||
const s = ring && ring.scales[ringIndex];
|
||||
const url = soundtrackUrl();
|
||||
const head = `ON · scale=${s ? s.id : "?"} · ring=${serverRing ? "server" : "SYNTH"} · url=${url || "NONE"}`;
|
||||
if (!url) return "audio: " + head + " ← no soundtrack URL for this scale";
|
||||
if (aud.error) return `audio: ${head} · MEDIA ERROR ${aud.error.code} (${AUD_ERR[aud.error.code] || "?"})`;
|
||||
if (audLastErr) return `audio: ${head} · ${audLastErr}`;
|
||||
if (aud.paused) return `audio: ${head} · PAUSED rs=${aud.readyState}`;
|
||||
return `audio: ${head} · ▶ ${aud.currentTime.toFixed(1)}s vol=${Math.round(aud.volume * 100)}% rs=${aud.readyState}`;
|
||||
}
|
||||
function updateAudioStatus() {
|
||||
const el = $("audio-status");
|
||||
if (!el) return;
|
||||
el.textContent = audioStatusText();
|
||||
const playing = $("audio").checked && !!soundtrackUrl() && !aud.paused && !aud.error && !audLastErr;
|
||||
el.classList.toggle("playing", playing);
|
||||
el.classList.toggle("blocked", $("audio").checked && (!soundtrackUrl() || !!aud.error || !!audLastErr));
|
||||
}
|
||||
aud.addEventListener("error", updateAudioStatus);
|
||||
setInterval(updateAudioStatus, 400);
|
||||
|
||||
function fadeVolume(el, to, ms, done) {
|
||||
// setInterval (not requestAnimationFrame): rAF throttles/pauses when the tab is
|
||||
// backgrounded, which would stall the gain ramp; a timer fires regardless.
|
||||
const from = el.volume, steps = Math.max(1, Math.round(ms / 30));
|
||||
let i = 0;
|
||||
return new Promise((resolve) => {
|
||||
const iv = setInterval(() => {
|
||||
i += 1;
|
||||
const k = Math.min(1, i / steps);
|
||||
el.volume = Math.min(1, Math.max(0, from + (to - from) * k));
|
||||
if (k >= 1) { clearInterval(iv); resolve(); }
|
||||
}, 30);
|
||||
});
|
||||
const iv = setInterval(() => {
|
||||
i += 1;
|
||||
const k = Math.min(1, i / steps);
|
||||
el.volume = Math.min(1, Math.max(0, from + (to - from) * k));
|
||||
if (k >= 1) { clearInterval(iv); if (done) done(); }
|
||||
}, 30);
|
||||
}
|
||||
|
||||
// Reconcile the audio layer to render.audio: crossfade the active element to the
|
||||
// new url (or fade to silence for off). White-noise and soundtrack are the same
|
||||
// mechanism — only the url differs; an unchanged url is a no-op so altitude
|
||||
// re-rolls on the same scale don't restart the bed.
|
||||
async function applyAudio(audio) {
|
||||
const url = audio ? audio.url : null;
|
||||
// Fallback per-scale soundtrack paths, used when the ring from /api/ring lacks the
|
||||
// `audio` field (e.g. a server process started before the audio field existed).
|
||||
// Mirrors simulator/build_pool_manifest.py SCALE_AUDIO — the 5 scales + files are
|
||||
// fixed, committed facts, so this keeps audio working without a server restart.
|
||||
const SCALE_AUDIO_FALLBACK = {
|
||||
cosmos: "cosmos/pillars.loop.mp3",
|
||||
orbit: "orbit/spaceamb.loop.mp3",
|
||||
coast: "coast/waves.loop.mp3",
|
||||
reef: "reef/soundscape.loop.mp3",
|
||||
abyss: "abyss/whale.loop.mp3",
|
||||
};
|
||||
|
||||
// The current altitude's soundtrack url: the ring's `audio` field, else the
|
||||
// scale-id fallback above.
|
||||
function soundtrackUrl() {
|
||||
const s = ring && ring.scales[ringIndex];
|
||||
if (!s) return null;
|
||||
const a = s.audio || SCALE_AUDIO_FALLBACK[s.id];
|
||||
return a ? "/media/audio/" + a : null;
|
||||
}
|
||||
|
||||
// Load `url` into the single element and fade it in. Call this SYNCHRONOUSLY from
|
||||
// the toggle's click the first time so play() is allowed; later calls reuse the
|
||||
// now-unlocked element.
|
||||
function playUrl(url) {
|
||||
aud.src = mediaUrl(url);
|
||||
aud.volume = 0;
|
||||
audLastErr = "";
|
||||
const pr = aud.play();
|
||||
if (pr) {
|
||||
pr.then(() => { audLastErr = ""; updateAudioStatus(); })
|
||||
.catch((e) => { audLastErr = "play BLOCKED: " + ((e && e.name) || e); updateAudioStatus(); });
|
||||
}
|
||||
fadeVolume(aud, 1, FADE_MS);
|
||||
}
|
||||
|
||||
// Reconcile audio to the Audio toggle + current altitude: on → the scale's
|
||||
// soundtrack, off → silence. Idempotent on an unchanged url (so a same-scale pool
|
||||
// re-roll doesn't restart it). MUST be called synchronously from the toggle gesture
|
||||
// the first time it plays.
|
||||
function applyAudio() {
|
||||
const url = $("audio").checked ? soundtrackUrl() : null;
|
||||
if (url === audUrl) return;
|
||||
audUrl = url;
|
||||
if (!audioReady) return; // deferred until the start gesture (§8)
|
||||
if (!url) { // fade current out to silence
|
||||
await fadeVolume(audActive, 0, XFADE_MS);
|
||||
audActive.pause();
|
||||
return;
|
||||
}
|
||||
audIdle.src = mediaUrl(url); // absolute /media/... passes through mediaUrl
|
||||
audIdle.volume = 0;
|
||||
audIdle.play().catch(() => {}); // start (fire-and-forget — awaiting can hang)
|
||||
await Promise.all([fadeVolume(audIdle, 1, XFADE_MS), fadeVolume(audActive, 0, XFADE_MS)]);
|
||||
audActive.pause();
|
||||
[audActive, audIdle] = [audIdle, audActive]; // swap roles
|
||||
if (!url) { fadeVolume(aud, 0, FADE_MS, () => aud.pause()); return; }
|
||||
if (aud.paused || !aud.src) playUrl(url); // start fresh
|
||||
else fadeVolume(aud, 0, 250, () => playUrl(url)); // dip out, then swap + fade in
|
||||
}
|
||||
|
||||
// Browsers block autoplay until a user gesture. The renderer is launched
|
||||
// full-screen by the operator; a one-time tap unlocks audio, then it follows
|
||||
// state (audio spec §8 — the documented launch step).
|
||||
function showStartGesture() {
|
||||
if (audioReady || document.getElementById("start-gesture")) return;
|
||||
const ov = document.createElement("div");
|
||||
ov.id = "start-gesture";
|
||||
ov.textContent = "▶ tap to start sound";
|
||||
ov.addEventListener("click", () => {
|
||||
// A click anywhere on the page grants Chrome's autoplay permission
|
||||
// document-wide for the session, so later programmatic play() is allowed —
|
||||
// no per-element priming needed (priming srcless elements only pollutes state).
|
||||
audioReady = true;
|
||||
ov.remove();
|
||||
audUrl = null; // force the next applyAudio to (re)load
|
||||
update(); // re-apply current state, now with audio
|
||||
}, { once: true });
|
||||
document.body.appendChild(ov);
|
||||
// "Loading Universe…" splash — hidden once media is preloaded; reflects progress.
|
||||
function setLoadingProgress(done, total) {
|
||||
const fill = document.getElementById("loading-fill");
|
||||
if (fill && total) fill.style.width = Math.round((done / total) * 100) + "%";
|
||||
}
|
||||
function hideLoading() {
|
||||
const el = document.getElementById("loading");
|
||||
if (!el) return;
|
||||
el.classList.add("done");
|
||||
setTimeout(() => el.remove(), 700);
|
||||
}
|
||||
|
||||
let videoEverOn = false; // has Video ever been switched on this session?
|
||||
|
||||
async function main() {
|
||||
devLiveReload();
|
||||
try { initPaint(); } catch (e) { paintOK = false; paint.style.display = "none"; showError("WebGL init: " + e.message); }
|
||||
await loadData();
|
||||
await landScale(); // pick the initial scale's pool member before first render
|
||||
preloadAllMedia(); // background: cache every clip in memory for instant altitude swaps
|
||||
buildDial(); // draw the altitude knob from the ring's scales
|
||||
initDev(); // wire the Dev Mode toggle + pool picker (reads persisted state)
|
||||
renderScaleReadout();
|
||||
for (const id of ["visual", "audio", "left", "right", "mood"]) {
|
||||
$(id).addEventListener("input", debounced);
|
||||
}
|
||||
showStartGesture(); // one-time tap unlocks audio (browser autoplay policy, §8)
|
||||
// Sliders stream on "input"; the toggles fire on "change".
|
||||
for (const id of ["left", "right", "mood"]) $(id).addEventListener("input", debounced);
|
||||
// Audio toggle: start/stop the soundtrack SYNCHRONOUSLY in this gesture so it
|
||||
// unlocks + plays on Safari (which blocks play() outside a user gesture).
|
||||
$("audio").addEventListener("change", () => { applyAudio(); debounced(); });
|
||||
// Video toggle: the FIRST time video turns on, bring audio with it (one flip = the
|
||||
// full experience). Audio toggled on its own first stays audio-only. The audio is
|
||||
// set + played in THIS gesture so it unlocks on Safari.
|
||||
$("visual").addEventListener("change", () => {
|
||||
if ($("visual").checked && !videoEverOn) {
|
||||
videoEverOn = true;
|
||||
if (!$("audio").checked) { $("audio").checked = true; applyAudio(); }
|
||||
}
|
||||
debounced();
|
||||
});
|
||||
// Altitude knob: drag to turn (commit detents on release), scroll to step, tap a label to jump.
|
||||
dial.addEventListener("pointerdown", onDialDown);
|
||||
window.addEventListener("pointermove", onDialMove);
|
||||
window.addEventListener("pointerup", onDialUp);
|
||||
dial.addEventListener("wheel", onWheel, { passive: false });
|
||||
$("stage").addEventListener("wheel", onWheel, { passive: false });
|
||||
update();
|
||||
update(); // render the initial state (both toggles off → black, silent)
|
||||
await preloadAllMedia(); // download all media, updating the loading bar
|
||||
hideLoading(); // experience is ready to run smoothly
|
||||
}
|
||||
main();
|
||||
|
||||
+27
-13
@@ -7,13 +7,18 @@
|
||||
<link rel="stylesheet" href="/style.css" />
|
||||
</head>
|
||||
<body>
|
||||
<div id="loading">
|
||||
<div class="loading-inner">
|
||||
<div class="loading-title">Loading Universe<span class="loading-dots"></span></div>
|
||||
<div class="loading-bar"><div id="loading-fill"></div></div>
|
||||
</div>
|
||||
</div>
|
||||
<header><h1>Human Experience Filter — Alteration Preview</h1></header>
|
||||
<main>
|
||||
<section class="stage" id="stage">
|
||||
<div class="screen">
|
||||
<video id="vid" loop muted playsinline></video>
|
||||
<audio id="audA" loop preload="auto"></audio>
|
||||
<audio id="audB" loop preload="auto"></audio>
|
||||
<audio id="aud" loop preload="auto"></audio>
|
||||
<canvas id="paint"></canvas>
|
||||
<div id="tint"></div>
|
||||
<svg id="overlay" viewBox="0 0 100 100" preserveAspectRatio="none"></svg>
|
||||
@@ -24,17 +29,17 @@
|
||||
|
||||
<section class="panel">
|
||||
<fieldset>
|
||||
<legend>Visual</legend>
|
||||
<label class="toggle"><input type="checkbox" id="visual" checked /> show video</label>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Audio</legend>
|
||||
<select id="audio">
|
||||
<option value="off">off (silence)</option>
|
||||
<option value="soundtrack">soundtrack (follows altitude)</option>
|
||||
<option value="white_noise">white noise</option>
|
||||
</select>
|
||||
<legend>Output</legend>
|
||||
<label class="dev-switch" for="visual">
|
||||
<input type="checkbox" id="visual" />
|
||||
<span class="dev-switch-track"><span class="dev-switch-thumb"></span></span>
|
||||
<span class="dev-switch-label">Video</span>
|
||||
</label>
|
||||
<label class="dev-switch" for="audio">
|
||||
<input type="checkbox" id="audio" />
|
||||
<span class="dev-switch-track"><span class="dev-switch-thumb"></span></span>
|
||||
<span class="dev-switch-label">Audio</span>
|
||||
</label>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
@@ -61,6 +66,15 @@
|
||||
</label>
|
||||
|
||||
<div id="dev-panel" class="dev-panel hidden">
|
||||
<fieldset>
|
||||
<legend>Audio diagnostics</legend>
|
||||
<div id="audio-status" class="audio-status">audio: off</div>
|
||||
<p class="hint">Native player — tests the soundtrack file directly (bypasses
|
||||
all app code). If it plays here but the Audio toggle is silent, it's a
|
||||
code/autoplay issue, not the file.</p>
|
||||
<audio id="aud-test" controls preload="auto" src="/media/audio/cosmos/pillars.loop.mp3" style="width:100%"></audio>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Pool — pick a clip</legend>
|
||||
<select id="pool-select" aria-label="Choose a clip from this altitude's pool"></select>
|
||||
|
||||
@@ -40,8 +40,19 @@ main { display: flex; gap: 1rem; padding: 1rem; flex-wrap: wrap; align-items: fl
|
||||
.hud-chip-text.measure { fill: #ffd79a; }
|
||||
.hud-conf { fill: #6cf; opacity: 0.8; }
|
||||
.hud-status { fill: #8fdcff; font: 2.4px monospace; opacity: 0.9; letter-spacing: 0.15px; }
|
||||
.black { position: absolute; inset: 0; background: #000; opacity: 1; transition: opacity 200ms ease; }
|
||||
/* z-index + own compositing layer: the <video> and WebGL <canvas> are GPU-
|
||||
composited layers that can paint ABOVE a plain auto-z sibling (Safari/Chrome
|
||||
with a real GPU) despite DOM order — so the black cover needs an explicit
|
||||
stacking order above them to actually blank the screen. */
|
||||
.black { position: absolute; inset: 0; background: #000; opacity: 1;
|
||||
z-index: 50; transform: translateZ(0); transition: opacity 200ms ease; }
|
||||
.hidden { display: none; }
|
||||
/* Stack the two Output toggles (Video / Audio) with a little breathing room. */
|
||||
.dev-switch + .dev-switch { margin-top: 0.45rem; }
|
||||
/* Live audio status readout (diagnostic) — turns green when actually playing. */
|
||||
.audio-status { margin-top: 0.5rem; font: 11px/1.4 monospace; color: #9ab; }
|
||||
.audio-status.playing { color: #4e9; }
|
||||
.audio-status.blocked { color: #f97; }
|
||||
.panel { flex: 0 0 280px; display: flex; flex-direction: column; gap: 0.8rem;
|
||||
max-height: calc(100vh - 2rem); overflow-y: auto;
|
||||
position: sticky; top: 1rem; }
|
||||
@@ -108,12 +119,28 @@ input[type=range], select { width: 100%; }
|
||||
.dev-anno .anno-meta { color: #678; }
|
||||
.dev-anno .anno-empty { color: #567; font-style: italic; }
|
||||
|
||||
/* One-time tap-to-start overlay — unlocks audio under the browser autoplay
|
||||
policy (audio spec §8). Removed after the first gesture. */
|
||||
#start-gesture {
|
||||
/* "Loading Universe…" splash — shown until all media is preloaded and the
|
||||
experience is ready to run smoothly, then faded out. */
|
||||
#loading {
|
||||
position: fixed; inset: 0; z-index: 10000;
|
||||
display: flex; align-items: center; justify-content: center;
|
||||
background: rgba(0, 0, 0, 0.82); color: #fff;
|
||||
font: 600 28px/1.2 system-ui, sans-serif; letter-spacing: 0.02em;
|
||||
cursor: pointer; user-select: none;
|
||||
background: radial-gradient(ellipse at center, #0a1230 0%, #02030a 70%);
|
||||
color: #cfe3ff; user-select: none;
|
||||
transition: opacity 0.6s ease;
|
||||
}
|
||||
#loading.done { opacity: 0; pointer-events: none; }
|
||||
.loading-inner { display: flex; flex-direction: column; align-items: center; gap: 1.1rem; }
|
||||
.loading-title { font: 600 30px/1.2 system-ui, sans-serif; letter-spacing: 0.04em; }
|
||||
.loading-dots::after {
|
||||
content: ""; animation: loading-dots 1.4s steps(4, end) infinite;
|
||||
}
|
||||
@keyframes loading-dots { 0% { content: ""; } 25% { content: "."; } 50% { content: ".."; } 75% { content: "..."; } 100% { content: ""; } }
|
||||
.loading-bar {
|
||||
width: 260px; height: 4px; border-radius: 2px;
|
||||
background: rgba(255, 255, 255, 0.12); overflow: hidden;
|
||||
}
|
||||
#loading-fill {
|
||||
height: 100%; width: 0%; border-radius: 2px;
|
||||
background: linear-gradient(90deg, #4e9cff, #9af);
|
||||
transition: width 0.25s ease;
|
||||
}
|
||||
|
||||
+65
-31
@@ -1,7 +1,9 @@
|
||||
"""Playwright E2E for the audio layer (audio spec §9). Skips cleanly when
|
||||
Playwright or its browser binary is absent (the headless dev box) — the unit +
|
||||
contract tiers cover the logic; this asserts the rendered <audio> element src and
|
||||
playback survives a Visual fade. Runs at the PPE stage where a browser exists.
|
||||
"""Playwright E2E for the audio + Video/Audio toggles (audio spec §9). Skips
|
||||
cleanly when Playwright or its browser binary is absent.
|
||||
|
||||
NOTE: headless engines relax BOTH autoplay and GPU compositing, so these assert
|
||||
the WIRING (toggle→play, the first-video-on→audio coupling, video-off blanking) —
|
||||
real Safari/iOS autoplay and real-GPU compositing still need a device by-ear/eye check.
|
||||
|
||||
Install: pip install -e '.[e2e]' && python -m playwright install chromium
|
||||
"""
|
||||
@@ -12,7 +14,6 @@ from contextlib import closing
|
||||
|
||||
import pytest
|
||||
|
||||
# Skip the whole module unless Playwright is importable.
|
||||
pytest.importorskip("playwright.sync_api")
|
||||
|
||||
|
||||
@@ -50,49 +51,82 @@ def page(app_url):
|
||||
|
||||
with sync_playwright() as p:
|
||||
try:
|
||||
# Headless Chromium blocks media playback without a real audio device;
|
||||
# this flag lets play() actually start so `paused` is meaningful (the
|
||||
# spec §9 asserts playback via element src/paused, not a waveform).
|
||||
# headless blocks media playback without a device; this flag lets play()
|
||||
# actually start so `paused` is meaningful (spec §9 asserts src/paused)
|
||||
browser = p.chromium.launch(args=["--autoplay-policy=no-user-gesture-required"])
|
||||
except Exception as exc: # no browser binary installed
|
||||
pytest.skip(f"chromium not available: {exc}")
|
||||
pg = browser.new_page()
|
||||
pg.goto(app_url)
|
||||
pg.click("#start-gesture") # unlock autoplay
|
||||
# wait out the "Loading Universe…" splash (it overlays + blocks clicks)
|
||||
pg.wait_for_selector("#loading", state="detached", timeout=60000)
|
||||
yield pg
|
||||
browser.close()
|
||||
|
||||
|
||||
def test_white_noise_loads_the_noise_asset(page):
|
||||
page.select_option("#audio", "white_noise")
|
||||
def _toggle(page, which):
|
||||
"""Click the visible toggle track for #visual / #audio (the input is hidden)."""
|
||||
page.click(f"label[for='{which}'] .dev-switch-track")
|
||||
|
||||
|
||||
def test_app_starts_blanked_and_silent(page):
|
||||
# both toggles default off → black screen, no audio (no tap-to-start wall)
|
||||
assert page.is_checked("#visual") is False
|
||||
assert page.is_checked("#audio") is False
|
||||
page.wait_for_selector("#black:not(.hidden)")
|
||||
assert page.evaluate("document.getElementById('aud').paused") is True
|
||||
|
||||
|
||||
def test_audio_toggle_plays_the_current_scale_soundtrack(page):
|
||||
_toggle(page, "audio")
|
||||
page.wait_for_function(
|
||||
"['audA','audB'].some(id => "
|
||||
"document.getElementById(id).src.includes('/media/audio/noise/pink.mp3'))"
|
||||
"(() => { const a = document.getElementById('aud');"
|
||||
" return /\\/media\\/audio\\/.+\\.mp3/.test(a.src) && !a.paused; })()"
|
||||
)
|
||||
|
||||
|
||||
def test_soundtrack_loads_the_current_scale_asset(page):
|
||||
# initial altitude is cosmos (index 0) → soundtrack resolves its scale asset,
|
||||
# not the noise bed (the altitude-coupling itself is covered at the API tier)
|
||||
page.select_option("#audio", "soundtrack")
|
||||
def test_first_video_on_also_enables_audio(page):
|
||||
_toggle(page, "visual") # first video-on couples audio
|
||||
page.wait_for_function("document.getElementById('audio').checked === true")
|
||||
page.wait_for_function(
|
||||
"['audA','audB'].some(id => {"
|
||||
" const s = document.getElementById(id).src;"
|
||||
" return /\\/media\\/audio\\/.+\\.mp3/.test(s) && !s.includes('noise');"
|
||||
"})"
|
||||
"(() => { const a = document.getElementById('aud'); return !!a.src && !a.paused; })()"
|
||||
)
|
||||
page.wait_for_selector("#black.hidden", state="attached") # video showing (black hidden=display:none)
|
||||
|
||||
|
||||
def test_visual_off_keeps_audio_and_fades_video(page):
|
||||
page.select_option("#audio", "white_noise")
|
||||
def test_audio_before_video_stays_audio_only(page):
|
||||
_toggle(page, "audio") # audio on first
|
||||
page.wait_for_function("!document.getElementById('aud').paused")
|
||||
assert page.is_checked("#visual") is False # video untouched
|
||||
page.wait_for_selector("#black:not(.hidden)") # still blanked
|
||||
|
||||
|
||||
def test_audio_then_video_leaves_both_on(page):
|
||||
_toggle(page, "audio") # audio first
|
||||
page.wait_for_function("!document.getElementById('aud').paused")
|
||||
_toggle(page, "visual") # then video
|
||||
page.wait_for_selector("#black.hidden", state="attached") # video showing
|
||||
assert page.is_checked("#audio") is True
|
||||
assert page.evaluate("!document.getElementById('aud').paused") is True # audio still plays
|
||||
|
||||
|
||||
def test_soundtrack_fallback_when_ring_lacks_audio(page):
|
||||
# simulate a stale server whose /api/ring omits the per-scale `audio` field;
|
||||
# the client falls back to the scale-id soundtrack map so audio still plays
|
||||
page.evaluate("ring.scales.forEach(s => { delete s.audio; })")
|
||||
_toggle(page, "audio")
|
||||
page.wait_for_function(
|
||||
"['audA','audB'].some(id => "
|
||||
"document.getElementById(id).src.includes('/media/audio/noise/pink.mp3'))"
|
||||
"(() => { const a = document.getElementById('aud');"
|
||||
" return a.src.includes('/media/audio/cosmos/') && !a.paused; })()"
|
||||
)
|
||||
page.uncheck("#visual")
|
||||
page.wait_for_selector("#black:not(.hidden)") # video faded to black
|
||||
playing = page.evaluate(
|
||||
"['audA','audB'].some(id => "
|
||||
"{const a = document.getElementById(id); return !a.paused && !!a.src;})"
|
||||
|
||||
|
||||
def test_video_off_blanks_after_being_on(page):
|
||||
_toggle(page, "visual") # on
|
||||
page.wait_for_selector("#black.hidden", state="attached")
|
||||
_toggle(page, "visual") # off
|
||||
page.wait_for_selector("#black:not(.hidden)")
|
||||
# the GPU-composited video layers are hidden too (settle the opacity transition)
|
||||
page.wait_for_function(
|
||||
"['vid','paint'].every(id => getComputedStyle(document.getElementById(id)).opacity === '0')"
|
||||
)
|
||||
assert playing is True # audio survives the video fade
|
||||
|
||||
+12
-16
@@ -10,9 +10,6 @@ from player.audio import (
|
||||
resolve_visual,
|
||||
)
|
||||
|
||||
NOISE = "/media/audio/noise/pink.mp3"
|
||||
|
||||
|
||||
def test_visual_on_off():
|
||||
assert resolve_visual("on") is True
|
||||
assert resolve_visual("off") is False
|
||||
@@ -27,37 +24,36 @@ def test_visual_positions_are_on_off():
|
||||
assert VISUAL_POSITIONS == frozenset({"on", "off"})
|
||||
|
||||
|
||||
def test_audio_sources_are_off_and_soundtrack():
|
||||
assert AUDIO_SOURCES == frozenset({"off", "soundtrack"})
|
||||
|
||||
|
||||
def test_soundtrack_resolves_to_the_current_scales_asset():
|
||||
url = resolve_audio_url("soundtrack", scale_audio="coast/waves.loop.mp3", noise_url=NOISE)
|
||||
url = resolve_audio_url("soundtrack", scale_audio="coast/waves.loop.mp3")
|
||||
assert url == "/media/audio/coast/waves.loop.mp3"
|
||||
|
||||
|
||||
def test_white_noise_resolves_to_the_global_bed_independent_of_altitude():
|
||||
url = resolve_audio_url("white_noise", scale_audio="coast/waves.loop.mp3", noise_url=NOISE)
|
||||
assert url == NOISE
|
||||
|
||||
|
||||
def test_off_resolves_to_silence():
|
||||
assert resolve_audio_url("off", scale_audio="coast/waves.loop.mp3", noise_url=NOISE) is None
|
||||
assert resolve_audio_url("off", scale_audio="coast/waves.loop.mp3") is None
|
||||
|
||||
|
||||
def test_soundtrack_with_no_scale_asset_is_silent():
|
||||
assert resolve_audio_url("soundtrack", scale_audio="", noise_url=NOISE) is None
|
||||
assert resolve_audio_url("soundtrack", scale_audio="") is None
|
||||
|
||||
|
||||
def test_resolve_audio_marks_only_soundtrack_altitude_coupled():
|
||||
r = resolve_audio("soundtrack", scale_audio="reef/soundscape.loop.mp3", noise_url=NOISE)
|
||||
r = resolve_audio("soundtrack", scale_audio="reef/soundscape.loop.mp3")
|
||||
assert isinstance(r, AudioResolution)
|
||||
assert r.source == "soundtrack" and r.altitude_coupled is True
|
||||
assert r.url == "/media/audio/reef/soundscape.loop.mp3"
|
||||
assert resolve_audio("white_noise", scale_audio="x/y.mp3", noise_url=NOISE).altitude_coupled is False
|
||||
assert resolve_audio("off", scale_audio="x/y.mp3", noise_url=NOISE).altitude_coupled is False
|
||||
assert resolve_audio("off", scale_audio="x/y.mp3").altitude_coupled is False
|
||||
|
||||
|
||||
def test_resolve_audio_url_rejects_unknown_source():
|
||||
with pytest.raises(ValueError):
|
||||
resolve_audio_url("podcast", scale_audio="", noise_url=NOISE)
|
||||
resolve_audio_url("white_noise", scale_audio="") # deferred, not a v1 source
|
||||
|
||||
|
||||
def test_music_is_not_a_v1_source():
|
||||
def test_music_and_white_noise_are_not_v1_sources():
|
||||
assert "music" not in AUDIO_SOURCES
|
||||
assert "white_noise" not in AUDIO_SOURCES
|
||||
|
||||
@@ -11,7 +11,7 @@ from player.controls import (
|
||||
|
||||
def test_visual_and_audio_enums_are_the_v1_positions():
|
||||
assert VISUAL_POSITIONS == frozenset({"on", "off"})
|
||||
assert AUDIO_SOURCES == frozenset({"off", "soundtrack", "white_noise"})
|
||||
assert AUDIO_SOURCES == frozenset({"off", "soundtrack"})
|
||||
|
||||
|
||||
def test_valid_controls_pass_validation():
|
||||
@@ -47,10 +47,10 @@ def test_out_of_range_or_non_int_knob_rejected(field, bad):
|
||||
|
||||
def test_parse_controls_from_mapping():
|
||||
c = parse_controls(
|
||||
{"visual": "on", "audio": "white_noise", "left": 1, "right": 2, "dark": 3,
|
||||
{"visual": "on", "audio": "soundtrack", "left": 1, "right": 2, "dark": 3,
|
||||
"light": 0, "volume": 2, "brightness": 1}
|
||||
)
|
||||
assert c == Controls("on", "white_noise", 1, 2, 3, 0, 2, 1)
|
||||
assert c == Controls("on", "soundtrack", 1, 2, 3, 0, 2, 1)
|
||||
|
||||
|
||||
def test_parse_controls_rejects_unknown_keys():
|
||||
|
||||
@@ -38,11 +38,11 @@ def test_visual_off_from_video_fades_to_black():
|
||||
|
||||
|
||||
def test_audio_survives_visual_off():
|
||||
# white noise + black is now reachable (the gap the bundled dial omitted)
|
||||
# soundtrack + black is reachable (a gap the bundled dial omitted)
|
||||
p = Player(LIB)
|
||||
t = p.update(_controls(visual="off", audio="white_noise"))
|
||||
t = p.update(_controls(visual="off", audio="soundtrack"))
|
||||
assert t.playback.video is False
|
||||
assert t.playback.audio_source == "white_noise"
|
||||
assert t.playback.audio_source == "soundtrack"
|
||||
|
||||
|
||||
def test_no_change_yields_none_transition():
|
||||
@@ -90,7 +90,7 @@ def test_volume_only_change_is_a_live_update():
|
||||
|
||||
def test_audio_source_change_while_black_is_a_live_update():
|
||||
p = Player(LIB)
|
||||
p.update(_controls(visual="off", audio="white_noise"))
|
||||
p.update(_controls(visual="off", audio="off"))
|
||||
t = p.update(_controls(visual="off", audio="soundtrack"))
|
||||
assert t.kind == TransitionKind.LIVE_UPDATE
|
||||
assert t.playback.audio_source == "soundtrack"
|
||||
|
||||
@@ -56,13 +56,11 @@ def test_alteration_visual_off_hides_video(client):
|
||||
assert body["render"]["video"]["shown"] is False
|
||||
|
||||
|
||||
def test_alteration_white_noise_is_altitude_independent(ring_client):
|
||||
for idx in (0, 1, 2):
|
||||
body = ring_client.post("/api/alteration", json={
|
||||
"controls": _controls(audio="white_noise"), "altitude_index": idx,
|
||||
}).json()
|
||||
assert body["render"]["audio"]["url"] == "/media/audio/noise/pink.mp3"
|
||||
assert body["render"]["audio"]["altitude_coupled"] is False
|
||||
def test_alteration_off_is_silent(ring_client):
|
||||
body = ring_client.post("/api/alteration", json={
|
||||
"controls": _controls(audio="off"), "altitude_index": 0,
|
||||
}).json()
|
||||
assert body["render"]["audio"] == {"source": "off", "url": None, "altitude_coupled": False}
|
||||
|
||||
|
||||
def test_alteration_soundtrack_couples_to_the_given_altitude(ring_client):
|
||||
|
||||
Reference in New Issue
Block a user