diff --git a/docs/superpowers/specs/2026-06-26-networked-control-surface-design.md b/docs/superpowers/specs/2026-06-26-networked-control-surface-design.md new file mode 100644 index 0000000..b31d076 --- /dev/null +++ b/docs/superpowers/specs/2026-06-26-networked-control-surface-design.md @@ -0,0 +1,311 @@ +# Networked Control Surface — design + +**Status:** proposed +**Date:** 2026-06-26 +**Session:** 0019 +**Supersedes/reframes:** ROADMAP sub-project 4 (Arduino Firmware / control panel) and +the "serial input" slice of sub-project 3 (Player Runtime). + +--- + +## 1. Goal + +Split the existing simulator into a **renderer** (the laptop full-screen +projection — "the experience") and a separate **controller** (a touch remote), +talking over the local network, so the experience can be driven from a handheld +device the way the real installation's panel drives its screen. + +This is **v1 hardware**, deliberately simulator-friendly: + +- The **laptop/computer renders** (no Raspberry Pi yet). +- The controller is a **separate device** over **wifi** (LAN), not a USB-serial + link to a Pi. +- The controller is **prototyped as a web page** (an iPad/tablet/phone twin), + and a **physical Arduino remote** drops in later as just another control source + speaking the same contract. + +Two goals, equally weighted (operator, session 0019): **decouple control from +display cleanly**, *and* get the **handheld "stand back and drive it" feel**. + +## 2. Core idea: one server of truth, interchangeable control sources + +``` +┌──────────────────┐ POST /api/control ┌─────────────────────┐ +│ CONTROL SOURCE │ ───── (knob change) ────▶ │ FastAPI server │ +│ │ │ (laptop) │ +│ • iPad web remote│ ◀──── GET /api/state ──── │ ┌───────────────┐ │ +│ (v1) │ (sync on load) │ │ SessionState │ │ +│ • Arduino (later)│ │ │ controls + │ │ +└──────────────────┘ │ │ ring index + │ │ + │ │ transitions + │ │ +┌──────────────────┐ GET /api/events (SSE) │ │ seq (version) │ │ +│ RENDERER │ ◀═══ state/ring/mode ════ │ └───────┬───────┘ │ +│ laptop full- │ (live push) └──────────┼──────────┘ +│ screen │ │ +│ (the experience) │ ──── POST /api/render/event ────────▶│ (settled ack) +└──────────────────┘ recompute RenderPlan + ring move +``` + +- **Server** owns the single authoritative `SessionState` (current controls, + altitude/ring index, per-field transition state, a monotonic `seq`). Every + change recomputes the derived view and broadcasts it. +- **Control source** = anything that `POST`s a control change. v1 has exactly + one: the iPad web remote. A future Arduino is *another instance of the same + role* — no new server concept. +- **Renderer** = the laptop projection. It holds **no** input UI; it subscribes + to the SSE stream and renders whatever state arrives. It is today's + `simulator/static/index.html`, stripped of its sliders, given a `state` + listener plus one small upstream ack channel. + +The property that matters: controller and renderer are now **separate web pages +off the same server**, decoupled exactly as a real installation's panel and +screen are. (Chosen architecture: server-authoritative session + **Server-Sent +Events** push. SSE fits because the directions are asymmetric — the renderer only +*receives*, the control source only *sends* — so no bidirectional WebSocket is +needed. Alternatives considered and rejected: renderer polling — laggy/wasteful; +peer-to-peer WebRTC — most plumbing and it breaks the clean single-contract seam +that makes the Arduino a drop-in.) + +## 3. Control surface: the five controls (the faithful twin) + +The iPad remote is a **faithful on-screen twin** of the intended physical panel — +what you like on the iPad is the spec for the Arduino panel. The control +inventory is exactly what the sim exposes today: + +| Control | Sim today | Physical form (later) | +|---|---|---| +| **Content** | 7-way ``, the Altitude SVG, the three + sliders) and their handlers. +- **Keeps** everything that draws the experience: the video element, the WebGL + Kuwahara dream shader, the HUD/annotation + affect overlay, the + ring-transition playback, the media preload pool, the crossfade-loop logic. +- **Adds** a thin `state` client: open `EventSource('/api/events')`; on `state` + apply controls→render live; on `ring` play the transition clip(s) then settle + and `POST /api/render/event {settled}`; on `mode` run the fade. On (re)connect, + pull `GET /api/state` to resync. SSE auto-reconnects, so a flaky link + self-heals. +- Runs full-screen on the laptop driving the projector; no keyboard/mouse needed + once launched. + +### 6.2 Controller — `/remote`, a new touch-first page (the faithful twin) + +- The five controls laid out like the intended physical panel: **Content** + selector, **Altitude** encoder dial (reuse the existing SVG dial component), + **Left** / **Right** / **Mood** knobs. +- On any input → `POST /api/control` with just the changed field. Live knobs post + on drag; Altitude posts encoder ticks; Content posts on selection. +- Per-knob transition state (§4): a transitional control goes **pending** on + touch, reflects **transitioning** from the SSE feed, clears on **settled**. + Altitude shows a settling indicator and its detents soft-lock (further ticks + queue as the next target rather than stacking); Content shows the selection as + pending until the fade completes. +- It **also** subscribes to `/api/events` so it stays in sync with other + sources (turn the future Arduino's knob and the iPad reflects it, and + vice-versa). On load it GETs `/api/state` to draw current positions. +- Touch-sized hit targets, no hover dependence, works in mobile Safari with no + install. + +## 7. The future-Arduino seam (contract only in v1) + +No firmware in this feature, but the design **proves the seam** so sub-project 4 +is a true drop-in: + +- **ESP32 (wifi Arduino)** → posts straight to `POST /api/control` over the LAN. + Zero server change. Recommended hardware target when the time comes. +- **Classic USB Arduino** → emits serial frames to a tiny `bridge.py` on the + laptop that translates each frame into a `POST /api/control`. The old roadmap + "3⇄4 serial framing contract" shrinks to *just* the Arduino↔bridge link; the + renderer/server never know the difference. +- Either way the Arduino sends **absolute pot values** (`set`) + **encoder + ticks** (`altitude_delta`) — exactly the iPad's message shape. + +This is what makes "faithful twin" real: the iPad and the Arduino are the same +control source speaking the same contract. + +## 8. Error handling / resilience + +- **Renderer drops:** SSE auto-reconnects; on reconnect it GETs `/api/state` and + resyncs. The per-field transition timeout clears any knob left `transitioning` + by a renderer that vanished mid-animation. +- **Controller drops / flaky wifi:** POSTs are fire-and-retry; absolute `set` + values are idempotent, so a retried knob value is harmless. `altitude_delta` is + the one relative input — on a failed POST the remote keeps the tick in its + local pending target and resends rather than double-counting. +- **No renderer connected:** the server still accepts control changes and holds + state; a renderer that connects later catches up from the snapshot. +- **Stale control source:** every event carries `seq`; a source ignores events + older than what it has applied. + +## 9. Testing (satisfies the mandatory pipeline tiers) + +- **Pure unit** — a `SessionState` store in Python: apply patch → live vs. + transitional classification; `altitude_delta` → ring move via the existing + `advance_ring`; `seq` bump; transition flag set/clear; timeout. No I/O, tested + like the rest of `player/`. +- **API/contract** — `GET /api/state`; `POST /api/control` (live + transitional + + bad input → 422); `POST /api/render/event` clears the flag; the SSE + event-builder tested as a pure function. +- **E2E (Playwright, required for the UI surface)** — open the renderer and + `/remote` as two pages: move a live knob → assert the renderer's render + updates; tick Altitude → assert a transition plays, the remote shows + `transitioning`, then clears on settle. + +## 10. Scope + +**In v1:** + +- Server `SessionState` + the four endpoints (`/api/state`, `/api/control`, + `/api/events`, `/api/render/event`). +- Renderer refactor to display-only + SSE client + settled ack. +- The `/remote` iPad twin: the five controls + per-knob transition state. +- Wifi/LAN only; laptop renders. + +**Explicitly NOT in v1 (deferred):** + +- Any Arduino/firmware (contract defined here; hardware later). +- Bluetooth / USB transport. +- Auth / room-token. +- The Pi renderer. +- Multi-room / multi-renderer fan-out (the SSE design allows it; not a v1 goal). + +## 11. Roadmap impact + +- **Sub-project 4** is reframed: the panel is now "any control source over the + `/api/control` contract," not USB-serial-to-Pi first. The iPad twin is the v1 + deliverable; the Arduino is a later drop-in. +- **Sub-project 3's "serial input" slice** is absorbed into a transport detail + (the Arduino↔bridge link), no longer a Pi-coupled concern. +- `ROADMAP.md` to be updated at implementation time.