# Networked Control Surface — design

**Status:** proposed
**Date:** 2026-06-26
**Session:** 0019
**Supersedes/reframes:** ROADMAP sub-project 4 (Arduino Firmware / control panel) and
the "serial input" slice of sub-project 3 (Player Runtime).

---

## 1. Goal

Split the existing simulator into a **renderer** (the laptop full-screen
projection — "the experience") and a separate **controller** (a touch remote),
talking over the local network, so the experience can be driven from a handheld
device the way the real installation's panel drives its screen.

This is **v1 hardware**, deliberately simulator-friendly:

- The **laptop/computer renders** (no Raspberry Pi yet).
- The controller is a **separate device** over **wifi** (LAN), not a USB-serial
  link to a Pi.
- The controller is **prototyped as a web page** (an iPad/tablet/phone twin),
  and a **physical Arduino remote** drops in later as just another control source
  speaking the same contract.

Two goals, equally weighted (operator, session 0019): **decouple control from
display cleanly**, *and* get the **handheld "stand back and drive it" feel**.

## 2. Core idea: one server of truth, interchangeable control sources

```
┌──────────────────┐   POST /api/control      ┌─────────────────────┐
│  CONTROL SOURCE  │ ───── (knob change) ────▶ │   FastAPI server    │
│                  │                           │  (laptop)           │
│ • iPad web remote│ ◀──── GET /api/state ──── │  ┌───────────────┐  │
│   (v1)           │        (sync on load)     │  │ SessionState  │  │
│ • Arduino (later)│                           │  │ controls +    │  │
└──────────────────┘                           │  │ ring index +  │  │
                                               │  │ transitions + │  │
┌──────────────────┐   GET /api/events (SSE)   │  │ seq (version) │  │
│    RENDERER      │ ◀═══ state/ring/mode ════ │  └───────┬───────┘  │
│ laptop full-     │      (live push)          └──────────┼──────────┘
│ screen           │                                      │
│ (the experience) │ ──── POST /api/render/event ────────▶│ (settled ack)
└──────────────────┘                          recompute RenderPlan + ring move
```

- **Server** owns the single authoritative `SessionState` (current controls,
  altitude/ring index, per-field transition state, a monotonic `seq`). Every
  change recomputes the derived view and broadcasts it.
- **Control source** = anything that `POST`s a control change. v1 has exactly
  one: the iPad web remote. A future Arduino is *another instance of the same
  role* — no new server concept.
- **Renderer** = the laptop projection. It holds **no** input UI; it subscribes
  to the SSE stream and renders whatever state arrives. It is today's
  `simulator/static/index.html`, stripped of its sliders, given a `state`
  listener plus one small upstream ack channel.

The property that matters: controller and renderer are now **separate web pages
off the same server**, decoupled exactly as a real installation's panel and
screen are. (Chosen architecture: server-authoritative session + **Server-Sent
Events** push. SSE fits because the directions are asymmetric — the renderer only
*receives*, the control source only *sends* — so no bidirectional WebSocket is
needed. Alternatives considered and rejected: renderer polling — laggy/wasteful;
peer-to-peer WebRTC — most plumbing and it breaks the clean single-contract seam
that makes the Arduino a drop-in.)

## 3. Control surface: the six controls (the faithful twin)

The iPad remote is a **faithful on-screen twin** of the intended physical panel —
what you like on the iPad is the spec for the Arduino panel. The control
inventory is exactly what the sim exposes today:

| Control | Sim today | Physical form (later) |
|---|---|---|
| **Visual** | toggle (show video on/off) | 2-position switch |
| **Audio** | on/off toggle (on = per-altitude soundtrack; ·white-noise & music deferred) | 2-position switch (rotary selector when more sources land) |
| **Altitude** | endless circular dial — walks the 5 scales (cosmos→abyss), wraps | rotary encoder (endless, detented) |
| **Left** (analytical) | slider 0–4 | knob/pot, 5 detents |
| **Right** (dreamlike) | slider 0–4 | knob/pot, 5 detents |
| **Mood** | slider −4…+4 (dark↔light) | center-detented knob |

(This is the original sub-project-4 inventory, *updated*: Dark+Light collapsed
into one **Mood** knob in session 0013, the **Altitude** encoder added in session
0018, and the bundled 7-way **Content** dial split into orthogonal **Visual** ×
**Audio** controls in session 0020 — see
[`2026-06-26-audio-video-separated-controls-design.md`](./2026-06-26-audio-video-separated-controls-design.md).)

## 4. Live vs. transitional controls (per-knob transition state)

The five controls split into two classes by how a **human** perceives the change:

- **Live controls — Left, Right, Mood, Audio.** The alteration engine treats
  Left/Right/Mood as `LIVE_UPDATE` (the WebGL Kuwahara dream + HUD overlay + color
  grade re-tune within a frame). **Audio** source changes are also live: a short
  (~0.6 s) renderer-side gain crossfade between two `<audio>` elements —
  perceptible but non-blocking, with no `settled` ack (audio spec §7). These are
  *always idle*; the remote just shows the value.
- **Transitional controls — Altitude, Visual.** These trigger a renderer
  animation that takes real, perceptible time: Altitude plays ring-transition
  clips (zoom/warp between scales); **Visual** on/off does fade-to-black /
  fade-from-black. During that window the control is **busy**, and the control
  source must model that so its feedback is honest.

Per-transitional-control lifecycle:

```
idle ──(source sends)──▶ pending ──(renderer starts)──▶ transitioning ──(renderer settles)──▶ idle
```

- **pending** — set locally by the remote the instant the knob is turned
  (immediate tactile feedback before the server replies).
- **transitioning** — the server flags the field once the renderer actually
  begins the animation; broadcast to *all* control sources.
- **idle** — cleared when the **renderer acks completion** (the only party that
  knows real playback timing), with a server-side max-duration timeout as a
  safety net so a dropped renderer can't wedge a knob.

Hardware payoff: a physical Altitude encoder can light an LED while
`transitioning`; the iPad twin shows the matching spinner/disabled-detent.

`SessionState` therefore carries
`transitions: { altitude: idle|transitioning, visual: idle|transitioning }`
alongside the controls and `seq`. (Live controls — Left/Right/Mood/Audio — need
no entry.)

## 5. Server contract

Four endpoints carry the whole thing. The existing `/api/alteration`,
`/api/ring`, `/api/ring/advance` remain as **internal helpers the server calls**;
control sources never hit them directly.

### 5.1 `GET /api/state` — snapshot (sync-on-load / polling fallback)

```json
{
  "seq": 42,
  "controls": { "visual": "on", "audio": "soundtrack", "left": 3, "right": 1, "mood": -2 },
  "altitude": { "index": 2, "scale": "coast", "clip_id": "coast_07" },
  "transitions": { "altitude": "idle", "visual": "idle" },
  "render": {
    "plan": { },
    "video": { "shown": true },
    "audio": { "source": "soundtrack", "url": "/media/audio/coast/waves.loop.mp3", "altitude_coupled": true }
  }
}
```

(The `render.video` + `render.audio` split and server-side `audio.url` resolution
are specified in the audio spec §6.1.)

A control source GETs this on load to draw its knobs in the right positions; the
renderer can use it to cold-start or resync after a reconnect.

### 5.2 `POST /api/control` — the one control-source contract

A partial patch — only the fields that changed:

```json
{ "set": { "left": 4 },               "altitude_delta": 0 }   // live knob
{ "set": { "audio": "white_noise" },  "altitude_delta": 0 }   // live (audio source)
{ "set": { "visual": "off" },         "altitude_delta": 0 }   // transitional (video fade)
{ "set": {},                          "altitude_delta": -1 }  // one encoder tick down
```

Server applies it, bumps `seq`, recomputes, and:

- **live field** (Left/Right/Mood, **or an `audio` source change**) → broadcasts
  a `state` event immediately (the renderer crossfades audio on its own, §7);
- **transitional field** (a `visual` change, or a non-zero `altitude_delta`) →
  computes the ring move / video fade, marks that field `transitioning`,
  broadcasts a `ring` (or `mode`) event for the renderer to animate.

Returns the new snapshot (an immediate authoritative echo for the poster).
Absolute `set` values make retries idempotent; `altitude_delta` is relative
(encoder ticks) and the server coalesces rapid ticks via the existing
`advance_ring` fast-spin logic.

### 5.3 `GET /api/events` — SSE stream (renderer subscribes)

Event types:

- `state` — full snapshot; sent on connect and on every live change (including an
  **audio** source change, which the renderer crossfades — §7).
- `ring` — an altitude move: the transition clip(s) to play + the landed
  scale/clip, **plus the new `audio.url`** so the renderer can crossfade the
  soundtrack in step (when `audio=soundtrack`). Renderer plays them, then settles.
- `mode` — a **Visual** on/off change: the video fade the renderer should run.
- `ping` — keepalive.

The SSE event payloads reuse the existing serializers
(`ring_move_to_dict`, `render_plan_to_dict`, etc.).

### 5.4 `POST /api/render/event` — the renderer's one upstream ack

```json
{ "type": "settled", "seq": 42, "field": "altitude" }
```

Sent when the renderer finishes a transition. Server flips that field `idle` and
broadcasts a `state` so every remote clears its spinner. A server-side timeout
(max transition duration + margin) clears it anyway if the renderer never acks.

### 5.5 End-to-end flow for one altitude tick

```
iPad: turn dial → field=pending locally → POST /api/control {altitude_delta:-1}
server: apply, seq++, transitions.altitude="transitioning"
        → SSE "ring" to renderer   → SSE "state" to all remotes (altitude busy)
renderer: play transition clip(s), settle on forest_07
        → POST /api/render/event {type:"settled", field:"altitude"}
server: transitions.altitude="idle" → SSE "state" to all → iPad clears spinner
```

### 5.6 Decisions flagged (not assumed silently)

- **Concurrency = last-write-wins.** Multiple control sources are allowed;
  per-field last write wins, `seq` orders everything. No locking in v1.
- **No auth.** v1 trusts the LAN — any device that can reach the laptop can drive
  it. A room-token is deferred until wanted.

## 6. The two pages

### 6.1 Renderer — `/` (display), refactored from today's `index.html`

- **Removes** the input controls (the Visual toggle, the Audio `<select>`, the
  Altitude SVG, the three sliders) and their handlers.
- **Keeps** everything that draws the experience: the video element, the **A/B
  `<audio>` crossfade layer**, the WebGL Kuwahara dream shader, the
  HUD/annotation + affect overlay, the ring-transition playback, the media
  preload pool, the crossfade-loop logic.
- **Adds** a thin `state` client: open `EventSource('/api/events')`; on `state`
  apply controls→render live (incl. the audio crossfade); on `ring` play the
  transition clip(s) — crossfading the soundtrack to the event's `audio.url` —
  then settle and `POST /api/render/event {settled}`; on `mode` run the video
  fade. On (re)connect,
  pull `GET /api/state` to resync. SSE auto-reconnects, so a flaky link
  self-heals.
- Runs full-screen on the laptop driving the projector; no keyboard/mouse needed
  once launched.

### 6.2 Controller — `/remote`, a new touch-first page (the faithful twin)

- The six controls laid out like the intended physical panel: **Visual** toggle +
  **Audio** selector, **Altitude** encoder dial (reuse the existing SVG dial
  component), **Left** / **Right** / **Mood** knobs.
- On any input → `POST /api/control` with just the changed field. Live knobs post
  on drag; Altitude posts encoder ticks; Visual/Audio post on change.
- Per-knob transition state (§4): a transitional control goes **pending** on
  touch, reflects **transitioning** from the SSE feed, clears on **settled**.
  Altitude shows a settling indicator and its detents soft-lock (further ticks
  queue as the next target rather than stacking); **Visual** shows the toggle as
  pending until the video fade completes; **Audio** is live (no pending state).
- It **also** subscribes to `/api/events` so it stays in sync with other
  sources (turn the future Arduino's knob and the iPad reflects it, and
  vice-versa). On load it GETs `/api/state` to draw current positions.
- Touch-sized hit targets, no hover dependence, works in mobile Safari with no
  install.

## 7. The future-Arduino seam (contract only in v1)

No firmware in this feature, but the design **proves the seam** so sub-project 4
is a true drop-in:

- **ESP32 (wifi Arduino)** → posts straight to `POST /api/control` over the LAN.
  Zero server change. Recommended hardware target when the time comes.
- **Classic USB Arduino** → emits serial frames to a tiny `bridge.py` on the
  laptop that translates each frame into a `POST /api/control`. The old roadmap
  "3⇄4 serial framing contract" shrinks to *just* the Arduino↔bridge link; the
  renderer/server never know the difference.
- Either way the Arduino sends **absolute pot values** (`set`) + **encoder
  ticks** (`altitude_delta`) — exactly the iPad's message shape.

This is what makes "faithful twin" real: the iPad and the Arduino are the same
control source speaking the same contract.

## 8. Error handling / resilience

- **Renderer drops:** SSE auto-reconnects; on reconnect it GETs `/api/state` and
  resyncs. The per-field transition timeout clears any knob left `transitioning`
  by a renderer that vanished mid-animation.
- **Controller drops / flaky wifi:** POSTs are fire-and-retry; absolute `set`
  values are idempotent, so a retried knob value is harmless. `altitude_delta` is
  the one relative input — on a failed POST the remote keeps the tick in its
  local pending target and resends rather than double-counting.
- **No renderer connected:** the server still accepts control changes and holds
  state; a renderer that connects later catches up from the snapshot.
- **Stale control source:** every event carries `seq`; a source ignores events
  older than what it has applied.

## 9. Testing (satisfies the mandatory pipeline tiers)

- **Pure unit** — a `SessionState` store in Python: apply patch → live vs.
  transitional classification; `altitude_delta` → ring move via the existing
  `advance_ring`; `seq` bump; transition flag set/clear; timeout. No I/O, tested
  like the rest of `player/`.
- **API/contract** — `GET /api/state`; `POST /api/control` (live + transitional +
  bad input → 422); `POST /api/render/event` clears the flag; the SSE
  event-builder tested as a pure function.
- **E2E (Playwright, required for the UI surface)** — open the renderer and
  `/remote` as two pages: move a live knob → assert the renderer's render
  updates; tick Altitude → assert a transition plays, the remote shows
  `transitioning`, then clears on settle.

## 10. Scope

**In v1:**

- Server `SessionState` + the four endpoints (`/api/state`, `/api/control`,
  `/api/events`, `/api/render/event`).
- Renderer refactor to display-only + SSE client + settled ack.
- The `/remote` iPad twin: the five controls + per-knob transition state.
- Wifi/LAN only; laptop renders.

**Explicitly NOT in v1 (deferred):**

- Any Arduino/firmware (contract defined here; hardware later).
- Bluetooth / USB transport.
- Auth / room-token.
- The Pi renderer.
- Multi-room / multi-renderer fan-out (the SSE design allows it; not a v1 goal).

## 11. Roadmap impact

- **Sub-project 4** is reframed: the panel is now "any control source over the
  `/api/control` contract," not USB-serial-to-Pi first. The iPad twin is the v1
  deliverable; the Arduino is a later drop-in.
- **Sub-project 3's "serial input" slice** is absorbed into a transport detail
  (the Arduino↔bridge link), no longer a Pi-coupled concern.
- `ROADMAP.md` to be updated at implementation time.
