Networked control surface — drive the experience from a separate control device #26

New Issue

2026-06-27T13:16:23Z

benstull commented

2026-06-27 13:16:23 +00:00

Networked control surface — drive the experience from a separate control device

Summary

The installation's experience can be driven by a separate controller device
over the local network, decoupled from the machine that renders it — enabling a
handheld software remote now and a physical control panel later, with no change to
the renderer when the controller changes.

Problem / pain

Today the controls and the rendered experience are the same browser tab on one
machine: to change altitude, left/right, mood, or content you must stand at the
rendering computer and operate on-screen sliders. For an installation this is
backwards — the person experiencing the piece can't drive it from where they're
standing, the operator can't tune it from across the room, and there is no path
to a physical control panel without re-architecting, because the control surface
and the screen are fused. Every future controller (a handheld, real knobs) is
blocked on that coupling today.

Who feels it

persona	role	what they need
Operator / artist	runs and tunes the installation	to drive and adjust the experience from where they stand, not chained to the render machine
Participant (later)	experiences the piece	to influence the experience from a handheld/panel in the room
Hardware builder	builds the eventual physical panel	a stable seam so a new controller drops in without touching the renderer

Desired outcome / value

Control is decoupled from display. The experience is driven by any "control
source" over the network, so a handheld remote works now and a physical panel
drops in later without re-architecting the renderer — proving the path to the
real installation while keeping today's simulator-first workflow.

Acceptance — "done right"

A person changes the experience (altitude, left, right, mood, content) from a
separate device while the rendering machine only renders.
Successive/concurrent control sources are interchangeable — swapping one for
another needs no renderer change.
Slow, human-perceptible changes (e.g. altitude/content transitions) give
honest feedback on the controller while they play.
The path to physical hardware is proven: a later hardware controller speaks
the same contract and needs no renderer or server redesign.

Scope / non-goals

In scope (across child Features): decoupling control from display; the
networked control contract every control source speaks; transition/busy feedback;
the first software control source (handheld remote); the eventual physical control
source.

Non-goals (deferred / child-level): the specific physical-panel build;
transports beyond local wifi; access control/auth; migrating rendering off the
laptop (e.g. to a Pi); multi-room / multi-screen fan-out.

Assumptions · constraints · dependencies

Builds on the existing simulator + alteration engine; honors the standing
simulator-first directive (prototype the controller in software before
hardware).
The engineering design already exists:
docs/superpowers/specs/2026-06-26-networked-control-surface-design.md (session
0019). Its control inventory should be reconciled with the then-current
simulator surface at implementation time (the content/mode surface has evolved
since — see session 0020's video+audio toggles).
Implement later — captured now as backlog, not scheduled.

Source / signal

Capture session human-experience-filter-art-0025. Originates from the session
0019 brainstorming and its design spec
docs/superpowers/specs/2026-06-26-networked-control-surface-design.md.

Priority (WSJF)

Value: medium-high (unblocks the whole hardware/installation path) ·
Time-criticality: low (deferred, experience-first) · Opportunity-enablement: high
(every controller depends on it) ÷ Size: large → provisional priority/P2.

Decomposition (one level — child Features)

Remote controller simulator (iPad web twin) — first / now-buildable. The
software remote on a separate device + the server-side networked-control
machinery it needs. Carries the design spec's v1 scope.
Physical control panel — later. A hardware control source (Arduino /
ESP32) over the same contract; reframes ROADMAP sub-project 4.
(Possible, deferred) Renderer migration / multi-renderer — move rendering
off the laptop and/or drive multiple surfaces; the contract already allows it.

Rolling wave: only Feature 1 is carved now; Features 2–3 stay high-level until
their wave.

Solution notes (optional — NOT a design, non-binding)

The design spec details the engineering approach: one server-authoritative session
state, a live push to the renderer, and a single control-source contract any
device posts to (software remote now, hardware later — e.g. wifi-capable
microcontroller posting directly, or a classic board via a small bridge). Captured
here only as orientation; the binding design is the spec.

# Networked control surface — drive the experience from a separate control device ## Summary The installation's experience can be driven by a **separate controller device over the local network**, decoupled from the machine that renders it — enabling a handheld software remote now and a physical control panel later, with no change to the renderer when the controller changes. ## Problem / pain Today the controls and the rendered experience are **the same browser tab on one machine**: to change altitude, left/right, mood, or content you must stand at the rendering computer and operate on-screen sliders. For an installation this is backwards — the person experiencing the piece can't drive it from where they're standing, the operator can't tune it from across the room, and there is **no path to a physical control panel** without re-architecting, because the control surface and the screen are fused. Every future controller (a handheld, real knobs) is blocked on that coupling today. ## Who feels it | persona | role | what they need | | --- | --- | --- | | Operator / artist | runs and tunes the installation | to drive and adjust the experience from where they stand, not chained to the render machine | | Participant (later) | experiences the piece | to influence the experience from a handheld/panel in the room | | Hardware builder | builds the eventual physical panel | a stable seam so a new controller drops in without touching the renderer | ## Desired outcome / value Control is **decoupled from display**. The experience is driven by any "control source" over the network, so a handheld remote works now *and* a physical panel drops in later **without re-architecting the renderer** — proving the path to the real installation while keeping today's simulator-first workflow. ## Acceptance — "done right" - A person changes the experience (altitude, left, right, mood, content) from a **separate device** while the rendering machine only renders. - Successive/concurrent control sources are **interchangeable** — swapping one for another needs no renderer change. - Slow, human-perceptible changes (e.g. altitude/content transitions) give **honest feedback** on the controller while they play. - The path to physical hardware is **proven**: a later hardware controller speaks the same contract and needs no renderer or server redesign. ## Scope / non-goals **In scope (across child Features):** decoupling control from display; the networked control contract every control source speaks; transition/busy feedback; the first software control source (handheld remote); the eventual physical control source. **Non-goals (deferred / child-level):** the specific physical-panel build; transports beyond local wifi; access control/auth; migrating rendering off the laptop (e.g. to a Pi); multi-room / multi-screen fan-out. ## Assumptions · constraints · dependencies - Builds on the existing simulator + alteration engine; honors the standing **simulator-first** directive (prototype the controller in software before hardware). - The engineering design already exists: `docs/superpowers/specs/2026-06-26-networked-control-surface-design.md` (session 0019). Its control inventory should be **reconciled with the then-current simulator surface** at implementation time (the content/mode surface has evolved since — see session 0020's video+audio toggles). - Implement **later** — captured now as backlog, not scheduled. ## Source / signal Capture session `human-experience-filter-art-0025`. Originates from the session 0019 brainstorming and its design spec `docs/superpowers/specs/2026-06-26-networked-control-surface-design.md`. ## Priority (WSJF) Value: medium-high (unblocks the whole hardware/installation path) · Time-criticality: low (deferred, experience-first) · Opportunity-enablement: high (every controller depends on it) ÷ Size: large → **provisional priority/P2**. ## Decomposition (one level — child Features) 1. **Remote controller simulator (iPad web twin)** — *first / now-buildable.* The software remote on a separate device + the server-side networked-control machinery it needs. Carries the design spec's v1 scope. 2. **Physical control panel** — *later.* A hardware control source (Arduino / ESP32) over the same contract; reframes ROADMAP sub-project 4. 3. *(Possible, deferred)* **Renderer migration / multi-renderer** — move rendering off the laptop and/or drive multiple surfaces; the contract already allows it. Rolling wave: only Feature 1 is carved now; Features 2–3 stay high-level until their wave. ## Solution notes (optional — NOT a design, non-binding) The design spec details the engineering approach: one server-authoritative session state, a live push to the renderer, and a single control-source contract any device posts to (software remote now, hardware later — e.g. wifi-capable microcontroller posting directly, or a classic board via a small bridge). Captured here only as orientation; the binding design is the spec.

benstull added the type/epic priority/P2 labels 2026-06-27 13:16:23 +00:00

benstull referenced this issue

2026-06-27 13:16:34 +00:00

Remote controller simulator — drive the experience from an iPad web twin over wifi #27

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: benstull/human-experience-filter-art#26