2026-06-06 00:49:42 +00:00
1 changed files with 360 additions and 0 deletions
@@ -0,0 +1,360 @@
+# Human Experience Filter — Machine-Altered Perception (Design Revision)
+
+**Date:** 2026-06-05
+**Status:** Approved design (pre-implementation)
+**Repo:** `human-experience-filter-art`
+**Supersedes:** the thesis (§1), selection model (§3), tagging division (§5), and
+sourcing strategy (§8) of
+[`2026-06-04-human-experience-filter-design.md`](./2026-06-04-human-experience-filter-design.md),
+and substantially expands its control panel (§2 selector) and hardware (§6). The
+coordinate model (§2 axes) and the single-panoramic-projector display are
+**preserved**.
+
+> **Why this revision exists.** The 2026-06-04 design framed the piece as an
+> *experience filter over found human artifacts* ("curation is the artwork; not
+> algorithmically generated filler"). This revision **changes the artistic
+> thesis** to one about **how humans interact with machines and how that interaction
+> reshapes their nervous systems** — and in doing so makes machine alteration of
+> the imagery the *subject* of the piece rather than a betrayal of it.
+
+---
+
+## 1. What this is (the new thesis)
+
+A single-viewer immersive installation about **a machine reshaping your
+perception — for better and for worse.**
+
+One person sits in a chair in a small room. A single panoramic projector wraps the
+walls they face. The source is **neutral**: real, calm, public-domain nature
+footage that sits at the *center* of the experience-space — neither analytical nor
+emotional, neither bright nor dark. The viewer turns knobs on a wooden control
+panel, and **the machine alters that neutral reality toward whatever they dial
+in.** The piece is the felt, bodily experience of a machine bending your inner
+state — and crucially it bends **both ways**: it can soothe you or disturb you,
+clarify the world or manipulate it, name your feelings or colonize them.
+
+You don't read that thesis on a wall label. You feel it in your nervous system as
+your own hand moves the knobs.
+
+Design constraints (carried from the original, one relaxed):
+
+- **One viewer, one room, all-local at runtime.** No networking, no streaming, no
+  multi-user, no session recording. (All AI alteration happens **offline at
+  authoring time**; the room plays pre-built media + cheap runtime grading.)
+- **Real footage as the substrate.** The base library is real public-domain
+  nature footage. The machine *alters* it — it does not invent from nothing. This
+  preserves real-world motion/composition (and some lineage to the original
+  found-media thesis) while making the alteration the point. *(This relaxes the
+  original "no generated content" rule: generation/alteration is now the
+  subject.)*
+- **Built to grow.** A small, deliberate base library; the data model and tooling
+  designed to expand.
+
+---
+
+## 2. What changes vs. 2026-06-04
+
+| Area | 2026-06-04 | This revision |
+|---|---|---|
+| Thesis (§1) | filter over found human artifacts | **machine reshapes perception, good & bad** |
+| Content | large found library pre-matched to coordinates | **small NEUTRAL base library, machine-altered to the knob state** |
+| Engine (§3) | nearest-match **lookup** in a tagged catalog | **transform from neutral** (knobs = a transformation, not a lookup) |
+| Sourcing (§8) | find media that already matches each cell | source **neutral** clips; the machine produces the variants |
+| Control panel (§2) | 4-way mode + 4 knobs | **7-way content dial + 4 experience knobs + volume + brightness**, on a tactile wooden panel |
+| Display | single pano projector | **unchanged** |
+| Coordinate axes (§2) | Left/Right/Dark/Light, two planes | **unchanged** (reinterpreted as transform targets) |
+
+---
+
+## 3. Coordinate model (preserved, reinterpreted)
+
+The four experience knobs and two planes from the 2026-06-04 §2 are unchanged:
+
+- **Brain plane** = Left × Right. Left = analytical / verbal / structured. Right =
+  artistic / emotional / abstract. Both high → "whole brain."
+- **Mood plane** = Dark × Light. Dark = somber / heavy. Light = uplifting / serene.
+
+Each knob is `0–4`. What changes is the *meaning of a knob position*: it is no
+longer "find the catalog piece nearest this coordinate" but **"how hard the machine
+pushes the neutral base toward this pole."** `(2,2,2,2)` is the neutral origin —
+the un-altered base clip.
+
+---
+
+## 4. The alteration engine (replaces the §3 selection algorithm)
+
+The engine's job changes from **choosing** a clip to **altering** one. Given a
+neutral base clip and a knob vector `(left, right, dark, light)`, it applies a
+**transformation toward each active pole** and composites the result.
+
+### 4.1 Per-axis alteration
+
+| Pole | What the machine does | How |
+|---|---|---|
+| **Left** (analytical/verbal) | **imposes analytical structure** on the scene — annotation, measurement, labels, grids, tracking boxes, data HUD; the machine *dissecting* the world into data | **overlay** layer composited on top |
+| **Right** (artistic/emotional) | **dissolves realism** toward painterly / dreamlike / abstract | **generative video-to-video (v2v)** restyle of the pixels |
+| **Dark** (somber/heavy) | drains toward the cool/melancholy/night grade (see §5) | **color grade** (deterministic) |
+| **Light** (uplifting/serene) | warms toward the bright/serene grade (see §5) | **color grade** (deterministic) |
+
+### 4.2 The composition rule
+
+The four axes are **two kinds of operation** that **layer** rather than blend-and-
+cancel:
+
+- **Substrate transforms** — Right (v2v restyle) + Dark + Light (color grades).
+  They alter the **pixels** and blend with each other.
+- **Overlay** — Left (annotation / labels / HUD). It composites **on top** of
+  whatever the substrate is.
+
+So Left and Right are **not** mutually-exclusive opposites; they stack on different
+layers. This single rule covers the whole `5×5×5×5` space:
+
+- **Max Left + Max Right** = maximally dissolved dreamlike substrate **with the
+  machine's labels on the feelings themselves** — *"awe," "grief," "detected:
+  longing 0.82,"* tracking boxes around emotional content. This is the "whole
+  brain" corner (§3): the machine that both **induces** emotion and
+  **names/quantifies** it — affective computing made visceral. (Good/bad duality:
+  is being told your own feeling clarifying, or invasive?)
+- **Dark + analytical** = a drained, melancholy scene with cold measurement
+  overlaid.
+- **Light + emotional** = a warm, dreamlike wash, unlabeled.
+
+### 4.3 Where each transform runs
+
+- **Runtime, on the Pi (free, continuous, full-res):** the Dark/Light color grade
+  and the Left analytical overlay. These are cheap (LUT/curves + luma key, and
+  text/graphics compositing) and can move continuously with the knob.
+- **Pre-baked offline (the only AI cost):** the Right generative v2v restyle.
+  Generative v2v cannot run real-time on a Pi, so a small set of restyled variants
+  is rendered offline and selected/blended at runtime.
+
+> **Crucial consequence — the Left labels are a RUNTIME OVERLAY, never baked into
+> the v2v pixels.** This is what makes multilingual support nearly free (§10).
+
+---
+
+## 5. Mood-axis color grade
+
+The Dark↔Light knob is a three-stop color ramp whose **center is the identity** —
+the raw, ungraded footage, no color transform applied. Turning toward a pole grades
+*away* from the raw clip. Hue carries emotional valence; the **negative-space
+value** (a luma-keyed grade — the brightest/darkest regions pushed toward
+white/black) carries the literal light/dark reading, so the image *is* lighter or
+darker even before the hue registers.
+
+| Mood knob | Hue | Negative space | Feeling |
+|---|---|---|---|
+| **Light** | warm **yellow** | → **white** | uplifting, serene, airy |
+| **Center** | **none — raw, ungraded video** | unchanged | the un-altered base |
+| **Dark** | cool **blue** | → **black** | melancholy, somber, night, cold |
+
+*(Red/alarm was considered for Dark and set aside: red reads as threat/alarm, not
+the intended melancholy. Dark is the cool, mournful blue. The center applies no
+grade at all — it is simply the raw clip.)*
+
+---
+
+## 6. Content model — the 7-way content dial
+
+The 2026-06-04 4-way selector (None/Audio/Video/A+V) expands to **seven positions**,
+letting the viewer choose channel **and audio source**:
+
+| Position | Audio source | Video |
+|---|---|---|
+| **Off** | — | black |
+| **White Noise** | generated white/pink noise (not from catalog) | black |
+| **Music** | public-domain classical (Musopen pool) | black |
+| **Audio Track** | the nature clip's **own** audio | black |
+| **Video** | — | the (altered) nature video |
+| **Music + Video** | public-domain classical (Musopen pool) | the (altered) nature video |
+| **Audio Track + Video** | the nature clip's own audio | the (altered) nature video |
+
+Notes:
+
+- "Music" (classical) and "Audio Track" (the clip's own field audio) are now
+  **distinct sources**; "White Noise" is **generated** at runtime, not a catalog
+  record. This touches the catalog/`mode` model (the 2026-06-04 §4 schema), which
+  must represent audio *source* in addition to channel.
+- **Off** is the void/rest state: black walls, silence.
+- Audio alteration (steering the *sound* by knob, parallel to the video) is **out
+  of scope for this revision** — the experience knobs alter video; audio is chosen
+  by source only. (Future work.)
+
+---
+
+## 7. Intensity controls
+
+Two knobs, orthogonal to the four experience axes, exist so a viewer can take the
+experience at **whatever intensity they want**:
+
+- **Volume** — audio output level.
+- **Brightness** — projector output luminance.
+
+These are pure output levels, distinct from the Dark/Light *mood grade* (which
+changes hue and tone, not raw output). Brightness is "how much light hits the
+wall"; Light is "how the world is colored."
+
+---
+
+## 8. Content sourcing — neutral base library
+
+The library is now **small and neutral**. Instead of hunting for footage that
+matches 625 coordinate cells, source a deliberate set of **neutral, calm,
+real public-domain nature clips** (centered on all four axes) and let the machine
+produce every variant.
+
+- **Reuse sub-project 2** (the built ingest/tagging/review tooling) to pull
+  neutral public-domain nature footage and populate mechanical fields.
+- **License stance** unchanged: prefer Public Domain / CC0; record license +
+  source per clip. Altered/generated variants record the model and that they are
+  machine-derived.
+- **Size:** dozens of neutral base clips (not hundreds), each yielding many
+  altered variants. Short, seamless loops (≈20–60 s) suit a looping installation
+  and are the v2v models' native sweet spot.
+
+---
+
+## 9. Economics (why this is feasible)
+
+At the original scale (120–800 pieces × ~10-min segments) AI involvement was
+~$15k–45k+ and not viable. Under this design it drops to **a few hundred to a few
+thousand dollars**, because:
+
+- The library is small (generate-to-order, not find-to-cover).
+- Pieces are short seamless loops (cheaper **and** higher-quality for the models).
+- Only the **Right** transform uses paid generative v2v; Dark/Light/Left are free
+  deterministic runtime operations.
+
+Indicative offline v2v build (2026 rates — Kling 3.0 / Sora 2 Standard ≈ $0.10/s,
+Veo 3.1 Lite / Runway Turbo ≈ $0.05/s; ×~4 retry yield): **~$300–3k** for a small
+base library × a handful of restyle variants. A flat authoring subscription
+(Runway $28–76/mo, Kling from ~$6/mo) may be cheaper than per-second API for a
+one-time build.
+
+---
+
+## 10. Accessibility, i18n, and the translation-cost finding
+
+The piece is operable **blind, in the dark, in your language**, via four redundant
+channels on the control panel: **touch** (engraved symbol shape), **low-light
+color** (LEDs), **braille**, and **audio** (a read-aloud button on a small *local*
+speaker, separate from the main audio system).
+
+Because "the human experience from a communication perspective is not universal" is
+itself on-thesis, the piece supports **many languages**. The cost of that hinges on
+one architectural decision:
+
+- **❌ Labels baked into the v2v render** → re-render every label-bearing variant
+  per language → video cost **× number of languages** (≈ $9k raw / $27–45k with
+  retries for ~50 languages). A budget landmine.
+- **✅ Labels as a runtime overlay** (which §4.2 already requires) → the v2v
+  substrate is rendered **once**, language-agnostic; a new language is a translated
+  string table + font + TTS voice. **Marginal video cost per language ≈ $0.**
+
+So multilingual support costs only cheap text work: machine-translating a small
+label/script vocabulary into ~100 languages is a few dollars; optional human review
+of a curated tier is ~$0.10/word (~$50–200/language); TTS for the read-aloud button
+is pennies per language. **Net: supporting *every* language is essentially free on
+the expensive part, provided labels stay a runtime overlay.**
+
+---
+
+## 11. The physical control panel
+
+A **wood** panel. Each control has a **tactile engraved symbol** below it —
+distinguishable by fingertip in the dark, and shaped to evoke the control's
+meaning — with **braille** beneath, a **dim color-coded LED** backlighting the
+engraving, and a **read-aloud button** that speaks the control's function on a
+small local speaker.
+
+Layout principle: the **four experience knobs are two opposing pairs**, with the
+opposition encoded *in the touch* (angular vs flowing, heavy-down vs radiant-up).
+The **utility controls** use a more conventional AV shape-language so they are
+never confused with the experience four.
+
+| Control | Engraved symbol | Touch | LED |
+|---|---|---|---|
+| **Left** (analytical) | parallel straight ridges / grid `≡` `▦` | flat, hard, orthogonal edges | — |
+| **Right** (emotional) | spiral / flowing wave `∿` | one continuous curve, no corners | — |
+| **Dark** (somber) | crescent / downward wedge `☾` `▼` | smooth solid mass, sinking | **blue** |
+| **Light** (serene) | radial sunburst `☀` | center with radial spikes | **yellow / white** |
+| **Content dial** (7) | per-position micro-icons: Off `○` · White-Noise `∴` · Music `♪` · Audio-Track `)))` · Video `▷` · Music+Video `♪▷` · A+V `▷)))` | detented clicks | neutral |
+| **Volume** | growing wedge `◁` | a slope that thickens | neutral |
+| **Brightness** | half-disc contrast `◐` | a circle, half raised | neutral |
+
+Design notes:
+
+- **Light vs Brightness** is the one collision trap (both are "bright"). Light =
+  radial sunburst (a *mood*, glows the hue it imparts); Brightness = half-disc
+  contrast (an *output level*) — distinct to finger and eye.
+- **LEDs equal the hue the knob imparts:** Dark glows blue, Light glows
+  yellow/white, the mood center is neutral — tying the panel to the projection.
+- **Negative-space symbols:** symbols differ by gross **edge profile** (curve vs
+  corner vs radial vs slope), not fine detail, so they read under a fingertip.
+
+This substantially fleshes out **sub-project 4 (Arduino firmware / control panel)**,
+previously only sketched: 7 controls + a read-aloud button + LEDs + braille, with
+the Arduino reading positions and the read-aloud/LED behavior coordinated with the
+Pi.
+
+---
+
+## 12. Hardware (revises §6)
+
+Unchanged from the 2026-06-04 single-pano revision except for the richer panel:
+
+- **Arduino** — reads the 7-way content dial, 4 experience knobs, volume,
+  brightness, and the read-aloud button; drives the panel LEDs; sends values to the
+  Pi over USB serial. (Bigger than the original 5-value panel.)
+- **Raspberry Pi 5 (or mini-PC)** — holds the base library + altered variants,
+  runs the alteration engine (runtime grading + overlay compositing + variant
+  selection), drives the single panoramic projector, plays audio, and serves the
+  read-aloud clips to the local panel speaker.
+- **Single panoramic projector** spanning the three walls — **unchanged**.
+- **Small local speaker** on/near the panel for the read-aloud button (separate
+  from the main audio).
+
+---
+
+## 13. Roadmap impact
+
+- **Sub-project 2 (ingest/tagging)** — reused to source the **neutral** base
+  clips; coordinate *drafting* matters less (the base is neutral by selection), but
+  mechanical tagging + license capture still apply; add a flag for "neutral base"
+  vs "altered variant," and model audio *source*.
+- **Sub-project 3 (player)** — grows an **alteration engine**: runtime color grade
+  (luma-keyed), runtime analytical-overlay compositor (multilingual, from string
+  tables), generated white noise, and selection/blending of pre-baked v2v variants.
+- **Sub-project 4 (firmware/panel)** — grows from a 5-value panel to the full
+  tactile/accessible panel in §11.
+- **New offline pipeline** — author neutral→restyled v2v variants (the only paid
+  AI step) and the label/string tables + TTS per language.
+
+---
+
+## 14. Open questions (for the plan, not blockers)
+
+- **Pano resolution strategy** — v2v models output ~16:9 720p–1080p; the three-wall
+  pano wants ultra-wide/high-res. Generate-at-max + upscale, native-ultrawide
+  tooling, or accept the look?
+- **Negative-space luma key** — exact curve/mask for "brights→white, darks→black"
+  on arbitrary nature footage.
+- **Transform composition order** — order of grade vs v2v substrate when several
+  poles are active; how variant blending interpolates between pre-baked restyle
+  levels.
+- **Neutral base count** — how many base clips for a satisfying launch.
+- **Language tier** — which languages get human-reviewed translation vs
+  machine-only; default UI language and switching.
+- **Runtime grading performance** — confirm the Pi can do luma-keyed grade +
+  overlay compositing at projector resolution/framerate (GPU shaders via mpv/ffmpeg).
+- **`approved`-only enforcement** in the player (carried from the original).
+
+---
+
+## 15. Out of scope (YAGNI)
+
+- Networking / streaming at runtime — all alteration is offline; the room is local.
+- Runtime/on-demand generation (breaks all-local; slow; unbounded cost).
+- Audio alteration by knob (video-only this revision).
+- Multi-user / multi-viewer; session recording or analytics.
+- Automatic ML coordinate tagging — base clips are chosen neutral by hand.