Merge pull request 'docs(design): machine-altered-perception thesis revision (session 0005)' (#4) from feature/machine-altered-perception-spec into main
This commit was merged in pull request #4.
This commit is contained in:
@@ -0,0 +1,360 @@
|
||||
# Human Experience Filter — Machine-Altered Perception (Design Revision)
|
||||
|
||||
**Date:** 2026-06-05
|
||||
**Status:** Approved design (pre-implementation)
|
||||
**Repo:** `human-experience-filter-art`
|
||||
**Supersedes:** the thesis (§1), selection model (§3), tagging division (§5), and
|
||||
sourcing strategy (§8) of
|
||||
[`2026-06-04-human-experience-filter-design.md`](./2026-06-04-human-experience-filter-design.md),
|
||||
and substantially expands its control panel (§2 selector) and hardware (§6). The
|
||||
coordinate model (§2 axes) and the single-panoramic-projector display are
|
||||
**preserved**.
|
||||
|
||||
> **Why this revision exists.** The 2026-06-04 design framed the piece as an
|
||||
> *experience filter over found human artifacts* ("curation is the artwork; not
|
||||
> algorithmically generated filler"). This revision **changes the artistic
|
||||
> thesis** to one about **how humans interact with machines and how that interaction
|
||||
> reshapes their nervous systems** — and in doing so makes machine alteration of
|
||||
> the imagery the *subject* of the piece rather than a betrayal of it.
|
||||
|
||||
---
|
||||
|
||||
## 1. What this is (the new thesis)
|
||||
|
||||
A single-viewer immersive installation about **a machine reshaping your
|
||||
perception — for better and for worse.**
|
||||
|
||||
One person sits in a chair in a small room. A single panoramic projector wraps the
|
||||
walls they face. The source is **neutral**: real, calm, public-domain nature
|
||||
footage that sits at the *center* of the experience-space — neither analytical nor
|
||||
emotional, neither bright nor dark. The viewer turns knobs on a wooden control
|
||||
panel, and **the machine alters that neutral reality toward whatever they dial
|
||||
in.** The piece is the felt, bodily experience of a machine bending your inner
|
||||
state — and crucially it bends **both ways**: it can soothe you or disturb you,
|
||||
clarify the world or manipulate it, name your feelings or colonize them.
|
||||
|
||||
You don't read that thesis on a wall label. You feel it in your nervous system as
|
||||
your own hand moves the knobs.
|
||||
|
||||
Design constraints (carried from the original, one relaxed):
|
||||
|
||||
- **One viewer, one room, all-local at runtime.** No networking, no streaming, no
|
||||
multi-user, no session recording. (All AI alteration happens **offline at
|
||||
authoring time**; the room plays pre-built media + cheap runtime grading.)
|
||||
- **Real footage as the substrate.** The base library is real public-domain
|
||||
nature footage. The machine *alters* it — it does not invent from nothing. This
|
||||
preserves real-world motion/composition (and some lineage to the original
|
||||
found-media thesis) while making the alteration the point. *(This relaxes the
|
||||
original "no generated content" rule: generation/alteration is now the
|
||||
subject.)*
|
||||
- **Built to grow.** A small, deliberate base library; the data model and tooling
|
||||
designed to expand.
|
||||
|
||||
---
|
||||
|
||||
## 2. What changes vs. 2026-06-04
|
||||
|
||||
| Area | 2026-06-04 | This revision |
|
||||
|---|---|---|
|
||||
| Thesis (§1) | filter over found human artifacts | **machine reshapes perception, good & bad** |
|
||||
| Content | large found library pre-matched to coordinates | **small NEUTRAL base library, machine-altered to the knob state** |
|
||||
| Engine (§3) | nearest-match **lookup** in a tagged catalog | **transform from neutral** (knobs = a transformation, not a lookup) |
|
||||
| Sourcing (§8) | find media that already matches each cell | source **neutral** clips; the machine produces the variants |
|
||||
| Control panel (§2) | 4-way mode + 4 knobs | **7-way content dial + 4 experience knobs + volume + brightness**, on a tactile wooden panel |
|
||||
| Display | single pano projector | **unchanged** |
|
||||
| Coordinate axes (§2) | Left/Right/Dark/Light, two planes | **unchanged** (reinterpreted as transform targets) |
|
||||
|
||||
---
|
||||
|
||||
## 3. Coordinate model (preserved, reinterpreted)
|
||||
|
||||
The four experience knobs and two planes from the 2026-06-04 §2 are unchanged:
|
||||
|
||||
- **Brain plane** = Left × Right. Left = analytical / verbal / structured. Right =
|
||||
artistic / emotional / abstract. Both high → "whole brain."
|
||||
- **Mood plane** = Dark × Light. Dark = somber / heavy. Light = uplifting / serene.
|
||||
|
||||
Each knob is `0–4`. What changes is the *meaning of a knob position*: it is no
|
||||
longer "find the catalog piece nearest this coordinate" but **"how hard the machine
|
||||
pushes the neutral base toward this pole."** `(2,2,2,2)` is the neutral origin —
|
||||
the un-altered base clip.
|
||||
|
||||
---
|
||||
|
||||
## 4. The alteration engine (replaces the §3 selection algorithm)
|
||||
|
||||
The engine's job changes from **choosing** a clip to **altering** one. Given a
|
||||
neutral base clip and a knob vector `(left, right, dark, light)`, it applies a
|
||||
**transformation toward each active pole** and composites the result.
|
||||
|
||||
### 4.1 Per-axis alteration
|
||||
|
||||
| Pole | What the machine does | How |
|
||||
|---|---|---|
|
||||
| **Left** (analytical/verbal) | **imposes analytical structure** on the scene — annotation, measurement, labels, grids, tracking boxes, data HUD; the machine *dissecting* the world into data | **overlay** layer composited on top |
|
||||
| **Right** (artistic/emotional) | **dissolves realism** toward painterly / dreamlike / abstract | **generative video-to-video (v2v)** restyle of the pixels |
|
||||
| **Dark** (somber/heavy) | drains toward the cool/melancholy/night grade (see §5) | **color grade** (deterministic) |
|
||||
| **Light** (uplifting/serene) | warms toward the bright/serene grade (see §5) | **color grade** (deterministic) |
|
||||
|
||||
### 4.2 The composition rule
|
||||
|
||||
The four axes are **two kinds of operation** that **layer** rather than blend-and-
|
||||
cancel:
|
||||
|
||||
- **Substrate transforms** — Right (v2v restyle) + Dark + Light (color grades).
|
||||
They alter the **pixels** and blend with each other.
|
||||
- **Overlay** — Left (annotation / labels / HUD). It composites **on top** of
|
||||
whatever the substrate is.
|
||||
|
||||
So Left and Right are **not** mutually-exclusive opposites; they stack on different
|
||||
layers. This single rule covers the whole `5×5×5×5` space:
|
||||
|
||||
- **Max Left + Max Right** = maximally dissolved dreamlike substrate **with the
|
||||
machine's labels on the feelings themselves** — *"awe," "grief," "detected:
|
||||
longing 0.82,"* tracking boxes around emotional content. This is the "whole
|
||||
brain" corner (§3): the machine that both **induces** emotion and
|
||||
**names/quantifies** it — affective computing made visceral. (Good/bad duality:
|
||||
is being told your own feeling clarifying, or invasive?)
|
||||
- **Dark + analytical** = a drained, melancholy scene with cold measurement
|
||||
overlaid.
|
||||
- **Light + emotional** = a warm, dreamlike wash, unlabeled.
|
||||
|
||||
### 4.3 Where each transform runs
|
||||
|
||||
- **Runtime, on the Pi (free, continuous, full-res):** the Dark/Light color grade
|
||||
and the Left analytical overlay. These are cheap (LUT/curves + luma key, and
|
||||
text/graphics compositing) and can move continuously with the knob.
|
||||
- **Pre-baked offline (the only AI cost):** the Right generative v2v restyle.
|
||||
Generative v2v cannot run real-time on a Pi, so a small set of restyled variants
|
||||
is rendered offline and selected/blended at runtime.
|
||||
|
||||
> **Crucial consequence — the Left labels are a RUNTIME OVERLAY, never baked into
|
||||
> the v2v pixels.** This is what makes multilingual support nearly free (§10).
|
||||
|
||||
---
|
||||
|
||||
## 5. Mood-axis color grade
|
||||
|
||||
The Dark↔Light knob is a three-stop color ramp whose **center is the identity** —
|
||||
the raw, ungraded footage, no color transform applied. Turning toward a pole grades
|
||||
*away* from the raw clip. Hue carries emotional valence; the **negative-space
|
||||
value** (a luma-keyed grade — the brightest/darkest regions pushed toward
|
||||
white/black) carries the literal light/dark reading, so the image *is* lighter or
|
||||
darker even before the hue registers.
|
||||
|
||||
| Mood knob | Hue | Negative space | Feeling |
|
||||
|---|---|---|---|
|
||||
| **Light** | warm **yellow** | → **white** | uplifting, serene, airy |
|
||||
| **Center** | **none — raw, ungraded video** | unchanged | the un-altered base |
|
||||
| **Dark** | cool **blue** | → **black** | melancholy, somber, night, cold |
|
||||
|
||||
*(Red/alarm was considered for Dark and set aside: red reads as threat/alarm, not
|
||||
the intended melancholy. Dark is the cool, mournful blue. The center applies no
|
||||
grade at all — it is simply the raw clip.)*
|
||||
|
||||
---
|
||||
|
||||
## 6. Content model — the 7-way content dial
|
||||
|
||||
The 2026-06-04 4-way selector (None/Audio/Video/A+V) expands to **seven positions**,
|
||||
letting the viewer choose channel **and audio source**:
|
||||
|
||||
| Position | Audio source | Video |
|
||||
|---|---|---|
|
||||
| **Off** | — | black |
|
||||
| **White Noise** | generated white/pink noise (not from catalog) | black |
|
||||
| **Music** | public-domain classical (Musopen pool) | black |
|
||||
| **Audio Track** | the nature clip's **own** audio | black |
|
||||
| **Video** | — | the (altered) nature video |
|
||||
| **Music + Video** | public-domain classical (Musopen pool) | the (altered) nature video |
|
||||
| **Audio Track + Video** | the nature clip's own audio | the (altered) nature video |
|
||||
|
||||
Notes:
|
||||
|
||||
- "Music" (classical) and "Audio Track" (the clip's own field audio) are now
|
||||
**distinct sources**; "White Noise" is **generated** at runtime, not a catalog
|
||||
record. This touches the catalog/`mode` model (the 2026-06-04 §4 schema), which
|
||||
must represent audio *source* in addition to channel.
|
||||
- **Off** is the void/rest state: black walls, silence.
|
||||
- Audio alteration (steering the *sound* by knob, parallel to the video) is **out
|
||||
of scope for this revision** — the experience knobs alter video; audio is chosen
|
||||
by source only. (Future work.)
|
||||
|
||||
---
|
||||
|
||||
## 7. Intensity controls
|
||||
|
||||
Two knobs, orthogonal to the four experience axes, exist so a viewer can take the
|
||||
experience at **whatever intensity they want**:
|
||||
|
||||
- **Volume** — audio output level.
|
||||
- **Brightness** — projector output luminance.
|
||||
|
||||
These are pure output levels, distinct from the Dark/Light *mood grade* (which
|
||||
changes hue and tone, not raw output). Brightness is "how much light hits the
|
||||
wall"; Light is "how the world is colored."
|
||||
|
||||
---
|
||||
|
||||
## 8. Content sourcing — neutral base library
|
||||
|
||||
The library is now **small and neutral**. Instead of hunting for footage that
|
||||
matches 625 coordinate cells, source a deliberate set of **neutral, calm,
|
||||
real public-domain nature clips** (centered on all four axes) and let the machine
|
||||
produce every variant.
|
||||
|
||||
- **Reuse sub-project 2** (the built ingest/tagging/review tooling) to pull
|
||||
neutral public-domain nature footage and populate mechanical fields.
|
||||
- **License stance** unchanged: prefer Public Domain / CC0; record license +
|
||||
source per clip. Altered/generated variants record the model and that they are
|
||||
machine-derived.
|
||||
- **Size:** dozens of neutral base clips (not hundreds), each yielding many
|
||||
altered variants. Short, seamless loops (≈20–60 s) suit a looping installation
|
||||
and are the v2v models' native sweet spot.
|
||||
|
||||
---
|
||||
|
||||
## 9. Economics (why this is feasible)
|
||||
|
||||
At the original scale (120–800 pieces × ~10-min segments) AI involvement was
|
||||
~$15k–45k+ and not viable. Under this design it drops to **a few hundred to a few
|
||||
thousand dollars**, because:
|
||||
|
||||
- The library is small (generate-to-order, not find-to-cover).
|
||||
- Pieces are short seamless loops (cheaper **and** higher-quality for the models).
|
||||
- Only the **Right** transform uses paid generative v2v; Dark/Light/Left are free
|
||||
deterministic runtime operations.
|
||||
|
||||
Indicative offline v2v build (2026 rates — Kling 3.0 / Sora 2 Standard ≈ $0.10/s,
|
||||
Veo 3.1 Lite / Runway Turbo ≈ $0.05/s; ×~4 retry yield): **~$300–3k** for a small
|
||||
base library × a handful of restyle variants. A flat authoring subscription
|
||||
(Runway $28–76/mo, Kling from ~$6/mo) may be cheaper than per-second API for a
|
||||
one-time build.
|
||||
|
||||
---
|
||||
|
||||
## 10. Accessibility, i18n, and the translation-cost finding
|
||||
|
||||
The piece is operable **blind, in the dark, in your language**, via four redundant
|
||||
channels on the control panel: **touch** (engraved symbol shape), **low-light
|
||||
color** (LEDs), **braille**, and **audio** (a read-aloud button on a small *local*
|
||||
speaker, separate from the main audio system).
|
||||
|
||||
Because "the human experience from a communication perspective is not universal" is
|
||||
itself on-thesis, the piece supports **many languages**. The cost of that hinges on
|
||||
one architectural decision:
|
||||
|
||||
- **❌ Labels baked into the v2v render** → re-render every label-bearing variant
|
||||
per language → video cost **× number of languages** (≈ $9k raw / $27–45k with
|
||||
retries for ~50 languages). A budget landmine.
|
||||
- **✅ Labels as a runtime overlay** (which §4.2 already requires) → the v2v
|
||||
substrate is rendered **once**, language-agnostic; a new language is a translated
|
||||
string table + font + TTS voice. **Marginal video cost per language ≈ $0.**
|
||||
|
||||
So multilingual support costs only cheap text work: machine-translating a small
|
||||
label/script vocabulary into ~100 languages is a few dollars; optional human review
|
||||
of a curated tier is ~$0.10/word (~$50–200/language); TTS for the read-aloud button
|
||||
is pennies per language. **Net: supporting *every* language is essentially free on
|
||||
the expensive part, provided labels stay a runtime overlay.**
|
||||
|
||||
---
|
||||
|
||||
## 11. The physical control panel
|
||||
|
||||
A **wood** panel. Each control has a **tactile engraved symbol** below it —
|
||||
distinguishable by fingertip in the dark, and shaped to evoke the control's
|
||||
meaning — with **braille** beneath, a **dim color-coded LED** backlighting the
|
||||
engraving, and a **read-aloud button** that speaks the control's function on a
|
||||
small local speaker.
|
||||
|
||||
Layout principle: the **four experience knobs are two opposing pairs**, with the
|
||||
opposition encoded *in the touch* (angular vs flowing, heavy-down vs radiant-up).
|
||||
The **utility controls** use a more conventional AV shape-language so they are
|
||||
never confused with the experience four.
|
||||
|
||||
| Control | Engraved symbol | Touch | LED |
|
||||
|---|---|---|---|
|
||||
| **Left** (analytical) | parallel straight ridges / grid `≡` `▦` | flat, hard, orthogonal edges | — |
|
||||
| **Right** (emotional) | spiral / flowing wave `∿` | one continuous curve, no corners | — |
|
||||
| **Dark** (somber) | crescent / downward wedge `☾` `▼` | smooth solid mass, sinking | **blue** |
|
||||
| **Light** (serene) | radial sunburst `☀` | center with radial spikes | **yellow / white** |
|
||||
| **Content dial** (7) | per-position micro-icons: Off `○` · White-Noise `∴` · Music `♪` · Audio-Track `)))` · Video `▷` · Music+Video `♪▷` · A+V `▷)))` | detented clicks | neutral |
|
||||
| **Volume** | growing wedge `◁` | a slope that thickens | neutral |
|
||||
| **Brightness** | half-disc contrast `◐` | a circle, half raised | neutral |
|
||||
|
||||
Design notes:
|
||||
|
||||
- **Light vs Brightness** is the one collision trap (both are "bright"). Light =
|
||||
radial sunburst (a *mood*, glows the hue it imparts); Brightness = half-disc
|
||||
contrast (an *output level*) — distinct to finger and eye.
|
||||
- **LEDs equal the hue the knob imparts:** Dark glows blue, Light glows
|
||||
yellow/white, the mood center is neutral — tying the panel to the projection.
|
||||
- **Negative-space symbols:** symbols differ by gross **edge profile** (curve vs
|
||||
corner vs radial vs slope), not fine detail, so they read under a fingertip.
|
||||
|
||||
This substantially fleshes out **sub-project 4 (Arduino firmware / control panel)**,
|
||||
previously only sketched: 7 controls + a read-aloud button + LEDs + braille, with
|
||||
the Arduino reading positions and the read-aloud/LED behavior coordinated with the
|
||||
Pi.
|
||||
|
||||
---
|
||||
|
||||
## 12. Hardware (revises §6)
|
||||
|
||||
Unchanged from the 2026-06-04 single-pano revision except for the richer panel:
|
||||
|
||||
- **Arduino** — reads the 7-way content dial, 4 experience knobs, volume,
|
||||
brightness, and the read-aloud button; drives the panel LEDs; sends values to the
|
||||
Pi over USB serial. (Bigger than the original 5-value panel.)
|
||||
- **Raspberry Pi 5 (or mini-PC)** — holds the base library + altered variants,
|
||||
runs the alteration engine (runtime grading + overlay compositing + variant
|
||||
selection), drives the single panoramic projector, plays audio, and serves the
|
||||
read-aloud clips to the local panel speaker.
|
||||
- **Single panoramic projector** spanning the three walls — **unchanged**.
|
||||
- **Small local speaker** on/near the panel for the read-aloud button (separate
|
||||
from the main audio).
|
||||
|
||||
---
|
||||
|
||||
## 13. Roadmap impact
|
||||
|
||||
- **Sub-project 2 (ingest/tagging)** — reused to source the **neutral** base
|
||||
clips; coordinate *drafting* matters less (the base is neutral by selection), but
|
||||
mechanical tagging + license capture still apply; add a flag for "neutral base"
|
||||
vs "altered variant," and model audio *source*.
|
||||
- **Sub-project 3 (player)** — grows an **alteration engine**: runtime color grade
|
||||
(luma-keyed), runtime analytical-overlay compositor (multilingual, from string
|
||||
tables), generated white noise, and selection/blending of pre-baked v2v variants.
|
||||
- **Sub-project 4 (firmware/panel)** — grows from a 5-value panel to the full
|
||||
tactile/accessible panel in §11.
|
||||
- **New offline pipeline** — author neutral→restyled v2v variants (the only paid
|
||||
AI step) and the label/string tables + TTS per language.
|
||||
|
||||
---
|
||||
|
||||
## 14. Open questions (for the plan, not blockers)
|
||||
|
||||
- **Pano resolution strategy** — v2v models output ~16:9 720p–1080p; the three-wall
|
||||
pano wants ultra-wide/high-res. Generate-at-max + upscale, native-ultrawide
|
||||
tooling, or accept the look?
|
||||
- **Negative-space luma key** — exact curve/mask for "brights→white, darks→black"
|
||||
on arbitrary nature footage.
|
||||
- **Transform composition order** — order of grade vs v2v substrate when several
|
||||
poles are active; how variant blending interpolates between pre-baked restyle
|
||||
levels.
|
||||
- **Neutral base count** — how many base clips for a satisfying launch.
|
||||
- **Language tier** — which languages get human-reviewed translation vs
|
||||
machine-only; default UI language and switching.
|
||||
- **Runtime grading performance** — confirm the Pi can do luma-keyed grade +
|
||||
overlay compositing at projector resolution/framerate (GPU shaders via mpv/ffmpeg).
|
||||
- **`approved`-only enforcement** in the player (carried from the original).
|
||||
|
||||
---
|
||||
|
||||
## 15. Out of scope (YAGNI)
|
||||
|
||||
- Networking / streaming at runtime — all alteration is offline; the room is local.
|
||||
- Runtime/on-demand generation (breaks all-local; slow; unbounded cost).
|
||||
- Audio alteration by knob (video-only this revision).
|
||||
- Multi-user / multi-viewer; session recording or analytics.
|
||||
- Automatic ML coordinate tagging — base clips are chosen neutral by hand.
|
||||
Reference in New Issue
Block a user