Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Ben Stull
|
3ef21fb7f7 |
@@ -0,0 +1,242 @@
|
||||
# Human Experience Filter — Simulator Alteration Preview (Design)
|
||||
|
||||
**Date:** 2026-06-06
|
||||
**Status:** Approved design (pre-implementation)
|
||||
**Repo:** `human-experience-filter-art`
|
||||
**Builds on:** the alteration engine of
|
||||
[`2026-06-05-machine-altered-perception-design.md`](./2026-06-05-machine-altered-perception-design.md)
|
||||
(the "design" below) and the simulator scaffold of
|
||||
[`2026-06-04-experience-simulator-design.md`](./2026-06-04-experience-simulator-design.md).
|
||||
**Revises the design:** §4.3 and §10 — the Left analytical HUD is **no longer a
|
||||
runtime overlay**; it is **baked into authored variant videos** (see §8 below).
|
||||
This is a deliberate trade of near-free multilingual support for authorial
|
||||
precision over the HUD.
|
||||
**Retires:** the simulator's selection-era "curator's X-ray" view and its
|
||||
`/api/select` + `/api/catalog/meta` endpoints (the selection model they
|
||||
visualize was superseded by the alteration model).
|
||||
|
||||
> **Why this exists.** Operator directive (session 0007): *build and design only
|
||||
> things that run in the simulator, and get the whole experience working the way
|
||||
> we like in the simulator before moving to hardware.* The slice-1 alteration
|
||||
> engine (PR #5) is pure logic with **no simulator surface** — you cannot yet turn
|
||||
> the experience knobs and *see* the result. This design brings the alteration
|
||||
> into the simulator so the look can be tuned and liked before any Pi/serial work.
|
||||
|
||||
---
|
||||
|
||||
## 1. Scope
|
||||
|
||||
A browser **alteration preview**: turn the four experience knobs and the content
|
||||
dial, and see the neutral base footage altered toward the knob state, in real
|
||||
time, on a looping clip. The purpose is to **tune the look** of the filter and to
|
||||
**settle the knob→strength calibration by eye** (the open §3-vs-§4.2/§5 question
|
||||
from session 0006).
|
||||
|
||||
**In scope**
|
||||
- Live preview of the four experience knobs (Left/Right/Dark/Light) on a looping
|
||||
base clip.
|
||||
- **Dark/Light** rendered as a live, deterministic runtime color grade.
|
||||
- **Left/Right** rendered by selecting a **pre-baked authored variant clip** from
|
||||
a 5×5 grid and crossfading on change.
|
||||
- A **calibration panel** that adjusts the grade curve live; the chosen values are
|
||||
baked back into `player/alteration.py` defaults.
|
||||
- A **RenderPlan readout** (the project's "X-ray" honesty): always show the exact
|
||||
numbers the engine produced.
|
||||
- The content dial's **video on/off** behavior (so "Off" goes to black), driven by
|
||||
the existing `resolve_content`.
|
||||
- Placeholder variant generation so the mechanism is testable before the operator
|
||||
authors real videos.
|
||||
|
||||
**Out of scope (this slice)** — real generative v2v; serial input / the 3⇄4
|
||||
framing contract; audio playback (music / white-noise / audio-track); the Pi/mpv
|
||||
runtime renderer; the full transition/crossfade timing engine; multilingual label
|
||||
tables; catalog-model changes. These remain later roadmap slices.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture — Python-canonical, thin browser renderer
|
||||
|
||||
`player/alteration.py` stays the **single source of truth** for the alteration
|
||||
math (handbook §4.2 — deterministic core, thin I/O). The browser owns only
|
||||
*rendering*. Data flow:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph Browser
|
||||
K[4 experience knobs +<br/>content dial +<br/>calibration sliders]
|
||||
V["<video> variant + canvas grade"]
|
||||
RO[RenderPlan readout]
|
||||
end
|
||||
subgraph FastAPI [simulator/app.py]
|
||||
EP["POST /api/alteration"]
|
||||
CL["GET /api/clips"]
|
||||
end
|
||||
ENG["player.alteration.plan_alteration(coord, calibration)"]
|
||||
K -- "debounced POST {controls, calibration}" --> EP
|
||||
EP --> ENG --> EP -- "RenderPlan {variant, grade}" --> V
|
||||
EP --> RO
|
||||
CL -- "base clip + variant manifest" --> V
|
||||
```
|
||||
|
||||
- The browser sends control/calibration changes (debounced) and receives a
|
||||
`RenderPlan`. Video filtering itself runs continuously in the browser; only a
|
||||
*plan recompute* makes a round-trip. On localhost these JSON round-trips are
|
||||
imperceptible.
|
||||
- "Bake the calibration winner in" = change the `Calibration` defaults in Python.
|
||||
Nothing in the browser is canonical.
|
||||
|
||||
---
|
||||
|
||||
## 3. The four axes — how each is rendered
|
||||
|
||||
| Axis | Engine output | Browser rendering |
|
||||
|---|---|---|
|
||||
| **Dark / Light** | `ColorGrade.tone` ∈ [−1, 1], center = identity | live color grade on the `<video>`: Light → warm/yellow + negative space toward white; Dark → cool/blue + negative space toward black; 0 = raw. (Confirmed direction in brainstorm.) |
|
||||
| **Left / Right** | `VariantRef(left, right)` | select the authored variant clip for `(left, right)`; **crossfade** when it changes; `(0,0)` → raw base. The analytical HUD (Left) and dreamlike restyle (Right) are **baked into the clip** (§8). |
|
||||
|
||||
Dark/Light is the only continuously-tunable axis in this slice; Left/Right is a
|
||||
discrete selection by coordinate.
|
||||
|
||||
---
|
||||
|
||||
## 4. The variant grid (per base clip)
|
||||
|
||||
Each base clip carries a 5×5 grid keyed by the two baked axes:
|
||||
|
||||
- Rows = **Left** (analytical / HUD), 0–4. Columns = **Right** (artistic /
|
||||
dreamlike restyle), 0–4.
|
||||
- `(0,0)` = the raw base clip — no authored file needed.
|
||||
- The other **24 cells are authored videos** (4 analytical-edge + 4 dreamlike-edge
|
||||
+ 16 combined core). This is **per base clip**; authoring load multiplies by the
|
||||
number of base clips (and by language, since the HUD is baked — §8).
|
||||
|
||||
The grid is **data**, not code: a manifest the simulator reads (see §6).
|
||||
|
||||
---
|
||||
|
||||
## 5. Engine reconciliation (`player/alteration.py`)
|
||||
|
||||
The slice-1 engine modeled Left/Right as continuous runtime layers
|
||||
(`AnalyticalOverlay.intensity`, `Restyle.blend`). Baked variants make that
|
||||
obsolete. Changes:
|
||||
|
||||
- **`RenderPlan`** becomes `{ grade: ColorGrade, variant: VariantRef }`.
|
||||
- `ColorGrade` — unchanged.
|
||||
- `VariantRef(left, right)` — new; identity selection by coordinate.
|
||||
- **Remove** `AnalyticalOverlay` and `Restyle` (their content is now baked into the
|
||||
variant clip) rather than leaving dead fields.
|
||||
- **`Calibration`** — a new frozen dataclass parameterizing the grade curve (e.g.
|
||||
`mood_center`, per-axis curve), defaulting to **today's exact behavior**
|
||||
(behavior-preserving). `plan_alteration(coord, calibration=DEFAULT_CALIBRATION)`.
|
||||
- **`player/state.py`** — the `CROSSFADE` trigger switches from "restyle changed"
|
||||
to "variant changed"; `LIVE_UPDATE` still covers a grade-only change. Video
|
||||
on/off fades unchanged.
|
||||
|
||||
These are framework-code changes that keep the engine pure and unit-tested; they
|
||||
do **not** bake any deployment-shape decision into the engine.
|
||||
|
||||
---
|
||||
|
||||
## 6. Data & endpoints
|
||||
|
||||
- **`simulator/clips.py`** (replaces `simulator/fixtures.py`): reads a base-clip +
|
||||
variant manifest. Each base clip: `{ id, title, base_file, license, source,
|
||||
variants: { "L,R": { file, model?, hud_lang? } } }`. Missing cells fall back to
|
||||
the raw base (and are flagged in the readout as "raw / unauthored").
|
||||
- **`POST /api/alteration`**: body `{ controls, calibration }` → `RenderPlan`
|
||||
(plus the `ContentResolution` from `resolve_content`, so the dial's video-on/off
|
||||
is honored). The endpoint calls the real `plan_alteration`.
|
||||
- **`GET /api/clips`**: the base-clip list + variant manifest for the active base.
|
||||
- Video files served as static assets from a sample-media directory.
|
||||
- **Removed**: `POST /api/select`, `GET /api/catalog/meta`.
|
||||
- `hef.selection` (the library, incl. `Coordinate`) is **untouched**; only the
|
||||
simulator's selection view/endpoints retire. (`ranked_candidates`, added for the
|
||||
old X-ray, may become unused by the simulator — noted, not removed here.)
|
||||
|
||||
---
|
||||
|
||||
## 7. Calibration tuning
|
||||
|
||||
The calibration panel exposes the grade-curve parameters (the two conventions in
|
||||
tension: `(2,2,2,2)`-centered vs the `value/4` reading currently implemented, plus
|
||||
curve shape). Changing them re-requests the plan and the footage responds live.
|
||||
When the operator is happy, the chosen values are written into
|
||||
`DEFAULT_CALIBRATION` in `player/alteration.py` and locked in by a unit test.
|
||||
|
||||
---
|
||||
|
||||
## 8. Baked HUD — the §4.3/§10 revision (accepted trade)
|
||||
|
||||
The design's §4.3/§10 kept the Left HUD a **runtime overlay** specifically so the
|
||||
v2v substrate is language-agnostic and multilingual support is near-free. This
|
||||
design **reverses that for the Left/Right plane**: the operator authors the HUD
|
||||
**into** the variant videos to get pixel-precise control over the HUD and the
|
||||
analytical↔feeling balance.
|
||||
|
||||
**Consequences (accepted):**
|
||||
- Multilingual support is **no longer near-free**: baked HUD text means a new
|
||||
language re-renders every label-bearing variant (the §10 "budget landmine").
|
||||
The piece is English-first; broad i18n is deferred/expensive.
|
||||
- The live Dark/Light grade is applied **on top** of the baked cell, so it **tints
|
||||
the baked HUD** too (a "Dark" mood cools/darkens the HUD). With baked overlays
|
||||
the grade cannot skip the HUD. Escape hatch: author HUD colors that survive
|
||||
grading, or (future) reinstate a runtime HUD layer.
|
||||
|
||||
The parent design's §4.3/§10 must be updated to point at this revision (a task for
|
||||
the implementation plan).
|
||||
|
||||
---
|
||||
|
||||
## 9. Bootstrapping before authored videos exist
|
||||
|
||||
The operator will author the 24 variants per base clip; none exist yet. So the
|
||||
mechanism is testable immediately, the build includes a **placeholder generator**:
|
||||
for at least one base clip, produce the 24 cells by burning the cell's `L,R` (and
|
||||
a stub HUD caption) into the base loop via ffmpeg. Real authored clips drop into
|
||||
the manifest with **no code change**.
|
||||
|
||||
---
|
||||
|
||||
## 10. Testing
|
||||
|
||||
- **`player/` unit tests:** `DEFAULT_CALIBRATION` is behavior-preserving vs the
|
||||
current helpers; the two calibration conventions produce the expected plans;
|
||||
`VariantRef` selection (incl. `(0,0)` → raw) and the `state.py` crossfade
|
||||
trigger on variant change.
|
||||
- **Simulator API tests** (rewrite `tests/test_simulator_api.py`):
|
||||
`POST /api/alteration` returns the engine's plan for given controls/calibration;
|
||||
`GET /api/clips` returns the manifest; the removed endpoints are gone.
|
||||
- **No browser/E2E automation** this slice (manual visual tuning is the point);
|
||||
the rendering JS is kept thin and the engine logic stays in tested Python.
|
||||
|
||||
---
|
||||
|
||||
## 11. What ships
|
||||
|
||||
- `player/alteration.py` (parameterized `Calibration`, `VariantRef`, slimmed
|
||||
`RenderPlan`) + `player/state.py` crossfade-trigger update.
|
||||
- `simulator/clips.py` (variant manifest) replacing `fixtures.py`.
|
||||
- `simulator/app.py` new endpoints; selection endpoints removed.
|
||||
- `simulator/static/` rewritten as the Player preview (variant `<video>` +
|
||||
crossfade, live grade, calibration panel, RenderPlan readout, content dial).
|
||||
- Placeholder-variant generator + a sample base clip.
|
||||
- Tests above; `docs/USER_GUIDE.md` "Playing with the simulator" rewritten;
|
||||
parent design §4.3/§10 pointer updated; `docs/ROADMAP.md` §3 updated.
|
||||
|
||||
---
|
||||
|
||||
## 12. Open questions (for the plan, not blockers)
|
||||
|
||||
- **Grade vs baked HUD interaction** — accepted that the grade tints the HUD;
|
||||
revisit only if it reads badly once real authored clips exist.
|
||||
- **Sample base clip** — pick one CC0 nature loop for the placeholder grid.
|
||||
- **Crossfade timing in the browser** — a simple opacity crossfade is enough for
|
||||
tuning; the real timing engine is a later slice.
|
||||
|
||||
---
|
||||
|
||||
## 13. Out of scope (YAGNI)
|
||||
|
||||
Real generative v2v; serial / firmware; audio playback; the Pi renderer; the full
|
||||
transition engine; multilingual tables; catalog-model changes; retiring
|
||||
`hef.selection.ranked_candidates`. All remain later roadmap work.
|
||||
@@ -1,236 +0,0 @@
|
||||
# HEF — Scales-of-Nature Library + Stabilized Right-Axis Pipeline (Design Revision)
|
||||
|
||||
**Date:** 2026-06-07
|
||||
**Status:** Approved design (pre-implementation) — operator-approved this session (0008)
|
||||
**Repo:** `human-experience-filter-art`
|
||||
**Refines:** [`2026-06-05-machine-altered-perception-design.md`](./2026-06-05-machine-altered-perception-design.md)
|
||||
— specifically its Right-axis pipeline (§4.1/§4.3), content sourcing/model (§6/§8),
|
||||
and economics (§9), and it **adds a scale-navigation control + zoom transitions**
|
||||
to the §2 selector / §11 control panel. The thesis (§1), coordinate model (§3),
|
||||
Dark/Light/Left treatment, and accessibility (§10) are **preserved**.
|
||||
**Grounded in:** a local proof-of-concept run this session on the operator's Mac
|
||||
mini (M4 Pro, 64 GB, MPS) — all numbers below are measured, not estimated.
|
||||
|
||||
> **Why this revision exists.** The 2026-06-05 design specified the Right axis as
|
||||
> "generative video-to-video restyle, pre-baked offline" and assumed that meant a
|
||||
> **paid cloud API** (§9 priced Kling/Sora/Veo/Runway at $0.05–0.10/s). A POC this
|
||||
> session established two things that change the design: (1) the Right restyle runs
|
||||
> **entirely locally and offline** on the operator's existing hardware, for the
|
||||
> cost of electricity; and (2) naïve per-frame restyle **boils/flickers** in a way
|
||||
> the operator found disorienting — disqualifying for a piece meant to be peaceful
|
||||
> — and the fix is **optical-flow keyframe propagation**. Separately, the operator
|
||||
> chose how the "scales of nature" idea enters the piece: as the curatorial theme
|
||||
> of a **small neutral base library**, not a single fixed journey.
|
||||
|
||||
---
|
||||
|
||||
## 1. The Right axis is a local, flow-stabilized restyle (refines §4.1, §4.3)
|
||||
|
||||
The §4.1 mapping is unchanged in spirit — **Right = dissolve realism toward
|
||||
painterly/dreamlike via generative video-to-video** — but the *implementation* is
|
||||
now pinned:
|
||||
|
||||
- **Engine:** Stable Diffusion **img2img** (POC used `stabilityai/sd-turbo`) run on
|
||||
**Apple MPS**, locally, offline, at authoring time. No cloud API.
|
||||
- **Temporal coherence is a hard requirement, not a nicety.** Per-frame img2img
|
||||
independently re-imagines each frame, producing a shimmering "boil" that reads as
|
||||
disorienting — the **opposite** of the piece's peaceful intent. This was caught
|
||||
in the POC and is now a named design constraint: *the Right substrate must be
|
||||
temporally coherent.*
|
||||
- **Stabilization: optical-flow keyframe propagation.** Fully stylize **keyframes**
|
||||
at a fixed interval; for in-between frames, **warp the previous stylized frame
|
||||
forward by optical flow** (so motion is continuous) and apply only a *light*
|
||||
diffusion refine. The flow warp removes the boil; periodic keyframes bound drift.
|
||||
(This is the EbSynth principle. Genuine `ebsynth`/`ezsynth` are NVIDIA/Windows-
|
||||
leaning and don't install cleanly on Apple Silicon, so the POC implemented the
|
||||
same idea directly with OpenCV Farneback flow + the existing diffusers pipeline.)
|
||||
|
||||
This stays consistent with §4.3's crucial invariant: **the Left analytical labels
|
||||
remain a runtime overlay, never baked into the restyled pixels** — the POC's Left
|
||||
HUD is composited deterministically on top, preserving the near-free i18n of §10.
|
||||
|
||||
### 1.1 Where each transform runs (updated §4.3 table)
|
||||
|
||||
| Pole | Operation | Where it runs | Measured cost (8 s, 1080p clip) |
|
||||
|---|---|---|---|
|
||||
| **Dark** | color grade | runtime, live on the Pi | ~2.4 s offline; live at runtime |
|
||||
| **Light** | color grade | runtime, live on the Pi | ~2.5 s offline; live at runtime |
|
||||
| **Left** | analytical overlay (HUD) | runtime, live on the Pi | ~2.2 s offline; live at runtime |
|
||||
| **Right** | local generative restyle + flow propagation | **pre-baked offline, locally** | ~2.7 min/restyle-strength |
|
||||
|
||||
The three deterministic axes are confirmed cheap enough to run **live**; only the
|
||||
Right restyle is pre-baked. A peaceful **deterministic** alternative for Right (soft
|
||||
edge-preserving smoothing + bloom, zero flicker by construction, ~5 s/clip) was
|
||||
prototyped and set aside — the operator preferred the true generative repaint once
|
||||
the flow stabilization made it calm. It remains a documented fallback.
|
||||
|
||||
### 1.2 Runtime label rendering & i18n (sharpens §10)
|
||||
|
||||
The Left analytical labels are drawn **live by the Pi as a 2D graphics overlay** —
|
||||
**not** baked into video, and specifically **not** a pre-rendered overlay *video*. A
|
||||
baked overlay would have to exist per language × per Left level × per scale, which
|
||||
re-introduces the "× number of languages" cost §10 exists to avoid. So the runtime
|
||||
path is:
|
||||
|
||||
- **Architecture.** Per base clip, an offline-authored **annotation track** (box
|
||||
positions, anchor points, which annotations appear at each Left level) referencing
|
||||
language-agnostic label **keys** (e.g. `detected.conifer`). Per language, a cheap
|
||||
**string table** (key → translated text) + font + TTS voice. At runtime the Pi
|
||||
reads the Left knob, selects the active annotations, **shapes** the current
|
||||
language's strings, and composites over the altered video — updating only on change
|
||||
(knob move, language switch, timeline cue), not every frame.
|
||||
- **Correctness needs a real shaping stack.** Rendering *every* language correctly
|
||||
(Arabic joining, Indic conjuncts, CJK, RTL) requires **Pango + HarfBuzz + Noto
|
||||
fonts**, not naïve text drawing. (The POC's HUD used ffmpeg `drawtext`/Menlo —
|
||||
Latin-only; it would mis-render complex scripts and is **not** the runtime path.)
|
||||
This shaping stack is the load-bearing requirement behind §10's "label things
|
||||
correctly."
|
||||
- **Feasibility.** This is OSD/subtitle-class compositing; a Pi 5 (VideoCore VII,
|
||||
GLES/Vulkan, hardware decode) handles it. Headroom at the *panoramic* resolution is
|
||||
the one unmeasured variable — see §6.
|
||||
|
||||
---
|
||||
|
||||
## 2. Content = a small NEUTRAL "scales of nature" library (refines §6, §8)
|
||||
|
||||
The operator's "cosmic zoom" concept (space → continents → birds → ocean → abyss →
|
||||
microscopic → galaxy) enters the piece **as a curatorial theme, not a fixed film.**
|
||||
|
||||
- **Structure:** a **small library (~4–6 to start) of calm, neutral base clips**,
|
||||
each drawn from a *different scale of nature* — e.g. an orbital Earth, a forest, a
|
||||
coral reef, the deep-sea abyss, the microscopic, the cosmos. The machine alters
|
||||
whichever clip is playing, exactly as for any neutral base.
|
||||
- **Why this and not a single stitched journey.** A literal galaxy→cell "how small
|
||||
we are" journey carries its **own** emotion (awe, cosmic insignificance) *before
|
||||
the machine acts*, which contradicts the §1 neutral-base thesis, and as a fixed
|
||||
film it becomes "a journey you watch" rather than "a reality you bend." Keeping the
|
||||
scales as the *theme of a neutral library* preserves the awe-of-scale richness and
|
||||
the piece's coherence **while keeping the base neutral and the experience
|
||||
interactive.** (Rejected alternatives: single stitched base; cosmic-zoom as
|
||||
intro/reset; rethinking the thesis.)
|
||||
- **Cost is not the constraint.** Per §4 below, ~5 base clips is ~1 hour of overnight
|
||||
local pre-bake — so this is an *artistic* choice, made on artistic grounds.
|
||||
- **Mechanism unchanged:** this slots into the existing §6 content model and the
|
||||
sub-project-2 ingest/tagging/review tooling; "scales of nature" is simply the
|
||||
selection principle for which neutral clips to source.
|
||||
|
||||
### 2.1 Strict-PD sourcing map (refines §8)
|
||||
|
||||
License stance is unchanged (prefer Public Domain / CC0; record license + source per
|
||||
clip). The "scales" theme maps onto genuinely public-domain pools cleanly at the
|
||||
*ends* and is softer in the terrestrial *middle*:
|
||||
|
||||
| Scale | Best strict-PD source | Status |
|
||||
|---|---|---|
|
||||
| Cosmos / galaxy / deep space | NASA, Hubble, JWST | 🟢 abundant, true PD |
|
||||
| Earth from orbit / continents | NASA / ISS | 🟢 true PD |
|
||||
| Ocean & **deep sea / abyss** (global) | **NOAA Ocean Exploration** | 🟢 true PD, worldwide |
|
||||
| Microscopic / single-celled | NIH / NSF | 🟢 thinner but PD |
|
||||
| US land / wildlife | NPS, USGS, USFWS | 🟢 true PD (US locations only) |
|
||||
| **Non-US terrestrial, high-flying birds** | — | 🟡 mostly CC-BY; the PD soft spot |
|
||||
|
||||
US-government works are public domain by statute (17 U.S.C. §105); the installation
|
||||
is US-based, so this is the cleanest possible legal footing. **Caveat the design
|
||||
must respect:** "free stock" sites (Pexels, Pixabay, Mitch Martinez's free 4K, etc.)
|
||||
are *royalty-free but NOT public domain* — they restrict redistribution and retain
|
||||
copyright. The ingest tool's "no explicit license → assume PD, verify before use"
|
||||
flag exists precisely for this trap and must not be trusted blindly.
|
||||
|
||||
---
|
||||
|
||||
## 3. Scale navigation & zoom transitions (new element; refines §2 selector, §11)
|
||||
|
||||
The scales-of-nature library is navigated as a **closed loop (a ring), not a line.**
|
||||
A dedicated **scale ("zoom") control** lets the viewer journey through scales;
|
||||
advancing it triggers a short **AI zoom/warp transition** to the next scale,
|
||||
pre-baked offline. Diving past the smallest (single-celled) **wraps around** to the
|
||||
largest (cosmos) — the infinite-zoom payoff that unifies micro and macro and makes
|
||||
the ring continuous.
|
||||
|
||||
- **The control is an *endless* rotary encoder — infinitely turnable, no end stops.**
|
||||
The form embodies the concept: a ring of scales has no beginning or end, so neither
|
||||
does the knob. Keep turning one way and you zoom inward forever (…reef →
|
||||
microscopic → **cosmos** → continents → …); turn back to zoom out. This sets it
|
||||
apart from the four **experience knobs**, which are *absolute* 0–4 pots: the zoom
|
||||
control reports **relative** rotation (encoder detents), and the player/firmware
|
||||
advances or retreats one ring-step per increment. It is distinct too from the §6
|
||||
content dial (audio/video channel) — it chooses *where in the ring* you are, while
|
||||
the knobs still bend whichever scale is present.
|
||||
- **Transitions:** between each adjacent pair of scale clips, a short (~few-second)
|
||||
generative morph — **first-last-frame-conditioned image-to-video** (Wan/LTX-class)
|
||||
or SD "infinite-zoom" outpainting for the literal zoom-through. Pre-baked offline,
|
||||
local; one clip per ring edge (N scales → N transitions, including the micro→cosmos
|
||||
closer). A fast spin may cross several scales — transitions chain, or past a speed
|
||||
threshold a faster blended pass is used.
|
||||
- **Thesis-safe:** dwells on a scale are the neutral, knob-altered interactive cores;
|
||||
transitions are fixed connective moments (un-altered, or at most carrying the
|
||||
current mood grade). The awe lives in the *movement between* scales, not the base.
|
||||
- **Heavier than the restyle, still bounded:** generative video synthesis costs more
|
||||
per second than img2img, but transitions are few and short — a handful of ~3–5 s
|
||||
morphs is an overnight local batch.
|
||||
|
||||
---
|
||||
|
||||
## 4. Economics update — local authoring ≈ free (refines §9)
|
||||
|
||||
§9 priced the Right axis at **~$300–3k** of cloud generative-v2v API. The POC
|
||||
collapses that: the restyle runs on **hardware the operator already owns**, offline,
|
||||
so the marginal cost is electricity.
|
||||
|
||||
- **Per base clip:** ~4 Right restyle strengths × ~2.7 min ≈ **~11 min of local AI
|
||||
pre-bake**, plus seconds for the deterministic Dark/Light/Left grid.
|
||||
- **Whole small library (~5 clips):** **~1 hour** of overnight batch rendering.
|
||||
- **Scale transitions:** ~N short generative-video morphs (one per ring edge), a few
|
||||
seconds each — a separate, heavier offline batch (video synthesis > img2img), but
|
||||
still overnight-local.
|
||||
- **Cloud API: no longer required** for the build. (It remains an option if a
|
||||
higher-quality video model than a local one is wanted for a final pass.)
|
||||
|
||||
This also tightens the piece's "all-local" ethos: not just *runtime* is local
|
||||
(§1 of the prior design) — now *authoring* is too.
|
||||
|
||||
---
|
||||
|
||||
## 5. POC evidence (this session)
|
||||
|
||||
A throwaway spike (outside the repo, `~/hef-poc/`) validated the full engine on one
|
||||
real nature clip (a 4K Yosemite waterfall, trimmed to 8 s @ 1080p):
|
||||
|
||||
- **All four axes rendered** and read as distinct: Dark (cold/somber), Light
|
||||
(warm/serene), Left (analytical HUD overlay), Right (painterly).
|
||||
- **Right per-frame:** ~3.4 min/8 s clip, **flickers badly** (disqualifying).
|
||||
- **Right flow-propagated:** ~2.7 min/8 s clip, **calm** (operator-approved).
|
||||
- **Deterministic axes:** ~2.4 s each, ~3× faster than real-time → confirmed
|
||||
runtime-capable.
|
||||
- **Stack:** `imageio-ffmpeg`, `diffusers` + `sd-turbo` on MPS, OpenCV Farneback
|
||||
flow; Python 3.13; 64 GB unified memory comfortably ran models that OOM consumer
|
||||
GPUs.
|
||||
|
||||
---
|
||||
|
||||
## 6. Open questions (for the plan, not blockers)
|
||||
|
||||
- **Flow quality at scale:** the OpenCV flow propagation was validated on one short
|
||||
clip; longer clips / faster motion may need shorter keyframe intervals, bidirectional
|
||||
blending, or a stronger flow model (RAFT).
|
||||
- **Painterly strength:** the POC kept the restyle gentle; the dreamlike *range* and
|
||||
the per-axis restyle-strength count (the §4.3 "small set of variants") are
|
||||
unfixed.
|
||||
- **Base-clip sourcing:** select and ingest the actual ~4–6 strictly-PD neutral
|
||||
scale clips (NASA/NOAA/NPS) via sub-project 2.
|
||||
- **Model choice:** `sd-turbo` was the POC's speed pick; a higher-quality local
|
||||
model (or a final cloud pass) may be worth a comparison for the shipped variants.
|
||||
- **Scale transitions:** generation method (first-last-frame i2v vs. infinite-zoom
|
||||
outpainting) and local model; per-transition length; whether transitions carry the
|
||||
current mood grade; the ring ordering of the scales; and behavior on fast or
|
||||
continuous spins of the endless encoder (chain transitions vs. blended skip).
|
||||
- **Pi compositing headroom:** confirm the Pi 5 can decode the altered video **and**
|
||||
render the live Pango/HarfBuzz label overlay at the *actual* panoramic projector
|
||||
resolution (ultra-wide / high-res) — low risk but unmeasured (see §1.2).
|
||||
|
||||
## 7. Out of scope (YAGNI)
|
||||
|
||||
- A single continuous **one-take** zoom through *all* scales (we use discrete neutral
|
||||
clips joined by short AI transitions on a navigable ring — not one unbroken shot).
|
||||
- Audio-axis alteration (already deferred by the prior §6).
|
||||
- Cloud rendering pipeline (local supersedes it for the base build).
|
||||
@@ -1,101 +0,0 @@
|
||||
# Session 0007.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Date: 2026-06-06 (PST) · 14:12 → 17:57
|
||||
> Type: coding
|
||||
> Goal: continue the roadmap; reframed by the operator to **simulator-first** —
|
||||
> bring the slice-1 alteration engine into the simulator so the experience can be
|
||||
> tuned before any hardware.
|
||||
> Outcome: **design spec written, committed, and pushed** on branch
|
||||
> `feature/simulator-alteration-preview`. No code changes. Implementation deferred
|
||||
> to the next session (design + implementation to ship as one PR).
|
||||
|
||||
## Plan
|
||||
|
||||
Bring the alteration engine into the simulator (scope: tune the look), per
|
||||
`docs/superpowers/specs/2026-06-06-simulator-alteration-preview-design.md`:
|
||||
- Python-canonical engine, thin browser renderer.
|
||||
- Dark/Light = live runtime color grade; Left/Right = 5×5 grid of pre-baked
|
||||
authored variant clips selected by coordinate (24 videos/base clip).
|
||||
- Revises design §4.3/§10: HUD baked into variant videos (authorial precision)
|
||||
rather than runtime overlay — trades away near-free i18n.
|
||||
- Engine reconciliation: parameterize `Calibration`, add `VariantRef`, slim
|
||||
`RenderPlan` (drop runtime `AnalyticalOverlay`/`Restyle`); `state.py` crossfade
|
||||
on variant change.
|
||||
- Simulator: retire selection X-ray + `/api/select`; add `/api/alteration` +
|
||||
`/api/clips` + variant manifest; placeholder-variant generator.
|
||||
|
||||
## Pre-session state
|
||||
|
||||
- `main` at `5290785`, clean and pushed. Sub-projects 1 & 2 done; sub-project 3
|
||||
(Player Runtime) in progress — slice 1 (pure-logic `player/` core) merged via
|
||||
PR #5 (session 0006).
|
||||
- A **stale nested worktree** (`.worktrees/feature-BuildSimulator11`, branch
|
||||
`feature/player-alteration-core`, already merged) was present and breaking the
|
||||
session resolver ("ambiguous app.json"), exactly as the session-0006 memory
|
||||
gotcha predicted.
|
||||
|
||||
## Turn-by-turn arc
|
||||
|
||||
1. **Session gate.** Opened with "What's next?" → classified as a coding session;
|
||||
ran `wgl-session-coding-init`.
|
||||
2. **Baseline cleanup.** Claim dry-run failed on the resolver ambiguity. Verified
|
||||
the stale worktree's branch was fully merged + clean, removed the worktree and
|
||||
branch, and deleted the empty `.worktrees/` dir. Tree back to clean `main`.
|
||||
3. **Claimed session 0007** (no other sessions in flight).
|
||||
4. **Operator steer:** "only design things that will run in the simulator; get
|
||||
everything working the way we like in the simulator before hardware." Saved as
|
||||
a `feedback` memory ([[simulator-first-before-hardware]]).
|
||||
5. **Discovered** the simulator (`simulator/`, built session 0004) is fully
|
||||
decoupled from `player/` — it exercises the OLD *selection* model and has no
|
||||
alteration surface. That gap became the work.
|
||||
6. **Brainstorming** (with the visual companion). Decisions, one at a time:
|
||||
scope = tune the alteration look; medium = short looping video; Right restyle =
|
||||
labeled stand-in shader (swappable for real variants); calibration =
|
||||
live-adjustable curves; architecture = **A, Python-canonical**; integration =
|
||||
new view, **retire the selection X-ray**.
|
||||
7. **Interactive look mockup** (browser): a real nature image with the 4 knobs as
|
||||
live sliders. Operator confirmed Dark/Light read well, but said for Left/Right
|
||||
they'll **author videos with the overlay baked in** for precise HUD control.
|
||||
8. **Surfaced the tradeoff:** baking the HUD reverses design §4.3/§10's near-free
|
||||
i18n. Operator chose **precision wins**. Confirmed the variant grid as **full
|
||||
5×5 = 24 authored clips per base** (shown as a grid mockup).
|
||||
9. **Revised model** presented and approved: base clip + variant manifest;
|
||||
`RenderPlan = {grade, variant}`; runtime overlay/restyle layers retired;
|
||||
placeholder-variant bootstrapping via ffmpeg.
|
||||
10. **Wrote the spec**, self-reviewed, committed on a feature branch.
|
||||
11. Operator: "do [implementation] next session. finalize this one." →
|
||||
`wgl-session-finalize`.
|
||||
|
||||
## Cut state (end of session)
|
||||
|
||||
| Repo | Branch | Commit | State |
|
||||
|---|---|---|---|
|
||||
| human-experience-filter-art | `feature/simulator-alteration-preview` | `3ef21fb` | pushed to origin, **not merged** |
|
||||
|
||||
- `docs/superpowers/specs/2026-06-06-simulator-alteration-preview-design.md` —
|
||||
new, committed.
|
||||
- `main` unchanged at `5290785`.
|
||||
- Working tree clean. Visual-companion server stopped; `.superpowers/` gitignored.
|
||||
- No tests run (no code changed).
|
||||
|
||||
## What lands on the operator's plate
|
||||
|
||||
- **Author the 24 variant videos** per base clip (the Left×Right grid), with HUD
|
||||
baked in. The simulator will consume them via the manifest; placeholder variants
|
||||
cover the gap until then.
|
||||
- **Deferred decisions** (also surfaced in chat):
|
||||
- *knob→strength calibration* — unresolved since session 0006; will be settled
|
||||
by eye in the simulator's calibration panel next session, then baked into
|
||||
`DEFAULT_CALIBRATION`.
|
||||
- *grade tints baked HUD* — accepted under "precision wins"; revisit only if it
|
||||
reads badly once real authored clips exist.
|
||||
|
||||
## Prompt the operator can paste into the next session
|
||||
|
||||
```
|
||||
/goal Implement the simulator alteration preview on branch feature/simulator-alteration-preview, per docs/superpowers/specs/2026-06-06-simulator-alteration-preview-design.md — begin with the writing-plans skill, then build: parameterize Calibration + add VariantRef and slim RenderPlan in player/, add /api/alteration + /api/clips + simulator/clips.py variant manifest, rewrite the simulator UI as the player preview, retire the selection X-ray + /api/select, add the placeholder-variant generator, and update tests + USER_GUIDE + ROADMAP §3 + the parent design §4.3/§10 pointer.
|
||||
```
|
||||
|
||||
(Or resume with `/goal next` — the `Next /goal:` field is stored in memory
|
||||
`sub-project-3-player-progress.md`.)
|
||||
@@ -1,20 +0,0 @@
|
||||
# Session 0008.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-07T19-42 (PST)
|
||||
> Type: spec
|
||||
> Status: **PLACEHOLDER — claimed at session start; finalized at session end.**
|
||||
>
|
||||
> This file reserves session ID 0008 for human-experience-filter-art. The driver replaces this
|
||||
> body with the full transcript and renames the file to its final
|
||||
> SESSION-0008.0-TRANSCRIPT-2026-06-07T19-42--<end>.md form at session end.
|
||||
|
||||
## Launch prompt
|
||||
|
||||
_(launch prompt not captured at claim time)_
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
_Autonomous-mode low-confidence calls the driver made and would have
|
||||
liked operator input on. Appended as the session runs; surfaced at
|
||||
finalize. Empty if none._
|
||||
@@ -16,11 +16,5 @@
|
||||
},
|
||||
"0006": {
|
||||
"title": ""
|
||||
},
|
||||
"0007": {
|
||||
"title": ""
|
||||
},
|
||||
"0008": {
|
||||
"title": ""
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user