2026-06-08 05:34:49 +00:00
1 changed files with 236 additions and 0 deletions
@@ -0,0 +1,236 @@
+# HEF — Scales-of-Nature Library + Stabilized Right-Axis Pipeline (Design Revision)
+
+**Date:** 2026-06-07
+**Status:** Approved design (pre-implementation) — operator-approved this session (0008)
+**Repo:** `human-experience-filter-art`
+**Refines:** [`2026-06-05-machine-altered-perception-design.md`](./2026-06-05-machine-altered-perception-design.md)
+— specifically its Right-axis pipeline (§4.1/§4.3), content sourcing/model (§6/§8),
+and economics (§9), and it **adds a scale-navigation control + zoom transitions**
+to the §2 selector / §11 control panel. The thesis (§1), coordinate model (§3),
+Dark/Light/Left treatment, and accessibility (§10) are **preserved**.
+**Grounded in:** a local proof-of-concept run this session on the operator's Mac
+mini (M4 Pro, 64 GB, MPS) — all numbers below are measured, not estimated.
+
+> **Why this revision exists.** The 2026-06-05 design specified the Right axis as
+> "generative video-to-video restyle, pre-baked offline" and assumed that meant a
+> **paid cloud API** (§9 priced Kling/Sora/Veo/Runway at $0.05–0.10/s). A POC this
+> session established two things that change the design: (1) the Right restyle runs
+> **entirely locally and offline** on the operator's existing hardware, for the
+> cost of electricity; and (2) naïve per-frame restyle **boils/flickers** in a way
+> the operator found disorienting — disqualifying for a piece meant to be peaceful
+> — and the fix is **optical-flow keyframe propagation**. Separately, the operator
+> chose how the "scales of nature" idea enters the piece: as the curatorial theme
+> of a **small neutral base library**, not a single fixed journey.
+
+---
+
+## 1. The Right axis is a local, flow-stabilized restyle (refines §4.1, §4.3)
+
+The §4.1 mapping is unchanged in spirit — **Right = dissolve realism toward
+painterly/dreamlike via generative video-to-video** — but the *implementation* is
+now pinned:
+
+- **Engine:** Stable Diffusion **img2img** (POC used `stabilityai/sd-turbo`) run on
+  **Apple MPS**, locally, offline, at authoring time. No cloud API.
+- **Temporal coherence is a hard requirement, not a nicety.** Per-frame img2img
+  independently re-imagines each frame, producing a shimmering "boil" that reads as
+  disorienting — the **opposite** of the piece's peaceful intent. This was caught
+  in the POC and is now a named design constraint: *the Right substrate must be
+  temporally coherent.*
+- **Stabilization: optical-flow keyframe propagation.** Fully stylize **keyframes**
+  at a fixed interval; for in-between frames, **warp the previous stylized frame
+  forward by optical flow** (so motion is continuous) and apply only a *light*
+  diffusion refine. The flow warp removes the boil; periodic keyframes bound drift.
+  (This is the EbSynth principle. Genuine `ebsynth`/`ezsynth` are NVIDIA/Windows-
+  leaning and don't install cleanly on Apple Silicon, so the POC implemented the
+  same idea directly with OpenCV Farneback flow + the existing diffusers pipeline.)
+
+This stays consistent with §4.3's crucial invariant: **the Left analytical labels
+remain a runtime overlay, never baked into the restyled pixels** — the POC's Left
+HUD is composited deterministically on top, preserving the near-free i18n of §10.
+
+### 1.1 Where each transform runs (updated §4.3 table)
+
+| Pole | Operation | Where it runs | Measured cost (8 s, 1080p clip) |
+|---|---|---|---|
+| **Dark** | color grade | runtime, live on the Pi | ~2.4 s offline; live at runtime |
+| **Light** | color grade | runtime, live on the Pi | ~2.5 s offline; live at runtime |
+| **Left** | analytical overlay (HUD) | runtime, live on the Pi | ~2.2 s offline; live at runtime |
+| **Right** | local generative restyle + flow propagation | **pre-baked offline, locally** | ~2.7 min/restyle-strength |
+
+The three deterministic axes are confirmed cheap enough to run **live**; only the
+Right restyle is pre-baked. A peaceful **deterministic** alternative for Right (soft
+edge-preserving smoothing + bloom, zero flicker by construction, ~5 s/clip) was
+prototyped and set aside — the operator preferred the true generative repaint once
+the flow stabilization made it calm. It remains a documented fallback.
+
+### 1.2 Runtime label rendering & i18n (sharpens §10)
+
+The Left analytical labels are drawn **live by the Pi as a 2D graphics overlay** —
+**not** baked into video, and specifically **not** a pre-rendered overlay *video*. A
+baked overlay would have to exist per language × per Left level × per scale, which
+re-introduces the "× number of languages" cost §10 exists to avoid. So the runtime
+path is:
+
+- **Architecture.** Per base clip, an offline-authored **annotation track** (box
+  positions, anchor points, which annotations appear at each Left level) referencing
+  language-agnostic label **keys** (e.g. `detected.conifer`). Per language, a cheap
+  **string table** (key → translated text) + font + TTS voice. At runtime the Pi
+  reads the Left knob, selects the active annotations, **shapes** the current
+  language's strings, and composites over the altered video — updating only on change
+  (knob move, language switch, timeline cue), not every frame.
+- **Correctness needs a real shaping stack.** Rendering *every* language correctly
+  (Arabic joining, Indic conjuncts, CJK, RTL) requires **Pango + HarfBuzz + Noto
+  fonts**, not naïve text drawing. (The POC's HUD used ffmpeg `drawtext`/Menlo —
+  Latin-only; it would mis-render complex scripts and is **not** the runtime path.)
+  This shaping stack is the load-bearing requirement behind §10's "label things
+  correctly."
+- **Feasibility.** This is OSD/subtitle-class compositing; a Pi 5 (VideoCore VII,
+  GLES/Vulkan, hardware decode) handles it. Headroom at the *panoramic* resolution is
+  the one unmeasured variable — see §6.
+
+---
+
+## 2. Content = a small NEUTRAL "scales of nature" library (refines §6, §8)
+
+The operator's "cosmic zoom" concept (space → continents → birds → ocean → abyss →
+microscopic → galaxy) enters the piece **as a curatorial theme, not a fixed film.**
+
+- **Structure:** a **small library (~4–6 to start) of calm, neutral base clips**,
+  each drawn from a *different scale of nature* — e.g. an orbital Earth, a forest, a
+  coral reef, the deep-sea abyss, the microscopic, the cosmos. The machine alters
+  whichever clip is playing, exactly as for any neutral base.
+- **Why this and not a single stitched journey.** A literal galaxy→cell "how small
+  we are" journey carries its **own** emotion (awe, cosmic insignificance) *before
+  the machine acts*, which contradicts the §1 neutral-base thesis, and as a fixed
+  film it becomes "a journey you watch" rather than "a reality you bend." Keeping the
+  scales as the *theme of a neutral library* preserves the awe-of-scale richness and
+  the piece's coherence **while keeping the base neutral and the experience
+  interactive.** (Rejected alternatives: single stitched base; cosmic-zoom as
+  intro/reset; rethinking the thesis.)
+- **Cost is not the constraint.** Per §4 below, ~5 base clips is ~1 hour of overnight
+  local pre-bake — so this is an *artistic* choice, made on artistic grounds.
+- **Mechanism unchanged:** this slots into the existing §6 content model and the
+  sub-project-2 ingest/tagging/review tooling; "scales of nature" is simply the
+  selection principle for which neutral clips to source.
+
+### 2.1 Strict-PD sourcing map (refines §8)
+
+License stance is unchanged (prefer Public Domain / CC0; record license + source per
+clip). The "scales" theme maps onto genuinely public-domain pools cleanly at the
+*ends* and is softer in the terrestrial *middle*:
+
+| Scale | Best strict-PD source | Status |
+|---|---|---|
+| Cosmos / galaxy / deep space | NASA, Hubble, JWST | 🟢 abundant, true PD |
+| Earth from orbit / continents | NASA / ISS | 🟢 true PD |
+| Ocean & **deep sea / abyss** (global) | **NOAA Ocean Exploration** | 🟢 true PD, worldwide |
+| Microscopic / single-celled | NIH / NSF | 🟢 thinner but PD |
+| US land / wildlife | NPS, USGS, USFWS | 🟢 true PD (US locations only) |
+| **Non-US terrestrial, high-flying birds** | — | 🟡 mostly CC-BY; the PD soft spot |
+
+US-government works are public domain by statute (17 U.S.C. §105); the installation
+is US-based, so this is the cleanest possible legal footing. **Caveat the design
+must respect:** "free stock" sites (Pexels, Pixabay, Mitch Martinez's free 4K, etc.)
+are *royalty-free but NOT public domain* — they restrict redistribution and retain
+copyright. The ingest tool's "no explicit license → assume PD, verify before use"
+flag exists precisely for this trap and must not be trusted blindly.
+
+---
+
+## 3. Scale navigation & zoom transitions (new element; refines §2 selector, §11)
+
+The scales-of-nature library is navigated as a **closed loop (a ring), not a line.**
+A dedicated **scale ("zoom") control** lets the viewer journey through scales;
+advancing it triggers a short **AI zoom/warp transition** to the next scale,
+pre-baked offline. Diving past the smallest (single-celled) **wraps around** to the
+largest (cosmos) — the infinite-zoom payoff that unifies micro and macro and makes
+the ring continuous.
+
+- **The control is an *endless* rotary encoder — infinitely turnable, no end stops.**
+  The form embodies the concept: a ring of scales has no beginning or end, so neither
+  does the knob. Keep turning one way and you zoom inward forever (…reef →
+  microscopic → **cosmos** → continents → …); turn back to zoom out. This sets it
+  apart from the four **experience knobs**, which are *absolute* 0–4 pots: the zoom
+  control reports **relative** rotation (encoder detents), and the player/firmware
+  advances or retreats one ring-step per increment. It is distinct too from the §6
+  content dial (audio/video channel) — it chooses *where in the ring* you are, while
+  the knobs still bend whichever scale is present.
+- **Transitions:** between each adjacent pair of scale clips, a short (~few-second)
+  generative morph — **first-last-frame-conditioned image-to-video** (Wan/LTX-class)
+  or SD "infinite-zoom" outpainting for the literal zoom-through. Pre-baked offline,
+  local; one clip per ring edge (N scales → N transitions, including the micro→cosmos
+  closer). A fast spin may cross several scales — transitions chain, or past a speed
+  threshold a faster blended pass is used.
+- **Thesis-safe:** dwells on a scale are the neutral, knob-altered interactive cores;
+  transitions are fixed connective moments (un-altered, or at most carrying the
+  current mood grade). The awe lives in the *movement between* scales, not the base.
+- **Heavier than the restyle, still bounded:** generative video synthesis costs more
+  per second than img2img, but transitions are few and short — a handful of ~3–5 s
+  morphs is an overnight local batch.
+
+---
+
+## 4. Economics update — local authoring ≈ free (refines §9)
+
+§9 priced the Right axis at **~$300–3k** of cloud generative-v2v API. The POC
+collapses that: the restyle runs on **hardware the operator already owns**, offline,
+so the marginal cost is electricity.
+
+- **Per base clip:** ~4 Right restyle strengths × ~2.7 min ≈ **~11 min of local AI
+  pre-bake**, plus seconds for the deterministic Dark/Light/Left grid.
+- **Whole small library (~5 clips):** **~1 hour** of overnight batch rendering.
+- **Scale transitions:** ~N short generative-video morphs (one per ring edge), a few
+  seconds each — a separate, heavier offline batch (video synthesis > img2img), but
+  still overnight-local.
+- **Cloud API: no longer required** for the build. (It remains an option if a
+  higher-quality video model than a local one is wanted for a final pass.)
+
+This also tightens the piece's "all-local" ethos: not just *runtime* is local
+(§1 of the prior design) — now *authoring* is too.
+
+---
+
+## 5. POC evidence (this session)
+
+A throwaway spike (outside the repo, `~/hef-poc/`) validated the full engine on one
+real nature clip (a 4K Yosemite waterfall, trimmed to 8 s @ 1080p):
+
+- **All four axes rendered** and read as distinct: Dark (cold/somber), Light
+  (warm/serene), Left (analytical HUD overlay), Right (painterly).
+- **Right per-frame:** ~3.4 min/8 s clip, **flickers badly** (disqualifying).
+- **Right flow-propagated:** ~2.7 min/8 s clip, **calm** (operator-approved).
+- **Deterministic axes:** ~2.4 s each, ~3× faster than real-time → confirmed
+  runtime-capable.
+- **Stack:** `imageio-ffmpeg`, `diffusers` + `sd-turbo` on MPS, OpenCV Farneback
+  flow; Python 3.13; 64 GB unified memory comfortably ran models that OOM consumer
+  GPUs.
+
+---
+
+## 6. Open questions (for the plan, not blockers)
+
+- **Flow quality at scale:** the OpenCV flow propagation was validated on one short
+  clip; longer clips / faster motion may need shorter keyframe intervals, bidirectional
+  blending, or a stronger flow model (RAFT).
+- **Painterly strength:** the POC kept the restyle gentle; the dreamlike *range* and
+  the per-axis restyle-strength count (the §4.3 "small set of variants") are
+  unfixed.
+- **Base-clip sourcing:** select and ingest the actual ~4–6 strictly-PD neutral
+  scale clips (NASA/NOAA/NPS) via sub-project 2.
+- **Model choice:** `sd-turbo` was the POC's speed pick; a higher-quality local
+  model (or a final cloud pass) may be worth a comparison for the shipped variants.
+- **Scale transitions:** generation method (first-last-frame i2v vs. infinite-zoom
+  outpainting) and local model; per-transition length; whether transitions carry the
+  current mood grade; the ring ordering of the scales; and behavior on fast or
+  continuous spins of the endless encoder (chain transitions vs. blended skip).
+- **Pi compositing headroom:** confirm the Pi 5 can decode the altered video **and**
+  render the live Pango/HarfBuzz label overlay at the *actual* panoramic projector
+  resolution (ultra-wide / high-res) — low risk but unmeasured (see §1.2).
+
+## 7. Out of scope (YAGNI)
+
+- A single continuous **one-take** zoom through *all* scales (we use discrete neutral
+  clips joined by short AI transitions on a navigable ring — not one unbroken shot).
+- Audio-axis alteration (already deferred by the prior §6).
+- Cloud rendering pipeline (local supersedes it for the base build).