diff --git a/docs/superpowers/specs/2026-06-07-scales-library-and-right-axis-pipeline-design.md b/docs/superpowers/specs/2026-06-07-scales-library-and-right-axis-pipeline-design.md new file mode 100644 index 0000000..d455e2c --- /dev/null +++ b/docs/superpowers/specs/2026-06-07-scales-library-and-right-axis-pipeline-design.md @@ -0,0 +1,236 @@ +# HEF — Scales-of-Nature Library + Stabilized Right-Axis Pipeline (Design Revision) + +**Date:** 2026-06-07 +**Status:** Approved design (pre-implementation) — operator-approved this session (0008) +**Repo:** `human-experience-filter-art` +**Refines:** [`2026-06-05-machine-altered-perception-design.md`](./2026-06-05-machine-altered-perception-design.md) +— specifically its Right-axis pipeline (§4.1/§4.3), content sourcing/model (§6/§8), +and economics (§9), and it **adds a scale-navigation control + zoom transitions** +to the §2 selector / §11 control panel. The thesis (§1), coordinate model (§3), +Dark/Light/Left treatment, and accessibility (§10) are **preserved**. +**Grounded in:** a local proof-of-concept run this session on the operator's Mac +mini (M4 Pro, 64 GB, MPS) — all numbers below are measured, not estimated. + +> **Why this revision exists.** The 2026-06-05 design specified the Right axis as +> "generative video-to-video restyle, pre-baked offline" and assumed that meant a +> **paid cloud API** (§9 priced Kling/Sora/Veo/Runway at $0.05–0.10/s). A POC this +> session established two things that change the design: (1) the Right restyle runs +> **entirely locally and offline** on the operator's existing hardware, for the +> cost of electricity; and (2) naïve per-frame restyle **boils/flickers** in a way +> the operator found disorienting — disqualifying for a piece meant to be peaceful +> — and the fix is **optical-flow keyframe propagation**. Separately, the operator +> chose how the "scales of nature" idea enters the piece: as the curatorial theme +> of a **small neutral base library**, not a single fixed journey. + +--- + +## 1. The Right axis is a local, flow-stabilized restyle (refines §4.1, §4.3) + +The §4.1 mapping is unchanged in spirit — **Right = dissolve realism toward +painterly/dreamlike via generative video-to-video** — but the *implementation* is +now pinned: + +- **Engine:** Stable Diffusion **img2img** (POC used `stabilityai/sd-turbo`) run on + **Apple MPS**, locally, offline, at authoring time. No cloud API. +- **Temporal coherence is a hard requirement, not a nicety.** Per-frame img2img + independently re-imagines each frame, producing a shimmering "boil" that reads as + disorienting — the **opposite** of the piece's peaceful intent. This was caught + in the POC and is now a named design constraint: *the Right substrate must be + temporally coherent.* +- **Stabilization: optical-flow keyframe propagation.** Fully stylize **keyframes** + at a fixed interval; for in-between frames, **warp the previous stylized frame + forward by optical flow** (so motion is continuous) and apply only a *light* + diffusion refine. The flow warp removes the boil; periodic keyframes bound drift. + (This is the EbSynth principle. Genuine `ebsynth`/`ezsynth` are NVIDIA/Windows- + leaning and don't install cleanly on Apple Silicon, so the POC implemented the + same idea directly with OpenCV Farneback flow + the existing diffusers pipeline.) + +This stays consistent with §4.3's crucial invariant: **the Left analytical labels +remain a runtime overlay, never baked into the restyled pixels** — the POC's Left +HUD is composited deterministically on top, preserving the near-free i18n of §10. + +### 1.1 Where each transform runs (updated §4.3 table) + +| Pole | Operation | Where it runs | Measured cost (8 s, 1080p clip) | +|---|---|---|---| +| **Dark** | color grade | runtime, live on the Pi | ~2.4 s offline; live at runtime | +| **Light** | color grade | runtime, live on the Pi | ~2.5 s offline; live at runtime | +| **Left** | analytical overlay (HUD) | runtime, live on the Pi | ~2.2 s offline; live at runtime | +| **Right** | local generative restyle + flow propagation | **pre-baked offline, locally** | ~2.7 min/restyle-strength | + +The three deterministic axes are confirmed cheap enough to run **live**; only the +Right restyle is pre-baked. A peaceful **deterministic** alternative for Right (soft +edge-preserving smoothing + bloom, zero flicker by construction, ~5 s/clip) was +prototyped and set aside — the operator preferred the true generative repaint once +the flow stabilization made it calm. It remains a documented fallback. + +### 1.2 Runtime label rendering & i18n (sharpens §10) + +The Left analytical labels are drawn **live by the Pi as a 2D graphics overlay** — +**not** baked into video, and specifically **not** a pre-rendered overlay *video*. A +baked overlay would have to exist per language × per Left level × per scale, which +re-introduces the "× number of languages" cost §10 exists to avoid. So the runtime +path is: + +- **Architecture.** Per base clip, an offline-authored **annotation track** (box + positions, anchor points, which annotations appear at each Left level) referencing + language-agnostic label **keys** (e.g. `detected.conifer`). Per language, a cheap + **string table** (key → translated text) + font + TTS voice. At runtime the Pi + reads the Left knob, selects the active annotations, **shapes** the current + language's strings, and composites over the altered video — updating only on change + (knob move, language switch, timeline cue), not every frame. +- **Correctness needs a real shaping stack.** Rendering *every* language correctly + (Arabic joining, Indic conjuncts, CJK, RTL) requires **Pango + HarfBuzz + Noto + fonts**, not naïve text drawing. (The POC's HUD used ffmpeg `drawtext`/Menlo — + Latin-only; it would mis-render complex scripts and is **not** the runtime path.) + This shaping stack is the load-bearing requirement behind §10's "label things + correctly." +- **Feasibility.** This is OSD/subtitle-class compositing; a Pi 5 (VideoCore VII, + GLES/Vulkan, hardware decode) handles it. Headroom at the *panoramic* resolution is + the one unmeasured variable — see §6. + +--- + +## 2. Content = a small NEUTRAL "scales of nature" library (refines §6, §8) + +The operator's "cosmic zoom" concept (space → continents → birds → ocean → abyss → +microscopic → galaxy) enters the piece **as a curatorial theme, not a fixed film.** + +- **Structure:** a **small library (~4–6 to start) of calm, neutral base clips**, + each drawn from a *different scale of nature* — e.g. an orbital Earth, a forest, a + coral reef, the deep-sea abyss, the microscopic, the cosmos. The machine alters + whichever clip is playing, exactly as for any neutral base. +- **Why this and not a single stitched journey.** A literal galaxy→cell "how small + we are" journey carries its **own** emotion (awe, cosmic insignificance) *before + the machine acts*, which contradicts the §1 neutral-base thesis, and as a fixed + film it becomes "a journey you watch" rather than "a reality you bend." Keeping the + scales as the *theme of a neutral library* preserves the awe-of-scale richness and + the piece's coherence **while keeping the base neutral and the experience + interactive.** (Rejected alternatives: single stitched base; cosmic-zoom as + intro/reset; rethinking the thesis.) +- **Cost is not the constraint.** Per §4 below, ~5 base clips is ~1 hour of overnight + local pre-bake — so this is an *artistic* choice, made on artistic grounds. +- **Mechanism unchanged:** this slots into the existing §6 content model and the + sub-project-2 ingest/tagging/review tooling; "scales of nature" is simply the + selection principle for which neutral clips to source. + +### 2.1 Strict-PD sourcing map (refines §8) + +License stance is unchanged (prefer Public Domain / CC0; record license + source per +clip). The "scales" theme maps onto genuinely public-domain pools cleanly at the +*ends* and is softer in the terrestrial *middle*: + +| Scale | Best strict-PD source | Status | +|---|---|---| +| Cosmos / galaxy / deep space | NASA, Hubble, JWST | 🟢 abundant, true PD | +| Earth from orbit / continents | NASA / ISS | 🟢 true PD | +| Ocean & **deep sea / abyss** (global) | **NOAA Ocean Exploration** | 🟢 true PD, worldwide | +| Microscopic / single-celled | NIH / NSF | 🟢 thinner but PD | +| US land / wildlife | NPS, USGS, USFWS | 🟢 true PD (US locations only) | +| **Non-US terrestrial, high-flying birds** | — | 🟡 mostly CC-BY; the PD soft spot | + +US-government works are public domain by statute (17 U.S.C. §105); the installation +is US-based, so this is the cleanest possible legal footing. **Caveat the design +must respect:** "free stock" sites (Pexels, Pixabay, Mitch Martinez's free 4K, etc.) +are *royalty-free but NOT public domain* — they restrict redistribution and retain +copyright. The ingest tool's "no explicit license → assume PD, verify before use" +flag exists precisely for this trap and must not be trusted blindly. + +--- + +## 3. Scale navigation & zoom transitions (new element; refines §2 selector, §11) + +The scales-of-nature library is navigated as a **closed loop (a ring), not a line.** +A dedicated **scale ("zoom") control** lets the viewer journey through scales; +advancing it triggers a short **AI zoom/warp transition** to the next scale, +pre-baked offline. Diving past the smallest (single-celled) **wraps around** to the +largest (cosmos) — the infinite-zoom payoff that unifies micro and macro and makes +the ring continuous. + +- **The control is an *endless* rotary encoder — infinitely turnable, no end stops.** + The form embodies the concept: a ring of scales has no beginning or end, so neither + does the knob. Keep turning one way and you zoom inward forever (…reef → + microscopic → **cosmos** → continents → …); turn back to zoom out. This sets it + apart from the four **experience knobs**, which are *absolute* 0–4 pots: the zoom + control reports **relative** rotation (encoder detents), and the player/firmware + advances or retreats one ring-step per increment. It is distinct too from the §6 + content dial (audio/video channel) — it chooses *where in the ring* you are, while + the knobs still bend whichever scale is present. +- **Transitions:** between each adjacent pair of scale clips, a short (~few-second) + generative morph — **first-last-frame-conditioned image-to-video** (Wan/LTX-class) + or SD "infinite-zoom" outpainting for the literal zoom-through. Pre-baked offline, + local; one clip per ring edge (N scales → N transitions, including the micro→cosmos + closer). A fast spin may cross several scales — transitions chain, or past a speed + threshold a faster blended pass is used. +- **Thesis-safe:** dwells on a scale are the neutral, knob-altered interactive cores; + transitions are fixed connective moments (un-altered, or at most carrying the + current mood grade). The awe lives in the *movement between* scales, not the base. +- **Heavier than the restyle, still bounded:** generative video synthesis costs more + per second than img2img, but transitions are few and short — a handful of ~3–5 s + morphs is an overnight local batch. + +--- + +## 4. Economics update — local authoring ≈ free (refines §9) + +§9 priced the Right axis at **~$300–3k** of cloud generative-v2v API. The POC +collapses that: the restyle runs on **hardware the operator already owns**, offline, +so the marginal cost is electricity. + +- **Per base clip:** ~4 Right restyle strengths × ~2.7 min ≈ **~11 min of local AI + pre-bake**, plus seconds for the deterministic Dark/Light/Left grid. +- **Whole small library (~5 clips):** **~1 hour** of overnight batch rendering. +- **Scale transitions:** ~N short generative-video morphs (one per ring edge), a few + seconds each — a separate, heavier offline batch (video synthesis > img2img), but + still overnight-local. +- **Cloud API: no longer required** for the build. (It remains an option if a + higher-quality video model than a local one is wanted for a final pass.) + +This also tightens the piece's "all-local" ethos: not just *runtime* is local +(§1 of the prior design) — now *authoring* is too. + +--- + +## 5. POC evidence (this session) + +A throwaway spike (outside the repo, `~/hef-poc/`) validated the full engine on one +real nature clip (a 4K Yosemite waterfall, trimmed to 8 s @ 1080p): + +- **All four axes rendered** and read as distinct: Dark (cold/somber), Light + (warm/serene), Left (analytical HUD overlay), Right (painterly). +- **Right per-frame:** ~3.4 min/8 s clip, **flickers badly** (disqualifying). +- **Right flow-propagated:** ~2.7 min/8 s clip, **calm** (operator-approved). +- **Deterministic axes:** ~2.4 s each, ~3× faster than real-time → confirmed + runtime-capable. +- **Stack:** `imageio-ffmpeg`, `diffusers` + `sd-turbo` on MPS, OpenCV Farneback + flow; Python 3.13; 64 GB unified memory comfortably ran models that OOM consumer + GPUs. + +--- + +## 6. Open questions (for the plan, not blockers) + +- **Flow quality at scale:** the OpenCV flow propagation was validated on one short + clip; longer clips / faster motion may need shorter keyframe intervals, bidirectional + blending, or a stronger flow model (RAFT). +- **Painterly strength:** the POC kept the restyle gentle; the dreamlike *range* and + the per-axis restyle-strength count (the §4.3 "small set of variants") are + unfixed. +- **Base-clip sourcing:** select and ingest the actual ~4–6 strictly-PD neutral + scale clips (NASA/NOAA/NPS) via sub-project 2. +- **Model choice:** `sd-turbo` was the POC's speed pick; a higher-quality local + model (or a final cloud pass) may be worth a comparison for the shipped variants. +- **Scale transitions:** generation method (first-last-frame i2v vs. infinite-zoom + outpainting) and local model; per-transition length; whether transitions carry the + current mood grade; the ring ordering of the scales; and behavior on fast or + continuous spins of the endless encoder (chain transitions vs. blended skip). +- **Pi compositing headroom:** confirm the Pi 5 can decode the altered video **and** + render the live Pango/HarfBuzz label overlay at the *actual* panoramic projector + resolution (ultra-wide / high-res) — low risk but unmeasured (see §1.2). + +## 7. Out of scope (YAGNI) + +- A single continuous **one-take** zoom through *all* scales (we use discrete neutral + clips joined by short AI transitions on a navigable ring — not one unbroken shot). +- Audio-axis alteration (already deferred by the prior §6). +- Cloud rendering pipeline (local supersedes it for the base build).