# HEF — Scales-of-Nature Library + Stabilized Right-Axis Pipeline (Design Revision)

**Date:** 2026-06-07
**Status:** Approved design (pre-implementation) — operator-approved this session (0008)
**Repo:** `human-experience-filter-art`
**Refines:** [`2026-06-05-machine-altered-perception-design.md`](./2026-06-05-machine-altered-perception-design.md)
— specifically its Right-axis pipeline (§4.1/§4.3), content sourcing/model (§6/§8),
and economics (§9), and it **adds a scale-navigation control + zoom transitions**
to the §2 selector / §11 control panel. The thesis (§1), coordinate model (§3),
Dark/Light/Left treatment, and accessibility (§10) are **preserved**.
**Grounded in:** a local proof-of-concept run this session on the operator's Mac
mini (M4 Pro, 64 GB, MPS) — all numbers below are measured, not estimated.

> **Why this revision exists.** The 2026-06-05 design specified the Right axis as
> "generative video-to-video restyle, pre-baked offline" and assumed that meant a
> **paid cloud API** (§9 priced Kling/Sora/Veo/Runway at $0.05–0.10/s). A POC this
> session established two things that change the design: (1) the Right restyle runs
> **entirely locally and offline** on the operator's existing hardware, for the
> cost of electricity; and (2) naïve per-frame restyle **boils/flickers** in a way
> the operator found disorienting — disqualifying for a piece meant to be peaceful
> — and the fix is **optical-flow keyframe propagation**. Separately, the operator
> chose how the "scales of nature" idea enters the piece: as the curatorial theme
> of a **small neutral base library**, not a single fixed journey.

---

## 1. The Right axis is a local, flow-stabilized restyle (refines §4.1, §4.3)

The §4.1 mapping is unchanged in spirit — **Right = dissolve realism toward
painterly/dreamlike via generative video-to-video** — but the *implementation* is
now pinned:

- **Engine:** Stable Diffusion **img2img** (POC used `stabilityai/sd-turbo`) run on
  **Apple MPS**, locally, offline, at authoring time. No cloud API.
- **Temporal coherence is a hard requirement, not a nicety.** Per-frame img2img
  independently re-imagines each frame, producing a shimmering "boil" that reads as
  disorienting — the **opposite** of the piece's peaceful intent. This was caught
  in the POC and is now a named design constraint: *the Right substrate must be
  temporally coherent.*
- **Stabilization: optical-flow keyframe propagation.** Fully stylize **keyframes**
  at a fixed interval; for in-between frames, **warp the previous stylized frame
  forward by optical flow** (so motion is continuous) and apply only a *light*
  diffusion refine. The flow warp removes the boil; periodic keyframes bound drift.
  (This is the EbSynth principle. Genuine `ebsynth`/`ezsynth` are NVIDIA/Windows-
  leaning and don't install cleanly on Apple Silicon, so the POC implemented the
  same idea directly with OpenCV Farneback flow + the existing diffusers pipeline.)

This stays consistent with §4.3's crucial invariant: **the Left analytical labels
remain a runtime overlay, never baked into the restyled pixels** — the POC's Left
HUD is composited deterministically on top, preserving the near-free i18n of §10.

### 1.1 Where each transform runs (updated §4.3 table)

| Pole | Operation | Where it runs | Measured cost (8 s, 1080p clip) |
|---|---|---|---|
| **Dark** | color grade | runtime, live on the Pi | ~2.4 s offline; live at runtime |
| **Light** | color grade | runtime, live on the Pi | ~2.5 s offline; live at runtime |
| **Left** | analytical overlay (HUD) | runtime, live on the Pi | ~2.2 s offline; live at runtime |
| **Right** | local generative restyle + flow propagation | **pre-baked offline, locally** | ~2.7 min/restyle-strength |

The three deterministic axes are confirmed cheap enough to run **live**; only the
Right restyle is pre-baked. A peaceful **deterministic** alternative for Right (soft
edge-preserving smoothing + bloom, zero flicker by construction, ~5 s/clip) was
prototyped and set aside — the operator preferred the true generative repaint once
the flow stabilization made it calm. It remains a documented fallback.

### 1.2 Runtime label rendering & i18n (sharpens §10)

The Left analytical labels are drawn **live by the Pi as a 2D graphics overlay** —
**not** baked into video, and specifically **not** a pre-rendered overlay *video*. A
baked overlay would have to exist per language × per Left level × per scale, which
re-introduces the "× number of languages" cost §10 exists to avoid. So the runtime
path is:

- **Architecture.** Per base clip, an offline-authored **annotation track** (box
  positions, anchor points, which annotations appear at each Left level) referencing
  language-agnostic label **keys** (e.g. `detected.conifer`). Per language, a cheap
  **string table** (key → translated text) + font + TTS voice. At runtime the Pi
  reads the Left knob, selects the active annotations, **shapes** the current
  language's strings, and composites over the altered video — updating only on change
  (knob move, language switch, timeline cue), not every frame.
- **Correctness needs a real shaping stack.** Rendering *every* language correctly
  (Arabic joining, Indic conjuncts, CJK, RTL) requires **Pango + HarfBuzz + Noto
  fonts**, not naïve text drawing. (The POC's HUD used ffmpeg `drawtext`/Menlo —
  Latin-only; it would mis-render complex scripts and is **not** the runtime path.)
  This shaping stack is the load-bearing requirement behind §10's "label things
  correctly."
- **Feasibility.** This is OSD/subtitle-class compositing; a Pi 5 (VideoCore VII,
  GLES/Vulkan, hardware decode) handles it. Headroom at the *panoramic* resolution is
  the one unmeasured variable — see §6.

---

## 2. Content = a small NEUTRAL "scales of nature" library (refines §6, §8)

The operator's "cosmic zoom" concept (space → continents → birds → ocean → abyss →
microscopic → galaxy) enters the piece **as a curatorial theme, not a fixed film.**

- **Structure:** a **small library (~4–6 to start) of calm, neutral base clips**,
  each drawn from a *different scale of nature* — e.g. an orbital Earth, a forest, a
  coral reef, the deep-sea abyss, the microscopic, the cosmos. The machine alters
  whichever clip is playing, exactly as for any neutral base.
- **Why this and not a single stitched journey.** A literal galaxy→cell "how small
  we are" journey carries its **own** emotion (awe, cosmic insignificance) *before
  the machine acts*, which contradicts the §1 neutral-base thesis, and as a fixed
  film it becomes "a journey you watch" rather than "a reality you bend." Keeping the
  scales as the *theme of a neutral library* preserves the awe-of-scale richness and
  the piece's coherence **while keeping the base neutral and the experience
  interactive.** (Rejected alternatives: single stitched base; cosmic-zoom as
  intro/reset; rethinking the thesis.)
- **Cost is not the constraint.** Per §4 below, ~5 base clips is ~1 hour of overnight
  local pre-bake — so this is an *artistic* choice, made on artistic grounds.
- **Mechanism unchanged:** this slots into the existing §6 content model and the
  sub-project-2 ingest/tagging/review tooling; "scales of nature" is simply the
  selection principle for which neutral clips to source.

### 2.1 Strict-PD sourcing map (refines §8)

License stance is unchanged (prefer Public Domain / CC0; record license + source per
clip). The "scales" theme maps onto genuinely public-domain pools cleanly at the
*ends* and is softer in the terrestrial *middle*:

| Scale | Best strict-PD source | Status |
|---|---|---|
| Cosmos / galaxy / deep space | NASA, Hubble, JWST | 🟢 abundant, true PD |
| Earth from orbit / continents | NASA / ISS | 🟢 true PD |
| Ocean & **deep sea / abyss** (global) | **NOAA Ocean Exploration** | 🟢 true PD, worldwide |
| Microscopic / single-celled | NIH / NSF | 🟢 thinner but PD |
| US land / wildlife | NPS, USGS, USFWS | 🟢 true PD (US locations only) |
| **Non-US terrestrial, high-flying birds** | — | 🟡 mostly CC-BY; the PD soft spot |

US-government works are public domain by statute (17 U.S.C. §105); the installation
is US-based, so this is the cleanest possible legal footing. **Caveat the design
must respect:** "free stock" sites (Pexels, Pixabay, Mitch Martinez's free 4K, etc.)
are *royalty-free but NOT public domain* — they restrict redistribution and retain
copyright. The ingest tool's "no explicit license → assume PD, verify before use"
flag exists precisely for this trap and must not be trusted blindly.

---

## 3. Scale navigation & zoom transitions (new element; refines §2 selector, §11)

The scales-of-nature library is navigated as a **closed loop (a ring), not a line.**
A dedicated **scale ("zoom") control** lets the viewer journey through scales;
advancing it triggers a short **AI zoom/warp transition** to the next scale,
pre-baked offline. Diving past the smallest (single-celled) **wraps around** to the
largest (cosmos) — the infinite-zoom payoff that unifies micro and macro and makes
the ring continuous.

- **The control is an *endless* rotary encoder — infinitely turnable, no end stops.**
  The form embodies the concept: a ring of scales has no beginning or end, so neither
  does the knob. Keep turning one way and you zoom inward forever (…reef →
  microscopic → **cosmos** → continents → …); turn back to zoom out. This sets it
  apart from the four **experience knobs**, which are *absolute* 0–4 pots: the zoom
  control reports **relative** rotation (encoder detents), and the player/firmware
  advances or retreats one ring-step per increment. It is distinct too from the §6
  content dial (audio/video channel) — it chooses *where in the ring* you are, while
  the knobs still bend whichever scale is present.
- **Transitions:** between each adjacent pair of scale clips, a short (~few-second)
  generative morph — **first-last-frame-conditioned image-to-video** (Wan/LTX-class)
  or SD "infinite-zoom" outpainting for the literal zoom-through. Pre-baked offline,
  local; one clip per ring edge (N scales → N transitions, including the micro→cosmos
  closer). A fast spin may cross several scales — transitions chain, or past a speed
  threshold a faster blended pass is used.
  - **Implemented (session 0012):** `player.ring.advance_ring` takes an opt-in
    `fast_spin_threshold` (canonical default `DEFAULT_FAST_SPIN_THRESHOLD = 3`).
    A spin batches its detents into one `advance` call, so `abs(delta)` is the
    input layer's proxy for spin speed; at/above the threshold the move collapses
    to a **single blended pass** — one `TransitionStep` (the arrival edge, marked
    `blended`, landing straight on the destination, `RingMove.fast=True`) instead
    of chaining every transition. The policy is opt-in because the pure function
    cannot observe wall-clock speed — the simulator/firmware enables it. The
    simulator plays the blended step at 2.5× (`FAST_BLEND_RATE`); when real
    transition clips exist a dedicated fast-blend clip can replace the
    accelerated arrival-edge placeholder. Threshold + rate are tunable by eye.
- **Thesis-safe:** dwells on a scale are the neutral, knob-altered interactive cores;
  transitions are fixed connective moments (un-altered, or at most carrying the
  current mood grade). The awe lives in the *movement between* scales, not the base.
- **Heavier than the restyle, still bounded:** generative video synthesis costs more
  per second than img2img, but transitions are few and short — a handful of ~3–5 s
  morphs is an overnight local batch.

---

## 4. Economics update — local authoring ≈ free (refines §9)

§9 priced the Right axis at **~$300–3k** of cloud generative-v2v API. The POC
collapses that: the restyle runs on **hardware the operator already owns**, offline,
so the marginal cost is electricity.

- **Per base clip:** ~4 Right restyle strengths × ~2.7 min ≈ **~11 min of local AI
  pre-bake**, plus seconds for the deterministic Dark/Light/Left grid.
- **Whole small library (~5 clips):** **~1 hour** of overnight batch rendering.
- **Scale transitions:** ~N short generative-video morphs (one per ring edge), a few
  seconds each — a separate, heavier offline batch (video synthesis > img2img), but
  still overnight-local.
- **Cloud API: no longer required** for the build. (It remains an option if a
  higher-quality video model than a local one is wanted for a final pass.)

This also tightens the piece's "all-local" ethos: not just *runtime* is local
(§1 of the prior design) — now *authoring* is too.

---

## 5. POC evidence (this session)

A throwaway spike (outside the repo, `~/hef-poc/`) validated the full engine on one
real nature clip (a 4K Yosemite waterfall, trimmed to 8 s @ 1080p):

- **All four axes rendered** and read as distinct: Dark (cold/somber), Light
  (warm/serene), Left (analytical HUD overlay), Right (painterly).
- **Right per-frame:** ~3.4 min/8 s clip, **flickers badly** (disqualifying).
- **Right flow-propagated:** ~2.7 min/8 s clip, **calm** (operator-approved).
- **Deterministic axes:** ~2.4 s each, ~3× faster than real-time → confirmed
  runtime-capable.
- **Stack:** `imageio-ffmpeg`, `diffusers` + `sd-turbo` on MPS, OpenCV Farneback
  flow; Python 3.13; 64 GB unified memory comfortably ran models that OOM consumer
  GPUs.

---

## 6. Open questions (for the plan, not blockers)

- **Flow quality at scale:** the OpenCV flow propagation was validated on one short
  clip; longer clips / faster motion may need shorter keyframe intervals, bidirectional
  blending, or a stronger flow model (RAFT).
- **Painterly strength:** the POC kept the restyle gentle; the dreamlike *range* and
  the per-axis restyle-strength count (the §4.3 "small set of variants") are
  unfixed.
- **Base-clip sourcing:** select and ingest the actual ~4–6 strictly-PD neutral
  scale clips (NASA/NOAA/NPS) via sub-project 2.
- **Model choice:** `sd-turbo` was the POC's speed pick; a higher-quality local
  model (or a final cloud pass) may be worth a comparison for the shipped variants.
- **Scale transitions:** generation method (first-last-frame i2v vs. infinite-zoom
  outpainting) and local model; per-transition length; whether transitions carry the
  current mood grade; the ring ordering of the scales; and behavior on fast or
  continuous spins of the endless encoder (chain transitions vs. blended skip).
- **Pi compositing headroom:** confirm the Pi 5 can decode the altered video **and**
  render the live Pango/HarfBuzz label overlay at the *actual* panoramic projector
  resolution (ultra-wide / high-res) — low risk but unmeasured (see §1.2).

## 7. Out of scope (YAGNI)

- A single continuous **one-take** zoom through *all* scales (we use discrete neutral
  clips joined by short AI transitions on a navigable ring — not one unbroken shot).
- Audio-axis alteration (already deferred by the prior §6).
- Cloud rendering pipeline (local supersedes it for the base build).
