# HEF — Reconciled Simulator-First Alteration Slice (Design)

**Date:** 2026-06-07
**Status:** Approved design (pre-implementation) — reconciliation approved this session (0009)
**Repo:** `human-experience-filter-art`
**Reconciles:**
[`2026-06-06-simulator-alteration-preview-design.md`](./2026-06-06-simulator-alteration-preview-design.md)
(session 0007, the *unmerged* `feature/simulator-alteration-preview` branch) **with**
[`2026-06-07-scales-library-and-right-axis-pipeline-design.md`](./2026-06-07-scales-library-and-right-axis-pipeline-design.md)
(session 0008, merged to `main`).
**Parents:**
[`2026-06-05-machine-altered-perception-design.md`](./2026-06-05-machine-altered-perception-design.md)
(the alteration engine) and
[`2026-06-04-experience-simulator-design.md`](./2026-06-04-experience-simulator-design.md)
(the simulator scaffold).
**Supersedes:** the **baked-HUD / 5×5-grid** position of the 0007 design (§4, §5, §8 there).

> **Why this exists.** Sessions 0007 and 0008 left two unmerged design threads that
> disagree on one load-bearing point — how the **Left** analytical HUD is rendered —
> and, downstream of that, on the **shape of the pre-baked variant set**. 0007 baked
> the Left HUD into a 5×5 grid of authored Left×Right variant clips (pixel-precise,
> but i18n becomes expensive). 0008 (the later, merged design) kept the Left HUD a
> **runtime overlay** driven by an authored annotation track + per-language string
> tables (near-free i18n). This document picks the runtime-overlay position, follows
> its consequences through the engine and the simulator, and scopes the first
> **simulator-runnable** slice that realizes it. It then hands off to an
> implementation plan.

---

## 1. The decision (operator-approved this session)

**Left is a runtime overlay, not baked pixels.** Concretely:

- **Left axis** → a **runtime `AnalyticalOverlay`** driven by an **offline-authored
  annotation track** (box positions, anchor points, and which label *keys* appear at
  each Left level 0–4) plus a per-language **string table** (key → translated text).
  Text is **shaped live** — natively by the browser in the simulator; by
  **Pango + HarfBuzz + Noto** on the Pi at runtime (the 0008 §1.2 requirement). This
  is the "authored box positions + runtime-shaped text" hybrid: authorial control
  over *layout*, near-free *i18n*.
- **Right axis** → the **only pre-baked axis**: a small set of **flow-stabilized
  restyle-strength variants** (0008 §1), selected **discretely** by the Right knob
  (0–4), with a **crossfade** on change.
- **Dark/Light** → a **live runtime `ColorGrade`** (mood center = identity, §5 of the
  parent).

**What this supersedes.** The 0007 design's 5×5 Left×Right **grid of 24 authored
clips** and its **removal of `AnalyticalOverlay`/`Restyle` from `RenderPlan`** are
dropped. The Left HUD is *not* baked; the variant set is **1-D over Right strength**,
not a 2-D grid. (Accepted loss vs. fully-baked HUD: no pixel-painted HUD *artwork* —
the HUD is shaped text + drawn boxes. Accepted gain: near-free i18n + far less
authoring — N Right variants per clip, not 24.)

**What it preserves from 0007.** The valuable, non-conflicting parts: Python-canonical
engine with a thin browser renderer; a parameterized **`Calibration`** tuned by eye in
the sim; retiring the simulator's selection-era surface; placeholder-variant generation
so the mechanism is testable before real authored media exists.

### 1.1 The merged engine is already most of the way there

The slice-1 engine on `main` (`player/alteration.py`) **already** models
`RenderPlan = { grade, overlay, restyle }` with a runtime `AnalyticalOverlay` — only
0007's *unmerged doc* proposed removing it. So this reconciliation is **surgical**, not
a rewrite: keep `grade` and `overlay`; change only how the **Right** axis and the
**calibration** are modeled.

---

## 2. Engine reconciliation (`player/alteration.py`, `player/state.py`)

### 2.1 `RenderPlan` layers

| Layer | Axis | Type | Change from `main` |
|---|---|---|---|
| `grade: ColorGrade` | Dark/Light | `tone ∈ [−1,1]`, 0 = identity | unchanged shape; curve now from `Calibration` |
| `overlay: AnalyticalOverlay` | Left | gains discrete `level: int` (0–4) + keeps `intensity` | `level` added so the renderer selects which annotations show |
| `restyle: Restyle` | Right | **discrete** `variant: int` (0–4); 0 = raw | replaces continuous `blend: float` |

- **`Restyle.variant`** is a discrete index selecting a **pre-baked** Right-strength
  clip. `variant == 0` means the raw base (no restyle). This matches "select a
  pre-baked variant and crossfade," and replaces the continuous `blend` that no longer
  has a runtime meaning (restyle is pre-baked, not blended live).
- **`AnalyticalOverlay.level`** (0–4) is the Left knob value; the renderer uses it to
  choose which annotations from the authored track are active. `intensity` (0..1)
  stays for overlay opacity/strength and is derived from `level` via `Calibration`.
- **`RenderPlan.is_identity`** holds when `grade.is_identity and overlay.level == 0
  and restyle.variant == 0`.

### 2.2 `Calibration` (new frozen dataclass)

A frozen `Calibration` parameterizes the knob→strength maps so they can be tuned **by
eye in the sim** and then baked into a `DEFAULT_CALIBRATION` constant:

- `mood_center` and a per-axis curve for `_mood_tone` (Dark/Light).
- the Left `level → intensity` curve.
- the Right `knob → variant` map (which knob positions select which pre-baked
  strength; identity-preserving so knob 0 → variant 0).

`plan_alteration(coord, calibration: Calibration = DEFAULT_CALIBRATION) -> RenderPlan`.
**`DEFAULT_CALIBRATION` reproduces today's exact behavior** (the three current helpers:
`value/4` for Left intensity, `(light−dark)/4` for mood, `right` → `variant=right`), so
the change is behavior-preserving until the operator tunes it. The session-0006
knob→strength open decision is then settled **by eye** in the sim's calibration panel
and locked into `DEFAULT_CALIBRATION` by a unit test.

### 2.3 `player/state.py`

`state.py` already classifies a change to `plan.restyle` as a `CROSSFADE` and a grade-/
overlay-only change as `LIVE_UPDATE`. With `Restyle.variant` discrete, the same
classifier works unchanged: a new Right variant → `CROSSFADE`; a grade or Left-overlay
change → `LIVE_UPDATE`; clip swap or video on/off → crossfade / fade-to-black as today.
The only edit is to the `_classify` comparison if the field name changes
(`restyle.blend` → `restyle.variant`).

These are pure framework-code changes; no deployment-shape decision enters the engine.

---

## 3. Simulator (`simulator/`)

### 3.1 Retire the selection era

The simulator currently visualizes the **old selection model** (the "curator's X-ray").
Remove:

- `POST /api/select`, `GET /api/catalog/meta`;
- the X-ray static UI (`simulator/static/*` rewritten, see §3.4);
- `simulator/fixtures.py` (the 625-record synthetic *catalog*) — replaced by
  `simulator/clips.py`.

`hef.selection` (the library, incl. `Coordinate`, `ranked_candidates`) is **untouched**;
only the simulator's selection *surface* retires. `Coordinate` is still used by the
alteration engine.

### 3.2 `simulator/clips.py` (replaces `fixtures.py`)

Reads a base-clip + variant manifest. Per base clip:

```
{
  "id": "forest",
  "title": "...",
  "base_file": "forest/base.mp4",
  "license": "...", "source": "...",
  "right_variants": { "1": {"file": "forest/right1.mp4", "model": "..."},
                      "...": {...}, "4": {"file": "forest/right4.mp4"} },
  "annotations": [ {"key": "detected.conifer", "box": [x,y,w,h], "min_level": 1}, ... ],
  "strings": { "en": { "detected.conifer": "conifer", ... } }
}
```

- `right_variants` is keyed by Right strength `1..4` (strength `0` = raw `base_file`).
  Missing strengths fall back to the raw base and are flagged "raw / unauthored" in the
  readout.
- `annotations` is the **authored annotation track** (box + label key + the minimum
  Left level at which it appears). `strings` is the per-language table (English only
  this slice).

### 3.3 Endpoints

- **`POST /api/alteration`** — body `{ controls, calibration? }` → `RenderPlan`
  (serialized) **plus** the `ContentResolution` from `resolve_content` (so the content
  dial's video-on/off is honored — "Off" → black). Calls the real
  `plan_alteration(coord, calibration)`.
- **`GET /api/clips`** — the base-clip list + the active clip's manifest (variants +
  annotation track + string table).
- Video served as static assets from a sample-media directory.
- **Removed:** `POST /api/select`, `GET /api/catalog/meta`.

### 3.4 Browser preview (`simulator/static/`)

Thin renderer; all alteration math stays in Python.

- **Right** — `<video>` showing the selected Right-variant file; **opacity crossfade**
  to the new file when `restyle.variant` changes; variant 0 → the raw base.
- **Dark/Light** — **live color grade** over the video via CSS/canvas filters: Light →
  warm + lifted toward white; Dark → cool + crushed toward black; tone 0 → raw.
- **Left** — overlay drawn **live** from the annotation track: for each annotation with
  `min_level ≤ overlay.level`, draw its box and the **shaped** string (browser-native
  shaping) for the active language; opacity from `overlay.intensity`. No baked HUD.
- **Content dial** — drives `<video>` visibility; "Off"/audio-only → black walls.
- **Calibration panel** — sliders for the `Calibration` params; changing them
  re-requests the plan and the footage responds live.
- **RenderPlan readout** — always shows the exact engine numbers (grade tone, overlay
  level/intensity, restyle variant) — the project's honesty "X-ray," now over the
  alteration model.

The Left overlay being browser-drawn is the **simulator analogue** of the Pi's
Pango/HarfBuzz path: both take the *same* annotation track + string table; the browser
shapes natively, the Pi shapes with HarfBuzz. The manifest is the shared contract.

---

## 4. The sample clip + a real Right variant (this slice)

To make the look **evaluable now**, wire one real clip end-to-end using the session-0008
POC artifacts (`~/hef-poc/out/`, outside the repo):

- **Base clip** = the POC's `neutral.mp4` (an 8 s nature loop) → copied to the
  sample-media dir as the one base clip.
- **Right variant (top strength)** = the POC's **`right_flow.mp4`** — the real,
  operator-approved **flow-stabilized** restyle — wired as Right strength 4.
- **Intermediate Right strengths (1–3)** = generated by a small **ffmpeg placeholder
  generator** (e.g. graded/blended stand-ins) so the crossfade mechanism is exercised
  across the full knob range; the real high-end look is present for tuning.
- **Left annotation track** = a minimal authored track (a few boxes + English label
  keys) for that clip, so Left renders as real shaped text over drawn boxes.

> **Licensing note.** The simulator sample footage exists **only to tune the look**; it
> is not shipped installation content. Strict-PD scale-library sourcing (NASA/NOAA/NPS
> per 0008 §2.1) remains a later slice and is unaffected by this choice.

A real multi-strength SD re-bake (4 genuine flow-stabilized strengths) is **out of scope
this slice** — one real strength + placeholders is enough to settle the mechanism and the
calibration. The re-bake is a later offline-pipeline task.

---

## 5. Testing

- **`player/` unit tests** (`tests/test_player_alteration.py`, `test_player_state.py`):
  - `DEFAULT_CALIBRATION` reproduces the current helpers exactly (behavior-preserving).
  - `Restyle.variant` is discrete; knob 0 → variant 0 (identity); a non-default
    `Calibration` changes the plan as specified.
  - `AnalyticalOverlay.level` maps from the Left knob; `intensity` derives from it.
  - `state.py`: a Right-variant change → `CROSSFADE`; a grade-/overlay-only change →
    `LIVE_UPDATE`; video on/off unchanged.
- **Simulator API tests** (rewrite `tests/test_simulator_api.py`):
  - `POST /api/alteration` returns the engine's plan (+ `ContentResolution`) for given
    controls/calibration.
  - `GET /api/clips` returns the manifest.
  - the removed endpoints (`/api/select`, `/api/catalog/meta`) are gone (404).
  - `test_fixtures.py` retired/rewritten for `clips.py`.
- **No browser/E2E automation** this slice — manual visual tuning is the point; the JS
  stays thin and the tested logic stays in Python.

---

## 6. What ships

- `player/alteration.py` — `Calibration` + `DEFAULT_CALIBRATION`, discrete
  `Restyle.variant`, `AnalyticalOverlay.level`; `plan_alteration(coord, calibration)`.
- `player/state.py` — crossfade-trigger field rename only.
- `simulator/clips.py` (variant + annotation manifest) replacing `fixtures.py`.
- `simulator/app.py` — `/api/alteration` + `/api/clips`; selection endpoints removed.
- `simulator/static/` — rewritten as the alteration preview (variant `<video>` +
  crossfade, live grade, live Left overlay, content dial, calibration panel,
  RenderPlan readout).
- Sample base clip + one real Right variant + placeholder generator + minimal Left
  annotation track / English strings.
- Tests above; `docs/USER_GUIDE.md` "Playing with the simulator" rewritten; the parent
  design §4.3/§10 pointer updated to cite this reconciliation; `docs/ROADMAP.md` §3
  updated.

---

## 7. Out of scope (YAGNI) — later slices

- Serial input / the 3⇄4 framing contract; the Pi/mpv/GPU runtime renderer (deferred by
  `simulator-first-before-hardware`).
- Audio playback (music / white-noise / audio-track).
- The **endless rotary encoder** + **AI zoom/warp transitions** between scales
  (0008 §3) — a separate control + offline pipeline element.
- A real multi-strength SD flow-stabilized re-bake; strict-PD scale-library sourcing
  (0008 §2.1).
- Catalog-model changes (audio source / neutral-vs-variant flag); retiring
  `hef.selection.ranked_candidates`.
- Broad multilingual string tables (English-first; the runtime path keeps i18n cheap,
  but authoring other languages is later).

---

## 8. Open questions (for the plan, not blockers)

- **Calibration curve shape** — **RESOLVED (session 0010, by eye).**
  `DEFAULT_CALIBRATION` is **locked** to unity gains + a linear variant map
  (`mood_gain=1.0`, `overlay_gain=1.0`, `right_variant_map=(0,1,2,3,4)`), as a
  deliberate choice: with the dark-grade fix below, full knob is peaceful on
  every axis (POC + sim), so full tilt = full look and the 5 notches map 1:1 to
  the 5 discrete Right bakes. This also closes the **session-0006** convention
  question — knobs run 0=off..4=max, equal Dark/Light = identity; no
  "centered at 2 = no push." Guarded by `test_default_calibration_is_locked`.
- **Grade vs. Left overlay interaction** — **RESOLVED: overlay above the grade.**
  The simulator composites the Left HUD (SVG) above the mood grade and the cool
  tint, so the HUD stays legible regardless of mood. The Pi renderer should do
  the same.
- **Dark-pole grade look** — **FIXED (session 0010).** The first by-eye pass found
  the sim's dark grade used a full-frame `hue-rotate(-200deg)`, which turned the
  rock orange and trees purple — the disorienting look rejected in 0008, not the
  peaceful POC `dark_frame`. Replaced with darken + slight desaturate on the video
  filter plus a `multiply`-blended deep-blue wash (`#tint`) that lifts shadows
  toward blue while preserving natural greens. The Pi renderer (later slice) will
  do proper grading; this matches the approved POC dark look closely enough to tune
  by eye in the sim.
- **Crossfade timing in the browser** — a simple opacity crossfade is enough for tuning;
  the real timing engine is a later slice.
- **Placeholder fidelity** — how close the strength-1–3 placeholders should look to real
  restyle; cheap stand-ins are fine for mechanism + calibration.
