docs(plan): v2v vertical slice — real forest Right variants

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 06:32:04 -07:00
parent 49b1f0a549
commit 95d08d5c01
1 changed files with 239 additions and 0 deletions
@@ -0,0 +1,239 @@
+# v2v Vertical Slice — Real Forest Right Variants Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Produce the first *real* (not placeholder) multi-strength Right-axis
+restyle media for the simulator — flow-stabilized painterly variants of the
+forest scale at Right strengths 1–4 — so the dreamlike-axis *look* can finally be
+judged by eye on real footage, breaking the "can't judge the ring liked without
+real content" chicken-and-egg (operator chose a vertical slice, session 0012).
+
+**Architecture:** Productionize the proven session-0008 POC restyle algorithm
+(`~/hef-poc/flow_restyle.py`) into a repo-resident offline baker,
+`simulator/bake_right_variants.py`, a sibling of the existing
+`setup_sample_media.py` / `setup_scales_media.py` media-generation scripts. The
+algorithm is unchanged (extract frames → SD img2img on keyframes → Farneback
+optical-flow warp + light refine on tweens → reassemble), which sessions 0008
+proved is temporally calm (no boiling). The one new idea is a **strength curve**:
+Right level 1–4 maps to an increasing keyframe img2img `strength`, so the four
+discrete pre-baked variants form a subtle→strong dreamlike ramp. Only the curve
+helper is unit-tested; the rendered media is verified by eye (screenshots), as
+the other media-setup scripts are.
+
+**Tech Stack:** Python, diffusers `AutoPipelineForImage2Image` + `stabilityai/sd-turbo`
+(cached, 4.8 G) on Apple MPS (torch 2.12, available in the project `.venv`),
+OpenCV Farneback optical flow, imageio-ffmpeg. ~2.7 min/clip × 4 ≈ ~11 min local
+compute.
+
+**Scope note — what this slice is NOT:** the *real i2v ring transition* half of
+the v2v build is deferred: it needs a second real scale base (cosmos/abyss are
+still synthetic placeholders) plus a generative-video model not in the POC, and
+real strict-PD footage sourcing (NASA/NOAA, sub-project 2). This slice does the
+Right-variant half, which is fully achievable now (forest has a real base).
+
+---
+
+### Task 1: The Right strength curve (pure, testable)
+
+**Files:**
+- Create: `simulator/bake_right_variants.py`
+- Test: `tests/test_bake_right_variants.py`
+
+- [ ] **Step 1: Write the failing test**
+
+```python
+# tests/test_bake_right_variants.py
+import pytest
+
+from simulator.bake_right_variants import right_strength_params, RIGHT_LEVELS
+
+
+def test_levels_are_one_through_four():
+    assert RIGHT_LEVELS == (1, 2, 3, 4)
+
+
+def test_strength_increases_with_level():
+    keys = [right_strength_params(l)[0] for l in RIGHT_LEVELS]
+    assert keys == sorted(keys)            # monotonic ramp
+    assert keys[0] < keys[-1]              # subtle -> strong
+
+
+def test_tween_strength_is_a_fraction_of_key_strength():
+    for l in RIGHT_LEVELS:
+        key, tween = right_strength_params(l)
+        assert 0.0 < tween < key           # tween is a light refine only
+
+
+def test_rejects_out_of_range_level():
+    with pytest.raises(ValueError):
+        right_strength_params(0)
+    with pytest.raises(ValueError):
+        right_strength_params(5)
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `.venv/bin/python -m pytest tests/test_bake_right_variants.py -q`
+Expected: FAIL (ModuleNotFoundError / ImportError — module not created yet)
+
+- [ ] **Step 3: Write minimal implementation (curve only, no torch import at module load)**
+
+```python
+# simulator/bake_right_variants.py  (top of file)
+"""Offline baker: real flow-stabilized painterly Right-axis variants (scales
+design §1/§4). Productionizes the session-0008 POC (~/hef-poc/flow_restyle.py):
+SD img2img keyframes + Farneback optical-flow tweens = temporally calm restyle.
+
+Right level -> keyframe img2img strength ramp (subtle -> strong); tween strength
+is a light refine fraction of the keyframe strength. Curve is by-eye tunable.
+
+Usage:  .venv/bin/python -m simulator.bake_right_variants [--scale forest]
+Requires: torch+MPS, diffusers, sd-turbo (cached), opencv, imageio-ffmpeg.
+"""
+from __future__ import annotations
+
+RIGHT_LEVELS = (1, 2, 3, 4)
+
+# Keyframe img2img strength per Right level (by-eye ramp). Level 4 ~ the POC's
+# proven 0.5-0.58 painterly look; level 1 is a barely-there dream haze.
+_KEY_STRENGTH = {1: 0.28, 2: 0.38, 3: 0.48, 4: 0.58}
+_TWEEN_RATIO = 0.36  # POC ratio (0.18 / 0.5): tweens are a light refine only
+
+
+def right_strength_params(level: int) -> tuple[float, float]:
+    """(keyframe_strength, tween_strength) for a Right level in RIGHT_LEVELS."""
+    if level not in _KEY_STRENGTH:
+        raise ValueError(f"Right level must be one of {RIGHT_LEVELS}, got {level}")
+    key = _KEY_STRENGTH[level]
+    return key, round(key * _TWEEN_RATIO, 4)
+```
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run: `.venv/bin/python -m pytest tests/test_bake_right_variants.py -q`
+Expected: PASS (4 passed)
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add simulator/bake_right_variants.py tests/test_bake_right_variants.py
+git commit -m "feat(pipeline): Right strength curve for the v2v variant baker"
+```
+
+---
+
+### Task 2: The baker (flow restyle, productionized)
+
+**Files:**
+- Modify: `simulator/bake_right_variants.py`
+
+No unit test for the render itself (non-trivial ML output; verified by eye in
+Task 4). The pure curve is already covered by Task 1.
+
+- [ ] **Step 1: Add the restyle engine + per-level bake, porting the POC algorithm verbatim**
+
+Port `extract`, `warp`, and the keyframe/tween loop from
+`~/hef-poc/flow_restyle.py` unchanged (it is the approved calm algorithm). Wrap
+it as `restyle_clip(src, out, key_strength, tween_strength, *, fps=12.0, keyint=24)`
+and add a `bake_scale(scale="forest")` that loops `RIGHT_LEVELS`, calling
+`restyle_clip` with `right_strength_params(level)` and writing
+`simulator/sample_media/<scale>/right<level>.mp4`. Load the SD pipeline ONCE and
+reuse it across levels. `PROMPT`/`NEG`/Farneback params/seed = the POC's.
+
+Source base = `simulator/sample_media/<scale>/base.mp4` (forest = the real
+Yosemite POC clip). `main()` parses `--scale` (default `forest`), `--fps`,
+`--keyint` and calls `bake_scale`.
+
+- [ ] **Step 2: Smoke-check the module imports and the curve still passes**
+
+Run: `.venv/bin/python -m pytest tests/test_bake_right_variants.py -q`
+Expected: PASS (torch imported lazily inside the bake funcs, not at module load,
+so the test stays fast and import-safe)
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add simulator/bake_right_variants.py
+git commit -m "feat(pipeline): forest Right-variant baker (POC flow restyle, productionized)"
+```
+
+---
+
+### Task 3: Run the bake (real media, ~11 min)
+
+**Files:**
+- Writes (gitignored): `simulator/sample_media/forest/right{1,2,3,4}.mp4`
+
+- [ ] **Step 1: Run the baker**
+
+Run: `.venv/bin/python -m simulator.bake_right_variants --scale forest`
+Expected: 4 clips written; per-level timing printed; ~11 min total. Replaces the
+3 placeholder strengths (1–3) and the single real strength (4) with a consistent
+real ramp.
+
+- [ ] **Step 2: Confirm the outputs exist and are real video**
+
+Run: `ls -la simulator/sample_media/forest/right*.mp4`
+Expected: right1–4.mp4 present, each a multi-MB H.264 clip.
+
+(No commit — media is gitignored; it is reproduced by the baker.)
+
+---
+
+### Task 4: Wire the manifest + verify by eye
+
+**Files:**
+- Modify: `simulator/sample_media/manifest.json` (forest `right_variants` models)
+- Modify: `docs/USER_GUIDE.md` (note real forest Right variants + the baker)
+- Modify: `docs/ROADMAP.md` (v2v slice progress)
+
+- [ ] **Step 1: Update the forest variant models in the manifest**
+
+Set every forest `right_variants` entry (1–4) `model` to
+`"sd-turbo+farneback-flow"` (drop the `"placeholder"` labels) — they are all real
+bakes now.
+
+- [ ] **Step 2: Boot the sim and drive the Right axis across 0→4 on forest**
+
+Run: `.venv/bin/python -m uvicorn simulator.app:app --port 8013` (background),
+navigate to the forest scale, and capture headless screenshots at Right 0/1/2/3/4.
+Expected: a coherent subtle→strong painterly ramp on the real Yosemite footage,
+temporally calm (no boiling), grade/overlay still compose on top.
+
+- [ ] **Step 3: Update docs**
+
+USER_GUIDE: note forest now carries real flow-stabilized Right variants generated
+by `simulator/bake_right_variants.py`. ROADMAP: mark the v2v Right-variant vertical
+slice done; note the i2v ring-transition half still deferred (needs a 2nd real
+base + video model).
+
+- [ ] **Step 4: Run the full suite**
+
+Run: `.venv/bin/python -m pytest -q`
+Expected: PASS (existing + Task-1 curve tests).
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add simulator/sample_media/manifest.json docs/USER_GUIDE.md docs/ROADMAP.md
+git commit -m "feat(sim): forest carries real multi-strength Right variants; docs"
+```
+
+---
+
+## Self-Review
+
+**Spec coverage:** scales design §1 (local restyle pipeline) + §4 (economics:
+local bake) → Tasks 2–3 productionize + run it. The operator's vertical-slice
+choice (real Right variants for one scale to judge the look) → Tasks 3–4. The
+deferred i2v transition half is explicitly scoped out (noted in header + Task 4).
+The strength curve (the one new design element) → Task 1.
+
+**Placeholder scan:** none — the baker algorithm is a verbatim port of the
+existing `~/hef-poc/flow_restyle.py` (cited), the curve is concrete constants,
+commands are exact.
+
+**Type consistency:** `right_strength_params(level) -> (key, tween)` defined in
+Task 1, consumed by `bake_scale` in Task 2. `RIGHT_LEVELS` shared. Output paths
+`simulator/sample_media/forest/right<level>.mp4` consistent with the manifest's
+existing forest `right_variants` map (Task 4).