feat(v2v): real multi-strength Right variants for the forest scale (vertical slice) #11
@@ -0,0 +1,239 @@
|
||||
# v2v Vertical Slice — Real Forest Right Variants Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Produce the first *real* (not placeholder) multi-strength Right-axis
|
||||
restyle media for the simulator — flow-stabilized painterly variants of the
|
||||
forest scale at Right strengths 1–4 — so the dreamlike-axis *look* can finally be
|
||||
judged by eye on real footage, breaking the "can't judge the ring liked without
|
||||
real content" chicken-and-egg (operator chose a vertical slice, session 0012).
|
||||
|
||||
**Architecture:** Productionize the proven session-0008 POC restyle algorithm
|
||||
(`~/hef-poc/flow_restyle.py`) into a repo-resident offline baker,
|
||||
`simulator/bake_right_variants.py`, a sibling of the existing
|
||||
`setup_sample_media.py` / `setup_scales_media.py` media-generation scripts. The
|
||||
algorithm is unchanged (extract frames → SD img2img on keyframes → Farneback
|
||||
optical-flow warp + light refine on tweens → reassemble), which sessions 0008
|
||||
proved is temporally calm (no boiling). The one new idea is a **strength curve**:
|
||||
Right level 1–4 maps to an increasing keyframe img2img `strength`, so the four
|
||||
discrete pre-baked variants form a subtle→strong dreamlike ramp. Only the curve
|
||||
helper is unit-tested; the rendered media is verified by eye (screenshots), as
|
||||
the other media-setup scripts are.
|
||||
|
||||
**Tech Stack:** Python, diffusers `AutoPipelineForImage2Image` + `stabilityai/sd-turbo`
|
||||
(cached, 4.8 G) on Apple MPS (torch 2.12, available in the project `.venv`),
|
||||
OpenCV Farneback optical flow, imageio-ffmpeg. ~2.7 min/clip × 4 ≈ ~11 min local
|
||||
compute.
|
||||
|
||||
**Scope note — what this slice is NOT:** the *real i2v ring transition* half of
|
||||
the v2v build is deferred: it needs a second real scale base (cosmos/abyss are
|
||||
still synthetic placeholders) plus a generative-video model not in the POC, and
|
||||
real strict-PD footage sourcing (NASA/NOAA, sub-project 2). This slice does the
|
||||
Right-variant half, which is fully achievable now (forest has a real base).
|
||||
|
||||
---
|
||||
|
||||
### Task 1: The Right strength curve (pure, testable)
|
||||
|
||||
**Files:**
|
||||
- Create: `simulator/bake_right_variants.py`
|
||||
- Test: `tests/test_bake_right_variants.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
# tests/test_bake_right_variants.py
|
||||
import pytest
|
||||
|
||||
from simulator.bake_right_variants import right_strength_params, RIGHT_LEVELS
|
||||
|
||||
|
||||
def test_levels_are_one_through_four():
|
||||
assert RIGHT_LEVELS == (1, 2, 3, 4)
|
||||
|
||||
|
||||
def test_strength_increases_with_level():
|
||||
keys = [right_strength_params(l)[0] for l in RIGHT_LEVELS]
|
||||
assert keys == sorted(keys) # monotonic ramp
|
||||
assert keys[0] < keys[-1] # subtle -> strong
|
||||
|
||||
|
||||
def test_tween_strength_is_a_fraction_of_key_strength():
|
||||
for l in RIGHT_LEVELS:
|
||||
key, tween = right_strength_params(l)
|
||||
assert 0.0 < tween < key # tween is a light refine only
|
||||
|
||||
|
||||
def test_rejects_out_of_range_level():
|
||||
with pytest.raises(ValueError):
|
||||
right_strength_params(0)
|
||||
with pytest.raises(ValueError):
|
||||
right_strength_params(5)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/test_bake_right_variants.py -q`
|
||||
Expected: FAIL (ModuleNotFoundError / ImportError — module not created yet)
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation (curve only, no torch import at module load)**
|
||||
|
||||
```python
|
||||
# simulator/bake_right_variants.py (top of file)
|
||||
"""Offline baker: real flow-stabilized painterly Right-axis variants (scales
|
||||
design §1/§4). Productionizes the session-0008 POC (~/hef-poc/flow_restyle.py):
|
||||
SD img2img keyframes + Farneback optical-flow tweens = temporally calm restyle.
|
||||
|
||||
Right level -> keyframe img2img strength ramp (subtle -> strong); tween strength
|
||||
is a light refine fraction of the keyframe strength. Curve is by-eye tunable.
|
||||
|
||||
Usage: .venv/bin/python -m simulator.bake_right_variants [--scale forest]
|
||||
Requires: torch+MPS, diffusers, sd-turbo (cached), opencv, imageio-ffmpeg.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
RIGHT_LEVELS = (1, 2, 3, 4)
|
||||
|
||||
# Keyframe img2img strength per Right level (by-eye ramp). Level 4 ~ the POC's
|
||||
# proven 0.5-0.58 painterly look; level 1 is a barely-there dream haze.
|
||||
_KEY_STRENGTH = {1: 0.28, 2: 0.38, 3: 0.48, 4: 0.58}
|
||||
_TWEEN_RATIO = 0.36 # POC ratio (0.18 / 0.5): tweens are a light refine only
|
||||
|
||||
|
||||
def right_strength_params(level: int) -> tuple[float, float]:
|
||||
"""(keyframe_strength, tween_strength) for a Right level in RIGHT_LEVELS."""
|
||||
if level not in _KEY_STRENGTH:
|
||||
raise ValueError(f"Right level must be one of {RIGHT_LEVELS}, got {level}")
|
||||
key = _KEY_STRENGTH[level]
|
||||
return key, round(key * _TWEEN_RATIO, 4)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/test_bake_right_variants.py -q`
|
||||
Expected: PASS (4 passed)
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add simulator/bake_right_variants.py tests/test_bake_right_variants.py
|
||||
git commit -m "feat(pipeline): Right strength curve for the v2v variant baker"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: The baker (flow restyle, productionized)
|
||||
|
||||
**Files:**
|
||||
- Modify: `simulator/bake_right_variants.py`
|
||||
|
||||
No unit test for the render itself (non-trivial ML output; verified by eye in
|
||||
Task 4). The pure curve is already covered by Task 1.
|
||||
|
||||
- [ ] **Step 1: Add the restyle engine + per-level bake, porting the POC algorithm verbatim**
|
||||
|
||||
Port `extract`, `warp`, and the keyframe/tween loop from
|
||||
`~/hef-poc/flow_restyle.py` unchanged (it is the approved calm algorithm). Wrap
|
||||
it as `restyle_clip(src, out, key_strength, tween_strength, *, fps=12.0, keyint=24)`
|
||||
and add a `bake_scale(scale="forest")` that loops `RIGHT_LEVELS`, calling
|
||||
`restyle_clip` with `right_strength_params(level)` and writing
|
||||
`simulator/sample_media/<scale>/right<level>.mp4`. Load the SD pipeline ONCE and
|
||||
reuse it across levels. `PROMPT`/`NEG`/Farneback params/seed = the POC's.
|
||||
|
||||
Source base = `simulator/sample_media/<scale>/base.mp4` (forest = the real
|
||||
Yosemite POC clip). `main()` parses `--scale` (default `forest`), `--fps`,
|
||||
`--keyint` and calls `bake_scale`.
|
||||
|
||||
- [ ] **Step 2: Smoke-check the module imports and the curve still passes**
|
||||
|
||||
Run: `.venv/bin/python -m pytest tests/test_bake_right_variants.py -q`
|
||||
Expected: PASS (torch imported lazily inside the bake funcs, not at module load,
|
||||
so the test stays fast and import-safe)
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add simulator/bake_right_variants.py
|
||||
git commit -m "feat(pipeline): forest Right-variant baker (POC flow restyle, productionized)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Run the bake (real media, ~11 min)
|
||||
|
||||
**Files:**
|
||||
- Writes (gitignored): `simulator/sample_media/forest/right{1,2,3,4}.mp4`
|
||||
|
||||
- [ ] **Step 1: Run the baker**
|
||||
|
||||
Run: `.venv/bin/python -m simulator.bake_right_variants --scale forest`
|
||||
Expected: 4 clips written; per-level timing printed; ~11 min total. Replaces the
|
||||
3 placeholder strengths (1–3) and the single real strength (4) with a consistent
|
||||
real ramp.
|
||||
|
||||
- [ ] **Step 2: Confirm the outputs exist and are real video**
|
||||
|
||||
Run: `ls -la simulator/sample_media/forest/right*.mp4`
|
||||
Expected: right1–4.mp4 present, each a multi-MB H.264 clip.
|
||||
|
||||
(No commit — media is gitignored; it is reproduced by the baker.)
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Wire the manifest + verify by eye
|
||||
|
||||
**Files:**
|
||||
- Modify: `simulator/sample_media/manifest.json` (forest `right_variants` models)
|
||||
- Modify: `docs/USER_GUIDE.md` (note real forest Right variants + the baker)
|
||||
- Modify: `docs/ROADMAP.md` (v2v slice progress)
|
||||
|
||||
- [ ] **Step 1: Update the forest variant models in the manifest**
|
||||
|
||||
Set every forest `right_variants` entry (1–4) `model` to
|
||||
`"sd-turbo+farneback-flow"` (drop the `"placeholder"` labels) — they are all real
|
||||
bakes now.
|
||||
|
||||
- [ ] **Step 2: Boot the sim and drive the Right axis across 0→4 on forest**
|
||||
|
||||
Run: `.venv/bin/python -m uvicorn simulator.app:app --port 8013` (background),
|
||||
navigate to the forest scale, and capture headless screenshots at Right 0/1/2/3/4.
|
||||
Expected: a coherent subtle→strong painterly ramp on the real Yosemite footage,
|
||||
temporally calm (no boiling), grade/overlay still compose on top.
|
||||
|
||||
- [ ] **Step 3: Update docs**
|
||||
|
||||
USER_GUIDE: note forest now carries real flow-stabilized Right variants generated
|
||||
by `simulator/bake_right_variants.py`. ROADMAP: mark the v2v Right-variant vertical
|
||||
slice done; note the i2v ring-transition half still deferred (needs a 2nd real
|
||||
base + video model).
|
||||
|
||||
- [ ] **Step 4: Run the full suite**
|
||||
|
||||
Run: `.venv/bin/python -m pytest -q`
|
||||
Expected: PASS (existing + Task-1 curve tests).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add simulator/sample_media/manifest.json docs/USER_GUIDE.md docs/ROADMAP.md
|
||||
git commit -m "feat(sim): forest carries real multi-strength Right variants; docs"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
**Spec coverage:** scales design §1 (local restyle pipeline) + §4 (economics:
|
||||
local bake) → Tasks 2–3 productionize + run it. The operator's vertical-slice
|
||||
choice (real Right variants for one scale to judge the look) → Tasks 3–4. The
|
||||
deferred i2v transition half is explicitly scoped out (noted in header + Task 4).
|
||||
The strength curve (the one new design element) → Task 1.
|
||||
|
||||
**Placeholder scan:** none — the baker algorithm is a verbatim port of the
|
||||
existing `~/hef-poc/flow_restyle.py` (cited), the curve is concrete constants,
|
||||
commands are exact.
|
||||
|
||||
**Type consistency:** `right_strength_params(level) -> (key, tween)` defined in
|
||||
Task 1, consumed by `bake_scale` in Task 2. `RIGHT_LEVELS` shared. Output paths
|
||||
`simulator/sample_media/forest/right<level>.mp4` consistent with the manifest's
|
||||
existing forest `right_variants` map (Task 4).
|
||||
Reference in New Issue
Block a user