benstull/human-experience-filter-art
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(sim): Right axis = deterministic real-time painterly dream

Browse Source 
Replace the pre-baked SD restyle variants (too fluid — the painting re-rolled
every keyframe and the flow-warp melted frames) with a deterministic, real-time
dream that holds STILL across the loop; only the footage's own motion moves.
Design: docs/superpowers/specs/2026-06-22-right-axis-deterministic-dream-design.md

Engine (player/alteration.py): Restyle(variant) -> Dream(strength, intensity);
Calibration.right_variant_map -> dream_gain. player/state.py: a Right change is a
LIVE_UPDATE now, not a crossfade (only a clip swap crossfades). Plan dict
restyle -> dream. Tests migrated; 233 green. (No new Python behavior for the
client-render swap below.)

Renderer (simulator front-end): the Right dream is a WebGL2 Kuwahara shader on a
<canvas> over the live video — edge-preserving, so motion/structure stay crisp;
the dream is the same footage, just stylized (no blur). Ramped by the knob; max
goes "trippy" (vivid saturation + ~45deg hue drift + posterize, concentrated near
max via t=intensity^2). Phase-1 luminous-haze (CSS+bloom) was built then retired
by eye (blur read as out-of-focus video). Mood grade rides as a CSS filter on the
canvas; bloom layer removed.

Dev ergonomics that fell out of the iteration:
- /dev/version + a 1s client poll = live-reload (open tab never runs stale
  renderer code; reloads on my edits AND server restarts).
- no-cache on the dev server; an on-screen error banner + crash-proof render so a
  silent throw can't masquerade as "nothing is changing".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ben Stull
2026-06-23 09:55:35 -07:00
parent 6adb93407b
commit cbb7c9f53d
10 changed files with 404 additions and 92 deletions
Download Patch File Download Diff File Expand all files Collapse all files
simulator/static/app.js
+181 -34
View File
@@ -1,16 +1,39 @@
// Thin renderer: post controls+calibration -> RenderPlan; render grade, Right
// variant crossfade, and the live Left overlay. All alteration math stays in
// Python. The scale RING (endless encoder) is navigated via /api/ring/advance —
// Python owns the step/wrap/transition-selection math; the browser only plays
// the returned transition clip(s) then settles on the target scale's clip.
// Thin renderer: post controls+calibration -> RenderPlan; render the mood grade,
// the live Right DREAM (a deterministic luminous haze, NOT a baked variant — it
// holds still across the loop), and the live Left overlay + affect. All alteration
// math stays in Python. The scale RING (endless encoder) is navigated via
// /api/ring/advance — Python owns the step/wrap/transition-selection math; the
// browser only plays the returned transition clip(s) then settles on the target.
const $ = (id) => document.getElementById(id);
const vid = $("vid"), tint = $("tint"), overlay = $("overlay"), black = $("black"), readout = $("readout");
const vid = $("vid"), paint = $("paint"), tint = $("tint"), overlay = $("overlay"),
black = $("black"), readout = $("readout");
const affectLayer = $("affect");
// Right-dream state, read by the painterly render loop each frame. update() (debounced
// on knob moves) writes these; the WebGL loop renders the live video continuously.
let dreamIntensity = 0; // 0..1 — painterly strength (and dream pastel/luminous)
let gradeFilter = "none"; // CSS filter string for the Dark/Light mood grade
// Surface any uncaught JS error on-screen (a silent throw in update() otherwise
// looks like "nothing is changing" — the render dies but the page sits there).
function showError(msg) {
let b = document.getElementById("err-banner");
if (!b) {
b = document.createElement("div");
b.id = "err-banner";
b.style.cssText = "position:fixed;top:0;left:0;right:0;z-index:9999;background:#a00;" +
"color:#fff;font:12px/1.4 monospace;padding:6px 10px;white-space:pre-wrap;";
document.body.appendChild(b);
}
b.textContent = "⚠ " + msg;
}
window.addEventListener("error", (e) =>
showError(`${e.message} @ ${(e.filename || "").split("/").pop()}:${e.lineno}`));
let clipsById = {}; // id -> clip manifest entry
let ring = null; // {scales:[{id,clip_id,title}], transitions:[...]} or null
let ringIndex = 0; // current scale position on the ring
let currentVariant = -1; // last loaded Right strength (reset on clip change)
let currentClipId = null; // base media currently loaded (reset to force a reload)
let busy = false; // true while a ring transition is playing
async function loadData() {
@@ -32,36 +55,134 @@ function activeClip() {
function mediaUrl(file) { return "/media/" + file; }
function variantFile(strength) {
// Load the active scale's BASE footage into the video (the painterly canvas reads
// its frames live). The Right knob no longer swaps the source — it only drives the
// deterministic dream restyle — so media reloads only when the SCALE changes.
function ensureClipMedia() {
const clip = activeClip();
if (!clip) return "";
const v = clip.right_variants[String(strength)];
return v ? v.file : clip.base_file;
if (!clip || clip.id === currentClipId) return;
currentClipId = clip.id;
vid.src = mediaUrl(clip.base_file);
vid.loop = true;
vid.muted = true;
vid.play().catch(() => {});
vid.style.opacity = "1";
}
function applyGrade(tone) {
// Light: warm + brighten (sepia). Dark: cool + darken via a multiply-blended
// blue wash (#tint) that lifts shadows toward blue while keeping natural
// greens — the peaceful POC dark look, NOT a full-frame hue spin.
// Compose the video look. The Right DREAM is now a painterly WebGL restyle of the
// live frames (see the render loop below) — so here we only stash its intensity for
// that loop, and build the Dark/Light mood grade as a CSS filter. The grade rides
// on the painterly canvas (or #vid in the WebGL-less fallback). No blur: the dream
// must keep the base's crisp motion, just stylized.
function applyVideoLook(tone, dream) {
const warm = tone > 0 ? tone : 0, cool = tone < 0 ? -tone : 0;
const bright = 1 + 0.25 * warm - 0.35 * cool;
const sat = 1 + 0.15 * warm - 0.30 * cool;
vid.style.filter =
`brightness(${bright.toFixed(3)}) saturate(${sat.toFixed(3)}) ` +
`sepia(${(warm * 0.5).toFixed(3)})`;
const sepia = warm * 0.5;
gradeFilter = `brightness(${bright.toFixed(3)}) saturate(${sat.toFixed(3)}) sepia(${sepia.toFixed(3)})`;
dreamIntensity = dream;
tint.style.opacity = (cool * 0.6).toFixed(3);
// WebGL-less fallback: the canvas is hidden, so grade the visible #vid directly.
if (!paintOK) vid.style.filter = gradeFilter;
}
function loadVariant(strength) {
if (strength === currentVariant) return;
currentVariant = strength;
vid.style.opacity = "0";
setTimeout(() => {
vid.src = mediaUrl(variantFile(strength));
vid.loop = true;
vid.play().catch(() => {});
vid.style.opacity = "1";
}, 150);
// --- Right dream: real-time painterly restyle (WebGL2 Kuwahara) ---
// An edge-preserving "oil painting" filter on the LIVE video frames: each pixel
// becomes the mean of whichever surrounding quadrant is most uniform, flattening
// the image into painted regions WHILE KEEPING EDGES — so motion stays as crisp as
// the source. Strength + a gentle pastel/luminous dream-grade scale with intensity.
// Deterministic (a pure function of each frame) => the dream holds still across the
// loop; only the footage's own motion moves.
let paintOK = false;
const PAINT_VS = `#version 300 es
in vec2 p; out vec2 v_uv;
void main(){
// Fullscreen triangle (verts to +3); map the visible [-1,1] to uv [0,1] with Y
// flipped for the video's top-left origin. (Visible region stays in range; the
// off-screen excess is clipped.)
v_uv = vec2(p.x * 0.5 + 0.5, 0.5 - p.y * 0.5);
gl_Position = vec4(p, 0.0, 1.0);
}`;
const PAINT_FS = `#version 300 es
precision highp float;
in vec2 v_uv; out vec4 frag;
uniform sampler2D u_tex; uniform vec2 u_texel; uniform float u_amt;
#define R 6
#define QUAD(x0,x1,y0,y1,MO,VO) { vec3 m=vec3(0.0),s=vec3(0.0); \
for(int j=y0;j<=y1;j++){ for(int i=x0;i<=x1;i++){ \
vec3 c=texture(u_tex, v_uv+vec2(float(i),float(j))*u_texel).rgb; m+=c; s+=c*c; } } \
float n=float((x1-x0+1)*(y1-y0+1)); m/=n; vec3 vv=s/n-m*m; MO=m; VO=vv.r+vv.g+vv.b; }
vec3 hueRotate(vec3 c, float a){
float co=cos(a), si=sin(a);
return vec3(
c.r*(0.299+0.701*co+0.168*si) + c.g*(0.587-0.587*co+0.330*si) + c.b*(0.114-0.114*co-0.497*si),
c.r*(0.299-0.299*co-0.328*si) + c.g*(0.587+0.413*co+0.035*si) + c.b*(0.114-0.114*co+0.292*si),
c.r*(0.299-0.300*co+1.250*si) + c.g*(0.587-0.588*co-1.050*si) + c.b*(0.114+0.886*co-0.203*si));
}
void main(){
vec3 base = texture(u_tex, v_uv).rgb;
if (u_amt <= 0.001) { frag = vec4(base, 1.0); return; }
vec3 m0,m1,m2,m3; float v0,v1,v2,v3;
QUAD(-R,0,-R,0,m0,v0) QUAD(0,R,-R,0,m1,v1) QUAD(-R,0,0,R,m2,v2) QUAD(0,R,0,R,m3,v3)
vec3 painted=m0; float mv=v0;
if(v1<mv){mv=v1;painted=m1;} if(v2<mv){mv=v2;painted=m2;} if(v3<mv){mv=v3;painted=m3;}
vec3 styl = mix(base, painted, u_amt); // painterly restyle, linear in the knob
// The "trippy" terms ramp with t = amt^2, so low Right stays gently dreamy and
// only MAX goes psychedelic: vivid saturation, a surreal hue drift, poster banding.
float t = u_amt * u_amt;
float luma = dot(styl, vec3(0.299, 0.587, 0.114));
styl = mix(vec3(luma), styl, 1.0 + 1.1 * t); // saturation: vivid toward max
styl = hueRotate(styl, 0.8 * t); // surreal hue drift (~45deg at max)
float levels = mix(255.0, 6.0, t); // posterize: color banding toward max
styl = floor(styl * levels + 0.5) / levels;
styl *= 1.0 + 0.06 * u_amt; // luminous lift
frag = vec4(clamp(styl, 0.0, 1.0), 1.0);
}`;
let gl = null, uAmt = null, uTexel = null;
function _shader(kind, src) {
const s = gl.createShader(kind);
gl.shaderSource(s, src); gl.compileShader(s);
if (!gl.getShaderParameter(s, gl.COMPILE_STATUS)) throw new Error(gl.getShaderInfoLog(s));
return s;
}
function initPaint() {
gl = paint.getContext("webgl2");
if (!gl) { paint.style.display = "none"; return false; } // CSS fallback grades #vid
const prog = gl.createProgram();
gl.attachShader(prog, _shader(gl.VERTEX_SHADER, PAINT_VS));
gl.attachShader(prog, _shader(gl.FRAGMENT_SHADER, PAINT_FS));
gl.bindAttribLocation(prog, 0, "p"); gl.linkProgram(prog);
if (!gl.getProgramParameter(prog, gl.LINK_STATUS)) throw new Error(gl.getProgramInfoLog(prog));
gl.useProgram(prog);
const buf = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, buf);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-1, -1, 3, -1, -1, 3]), gl.STATIC_DRAW);
gl.enableVertexAttribArray(0); gl.vertexAttribPointer(0, 2, gl.FLOAT, false, 0, 0);
const t = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, t);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
uAmt = gl.getUniformLocation(prog, "u_amt");
uTexel = gl.getUniformLocation(prog, "u_texel");
paintOK = true;
requestAnimationFrame(paintLoop);
return true;
}
function paintLoop() {
if (paintOK && vid.readyState >= 2 && vid.videoWidth) {
const w = vid.videoWidth, h = vid.videoHeight;
if (paint.width !== w || paint.height !== h) { paint.width = w; paint.height = h; }
gl.viewport(0, 0, w, h);
try { gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGB, gl.RGB, gl.UNSIGNED_BYTE, vid); } catch (_) {}
// No painterly restyle during a ring transition (busy) — show it raw.
gl.uniform1f(uAmt, busy ? 0 : dreamIntensity);
gl.uniform2f(uTexel, 1 / w, 1 / h);
gl.drawArrays(gl.TRIANGLES, 0, 3);
paint.style.filter = busy ? "none" : gradeFilter;
}
requestAnimationFrame(paintLoop);
}
const SVGNS = "http://www.w3.org/2000/svg";
@@ -153,6 +274,7 @@ function renderOverlay(level, intensity) {
// softer than the clinical reticles — the dream leaking into the machine's read.
function renderAffect(strength, intensity) {
const clip = activeClip();
if (!affectLayer) return; // tolerate a stale page missing the affect layer
affectLayer.innerHTML = "";
if (!clip || !clip.affect || strength <= 0) { affectLayer.style.opacity = "0"; return; }
affectLayer.style.opacity = String(intensity);
@@ -182,7 +304,7 @@ function controls() {
function calibration() {
return { mood_gain: +$("mood_gain").value, overlay_gain: +$("overlay_gain").value,
right_variant_map: [0, 1, 2, 3, 4] };
dream_gain: 1.0 };
}
let timer = null;
@@ -197,10 +319,17 @@ async function update() {
readout.textContent = JSON.stringify(data, null, 2);
if (!data.content.video) { black.classList.remove("hidden"); return; }
black.classList.add("hidden");
applyGrade(data.plan.grade.tone);
loadVariant(data.plan.restyle.variant);
renderOverlay(data.plan.overlay.level, data.plan.overlay.intensity);
renderAffect(data.plan.affect.strength, data.plan.affect.intensity);
try {
ensureClipMedia();
applyVideoLook(data.plan.grade.tone, data.plan.dream.intensity);
renderOverlay(data.plan.overlay.level, data.plan.overlay.intensity);
renderAffect(data.plan.affect.strength, data.plan.affect.intensity);
const b = document.getElementById("err-banner");
if (b) b.remove(); // render succeeded — clear any prior error
} catch (err) {
showError("render failed: " + (err && err.message ? err.message : err));
throw err;
}
}
function debounced() { clearTimeout(timer); timer = setTimeout(update, 80); }
@@ -216,6 +345,7 @@ function playTransition(file, blended) {
// Resolves on 'ended' (or a safety timeout that scales with playback rate).
return new Promise((resolve) => {
overlay.style.opacity = "0";
affectLayer.style.opacity = "0";
tint.style.opacity = "0";
vid.style.filter = "none";
vid.loop = false;
@@ -252,7 +382,7 @@ async function advance(delta) {
await playTransition(step.file, step.blended);
}
ringIndex = move.to_index;
currentVariant = -1; // force the target clip's variant to (re)load
currentClipId = null; // force the target scale's base media to (re)load
renderScaleReadout();
} finally {
busy = false;
@@ -272,7 +402,24 @@ function onWheel(e) {
}, 90);
}
// Dev live-reload: poll the asset version and reload when it changes, so an open
// tab never keeps running a stale renderer while we iterate (the readout updates
// from the live API and masks it otherwise). Reloads on my edits AND on a server
// restart. Dev-only convenience; harmless if the endpoint is absent.
function devLiveReload() {
let seen = null;
setInterval(async () => {
try {
const v = (await (await fetch("/dev/version", { cache: "no-store" })).json()).version;
if (seen === null) seen = v;
else if (v !== seen) location.reload();
} catch (_) { /* server restarting / endpoint absent — ignore */ }
}, 1000);
}
async function main() {
devLiveReload();
try { initPaint(); } catch (e) { paintOK = false; paint.style.display = "none"; showError("WebGL init: " + e.message); }
await loadData();
renderScaleReadout();
for (const id of ["content", "left", "right", "dark", "light", "mood_gain", "overlay_gain"]) {
simulator/static/index.html
+1
View File
@@ -12,6 +12,7 @@
<section class="stage" id="stage">
<div class="screen">
<video id="vid" loop muted playsinline></video>
<canvas id="paint"></canvas>
<div id="tint"></div>
<svg id="overlay" viewBox="0 0 100 100" preserveAspectRatio="none"></svg>
<svg id="affect" viewBox="0 0 100 100" preserveAspectRatio="none"></svg>
simulator/static/style.css
+6
View File
@@ -7,6 +7,12 @@ main { display: flex; gap: 1rem; padding: 1rem; flex-wrap: wrap; }
.screen { position: relative; width: 100%; aspect-ratio: 16 / 9; background: #000;
border-radius: 6px; overflow: hidden; }
#vid { width: 100%; height: 100%; object-fit: cover; transition: opacity 0.15s ease; }
/* Right-dream painterly canvas: a WebGL Kuwahara restyle of the LIVE video frames,
drawn over the base. Edge-preserving, so motion stays as crisp as the source —
the dream is the same footage, just stylized. The mood grade rides as a CSS
filter here. Hidden when WebGL is unavailable (CSS fallback shows #vid). */
#paint { position: absolute; inset: 0; width: 100%; height: 100%; object-fit: cover;
pointer-events: none; transition: filter 0.2s ease; }
#tint { position: absolute; inset: 0; pointer-events: none; opacity: 0;
background: #28425f; mix-blend-mode: multiply;
transition: opacity 0.2s ease; }