Compare commits
18 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| d24f829276 | |||
| 129bb23cb9 | |||
| a2ef792f0f | |||
| 53cb89b4c6 | |||
| 0201690ce0 | |||
| 0928988abc | |||
| a1c9c5c760 | |||
| ab2edf77bb | |||
| d19fe4e8fa | |||
| 9a83c0132d | |||
| 7fd9aabd0e | |||
| 7a32ff1acc | |||
| 5fae4f9005 | |||
| 7628a5321f | |||
| bcd4d32b19 | |||
| c4452396a1 | |||
| 70fd367c70 | |||
| 12408e505e |
+48
-37
@@ -296,24 +296,17 @@ exists. (The earlier selection-era "curator's X-ray" view was retired when the
|
||||
piece moved from *selecting* clips to *altering* them.)
|
||||
|
||||
**One-time setup — populate the sample footage** (look-tuning only; not shipped
|
||||
content):
|
||||
content). The ring uses real strict-PD footage in a **rotating pool** per scale
|
||||
(`docs/content-candidate-pool.md`); regenerate the manifest + new transition
|
||||
placeholders with:
|
||||
|
||||
python simulator/setup_sample_media.py # stage the forest neutral base
|
||||
python -m simulator.bake_right_variants --scale forest # real Right variants (SD on MPS, ~11 min)
|
||||
python simulator/setup_scales_media.py # the cosmos + abyss scales + ring transitions
|
||||
python simulator/build_pool_manifest.py --media # write the pool manifest + coast-edge transitions
|
||||
|
||||
`setup_sample_media.py` stages the forest neutral base from the session-0008 POC
|
||||
artifacts (`~/hef-poc/out/neutral.mp4`). `bake_right_variants.py` then produces
|
||||
the **real** flow-stabilized painterly Right variants (strengths 1–4) for the
|
||||
forest scale — SD img2img keyframes + Farneback optical-flow tweens on Apple MPS,
|
||||
the temporally-calm POC algorithm, productionized (scales design §1/§4). The
|
||||
keyframe-strength ramp per Right level is by-eye tunable in
|
||||
`simulator/bake_right_variants.py`. `setup_scales_media.py` makes the scale
|
||||
**ring** demonstrable: cheap synthetic placeholder bases for the two true-PD
|
||||
scales (`cosmos` = NASA/Hubble, `abyss` = NOAA Ocean Exploration) plus the
|
||||
per-edge zoom/warp transition clips — these scales carry a raw base only (no real
|
||||
Right variants yet; the baker can extend to them once their real footage is
|
||||
sourced). The `.mp4` binaries are gitignored.
|
||||
`build_pool_manifest.py` holds the hand-authored label/affect content and emits
|
||||
`simulator/sample_media/manifest.json` (the 19-clip pool baseline). The pool clips
|
||||
themselves are sourced via the content pipeline (`tools/pipeline/run.py
|
||||
process_clip`); the `.mp4` binaries are gitignored. (`setup_scales_media.py` is the
|
||||
older one-base-per-scale generator, now superseded.)
|
||||
|
||||
**Run it (Docker):**
|
||||
|
||||
@@ -331,27 +324,45 @@ then open http://localhost:8000.
|
||||
- **Content dial** — picks audio/video channel; "off" and audio-only positions go
|
||||
to black walls.
|
||||
- **Scale ring (endless encoder)** — `⊖ out` / `in ⊕` (or scroll the stage) walk a
|
||||
*closed ring* of neutral "scales of nature" clips — cosmos → forest → abyss and
|
||||
back around (diving past the smallest **wraps** to the largest). It is *relative*
|
||||
(an endless encoder), distinct from the absolute 0–4 knobs: each step plays a
|
||||
short placeholder zoom/warp **transition**, then settles on the next scale, with
|
||||
the current knob alteration still applied. The current scale is named beside the
|
||||
buttons (`name (i/N)`). A **fast spin** (scroll several detents at once, ≥3)
|
||||
collapses to a single quick **blended pass** straight to the destination scale
|
||||
instead of grinding through every transition (scales design §3); slow single
|
||||
steps still chain one full transition per scale crossed.
|
||||
- **Four experience knobs (0–4):**
|
||||
- **Dark / Light** — a live runtime color grade (cool/dark ↔ warm/bright; equal
|
||||
or zero = the raw footage).
|
||||
- **Right (dreamlike)** — selects a discrete pre-baked, flow-stabilized restyle
|
||||
variant and crossfades to it (strength 0 = raw base).
|
||||
- **Left (analytical)** — a live overlay: labelled boxes from the clip's authored
|
||||
annotation track, with more annotations appearing at higher levels. Text is
|
||||
shaped live (the simulator analogue of the Pi's Pango/HarfBuzz path).
|
||||
- **Calibration sliders** — adjust the grade/overlay gain curves live; once a look
|
||||
is liked, bake the values into `DEFAULT_CALIBRATION` in `player/alteration.py`.
|
||||
*closed ring* of neutral "scales of nature" — **cosmos → orbit → coast → reef →
|
||||
abyss** and back around (diving past the smallest **wraps** to the largest). Each
|
||||
scale is a **rotating pool** of vetted clips: landing on a scale plays a random
|
||||
pool member, so the ring feels fresh each pass. It is *relative* (an endless
|
||||
encoder), distinct from the absolute 0–4 knobs: each step plays a short
|
||||
placeholder zoom/warp **transition**, then settles, with the current alteration
|
||||
still applied. The current scale + chosen member is named beside the buttons
|
||||
(`scale · member (i/N · pool N)`). A **fast spin** (≥3 detents at once) collapses
|
||||
to one quick **blended pass** instead of grinding through every transition.
|
||||
- **Three experience knobs:**
|
||||
- **Mood (−4 dark .. 0 .. +4 light)** — a live runtime color grade (cool/dark ↔
|
||||
warm/bright; 0 = the raw footage).
|
||||
- **Right (dreamlike, 0–4)** — a **deterministic real-time painterly dream** (a
|
||||
WebGL Kuwahara restyle of the live frames; holds still across the loop, goes
|
||||
trippy at max). It also drives **progressive emotion tiers**: when both knobs
|
||||
are up, the affect words escalate from basic to compound as Right rises.
|
||||
- **Left (analytical, 0–4)** — a live HUD of labelled boxes from the clip's
|
||||
authored annotation track. Labels use **progressive detail tiers**: low Left
|
||||
shows fewer, more general labels (high-salience objects only); raising Left
|
||||
brings in more objects *and* escalates each label general → specific →
|
||||
scientific → +fact. Time-windowed tracked labels appear only while their
|
||||
subject is on screen and follow it.
|
||||
- **RenderPlan readout** — always shows the exact numbers the engine produced (the
|
||||
project's honesty "X-ray," now over the alteration model).
|
||||
project's honesty "X-ray," over the alteration model).
|
||||
|
||||
The base clips, Right variants, Left annotation track, and string tables come from
|
||||
The base clips, the Left annotation tracks (with tiers + appear/disappear
|
||||
windows), affect anchors, and per-tier string tables all come from
|
||||
`simulator/sample_media/manifest.json`.
|
||||
|
||||
### Authoring labels — the author mode
|
||||
|
||||
Open **http://localhost:8000/author.html** to author a clip's labels by eye
|
||||
(content-pipeline §11.5). Pick a pool clip, **scrub** to where a subject is on
|
||||
screen, **drag a box** over it, set the label key + salience + the four detail
|
||||
tiers, then **Run tracker** to propagate the box into a keyframed track (classical
|
||||
OpenCV optical flow) with an appear/disappear window — or **Add as static box**
|
||||
for a fixed label. Shift-click the stage to place an **affect anchor** and type its
|
||||
emotion tiers. **Save to manifest** writes the entry back via the pipeline's
|
||||
idempotent upsert (keeping the clip's media + provenance, replacing only the
|
||||
authored content). The tracker only propagates **box geometry** — every label
|
||||
string, scientific name, and fact is **hand-authored**, because a generic detector
|
||||
can't produce "*Gymnothorax*" or a lifespan.
|
||||
|
||||
@@ -27,16 +27,27 @@ stays non‑commercial**.
|
||||
|
||||
| id | clip | source | window (s) | license / credit |
|
||||
|----|------|--------|-----------|------------------|
|
||||
| `cosmos` | Orion Nebula flythrough | NASA/JPL‑Caltech — https://images.nasa.gov/details/JPL-20221122-SOLSYSf-0001-Orion%20Dust%20and%20Death | ~30–54 | PD (17 U.S.C. §105) |
|
||||
| `cosmos` | Cosmic Cliffs — Carina Nebula flythrough | NASA SVS 31348 — https://svs.gsfc.nasa.gov/31348/ (`Clifs-3d-STScI.mp4`, *Exploring the Cosmic Cliffs in 3D*, Webb NIRCam, **4K**) | 10–34 | CC‑BY‑class — credit **NASA, ESA, CSA, STScI** |
|
||||
|
||||
> ℹ️ **Cosmos primary swapped to the Webb "Cosmic Cliffs" flythrough (2026-06-25, session 0018).**
|
||||
> History: the original primary was NASA/JPL's *"Orion: Dust and Death"* (captions
|
||||
> burned in → cropping lost framing); it was then swapped to STScI's text-free
|
||||
> *Flight Through the Orion Nebula* (SVS 30957, visible light). The operator found
|
||||
> the Orion clip merely good and asked for a more dramatic nebula, so the primary is
|
||||
> now STScI's **Exploring the Cosmic Cliffs in 3D** (Carina Nebula, Webb NIRCam,
|
||||
> **SVS 31348**) — a full-frame, text-free 3D flythrough sourced from its **4K**
|
||||
> master (`Clifs-3d-STScI.mp4`, 3840×2160) → supersampled 1080p base + 4K master.
|
||||
> Trim 10–34s clears the opening title card (gone by ~8s) and the end-credit card
|
||||
> (~102s). The earlier `decaption_cosmos` crop helper remains removed.
|
||||
| `cosmos_galaxies` | Flying Through Galaxies | NASA SVS — https://svs.gsfc.nasa.gov/vis/a010000/a014900/a014950/14950_Galaxies_FlyThrough_4k.mp4 | 8–32 | PD |
|
||||
| `cosmos_hudf` | Hubble Ultra Deep Field zoom | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030600/a030687/hudf-b-1920x1080p30.mov | 20–44 | CC‑BY‑class — credit **NASA, ESA, STScI** |
|
||||
| `cosmos_xdf` | eXtreme Deep Field flythrough | NASA SVS — https://svs.gsfc.nasa.gov/vis/a030000/a030600/a030681/hxdf_fly-b-1920x1080p30.mov | 2–26 | CC‑BY‑class — credit **NASA, ESA, STScI** |
|
||||
|
||||
> ⚠️ **DISCREPANCY to fix next session.** Operator selected cosmos **#1, #3
|
||||
> (Orion), #4, #5**. During processing, **#6 Galaxy Traverse was run by mistake
|
||||
> instead of #3 Orion**. The Orion clip (#3) is text‑free and already lives at
|
||||
> `simulator/sample_media/cosmos/base.mp4` (the prior real base), so the pool above
|
||||
> is correct. **`cosmos_traverse` is an UNSELECTED extra on disk — drop it** (or
|
||||
> instead of #3 Orion**. The cosmos primary now lives at
|
||||
> `simulator/sample_media/cosmos/base.mp4` as the Webb Cosmic Cliffs flythrough
|
||||
> (SVS 31348 — see the cosmos-primary note above). **`cosmos_traverse` is an UNSELECTED extra on disk — drop it** (or
|
||||
> confirm with operator). The current ring scale id is `cosmos`; when the pool is
|
||||
> wired, `cosmos` becomes one pool member.
|
||||
|
||||
@@ -108,5 +119,5 @@ Removed from local media (all were gitignored, so repo-invisible):
|
||||
- `forest` — the old Yosemite base, superseded by the `coast` pool. **dropped.**
|
||||
- `reef` — the original placeholder/early reef base, superseded by `reef_*`. **dropped.**
|
||||
- `orbit` / `abyss` — the Increment‑1 single real bases (NASA ISS Exp65 / ctenophore),
|
||||
not in the new pools. **dropped** (the cosmos Orion base stays as the cosmos pool primary).
|
||||
not in the new pools. **dropped** (the cosmos base — now the Webb Cosmic Cliffs flythrough, SVS 31348 — stays as the cosmos pool primary).
|
||||
- `review.html` gallery — now `.gitignore`d (local‑only, re‑buildable).
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# HEF — Content Pipeline (source → process → label → manifest)
|
||||
|
||||
**Date:** 2026-06-24
|
||||
**Status:** Graduated — operator-approved (session 0014); **Increment 1 built & merged** (PR #13); real PD sourcing + rotating pool curated (PR #14/#15). **Increment 2 schema pinned + building** (session 0016 — see §11).
|
||||
**Status:** Graduated — operator-approved (session 0014); **Increment 1 built & merged** (PR #13); real PD sourcing + rotating pool curated (PR #14/#15). **Increment 2 BUILT** (session 0016 — pool model + tiers + windows in PR #16; `track.py` + author mode in PR B; see §11).
|
||||
**Repo:** `human-experience-filter-art`
|
||||
**Anchor:** `docs/ROADMAP.md` sub-project 3 (Player Runtime) content needs; the
|
||||
spiritual successor to sub-project 2 (Ingest & Tagging tools), retargeted from the
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# Simulator Dev Mode — design
|
||||
|
||||
> 2026-06-25 · session 0017 · planning-and-executing
|
||||
> Anchor: leaf task (simulator dev-tooling — R2b, no upstream design needed)
|
||||
|
||||
## Problem
|
||||
|
||||
While previewing the experience, the operator can't choose *which* clip in an
|
||||
altitude's rotating pool is shown — the server picks a random pool member on each
|
||||
landing (content-pipeline §11.1). For by-eye review they want to step through every
|
||||
clip in a pool on demand, and to see the engine/clip data behind what's on screen.
|
||||
The RenderPlan JSON readout is also always visible, cluttering what should read as
|
||||
an audience-facing preview.
|
||||
|
||||
## Goal
|
||||
|
||||
A single **Dev Mode** toggle at the bottom of the control panel. Off by default
|
||||
(persisted in `localStorage`). When on, a panel appears directly below it with all
|
||||
dev controls and data; when off, the normal view is just the experience + the three
|
||||
control fieldsets.
|
||||
|
||||
## Scope
|
||||
|
||||
Frontend only — `simulator/static/{index.html,app.js,style.css}`. No API changes:
|
||||
every datum Dev Mode shows is already available client-side (`/api/clips`,
|
||||
`/api/ring`, the `/api/alteration` response, and the in-memory preload cache). The
|
||||
separate `/author.html` remains the place to *edit* labels; Dev Mode is
|
||||
inspect-and-select only.
|
||||
|
||||
## Dev panel contents (top → bottom)
|
||||
|
||||
1. **Pool picker** — a `<select>` of the current altitude's pool members
|
||||
(`clip_id · title`), its value reflecting the clip currently playing. Choosing a
|
||||
member sets `activeClipId` to it and reloads the base media, overriding the random
|
||||
landing pick. A `🎲 Re-roll random` button re-runs the server-canonical random
|
||||
pick for the current scale. The dropdown is rebuilt on every scale landing, so it
|
||||
always lists the current pool; single-clip scales (cosmos/orbit) show one entry.
|
||||
2. **Active clip** — `id`, `title`, `source`, `license`, `base_file`.
|
||||
3. **Annotations & affect** — each annotation key with salience (or legacy
|
||||
`min_level`), time window (`appear`–`disappear`), and tier count; each affect key
|
||||
with `min_level` and tier count. Read-only.
|
||||
4. **RenderPlan** — the relocated `#readout` (the `/api/alteration` JSON).
|
||||
5. **Cache & playback** — preload status (`N/total cached`), whether the active
|
||||
clip is served **from memory** (a preloaded blob) vs the network, and live video
|
||||
resolution / duration / loop position.
|
||||
|
||||
## Behavior
|
||||
|
||||
- **Pool override is per-landing.** Changing altitude re-rolls randomly as today;
|
||||
the operator then picks again from the new scale's pool. The pick is not persisted
|
||||
across landings (YAGNI — "select any video while on an altitude").
|
||||
- **Refresh points.** Pool picker + clip metadata + annotations rebuild whenever the
|
||||
scale lands (hooked into `renderScaleReadout()`) and after a manual pick/re-roll.
|
||||
The RenderPlan readout updates on each `update()` (knob change) as today. The
|
||||
cache/playback stats refresh on a light interval (~500 ms) while Dev Mode is on,
|
||||
and the cache count also ticks up as the preloader fills.
|
||||
- **`pickRandomMember()`** is factored out of `landScale()` so the initial landing,
|
||||
ring advances, and the Re-roll button all share the one server-canonical pick.
|
||||
|
||||
## Testing
|
||||
|
||||
No JS unit harness in the repo and no API change, so verification is: `node --check`
|
||||
on `app.js`, the Python suite stays green (unaffected), and by-eye review by the
|
||||
operator (no headless browser yet — §9 E2E policy noted, machinery pending).
|
||||
@@ -0,0 +1,117 @@
|
||||
# Session 0016.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-24T17-48 (PST)
|
||||
> Type: planning-and-executing
|
||||
> Posture: yolo
|
||||
> Claude-Session: 0fde1d36-7c64-414f-8f49-8669b5dd2263
|
||||
> Status: **FINALIZED**
|
||||
|
||||
## Launch prompt
|
||||
|
||||
`/goal` Content pipeline Increment 2 (expanded) — rotating clip pools + time-windowed
|
||||
TRACKED labels + progressive left-brain detail + progressive right-brain emotions.
|
||||
Short design pass first (schema changes), then build.
|
||||
|
||||
> Anchor: design `docs/superpowers/specs/2026-06-24-content-pipeline-design.md`
|
||||
> (graduated, R2a-eligible). Repo is roadmap-anchored, no Gitea issue tracker (R1
|
||||
> deviation, as all prior sessions). Increment 2 of that design.
|
||||
|
||||
## Pre-state
|
||||
|
||||
`main` at `b28f930` (PR #15 — operator-curated rotating-pool doc merged). All 19
|
||||
vetted pool clips on disk (gitignored) plus the leftovers to drop. The ring was the
|
||||
Increment-1 one-base-per-scale model (cosmos/orbit/forest/reef/abyss), flat
|
||||
`min_level` label gating, static or simple-track boxes, scene-level affect gated by
|
||||
`min(left,right)`. `tools/pipeline/` had stages 1–4 + 6 (no `track.py`). 246 tests.
|
||||
|
||||
## Arc
|
||||
|
||||
1. **Session gate + init.** Classified the `/goal` as planning-and-executing,
|
||||
claimed session **0016** (peeked an ENDED-UNFINALIZED 0015 — its PR #15 work was
|
||||
already merged, not live — and proceeded; no worktree needed, solo session).
|
||||
Verified clean `main`, read the pool doc + spec + sub-project-3 memory + the live
|
||||
code (ring, clips, app, app.js, alteration, pipeline).
|
||||
2. **Design pass (operator asked for it — schema changes).** Appended **§11** to the
|
||||
content-pipeline spec pinning the concrete shapes: rotating pool, time-windowed
|
||||
tracks, progressive Left detail tiers (salience + tiered strings), progressive
|
||||
Right emotion tiers, and the `track.py` + author-mode tooling. Key call: tiers
|
||||
need **no engine change** — the client reads `overlay.level` (Left) and
|
||||
`dream.strength` (Right) already in the `RenderPlan`.
|
||||
3. **PR #16 — the experience.** `Scale.pool` + pure `pick_clip_id` (injected
|
||||
randomness); server-side random pick at the API boundary (delta=0 = initial pick).
|
||||
`forest`→`coast` rename + ring reorder + orphan-media cleanup. Time-windowed
|
||||
tracked labels (`appear`/`disappear`, seam-wrapping). Salience-gated +
|
||||
tier-escalating Left labels; tier-escalating Right emotions; tiered-or-static
|
||||
strings (back-compat). New `build_pool_manifest.py` emits the 19-clip pool
|
||||
manifest (abyss+reef authored as tracked/windowed showcase). Merged via Gitea API.
|
||||
4. **PR #17 — the tooling.** `tools/pipeline/track.py`: classical LK optical-flow box
|
||||
tracker (this opencv build has no CSRT) → sparse keyframed track + window; optional
|
||||
lazy ML path. Simulator AUTHOR MODE (`/author.html` + `/api/author/{track,clip}`):
|
||||
scrub, drag a seed box, run tracker, author tiers/affect, save to manifest
|
||||
(idempotent upsert, keeps provenance). Repointed the pipeline integration test off
|
||||
the retired forest base; USER_GUIDE sim section brought current. Merged.
|
||||
|
||||
## Cut state
|
||||
|
||||
`main` at `c445239` (PR #17 merge). **267 passed / 2 skipped.** Clean tree, fully
|
||||
pushed, no open PRs. Both feature branches deleted. All 5 goal items delivered.
|
||||
|
||||
Verification was API/logic-level + JS syntax + live-server probes (5 scales with
|
||||
pools, rotating picks, new edges, 19 clips served, tracker returns keyframes on real
|
||||
abyss_wow footage, committed manifest untouched by the read-only probe). **The
|
||||
VISUAL look was not screenshot-verified — no Chrome on this box** — so operator
|
||||
by-eye review is the next step (matches the established deferred-review pattern).
|
||||
|
||||
## Follow-up (post-finalize, same conversation) — PR #18
|
||||
|
||||
Operator asked to replace the scale-ring `⊖ out / in ⊕` buttons with a **circular
|
||||
"Altitude" knob** that turns all the way around (including abyss→cosmos). Built a
|
||||
turnable SVG dial in `simulator/static/app.js` (`buildDial` / `renderDial` / drag +
|
||||
wheel handlers) + `style.css`: cosmos (highest) at top, clockwise descends
|
||||
cosmos→orbit→coast→reef→abyss and **wraps** back up. Dragging accumulates rotation
|
||||
and commits whole detents on release (a big spin → one fast blended pass, reusing
|
||||
the proven wheel/detent path); scroll the knob to step; tap a label to jump the
|
||||
shortest way around. **Pure frontend** — the endless ring + wrap already existed
|
||||
server-side, so no engine/API change. JS syntax-checked; 23 sim-API tests green;
|
||||
live probe confirmed the dial replaced the buttons and the abyss→cosmos wrap is
|
||||
intact. Merged as **PR #18** (`main` at `7a32ff1`). Visual look still operator-by-eye
|
||||
(no Chrome on this box).
|
||||
|
||||
## Deployment pipeline (§9)
|
||||
|
||||
No stage run: the HEF simulator is a **local dev tool** (simulator-first directive),
|
||||
not a deployed app — it has no PPE/prod infra. Nothing shippable through the §9
|
||||
pipeline this session; the session output is the merged code on `main`.
|
||||
|
||||
## Plan archival
|
||||
|
||||
No separate `superpowers:writing-plans` artifact was produced — the design pass was a
|
||||
spec §11 edit (merged in PR #16) and execution was direct TDD tracked by the
|
||||
transcript `## Plan` block + the task list. Nothing to `submit-plan.sh`; this app has
|
||||
no content repo anyway (roadmap-anchored).
|
||||
|
||||
## Deferred decisions
|
||||
|
||||
- **Adopting/finalizing the ENDED-UNFINALIZED session 0015 placeholder** was skipped
|
||||
— its PR #15 work is already on `main` and its outcomes are fully recorded in
|
||||
memory; finalizing a prior dead session was judged an unnecessary detour. Low
|
||||
confidence; flagging for the operator.
|
||||
- **By-eye visual review deferred** to the operator (no Chrome here) — consistent with
|
||||
the prior-session pattern, but the look of the tiers/windows/pool rotation AND the
|
||||
new Altitude knob is unconfirmed visually.
|
||||
- **Light-vs-rich label authoring:** only abyss + reef + the scale primaries are
|
||||
richly authored; the other pool members carry lighter labels. Finishing them is the
|
||||
author-mode follow-up (next goal).
|
||||
|
||||
## Next-session prompt
|
||||
|
||||
`/goal` Operator by-eye review of Increment 2 + the Altitude knob in the sim
|
||||
(`make sim-local`, :8000): turn the Altitude knob (drag/scroll, wraps abyss→cosmos;
|
||||
pools rotate per landing), sweep Left (progressive label tiers
|
||||
general→scientific+fact), sweep Right with Left up (progressive emotions
|
||||
basic→compound), watch abyss/reef time-windowed tracked labels enter/leave + track,
|
||||
and exercise `/author.html`. Then author real labels for the remaining pool members
|
||||
via author mode. After the content is liked, the next build frontier is the deferred
|
||||
i2v ring transitions (generative-video model + adjacent real bases) and/or the Pi
|
||||
renderer + serial/firmware (per [[simulator-first-before-hardware]]), per ROADMAP.md.
|
||||
+5
-5
@@ -1,15 +1,15 @@
|
||||
# Session 0016.0 — Transcript
|
||||
# Session 0017.0 — Transcript
|
||||
|
||||
> App: human-experience-filter-art
|
||||
> Start: 2026-06-24T17-48 (PST)
|
||||
> Start: 2026-06-25T07-41 (PST)
|
||||
> Type: planning-and-executing
|
||||
> Posture: yolo
|
||||
> Claude-Session: 0fde1d36-7c64-414f-8f49-8669b5dd2263
|
||||
> Claude-Session: bcb069f2-0fba-4a66-ae8e-f40b7917a48c
|
||||
> Status: **PLACEHOLDER — claimed at session start; finalized at session end.**
|
||||
>
|
||||
> This file reserves session ID 0016 for human-experience-filter-art. The driver replaces this
|
||||
> This file reserves session ID 0017 for human-experience-filter-art. The driver replaces this
|
||||
> body with the full transcript and renames the file to its final
|
||||
> SESSION-0016.0-TRANSCRIPT-2026-06-24T17-48--<end>.md form at session end.
|
||||
> SESSION-0017.0-TRANSCRIPT-2026-06-25T07-41--<end>.md form at session end.
|
||||
|
||||
## Launch prompt
|
||||
|
||||
@@ -46,5 +46,8 @@
|
||||
},
|
||||
"0016": {
|
||||
"title": ""
|
||||
},
|
||||
"0017": {
|
||||
"title": ""
|
||||
}
|
||||
}
|
||||
|
||||
+88
-5
@@ -8,6 +8,7 @@ endpoints (/api/select, /api/catalog/meta) and the X-ray are retired.
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import time
|
||||
@@ -84,6 +85,29 @@ class RingAdvanceRequest(BaseModel):
|
||||
delta: int
|
||||
|
||||
|
||||
class AuthorTrackRequest(BaseModel):
|
||||
"""Author mode: propagate a hand-seeded box on a clip into a keyframed track
|
||||
(content-pipeline §11.5). `seed_box` + `seed_t` are normalized."""
|
||||
|
||||
clip_id: str
|
||||
key: str
|
||||
seed_box: list[float] = Field(min_length=4, max_length=4)
|
||||
seed_t: float = Field(default=0.0, ge=0.0, le=1.0)
|
||||
salience: int = Field(default=4, ge=1, le=4)
|
||||
n_keyframes: int = Field(default=5, ge=2, le=20)
|
||||
max_frames: Optional[int] = None
|
||||
|
||||
|
||||
class AuthorClipRequest(BaseModel):
|
||||
"""Author mode: persist the authored labels/affect/strings for an existing
|
||||
pool clip into the manifest (geometry from the tracker, semantics by hand)."""
|
||||
|
||||
clip_id: str
|
||||
annotations: list = []
|
||||
affect: list = []
|
||||
strings: dict = {}
|
||||
|
||||
|
||||
def _load_clips(manifest_path: Optional[Path]):
|
||||
path = Path(manifest_path) if manifest_path else DEFAULT_MANIFEST
|
||||
if path.exists():
|
||||
@@ -100,6 +124,7 @@ def _load_ring(manifest_path: Optional[Path]):
|
||||
|
||||
def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
|
||||
app = FastAPI(title="HEF Alteration Simulator")
|
||||
app.state.manifest_path = Path(manifest_path) if manifest_path else DEFAULT_MANIFEST
|
||||
app.state.clips = _load_clips(manifest_path)
|
||||
app.state.ring = _load_ring(manifest_path)
|
||||
|
||||
@@ -158,13 +183,71 @@ def create_app(manifest_path: Optional[Path] = None) -> FastAPI:
|
||||
chosen = pick_clip_id(landed, random.random())
|
||||
return ring_move_to_dict(move, app.state.ring, chosen)
|
||||
|
||||
@app.post("/api/author/track")
|
||||
def api_author_track(req: AuthorTrackRequest):
|
||||
# Author mode (§11.5): run the classical optical-flow tracker on a clip's
|
||||
# base footage, returning the keyframed track geometry for the author to
|
||||
# accept/correct. Semantics (key/strings/tiers) stay the author's.
|
||||
clip = next((c for c in app.state.clips if c.id == req.clip_id), None)
|
||||
if clip is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown clip {req.clip_id!r}")
|
||||
base = MEDIA_DIR / clip.base_file
|
||||
if not base.exists():
|
||||
raise HTTPException(status_code=404, detail=f"media absent: {clip.base_file}")
|
||||
try:
|
||||
import cv2
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="opencv-python not installed")
|
||||
from tools.pipeline.track import track_seed
|
||||
|
||||
cap = cv2.VideoCapture(str(base))
|
||||
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
|
||||
cap.release()
|
||||
seed_frame = max(0, min(total - 1, round(req.seed_t * total))) if total else 0
|
||||
return track_seed(
|
||||
base, req.key, tuple(req.seed_box), seed_frame=seed_frame,
|
||||
salience=req.salience, n_keyframes=req.n_keyframes, max_frames=req.max_frames,
|
||||
)
|
||||
|
||||
@app.post("/api/author/clip")
|
||||
def api_author_clip(req: AuthorClipRequest):
|
||||
# Author mode (§11.5 / stage 6): persist authored labels/affect/strings for
|
||||
# an EXISTING pool clip — idempotent upsert that keeps the clip's media +
|
||||
# provenance and replaces only the authored content. Reloads in place so
|
||||
# the preview reflects the edit without a restart.
|
||||
from tools.pipeline.manifest import upsert_clip
|
||||
|
||||
path = app.state.manifest_path
|
||||
data = json.loads(path.read_text())
|
||||
existing = next((c for c in data.get("clips", []) if c["id"] == req.clip_id), None)
|
||||
if existing is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown clip {req.clip_id!r}")
|
||||
merged = dict(existing)
|
||||
merged["annotations"] = req.annotations
|
||||
merged["affect"] = req.affect
|
||||
if req.strings:
|
||||
merged["strings"] = req.strings
|
||||
data = upsert_clip(data, merged)
|
||||
path.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n")
|
||||
app.state.clips = load_manifest(path)
|
||||
app.state.ring = load_ring(path)
|
||||
return {"ok": True, "clip": merged}
|
||||
|
||||
@app.middleware("http")
|
||||
async def _no_cache(request, call_next):
|
||||
# Dev preview server: never let a browser serve a stale app.js/style.css.
|
||||
# The whole point is by-eye iteration, so a plain refresh must always get
|
||||
# the latest assets (the heuristic cache otherwise hides JS/CSS edits).
|
||||
async def _cache_policy(request, call_next):
|
||||
# Dev preview server: never let a browser serve a stale app.js/style.css —
|
||||
# by-eye iteration needs a plain refresh to always get the latest assets.
|
||||
# Media uses `no-cache` (store, but REVALIDATE every load): instant altitude
|
||||
# swaps already come from the in-memory blob preload (static/app.js), so the
|
||||
# HTTP cache only matters on (re)load — where a conditional request returns a
|
||||
# cheap 304 when unchanged, yet picks up a re-sourced/re-cropped clip
|
||||
# immediately. `immutable` was wrong here: it pinned stale footage for a year
|
||||
# so edited clips never showed without a manual cache clear.
|
||||
response = await call_next(request)
|
||||
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
|
||||
if request.url.path.startswith("/media/"):
|
||||
response.headers["Cache-Control"] = "no-cache"
|
||||
else:
|
||||
response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
|
||||
return response
|
||||
|
||||
if MEDIA_DIR.exists():
|
||||
|
||||
@@ -51,12 +51,14 @@ RING_ORDER = ["cosmos", "orbit", "coast", "reef", "abyss"]
|
||||
PD = "public-domain (US Gov, 17 U.S.C. §105)"
|
||||
PD_NPS = "public-domain (NPS, no © — 17 U.S.C. §105)"
|
||||
CCBY_STSCI = "CC-BY-class — credit NASA, ESA, STScI"
|
||||
CCBY_WEBB = "CC-BY-class — credit NASA, ESA, CSA, STScI"
|
||||
|
||||
# --- Per-clip provenance: id -> (title, license, source) ---
|
||||
META: dict[str, tuple[str, str, str]] = {
|
||||
# cosmos
|
||||
"cosmos": ("Orion Nebula flythrough (NASA/JPL-Caltech)", PD,
|
||||
"NASA/JPL-Caltech — images.nasa.gov JPL-20221122-SOLSYSf-0001 (Orion); trim ~30–54s, crossfade-loop"),
|
||||
"cosmos": ("Cosmic Cliffs — Carina Nebula flythrough (NASA/ESA/CSA/STScI)", CCBY_WEBB,
|
||||
"NASA SVS 31348 Clifs-3d-STScI (Exploring the Cosmic Cliffs in 3D, Webb NIRCam); "
|
||||
"4K source, trim 10–34s, crossfade-loop. Text-free — replaces the Orion SVS 30957 flythrough."),
|
||||
"cosmos_galaxies": ("Flying Through Galaxies (NASA SVS)", PD,
|
||||
"NASA SVS a014950 14950_Galaxies_FlyThrough_4k; trim 8–32s, crossfade-loop"),
|
||||
"cosmos_hudf": ("Hubble Ultra Deep Field zoom (NASA/ESA/STScI)", CCBY_STSCI,
|
||||
@@ -161,9 +163,9 @@ def measure(key, value, box, min_level):
|
||||
LABELS: dict[str, list[dict]] = {
|
||||
# ---------- cosmos ----------
|
||||
"cosmos": [
|
||||
static_label("detected.nebula", 4, ["cloud", "nebula", "emission nebula", "emission nebula · ionized H II region"], [0.32, 0.28, 0.34, 0.36]),
|
||||
static_label("detected.star", 2, ["star", "young star", "protostar", "protostar · <1 Myr old"], [0.60, 0.30, 0.10, 0.12]),
|
||||
measure("measure.redshift", "z ≈ 0.0", [0.36, 0.70, 0.18, 0.08], 4),
|
||||
static_label("detected.nebula", 4, ["cloud", "nebula", "emission nebula", "emission nebula · the Carina star-forming region"], [0.20, 0.42, 0.5, 0.42]),
|
||||
static_label("detected.star", 2, ["star", "young star", "protostar", "protostar · a newborn star, <1 Myr old"], [0.54, 0.12, 0.12, 0.14]),
|
||||
measure("measure.distance", "≈7,500 ly", [0.06, 0.06, 0.2, 0.08], 3),
|
||||
],
|
||||
"cosmos_galaxies": [
|
||||
static_label("detected.galaxy", 4, ["galaxy", "spiral galaxy", "barred spiral", "barred spiral · ~10¹¹ stars"], [0.34, 0.30, 0.30, 0.34]),
|
||||
@@ -355,7 +357,7 @@ def build_manifest() -> dict:
|
||||
return {"clips": clips, "ring": {"scales": scales, "transitions": transitions}}
|
||||
|
||||
|
||||
# --- Optional: generate the NEW transition placeholders (orbit-coast, coast-reef) ---
|
||||
# --- Generate the per-edge transition clips (zoom morph + reversed companion) ---
|
||||
|
||||
def _ffmpeg() -> str:
|
||||
if shutil.which("ffmpeg"):
|
||||
@@ -365,8 +367,10 @@ def _ffmpeg() -> str:
|
||||
|
||||
|
||||
def _make_transition(ff: str, scale_a: str, scale_b: str) -> Path:
|
||||
"""A placeholder zoom/warp morph between two scales, from their PRIMARY pool
|
||||
members' bases (mirrors setup_scales_media.py but keyed by scale->primary)."""
|
||||
"""A zoom/warp morph between two scales, from their PRIMARY pool members'
|
||||
bases (mirrors setup_scales_media.py but keyed by scale->primary). This is the
|
||||
recipe behind the well-liked orbit-coast edge; generate_media() now applies it
|
||||
uniformly to every edge so none carry stale footage from an earlier ring."""
|
||||
a = MEDIA / POOLS[scale_a][0] / "base.mp4"
|
||||
b = MEDIA / POOLS[scale_b][0] / "base.mp4"
|
||||
out = MEDIA / "transitions" / f"{scale_a}-{scale_b}.mp4"
|
||||
@@ -381,13 +385,29 @@ def _make_transition(ff: str, scale_a: str, scale_b: str) -> Path:
|
||||
return out
|
||||
|
||||
|
||||
def _make_reverse(ff: str, forward: Path) -> Path:
|
||||
"""The zoom-OUT companion of a forward (zoom-in) edge: the same clip played
|
||||
backward, written alongside it as `<edge>.rev.mp4`. An ascending ring move
|
||||
crosses its edge `reversed`; the renderer then plays this file, so zooming out
|
||||
recedes from the current scale back to the higher one instead of zooming in."""
|
||||
out = forward.with_suffix(".rev.mp4")
|
||||
subprocess.run([
|
||||
ff, "-y", "-i", str(forward), "-vf", "reverse", "-an", str(out),
|
||||
], check=True, capture_output=True)
|
||||
return out
|
||||
|
||||
|
||||
def generate_media() -> None:
|
||||
"""Generate the new ring edges introduced by the coast scale; the cosmos-orbit,
|
||||
reef-abyss and abyss-cosmos edges already exist from the prior 5-scale ring."""
|
||||
"""(Re)build EVERY ring-edge transition from the current real primary bases —
|
||||
the orbit-coast recipe applied uniformly, overwriting any stale clips left from
|
||||
an earlier ring — plus a reversed companion per edge for zoom-out moves."""
|
||||
ff = _ffmpeg()
|
||||
for a, b in [("orbit", "coast"), ("coast", "reef")]:
|
||||
_make_transition(ff, a, b)
|
||||
print(f"generated transitions/{a}-{b}.mp4 (placeholder zoom)")
|
||||
n = len(RING_ORDER)
|
||||
for i in range(n):
|
||||
a, b = RING_ORDER[i], RING_ORDER[(i + 1) % n]
|
||||
fwd = _make_transition(ff, a, b)
|
||||
_make_reverse(ff, fwd)
|
||||
print(f"generated transitions/{a}-{b}.mp4 (+ .rev) from real bases")
|
||||
|
||||
|
||||
def main(argv: list[str]) -> None:
|
||||
|
||||
@@ -2,41 +2,41 @@
|
||||
"clips": [
|
||||
{
|
||||
"id": "cosmos",
|
||||
"title": "Orion Nebula flythrough (NASA/JPL-Caltech)",
|
||||
"title": "Cosmic Cliffs — Carina Nebula flythrough (NASA/ESA/CSA/STScI)",
|
||||
"base_file": "cosmos/base.mp4",
|
||||
"license": "public-domain (US Gov, 17 U.S.C. §105)",
|
||||
"source": "NASA/JPL-Caltech — images.nasa.gov JPL-20221122-SOLSYSf-0001 (Orion); trim ~30–54s, crossfade-loop",
|
||||
"license": "CC-BY-class — credit NASA, ESA, CSA, STScI",
|
||||
"source": "NASA SVS 31348 Clifs-3d-STScI (Exploring the Cosmic Cliffs in 3D, Webb NIRCam); 4K source, trim 10–34s, crossfade-loop. Text-free — replaces the Orion SVS 30957 flythrough.",
|
||||
"right_variants": {},
|
||||
"annotations": [
|
||||
{
|
||||
"key": "detected.nebula",
|
||||
"salience": 4,
|
||||
"box": [
|
||||
0.32,
|
||||
0.28,
|
||||
0.34,
|
||||
0.36
|
||||
0.2,
|
||||
0.42,
|
||||
0.5,
|
||||
0.42
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "detected.star",
|
||||
"salience": 2,
|
||||
"box": [
|
||||
0.6,
|
||||
0.3,
|
||||
0.1,
|
||||
0.12
|
||||
0.54,
|
||||
0.12,
|
||||
0.12,
|
||||
0.14
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "measure.redshift",
|
||||
"key": "measure.distance",
|
||||
"box": [
|
||||
0.36,
|
||||
0.7,
|
||||
0.18,
|
||||
0.06,
|
||||
0.06,
|
||||
0.2,
|
||||
0.08
|
||||
],
|
||||
"min_level": 4
|
||||
"min_level": 3
|
||||
}
|
||||
],
|
||||
"affect": [
|
||||
@@ -79,15 +79,15 @@
|
||||
"cloud",
|
||||
"nebula",
|
||||
"emission nebula",
|
||||
"emission nebula · ionized H II region"
|
||||
"emission nebula · the Carina star-forming region"
|
||||
],
|
||||
"detected.star": [
|
||||
"star",
|
||||
"young star",
|
||||
"protostar",
|
||||
"protostar · <1 Myr old"
|
||||
"protostar · a newborn star, <1 Myr old"
|
||||
],
|
||||
"measure.redshift": "z ≈ 0.0",
|
||||
"measure.distance": "≈7,500 ly",
|
||||
"feel.wonder": [
|
||||
"wow",
|
||||
"wonder",
|
||||
|
||||
+352
-14
@@ -52,22 +52,28 @@ async function loadData() {
|
||||
ringIndex = 0;
|
||||
}
|
||||
|
||||
// Land on the current scale: pick a random pool member (content-pipeline §11.1).
|
||||
// A real server ring picks via a delta=0 advance (Python-canonical pick); the
|
||||
// synthesized fallback ring has a pool of one, so it just takes that member.
|
||||
async function landScale() {
|
||||
// Server-canonical random pick of a pool member for the current scale: a delta=0
|
||||
// advance (content-pipeline §11.1, Python owns the randomness). The synthesized
|
||||
// fallback ring has a pool of one, so it just returns that member. Returns a
|
||||
// clip_id or null. Shared by the initial landing AND the Dev Mode re-roll button.
|
||||
async function pickRandomMember() {
|
||||
const scale = ring && ring.scales[ringIndex];
|
||||
if (!scale) { activeClipId = null; return; }
|
||||
if (!scale) return null;
|
||||
if (serverRing) {
|
||||
try {
|
||||
const resp = await fetch("/api/ring/advance", {
|
||||
method: "POST", headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ from_index: ringIndex, delta: 0 }),
|
||||
});
|
||||
if (resp.ok) { activeClipId = (await resp.json()).target_clip_id; currentClipId = null; return; }
|
||||
if (resp.ok) return (await resp.json()).target_clip_id;
|
||||
} catch (_) { /* fall through to the primary member */ }
|
||||
}
|
||||
activeClipId = scale.clip_id;
|
||||
return scale.clip_id;
|
||||
}
|
||||
|
||||
// Land on the current scale: pick a random pool member and force its media to load.
|
||||
async function landScale() {
|
||||
activeClipId = await pickRandomMember();
|
||||
currentClipId = null;
|
||||
}
|
||||
|
||||
@@ -75,7 +81,83 @@ function activeClip() {
|
||||
return clipsById[activeClipId] || null;
|
||||
}
|
||||
|
||||
function mediaUrl(file) { return "/media/" + file; }
|
||||
// In-memory preload cache: media file path -> blob object URL. Once a clip's bytes
|
||||
// are cached, mediaUrl() hands the <video> an in-memory blob instead of a network
|
||||
// path, so swapping the altitude scale loads near-instantly (no fetch round-trip).
|
||||
// Falls back to the network path for anything not yet (or never) cached.
|
||||
const mediaBlobs = {};
|
||||
function mediaUrl(file) { return mediaBlobs[file] || ("/media/" + file); }
|
||||
|
||||
// Every media file the ring can show: each scale-pool clip's base footage plus the
|
||||
// per-edge transition clips. This is the full preload set (~two dozen files).
|
||||
function preloadList() {
|
||||
const files = new Set();
|
||||
for (const c of Object.values(clipsById)) if (c && c.base_file) files.add(c.base_file);
|
||||
// Each ring edge ships both directions: the forward (zoom-in) clip and its baked
|
||||
// reverse (zoom-out) companion — cache both so up and down moves are instant.
|
||||
if (ring && ring.transitions) for (const t of ring.transitions) {
|
||||
if (t && t.file) { files.add(t.file); files.add(reverseFile(t.file)); }
|
||||
}
|
||||
return [...files];
|
||||
}
|
||||
|
||||
// Order the preload so the clips most likely to be shown next come first: the
|
||||
// current scale's pool, then outward to neighboring scales, then transitions.
|
||||
function preloadOrder() {
|
||||
if (!ring || !ring.scales) return preloadList();
|
||||
const n = ring.scales.length, seen = new Set(), ordered = [];
|
||||
const add = (f) => { if (f && !seen.has(f)) { seen.add(f); ordered.push(f); } };
|
||||
for (let d = 0; d < n; d++) {
|
||||
for (const sign of d === 0 ? [0] : [1, -1]) {
|
||||
const s = ring.scales[((ringIndex + sign * d) % n + n) % n];
|
||||
if (s && s.pool) for (const m of s.pool) add((clipsById[m.clip_id] || {}).base_file);
|
||||
else if (s) add((clipsById[s.clip_id] || {}).base_file);
|
||||
}
|
||||
}
|
||||
if (ring.transitions) for (const t of ring.transitions) { add(t.file); add(reverseFile(t.file)); }
|
||||
for (const f of preloadList()) add(f); // sweep up anything not on the ring
|
||||
return ordered;
|
||||
}
|
||||
|
||||
// Fetch every clip into the blob cache in the background — non-blocking, so the
|
||||
// first scale plays immediately while the rest stream in. Bounded concurrency keeps
|
||||
// the ~350 MB fetch from stampeding. A tiny progress chip self-removes when done.
|
||||
async function preloadAllMedia(concurrency = 4) {
|
||||
const files = preloadOrder();
|
||||
if (!files.length) return;
|
||||
let done = 0;
|
||||
const total = files.length;
|
||||
const chip = preloadChip();
|
||||
const tick = () => { if (chip) chip.textContent = `⬇ caching clips ${done}/${total}`; };
|
||||
tick();
|
||||
let i = 0;
|
||||
async function worker() {
|
||||
while (i < files.length) {
|
||||
const file = files[i++];
|
||||
if (!mediaBlobs[file]) {
|
||||
try {
|
||||
const blob = await (await fetch(mediaUrl(file))).blob();
|
||||
mediaBlobs[file] = URL.createObjectURL(blob);
|
||||
} catch (_) { /* leave it to the network path on demand */ }
|
||||
}
|
||||
done++; tick();
|
||||
}
|
||||
}
|
||||
await Promise.all(Array.from({ length: Math.min(concurrency, files.length) }, worker));
|
||||
if (chip) chip.remove();
|
||||
}
|
||||
|
||||
function preloadChip() {
|
||||
let c = document.getElementById("preload-chip");
|
||||
if (!c) {
|
||||
c = document.createElement("div");
|
||||
c.id = "preload-chip";
|
||||
c.style.cssText = "position:fixed;bottom:8px;right:10px;z-index:9998;background:rgba(0,0,0,.55);" +
|
||||
"color:#9fe;font:11px/1.4 monospace;padding:4px 8px;border-radius:4px;pointer-events:none;";
|
||||
document.body.appendChild(c);
|
||||
}
|
||||
return c;
|
||||
}
|
||||
|
||||
// Load the active scale's BASE footage into the video (the painterly canvas reads
|
||||
// its frames live). The Right knob no longer swaps the source — it only drives the
|
||||
@@ -389,6 +471,8 @@ function renderScaleReadout() {
|
||||
// scale id · the chosen pool member · position on the ring (pool size if >1)
|
||||
const poolTag = poolN > 1 ? ` · pool ${poolN}` : "";
|
||||
$("scale-name").textContent = `${s.id} · ${member} (${ringIndex + 1}/${ring.scales.length}${poolTag})`;
|
||||
renderDial();
|
||||
refreshDevClip(); // keep the Dev Mode pool picker + clip data in sync with the landing
|
||||
}
|
||||
|
||||
function controls() {
|
||||
@@ -435,11 +519,17 @@ function debounced() { clearTimeout(timer); timer = setTimeout(update, 80); }
|
||||
|
||||
const FAST_BLEND_RATE = 2.5; // playback speed for a collapsed fast-spin pass
|
||||
|
||||
function playTransition(file, blended) {
|
||||
// The zoom-OUT companion of an edge clip: the baked `<edge>.rev.mp4` reverse. A
|
||||
// `reversed` step (an ascending / zoom-out ring move) plays this instead of the
|
||||
// forward zoom-in clip, so going up recedes from the current scale.
|
||||
function reverseFile(file) { return file.replace(/\.mp4$/, ".rev.mp4"); }
|
||||
|
||||
function playTransition(file, blended, reversed) {
|
||||
// Play one zoom/warp transition clip start-to-finish, with the alteration
|
||||
// layers hidden. A `blended` (fast-spin) pass runs at FAST_BLEND_RATE so a
|
||||
// quick encoder spin lands fast instead of grinding through every morph.
|
||||
// Resolves on 'ended' (or a safety timeout that scales with playback rate).
|
||||
// Zoom-out moves (`reversed`) play the baked reverse clip. Resolves on 'ended'
|
||||
// (or a safety timeout that scales with playback rate).
|
||||
return new Promise((resolve) => {
|
||||
overlay.style.opacity = "0";
|
||||
affectLayer.style.opacity = "0";
|
||||
@@ -447,7 +537,7 @@ function playTransition(file, blended) {
|
||||
vid.style.filter = "none";
|
||||
vid.loop = false;
|
||||
vid.style.opacity = "1";
|
||||
vid.src = mediaUrl(file);
|
||||
vid.src = mediaUrl(reversed ? reverseFile(file) : file);
|
||||
vid.playbackRate = blended ? FAST_BLEND_RATE : 1;
|
||||
let done = false;
|
||||
const finish = () => {
|
||||
@@ -476,7 +566,7 @@ async function advance(delta) {
|
||||
// A fast spin comes back collapsed to a single blended step (move.fast);
|
||||
// a slow spin chains one full transition per scale crossed.
|
||||
for (const step of move.steps) {
|
||||
await playTransition(step.file, step.blended);
|
||||
await playTransition(step.file, step.blended, step.reversed);
|
||||
}
|
||||
ringIndex = move.to_index;
|
||||
activeClipId = move.target_clip_id; // the randomly-picked pool member to load
|
||||
@@ -500,6 +590,248 @@ function onWheel(e) {
|
||||
}, 90);
|
||||
}
|
||||
|
||||
// --- Altitude knob: the endless rotary encoder, drawn as a turnable dial ---
|
||||
// cosmos (highest) sits at the top; turning CLOCKWISE descends through the scales
|
||||
// (cosmos → orbit → coast → reef → abyss) and past the deepest WRAPS back to the
|
||||
// top — the same endless ring the server already models. The drag accumulates a
|
||||
// rotation and commits whole detents on release (so a big spin becomes one fast
|
||||
// blended pass, exactly like the wheel); a tap on a label jumps to that scale.
|
||||
const dial = $("dial");
|
||||
const DIAL_C = 50, DIAL_LABEL_R = 41, DIAL_BODY_R = 27;
|
||||
let needleDeg = 0;
|
||||
let dialDrag = null; // {lastAng, accum, moved} while turning
|
||||
|
||||
function dialStep() { return ring && ring.scales.length ? 360 / ring.scales.length : 360; }
|
||||
|
||||
function _xy(r, deg) {
|
||||
const a = (deg - 90) * Math.PI / 180; // -90° so 0° points UP
|
||||
return [DIAL_C + r * Math.cos(a), DIAL_C + r * Math.sin(a)];
|
||||
}
|
||||
|
||||
function buildDial() {
|
||||
if (!dial || !ring) return;
|
||||
dial.innerHTML = "";
|
||||
const n = ring.scales.length, step = 360 / n;
|
||||
svg("circle", { cx: DIAL_C, cy: DIAL_C, r: DIAL_LABEL_R + 6, class: "dial-rim" }, dial);
|
||||
svg("circle", { cx: DIAL_C, cy: DIAL_C, r: DIAL_BODY_R, class: "dial-body" }, dial);
|
||||
for (let i = 0; i < n; i++) {
|
||||
const deg = i * step;
|
||||
const [tx0, ty0] = _xy(DIAL_BODY_R - 1.5, deg);
|
||||
const [tx1, ty1] = _xy(DIAL_BODY_R + 2.5, deg);
|
||||
svg("line", { x1: tx0, y1: ty0, x2: tx1, y2: ty1, class: "dial-tick" }, dial);
|
||||
const [lx, ly] = _xy(DIAL_LABEL_R, deg);
|
||||
const t = svg("text", {
|
||||
x: lx, y: ly, "text-anchor": "middle", "dominant-baseline": "central",
|
||||
class: "dial-label", "data-index": String(i),
|
||||
}, dial);
|
||||
t.textContent = ring.scales[i].id;
|
||||
}
|
||||
svg("text", { x: DIAL_C, y: DIAL_C - 7, "text-anchor": "middle",
|
||||
"dominant-baseline": "central", class: "dial-caption" }, dial)
|
||||
.textContent = "ALTITUDE";
|
||||
const needle = svg("g", { id: "needle", class: "dial-needle" }, dial);
|
||||
const tip = DIAL_C - (DIAL_BODY_R - 4);
|
||||
svg("polygon", { points: `${DIAL_C - 2.4},${DIAL_C} ${DIAL_C + 2.4},${DIAL_C} ${DIAL_C},${tip}` }, needle);
|
||||
svg("circle", { cx: DIAL_C, cy: DIAL_C, r: 3, class: "dial-hub" }, dial);
|
||||
renderDial();
|
||||
}
|
||||
|
||||
function setNeedle(deg) {
|
||||
const needle = $("needle");
|
||||
if (needle) needle.setAttribute("transform", `rotate(${deg} ${DIAL_C} ${DIAL_C})`);
|
||||
}
|
||||
|
||||
function renderDial() {
|
||||
if (!dial || !ring) return;
|
||||
needleDeg = ringIndex * dialStep();
|
||||
setNeedle(needleDeg);
|
||||
for (const el of dial.querySelectorAll(".dial-label")) {
|
||||
el.classList.toggle("active", +el.getAttribute("data-index") === ringIndex);
|
||||
}
|
||||
}
|
||||
|
||||
function dialAngle(e) {
|
||||
const r = dial.getBoundingClientRect();
|
||||
const dx = e.clientX - (r.left + r.width / 2);
|
||||
const dy = e.clientY - (r.top + r.height / 2);
|
||||
return (Math.atan2(dx, -dy) * 180 / Math.PI + 360) % 360; // 0 at top, clockwise +
|
||||
}
|
||||
function angDelta(a, b) {
|
||||
let d = a - b;
|
||||
while (d > 180) d -= 360;
|
||||
while (d < -180) d += 360;
|
||||
return d;
|
||||
}
|
||||
|
||||
function onDialDown(e) {
|
||||
if (busy || !ring || ring.scales.length < 2) return;
|
||||
e.preventDefault();
|
||||
dialDrag = { lastAng: dialAngle(e), accum: 0, moved: 0, target: e.target };
|
||||
}
|
||||
function onDialMove(e) {
|
||||
if (!dialDrag) return;
|
||||
const a = dialAngle(e);
|
||||
const d = angDelta(a, dialDrag.lastAng);
|
||||
dialDrag.lastAng = a;
|
||||
dialDrag.accum += d;
|
||||
dialDrag.moved += Math.abs(d);
|
||||
setNeedle(ringIndex * dialStep() + dialDrag.accum); // live feedback while turning
|
||||
}
|
||||
function onDialUp(e) {
|
||||
if (!dialDrag) return;
|
||||
const { accum, moved, target } = dialDrag;
|
||||
dialDrag = null;
|
||||
if (moved < 6) { // a tap, not a turn
|
||||
if (target && target.classList && target.classList.contains("dial-label")) {
|
||||
jumpToScale(+target.getAttribute("data-index"));
|
||||
}
|
||||
renderDial();
|
||||
return;
|
||||
}
|
||||
const detents = Math.round(accum / dialStep());
|
||||
if (detents) advance(detents); else renderDial(); // snap back if it didn't cross a detent
|
||||
}
|
||||
|
||||
// Click a label → travel the SHORTEST signed way around the ring to that scale.
|
||||
function jumpToScale(idx) {
|
||||
if (!ring) return;
|
||||
const n = ring.scales.length;
|
||||
let d = (idx - ringIndex) % n;
|
||||
if (d > n / 2) d -= n;
|
||||
if (d < -n / 2) d += n;
|
||||
if (d) advance(d);
|
||||
}
|
||||
|
||||
// --- Dev Mode: pool picker + live analysis, all under one toggle ---
|
||||
// Off by default, state persisted in localStorage. Everything here is read from
|
||||
// data the renderer already has (clips, ring, the alteration response, the preload
|
||||
// cache) — no server endpoints. Editing labels stays in /author.html.
|
||||
const DEV_KEY = "hef.devMode";
|
||||
let devMode = false;
|
||||
let devStatsTimer = null;
|
||||
|
||||
const escapeHtml = (s) => String(s).replace(/[&<>"]/g,
|
||||
(c) => ({ "&": "&", "<": "<", ">": ">", '"': """ }[c]));
|
||||
|
||||
function applyDevVisibility() {
|
||||
const panel = $("dev-panel");
|
||||
if (panel) panel.classList.toggle("hidden", !devMode);
|
||||
if (devMode) { refreshDevClip(); renderDevStats(); }
|
||||
}
|
||||
|
||||
function initDev() {
|
||||
const toggle = $("dev-mode");
|
||||
if (!toggle) return;
|
||||
try { devMode = localStorage.getItem(DEV_KEY) === "1"; } catch (_) { devMode = false; }
|
||||
toggle.checked = devMode;
|
||||
toggle.addEventListener("change", () => {
|
||||
devMode = toggle.checked;
|
||||
try { localStorage.setItem(DEV_KEY, devMode ? "1" : "0"); } catch (_) {}
|
||||
applyDevVisibility();
|
||||
});
|
||||
$("pool-select").addEventListener("change", onPoolSelect);
|
||||
$("pool-reroll").addEventListener("click", reroll);
|
||||
// Cache + playback stats are live-ish; cheap to repaint twice a second when shown.
|
||||
devStatsTimer = setInterval(() => { if (devMode) renderDevStats(); }, 500);
|
||||
applyDevVisibility();
|
||||
}
|
||||
|
||||
// Rebuild the pool picker for the current scale and repaint the clip data. Cheap
|
||||
// (pools are <= a handful), so just rebuild on every landing/pick.
|
||||
function refreshDevClip() {
|
||||
if (!devMode) return;
|
||||
const sel = $("pool-select");
|
||||
const scale = ring && ring.scales[ringIndex];
|
||||
if (sel && scale) {
|
||||
const pool = (scale.pool && scale.pool.length) ? scale.pool : [{ clip_id: scale.clip_id, title: scale.title }];
|
||||
sel.innerHTML = "";
|
||||
for (const m of pool) {
|
||||
const o = document.createElement("option");
|
||||
o.value = m.clip_id;
|
||||
o.textContent = `${m.clip_id} · ${m.title || (clipsById[m.clip_id] || {}).title || ""}`;
|
||||
sel.appendChild(o);
|
||||
}
|
||||
if (activeClipId) sel.value = activeClipId;
|
||||
sel.disabled = pool.length < 2;
|
||||
}
|
||||
renderDevMeta();
|
||||
renderDevAnno();
|
||||
}
|
||||
|
||||
function renderDevMeta() {
|
||||
const dl = $("clip-meta");
|
||||
const c = activeClip();
|
||||
if (!dl) return;
|
||||
if (!c) { dl.innerHTML = "<dt>—</dt><dd>no clip</dd>"; return; }
|
||||
const rows = [["id", c.id], ["title", c.title], ["source", c.source],
|
||||
["license", c.license], ["base_file", c.base_file]];
|
||||
dl.innerHTML = rows.map(([k, v]) =>
|
||||
`<dt>${escapeHtml(k)}</dt><dd>${escapeHtml(v != null ? v : "—")}</dd>`).join("");
|
||||
}
|
||||
|
||||
function renderDevAnno() {
|
||||
const box = $("clip-anno");
|
||||
const c = activeClip();
|
||||
if (!box) return;
|
||||
const strings = (c && c.strings && c.strings.en) || {};
|
||||
const win = (a) => (typeof a.appear === "number" || typeof a.disappear === "number")
|
||||
? ` win ${(a.appear ?? 0).toFixed(2)}–${(a.disappear ?? 1).toFixed(2)}` : "";
|
||||
const rows = [];
|
||||
for (const a of (c && c.annotations) || []) {
|
||||
const measure = a.key.startsWith("measure.");
|
||||
const sal = typeof a.salience === "number" ? `sal ${a.salience}`
|
||||
: typeof a.min_level === "number" ? `min L${a.min_level}` : "sal 1";
|
||||
const tiers = tierCount(strings[a.key]);
|
||||
rows.push(`<div class="anno-row"><span class="anno-key ${measure ? "measure" : ""}">${escapeHtml(a.key)}</span>` +
|
||||
`<span class="anno-meta"> · ${sal} · ${tiers} tier${tiers > 1 ? "s" : ""}${win(a)}</span></div>`);
|
||||
}
|
||||
for (const f of (c && c.affect) || []) {
|
||||
const tiers = tierCount(strings[f.key]);
|
||||
rows.push(`<div class="anno-row"><span class="anno-key affect">${escapeHtml(f.key)}</span>` +
|
||||
`<span class="anno-meta"> · min L${f.min_level ?? 1} · ${tiers} tier${tiers > 1 ? "s" : ""}</span></div>`);
|
||||
}
|
||||
box.innerHTML = rows.length ? rows.join("") : `<div class="anno-empty">no annotations or affect</div>`;
|
||||
}
|
||||
|
||||
function renderDevStats() {
|
||||
const dl = $("dev-stats");
|
||||
if (!dl) return;
|
||||
const c = activeClip();
|
||||
const total = preloadList().length;
|
||||
const cached = Object.keys(mediaBlobs).length;
|
||||
const fromMem = !!(c && mediaBlobs[c.base_file]);
|
||||
const res = vid.videoWidth ? `${vid.videoWidth}×${vid.videoHeight}` : "—";
|
||||
const dur = vid.duration ? `${vid.duration.toFixed(1)}s` : "—";
|
||||
const loop = vid.duration ? `${Math.round(loopT() * 100)}%` : "—";
|
||||
dl.innerHTML =
|
||||
`<dt>cache</dt><dd>${cached}/${total} cached</dd>` +
|
||||
`<dt>active source</dt><dd class="${fromMem ? "mem" : "net"}">${fromMem ? "memory (blob)" : "network"}</dd>` +
|
||||
`<dt>resolution</dt><dd>${res}</dd>` +
|
||||
`<dt>duration</dt><dd>${dur}</dd>` +
|
||||
`<dt>loop pos</dt><dd>${loop}</dd>`;
|
||||
}
|
||||
|
||||
// Pick a specific pool member (Dev Mode override of the random landing pick).
|
||||
function onPoolSelect() {
|
||||
const id = $("pool-select").value;
|
||||
if (!id || id === activeClipId) return;
|
||||
activeClipId = id;
|
||||
currentClipId = null; // force the base media to reload
|
||||
renderScaleReadout(); // refresh scale name + dev clip data
|
||||
update(); // reload media + plan
|
||||
}
|
||||
|
||||
// Re-roll: a fresh server-canonical random pick for the current scale.
|
||||
async function reroll() {
|
||||
if (busy) return;
|
||||
const id = await pickRandomMember();
|
||||
if (!id) return;
|
||||
activeClipId = id;
|
||||
currentClipId = null;
|
||||
renderScaleReadout();
|
||||
update();
|
||||
}
|
||||
|
||||
// Dev live-reload: poll the asset version and reload when it changes, so an open
|
||||
// tab never keeps running a stale renderer while we iterate (the readout updates
|
||||
// from the live API and masks it otherwise). Reloads on my edits AND on a server
|
||||
@@ -520,12 +852,18 @@ async function main() {
|
||||
try { initPaint(); } catch (e) { paintOK = false; paint.style.display = "none"; showError("WebGL init: " + e.message); }
|
||||
await loadData();
|
||||
await landScale(); // pick the initial scale's pool member before first render
|
||||
preloadAllMedia(); // background: cache every clip in memory for instant altitude swaps
|
||||
buildDial(); // draw the altitude knob from the ring's scales
|
||||
initDev(); // wire the Dev Mode toggle + pool picker (reads persisted state)
|
||||
renderScaleReadout();
|
||||
for (const id of ["content", "left", "right", "mood"]) {
|
||||
$(id).addEventListener("input", debounced);
|
||||
}
|
||||
$("zoom-in").addEventListener("click", () => advance(1));
|
||||
$("zoom-out").addEventListener("click", () => advance(-1));
|
||||
// Altitude knob: drag to turn (commit detents on release), scroll to step, tap a label to jump.
|
||||
dial.addEventListener("pointerdown", onDialDown);
|
||||
window.addEventListener("pointermove", onDialMove);
|
||||
window.addEventListener("pointerup", onDialUp);
|
||||
dial.addEventListener("wheel", onWheel, { passive: false });
|
||||
$("stage").addEventListener("wheel", onWheel, { passive: false });
|
||||
update();
|
||||
}
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
/* Author-mode-only styling (content-pipeline §11.5). Reuses style.css for the
|
||||
shared stage/panel chrome; this adds the box-draw layer + editor widgets. */
|
||||
|
||||
.author #draw {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
cursor: crosshair;
|
||||
}
|
||||
.author #draw .seed-box {
|
||||
fill: rgba(80, 200, 255, 0.12);
|
||||
stroke: #50c8ff;
|
||||
stroke-width: 0.4;
|
||||
vector-effect: non-scaling-stroke;
|
||||
}
|
||||
.author #draw .track-box {
|
||||
fill: rgba(120, 255, 160, 0.10);
|
||||
stroke: #78ffa0;
|
||||
stroke-width: 0.4;
|
||||
stroke-dasharray: 1.5 1;
|
||||
vector-effect: non-scaling-stroke;
|
||||
}
|
||||
.author #draw .affect-dot {
|
||||
fill: rgba(190, 150, 255, 0.85);
|
||||
stroke: #fff;
|
||||
stroke-width: 0.25;
|
||||
vector-effect: non-scaling-stroke;
|
||||
}
|
||||
|
||||
.author .scrub {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-top: 8px;
|
||||
}
|
||||
.author .scrub input[type="range"] { flex: 1; }
|
||||
.author .mono { font-family: ui-monospace, monospace; font-size: 12px; opacity: 0.85; }
|
||||
|
||||
.author .row { display: flex; gap: 8px; margin-top: 6px; flex-wrap: wrap; }
|
||||
.author fieldset label { display: block; margin: 4px 0; font-size: 13px; }
|
||||
.author fieldset input[type="text"],
|
||||
.author fieldset input[type="number"] { width: 100%; box-sizing: border-box; }
|
||||
.author fieldset input[type="number"] { width: 5em; }
|
||||
|
||||
.author ul.authored { list-style: none; padding: 0; margin: 6px 0; }
|
||||
.author ul.authored li {
|
||||
font-size: 12px;
|
||||
padding: 3px 6px;
|
||||
margin: 2px 0;
|
||||
background: rgba(255, 255, 255, 0.05);
|
||||
border-radius: 3px;
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
gap: 6px;
|
||||
}
|
||||
.author ul.authored .rm {
|
||||
background: none;
|
||||
border: none;
|
||||
color: #f88;
|
||||
cursor: pointer;
|
||||
font-size: 12px;
|
||||
}
|
||||
.author header .hint { font-size: 12px; opacity: 0.8; max-width: 70ch; }
|
||||
@@ -0,0 +1,79 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>HEF — Label Author Mode</title>
|
||||
<link rel="stylesheet" href="/style.css" />
|
||||
<link rel="stylesheet" href="/author.css" />
|
||||
</head>
|
||||
<body class="author">
|
||||
<header><h1>HEF — Label Author Mode</h1>
|
||||
<p class="hint">Scrub a pool clip · drag a box on a subject · run the tracker · author the tiers · save to the manifest. Geometry is propagated; <strong>semantics are hand-authored</strong> (content-pipeline §11.5). <a href="/">← back to preview</a></p>
|
||||
</header>
|
||||
<main>
|
||||
<section class="stage" id="stage">
|
||||
<div class="screen">
|
||||
<video id="vid" muted playsinline></video>
|
||||
<svg id="draw" viewBox="0 0 100 100" preserveAspectRatio="none"></svg>
|
||||
</div>
|
||||
<div class="scrub">
|
||||
<button type="button" id="play">▶/⏸</button>
|
||||
<input type="range" id="seek" min="0" max="1000" value="0" />
|
||||
<span id="time" class="mono">0.00</span>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="panel">
|
||||
<fieldset>
|
||||
<legend>Clip</legend>
|
||||
<select id="clip"></select>
|
||||
<p class="hint" id="clip-meta">—</p>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Seed box (drag on the video)</legend>
|
||||
<p class="mono" id="seed-readout">no box — drag on the stage</p>
|
||||
<label>label key <input type="text" id="ann-key" placeholder="detected.jelly" /></label>
|
||||
<label>salience (1–4; 4 = shows first/at low Left)
|
||||
<input type="number" id="ann-salience" min="1" max="4" value="4" /></label>
|
||||
<label>tier 1 — general <input type="text" id="t1" placeholder="jelly" /></label>
|
||||
<label>tier 2 — specific <input type="text" id="t2" placeholder="comb jelly" /></label>
|
||||
<label>tier 3 — scientific <input type="text" id="t3" placeholder="Ctenophora" /></label>
|
||||
<label>tier 4 — +fact <input type="text" id="t4" placeholder="Ctenophora · beats rows of cilia" /></label>
|
||||
<label>keyframes <input type="number" id="ann-keyframes" min="2" max="20" value="5" /></label>
|
||||
<div class="row">
|
||||
<button type="button" id="run-track">Run tracker → track</button>
|
||||
<button type="button" id="add-static">Add as static box</button>
|
||||
</div>
|
||||
<p class="mono" id="track-readout"></p>
|
||||
<button type="button" id="add-ann" disabled>+ Add label</button>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Affect anchor (click the stage to place)</legend>
|
||||
<p class="mono" id="affect-readout">no point — shift-click the stage</p>
|
||||
<label>feel key <input type="text" id="aff-key" placeholder="feel.unease" /></label>
|
||||
<label>min strength (1–4) <input type="number" id="aff-min" min="1" max="4" value="1" /></label>
|
||||
<label>tier 1 — basic <input type="text" id="a1" placeholder="uh" /></label>
|
||||
<label>tier 2 <input type="text" id="a2" placeholder="unease" /></label>
|
||||
<label>tier 3 <input type="text" id="a3" placeholder="disquiet" /></label>
|
||||
<label>tier 4 — compound <input type="text" id="a4" placeholder="a creeping disquiet" /></label>
|
||||
<button type="button" id="add-aff" disabled>+ Add emotion</button>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Authored for this clip</legend>
|
||||
<ul id="ann-list" class="authored"></ul>
|
||||
<ul id="aff-list" class="authored"></ul>
|
||||
<div class="row">
|
||||
<button type="button" id="load-existing">Load existing</button>
|
||||
<button type="button" id="save">Save to manifest</button>
|
||||
</div>
|
||||
<p class="mono" id="save-readout"></p>
|
||||
</fieldset>
|
||||
</section>
|
||||
</main>
|
||||
<script src="/author.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,254 @@
|
||||
// Label author mode (content-pipeline §11.5): scrub a pool clip, drag a seed box
|
||||
// on a subject, run the classical tracker to propagate a keyframed track, author
|
||||
// the LEFT detail tiers + RIGHT emotion tiers, and save the entry to the manifest.
|
||||
// Geometry comes from the tracker; SEMANTICS (keys, strings, tiers) are hand-typed.
|
||||
const $ = (id) => document.getElementById(id);
|
||||
const vid = $("vid"), draw = $("draw");
|
||||
const SVGNS = "http://www.w3.org/2000/svg";
|
||||
|
||||
let clips = [];
|
||||
let currentClip = null;
|
||||
let seedBox = null; // [x,y,w,h] normalized, from a drag
|
||||
let pendingTrack = null; // {track, appear, disappear} from the tracker (or static)
|
||||
let affectPoint = null; // [x,y] normalized, from a shift-click
|
||||
let annotations = []; // authored annotation dicts (carry _tiers)
|
||||
let affect = []; // authored affect dicts (carry _tiers)
|
||||
|
||||
function svg(tag, attrs, parent) {
|
||||
const el = document.createElementNS(SVGNS, tag);
|
||||
for (const k in attrs) el.setAttribute(k, attrs[k]);
|
||||
if (parent) parent.appendChild(el);
|
||||
return el;
|
||||
}
|
||||
const mono = (n) => Number(n).toFixed(3);
|
||||
|
||||
async function loadClips() {
|
||||
clips = (await (await fetch("/api/clips")).json()).clips || [];
|
||||
const sel = $("clip");
|
||||
sel.innerHTML = "";
|
||||
for (const c of clips) {
|
||||
const o = document.createElement("option");
|
||||
o.value = c.id; o.textContent = c.id;
|
||||
sel.appendChild(o);
|
||||
}
|
||||
if (clips.length) selectClip(clips[0].id);
|
||||
}
|
||||
|
||||
function selectClip(id) {
|
||||
currentClip = clips.find((c) => c.id === id) || null;
|
||||
if (!currentClip) return;
|
||||
$("clip-meta").textContent = `${currentClip.title} — ${currentClip.license}`;
|
||||
vid.src = "/media/" + currentClip.base_file;
|
||||
vid.currentTime = 0;
|
||||
annotations = []; affect = []; seedBox = null; pendingTrack = null; affectPoint = null;
|
||||
renderLists(); renderDrawLayer();
|
||||
$("seed-readout").textContent = "no box — drag on the stage";
|
||||
$("track-readout").textContent = "";
|
||||
$("affect-readout").textContent = "no point — shift-click the stage";
|
||||
}
|
||||
|
||||
// --- scrub ---
|
||||
function seekT() { return vid.duration ? vid.currentTime / vid.duration : 0; }
|
||||
$("play").addEventListener("click", () => { vid.paused ? vid.play() : vid.pause(); });
|
||||
$("seek").addEventListener("input", () => {
|
||||
if (vid.duration) vid.currentTime = (+$("seek").value / 1000) * vid.duration;
|
||||
});
|
||||
vid.addEventListener("timeupdate", () => {
|
||||
if (!vid.duration) return;
|
||||
$("seek").value = String(Math.round(seekT() * 1000));
|
||||
$("time").textContent = seekT().toFixed(3);
|
||||
});
|
||||
|
||||
// --- draw a seed box / place an affect point on the stage ---
|
||||
function evtNorm(e) {
|
||||
const r = draw.getBoundingClientRect();
|
||||
return [(e.clientX - r.left) / r.width, (e.clientY - r.top) / r.height];
|
||||
}
|
||||
let dragStart = null;
|
||||
draw.addEventListener("mousedown", (e) => {
|
||||
if (e.shiftKey) { // shift-click = affect anchor
|
||||
affectPoint = evtNorm(e).map((v) => Math.min(Math.max(v, 0), 1));
|
||||
$("affect-readout").textContent = `affect @ ${affectPoint.map(mono).join(", ")}`;
|
||||
$("add-aff").disabled = false;
|
||||
renderDrawLayer();
|
||||
return;
|
||||
}
|
||||
dragStart = evtNorm(e);
|
||||
});
|
||||
draw.addEventListener("mousemove", (e) => {
|
||||
if (!dragStart) return;
|
||||
seedBox = boxFrom(dragStart, evtNorm(e));
|
||||
renderDrawLayer();
|
||||
});
|
||||
window.addEventListener("mouseup", (e) => {
|
||||
if (!dragStart) return;
|
||||
seedBox = boxFrom(dragStart, evtNorm(e));
|
||||
dragStart = null;
|
||||
pendingTrack = null;
|
||||
$("seed-readout").textContent = `seed [${seedBox.map(mono).join(", ")}] @ t=${seekT().toFixed(3)}`;
|
||||
$("add-ann").disabled = false;
|
||||
renderDrawLayer();
|
||||
});
|
||||
function boxFrom(a, b) {
|
||||
const x = Math.min(a[0], b[0]), y = Math.min(a[1], b[1]);
|
||||
const w = Math.abs(b[0] - a[0]), h = Math.abs(b[1] - a[1]);
|
||||
const cl = (v) => Math.min(Math.max(v, 0), 1);
|
||||
return [cl(x), cl(y), Math.min(w, 1 - cl(x)), Math.min(h, 1 - cl(y))];
|
||||
}
|
||||
|
||||
function renderDrawLayer() {
|
||||
draw.innerHTML = "";
|
||||
if (seedBox) {
|
||||
const [x, y, w, h] = seedBox.map((n) => n * 100);
|
||||
svg("rect", { x, y, width: w, height: h, class: "seed-box" }, draw);
|
||||
}
|
||||
if (pendingTrack && pendingTrack.track && vid.duration) {
|
||||
const b = boxAt(pendingTrack.track, seekT());
|
||||
if (b) {
|
||||
const [x, y, w, h] = b.map((n) => n * 100);
|
||||
svg("rect", { x, y, width: w, height: h, class: "track-box" }, draw);
|
||||
}
|
||||
}
|
||||
if (affectPoint) {
|
||||
const [x, y] = affectPoint.map((n) => n * 100);
|
||||
svg("circle", { cx: x, cy: y, r: 1.2, class: "affect-dot" }, draw);
|
||||
}
|
||||
}
|
||||
// Interpolate a track box at loop-normalized t (mirrors the preview renderer).
|
||||
function boxAt(track, t) {
|
||||
if (!track || !track.length) return null;
|
||||
if (t <= track[0].t) return track[0].box;
|
||||
if (t >= track[track.length - 1].t) return track[track.length - 1].box;
|
||||
for (let i = 1; i < track.length; i++) {
|
||||
if (t <= track[i].t) {
|
||||
const a = track[i - 1], b = track[i], f = (t - a.t) / (b.t - a.t);
|
||||
return a.box.map((v, j) => v + (b.box[j] - v) * f);
|
||||
}
|
||||
}
|
||||
return track[track.length - 1].box;
|
||||
}
|
||||
// Animate the pending track preview while playing.
|
||||
function previewLoop() { renderDrawLayer(); requestAnimationFrame(previewLoop); }
|
||||
|
||||
// --- run the tracker on the seed box ---
|
||||
$("run-track").addEventListener("click", async () => {
|
||||
if (!seedBox || !currentClip) { $("track-readout").textContent = "draw a seed box first"; return; }
|
||||
const key = $("ann-key").value.trim() || "detected.object";
|
||||
$("track-readout").textContent = "tracking…";
|
||||
const resp = await fetch("/api/author/track", {
|
||||
method: "POST", headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
clip_id: currentClip.id, key, seed_box: seedBox, seed_t: seekT(),
|
||||
salience: +$("ann-salience").value, n_keyframes: +$("ann-keyframes").value,
|
||||
}),
|
||||
});
|
||||
if (!resp.ok) { $("track-readout").textContent = "tracker error " + resp.status; return; }
|
||||
const ann = await resp.json();
|
||||
pendingTrack = { track: ann.track, appear: ann.appear, disappear: ann.disappear };
|
||||
$("track-readout").textContent =
|
||||
`track: ${ann.track.length} keyframes · window ${mono(ann.appear)}–${mono(ann.disappear)}`;
|
||||
$("add-ann").disabled = false;
|
||||
});
|
||||
|
||||
// Add the current seed as a STATIC (untracked) box at the scrub position.
|
||||
$("add-static").addEventListener("click", () => {
|
||||
if (!seedBox) return;
|
||||
pendingTrack = { static: true, box: seedBox.slice() };
|
||||
$("track-readout").textContent = "static box (no track) at the drawn position";
|
||||
$("add-ann").disabled = false;
|
||||
});
|
||||
|
||||
function tiers(ids) {
|
||||
const vals = ids.map((id) => $(id).value.trim());
|
||||
while (vals.length && !vals[vals.length - 1]) vals.pop(); // drop empty trailing tiers
|
||||
return vals.length ? vals : null;
|
||||
}
|
||||
|
||||
$("add-ann").addEventListener("click", () => {
|
||||
const key = $("ann-key").value.trim();
|
||||
if (!key) { $("track-readout").textContent = "a label key is required"; return; }
|
||||
const t = tiers(["t1", "t2", "t3", "t4"]);
|
||||
const ann = { key, salience: +$("ann-salience").value, _tiers: t || key };
|
||||
if (pendingTrack && pendingTrack.track) {
|
||||
ann.track = pendingTrack.track;
|
||||
ann.appear = pendingTrack.appear; ann.disappear = pendingTrack.disappear;
|
||||
} else if (pendingTrack && pendingTrack.static) {
|
||||
ann.box = pendingTrack.box;
|
||||
} else if (seedBox) {
|
||||
ann.box = seedBox.slice();
|
||||
} else { $("track-readout").textContent = "draw + (optionally) track a box first"; return; }
|
||||
annotations.push(ann);
|
||||
seedBox = null; pendingTrack = null; $("add-ann").disabled = true;
|
||||
renderLists(); renderDrawLayer();
|
||||
});
|
||||
|
||||
$("add-aff").addEventListener("click", () => {
|
||||
const key = $("aff-key").value.trim();
|
||||
if (!key || !affectPoint) return;
|
||||
const t = tiers(["a1", "a2", "a3", "a4"]);
|
||||
affect.push({ key, at: affectPoint.slice(), min_level: +$("aff-min").value, _tiers: t || key });
|
||||
affectPoint = null; $("add-aff").disabled = true;
|
||||
renderLists(); renderDrawLayer();
|
||||
});
|
||||
|
||||
function renderLists() {
|
||||
const al = $("ann-list"); al.innerHTML = "";
|
||||
annotations.forEach((a, i) => {
|
||||
const li = document.createElement("li");
|
||||
const kind = a.track ? `track ${a.track.length}kf ${mono(a.appear)}–${mono(a.disappear)}` : "static";
|
||||
const tip = Array.isArray(a._tiers) ? a._tiers.join(" → ") : a._tiers;
|
||||
li.textContent = `${a.key} (s${a.salience}, ${kind}) — ${tip}`;
|
||||
li.appendChild(rm(() => { annotations.splice(i, 1); renderLists(); }));
|
||||
al.appendChild(li);
|
||||
});
|
||||
const fl = $("aff-list"); fl.innerHTML = "";
|
||||
affect.forEach((f, i) => {
|
||||
const li = document.createElement("li");
|
||||
const tip = Array.isArray(f._tiers) ? f._tiers.join(" → ") : f._tiers;
|
||||
li.textContent = `${f.key} (min ${f.min_level}) — ${tip}`;
|
||||
li.appendChild(rm(() => { affect.splice(i, 1); renderLists(); }));
|
||||
fl.appendChild(li);
|
||||
});
|
||||
}
|
||||
function rm(fn) {
|
||||
const b = document.createElement("button");
|
||||
b.textContent = "✕"; b.className = "rm"; b.onclick = fn;
|
||||
return b;
|
||||
}
|
||||
|
||||
// Load whatever is already authored for this clip into the editor.
|
||||
$("load-existing").addEventListener("click", () => {
|
||||
if (!currentClip) return;
|
||||
const strings = (currentClip.strings && currentClip.strings.en) || {};
|
||||
annotations = (currentClip.annotations || []).map((a) => ({ ...a, _tiers: strings[a.key] || a.key }));
|
||||
affect = (currentClip.affect || []).map((f) => ({ ...f, _tiers: strings[f.key] || f.key }));
|
||||
renderLists();
|
||||
});
|
||||
|
||||
// Assemble strings.en from each entry's _tiers and POST to the manifest.
|
||||
$("save").addEventListener("click", async () => {
|
||||
if (!currentClip) return;
|
||||
const en = {};
|
||||
const strip = (x) => { const { _tiers, ...rest } = x; return rest; };
|
||||
const anns = annotations.map((a) => { en[a.key] = a._tiers; return strip(a); });
|
||||
const affs = affect.map((f) => { en[f.key] = f._tiers; return strip(f); });
|
||||
const resp = await fetch("/api/author/clip", {
|
||||
method: "POST", headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({ clip_id: currentClip.id, annotations: anns, affect: affs, strings: { en } }),
|
||||
});
|
||||
$("save-readout").textContent = resp.ok
|
||||
? `saved ${anns.length} labels + ${affs.length} emotions to the manifest ✓`
|
||||
: "save error " + resp.status;
|
||||
if (resp.ok) { // refresh the in-memory clip copy
|
||||
clips = (await (await fetch("/api/clips")).json()).clips || [];
|
||||
currentClip = clips.find((c) => c.id === currentClip.id) || currentClip;
|
||||
}
|
||||
});
|
||||
|
||||
$("clip").addEventListener("change", (e) => selectClip(e.target.value));
|
||||
|
||||
async function main() {
|
||||
await loadClips();
|
||||
requestAnimationFrame(previewLoop);
|
||||
}
|
||||
main();
|
||||
+38
-10
@@ -35,13 +35,12 @@
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Scale ring (endless encoder)</legend>
|
||||
<div class="ring-control">
|
||||
<button type="button" id="zoom-out" title="zoom out (toward cosmos)">⊖ out</button>
|
||||
<span id="scale-name" class="scale-name">—</span>
|
||||
<button type="button" id="zoom-in" title="zoom in (toward microscopic)">in ⊕</button>
|
||||
<legend>Altitude</legend>
|
||||
<div class="dial-wrap">
|
||||
<svg id="dial" viewBox="0 0 100 100" aria-label="Altitude knob (turn to change scale)"></svg>
|
||||
</div>
|
||||
<p class="hint">Relative, endless — wraps past the smallest back to the largest. Scroll the stage too.</p>
|
||||
<span id="scale-name" class="scale-name">—</span>
|
||||
<p class="hint">Turn the knob (drag it, or scroll) to change altitude — endless: past the deepest it wraps back up to the highest. Click a label to jump there.</p>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
@@ -51,11 +50,40 @@
|
||||
<label>Mood — dark ◀ 0 ▶ light <input type="range" id="mood" min="-4" max="4" value="0" /></label>
|
||||
</fieldset>
|
||||
|
||||
<!-- Dev Mode: a single toggle; everything dev (pool picker + analysis) lives below it. -->
|
||||
<label class="dev-switch" for="dev-mode">
|
||||
<input type="checkbox" id="dev-mode" />
|
||||
<span class="dev-switch-track"><span class="dev-switch-thumb"></span></span>
|
||||
<span class="dev-switch-label">Dev Mode</span>
|
||||
</label>
|
||||
|
||||
<fieldset>
|
||||
<legend>RenderPlan readout</legend>
|
||||
<pre id="readout">—</pre>
|
||||
</fieldset>
|
||||
<div id="dev-panel" class="dev-panel hidden">
|
||||
<fieldset>
|
||||
<legend>Pool — pick a clip</legend>
|
||||
<select id="pool-select" aria-label="Choose a clip from this altitude's pool"></select>
|
||||
<button type="button" id="pool-reroll" class="dev-btn">🎲 Re-roll random</button>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Active clip</legend>
|
||||
<dl id="clip-meta" class="dev-dl"></dl>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Annotations & affect</legend>
|
||||
<div id="clip-anno" class="dev-anno"></div>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>RenderPlan</legend>
|
||||
<pre id="readout">—</pre>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Cache & playback</legend>
|
||||
<dl id="dev-stats" class="dev-dl"></dl>
|
||||
</fieldset>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
<script src="/app.js"></script>
|
||||
|
||||
@@ -48,10 +48,55 @@ label { display: block; margin: 0.4rem 0; }
|
||||
input[type=range], select { width: 100%; }
|
||||
#readout { background: #000; padding: 0.5rem; border-radius: 4px; font-size: 12px;
|
||||
white-space: pre-wrap; max-height: 240px; overflow: auto; }
|
||||
.ring-control { display: flex; align-items: center; gap: 0.4rem; }
|
||||
.ring-control button { flex: 0 0 auto; background: #1a2436; color: #9af;
|
||||
border: 1px solid #345; border-radius: 4px; padding: 0.3rem 0.5rem;
|
||||
cursor: pointer; font-size: 13px; }
|
||||
.ring-control button:hover { background: #243352; }
|
||||
.scale-name { flex: 1 1 auto; text-align: center; font-size: 12px; color: #cde; }
|
||||
/* Altitude knob */
|
||||
.dial-wrap { display: flex; justify-content: center; padding: 0.3rem 0 0.1rem; }
|
||||
#dial { width: 190px; height: 190px; touch-action: none; cursor: grab; user-select: none; }
|
||||
#dial:active { cursor: grabbing; }
|
||||
.dial-rim { fill: #0d1320; stroke: #243352; stroke-width: 1.2; }
|
||||
.dial-body { fill: #16203200; stroke: #2c3c5c; stroke-width: 1; }
|
||||
.dial-tick { stroke: #3a4d70; stroke-width: 0.8; }
|
||||
.dial-label { fill: #789ac0; font-size: 6px; font-family: ui-monospace, monospace;
|
||||
letter-spacing: 0.2px; cursor: pointer; }
|
||||
.dial-label:hover { fill: #cde; }
|
||||
.dial-label.active { fill: #9cf; font-weight: 700; }
|
||||
.dial-caption { fill: #4d6184; font-size: 4.4px; letter-spacing: 1.2px;
|
||||
font-family: ui-monospace, monospace; }
|
||||
.dial-needle { fill: #9cf; }
|
||||
.dial-needle polygon { filter: drop-shadow(0 0 1px #9cf); }
|
||||
.dial-hub { fill: #2c3c5c; stroke: #9cf; stroke-width: 0.6; }
|
||||
.scale-name { display: block; text-align: center; font-size: 12px; color: #cde; }
|
||||
.hint { margin: 0.3rem 0 0; font-size: 11px; color: #789; }
|
||||
|
||||
/* --- Dev Mode --- a single switch; all dev controls/data live in #dev-panel below it. */
|
||||
.dev-switch { display: flex; align-items: center; gap: 0.5rem; cursor: pointer;
|
||||
padding: 0.5rem 0.2rem 0.2rem; border-top: 1px solid #222; user-select: none; }
|
||||
.dev-switch input { position: absolute; opacity: 0; width: 0; height: 0; }
|
||||
.dev-switch-track { position: relative; width: 34px; height: 18px; border-radius: 9px;
|
||||
background: #2a2a2a; border: 1px solid #444; transition: background 0.15s; }
|
||||
.dev-switch-thumb { position: absolute; top: 1px; left: 1px; width: 14px; height: 14px;
|
||||
border-radius: 50%; background: #888; transition: transform 0.15s, background 0.15s; }
|
||||
.dev-switch input:checked + .dev-switch-track { background: #1d3a2a; border-color: #2e6; }
|
||||
.dev-switch input:checked + .dev-switch-track .dev-switch-thumb { transform: translateX(16px); background: #4e9; }
|
||||
.dev-switch input:focus-visible + .dev-switch-track { outline: 2px solid #9af; outline-offset: 1px; }
|
||||
.dev-switch-label { color: #9af; font-weight: 600; letter-spacing: 0.3px; }
|
||||
.dev-panel { display: flex; flex-direction: column; gap: 0.8rem; }
|
||||
/* Beat the generic `.hidden` (defined earlier, equal specificity): two classes
|
||||
win, so the panel truly collapses when Dev Mode is off. */
|
||||
.dev-panel.hidden { display: none; }
|
||||
.dev-panel legend { color: #4e9; }
|
||||
.dev-btn { margin-top: 0.5rem; width: 100%; padding: 0.4rem; cursor: pointer;
|
||||
background: #182028; color: #cde; border: 1px solid #2c3c5c; border-radius: 4px; }
|
||||
.dev-btn:hover { background: #20303f; }
|
||||
.dev-dl { margin: 0; display: grid; grid-template-columns: auto 1fr; gap: 0.15rem 0.6rem; font-size: 12px; }
|
||||
.dev-dl dt { color: #789; }
|
||||
.dev-dl dd { margin: 0; color: #dde; word-break: break-word; }
|
||||
.dev-dl dd.mem { color: #4e9; }
|
||||
.dev-dl dd.net { color: #fc6; }
|
||||
.dev-anno { font-size: 11px; max-height: 220px; overflow: auto;
|
||||
font-family: ui-monospace, monospace; }
|
||||
.dev-anno .anno-row { padding: 0.2rem 0; border-bottom: 1px solid #1d1d1d; }
|
||||
.dev-anno .anno-key { color: #aee6ff; }
|
||||
.dev-anno .anno-key.affect { color: #e3cfff; }
|
||||
.dev-anno .anno-key.measure { color: #ffd79a; }
|
||||
.dev-anno .anno-meta { color: #678; }
|
||||
.dev-anno .anno-empty { color: #567; font-style: italic; }
|
||||
|
||||
@@ -4,7 +4,9 @@ import pytest
|
||||
|
||||
from tools.pipeline.run import probe_duration, process_clip, resolve_ffmpeg
|
||||
|
||||
_FOREST = Path("simulator/sample_media/forest/base.mp4")
|
||||
# A real pool base (forest was retired with the rotating-pool rename, session 0016);
|
||||
# cosmos/base.mp4 is the cosmos pool primary and the most stable PD clip on disk.
|
||||
_BASE = Path("simulator/sample_media/cosmos/base.mp4")
|
||||
|
||||
|
||||
def _ffmpeg_available() -> bool:
|
||||
@@ -18,20 +20,18 @@ def _ffmpeg_available() -> bool:
|
||||
_HAS_FF = _ffmpeg_available()
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _FOREST.exists(),
|
||||
reason="forest POC base absent (run simulator/setup_sample_media.py)")
|
||||
def test_probe_duration_reads_forest_base():
|
||||
if not _FOREST.exists():
|
||||
pytest.skip("forest POC base absent")
|
||||
assert probe_duration(_FOREST) > 0
|
||||
@pytest.mark.skipif(not _BASE.exists(),
|
||||
reason="cosmos base absent (run simulator/build_pool_manifest.py)")
|
||||
def test_probe_duration_reads_real_base():
|
||||
assert probe_duration(_BASE) > 0
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _HAS_FF,
|
||||
reason="neither system ffmpeg nor imageio-ffmpeg available")
|
||||
@pytest.mark.skipif(not _FOREST.exists(),
|
||||
reason="forest POC base absent (run simulator/setup_sample_media.py)")
|
||||
@pytest.mark.skipif(not _BASE.exists(),
|
||||
reason="cosmos base absent (run simulator/build_pool_manifest.py)")
|
||||
def test_process_clip_emits_proxy_and_master(tmp_path):
|
||||
out = process_clip(_FOREST, tmp_path, overlap=0.5, start=0.0, duration=4.0)
|
||||
out = process_clip(_BASE, tmp_path, overlap=0.5, start=0.0, duration=4.0)
|
||||
assert out["proxy"].exists() and out["master"].exists()
|
||||
# Proxy is exactly 1920×1080 — verified with cv2, no ffprobe dependency.
|
||||
import cv2
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
"""tools/pipeline/track.py — the hybrid motion-track pass (content-pipeline §11.5).
|
||||
|
||||
Pure geometry helpers are unit-tested here; the cv2 optical-flow propagation is an
|
||||
opt-in integration test that runs on a real pool clip when present."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tools.pipeline.track import (
|
||||
clamp01_box,
|
||||
denormalize_box,
|
||||
infer_window,
|
||||
loop_t,
|
||||
normalize_box,
|
||||
sample_track,
|
||||
track_to_annotation,
|
||||
)
|
||||
|
||||
|
||||
# --- pure helpers -----------------------------------------------------------
|
||||
|
||||
|
||||
def test_clamp01_box_keeps_box_on_screen():
|
||||
assert clamp01_box((-0.2, 0.1, 0.5, 0.5)) == [0.0, 0.1, 0.5, 0.5]
|
||||
# size trimmed so the box never runs past the right/bottom edge
|
||||
assert clamp01_box((0.8, 0.8, 0.5, 0.5)) == [0.8, 0.8, pytest.approx(0.2), pytest.approx(0.2)]
|
||||
|
||||
|
||||
def test_normalize_denormalize_round_trip():
|
||||
px = (192, 108, 384, 216)
|
||||
norm = normalize_box(px, 1920, 1080)
|
||||
assert norm == [pytest.approx(0.1), pytest.approx(0.1), pytest.approx(0.2), pytest.approx(0.2)]
|
||||
assert denormalize_box(norm, 1920, 1080) == (192, 108, 384, 216)
|
||||
|
||||
|
||||
def test_loop_t_is_frame_over_total():
|
||||
assert loop_t(0, 100) == 0.0
|
||||
assert loop_t(50, 100) == 0.5
|
||||
assert loop_t(5, 0) == 0.0 # guard divide-by-zero
|
||||
|
||||
|
||||
def test_infer_window_spans_tracked_frames():
|
||||
appear, disappear = infer_window([10, 11, 30, 31], 100)
|
||||
assert appear == pytest.approx(0.1)
|
||||
assert disappear == pytest.approx(0.31)
|
||||
# no frames -> full clip
|
||||
assert infer_window([], 100) == (0.0, 1.0)
|
||||
|
||||
|
||||
def test_sample_track_downsamples_to_sparse_keyframes():
|
||||
# 21 dense frames -> at most n_keyframes, including the first and last
|
||||
per_frame = {i: [i / 100, 0.2, 0.1, 0.1] for i in range(0, 21)}
|
||||
track = sample_track(per_frame, total_frames=100, n_keyframes=5)
|
||||
assert len(track) <= 5
|
||||
ts = [k["t"] for k in track]
|
||||
assert ts == sorted(ts) # sorted by t
|
||||
assert ts[0] == pytest.approx(0.0) # first tracked frame
|
||||
assert ts[-1] == pytest.approx(0.2) # last tracked frame (frame 20 / 100)
|
||||
assert len(set(ts)) == len(ts) # de-duplicated on t
|
||||
|
||||
|
||||
def test_sample_track_keeps_all_when_already_sparse():
|
||||
per_frame = {0: [0.1, 0.1, 0.1, 0.1], 60: [0.5, 0.3, 0.1, 0.1]}
|
||||
track = sample_track(per_frame, total_frames=120, n_keyframes=5)
|
||||
assert [k["t"] for k in track] == [pytest.approx(0.0), pytest.approx(0.5)]
|
||||
|
||||
|
||||
def test_sample_track_empty_is_empty():
|
||||
assert sample_track({}, total_frames=100) == []
|
||||
|
||||
|
||||
def test_track_to_annotation_assembles_geometry_only():
|
||||
track = [{"t": 0.0, "box": [0.1, 0.2, 0.1, 0.1]}, {"t": 0.5, "box": [0.4, 0.2, 0.1, 0.1]}]
|
||||
ann = track_to_annotation("detected.jelly", track, salience=3, appear=0.0, disappear=0.55)
|
||||
assert ann["key"] == "detected.jelly"
|
||||
assert ann["salience"] == 3
|
||||
assert ann["track"] == track
|
||||
assert ann["appear"] == 0.0 and ann["disappear"] == 0.55
|
||||
# no semantics leaked in — strings/tiers are the author's, added elsewhere
|
||||
assert "strings" not in ann and "_tiers" not in ann
|
||||
|
||||
|
||||
def test_track_to_annotation_omits_window_when_absent():
|
||||
ann = track_to_annotation("detected.x", [], salience=4)
|
||||
assert "appear" not in ann and "disappear" not in ann
|
||||
|
||||
|
||||
# --- opt-in: classical optical-flow propagation on a real clip --------------
|
||||
|
||||
_CLIP = Path("simulator/sample_media/abyss_wow/base.mp4")
|
||||
|
||||
|
||||
def _cv2_available() -> bool:
|
||||
try:
|
||||
import cv2 # noqa: F401
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _CLIP.exists(),
|
||||
reason="abyss_wow base absent (run simulator/build_pool_manifest.py)")
|
||||
@pytest.mark.skipif(not _cv2_available(), reason="opencv-python not available")
|
||||
def test_track_seed_produces_a_sparse_annotation_on_real_footage():
|
||||
from tools.pipeline.track import track_box, track_seed
|
||||
|
||||
seed = (0.2, 0.3, 0.15, 0.18)
|
||||
per_frame, total = track_box(_CLIP, seed, seed_frame=0, max_frames=40)
|
||||
assert total > 0
|
||||
assert per_frame[0] == pytest.approx(list(seed), abs=0.02)
|
||||
assert len(per_frame) > 1 # propagated beyond the seed frame
|
||||
|
||||
ann = track_seed(_CLIP, "detected.jelly", seed, seed_frame=0, n_keyframes=5, max_frames=40)
|
||||
assert ann["key"] == "detected.jelly"
|
||||
assert 2 <= len(ann["track"]) <= 5 # sparse keyframes
|
||||
for kf in ann["track"]:
|
||||
assert 0.0 <= kf["t"] <= 1.0
|
||||
assert all(0.0 <= v <= 1.0 for v in kf["box"])
|
||||
assert 0.0 <= ann["appear"] <= ann["disappear"] <= 1.0
|
||||
@@ -91,6 +91,35 @@ def test_index_is_served():
|
||||
assert "Alteration" in resp.text
|
||||
|
||||
|
||||
def test_app_assets_are_not_cached(client):
|
||||
# Dev iteration relies on the browser never serving stale renderer code.
|
||||
resp = client.get("/")
|
||||
assert "no-store" in resp.headers.get("cache-control", "")
|
||||
|
||||
|
||||
def test_media_is_cacheable(tmp_path, monkeypatch):
|
||||
# Media is static: it must be browser-cacheable so the preloader's fetch (and
|
||||
# any fallback) doesn't re-hit the network on every altitude change.
|
||||
import simulator.app as appmod
|
||||
|
||||
media = tmp_path / "media"
|
||||
(media / "cosmos").mkdir(parents=True)
|
||||
(media / "cosmos" / "base.mp4").write_bytes(b"\x00\x00\x00\x18ftyp")
|
||||
monkeypatch.setattr(appmod, "MEDIA_DIR", media)
|
||||
manifest = tmp_path / "manifest.json"
|
||||
manifest.write_text(json.dumps({"clips": []}))
|
||||
|
||||
client = TestClient(create_app(manifest_path=manifest))
|
||||
resp = client.get("/media/cosmos/base.mp4")
|
||||
assert resp.status_code == 200
|
||||
cc = resp.headers.get("cache-control", "")
|
||||
# Stored but revalidated: not no-store (so the blob preload can cache it), and
|
||||
# not pinned immutable (so a re-sourced/re-cropped clip shows on a plain reload).
|
||||
assert "no-store" not in cc
|
||||
assert "immutable" not in cc
|
||||
assert "no-cache" in cc
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def ring_manifest_path(tmp_path):
|
||||
p = tmp_path / "manifest.json"
|
||||
@@ -250,3 +279,78 @@ def test_delta_zero_is_an_initial_pool_pick(pool_client):
|
||||
assert data["to_index"] == 1
|
||||
assert data["steps"] == []
|
||||
assert data["target_clip_id"] in members
|
||||
|
||||
|
||||
# --- author mode (content-pipeline §11.5) ---
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def author_client(tmp_path):
|
||||
p = tmp_path / "manifest.json"
|
||||
p.write_text(json.dumps({
|
||||
"clips": [{
|
||||
"id": "abyss_wow", "title": "World of Water", "base_file": "abyss_wow/base.mp4",
|
||||
"license": "PD", "source": "NOAA", "right_variants": {},
|
||||
"annotations": [], "affect": [], "strings": {"en": {}},
|
||||
}],
|
||||
}))
|
||||
return TestClient(create_app(manifest_path=p)), p
|
||||
|
||||
|
||||
def test_author_clip_persists_labels_and_keeps_provenance(author_client):
|
||||
client, path = author_client
|
||||
body = {
|
||||
"clip_id": "abyss_wow",
|
||||
"annotations": [{"key": "detected.jelly", "salience": 4,
|
||||
"appear": 0.0, "disappear": 0.5,
|
||||
"track": [{"t": 0.0, "box": [0.1, 0.2, 0.1, 0.1]}]}],
|
||||
"affect": [{"key": "feel.unease", "at": [0.5, 0.5], "min_level": 1}],
|
||||
"strings": {"en": {"detected.jelly": ["jelly", "comb jelly", "Ctenophora", "Ctenophora · cilia rows"],
|
||||
"feel.unease": ["uh", "unease", "disquiet", "a creeping disquiet"]}},
|
||||
}
|
||||
resp = client.post("/api/author/clip", json=body)
|
||||
assert resp.status_code == 200 and resp.json()["ok"] is True
|
||||
# persisted to disk
|
||||
saved = json.loads(path.read_text())["clips"][0]
|
||||
assert saved["annotations"][0]["key"] == "detected.jelly"
|
||||
assert saved["affect"][0]["key"] == "feel.unease"
|
||||
assert saved["strings"]["en"]["detected.jelly"][2] == "Ctenophora"
|
||||
# provenance preserved (only authored content replaced)
|
||||
assert saved["base_file"] == "abyss_wow/base.mp4"
|
||||
assert saved["license"] == "PD"
|
||||
# reflected by /api/clips without a restart
|
||||
served = client.get("/api/clips").json()["clips"][0]
|
||||
assert served["annotations"][0]["key"] == "detected.jelly"
|
||||
|
||||
|
||||
def test_author_clip_unknown_clip_404(author_client):
|
||||
client, _ = author_client
|
||||
resp = client.post("/api/author/clip", json={"clip_id": "nope", "annotations": []})
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
def test_author_track_unknown_clip_404(author_client):
|
||||
client, _ = author_client
|
||||
resp = client.post("/api/author/track", json={
|
||||
"clip_id": "nope", "key": "detected.x", "seed_box": [0.1, 0.1, 0.1, 0.1]})
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
_REAL_CLIP = __import__("pathlib").Path("simulator/sample_media/abyss_wow/base.mp4")
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _REAL_CLIP.exists(),
|
||||
reason="abyss_wow base absent (run simulator/build_pool_manifest.py)")
|
||||
def test_author_track_on_real_clip_returns_keyframes():
|
||||
# Against the real default manifest + footage: the tracker yields a sparse track.
|
||||
client = TestClient(create_app())
|
||||
resp = client.post("/api/author/track", json={
|
||||
"clip_id": "abyss_wow", "key": "detected.jelly",
|
||||
"seed_box": [0.2, 0.3, 0.15, 0.18], "seed_t": 0.0,
|
||||
"salience": 4, "n_keyframes": 5, "max_frames": 30,
|
||||
})
|
||||
assert resp.status_code == 200
|
||||
ann = resp.json()
|
||||
assert ann["key"] == "detected.jelly"
|
||||
assert 2 <= len(ann["track"]) <= 5
|
||||
assert 0.0 <= ann["appear"] <= ann["disappear"] <= 1.0
|
||||
|
||||
@@ -0,0 +1,297 @@
|
||||
"""Stage 5 geometry — the hybrid motion-track pass (content-pipeline §3.5 / §11.5).
|
||||
|
||||
Turns a hand-seeded box (or an ML detection) on one keyframe into the sparse,
|
||||
loop-normalized keyframed `track` the manifest stores (design §4) plus an
|
||||
appear/disappear window (§11.2). SEMANTICS are never produced here — the author
|
||||
owns the `key`, strings, `salience` and tiers; this module only propagates BOX
|
||||
GEOMETRY.
|
||||
|
||||
Two geometry paths, same artifact:
|
||||
- **classical** (always available): a hand-seeded box propagated by OpenCV
|
||||
Lucas-Kanade optical flow (a CSRT tracker is used instead when a contrib
|
||||
build provides one). Translation model — right for calm drifting subjects
|
||||
(the abyss/reef creatures). Re-seeds features when too few survive.
|
||||
- **ML detect+track** (optional, lazy-imported): a YOLO+tracker pass proposing
|
||||
boxes the author maps to a key. Imported only on demand so the base install
|
||||
needs no torch/ultralytics; a clear ImportError tells you what to `pip
|
||||
install` if you ask for it without it present.
|
||||
|
||||
The PURE helpers (normalization, keyframe sampling, window inference, annotation
|
||||
assembly) are unit-tested; the cv2/ML propagation is covered by the opt-in
|
||||
integration test on real footage.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Optional
|
||||
|
||||
Box = tuple[float, float, float, float] # (x, y, w, h)
|
||||
|
||||
|
||||
# --- pure geometry helpers --------------------------------------------------
|
||||
|
||||
def clamp01_box(box: Box) -> list[float]:
|
||||
"""Clamp a normalized box into the visible [0,1] frame, keeping it on-screen
|
||||
(origin clamped, then size trimmed so x+w<=1, y+h<=1)."""
|
||||
x, y, w, h = box
|
||||
x = min(max(x, 0.0), 1.0)
|
||||
y = min(max(y, 0.0), 1.0)
|
||||
w = min(max(w, 0.0), 1.0 - x)
|
||||
h = min(max(h, 0.0), 1.0 - y)
|
||||
return [x, y, w, h]
|
||||
|
||||
|
||||
def normalize_box(box_px: Box, width: int, height: int) -> list[float]:
|
||||
"""Pixel box -> normalized [0,1] box (the manifest's resolution-independent
|
||||
form). Clamped on-screen."""
|
||||
x, y, w, h = box_px
|
||||
return clamp01_box((x / width, y / height, w / width, h / height))
|
||||
|
||||
|
||||
def denormalize_box(box_norm: Box, width: int, height: int) -> tuple[int, int, int, int]:
|
||||
"""Normalized [0,1] box -> integer pixel box for the cv2 tracker."""
|
||||
x, y, w, h = box_norm
|
||||
return (round(x * width), round(y * height), round(w * width), round(h * height))
|
||||
|
||||
|
||||
def loop_t(frame_idx: int, total_frames: int) -> float:
|
||||
"""Loop-normalized time of a frame in [0,1), matching the client's
|
||||
`currentTime / duration` clock (design §4)."""
|
||||
if total_frames <= 0:
|
||||
return 0.0
|
||||
return frame_idx / total_frames
|
||||
|
||||
|
||||
def infer_window(tracked_frames: list[int], total_frames: int) -> tuple[float, float]:
|
||||
"""The appear/disappear window (§11.2) covering the frames where a box was
|
||||
tracked, as loop-normalized t. The first/last tracked frame bound it."""
|
||||
if not tracked_frames:
|
||||
return (0.0, 1.0)
|
||||
lo, hi = min(tracked_frames), max(tracked_frames)
|
||||
return (loop_t(lo, total_frames), loop_t(hi, total_frames))
|
||||
|
||||
|
||||
def sample_track(
|
||||
per_frame: dict[int, Box], total_frames: int, n_keyframes: int = 5
|
||||
) -> list[dict]:
|
||||
"""Down-sample a dense per-frame box map to a SPARSE keyframed track
|
||||
(design §4: sparse keyframes interpolated at runtime, never dense per-frame).
|
||||
|
||||
Picks `n_keyframes` frames evenly across the tracked span (always including
|
||||
the first and last tracked frame) and emits `[{t, box}, ...]` sorted by t,
|
||||
de-duplicated on t. Pure — takes already-normalized boxes."""
|
||||
if not per_frame:
|
||||
return []
|
||||
frames = sorted(per_frame)
|
||||
n = max(2, n_keyframes)
|
||||
if len(frames) <= n:
|
||||
picks = frames
|
||||
else:
|
||||
lo, hi = frames[0], frames[-1]
|
||||
step = (hi - lo) / (n - 1)
|
||||
targets = [lo + round(step * i) for i in range(n)]
|
||||
# snap each target to the nearest actually-tracked frame
|
||||
picks = sorted({min(frames, key=lambda f: abs(f - t)) for t in targets})
|
||||
out, seen = [], set()
|
||||
for f in picks:
|
||||
t = round(loop_t(f, total_frames), 4)
|
||||
if t in seen:
|
||||
continue
|
||||
seen.add(t)
|
||||
out.append({"t": t, "box": clamp01_box(per_frame[f])})
|
||||
return out
|
||||
|
||||
|
||||
def track_to_annotation(
|
||||
key: str,
|
||||
track: list[dict],
|
||||
*,
|
||||
salience: int = 4,
|
||||
appear: Optional[float] = None,
|
||||
disappear: Optional[float] = None,
|
||||
) -> dict:
|
||||
"""Assemble a manifest annotation from a propagated track + the author's
|
||||
semantics. Geometry (track + window) comes from this module; `key`,
|
||||
`salience` and the tiered strings are the author's (added separately)."""
|
||||
ann: dict = {"key": key, "salience": salience, "track": track}
|
||||
if appear is not None:
|
||||
ann["appear"] = round(appear, 4)
|
||||
if disappear is not None:
|
||||
ann["disappear"] = round(disappear, 4)
|
||||
return ann
|
||||
|
||||
|
||||
# --- classical propagation (cv2; covered by the opt-in integration test) -----
|
||||
|
||||
def _new_csrt():
|
||||
"""A CSRT tracker if this OpenCV build ships one (contrib), else None — then
|
||||
the optical-flow path is used. Kept tiny so the import stays lazy."""
|
||||
import cv2
|
||||
for factory in ("TrackerCSRT_create",):
|
||||
if hasattr(cv2, factory):
|
||||
return getattr(cv2, factory)()
|
||||
legacy = getattr(cv2, "legacy", None)
|
||||
if legacy is not None and hasattr(legacy, "TrackerCSRT_create"):
|
||||
return legacy.TrackerCSRT_create()
|
||||
return None
|
||||
|
||||
|
||||
def _features_in_box(gray, box_px, max_corners: int = 60):
|
||||
"""Good-feature points inside a pixel box, in full-frame coords (for LK)."""
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
x, y, w, h = (int(v) for v in box_px)
|
||||
h_img, w_img = gray.shape[:2]
|
||||
x0, y0 = max(x, 0), max(y, 0)
|
||||
x1, y1 = min(x + w, w_img), min(y + h, h_img)
|
||||
if x1 - x0 < 4 or y1 - y0 < 4:
|
||||
return None
|
||||
roi = gray[y0:y1, x0:x1]
|
||||
pts = cv2.goodFeaturesToTrack(roi, max_corners, 0.01, 5)
|
||||
if pts is None:
|
||||
return None
|
||||
pts = pts.reshape(-1, 2) + np.array([x0, y0], dtype="float32")
|
||||
return pts.reshape(-1, 1, 2)
|
||||
|
||||
|
||||
def track_box(
|
||||
video_path,
|
||||
seed_box_norm: Box,
|
||||
*,
|
||||
seed_frame: int = 0,
|
||||
max_frames: Optional[int] = None,
|
||||
min_features: int = 8,
|
||||
) -> tuple[dict[int, list[float]], int]:
|
||||
"""Propagate a hand-seeded normalized box FORWARD from `seed_frame` with
|
||||
optical flow (or CSRT when available). Returns `(per_frame_norm_boxes,
|
||||
total_frames)` — feed `per_frame` to `sample_track`. Translation model: the
|
||||
box follows the median displacement of features inside it, re-seeding when too
|
||||
few survive. Impure (reads the video) — opt-in integration test."""
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
cap = cv2.VideoCapture(str(video_path))
|
||||
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
|
||||
if seed_frame:
|
||||
cap.set(cv2.CAP_PROP_POS_FRAMES, seed_frame)
|
||||
ok, frame = cap.read()
|
||||
if not ok:
|
||||
cap.release()
|
||||
raise ValueError(f"could not read frame {seed_frame} of {video_path}")
|
||||
h_img, w_img = frame.shape[:2]
|
||||
box = list(denormalize_box(seed_box_norm, w_img, h_img)) # px [x,y,w,h]
|
||||
|
||||
csrt = _new_csrt()
|
||||
if csrt is not None:
|
||||
csrt.init(frame, tuple(int(v) for v in box))
|
||||
|
||||
prev_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
|
||||
pts = None if csrt is not None else _features_in_box(prev_gray, box)
|
||||
out: dict[int, list[float]] = {seed_frame: normalize_box(box, w_img, h_img)}
|
||||
|
||||
fidx = seed_frame
|
||||
while True:
|
||||
ok, frame = cap.read()
|
||||
if not ok:
|
||||
break
|
||||
fidx += 1
|
||||
if csrt is not None:
|
||||
ok2, b = csrt.update(frame)
|
||||
if ok2:
|
||||
box = [b[0], b[1], b[2], b[3]]
|
||||
else:
|
||||
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
|
||||
if pts is None or len(pts) < min_features:
|
||||
pts = _features_in_box(prev_gray, box)
|
||||
if pts is not None and len(pts):
|
||||
nxt, status, _ = cv2.calcOpticalFlowPyrLK(prev_gray, gray, pts, None)
|
||||
if nxt is not None:
|
||||
good_old = pts[status == 1]
|
||||
good_new = nxt[status == 1]
|
||||
if len(good_new) >= 3:
|
||||
dx = float(np.median(good_new[:, 0] - good_old[:, 0]))
|
||||
dy = float(np.median(good_new[:, 1] - good_old[:, 1]))
|
||||
box[0] += dx
|
||||
box[1] += dy
|
||||
pts = good_new.reshape(-1, 1, 2)
|
||||
else:
|
||||
pts = None
|
||||
prev_gray = gray
|
||||
out[fidx] = normalize_box(box, w_img, h_img)
|
||||
if max_frames is not None and fidx - seed_frame >= max_frames:
|
||||
break
|
||||
cap.release()
|
||||
return out, total
|
||||
|
||||
|
||||
def track_seed(
|
||||
video_path,
|
||||
key: str,
|
||||
seed_box_norm: Box,
|
||||
*,
|
||||
seed_frame: int = 0,
|
||||
salience: int = 4,
|
||||
n_keyframes: int = 5,
|
||||
max_frames: Optional[int] = None,
|
||||
) -> dict:
|
||||
"""End-to-end classical pass: seed -> propagate -> sample -> annotation
|
||||
(geometry only; the author adds strings/tiers). Impure (reads the video)."""
|
||||
per_frame, total = track_box(
|
||||
video_path, seed_box_norm, seed_frame=seed_frame, max_frames=max_frames
|
||||
)
|
||||
track = sample_track(per_frame, total, n_keyframes=n_keyframes)
|
||||
appear, disappear = infer_window(list(per_frame), total)
|
||||
return track_to_annotation(
|
||||
key, track, salience=salience, appear=appear, disappear=disappear
|
||||
)
|
||||
|
||||
|
||||
# --- optional ML detect+track (lazy-imported) --------------------------------
|
||||
|
||||
def detect_and_track(video_path, *, model: str = "yolov8n.pt", classes=None) -> list[dict]:
|
||||
"""Optional ML path (content-pipeline §3.5): run a detect+track model and
|
||||
return candidate tracks `[{track_id, label, salience, track:[{t,box}],
|
||||
appear, disappear}]` for the author to map to keys. Heavy deps are
|
||||
lazy-imported so the base install stays light.
|
||||
|
||||
Raises a clear ImportError naming the install if `ultralytics` is absent —
|
||||
this path is opt-in by design; the classical `track_seed` needs only cv2."""
|
||||
try:
|
||||
from ultralytics import YOLO # noqa: F401
|
||||
except ImportError as e: # pragma: no cover - exercised only without the dep
|
||||
raise ImportError(
|
||||
"the ML detect+track path needs ultralytics — "
|
||||
"`pip install ultralytics` (optional; the classical track_seed path "
|
||||
"uses only opencv-python)."
|
||||
) from e
|
||||
import cv2 # noqa: F401 (pragma: no cover below — needs the real model)
|
||||
|
||||
yolo = YOLO(model) # pragma: no cover
|
||||
cap = cv2.VideoCapture(str(video_path)) # pragma: no cover
|
||||
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0) # pragma: no cover
|
||||
cap.release() # pragma: no cover
|
||||
per_track: dict[int, dict[int, list[float]]] = {} # pragma: no cover
|
||||
labels: dict[int, str] = {} # pragma: no cover
|
||||
for fidx, res in enumerate( # pragma: no cover
|
||||
yolo.track(source=str(video_path), persist=True, classes=classes, stream=True)
|
||||
):
|
||||
if res.boxes is None or res.boxes.id is None: # pragma: no cover
|
||||
continue
|
||||
w_img, h_img = res.orig_shape[1], res.orig_shape[0] # pragma: no cover
|
||||
for box, tid, cls in zip( # pragma: no cover
|
||||
res.boxes.xywh.tolist(), res.boxes.id.tolist(), res.boxes.cls.tolist()
|
||||
):
|
||||
cx, cy, bw, bh = box # pragma: no cover
|
||||
nb = normalize_box((cx - bw / 2, cy - bh / 2, bw, bh), w_img, h_img)
|
||||
per_track.setdefault(int(tid), {})[fidx] = nb # pragma: no cover
|
||||
labels[int(tid)] = yolo.names.get(int(cls), str(cls))
|
||||
out = [] # pragma: no cover
|
||||
for tid, frames in per_track.items(): # pragma: no cover
|
||||
track = sample_track(frames, total) # pragma: no cover
|
||||
appear, disappear = infer_window(list(frames), total) # pragma: no cover
|
||||
out.append({ # pragma: no cover
|
||||
"track_id": tid, "label": labels.get(tid, ""), "salience": 4,
|
||||
"track": track, "appear": round(appear, 4), "disappear": round(disappear, 4),
|
||||
})
|
||||
return out # pragma: no cover
|
||||
Reference in New Issue
Block a user