diff --git a/docs/superpowers/plans/2026-06-04-ingest-tagging-review-tools.md b/docs/superpowers/plans/2026-06-04-ingest-tagging-review-tools.md new file mode 100644 index 0000000..0fece86 --- /dev/null +++ b/docs/superpowers/plans/2026-06-04-ingest-tagging-review-tools.md @@ -0,0 +1,500 @@ +# Ingest & Tagging / Review Tools — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the `tools/` package that turns the hand-authored catalog into the assisted *draft-then-review* pipeline: per-archive ingest fetchers, mechanical tagging (ffprobe/ffmpeg + origin license), heuristic coordinate drafting (`review_status=proposed` + one-line rationale), and an interactive review CLI that flips records to `approved` with a `reviewed_at` stamp. + +**Spec:** [`2026-06-04-ingest-tagging-review-tools.md`](../specs/2026-06-04-ingest-tagging-review-tools.md). Read it first — this plan implements it task-by-task and inherits its decisions (esp. §3 the single required `hef.catalog` addition, §5.2 `dominant_color` is optional/opt-in, §6.4 first-ship = LibriVox + NASA + Internet Archive). + +**Architecture:** A `tools/` package alongside the existing `hef/` (which it imports, never forks). The two external boundaries — `ffmpeg`/`ffprobe` (subprocess) and archive HTTP — are isolated behind injectable seams (a `Prober`, a `download` callable, an HTTP client) so the unit suite stays **hermetic** (no network, no binaries). Integration tests that actually invoke `ffprobe`/`ffmpeg` are opt-in and skipped when the binaries are absent. + +**Tech Stack:** Python 3.11+, stdlib only in the testable core (`dataclasses`, `json`, `subprocess`, `urllib`, `pathlib`, `datetime`); `ffmpeg`/`ffprobe` system binaries; pytest. No `requests`, no image library (dominant color, when computed, is ffmpeg-only). + +**Where this sits:** sub-project **2 of 5**. Depends on sub-project 1 (`hef.catalog`, `hef.selection`, built). Feeds sub-project 3 (the Pi player). Sub-project 5 (side walls) was dropped (design §7 revision), so `dominant_color` is optional here. + +--- + +## File Structure + +``` +pyproject.toml + add `tools` to packages; optional [project.scripts] +.gitignore + media/ +hef/catalog.py + validate_catalog(), index_by_id() (ONLY hef change) +tools/__init__.py package marker +tools/http.py stdlib-urllib client (timeout/retry/user-agent), injectable +tools/probe.py Prober: ffprobe wrapper -> parsed Probe result +tools/tagging.py mode/duration_s/resolution from a Probe (+ cover-art guard) +tools/mediatools.py ffmpeg: representative frame; optional dominant_color +tools/licensing.py per-archive license/attribution -> hef LICENSES vocab +tools/drafting.py Signals, Draft, Proposer, HeuristicProposer +tools/ingest/__init__.py +tools/ingest/base.py Candidate, Fetcher, ingest_candidate() pipeline +tools/ingest/librivox.py first-ship fetcher +tools/ingest/nasa.py first-ship fetcher +tools/ingest/internet_archive.py first-ship fetcher +tools/ingest/musopen.py deferred: docstring stub raising NotImplementedError +tools/ingest/fma.py deferred: docstring stub +tools/ingest/freesound.py deferred: docstring stub (needs FREESOUND_API_TOKEN) +tools/ingest_cli.py `python -m tools.ingest_cli` +tools/review.py proposed_records(), approve() (pure transition core) +tools/review_cli.py `python -m tools.review_cli` (interactive walk) +tests/test_catalog_integrity.py validate_catalog / index_by_id +tests/test_probe.py ffprobe JSON parsing (canned) +tests/test_tagging.py mode/duration/resolution + cover-art guard +tests/test_mediatools.py dominant_color from canned rgb bytes; opt-in default off +tests/test_licensing.py per-archive license normalization +tests/test_drafting.py HeuristicProposer +tests/test_fetchers.py librivox/nasa/internet_archive via fake HTTP client +tests/test_ingest_pipeline.py ingest_candidate end-to-end with fakes +tests/test_review.py proposed_records / approve transition +tests/test_tools_integration.py end-to-end mocked; opt-in real-ffprobe (skipped if absent) +``` + +`tools/` holds everything new; `hef/` gains only the two additive symbols of spec §3. + +--- + +### Task 1: Scaffolding — `tools/` package, deps, http client, smoke test + +**Files:** `pyproject.toml`, `.gitignore`, `tools/__init__.py`, `tools/http.py`, `tests/test_tools_smoke.py` + +- [ ] **Step 1: Add `tools` to packaging.** In `pyproject.toml`, change `packages = ["hef"]` → `packages = ["hef", "tools"]`. (Optional: add `[project.scripts]` `hef-ingest = "tools.ingest_cli:main"` and `hef-review = "tools.review_cli:main"` for installed checkouts; `python -m` is the documented path either way.) + +- [ ] **Step 2: gitignore media.** Append `media/` to `.gitignore`. + +- [ ] **Step 3: Write the failing smoke test** `tests/test_tools_smoke.py`: +```python +def test_tools_package_imports(): + import tools + import tools.http + assert tools is not None +``` +Run `python -m pytest tests/test_tools_smoke.py -v` → FAIL (`ModuleNotFoundError: tools`). + +- [ ] **Step 4: Create `tools/__init__.py`** (empty) and `tools/http.py` — a tiny stdlib client: +```python +"""Minimal HTTP client over urllib: timeout, small retry, user-agent. Injectable.""" +from __future__ import annotations +import json as _json +import time +import urllib.request + +USER_AGENT = "human-experience-filter-ingest/0.1 (+local art installation)" + +class HttpClient: + def __init__(self, *, timeout=30.0, retries=2, opener=None): + self.timeout, self.retries = timeout, retries + self._opener = opener or urllib.request.urlopen + + def get_bytes(self, url, *, headers=None): + last = None + for attempt in range(self.retries + 1): + try: + req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT, **(headers or {})}) + with self._opener(req, timeout=self.timeout) as resp: + return resp.read() + except Exception as exc: # noqa: BLE001 - retried below + last = exc + if attempt < self.retries: + time.sleep(0.2 * (attempt + 1)) + raise last + + def get_json(self, url, *, headers=None): + return _json.loads(self.get_bytes(url, headers=headers).decode("utf-8")) +``` +The `opener` injection point lets tests pass a fake (no network). + +- [ ] **Step 5:** Run smoke test → PASS. Commit: +``` +git add pyproject.toml .gitignore tools/__init__.py tools/http.py tests/test_tools_smoke.py +git commit -m "chore: scaffold tools/ package + stdlib http client" +``` + +--- + +### Task 2: `hef.catalog` additions — `validate_catalog` + `index_by_id` + +The ONLY change to the shared spine (spec §3). Purely additive. + +**Files:** `hef/catalog.py`, `tests/test_catalog_integrity.py` + +- [ ] **Step 1: Failing tests** `tests/test_catalog_integrity.py` (reuse a local `make_record` like `tests/test_catalog.py`): +```python +import pytest +from hef.catalog import Record, validate_catalog, index_by_id, CatalogError + +def make_record(**o): + base = dict(id="a", title="t", source_url="u", source_archive="nasa", + license="public_domain", mode="video", left=0, right=0, dark=0, + light=0, duration_s=1, file_path="p") + base.update(o); return Record(**base) + +def test_validate_catalog_accepts_unique_ids(): + validate_catalog([make_record(id="a"), make_record(id="b")]) # no raise + +def test_validate_catalog_rejects_duplicate_id(): + with pytest.raises(CatalogError) as e: + validate_catalog([make_record(id="dup"), make_record(id="dup")]) + assert "dup" in str(e.value) + +def test_validate_catalog_validates_each_record(): + with pytest.raises(CatalogError): + validate_catalog([make_record(left=9)]) + +def test_index_by_id_round_trips(): + recs = [make_record(id="a"), make_record(id="b")] + idx = index_by_id(recs) + assert idx["a"].id == "a" and idx["b"].id == "b" + +def test_index_by_id_rejects_duplicates(): + with pytest.raises(CatalogError): + index_by_id([make_record(id="x"), make_record(id="x")]) +``` +Run → FAIL (ImportError). + +- [ ] **Step 2: Implement** — append to `hef/catalog.py` (do NOT touch existing functions): +```python +def index_by_id(records) -> dict: + """Map id -> Record, raising CatalogError on a duplicate id.""" + idx: dict = {} + for r in records: + if r.id in idx: + raise CatalogError(f"duplicate record id: {r.id!r}") + idx[r.id] = r + return idx + + +def validate_catalog(records) -> None: + """Validate every record AND cross-record invariants (currently: unique ids).""" + for r in records: + validate(r) + index_by_id(records) # raises on duplicate id +``` + +- [ ] **Step 3:** Run `python -m pytest tests/test_catalog_integrity.py -v` → PASS. Run full suite `python -m pytest -q` → still green (37 prior + new). Commit: +``` +git add hef/catalog.py tests/test_catalog_integrity.py +git commit -m "feat: catalog-level validate_catalog + index_by_id (unique ids)" +``` + +--- + +### Task 3: `tools/probe.py` — ffprobe wrapper + +**Files:** `tools/probe.py`, `tests/test_probe.py` + +- [ ] **Step 1: Failing tests** — feed canned `ffprobe` JSON via an injected runner; assert a parsed `Probe` (list of streams + format dict). Cover: a video+audio file, an audio-only file, and an audio file with an `attached_pic` cover stream. Example: +```python +import json +from tools.probe import Probe, probe_file + +def fake_runner(args): # mimics subprocess.run(...).stdout + return json.dumps({ + "streams": [ + {"codec_type": "video", "width": 1920, "height": 1080, "disposition": {"attached_pic": 0}}, + {"codec_type": "audio"}, + ], + "format": {"duration": "12.5"}, + }) + +def test_probe_parses_streams_and_format(): + p = probe_file("x.mp4", runner=fake_runner) + assert any(s["codec_type"] == "video" for s in p.streams) + assert p.format["duration"] == "12.5" +``` + +- [ ] **Step 2: Implement** `tools/probe.py`: +```python +"""ffprobe wrapper -> parsed streams/format. Subprocess runner is injectable.""" +from __future__ import annotations +import json, subprocess +from dataclasses import dataclass + +@dataclass +class Probe: + streams: list + format: dict + +def _default_runner(args) -> str: + return subprocess.run(args, capture_output=True, text=True, check=True).stdout + +def probe_file(path, *, runner=_default_runner) -> Probe: + out = runner(["ffprobe", "-v", "quiet", "-print_format", "json", + "-show_format", "-show_streams", str(path)]) + data = json.loads(out) + return Probe(streams=data.get("streams", []), format=data.get("format", {})) +``` + +- [ ] **Step 3:** Run → PASS. Commit `feat: ffprobe wrapper (tools.probe)`. + +--- + +### Task 4: `tools/tagging.py` — mode / duration_s / resolution (with cover-art guard) + +**Files:** `tools/tagging.py`, `tests/test_tagging.py` + +- [ ] **Step 1: Failing tests** covering the spec §5.1 cases: + - video+audio → `mode="av"`, `resolution="1920x1080"`, `duration_s=13` (round 12.5). + - audio only → `mode="audio"`, `resolution=""`. + - **audio + `attached_pic` cover image → `mode="audio"`** (the guard), `resolution=""`. + - video only → `mode="video"`. + - duration fallback to longest stream when `format.duration` absent. +```python +from tools.probe import Probe +from tools.tagging import derive_tags + +def test_cover_art_stays_audio(): + p = Probe(streams=[ + {"codec_type": "audio", "duration": "30.0"}, + {"codec_type": "video", "width": 600, "height": 600, "disposition": {"attached_pic": 1}}, + ], format={"duration": "30.0"}) + t = derive_tags(p) + assert t.mode == "audio" and t.resolution == "" and t.duration_s == 30 +``` + +- [ ] **Step 2: Implement** — a `Tags(mode, duration_s, resolution)` dataclass and `derive_tags(probe)`: + - real video streams = `codec_type == "video"` AND `disposition.attached_pic != 1`. + - mode: both→`av`, video-only→`video`, else→`audio`. + - resolution from the first real video stream (`f"{w}x{h}"`), else `""`. + - duration: `round(float(format["duration"]))` else max stream duration else 0; never negative. + +- [ ] **Step 3:** Run → PASS. Commit `feat: mechanical mode/duration/resolution tagging`. + +--- + +### Task 5: `tools/mediatools.py` — representative frame + optional dominant_color + +**Files:** `tools/mediatools.py`, `tests/test_mediatools.py` + +- [ ] **Step 1: Failing tests** (hermetic — inject the ffmpeg runner returning canned bytes): + - `dominant_color_from_rgb(b"\xff\x00\x00") == "#ff0000"`. + - `compute_dominant_color(path, runner=fake)` returns the hex when enabled. + - The default ingest path does NOT call this (asserted in Task 8, not here). + +- [ ] **Step 2: Implement**: +```python +"""ffmpeg helpers: representative frame; OPTIONAL dominant color. Runner injectable.""" +from __future__ import annotations +import subprocess + +def _run_bytes(args) -> bytes: + return subprocess.run(args, capture_output=True, check=True).stdout + +def dominant_color_from_rgb(rgb: bytes) -> str: + r, g, b = rgb[0], rgb[1], rgb[2] + return f"#{r:02x}{g:02x}{b:02x}" + +def compute_dominant_color(path, *, midpoint_s=0.0, runner=_run_bytes) -> str: + """ffmpeg-only single-color palette of a mid-segment frame -> #rrggbb.""" + args = ["ffmpeg", "-v", "quiet", "-ss", str(midpoint_s), "-i", str(path), + "-vf", "thumbnail,palettegen=max_colors=1", "-frames:v", "1", + "-f", "rawvideo", "-pix_fmt", "rgb24", "-"] + return dominant_color_from_rgb(runner(args)) + +def extract_frame(path, dest, *, midpoint_s=0.0, runner=None): + """Write one representative frame to dest (PNG) for the review preview.""" + runner = runner or (lambda a: subprocess.run(a, check=True)) + runner(["ffmpeg", "-v", "quiet", "-y", "-ss", str(midpoint_s), "-i", str(path), + "-frames:v", "1", str(dest)]) + return dest +``` + +- [ ] **Step 3:** Run → PASS. Commit `feat: ffmpeg frame extraction + optional dominant_color`. + +--- + +### Task 6: `tools/licensing.py` — origin → license/attribution + +**Files:** `tools/licensing.py`, `tests/test_licensing.py` + +- [ ] **Step 1: Failing tests** mapping sample origin metadata per archive to the `hef.catalog.LICENSES` vocab: + - CC-BY url/identifier → `("cc_by", "")` (non-empty attribution). + - CC0 / public-domain markers → `("cc0"|"public_domain", "")`. + - LibriVox → always `("public_domain", "")`. + - unmappable → raises a clear error (rejected at ingest). +```python +from tools.licensing import normalize_license + +def test_cc_by_requires_attribution(): + lic, attr = normalize_license("https://creativecommons.org/licenses/by/4.0/", + creator="Jane Doe") + assert lic == "cc_by" and "Jane Doe" in attr + +def test_unmappable_rejected(): + import pytest + with pytest.raises(ValueError): + normalize_license("All Rights Reserved") +``` + +- [ ] **Step 2: Implement** `normalize_license(raw, *, creator="", license_name="")`: + - regex/string-match CC URLs and identifiers → `cc0`/`cc_by`/`cc_by_nc`. + - public-domain / "no known copyright" / "publicdomain" → `public_domain`. + - for `ATTRIBUTION_LICENSES` build `attribution` from `creator` + license name/URL (must be non-empty; `validate()` enforces it downstream). + - anything else → `raise ValueError(f"unmappable license: {raw!r}")`. + - Per-archive helpers may wrap it (e.g. `librivox_license()` returns `("public_domain","")`). + +- [ ] **Step 3:** Run → PASS. Commit `feat: per-archive license normalization`. + +--- + +### Task 7: `tools/drafting.py` — heuristic coordinate proposer + +**Files:** `tools/drafting.py`, `tests/test_drafting.py` + +- [ ] **Step 1: Failing tests** for `HeuristicProposer` (spec §7.1): + - `librivox` archive → high `left`; `nasa`/`musopen`/`fma` → high `right`. + - title/description containing a dark keyword ("storm") raises `dark`; a light keyword ("sunrise") raises `light`. + - all coords clamped to 0..4. + - `rationale` is a non-empty single line citing a signal. +```python +from tools.drafting import Signals, HeuristicProposer + +def test_librivox_seeds_left(): + d = HeuristicProposer().propose(Signals(title="Meditations", description="", + source_archive="librivox", mode="audio", duration_s=600)) + assert d.coordinate.left >= 3 and d.rationale + +def test_storm_seeds_dark(): + d = HeuristicProposer().propose(Signals(title="Thunderstorm at Night", + description="", source_archive="nasa", mode="video", duration_s=600)) + assert d.coordinate.dark >= 2 +``` + +- [ ] **Step 2: Implement** `Signals`, `Draft` (wrapping `hef.selection.Coordinate`), `Proposer` protocol, and `HeuristicProposer` with: archive priors (brain plane), keyword sets for dark/light (mood plane), `max(0, min(4, v))` clamping, one-line rationale naming the dominant signal. Deterministic — no I/O. + +- [ ] **Step 3:** Run → PASS. Commit `feat: heuristic coordinate proposer (drafting)`. + +--- + +### Task 8: `tools/ingest/base.py` — Candidate, Fetcher, pipeline + +**Files:** `tools/ingest/__init__.py`, `tools/ingest/base.py`, `tests/test_ingest_pipeline.py` + +- [ ] **Step 1: Failing tests** for `ingest_candidate` with ALL boundaries faked (no network, no ffprobe, no disk writes beyond tmp_path): + - given a `Candidate` + fake `prober` (returns a Probe) + fake `downloader` (writes a tmp file) + `HeuristicProposer`, it appends ONE `proposed` record whose mechanical fields match the probe, `review_status=="proposed"`, `reviewed_at is None`, `rationale` non-empty, `file_path` relative to media-root. + - re-running with the same `suggested_id` is idempotent (no duplicate; skipped). + - `dominant_color` stays `""` by default; computed only when `compute_color=True`. + - a candidate whose license is unmappable raises before writing (nothing appended). + +- [ ] **Step 2: Implement** `Candidate` (spec §6.1), the `Fetcher` Protocol (`archive`, `search`, `resolve`), and: +```python +def ingest_candidate(candidate, *, catalog_path, media_root, proposer, + prober=probe_file, downloader=..., compute_color=False): + # 1. load_catalog + validate_catalog; skip if candidate.suggested_id exists + # 2. download media_url -> //. (skip if present) + # 3. tags = derive_tags(prober(file)); color = compute_dominant_color(...) if compute_color and video + # 4. draft = proposer.propose(Signals(...)) + # 5. build hef.catalog.Record(...) (proposed shape via defaults) + # 6. validate(record); re-check uniqueness; append_record(record, catalog_path) +``` +Plus `ingest_search(fetcher, query, *, limit, **kw)` looping the above over `fetcher.search`. + +- [ ] **Step 3:** Run → PASS. Commit `feat: ingest pipeline (Candidate/Fetcher/ingest_candidate)`. + +--- + +### Task 9: First-ship fetchers — LibriVox, NASA, Internet Archive + +**Files:** `tools/ingest/librivox.py`, `tools/ingest/nasa.py`, `tools/ingest/internet_archive.py`, `tools/ingest/{musopen,fma,freesound}.py` (deferred stubs), `tests/test_fetchers.py` + +- [ ] **Step 1: Failing tests** — one per first-ship fetcher, feeding canned API JSON through `HttpClient(opener=fake)`; assert `Candidate` fields + a stable `suggested_id` derivation and the normalized license: + - **LibriVox** (`librivox.org/api/feed/audiobooks?...&format=json`): `license="public_domain"`, `mode` hint audio, id like `librivox-`. + - **NASA** (`images-api.nasa.gov/search?q=...`): `license="public_domain"`, id like `nasa-`, media_url from the asset collection. + - **Internet Archive** (`archive.org/metadata/`): license from `licenseurl`/`rights` via `tools.licensing`; ambiguous "no known copyright" → `public_domain` with a `notes` flag; id like `ia-`. + +- [ ] **Step 2: Implement** the three fetchers against the documented JSON APIs, each taking an injected `HttpClient`, each exposing `archive`, `search(query, limit)`, `resolve(identifier)`. Use `tools.licensing` for license/attribution. + +- [ ] **Step 3: Deferred stubs.** `musopen.py`, `fma.py`, `freesound.py` define the class with the `Fetcher` shape but `raise NotImplementedError("deferred — see spec §6.4")`; `freesound.py` documents the `FREESOUND_API_TOKEN` env requirement (secret; never logged). A test asserts they raise `NotImplementedError` (so the seam is wired and the deferral is explicit, not forgotten). + +- [ ] **Step 4:** Run → PASS. Commit `feat: LibriVox/NASA/Internet Archive fetchers (+ deferred stubs)`. + +--- + +### Task 10: `tools/ingest_cli.py` + +**Files:** `tools/ingest_cli.py` (light/manual test) + +- [ ] **Step 1:** `argparse` entry: `python -m tools.ingest_cli --query "..." [--limit N] [--catalog catalog/library.jsonl] [--media-root ./media] [--dominant-color] [--resolve ]`. Wires the named fetcher + `HeuristicProposer` into `ingest_search`/`ingest_candidate`. `main()` returns an exit code; secrets read from env only. + +- [ ] **Step 2:** A small test that `--help` parses and that an unknown archive errors cleanly (no network). Real fetches are manual/integration. + +- [ ] **Step 3:** Commit `feat: ingest CLI entry point`. + +--- + +### Task 11: `tools/review.py` — the transition core + +**Files:** `tools/review.py`, `tests/test_review.py` + +- [ ] **Step 1: Failing tests** (spec §8.1): +```python +from dataclasses import replace +from tools.review import proposed_records, approve +from hef.selection import Coordinate +# make_record helper as elsewhere + +def test_proposed_records_filters(): + a = make_record(id="a", review_status="proposed") + b = make_record(id="b", review_status="approved") + assert [r.id for r in proposed_records([a, b])] == ["a"] + +def test_approve_sets_status_and_timestamp(): + r = make_record(review_status="proposed") + out = approve(r, reviewed_at="2026-06-04T13:00:00+00:00") + assert out.review_status == "approved" and out.reviewed_at == "2026-06-04T13:00:00+00:00" + +def test_approve_can_override_coordinates(): + r = make_record(left=0, right=0, dark=0, light=0, review_status="proposed") + out = approve(r, reviewed_at="t", coordinate=Coordinate(4, 1, 2, 3)) + assert (out.left, out.right, out.dark, out.light) == (4, 1, 2, 3) +``` + +- [ ] **Step 2: Implement** `proposed_records(records)` and `approve(record, *, reviewed_at, coordinate=None)` using `dataclasses.replace` to return an approved copy (no in-place surprise); caller re-validates. Pure, no I/O, no clock (timestamp injected). + +- [ ] **Step 3:** Run → PASS. Commit `feat: review transition core (proposed -> approved)`. + +--- + +### Task 12: `tools/review_cli.py` — interactive walk + +**Files:** `tools/review_cli.py` (manual; preview reuses `tools.mediatools`) + +- [ ] **Step 1:** Implement the walk (spec §8.2): `load_catalog` → `validate_catalog`; for each `proposed` record print id/title/source/license/mode/duration/resolution + proposed coords + rationale; render a preview (video/av → `extract_frame` then `open`/`xdg-open`; audio → `showwavespic` thumbnail and/or optional `ffplay`); prompt `[a]ccept/[e]dit/[s]kip/[q]uit`; on accept/edit call `approve(...)` with `datetime.now(timezone.utc).isoformat()` and persist via `save_catalog` (rewrite) after each approval. `--catalog`, `--media-root` args. + +- [ ] **Step 2:** Keep the shell thin — all decision logic already tested in Task 11. A tiny test that `--help` parses; the interactive loop is manual. + +- [ ] **Step 3:** Commit `feat: interactive review CLI`. + +--- + +### Task 13: End-to-end mocked integration + opt-in real-ffprobe + +**Files:** `tests/test_tools_integration.py` + +- [ ] **Step 1: Hermetic end-to-end** (spec §10 test 8): fake fetcher → `Candidate`; fake downloader + fake prober + `HeuristicProposer` → `ingest_candidate` appends a `proposed` record; `proposed_records` + `approve` flips it; `load_catalog` + `validate_catalog` pass; `select(loaded, coord, mode, approved_only=True)` returns it. + +- [ ] **Step 2: Opt-in real-ffprobe test** — guarded by `shutil.which("ffprobe")` (`pytest.mark.skipif` when absent): generate a 1-second test clip with `ffmpeg lavfi` into `tmp_path`, probe it, assert `mode`/`duration_s`/`resolution`. Same pattern for a real `compute_dominant_color`. + +- [ ] **Step 3:** Run the FULL suite `python -m pytest -q` → green. Commit `test: end-to-end ingest+review integration (+ opt-in ffprobe)`. + +--- + +### Task 14: Docs — extend the User Guide for the tools + +**Files:** `docs/USER_GUIDE.md` + +- [ ] **Step 1:** Add an "Ingesting & reviewing media" section: prerequisites (`ffmpeg`/`ffprobe`), `--media-root`, the first-ship archives, `python -m tools.ingest_cli ...` examples, the review walk, the Freesound token note (env only), and that `dominant_color` is optional/opt-in. Update the scope banner (no longer "only catalog core is built"). Reconcile the absolute-vs-relative `file_path` note (spec §6.3). + +- [ ] **Step 2:** Commit `docs: user guide for ingest + review tools`. + +--- + +## Done criteria + +- `python -m pytest -q` is green from the repo root inside `.venv` (prior 37 + all new unit tests; opt-in ffprobe tests skip cleanly when binaries are absent). +- `hef.catalog` gains exactly `validate_catalog` and `index_by_id`; every pre-existing symbol/behavior is unchanged. +- `python -m tools.ingest_cli --query ...` against LibriVox / NASA / Internet Archive produces `proposed` records with mechanical fields filled and a one-line rationale. +- `python -m tools.review_cli` walks the `proposed` records (coords + rationale + preview frame) and flips accepted ones to `approved` with a `reviewed_at` stamp; `load_catalog` + `validate_catalog` validate the result. +- `dominant_color` is computed only with `--dominant-color`; Freesound/Musopen/FMA fetchers are explicit deferred stubs (raise `NotImplementedError`). + +This populated, human-reviewed catalog is what sub-project 3 (the Pi player) drives to the single panoramic projector.