|
|
|
@@ -0,0 +1,500 @@
|
|
|
|
|
# Ingest & Tagging / Review Tools — Implementation Plan
|
|
|
|
|
|
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
|
|
|
|
|
|
**Goal:** Build the `tools/` package that turns the hand-authored catalog into the assisted *draft-then-review* pipeline: per-archive ingest fetchers, mechanical tagging (ffprobe/ffmpeg + origin license), heuristic coordinate drafting (`review_status=proposed` + one-line rationale), and an interactive review CLI that flips records to `approved` with a `reviewed_at` stamp.
|
|
|
|
|
|
|
|
|
|
**Spec:** [`2026-06-04-ingest-tagging-review-tools.md`](../specs/2026-06-04-ingest-tagging-review-tools.md). Read it first — this plan implements it task-by-task and inherits its decisions (esp. §3 the single required `hef.catalog` addition, §5.2 `dominant_color` is optional/opt-in, §6.4 first-ship = LibriVox + NASA + Internet Archive).
|
|
|
|
|
|
|
|
|
|
**Architecture:** A `tools/` package alongside the existing `hef/` (which it imports, never forks). The two external boundaries — `ffmpeg`/`ffprobe` (subprocess) and archive HTTP — are isolated behind injectable seams (a `Prober`, a `download` callable, an HTTP client) so the unit suite stays **hermetic** (no network, no binaries). Integration tests that actually invoke `ffprobe`/`ffmpeg` are opt-in and skipped when the binaries are absent.
|
|
|
|
|
|
|
|
|
|
**Tech Stack:** Python 3.11+, stdlib only in the testable core (`dataclasses`, `json`, `subprocess`, `urllib`, `pathlib`, `datetime`); `ffmpeg`/`ffprobe` system binaries; pytest. No `requests`, no image library (dominant color, when computed, is ffmpeg-only).
|
|
|
|
|
|
|
|
|
|
**Where this sits:** sub-project **2 of 5**. Depends on sub-project 1 (`hef.catalog`, `hef.selection`, built). Feeds sub-project 3 (the Pi player). Sub-project 5 (side walls) was dropped (design §7 revision), so `dominant_color` is optional here.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## File Structure
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
pyproject.toml + add `tools` to packages; optional [project.scripts]
|
|
|
|
|
.gitignore + media/
|
|
|
|
|
hef/catalog.py + validate_catalog(), index_by_id() (ONLY hef change)
|
|
|
|
|
tools/__init__.py package marker
|
|
|
|
|
tools/http.py stdlib-urllib client (timeout/retry/user-agent), injectable
|
|
|
|
|
tools/probe.py Prober: ffprobe wrapper -> parsed Probe result
|
|
|
|
|
tools/tagging.py mode/duration_s/resolution from a Probe (+ cover-art guard)
|
|
|
|
|
tools/mediatools.py ffmpeg: representative frame; optional dominant_color
|
|
|
|
|
tools/licensing.py per-archive license/attribution -> hef LICENSES vocab
|
|
|
|
|
tools/drafting.py Signals, Draft, Proposer, HeuristicProposer
|
|
|
|
|
tools/ingest/__init__.py
|
|
|
|
|
tools/ingest/base.py Candidate, Fetcher, ingest_candidate() pipeline
|
|
|
|
|
tools/ingest/librivox.py first-ship fetcher
|
|
|
|
|
tools/ingest/nasa.py first-ship fetcher
|
|
|
|
|
tools/ingest/internet_archive.py first-ship fetcher
|
|
|
|
|
tools/ingest/musopen.py deferred: docstring stub raising NotImplementedError
|
|
|
|
|
tools/ingest/fma.py deferred: docstring stub
|
|
|
|
|
tools/ingest/freesound.py deferred: docstring stub (needs FREESOUND_API_TOKEN)
|
|
|
|
|
tools/ingest_cli.py `python -m tools.ingest_cli`
|
|
|
|
|
tools/review.py proposed_records(), approve() (pure transition core)
|
|
|
|
|
tools/review_cli.py `python -m tools.review_cli` (interactive walk)
|
|
|
|
|
tests/test_catalog_integrity.py validate_catalog / index_by_id
|
|
|
|
|
tests/test_probe.py ffprobe JSON parsing (canned)
|
|
|
|
|
tests/test_tagging.py mode/duration/resolution + cover-art guard
|
|
|
|
|
tests/test_mediatools.py dominant_color from canned rgb bytes; opt-in default off
|
|
|
|
|
tests/test_licensing.py per-archive license normalization
|
|
|
|
|
tests/test_drafting.py HeuristicProposer
|
|
|
|
|
tests/test_fetchers.py librivox/nasa/internet_archive via fake HTTP client
|
|
|
|
|
tests/test_ingest_pipeline.py ingest_candidate end-to-end with fakes
|
|
|
|
|
tests/test_review.py proposed_records / approve transition
|
|
|
|
|
tests/test_tools_integration.py end-to-end mocked; opt-in real-ffprobe (skipped if absent)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`tools/` holds everything new; `hef/` gains only the two additive symbols of spec §3.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 1: Scaffolding — `tools/` package, deps, http client, smoke test
|
|
|
|
|
|
|
|
|
|
**Files:** `pyproject.toml`, `.gitignore`, `tools/__init__.py`, `tools/http.py`, `tests/test_tools_smoke.py`
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1: Add `tools` to packaging.** In `pyproject.toml`, change `packages = ["hef"]` → `packages = ["hef", "tools"]`. (Optional: add `[project.scripts]` `hef-ingest = "tools.ingest_cli:main"` and `hef-review = "tools.review_cli:main"` for installed checkouts; `python -m` is the documented path either way.)
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2: gitignore media.** Append `media/` to `.gitignore`.
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3: Write the failing smoke test** `tests/test_tools_smoke.py`:
|
|
|
|
|
```python
|
|
|
|
|
def test_tools_package_imports():
|
|
|
|
|
import tools
|
|
|
|
|
import tools.http
|
|
|
|
|
assert tools is not None
|
|
|
|
|
```
|
|
|
|
|
Run `python -m pytest tests/test_tools_smoke.py -v` → FAIL (`ModuleNotFoundError: tools`).
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 4: Create `tools/__init__.py`** (empty) and `tools/http.py` — a tiny stdlib client:
|
|
|
|
|
```python
|
|
|
|
|
"""Minimal HTTP client over urllib: timeout, small retry, user-agent. Injectable."""
|
|
|
|
|
from __future__ import annotations
|
|
|
|
|
import json as _json
|
|
|
|
|
import time
|
|
|
|
|
import urllib.request
|
|
|
|
|
|
|
|
|
|
USER_AGENT = "human-experience-filter-ingest/0.1 (+local art installation)"
|
|
|
|
|
|
|
|
|
|
class HttpClient:
|
|
|
|
|
def __init__(self, *, timeout=30.0, retries=2, opener=None):
|
|
|
|
|
self.timeout, self.retries = timeout, retries
|
|
|
|
|
self._opener = opener or urllib.request.urlopen
|
|
|
|
|
|
|
|
|
|
def get_bytes(self, url, *, headers=None):
|
|
|
|
|
last = None
|
|
|
|
|
for attempt in range(self.retries + 1):
|
|
|
|
|
try:
|
|
|
|
|
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT, **(headers or {})})
|
|
|
|
|
with self._opener(req, timeout=self.timeout) as resp:
|
|
|
|
|
return resp.read()
|
|
|
|
|
except Exception as exc: # noqa: BLE001 - retried below
|
|
|
|
|
last = exc
|
|
|
|
|
if attempt < self.retries:
|
|
|
|
|
time.sleep(0.2 * (attempt + 1))
|
|
|
|
|
raise last
|
|
|
|
|
|
|
|
|
|
def get_json(self, url, *, headers=None):
|
|
|
|
|
return _json.loads(self.get_bytes(url, headers=headers).decode("utf-8"))
|
|
|
|
|
```
|
|
|
|
|
The `opener` injection point lets tests pass a fake (no network).
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 5:** Run smoke test → PASS. Commit:
|
|
|
|
|
```
|
|
|
|
|
git add pyproject.toml .gitignore tools/__init__.py tools/http.py tests/test_tools_smoke.py
|
|
|
|
|
git commit -m "chore: scaffold tools/ package + stdlib http client"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 2: `hef.catalog` additions — `validate_catalog` + `index_by_id`
|
|
|
|
|
|
|
|
|
|
The ONLY change to the shared spine (spec §3). Purely additive.
|
|
|
|
|
|
|
|
|
|
**Files:** `hef/catalog.py`, `tests/test_catalog_integrity.py`
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1: Failing tests** `tests/test_catalog_integrity.py` (reuse a local `make_record` like `tests/test_catalog.py`):
|
|
|
|
|
```python
|
|
|
|
|
import pytest
|
|
|
|
|
from hef.catalog import Record, validate_catalog, index_by_id, CatalogError
|
|
|
|
|
|
|
|
|
|
def make_record(**o):
|
|
|
|
|
base = dict(id="a", title="t", source_url="u", source_archive="nasa",
|
|
|
|
|
license="public_domain", mode="video", left=0, right=0, dark=0,
|
|
|
|
|
light=0, duration_s=1, file_path="p")
|
|
|
|
|
base.update(o); return Record(**base)
|
|
|
|
|
|
|
|
|
|
def test_validate_catalog_accepts_unique_ids():
|
|
|
|
|
validate_catalog([make_record(id="a"), make_record(id="b")]) # no raise
|
|
|
|
|
|
|
|
|
|
def test_validate_catalog_rejects_duplicate_id():
|
|
|
|
|
with pytest.raises(CatalogError) as e:
|
|
|
|
|
validate_catalog([make_record(id="dup"), make_record(id="dup")])
|
|
|
|
|
assert "dup" in str(e.value)
|
|
|
|
|
|
|
|
|
|
def test_validate_catalog_validates_each_record():
|
|
|
|
|
with pytest.raises(CatalogError):
|
|
|
|
|
validate_catalog([make_record(left=9)])
|
|
|
|
|
|
|
|
|
|
def test_index_by_id_round_trips():
|
|
|
|
|
recs = [make_record(id="a"), make_record(id="b")]
|
|
|
|
|
idx = index_by_id(recs)
|
|
|
|
|
assert idx["a"].id == "a" and idx["b"].id == "b"
|
|
|
|
|
|
|
|
|
|
def test_index_by_id_rejects_duplicates():
|
|
|
|
|
with pytest.raises(CatalogError):
|
|
|
|
|
index_by_id([make_record(id="x"), make_record(id="x")])
|
|
|
|
|
```
|
|
|
|
|
Run → FAIL (ImportError).
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2: Implement** — append to `hef/catalog.py` (do NOT touch existing functions):
|
|
|
|
|
```python
|
|
|
|
|
def index_by_id(records) -> dict:
|
|
|
|
|
"""Map id -> Record, raising CatalogError on a duplicate id."""
|
|
|
|
|
idx: dict = {}
|
|
|
|
|
for r in records:
|
|
|
|
|
if r.id in idx:
|
|
|
|
|
raise CatalogError(f"duplicate record id: {r.id!r}")
|
|
|
|
|
idx[r.id] = r
|
|
|
|
|
return idx
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def validate_catalog(records) -> None:
|
|
|
|
|
"""Validate every record AND cross-record invariants (currently: unique ids)."""
|
|
|
|
|
for r in records:
|
|
|
|
|
validate(r)
|
|
|
|
|
index_by_id(records) # raises on duplicate id
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3:** Run `python -m pytest tests/test_catalog_integrity.py -v` → PASS. Run full suite `python -m pytest -q` → still green (37 prior + new). Commit:
|
|
|
|
|
```
|
|
|
|
|
git add hef/catalog.py tests/test_catalog_integrity.py
|
|
|
|
|
git commit -m "feat: catalog-level validate_catalog + index_by_id (unique ids)"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 3: `tools/probe.py` — ffprobe wrapper
|
|
|
|
|
|
|
|
|
|
**Files:** `tools/probe.py`, `tests/test_probe.py`
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1: Failing tests** — feed canned `ffprobe` JSON via an injected runner; assert a parsed `Probe` (list of streams + format dict). Cover: a video+audio file, an audio-only file, and an audio file with an `attached_pic` cover stream. Example:
|
|
|
|
|
```python
|
|
|
|
|
import json
|
|
|
|
|
from tools.probe import Probe, probe_file
|
|
|
|
|
|
|
|
|
|
def fake_runner(args): # mimics subprocess.run(...).stdout
|
|
|
|
|
return json.dumps({
|
|
|
|
|
"streams": [
|
|
|
|
|
{"codec_type": "video", "width": 1920, "height": 1080, "disposition": {"attached_pic": 0}},
|
|
|
|
|
{"codec_type": "audio"},
|
|
|
|
|
],
|
|
|
|
|
"format": {"duration": "12.5"},
|
|
|
|
|
})
|
|
|
|
|
|
|
|
|
|
def test_probe_parses_streams_and_format():
|
|
|
|
|
p = probe_file("x.mp4", runner=fake_runner)
|
|
|
|
|
assert any(s["codec_type"] == "video" for s in p.streams)
|
|
|
|
|
assert p.format["duration"] == "12.5"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2: Implement** `tools/probe.py`:
|
|
|
|
|
```python
|
|
|
|
|
"""ffprobe wrapper -> parsed streams/format. Subprocess runner is injectable."""
|
|
|
|
|
from __future__ import annotations
|
|
|
|
|
import json, subprocess
|
|
|
|
|
from dataclasses import dataclass
|
|
|
|
|
|
|
|
|
|
@dataclass
|
|
|
|
|
class Probe:
|
|
|
|
|
streams: list
|
|
|
|
|
format: dict
|
|
|
|
|
|
|
|
|
|
def _default_runner(args) -> str:
|
|
|
|
|
return subprocess.run(args, capture_output=True, text=True, check=True).stdout
|
|
|
|
|
|
|
|
|
|
def probe_file(path, *, runner=_default_runner) -> Probe:
|
|
|
|
|
out = runner(["ffprobe", "-v", "quiet", "-print_format", "json",
|
|
|
|
|
"-show_format", "-show_streams", str(path)])
|
|
|
|
|
data = json.loads(out)
|
|
|
|
|
return Probe(streams=data.get("streams", []), format=data.get("format", {}))
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3:** Run → PASS. Commit `feat: ffprobe wrapper (tools.probe)`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 4: `tools/tagging.py` — mode / duration_s / resolution (with cover-art guard)
|
|
|
|
|
|
|
|
|
|
**Files:** `tools/tagging.py`, `tests/test_tagging.py`
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1: Failing tests** covering the spec §5.1 cases:
|
|
|
|
|
- video+audio → `mode="av"`, `resolution="1920x1080"`, `duration_s=13` (round 12.5).
|
|
|
|
|
- audio only → `mode="audio"`, `resolution=""`.
|
|
|
|
|
- **audio + `attached_pic` cover image → `mode="audio"`** (the guard), `resolution=""`.
|
|
|
|
|
- video only → `mode="video"`.
|
|
|
|
|
- duration fallback to longest stream when `format.duration` absent.
|
|
|
|
|
```python
|
|
|
|
|
from tools.probe import Probe
|
|
|
|
|
from tools.tagging import derive_tags
|
|
|
|
|
|
|
|
|
|
def test_cover_art_stays_audio():
|
|
|
|
|
p = Probe(streams=[
|
|
|
|
|
{"codec_type": "audio", "duration": "30.0"},
|
|
|
|
|
{"codec_type": "video", "width": 600, "height": 600, "disposition": {"attached_pic": 1}},
|
|
|
|
|
], format={"duration": "30.0"})
|
|
|
|
|
t = derive_tags(p)
|
|
|
|
|
assert t.mode == "audio" and t.resolution == "" and t.duration_s == 30
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2: Implement** — a `Tags(mode, duration_s, resolution)` dataclass and `derive_tags(probe)`:
|
|
|
|
|
- real video streams = `codec_type == "video"` AND `disposition.attached_pic != 1`.
|
|
|
|
|
- mode: both→`av`, video-only→`video`, else→`audio`.
|
|
|
|
|
- resolution from the first real video stream (`f"{w}x{h}"`), else `""`.
|
|
|
|
|
- duration: `round(float(format["duration"]))` else max stream duration else 0; never negative.
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3:** Run → PASS. Commit `feat: mechanical mode/duration/resolution tagging`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 5: `tools/mediatools.py` — representative frame + optional dominant_color
|
|
|
|
|
|
|
|
|
|
**Files:** `tools/mediatools.py`, `tests/test_mediatools.py`
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1: Failing tests** (hermetic — inject the ffmpeg runner returning canned bytes):
|
|
|
|
|
- `dominant_color_from_rgb(b"\xff\x00\x00") == "#ff0000"`.
|
|
|
|
|
- `compute_dominant_color(path, runner=fake)` returns the hex when enabled.
|
|
|
|
|
- The default ingest path does NOT call this (asserted in Task 8, not here).
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2: Implement**:
|
|
|
|
|
```python
|
|
|
|
|
"""ffmpeg helpers: representative frame; OPTIONAL dominant color. Runner injectable."""
|
|
|
|
|
from __future__ import annotations
|
|
|
|
|
import subprocess
|
|
|
|
|
|
|
|
|
|
def _run_bytes(args) -> bytes:
|
|
|
|
|
return subprocess.run(args, capture_output=True, check=True).stdout
|
|
|
|
|
|
|
|
|
|
def dominant_color_from_rgb(rgb: bytes) -> str:
|
|
|
|
|
r, g, b = rgb[0], rgb[1], rgb[2]
|
|
|
|
|
return f"#{r:02x}{g:02x}{b:02x}"
|
|
|
|
|
|
|
|
|
|
def compute_dominant_color(path, *, midpoint_s=0.0, runner=_run_bytes) -> str:
|
|
|
|
|
"""ffmpeg-only single-color palette of a mid-segment frame -> #rrggbb."""
|
|
|
|
|
args = ["ffmpeg", "-v", "quiet", "-ss", str(midpoint_s), "-i", str(path),
|
|
|
|
|
"-vf", "thumbnail,palettegen=max_colors=1", "-frames:v", "1",
|
|
|
|
|
"-f", "rawvideo", "-pix_fmt", "rgb24", "-"]
|
|
|
|
|
return dominant_color_from_rgb(runner(args))
|
|
|
|
|
|
|
|
|
|
def extract_frame(path, dest, *, midpoint_s=0.0, runner=None):
|
|
|
|
|
"""Write one representative frame to dest (PNG) for the review preview."""
|
|
|
|
|
runner = runner or (lambda a: subprocess.run(a, check=True))
|
|
|
|
|
runner(["ffmpeg", "-v", "quiet", "-y", "-ss", str(midpoint_s), "-i", str(path),
|
|
|
|
|
"-frames:v", "1", str(dest)])
|
|
|
|
|
return dest
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3:** Run → PASS. Commit `feat: ffmpeg frame extraction + optional dominant_color`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 6: `tools/licensing.py` — origin → license/attribution
|
|
|
|
|
|
|
|
|
|
**Files:** `tools/licensing.py`, `tests/test_licensing.py`
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1: Failing tests** mapping sample origin metadata per archive to the `hef.catalog.LICENSES` vocab:
|
|
|
|
|
- CC-BY url/identifier → `("cc_by", "<attribution string>")` (non-empty attribution).
|
|
|
|
|
- CC0 / public-domain markers → `("cc0"|"public_domain", "")`.
|
|
|
|
|
- LibriVox → always `("public_domain", "")`.
|
|
|
|
|
- unmappable → raises a clear error (rejected at ingest).
|
|
|
|
|
```python
|
|
|
|
|
from tools.licensing import normalize_license
|
|
|
|
|
|
|
|
|
|
def test_cc_by_requires_attribution():
|
|
|
|
|
lic, attr = normalize_license("https://creativecommons.org/licenses/by/4.0/",
|
|
|
|
|
creator="Jane Doe")
|
|
|
|
|
assert lic == "cc_by" and "Jane Doe" in attr
|
|
|
|
|
|
|
|
|
|
def test_unmappable_rejected():
|
|
|
|
|
import pytest
|
|
|
|
|
with pytest.raises(ValueError):
|
|
|
|
|
normalize_license("All Rights Reserved")
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2: Implement** `normalize_license(raw, *, creator="", license_name="")`:
|
|
|
|
|
- regex/string-match CC URLs and identifiers → `cc0`/`cc_by`/`cc_by_nc`.
|
|
|
|
|
- public-domain / "no known copyright" / "publicdomain" → `public_domain`.
|
|
|
|
|
- for `ATTRIBUTION_LICENSES` build `attribution` from `creator` + license name/URL (must be non-empty; `validate()` enforces it downstream).
|
|
|
|
|
- anything else → `raise ValueError(f"unmappable license: {raw!r}")`.
|
|
|
|
|
- Per-archive helpers may wrap it (e.g. `librivox_license()` returns `("public_domain","")`).
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3:** Run → PASS. Commit `feat: per-archive license normalization`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 7: `tools/drafting.py` — heuristic coordinate proposer
|
|
|
|
|
|
|
|
|
|
**Files:** `tools/drafting.py`, `tests/test_drafting.py`
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1: Failing tests** for `HeuristicProposer` (spec §7.1):
|
|
|
|
|
- `librivox` archive → high `left`; `nasa`/`musopen`/`fma` → high `right`.
|
|
|
|
|
- title/description containing a dark keyword ("storm") raises `dark`; a light keyword ("sunrise") raises `light`.
|
|
|
|
|
- all coords clamped to 0..4.
|
|
|
|
|
- `rationale` is a non-empty single line citing a signal.
|
|
|
|
|
```python
|
|
|
|
|
from tools.drafting import Signals, HeuristicProposer
|
|
|
|
|
|
|
|
|
|
def test_librivox_seeds_left():
|
|
|
|
|
d = HeuristicProposer().propose(Signals(title="Meditations", description="",
|
|
|
|
|
source_archive="librivox", mode="audio", duration_s=600))
|
|
|
|
|
assert d.coordinate.left >= 3 and d.rationale
|
|
|
|
|
|
|
|
|
|
def test_storm_seeds_dark():
|
|
|
|
|
d = HeuristicProposer().propose(Signals(title="Thunderstorm at Night",
|
|
|
|
|
description="", source_archive="nasa", mode="video", duration_s=600))
|
|
|
|
|
assert d.coordinate.dark >= 2
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2: Implement** `Signals`, `Draft` (wrapping `hef.selection.Coordinate`), `Proposer` protocol, and `HeuristicProposer` with: archive priors (brain plane), keyword sets for dark/light (mood plane), `max(0, min(4, v))` clamping, one-line rationale naming the dominant signal. Deterministic — no I/O.
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3:** Run → PASS. Commit `feat: heuristic coordinate proposer (drafting)`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 8: `tools/ingest/base.py` — Candidate, Fetcher, pipeline
|
|
|
|
|
|
|
|
|
|
**Files:** `tools/ingest/__init__.py`, `tools/ingest/base.py`, `tests/test_ingest_pipeline.py`
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1: Failing tests** for `ingest_candidate` with ALL boundaries faked (no network, no ffprobe, no disk writes beyond tmp_path):
|
|
|
|
|
- given a `Candidate` + fake `prober` (returns a Probe) + fake `downloader` (writes a tmp file) + `HeuristicProposer`, it appends ONE `proposed` record whose mechanical fields match the probe, `review_status=="proposed"`, `reviewed_at is None`, `rationale` non-empty, `file_path` relative to media-root.
|
|
|
|
|
- re-running with the same `suggested_id` is idempotent (no duplicate; skipped).
|
|
|
|
|
- `dominant_color` stays `""` by default; computed only when `compute_color=True`.
|
|
|
|
|
- a candidate whose license is unmappable raises before writing (nothing appended).
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2: Implement** `Candidate` (spec §6.1), the `Fetcher` Protocol (`archive`, `search`, `resolve`), and:
|
|
|
|
|
```python
|
|
|
|
|
def ingest_candidate(candidate, *, catalog_path, media_root, proposer,
|
|
|
|
|
prober=probe_file, downloader=..., compute_color=False):
|
|
|
|
|
# 1. load_catalog + validate_catalog; skip if candidate.suggested_id exists
|
|
|
|
|
# 2. download media_url -> <media_root>/<archive>/<id>.<ext> (skip if present)
|
|
|
|
|
# 3. tags = derive_tags(prober(file)); color = compute_dominant_color(...) if compute_color and video
|
|
|
|
|
# 4. draft = proposer.propose(Signals(...))
|
|
|
|
|
# 5. build hef.catalog.Record(...) (proposed shape via defaults)
|
|
|
|
|
# 6. validate(record); re-check uniqueness; append_record(record, catalog_path)
|
|
|
|
|
```
|
|
|
|
|
Plus `ingest_search(fetcher, query, *, limit, **kw)` looping the above over `fetcher.search`.
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3:** Run → PASS. Commit `feat: ingest pipeline (Candidate/Fetcher/ingest_candidate)`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 9: First-ship fetchers — LibriVox, NASA, Internet Archive
|
|
|
|
|
|
|
|
|
|
**Files:** `tools/ingest/librivox.py`, `tools/ingest/nasa.py`, `tools/ingest/internet_archive.py`, `tools/ingest/{musopen,fma,freesound}.py` (deferred stubs), `tests/test_fetchers.py`
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1: Failing tests** — one per first-ship fetcher, feeding canned API JSON through `HttpClient(opener=fake)`; assert `Candidate` fields + a stable `suggested_id` derivation and the normalized license:
|
|
|
|
|
- **LibriVox** (`librivox.org/api/feed/audiobooks?...&format=json`): `license="public_domain"`, `mode` hint audio, id like `librivox-<slug>`.
|
|
|
|
|
- **NASA** (`images-api.nasa.gov/search?q=...`): `license="public_domain"`, id like `nasa-<nasa_id>`, media_url from the asset collection.
|
|
|
|
|
- **Internet Archive** (`archive.org/metadata/<id>`): license from `licenseurl`/`rights` via `tools.licensing`; ambiguous "no known copyright" → `public_domain` with a `notes` flag; id like `ia-<identifier>`.
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2: Implement** the three fetchers against the documented JSON APIs, each taking an injected `HttpClient`, each exposing `archive`, `search(query, limit)`, `resolve(identifier)`. Use `tools.licensing` for license/attribution.
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3: Deferred stubs.** `musopen.py`, `fma.py`, `freesound.py` define the class with the `Fetcher` shape but `raise NotImplementedError("deferred — see spec §6.4")`; `freesound.py` documents the `FREESOUND_API_TOKEN` env requirement (secret; never logged). A test asserts they raise `NotImplementedError` (so the seam is wired and the deferral is explicit, not forgotten).
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 4:** Run → PASS. Commit `feat: LibriVox/NASA/Internet Archive fetchers (+ deferred stubs)`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 10: `tools/ingest_cli.py`
|
|
|
|
|
|
|
|
|
|
**Files:** `tools/ingest_cli.py` (light/manual test)
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1:** `argparse` entry: `python -m tools.ingest_cli <archive> --query "..." [--limit N] [--catalog catalog/library.jsonl] [--media-root ./media] [--dominant-color] [--resolve <identifier>]`. Wires the named fetcher + `HeuristicProposer` into `ingest_search`/`ingest_candidate`. `main()` returns an exit code; secrets read from env only.
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2:** A small test that `--help` parses and that an unknown archive errors cleanly (no network). Real fetches are manual/integration.
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3:** Commit `feat: ingest CLI entry point`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 11: `tools/review.py` — the transition core
|
|
|
|
|
|
|
|
|
|
**Files:** `tools/review.py`, `tests/test_review.py`
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1: Failing tests** (spec §8.1):
|
|
|
|
|
```python
|
|
|
|
|
from dataclasses import replace
|
|
|
|
|
from tools.review import proposed_records, approve
|
|
|
|
|
from hef.selection import Coordinate
|
|
|
|
|
# make_record helper as elsewhere
|
|
|
|
|
|
|
|
|
|
def test_proposed_records_filters():
|
|
|
|
|
a = make_record(id="a", review_status="proposed")
|
|
|
|
|
b = make_record(id="b", review_status="approved")
|
|
|
|
|
assert [r.id for r in proposed_records([a, b])] == ["a"]
|
|
|
|
|
|
|
|
|
|
def test_approve_sets_status_and_timestamp():
|
|
|
|
|
r = make_record(review_status="proposed")
|
|
|
|
|
out = approve(r, reviewed_at="2026-06-04T13:00:00+00:00")
|
|
|
|
|
assert out.review_status == "approved" and out.reviewed_at == "2026-06-04T13:00:00+00:00"
|
|
|
|
|
|
|
|
|
|
def test_approve_can_override_coordinates():
|
|
|
|
|
r = make_record(left=0, right=0, dark=0, light=0, review_status="proposed")
|
|
|
|
|
out = approve(r, reviewed_at="t", coordinate=Coordinate(4, 1, 2, 3))
|
|
|
|
|
assert (out.left, out.right, out.dark, out.light) == (4, 1, 2, 3)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2: Implement** `proposed_records(records)` and `approve(record, *, reviewed_at, coordinate=None)` using `dataclasses.replace` to return an approved copy (no in-place surprise); caller re-validates. Pure, no I/O, no clock (timestamp injected).
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3:** Run → PASS. Commit `feat: review transition core (proposed -> approved)`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 12: `tools/review_cli.py` — interactive walk
|
|
|
|
|
|
|
|
|
|
**Files:** `tools/review_cli.py` (manual; preview reuses `tools.mediatools`)
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1:** Implement the walk (spec §8.2): `load_catalog` → `validate_catalog`; for each `proposed` record print id/title/source/license/mode/duration/resolution + proposed coords + rationale; render a preview (video/av → `extract_frame` then `open`/`xdg-open`; audio → `showwavespic` thumbnail and/or optional `ffplay`); prompt `[a]ccept/[e]dit/[s]kip/[q]uit`; on accept/edit call `approve(...)` with `datetime.now(timezone.utc).isoformat()` and persist via `save_catalog` (rewrite) after each approval. `--catalog`, `--media-root` args.
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2:** Keep the shell thin — all decision logic already tested in Task 11. A tiny test that `--help` parses; the interactive loop is manual.
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3:** Commit `feat: interactive review CLI`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 13: End-to-end mocked integration + opt-in real-ffprobe
|
|
|
|
|
|
|
|
|
|
**Files:** `tests/test_tools_integration.py`
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1: Hermetic end-to-end** (spec §10 test 8): fake fetcher → `Candidate`; fake downloader + fake prober + `HeuristicProposer` → `ingest_candidate` appends a `proposed` record; `proposed_records` + `approve` flips it; `load_catalog` + `validate_catalog` pass; `select(loaded, coord, mode, approved_only=True)` returns it.
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2: Opt-in real-ffprobe test** — guarded by `shutil.which("ffprobe")` (`pytest.mark.skipif` when absent): generate a 1-second test clip with `ffmpeg lavfi` into `tmp_path`, probe it, assert `mode`/`duration_s`/`resolution`. Same pattern for a real `compute_dominant_color`.
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 3:** Run the FULL suite `python -m pytest -q` → green. Commit `test: end-to-end ingest+review integration (+ opt-in ffprobe)`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### Task 14: Docs — extend the User Guide for the tools
|
|
|
|
|
|
|
|
|
|
**Files:** `docs/USER_GUIDE.md`
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 1:** Add an "Ingesting & reviewing media" section: prerequisites (`ffmpeg`/`ffprobe`), `--media-root`, the first-ship archives, `python -m tools.ingest_cli ...` examples, the review walk, the Freesound token note (env only), and that `dominant_color` is optional/opt-in. Update the scope banner (no longer "only catalog core is built"). Reconcile the absolute-vs-relative `file_path` note (spec §6.3).
|
|
|
|
|
|
|
|
|
|
- [ ] **Step 2:** Commit `docs: user guide for ingest + review tools`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Done criteria
|
|
|
|
|
|
|
|
|
|
- `python -m pytest -q` is green from the repo root inside `.venv` (prior 37 + all new unit tests; opt-in ffprobe tests skip cleanly when binaries are absent).
|
|
|
|
|
- `hef.catalog` gains exactly `validate_catalog` and `index_by_id`; every pre-existing symbol/behavior is unchanged.
|
|
|
|
|
- `python -m tools.ingest_cli <archive> --query ...` against LibriVox / NASA / Internet Archive produces `proposed` records with mechanical fields filled and a one-line rationale.
|
|
|
|
|
- `python -m tools.review_cli` walks the `proposed` records (coords + rationale + preview frame) and flips accepted ones to `approved` with a `reviewed_at` stamp; `load_catalog` + `validate_catalog` validate the result.
|
|
|
|
|
- `dominant_color` is computed only with `--dominant-color`; Freesound/Musopen/FMA fetchers are explicit deferred stubs (raise `NotImplementedError`).
|
|
|
|
|
|
|
|
|
|
This populated, human-reviewed catalog is what sub-project 3 (the Pi player) drives to the single panoramic projector.
|
Ben Stull