spec: sub-project 2 — ingest & tagging / review tools

Develops the draft-then-review pipeline spec (design §5/§8, ROADMAP §2): per-archive fetchers, mechanical tagging (ffprobe mode/duration/resolution, ffmpeg dominant_color, origin license/attribution), heuristic coordinate drafting (review_status=proposed + one-line rationale), and the review CLI (walk proposed -> approve + reviewed_at). Builds on hef.catalog additively: schema unchanged (Record defaults already match the proposed shape); the only required core addition is a catalog-level validate_catalog (unique-id integrity), with optional index_by_id. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 05:45:37 -07:00
parent 5e4b79486a
commit f80ec65553
1 changed files with 535 additions and 0 deletions
@@ -0,0 +1,535 @@
+# Sub-project 2 — Ingest & Tagging / Review Tools — Spec
+
+**Date:** 2026-06-04
+**Status:** Proposed (sub-project 2 of 5) — pending operator approval
+**Repo:** `human-experience-filter-art`
+**Design reference:** [`2026-06-04-human-experience-filter-design.md`](./2026-06-04-human-experience-filter-design.md) (esp. §4 catalog, §5 division-of-labor, §8 sourcing)
+**Roadmap item:** [`docs/ROADMAP.md`](../../ROADMAP.md) §2
+**Builds on:** sub-project 1 — `hef.catalog` + `hef.selection` (done, merged, 37 tests)
+
+---
+
+## 1. What this is
+
+Sub-project 1 gave us the **spine**: a validated JSONL catalog (`hef.catalog`) and
+a nearest-match selector (`hef.selection`). Today the only way to fill that
+catalog is hand-authoring records (per [`docs/USER_GUIDE.md`](../../USER_GUIDE.md)).
+
+Sub-project 2 builds the **assisted *draft-then-review* pipeline** the design spec
+calls for (§5), under `tools/`:
+
+1. **Ingest** — per-archive fetchers pull a candidate piece from a public-domain
+   pool and write a catalog record.
+2. **Mechanical tagging** — tooling auto-fills the non-curatorial fields: `mode`
+   (via `ffprobe`), `license`/`attribution`/`source_*` (from origin),
+   `duration_s`, `resolution`, and `dominant_color` (computed from the video).
+3. **Coordinate drafting** — a proposer suggests `left/right/dark/light` with a
+   one-line `rationale`, written as `review_status: proposed`.
+4. **Review CLI** — walks `proposed` records one at a time (proposed coords +
+   rationale + preview frame), accepts or corrects, flips to `approved`, and
+   stamps `reviewed_at`.
+
+The output is a populated catalog that `load_catalog` validates and the player
+(sub-project 3) can `select()` over. "How far from done" stays exactly what §4
+defines: the count of records still in `review_status: proposed`.
+
+### Design invariants this honors
+
+- **Curation is the artwork** (design §1). The four coordinates are a human
+  curatorial act (design §11: *no automatic ML coordinate tagging*). The pipeline
+  **drafts** coordinates as a starting point, but every coordinate is human-blessed
+  before it can be selected. Drafting is deterministic and rule-based — there is no
+  ML model in the tool.
+- **The catalog is the spine.** Tools only **read** and **append/rewrite** through
+  `hef.catalog`; they never define a parallel record shape.
+- **Local, dependency-light, hermetically testable.** Like sub-project 1, the
+  testable core is pure-stdlib Python. The two external boundaries — `ffmpeg`/`ffprobe`
+  (subprocess) and archive HTTP — are isolated behind injectable seams so the unit
+  suite never shells out or hits the network.
+
+---
+
+## 2. What `hef.catalog` already gives us (and why the schema needs no change)
+
+The headline of the "build on without breaking" constraint: **the `Record` schema
+is already complete for this flow.** Sub-project 1 shipped, with defaults chosen
+for exactly the draft-then-review pipeline:
+
+| Field group | Fields | Who sets it here |
+|---|---|---|
+| Identity / origin | `id`, `title`, `source_url`, `source_archive` | ingest (from the fetcher) |
+| License | `license`, `attribution` | mechanical tagging (from origin) |
+| Mechanical media | `mode`, `duration_s`, `resolution`, `dominant_color` | mechanical tagging (ffprobe/ffmpeg) |
+| Curatorial | `left`, `right`, `dark`, `light` | drafting (proposed) → review (approved) |
+| Review state | `review_status` (default `"proposed"`), `rationale`, `reviewed_at` (default `null`) | drafting sets `rationale`; review sets `reviewed_at` |
+| Misc | `file_path`, `notes` | ingest (`file_path`); either (`notes`) |
+
+`Record`'s defaults — `review_status="proposed"`, `reviewed_at=None`,
+`rationale=""`, plus `attribution`/`resolution`/`dominant_color` defaulting to
+`""` — are precisely an ingested-but-unreviewed record. **Ingest constructs a
+`Record` and the proposed-state shape falls out of the defaults; nothing in the
+schema is added or changed.**
+
+The tools consume these existing symbols unchanged:
+
+- `hef.catalog`: `Record`, `validate`, `record_to_dict`, `record_from_dict`,
+  `load_catalog`, `save_catalog`, `append_record`, `CatalogError`, and the
+  vocab constants `MODES`, `LICENSES`, `REVIEW_STATUSES`, `ATTRIBUTION_LICENSES`,
+  `COORD_FIELDS`, `COORD_MIN`, `COORD_MAX`.
+- `hef.selection`: `Coordinate`, `select` (used by the end-to-end test and by
+  the player later).
+
+---
+
+## 3. The one required addition to `hef.catalog`
+
+Per the constraint, every addition the tools need is called out explicitly. There
+is exactly **one** required change to the shared library, and it is purely
+additive (no existing symbol changes signature or behavior):
+
+### 3.1 `validate_catalog(records)` — catalog-level integrity (unique ids)
+
+`validate()` is per-record; it cannot see the whole file, so it cannot catch a
+**duplicate `id`**. The ingest pipeline appends repeatedly and the review CLI
+looks records up *by id* — both are unsafe if ids collide (ingest silently
+double-adds a re-fetched piece; review can't tell two records apart). So add a
+catalog-level check to the shared spine, where the player also benefits:
+
+```python
+def validate_catalog(records) -> None:
+    """Validate every record AND the cross-record invariants (currently: unique ids).
+    Raises CatalogError naming the first duplicate id."""
+```
+
+- It calls `validate()` on each record, then asserts `id` uniqueness.
+- **Non-breaking:** new symbol; existing `validate`/`load_catalog`/`save_catalog`/
+  `append_record` are untouched. Any catalog valid today (unique ids) stays valid.
+- **Used by tools** at two points: before `append_record` during ingest (reject a
+  candidate whose derived `id` already exists), and after `load_catalog` in the
+  review CLI (fail loudly on a corrupted/duplicated catalog before editing it).
+
+> A companion convenience, `index_by_id(records) -> dict[str, Record]`, may be
+> added alongside it (the review CLI and dedupe both want id-keyed lookup). It is
+> a trivial helper; if we prefer to keep the spine minimal it can live in the
+> tools layer instead. **Recommendation:** add `index_by_id` to `hef.catalog`
+> too, since by-id lookup is a catalog concern the player will also want.
+
+**Everything else this sub-project needs lives in `tools/` and only *consumes*
+`hef.catalog`.** No other change to `hef/` is required. (Considered and
+deliberately *not* added now — see §11: a `source_id` archive-identifier field,
+a `media_sha256`, an `ingested_at` timestamp. They are future schema extensions,
+not needed for the done-criteria.)
+
+---
+
+## 4. Package layout
+
+Extends the existing repo (design §10; the sub-project-1 plan put shared code in
+`hef/` and data in `catalog/`):
+
+```
+tools/
+  __init__.py
+  http.py              # thin stdlib-urllib client: timeout, retry, user-agent (injectable)
+  probe.py             # Prober: ffprobe wrapper -> parsed streams (injectable)
+  mediatools.py        # ffmpeg helpers: extract frame, dominant color, waveform thumb
+  tagging.py           # mechanical tagging: assemble mode/duration/resolution/dominant_color
+  licensing.py         # per-archive license/attribution normalization -> LICENSES vocab
+  drafting.py          # Draft, Proposer protocol, HeuristicProposer
+  ingest/
+    __init__.py
+    base.py            # Candidate dataclass + Fetcher protocol + the ingest pipeline
+    internet_archive.py
+    musopen.py
+    librivox.py
+    nasa.py
+    fma.py             # Free Music Archive
+    freesound.py
+  ingest_cli.py        # `python -m tools.ingest_cli` — run a fetcher, write proposed records
+  review.py            # review state-transition core (pure, tested)
+  review_cli.py        # `python -m tools.review_cli` — interactive walk of proposed records
+```
+
+- `tools/` is added to `[tool.setuptools] packages` in `pyproject.toml`.
+- **Invocation matches the repo convention** (USER_GUIDE: run from repo root, no
+  install, `import hef` resolves via the root `conftest.py`): the CLIs run as
+  `.venv/bin/python -m tools.ingest_cli …` and `… -m tools.review_cli …`.
+  `[project.scripts]` console-script aliases (`hef-ingest`, `hef-review`) may be
+  declared too for an installed checkout, but `python -m` is the documented path.
+- `media/` (download target, §6.3) is added to `.gitignore`. Media never enters
+  git — the repo holds metadata + pointers only (design §4).
+
+---
+
+## 5. Mechanical tagging (`tools/probe.py`, `tools/mediatools.py`, `tools/tagging.py`, `tools/licensing.py`)
+
+All of §5's "mechanical / automatic" fields, set from the downloaded file and the
+origin metadata.
+
+### 5.1 `mode`, `duration_s`, `resolution` — from `ffprobe`
+
+`tools/probe.py` wraps:
+
+```
+ffprobe -v quiet -print_format json -show_format -show_streams <file>
+```
+
+and parses the JSON into a small `Probe` result (streams + format). `tools/tagging.py`
+derives:
+
+- **`mode`** — from which *meaningful* streams exist:
+  - audio stream present, no real video → `"audio"`
+  - real video stream present, no audio → `"video"`
+  - both → `"av"`
+  - **Cover-art guard:** an embedded cover image in an audio file shows up as a
+    video stream (`mjpeg`/`png`, `disposition.attached_pic == 1`, often 1 frame).
+    These are **excluded** when deciding `mode`, so an MP3 with album art is
+    correctly `"audio"`, not `"av"`. This guard is a tested case.
+- **`duration_s`** — `round(float(format.duration))`, falling back to the longest
+  stream `duration` if `format.duration` is absent. Integer (schema requires int ≥ 0).
+- **`resolution`** — `"{width}x{height}"` from the chosen video stream; `""` for
+  audio-only (schema default).
+
+`tools/probe.py` takes the subprocess runner as an injectable dependency
+(default: `subprocess.run`); tests feed canned `ffprobe` JSON and never shell out.
+
+### 5.2 `dominant_color` — computed from the video (`tools/mediatools.py`)
+
+For `video`/`av` records, extract a representative frame mid-segment and reduce it
+to one hex color. **Primary approach is `ffmpeg`-only** (keeps the dependency
+surface to just `ffmpeg`/`ffprobe` — no image library):
+
+```
+# representative frame near the middle, reduced to a single-entry palette
+ffmpeg -v quiet -ss <mid> -i <file> -vf "thumbnail,palettegen=max_colors=1" -frames:v 1 -f rawvideo -pix_fmt rgb24 -
+```
+
+The 3 output bytes are the dominant palette color → `"#rrggbb"`. (A simpler
+`scale=1:1` average-color path is the fallback if `palettegen` is unavailable.)
+Audio-only records keep `dominant_color=""`.
+
+> **Open decision (logged):** the roadmap assumed "an image lib for dominant
+> color." This spec instead recommends the **ffmpeg-only** path so the whole
+> sub-project depends on nothing beyond `ffmpeg`/`ffprobe`. **Pillow** (k-means /
+> `quantize` over a sampled frame) is the documented fallback if ffmpeg-derived
+> color quality proves insufficient in review. Chosen for the lean dependency
+> story; reversible.
+
+`dominant_color` feeds the side walls (sub-project 5, design §7).
+
+### 5.3 `license` / `attribution` / `source_*` — from origin (`tools/licensing.py`)
+
+Each fetcher resolves origin metadata; `tools/licensing.py` normalizes it to the
+`hef.catalog` vocab (`LICENSES = {public_domain, cc0, cc_by, cc_by_nc}`):
+
+- Map CC license URLs/identifiers → `cc0` | `cc_by` | `cc_by_nc`; explicit
+  public-domain / "no known copyright" → `public_domain`.
+- For `cc_by`/`cc_by_nc` (the `ATTRIBUTION_LICENSES`), build the required
+  `attribution` string (creator + license name/URL) — `validate()` already rejects
+  these licenses with empty `attribution`, so the normalizer **must** produce it.
+- Anything that doesn't map to an allowed license is **rejected at ingest** with a
+  clear error (the piece is not defensible per design §8's license stance) — it is
+  never written as an invalid record.
+
+This is the per-archive "license/attribution/source from origin" tagging; it is
+unit-tested with sample metadata payloads per archive.
+
+---
+
+## 6. Ingest (`tools/ingest/`)
+
+### 6.1 The `Candidate` + `Fetcher` seam
+
+Every archive differs in API but funnels into one shape. `tools/ingest/base.py`:
+
+```python
+@dataclass
+class Candidate:
+    source_archive: str      # origin label, e.g. internet_archive | librivox | nasa (§6.4)
+    source_url: str          # human/landing URL recorded in the record
+    media_url: str           # direct download URL of the chosen file
+    title: str
+    license: str             # normalized (tools.licensing) -> hef LICENSES vocab
+    attribution: str         # "" unless the license requires it
+    suggested_id: str        # stable id derived from archive + archive-identifier
+    media_ext: str           # file extension for the download target
+    description: str = ""     # free text used by drafting signals; lands in notes
+```
+
+```python
+class Fetcher(Protocol):
+    archive: str                                  # the source_archive label
+    def search(self, query: str, *, limit: int) -> list[Candidate]: ...
+    def resolve(self, identifier: str) -> Candidate: ...   # one item by id/URL
+```
+
+Fetchers receive an HTTP client (`tools/http.py`) by injection so tests feed canned
+API responses. Fetchers do **not** download or probe — they only resolve metadata.
+
+### 6.2 The ingest pipeline
+
+`tools/ingest/base.py` provides `ingest_candidate(candidate, *, catalog_path, media_root, proposer, prober, downloader)`:
+
+1. **Dedupe.** `load_catalog` + `validate_catalog`; if `candidate.suggested_id`
+   already exists, skip (log "already in catalog") — idempotent re-runs.
+2. **Download.** Fetch `candidate.media_url` → `<media_root>/<archive>/<id>.<ext>`
+   (see §6.3). Skip the download if the file already exists.
+3. **Mechanical tag.** `prober` → `mode`, `duration_s`, `resolution`; `mediatools`
+   → `dominant_color` (video/av only). `licensing` already normalized
+   `license`/`attribution` on the `Candidate`.
+4. **Draft coordinates.** `proposer.propose(signals)` → `Draft(coordinate, rationale)` (§7).
+5. **Build the record.** Construct a `hef.catalog.Record` from the above; the
+   `proposed`/`reviewed_at=None` shape comes from the defaults (§2). `notes` gets
+   `candidate.description`.
+6. **Validate + append.** `validate(record)`, then re-check uniqueness, then
+   `append_record(record, catalog_path)`.
+
+Batch helper `ingest_search(fetcher, query, *, limit, …)` runs steps over each
+search hit. Everything network/subprocess is injected, so the pipeline is tested
+end-to-end with a fake fetcher + fake prober + fake downloader (no I/O).
+
+### 6.3 Download / caching layout (settles a roadmap open decision)
+
+- **Media root:** `--media-root DIR` / env `HEF_MEDIA_ROOT`, default `./media/`
+  (gitignored).
+- **On-disk path:** `<media_root>/<source_archive>/<suggested_id>.<media_ext>`.
+- **`file_path` recorded in the record:** the path **relative to the media root**
+  (e.g. `nasa/nasa-as08-14-2383.mp4`). Relative-to-root is portable: the ingest
+  workstation and the Pi's mounted drive store the same tree under different
+  mounts, and the **player (sub-project 3) joins `file_path` with its own drive
+  mount**. `validate()` does not constrain `file_path` format, so this is
+  compatible with the hand-authoring guide's absolute-path examples.
+
+> **Open decision (logged):** relative-to-media-root vs. absolute `file_path`. This
+> spec recommends **relative-to-root** for portability between the tagging
+> workstation and the Pi. The USER_GUIDE currently shows absolute paths for
+> hand-authoring; both load fine. A doc note will reconcile them; the player spec
+> will define mount resolution. Reversible.
+
+### 6.4 The six archives (all specified; phased delivery)
+
+License normalization and access per pool (design §8 maps these to the axes):
+
+| Archive (`source_archive`) | Access | Typical license | Auth | Notes / risk |
+|---|---|---|---|---|
+| `internet_archive` (incl. Prelinger) | Metadata API `archive.org/metadata/<id>`; direct file URLs | `public_domain` / CC (per item `licenseurl`/`rights`) | none | Ambiguous "no known copyright" → `public_domain`, flagged in `notes` for review |
+| `librivox` | JSON API `librivox.org/api/feed/audiobooks` | `public_domain` (charter) | none | Reader credited in `notes`; attribution not required |
+| `nasa` | Images/video API `images-api.nasa.gov` | `public_domain` | none | NASA media guideline caveat: some items embed third-party content — flag in `notes` |
+| `musopen` | Musopen API / catalog | `public_domain` / CC | possible key | Access historically gated — may need an API key (secret, §9) |
+| `fma` (Free Music Archive) | Track/page resolution | CC (`cc_by`/`cc_by_nc`/`cc0`) | varies | **Highest risk:** the public FMA API has been deprecated/changed; fetcher resolves from a track URL + page metadata |
+| `freesound` | API v2 `freesound.org/apiv2` | `cc0` / `cc_by` / `cc_by_nc` | **token required** | API token is a secret (§9); uploader → `attribution` when license requires |
+
+> **Recommended first-ship set:** the keyless, clean-license, stable-API pools —
+> **LibriVox, NASA, Internet Archive/Prelinger.** Defer the ones with an auth or
+> API-stability tax — **Freesound** (token), **Musopen** (possible gate),
+> **Free Music Archive** (deprecated API) — to a second pass. The `Fetcher` seam
+> means deferring them costs nothing structurally; the spec still defines all six.
+> *(Logged as an open decision for operator confirmation — see §11.)*
+
+---
+
+## 7. Coordinate drafting (`tools/drafting.py`)
+
+The curatorial coordinates are a human act (design §11). The tool's job is to seed
+a **draft** to be reviewed, never to auto-tag authoritatively.
+
+```python
+@dataclass
+class Draft:
+    coordinate: Coordinate   # hef.selection.Coordinate(left, right, dark, light)
+    rationale: str           # one line, cites the signal that drove the guess
+
+@dataclass
+class Signals:
+    title: str
+    description: str
+    source_archive: str
+    mode: str
+    duration_s: int
+
+class Proposer(Protocol):
+    def propose(self, signals: Signals) -> Draft: ...
+```
+
+### 7.1 `HeuristicProposer` (the deterministic baseline)
+
+A rule-based seed grounded in the §8 sourcing→axis map and the §2 rubric — **not
+an ML model** (honors design §11):
+
+- **Archive priors** seed the brain plane: `librivox` → high `left`
+  (spoken/verbal); `internet_archive`/Prelinger → high `left` (educational/
+  industrial) by default; `musopen`/`fma` → high `right` (music); `nasa` → high
+  `right` (wordless awe); `freesound` → high `right` (abstract field recordings).
+- **Keyword nudges** on title+description seed the mood plane: dark words
+  (`storm`, `night`, `noir`, `requiem`, `minor`, `war`, `funeral`, `decay`, …)
+  raise `dark`; light words (`sunrise`, `dawn`, `garden`, `spring`, `joy`, `hope`,
+  `major`, `bright`, …) raise `light`.
+- Values **clamped to 0..4**.
+- `rationale` is one line naming the dominant signal, e.g.
+  `"librivox spoken reading → strong left; 'storm' in title → dark"`.
+
+Deterministic given `Signals` → fully unit-testable.
+
+### 7.2 The assistant seam
+
+`Proposer` is an interface. The `HeuristicProposer` is always-available and
+dependency-free. When a **session assistant** (e.g. Claude, or the operator) does
+the ingest, it can supply a considered coordinate + rationale through the *same*
+seam (a manual proposer that reads a per-candidate draft, or an LLM-backed
+proposer the operator wires in). Either way the record is written
+`review_status="proposed"` and is **still reviewed by a human**. The tool ships
+**no** ML dependency; the "assistant" is an optional, out-of-core seam.
+
+---
+
+## 8. Review CLI (`tools/review.py` + `tools/review_cli.py`)
+
+### 8.1 The transition core (pure, tested) — `tools/review.py`
+
+```python
+def proposed_records(records) -> list[Record]:
+    """Records still awaiting review."""
+
+def approve(record, *, reviewed_at, coordinate=None) -> Record:
+    """Return an approved copy: review_status='approved', reviewed_at set,
+    optionally overriding the four coordinates with a human correction.
+    The returned record is re-validated by the caller before save."""
+```
+
+- `reviewed_at` is an ISO-8601 UTC timestamp string
+  (`datetime.now(timezone.utc).isoformat()`), injected by the CLI so the core stays
+  pure/testable. `Record.reviewed_at` is `Optional[str]` — no schema change.
+- `approve` does not mutate in place gratuitously; it produces the approved record
+  and the CLI persists via load → replace-by-id → `save_catalog` (rewrite). At
+  120–800 records a full rewrite is trivial and crash-resilient when done after
+  each approval.
+
+### 8.2 The interactive walk — `tools/review_cli.py`
+
+`python -m tools.review_cli --catalog catalog/library.jsonl [--media-root ./media]`
+
+1. `load_catalog` → `validate_catalog` (fail loudly on a corrupt catalog before editing).
+2. For each `proposed` record, show:
+   - `id`, `title`, `source_archive`, `source_url`, `license`/`attribution`
+   - `mode`, `duration_s`, `resolution`, `dominant_color`
+   - the **proposed coordinates** `left/right/dark/light` and the one-line `rationale`
+   - a **preview frame**: for `video`/`av`, extract a representative frame (reusing
+     the §5.2 mid-segment frame) to a temp PNG and open it with the OS viewer
+     (`open` on macOS / `xdg-open` on Linux); for `audio`, render a waveform
+     thumbnail (`ffmpeg … showwavespic`) and/or offer optional 10-second playback
+     via `ffplay`. The preview resolves `file_path` against `--media-root`.
+3. Prompt: **[a]ccept · [e]dit coords · [s]kip · [q]uit**.
+   - **accept** → `approve(record, reviewed_at=now)` (keep coords).
+   - **edit** → prompt for new `left/right/dark/light` (and optionally a new
+     `rationale`), then `approve(..., coordinate=corrected)`.
+   - **skip** → leave `proposed`, advance.
+   - **quit** → save and exit.
+4. Persist after each approval (rewrite via `save_catalog`), so an interrupted
+   session keeps its progress.
+
+The preview/prompt shell is thin; the **tested** logic is the §8.1 transition core
+(deterministic, no terminal, no subprocess). Preview rendering reuses
+`tools/mediatools.py` and is exercised only by opt-in integration tests.
+
+---
+
+## 9. Dependencies, configuration, secrets
+
+- **System binaries:** `ffmpeg` + `ffprobe` (subprocess). Required for mechanical
+  tagging, dominant color, and previews. The dominant-color recommendation (§5.2)
+  keeps the dep surface to *just* these two — no Python image library in the core.
+  Pillow is an optional documented fallback only.
+- **HTTP:** stdlib `urllib.request` via `tools/http.py` (timeout, small retry,
+  user-agent) — no `requests` dependency, preserving the sub-project-1 ethos.
+  `internetarchive`/other archive SDKs are explicitly avoided in favor of the
+  documented JSON APIs.
+- **Config:** `--catalog` (default `catalog/library.jsonl`), `--media-root` /
+  `HEF_MEDIA_ROOT` (default `./media/`, gitignored).
+- **Secrets (wgl hard rule — never in catalog, code, or transcript):** Freesound
+  API token via `FREESOUND_API_TOKEN` (env or macOS Keychain); any Musopen/FMA key
+  likewise. Referenced by name only; never written into a record or logged.
+- **Testing:** `pytest` (already a dep). The unit suite stays **hermetic** —
+  `ffprobe`/`ffmpeg`/HTTP are injected and faked; no network, no binaries needed to
+  run the core tests. Integration tests that actually invoke `ffprobe`/`ffmpeg` are
+  opt-in and **skipped** when the binaries are absent.
+- **pyproject:** add `tools` to `[tool.setuptools] packages`; optionally add
+  `[project.scripts]` (`hef-ingest`, `hef-review`) and a `[project.optional-dependencies]`
+  `tools` extra (empty unless Pillow is later adopted).
+
+---
+
+## 10. Testing strategy & done-criteria
+
+Mirrors sub-project 1's TDD discipline (test-first, stdlib, hermetic).
+
+**Unit tests (hermetic):**
+
+1. `hef.catalog.validate_catalog` — accepts unique ids; raises naming a duplicate;
+   (`index_by_id` round-trip if added).
+2. **Mechanical tagging** — canned `ffprobe` JSON → expected `mode`/`duration_s`/
+   `resolution`; the **cover-art `attached_pic` guard** keeps an art-bearing MP3 at
+   `mode="audio"`.
+3. **Dominant color** — canned ffmpeg `rgb24` bytes → expected `"#rrggbb"`;
+   audio-only → `""`.
+4. **License normalization** — sample per-archive metadata → correct `license`;
+   `cc_by`/`cc_by_nc` produce non-empty `attribution`; unmappable license is
+   rejected.
+5. **Fetchers** — canned API JSON via a fake HTTP client → `Candidate` fields and
+   stable `suggested_id` derivation (one test per archive that ships).
+6. **Drafting** — `HeuristicProposer`: archive priors, keyword nudges, 0..4
+   clamping, one-line rationale content.
+7. **Review transition** — `proposed → approved` sets `reviewed_at` + status; edit
+   overrides coordinates; skip leaves `proposed`; result re-validates and
+   round-trips through `save_catalog`/`load_catalog`.
+8. **End-to-end (mocked)** — fake fetcher + fake prober + fake downloader +
+   `HeuristicProposer` → a `proposed` record appended; the review core approves it;
+   `load_catalog` + `validate_catalog` pass; `select(..., approved_only=True)` can
+   pick it.
+
+**Opt-in integration tests** (skipped without binaries): real `ffprobe` on a tiny
+fixture clip → `mode`/`duration`/`resolution`; real dominant-color extraction.
+
+**Done when (from ROADMAP §2):**
+
+- You can run ingest against a source and get `proposed` records.
+- You can review them to `approved`.
+- `load_catalog` (and `validate_catalog`) validate the result.
+- Unit tests on mechanical tagging and the review state transitions pass, and the
+  full suite is green from the repo root inside `.venv`.
+
+---
+
+## 11. Open decisions (recommended defaults; logged for operator)
+
+None block writing the implementation plan; each has a recommended default the
+plan will assume unless the operator overrides.
+
+1. **Dominant-color dependency** — *Recommend:* ffmpeg-only (`palettegen`), no
+   image lib; Pillow as fallback. (Roadmap had assumed an image lib.)
+2. **`file_path` style** — *Recommend:* relative-to-`media-root` for portability;
+   reconcile the USER_GUIDE's absolute examples with a doc note; player spec
+   defines mount resolution.
+3. **Which archives ship first** — *Recommend:* LibriVox, NASA, Internet
+   Archive/Prelinger first; defer Freesound, Musopen, FMA (auth / API-stability tax).
+4. **CLI surface** — *Recommend:* two `python -m` entry points (`tools.ingest_cli`,
+   `tools.review_cli`), matching the no-install repo convention; console-script
+   aliases optional.
+5. **`index_by_id` placement** — *Recommend:* add it to `hef.catalog` alongside
+   `validate_catalog` (the player wants by-id lookup too).
+
+**Deliberately deferred (YAGNI, no schema change now):** a `source_id`
+archive-identifier field, `media_sha256` integrity, an `ingested_at` timestamp, and
+any LLM-backed proposer in the core. All are future, additive extensions; none are
+needed for the done-criteria.
+
+---
+
+## 12. Relationship to the other sub-projects
+
+- **Consumes** sub-project 1 (`hef.catalog`, `hef.selection`) — the only required
+  change is the additive `validate_catalog` (+ optional `index_by_id`), §3.
+- **Feeds** sub-project 3 (the Pi player): a populated, `approved` catalog with
+  `dominant_color` ready for sub-project 5's side walls. The player resolving
+  `file_path` against its drive mount (§6.3) is a player-spec concern.
+- **Settles** two roadmap open decisions (download/caching layout §6.3; ingest
+  language = Python under `tools/`, §4).