Sub-project 2: ingest & tagging / review tools #1

2026-06-04T14:16:20Z

benstull commented

2026-06-04 14:16:20 +00:00

Summary

Sub-project 2 — the tools/ ingest & tagging / review pipeline (spec/plan: docs/superpowers/{specs,plans}/2026-06-04-ingest-tagging-review-tools.md).

Assisted draft-then-review pipeline under tools/: per-archive ingest fetchers, mechanical tagging (ffprobe/ffmpeg + origin license), heuristic coordinate drafting (review_status=proposed + rationale), and an interactive review CLI that flips records to approved with a reviewed_at stamp.
First-ship fetchers: LibriVox, NASA, Internet Archive; Musopen/FMA/Freesound are explicit deferred stubs.
Only hef.catalog change is additive: validate_catalog + index_by_id (unique-id integrity). No existing symbol changed.
dominant_color is opt-in (ffmpeg-only); media is gitignored.

Test Plan

python -m pytest -q green: 111 passed, 2 skipped (opt-in real-ffprobe tests skip cleanly without binaries).
Unit suite hermetic — HTTP, ffprobe/ffmpeg, downloads all injected/faked.
tools.ingest_cli / tools.review_cli parse; existing catalog still validates under validate_catalog.

🤖 Generated with Claude Code

## Summary Sub-project 2 — the `tools/` ingest & tagging / review pipeline (spec/plan: docs/superpowers/{specs,plans}/2026-06-04-ingest-tagging-review-tools.md). - Assisted **draft-then-review** pipeline under `tools/`: per-archive ingest fetchers, mechanical tagging (ffprobe/ffmpeg + origin license), heuristic coordinate drafting (`review_status=proposed` + rationale), and an interactive review CLI that flips records to `approved` with a `reviewed_at` stamp. - First-ship fetchers: **LibriVox, NASA, Internet Archive**; Musopen/FMA/Freesound are explicit deferred stubs. - Only `hef.catalog` change is additive: `validate_catalog` + `index_by_id` (unique-id integrity). No existing symbol changed. - `dominant_color` is opt-in (ffmpeg-only); media is gitignored. ## Test Plan - [x] `python -m pytest -q` green: **111 passed, 2 skipped** (opt-in real-ffprobe tests skip cleanly without binaries). - [x] Unit suite hermetic — HTTP, ffprobe/ffmpeg, downloads all injected/faked. - [x] `tools.ingest_cli` / `tools.review_cli` parse; existing catalog still validates under `validate_catalog`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

benstull added 14 commits 2026-06-04 14:16:21 +00:00

chore: scaffold tools/ package + stdlib http client db4f036729

Per sub-project-2 plan Task 1. Adds tools/ to setuptools packages,
gitignores media/, and a minimal injectable urllib HTTP client.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: catalog-level validate_catalog + index_by_id (unique ids) b6fb635ef4

Per sub-project-2 plan Task 2 / spec §3. Purely additive to hef.catalog;
no existing symbol changes. validate_catalog runs per-record validate() then
asserts id uniqueness; index_by_id gives by-id lookup the player also wants.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: ffprobe wrapper (tools.probe) 19104f94cd

Per sub-project-2 plan Task 3 / spec §5.1. Parses ffprobe JSON into a Probe
(streams + format); subprocess runner injectable so tests never shell out.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: mechanical mode/duration/resolution tagging 542b5a5641

Per sub-project-2 plan Task 4 / spec §5.1. derive_tags(Probe) -> Tags with the
attached_pic cover-art guard so an art-bearing audio file stays mode=audio.
Duration falls back to the longest stream when format.duration is absent, and
rounds half-up so 12.5s -> 13s (Python's round() would give 12 via banker's
rounding) to match the spec's stated behavior.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: ffmpeg frame extraction + optional dominant_color d2a97823f5

Per sub-project-2 plan Task 5 / spec §5.2. ffmpeg-only dominant color (no image
library), opt-in by design; extract_frame for review previews. Runners injectable
so the unit suite never shells out.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: per-archive license normalization 222421e773

Per sub-project-2 plan Task 6 / spec §5.3. normalize_license maps CC URLs /
identifiers and public-domain markers to the LICENSES vocab, builds non-empty
attribution for cc_by/cc_by_nc, and rejects unmappable licenses at ingest.
librivox_license() helper returns public_domain.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: heuristic coordinate proposer (drafting) 3d5c34491c

Per sub-project-2 plan Task 7 / spec §7. HeuristicProposer seeds the brain plane
from archive priors and the mood plane from title/description keyword nudges,
clamps to 0..4, and emits a one-line rationale. Deterministic, no I/O, no ML
(honors design §11) — only a DRAFT a human reviews.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: ingest pipeline (Candidate/Fetcher/ingest_candidate) 2db2b7ac16

Per sub-project-2 plan Task 8 / spec §6. ingest_candidate dedupes, downloads,
mechanically tags, drafts coordinates, builds a proposed Record and appends it;
ingest_search loops over fetcher hits. All boundaries (download/prober/color_fn)
injectable, so the pipeline is fully hermetic.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: LibriVox/NASA/Internet Archive fetchers (+ deferred stubs) cecc5a0f61

Per sub-project-2 plan Task 9 / spec §6.4. Three keyless first-ship fetchers
parse documented JSON APIs via an injected HttpClient; license/attribution go
through tools.licensing; NASA third-party + IA no-license cases are flagged in
notes for review. musopen/fma/freesound are explicit deferred stubs raising
NotImplementedError; freesound documents FREESOUND_API_TOKEN (secret, env-only).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: ingest CLI entry point 1d7f821ab8

Per sub-project-2 plan Task 10. argparse entry wiring named fetcher +
HeuristicProposer into ingest_search/ingest_candidate; --query/--resolve,
--limit, --catalog, --media-root (env HEF_MEDIA_ROOT), --dominant-color.
Unknown archive -> exit 2, deferred archive -> exit 3. Secrets env-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: review transition core (proposed -> approved) 3e07e86187

Per sub-project-2 plan Task 11 / spec §8.1. proposed_records filter and approve()
return an approved copy via dataclasses.replace (no in-place mutation), with
optional coordinate/rationale override and an injected reviewed_at timestamp.
Pure, no I/O, no clock.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat: interactive review CLI 376edd7819

Per sub-project-2 plan Task 12 / spec §8.2. Walks proposed records (fields +
coords + rationale + best-effort ffmpeg preview), prompts accept/edit/skip/quit,
and persists each approval via save_catalog rewrite. I/O seams (input_fn/now_fn/
out) and --no-preview make the walk testable hermetically; decision logic lives
in the tested review core.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test: end-to-end ingest+review integration (+ opt-in ffprobe) deb26b2575

Per sub-project-2 plan Task 13 / spec §10 test 8. Hermetic e2e: fake fetcher ->
ingest_candidate -> proposed record; approve flips it; reload + validate_catalog;
select(approved_only=True) finds it. Opt-in real-ffprobe/ffmpeg tests generate a
lavfi clip and assert mode/duration/resolution + dominant color, skipped when the
binaries are absent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: user guide for ingest + review tools 1e2a4dd966

Per sub-project-2 plan Task 14. Updates the scope banner (catalog core + tools
now built), adds an 'Ingesting & reviewing media' section (ffmpeg prereqs,
--media-root, first-ship archives, ingest_cli/review_cli usage, Freesound token
note, opt-in dominant_color), and reconciles the absolute-vs-relative file_path
note (spec §6.3).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

benstull merged commit d7a2cbaa40 into main

2026-06-04 14:16:28 +00:00

benstull deleted branch sub-project-2-ingest-tagging-review

2026-06-04 14:16:28 +00:00

benstull referenced this issue from a commit

2026-06-04 14:33:34 +00:00

docs(roadmap): mark sub-project 2 done, sub-project 3 next

benstull referenced this pull request

2026-06-04 14:33:44 +00:00

docs(roadmap): mark sub-project 2 done, sub-project 3 next #2

benstull referenced this pull request

2026-06-26 12:09:00 +00:00