Sub-project 2: ingest & tagging / review tools #1

Merged
benstull merged 14 commits from sub-project-2-ingest-tagging-review into main 2026-06-04 14:16:28 +00:00

14 Commits

Author SHA1 Message Date
Ben Stull 1e2a4dd966 docs: user guide for ingest + review tools
Per sub-project-2 plan Task 14. Updates the scope banner (catalog core + tools
now built), adds an 'Ingesting & reviewing media' section (ffmpeg prereqs,
--media-root, first-ship archives, ingest_cli/review_cli usage, Freesound token
note, opt-in dominant_color), and reconciles the absolute-vs-relative file_path
note (spec §6.3).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:36:15 -07:00
Ben Stull deb26b2575 test: end-to-end ingest+review integration (+ opt-in ffprobe)
Per sub-project-2 plan Task 13 / spec §10 test 8. Hermetic e2e: fake fetcher ->
ingest_candidate -> proposed record; approve flips it; reload + validate_catalog;
select(approved_only=True) finds it. Opt-in real-ffprobe/ffmpeg tests generate a
lavfi clip and assert mode/duration/resolution + dominant color, skipped when the
binaries are absent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:35:17 -07:00
Ben Stull 376edd7819 feat: interactive review CLI
Per sub-project-2 plan Task 12 / spec §8.2. Walks proposed records (fields +
coords + rationale + best-effort ffmpeg preview), prompts accept/edit/skip/quit,
and persists each approval via save_catalog rewrite. I/O seams (input_fn/now_fn/
out) and --no-preview make the walk testable hermetically; decision logic lives
in the tested review core.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:34:34 -07:00
Ben Stull 3e07e86187 feat: review transition core (proposed -> approved)
Per sub-project-2 plan Task 11 / spec §8.1. proposed_records filter and approve()
return an approved copy via dataclasses.replace (no in-place mutation), with
optional coordinate/rationale override and an injected reviewed_at timestamp.
Pure, no I/O, no clock.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:33:27 -07:00
Ben Stull 1d7f821ab8 feat: ingest CLI entry point
Per sub-project-2 plan Task 10. argparse entry wiring named fetcher +
HeuristicProposer into ingest_search/ingest_candidate; --query/--resolve,
--limit, --catalog, --media-root (env HEF_MEDIA_ROOT), --dominant-color.
Unknown archive -> exit 2, deferred archive -> exit 3. Secrets env-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:33:01 -07:00
Ben Stull cecc5a0f61 feat: LibriVox/NASA/Internet Archive fetchers (+ deferred stubs)
Per sub-project-2 plan Task 9 / spec §6.4. Three keyless first-ship fetchers
parse documented JSON APIs via an injected HttpClient; license/attribution go
through tools.licensing; NASA third-party + IA no-license cases are flagged in
notes for review. musopen/fma/freesound are explicit deferred stubs raising
NotImplementedError; freesound documents FREESOUND_API_TOKEN (secret, env-only).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:32:18 -07:00
Ben Stull 2db2b7ac16 feat: ingest pipeline (Candidate/Fetcher/ingest_candidate)
Per sub-project-2 plan Task 8 / spec §6. ingest_candidate dedupes, downloads,
mechanically tags, drafts coordinates, builds a proposed Record and appends it;
ingest_search loops over fetcher hits. All boundaries (download/prober/color_fn)
injectable, so the pipeline is fully hermetic.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:29:09 -07:00
Ben Stull 3d5c34491c feat: heuristic coordinate proposer (drafting)
Per sub-project-2 plan Task 7 / spec §7. HeuristicProposer seeds the brain plane
from archive priors and the mood plane from title/description keyword nudges,
clamps to 0..4, and emits a one-line rationale. Deterministic, no I/O, no ML
(honors design §11) — only a DRAFT a human reviews.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:27:37 -07:00
Ben Stull 222421e773 feat: per-archive license normalization
Per sub-project-2 plan Task 6 / spec §5.3. normalize_license maps CC URLs /
identifiers and public-domain markers to the LICENSES vocab, builds non-empty
attribution for cc_by/cc_by_nc, and rejects unmappable licenses at ingest.
librivox_license() helper returns public_domain.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:26:21 -07:00
Ben Stull d2a97823f5 feat: ffmpeg frame extraction + optional dominant_color
Per sub-project-2 plan Task 5 / spec §5.2. ffmpeg-only dominant color (no image
library), opt-in by design; extract_frame for review previews. Runners injectable
so the unit suite never shells out.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:25:11 -07:00
Ben Stull 542b5a5641 feat: mechanical mode/duration/resolution tagging
Per sub-project-2 plan Task 4 / spec §5.1. derive_tags(Probe) -> Tags with the
attached_pic cover-art guard so an art-bearing audio file stays mode=audio.
Duration falls back to the longest stream when format.duration is absent, and
rounds half-up so 12.5s -> 13s (Python's round() would give 12 via banker's
rounding) to match the spec's stated behavior.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:24:33 -07:00
Ben Stull 19104f94cd feat: ffprobe wrapper (tools.probe)
Per sub-project-2 plan Task 3 / spec §5.1. Parses ffprobe JSON into a Probe
(streams + format); subprocess runner injectable so tests never shell out.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:23:24 -07:00
Ben Stull b6fb635ef4 feat: catalog-level validate_catalog + index_by_id (unique ids)
Per sub-project-2 plan Task 2 / spec §3. Purely additive to hef.catalog;
no existing symbol changes. validate_catalog runs per-record validate() then
asserts id uniqueness; index_by_id gives by-id lookup the player also wants.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:23:01 -07:00
Ben Stull db4f036729 chore: scaffold tools/ package + stdlib http client
Per sub-project-2 plan Task 1. Adds tools/ to setuptools packages,
gitignores media/, and a minimal injectable urllib HTTP client.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:22:30 -07:00