Ben Stull
f05ee59763
§22.4a SLICE-1 of docs/design/2026-06-06-configurable-collection-metadata.md (§7.2). Entry metadata can live in a per-entry `<slug>.meta.yaml` sidecar with the `.md` kept as pure prose (INV-2). Additive and non-breaking — with no sidecars present every corpus stays on the legacy frontmatter path, byte-identical (N=1 unchanged). - Dual-read (app/metadata.py `read_entry`) — sidecar-else-legacy-frontmatter, identical records (INV-6); unknown/forward-compat keys ride along through parse→serialize and migration (INV-7, `Entry.extra`). A degenerate sidecar (malformed/empty/slug-less) never drops the entry — slug backstopped from the filename stem, flagged not lost (INV-3). - Migration tool (`metadata.migrate_collection`) — idempotent, one ChangeFiles commit per collection (new `gitea.change_files`). Tested as a function; its Owner-gated operator trigger is DEFERRED to SLICE-4 (write paths must become sidecar-aware first — see the design's SLICE-4 note + INV-8). No production trigger ships here, so no corpus is rewritten. - Malformed flag — migration 033 adds `cached_rfcs.metadata_malformed` (additive); the corpus mirror derives it; catalog list + entry-detail APIs surface `metadata_malformed`. - INV-7 at graduation — graduation now carries `Entry.extra` through the rebuild instead of dropping forward-compat keys. Gate: backend 575 passed (28 new: test_metadata / _migration / _cache + graduation extra-preservation). Frontend untouched. CHANGELOG 0.47.0 + upgrade-steps; VERSION + frontend/package.json -> 0.47.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
109 lines
5.8 KiB
Markdown
109 lines
5.8 KiB
Markdown
# SLICE-1 plan — sidecar storage + dual-read + migration + malformed flag
|
|
|
|
Just-in-time implementation plan for **SLICE-1** of
|
|
[Configurable Collection Metadata](../2026-06-06-configurable-collection-metadata.md)
|
|
(§7.2). Authored at the start of the SLICE-1 coding session (session 0084,
|
|
2026-06-07), against the code SLICE-0 (v0.46.2) landed.
|
|
|
|
## Scope (and non-scope)
|
|
|
|
**In:** the storage/compat layer only — sidecar files become the source of
|
|
truth for entry metadata, with a dual-read parser, an idempotent
|
|
frontmatter→sidecar migration tool, and a derived `metadata_malformed` flag.
|
|
|
|
**Out (later slices):** the `.collection.yaml` `fields:` schema + validation
|
|
(SLICE-2), faceted filtering (SLICE-3), the edit/bulk UIs (SLICE-4/5). SLICE-1
|
|
maps sidecar values onto the **existing** typed `cached_rfcs` columns; it does
|
|
not add per-field schema columns or facet aggregation.
|
|
|
|
## Invariants honored
|
|
|
|
- **INV-6 dual-read:** parser reads the sidecar if present, else legacy
|
|
top-of-doc frontmatter, with identical resulting in-memory records.
|
|
- **INV-7 unknown keys ride along:** preserved through parse→serialize and
|
|
through the migration (never dropped).
|
|
- **INV-2:** a migrated `.md` body contains no metadata.
|
|
- **INV-1:** `cached_rfcs` stays a derived, rebuildable index; the sidecar in
|
|
git is the source of truth.
|
|
- **INV-3:** bad metadata never hard-fails a read — the entry still loads and
|
|
the catalog flags it (`metadata_malformed`).
|
|
- **INV-5 / byte-identity:** a collection with no sidecars behaves exactly as
|
|
today (legacy frontmatter path); existing entries load identically.
|
|
|
|
## Components
|
|
|
|
1. **`app/entry.py` — unknown-key preservation (INV-7).** Add
|
|
`extra: dict[str, Any]` to `Entry`. `parse()` collects frontmatter keys
|
|
outside the known set into `extra`; `serialize()` re-emits them after the
|
|
known keys. Makes frontmatter round-trips lossless.
|
|
|
|
2. **`app/metadata.py` — new module (sidecar concerns).**
|
|
- `SIDECAR_SUFFIX = ".meta.yaml"`; `sidecar_name(slug)`,
|
|
`is_sidecar(name)`, `slug_of_sidecar(name)`.
|
|
- `metadata_dict(entry) -> dict` — the full metadata mapping (known
|
|
emit-rules + `extra`), shared by the sidecar writer and the frontmatter
|
|
serializer.
|
|
- `sidecar_yaml(entry) -> str` — canonical YAML for a sidecar from
|
|
`metadata_dict`.
|
|
- `strip_frontmatter(md_text) -> str` — body-only (drops a leading
|
|
`---…---` block if present; whole text otherwise).
|
|
- `parse_sidecar(text) -> tuple[dict, bool]` — lenient: `(values, malformed)`;
|
|
non-mapping / YAML error → `({}, True)`.
|
|
- `read_entry(md_text, sidecar_text|None) -> tuple[Entry, bool]` — dual-read:
|
|
sidecar present → metadata from sidecar values, body from
|
|
`strip_frontmatter(md_text)`, `malformed` from `parse_sidecar`; absent →
|
|
`entry.parse(md_text)`, `malformed=False`.
|
|
|
|
3. **`app/gitea.py` — `change_files(...)` batch commit.** `POST
|
|
/repos/{owner}/{repo}/contents` (Gitea ChangeFiles) with a `files[]` array
|
|
of `{operation, path, content(b64), sha?}` — one commit for N files. Backs
|
|
the migration's "one commit per collection".
|
|
|
|
4. **`metadata.migrate_collection(gitea, org, repo, subfolder, actor)`.**
|
|
Lists `<subfolder>/rfcs`; for each `<slug>.md` **without** a `<slug>.meta.yaml`
|
|
sibling and **with** legacy frontmatter, batch: create the sidecar
|
|
(`metadata_dict` → YAML) + update the `.md` to body-only. One ChangeFiles
|
|
commit per collection. Idempotent (skip entries already migrated; no-op when
|
|
none remain). Returns a summary (`migrated`, `skipped`, `committed`).
|
|
|
|
> **Deferred (decided mid-slice, after code review):** the **operator
|
|
> trigger** for this tool (an Owner-gated endpoint) is held back to SLICE-4.
|
|
> The propose/graduate/mark-reviewed/edit write paths still `entry.parse` the
|
|
> `.md` directly, so migrating a corpus to body-only `.md`s before those
|
|
> paths are sidecar-aware would break them (crash / re-introduce
|
|
> frontmatter). SLICE-1 ships the tool as tested groundwork; SLICE-4 makes
|
|
> the write paths sidecar-aware (and adds lazy migration) and is where the
|
|
> trigger belongs (INV-8: engine write paths unchanged this slice).
|
|
|
|
5. **`app/cache.py` — dual-read in `_refresh_collection_corpus`.** Build a
|
|
sidecar-by-stem map from the dir listing; for each `.md`, read its sidecar
|
|
sibling (if any), `metadata.read_entry(...)`, thread `malformed` into
|
|
`_upsert_cached_rfc(metadata_malformed=…)`.
|
|
|
|
6. **`backend/migrations/033_metadata_malformed.sql`** — additive
|
|
`ALTER TABLE cached_rfcs ADD COLUMN metadata_malformed INTEGER NOT NULL DEFAULT 0`.
|
|
|
|
7. **`app/api.py` — surface the flag.** Add `metadata_malformed` (bool) to the
|
|
two catalog list dicts and `get_rfc`/`_get_rfc_for_collection`. (Frontend
|
|
badge + `?malformed=` filter are SLICE-3.)
|
|
|
|
## Tests (TDD — write first)
|
|
|
|
- `test_metadata.py` (unit, pure): dual-read equivalence (sidecar vs legacy →
|
|
identical Entry); unknown-key preservation through parse→serialize and
|
|
through `metadata_dict`; `strip_frontmatter` (with/without frontmatter);
|
|
`parse_sidecar` malformed cases.
|
|
- `test_metadata_migration.py` (integration, FakeGitea): migrate a collection
|
|
→ sidecars written + `.md` bodies stripped + one commit; **idempotent**
|
|
(second run is a no-op); unknown keys preserved in the sidecar.
|
|
- extend the cache/propose vertical: a collection with a sidecar mirrors from
|
|
the sidecar; a malformed sidecar sets `metadata_malformed` and still loads
|
|
the entry (INV-3); a no-sidecar collection is byte-identical to today.
|
|
|
|
## Release
|
|
|
|
Minor bump **0.46.2 → 0.47.0** (new functionality: sidecar storage + migration
|
|
tool; non-breaking — additive migration 033, dual-read keeps legacy corpora
|
|
working, opt-in). §20 CHANGELOG + upgrade-steps: migration 033 auto-applies;
|
|
running the migration tool per collection is optional (**MAY**).
|