rfc-app/docs/design/plans/2026-06-07-slice1-sidecar-storage.md

SLICE-1 plan — sidecar storage + dual-read + migration + malformed flag

Just-in-time implementation plan for SLICE-1 of Configurable Collection Metadata (§7.2). Authored at the start of the SLICE-1 coding session (session 0084, 2026-06-07), against the code SLICE-0 (v0.46.2) landed.

Scope (and non-scope)

In: the storage/compat layer only — sidecar files become the source of truth for entry metadata, with a dual-read parser, an idempotent frontmatter→sidecar migration tool, and a derived metadata_malformed flag.

Out (later slices): the .collection.yaml fields: schema + validation (SLICE-2), faceted filtering (SLICE-3), the edit/bulk UIs (SLICE-4/5). SLICE-1 maps sidecar values onto the existing typed cached_rfcs columns; it does not add per-field schema columns or facet aggregation.

Invariants honored

• INV-6 dual-read: parser reads the sidecar if present, else legacy top-of-doc frontmatter, with identical resulting in-memory records.
• INV-7 unknown keys ride along: preserved through parse→serialize and through the migration (never dropped).
• INV-2: a migrated .md body contains no metadata.
• INV-1: cached_rfcs stays a derived, rebuildable index; the sidecar in git is the source of truth.
• INV-3: bad metadata never hard-fails a read — the entry still loads and the catalog flags it (metadata_malformed).
• INV-5 / byte-identity: a collection with no sidecars behaves exactly as today (legacy frontmatter path); existing entries load identically.

Components

1. app/entry.py — unknown-key preservation (INV-7). Add extra: dict[str, Any] to Entry. parse() collects frontmatter keys outside the known set into extra; serialize() re-emits them after the known keys. Makes frontmatter round-trips lossless.

2. app/metadata.py — new module (sidecar concerns).

   • SIDECAR_SUFFIX = ".meta.yaml"; sidecar_name(slug), is_sidecar(name), slug_of_sidecar(name).
   • metadata_dict(entry) -> dict — the full metadata mapping (known emit-rules + extra), shared by the sidecar writer and the frontmatter serializer.
   • sidecar_yaml(entry) -> str — canonical YAML for a sidecar from metadata_dict.
   • strip_frontmatter(md_text) -> str — body-only (drops a leading ---…--- block if present; whole text otherwise).
   • parse_sidecar(text) -> tuple[dict, bool] — lenient: (values, malformed); non-mapping / YAML error → ({}, True).
   • read_entry(md_text, sidecar_text|None) -> tuple[Entry, bool] — dual-read: sidecar present → metadata from sidecar values, body from strip_frontmatter(md_text), malformed from parse_sidecar; absent → entry.parse(md_text), malformed=False.

3. app/gitea.py — change_files(...) batch commit. POST /repos/{owner}/{repo}/contents (Gitea ChangeFiles) with a files[] array of {operation, path, content(b64), sha?} — one commit for N files. Backs the migration's "one commit per collection".

4. metadata.migrate_collection(gitea, org, repo, subfolder, actor). Lists <subfolder>/rfcs; for each <slug>.md without a <slug>.meta.yaml sibling and with legacy frontmatter, batch: create the sidecar (metadata_dict → YAML) + update the .md to body-only. One ChangeFiles commit per collection. Idempotent (skip entries already migrated; no-op when none remain). Returns a summary (migrated, skipped, committed).

   Deferred (decided mid-slice, after code review): the operator trigger for this tool (an Owner-gated endpoint) is held back to SLICE-4. The propose/graduate/mark-reviewed/edit write paths still entry.parse the .md directly, so migrating a corpus to body-only .mds before those paths are sidecar-aware would break them (crash / re-introduce frontmatter). SLICE-1 ships the tool as tested groundwork; SLICE-4 makes the write paths sidecar-aware (and adds lazy migration) and is where the trigger belongs (INV-8: engine write paths unchanged this slice).

5. app/cache.py — dual-read in _refresh_collection_corpus. Build a sidecar-by-stem map from the dir listing; for each .md, read its sidecar sibling (if any), metadata.read_entry(...), thread malformed into _upsert_cached_rfc(metadata_malformed=…).

6. backend/migrations/033_metadata_malformed.sql — additive ALTER TABLE cached_rfcs ADD COLUMN metadata_malformed INTEGER NOT NULL DEFAULT 0.

7. app/api.py — surface the flag. Add metadata_malformed (bool) to the two catalog list dicts and get_rfc/_get_rfc_for_collection. (Frontend badge + ?malformed= filter are SLICE-3.)

Tests (TDD — write first)

• test_metadata.py (unit, pure): dual-read equivalence (sidecar vs legacy → identical Entry); unknown-key preservation through parse→serialize and through metadata_dict; strip_frontmatter (with/without frontmatter); parse_sidecar malformed cases.
• test_metadata_migration.py (integration, FakeGitea): migrate a collection → sidecars written + .md bodies stripped + one commit; idempotent (second run is a no-op); unknown keys preserved in the sidecar.
• extend the cache/propose vertical: a collection with a sidecar mirrors from the sidecar; a malformed sidecar sets metadata_malformed and still loads the entry (INV-3); a no-sidecar collection is byte-identical to today.

Release

Minor bump 0.46.2 → 0.47.0 (new functionality: sidecar storage + migration tool; non-breaking — additive migration 033, dual-read keeps legacy corpora working, opt-in). §20 CHANGELOG + upgrade-steps: migration 033 auto-applies; running the migration tool per collection is optional (MAY).