# SLICE-1 plan — sidecar storage + dual-read + migration + malformed flag

Just-in-time implementation plan for **SLICE-1** of
[Configurable Collection Metadata](../2026-06-06-configurable-collection-metadata.md)
(§7.2). Authored at the start of the SLICE-1 coding session (session 0084,
2026-06-07), against the code SLICE-0 (v0.46.2) landed.

## Scope (and non-scope)

**In:** the storage/compat layer only — sidecar files become the source of
truth for entry metadata, with a dual-read parser, an idempotent
frontmatter→sidecar migration tool, and a derived `metadata_malformed` flag.

**Out (later slices):** the `.collection.yaml` `fields:` schema + validation
(SLICE-2), faceted filtering (SLICE-3), the edit/bulk UIs (SLICE-4/5). SLICE-1
maps sidecar values onto the **existing** typed `cached_rfcs` columns; it does
not add per-field schema columns or facet aggregation.

## Invariants honored

- **INV-6 dual-read:** parser reads the sidecar if present, else legacy
  top-of-doc frontmatter, with identical resulting in-memory records.
- **INV-7 unknown keys ride along:** preserved through parse→serialize and
  through the migration (never dropped).
- **INV-2:** a migrated `.md` body contains no metadata.
- **INV-1:** `cached_rfcs` stays a derived, rebuildable index; the sidecar in
  git is the source of truth.
- **INV-3:** bad metadata never hard-fails a read — the entry still loads and
  the catalog flags it (`metadata_malformed`).
- **INV-5 / byte-identity:** a collection with no sidecars behaves exactly as
  today (legacy frontmatter path); existing entries load identically.

## Components

1. **`app/entry.py` — unknown-key preservation (INV-7).** Add
   `extra: dict[str, Any]` to `Entry`. `parse()` collects frontmatter keys
   outside the known set into `extra`; `serialize()` re-emits them after the
   known keys. Makes frontmatter round-trips lossless.

2. **`app/metadata.py` — new module (sidecar concerns).**
   - `SIDECAR_SUFFIX = ".meta.yaml"`; `sidecar_name(slug)`,
     `is_sidecar(name)`, `slug_of_sidecar(name)`.
   - `metadata_dict(entry) -> dict` — the full metadata mapping (known
     emit-rules + `extra`), shared by the sidecar writer and the frontmatter
     serializer.
   - `sidecar_yaml(entry) -> str` — canonical YAML for a sidecar from
     `metadata_dict`.
   - `strip_frontmatter(md_text) -> str` — body-only (drops a leading
     `---…---` block if present; whole text otherwise).
   - `parse_sidecar(text) -> tuple[dict, bool]` — lenient: `(values, malformed)`;
     non-mapping / YAML error → `({}, True)`.
   - `read_entry(md_text, sidecar_text|None) -> tuple[Entry, bool]` — dual-read:
     sidecar present → metadata from sidecar values, body from
     `strip_frontmatter(md_text)`, `malformed` from `parse_sidecar`; absent →
     `entry.parse(md_text)`, `malformed=False`.

3. **`app/gitea.py` — `change_files(...)` batch commit.** `POST
   /repos/{owner}/{repo}/contents` (Gitea ChangeFiles) with a `files[]` array
   of `{operation, path, content(b64), sha?}` — one commit for N files. Backs
   the migration's "one commit per collection".

4. **`metadata.migrate_collection(gitea, org, repo, subfolder, actor)`.**
   Lists `<subfolder>/rfcs`; for each `<slug>.md` **without** a `<slug>.meta.yaml`
   sibling and **with** legacy frontmatter, batch: create the sidecar
   (`metadata_dict` → YAML) + update the `.md` to body-only. One ChangeFiles
   commit per collection. Idempotent (skip entries already migrated; no-op when
   none remain). Returns a summary (`migrated`, `skipped`, `committed`).

   > **Deferred (decided mid-slice, after code review):** the **operator
   > trigger** for this tool (an Owner-gated endpoint) is held back to SLICE-4.
   > The propose/graduate/mark-reviewed/edit write paths still `entry.parse` the
   > `.md` directly, so migrating a corpus to body-only `.md`s before those
   > paths are sidecar-aware would break them (crash / re-introduce
   > frontmatter). SLICE-1 ships the tool as tested groundwork; SLICE-4 makes
   > the write paths sidecar-aware (and adds lazy migration) and is where the
   > trigger belongs (INV-8: engine write paths unchanged this slice).

5. **`app/cache.py` — dual-read in `_refresh_collection_corpus`.** Build a
   sidecar-by-stem map from the dir listing; for each `.md`, read its sidecar
   sibling (if any), `metadata.read_entry(...)`, thread `malformed` into
   `_upsert_cached_rfc(metadata_malformed=…)`.

6. **`backend/migrations/033_metadata_malformed.sql`** — additive
   `ALTER TABLE cached_rfcs ADD COLUMN metadata_malformed INTEGER NOT NULL DEFAULT 0`.

7. **`app/api.py` — surface the flag.** Add `metadata_malformed` (bool) to the
   two catalog list dicts and `get_rfc`/`_get_rfc_for_collection`. (Frontend
   badge + `?malformed=` filter are SLICE-3.)

## Tests (TDD — write first)

- `test_metadata.py` (unit, pure): dual-read equivalence (sidecar vs legacy →
  identical Entry); unknown-key preservation through parse→serialize and
  through `metadata_dict`; `strip_frontmatter` (with/without frontmatter);
  `parse_sidecar` malformed cases.
- `test_metadata_migration.py` (integration, FakeGitea): migrate a collection
  → sidecars written + `.md` bodies stripped + one commit; **idempotent**
  (second run is a no-op); unknown keys preserved in the sidecar.
- extend the cache/propose vertical: a collection with a sidecar mirrors from
  the sidecar; a malformed sidecar sets `metadata_malformed` and still loads
  the entry (INV-3); a no-sidecar collection is byte-identical to today.

## Release

Minor bump **0.46.2 → 0.47.0** (new functionality: sidecar storage + migration
tool; non-breaking — additive migration 033, dual-read keeps legacy corpora
working, opt-in). §20 CHANGELOG + upgrade-steps: migration 033 auto-applies;
running the migration tool per collection is optional (**MAY**).