v0.47.0 — SLICE-1: metadata sidecars — storage + dual-read + migration tool

§22.4a SLICE-1 of docs/design/2026-06-06-configurable-collection-metadata.md
(§7.2). Entry metadata can live in a per-entry `<slug>.meta.yaml` sidecar with
the `.md` kept as pure prose (INV-2). Additive and non-breaking — with no
sidecars present every corpus stays on the legacy frontmatter path,
byte-identical (N=1 unchanged).

- Dual-read (app/metadata.py `read_entry`) — sidecar-else-legacy-frontmatter,
  identical records (INV-6); unknown/forward-compat keys ride along through
  parse→serialize and migration (INV-7, `Entry.extra`). A degenerate sidecar
  (malformed/empty/slug-less) never drops the entry — slug backstopped from the
  filename stem, flagged not lost (INV-3).
- Migration tool (`metadata.migrate_collection`) — idempotent, one ChangeFiles
  commit per collection (new `gitea.change_files`). Tested as a function; its
  Owner-gated operator trigger is DEFERRED to SLICE-4 (write paths must become
  sidecar-aware first — see the design's SLICE-4 note + INV-8). No production
  trigger ships here, so no corpus is rewritten.
- Malformed flag — migration 033 adds `cached_rfcs.metadata_malformed`
  (additive); the corpus mirror derives it; catalog list + entry-detail APIs
  surface `metadata_malformed`.
- INV-7 at graduation — graduation now carries `Entry.extra` through the rebuild
  instead of dropping forward-compat keys.

Gate: backend 575 passed (28 new: test_metadata / _migration / _cache +
graduation extra-preservation). Frontend untouched. CHANGELOG 0.47.0 +
upgrade-steps; VERSION + frontend/package.json -> 0.47.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ben Stull
2026-06-07 08:24:27 -07:00
parent b1acc2382d
commit f05ee59763
17 changed files with 1025 additions and 11 deletions
+51
View File
@@ -23,6 +23,57 @@ skip versions are the composition of each intervening adjacent
release's steps in order — no A-to-B path is pre-computed beyond release's steps in order — no A-to-B path is pre-computed beyond
that. that.
## 0.47.0 — 2026-06-07
**Minor — metadata sidecars: storage + dual-read + migration tool (§22.4a
SLICE-1).** Entry metadata can now live in a per-entry `<slug>.meta.yaml`
**sidecar**, with the `.md` kept as pure prose (INV-2). The corpus mirror reads
the sidecar when present and falls back to legacy top-of-document frontmatter
otherwise (**dual-read**, INV-6), so existing corpora load byte-identically.
This is SLICE-1 of the
[Configurable Collection Metadata](./docs/design/2026-06-06-configurable-collection-metadata.md)
design (§7.2); the collection `fields:` schema + validation (SLICE-2), faceted
filtering (SLICE-3), and the edit UIs (SLICE-4/5) follow.
Non-breaking and opt-in: a collection with no sidecars behaves exactly as
before, and the §22.13 default collection is untouched (**N=1 sees no change**).
Added:
- **Dual-read parser** (`app/metadata.py`) — `read_entry(md, sidecar)` yields
identical records whether metadata comes from a sidecar or legacy
frontmatter (INV-6); a malformed sidecar never hard-fails a read — the entry
still loads and is flagged (INV-3). Unknown / forward-compat keys ride along
untouched through parse→serialize and the migration (INV-7;
`Entry.extra`).
- **Frontmatter→sidecar migration tool** (`metadata.migrate_collection`) — a
deterministic, idempotent tool that lifts a collection's legacy entries into
sidecars + body-only `.md`s in **one commit** (new Gitea `change_files`
batch); a fully-migrated collection is a no-op. The **operator trigger** for
it (an Owner-gated endpoint) is intentionally **deferred to SLICE-4**: the
propose/graduate/mark-reviewed/edit write paths still read `.md` frontmatter
directly, so they must become sidecar-aware before a corpus is migrated in
production. Until then the tool is shippable groundwork, not yet wired to a
production trigger (INV-8: the engine write paths are unchanged this slice).
- **Malformed-metadata flag** — migration `033_metadata_malformed.sql` adds
`cached_rfcs.metadata_malformed` (additive); the corpus mirror derives it and
the catalog + entry-detail APIs surface `metadata_malformed`. A degenerate
sidecar (malformed / empty / slug-less) never drops the entry — it loads with
its slug backstopped from the filename and is flagged (INV-3).
- **INV-7 at graduation** — graduation now carries an entry's unknown /
forward-compat frontmatter keys through the rebuild instead of dropping them.
### Upgrade steps (0.46.2 → 0.47.0)
- The framework **MUST** apply migration `033_metadata_malformed.sql` — it runs
automatically at startup (additive column, no rebuild, default `0`).
- **No operator action** otherwise. With no sidecars present (the default after
this upgrade) every corpus stays on the legacy frontmatter path, byte-for-byte
as before. The frontmatter→sidecar migration is **not** operator-triggerable
yet (its endpoint lands in SLICE-4); dual-read makes the storage change
invisible until then.
- No config change. No content change is required.
## 0.46.2 — 2026-06-07 ## 0.46.2 — 2026-06-07
**Patch — `SPEC.md` §22.4a contract amendment: entry metadata is **Patch — `SPEC.md` §22.4a contract amendment: entry metadata is
+1 -1
View File
@@ -1 +1 @@
0.46.2 0.47.0
+5 -1
View File
@@ -651,6 +651,7 @@ def make_router(
f""" f"""
SELECT r.slug, r.title, r.state, r.rfc_id, r.repo, SELECT r.slug, r.title, r.state, r.rfc_id, r.repo,
r.owners_json, r.arbiters_json, r.tags_json, r.owners_json, r.arbiters_json, r.tags_json,
r.metadata_malformed,
r.last_main_commit_at, r.last_entry_commit_at, r.updated_at r.last_main_commit_at, r.last_entry_commit_at, r.updated_at
FROM cached_rfcs r JOIN collections c ON c.id = r.collection_id FROM cached_rfcs r JOIN collections c ON c.id = r.collection_id
WHERE r.state IN ('super-draft', 'active') WHERE r.state IN ('super-draft', 'active')
@@ -684,6 +685,7 @@ def make_router(
"last_active_at": r["last_main_commit_at"] or r["last_entry_commit_at"] or r["updated_at"], "last_active_at": r["last_main_commit_at"] or r["last_entry_commit_at"] or r["updated_at"],
"starred_by_me": r["slug"] in starred, "starred_by_me": r["slug"] in starred,
"has_open_prs": False, # wired in Slice 2 when per-RFC repos exist "has_open_prs": False, # wired in Slice 2 when per-RFC repos exist
"metadata_malformed": bool(r["metadata_malformed"]),
} }
) )
return {"items": items} return {"items": items}
@@ -744,7 +746,7 @@ def make_router(
rows = db.conn().execute( rows = db.conn().execute(
f""" f"""
SELECT slug, title, state, rfc_id, repo, SELECT slug, title, state, rfc_id, repo,
owners_json, arbiters_json, tags_json, owners_json, arbiters_json, tags_json, metadata_malformed,
last_main_commit_at, last_entry_commit_at, updated_at last_main_commit_at, last_entry_commit_at, updated_at
FROM cached_rfcs FROM cached_rfcs
WHERE state IN ('super-draft', 'active') WHERE state IN ('super-draft', 'active')
@@ -775,6 +777,7 @@ def make_router(
"last_active_at": r["last_main_commit_at"] or r["last_entry_commit_at"] or r["updated_at"], "last_active_at": r["last_main_commit_at"] or r["last_entry_commit_at"] or r["updated_at"],
"starred_by_me": r["slug"] in starred, "starred_by_me": r["slug"] in starred,
"has_open_prs": False, "has_open_prs": False,
"metadata_malformed": bool(r["metadata_malformed"]),
} }
for r in rows for r in rows
] ]
@@ -1342,6 +1345,7 @@ def _serialize_rfc(row) -> dict[str, Any]:
"arbiters": json.loads(row["arbiters_json"] or "[]"), "arbiters": json.loads(row["arbiters_json"] or "[]"),
"tags": json.loads(row["tags_json"] or "[]"), "tags": json.loads(row["tags_json"] or "[]"),
"body": row["body"] or "", "body": row["body"] or "",
"metadata_malformed": bool(row["metadata_malformed"]),
} }
+3
View File
@@ -376,6 +376,9 @@ def make_router(
models=super_draft_entry.models, models=super_draft_entry.models,
funder=super_draft_entry.funder, funder=super_draft_entry.funder,
body=super_draft_entry.body, body=super_draft_entry.body,
# INV-7 (§22.4a): carry forward-compat / unknown frontmatter keys
# through graduation rather than dropping them on the rebuild.
extra=dict(super_draft_entry.extra),
) )
graduated_contents = entry_mod.serialize(graduated_entry) graduated_contents = entry_mod.serialize(graduated_entry)
+38 -6
View File
@@ -27,7 +27,13 @@ import asyncio
import json import json
import logging import logging
from . import db, entry as entry_mod, projects as projects_mod, registry as registry_mod from . import (
db,
entry as entry_mod,
metadata as metadata_mod,
projects as projects_mod,
registry as registry_mod,
)
from .config import Config from .config import Config
from .gitea import Gitea, GiteaError from .gitea import Gitea, GiteaError
@@ -77,6 +83,15 @@ async def _refresh_collection_corpus(
project_id, collection_id, rfcs_dir, e) project_id, collection_id, rfcs_dir, e)
return return
# §22.4a SLICE-1: an entry's metadata may live in a `<slug>.meta.yaml`
# sidecar (the source of truth) with the `.md` kept as pure prose. Map the
# sidecars surfaced by this listing so each `.md` can dual-read its sibling.
sidecar_path_by_slug = {
metadata_mod.slug_of_sidecar(f["name"]): f["path"]
for f in files
if f.get("type") == "file" and metadata_mod.is_sidecar(f.get("name", ""))
}
seen_slugs: set[str] = set() seen_slugs: set[str] = set()
for f in files: for f in files:
if f.get("type") != "file" or not f.get("name", "").endswith(".md"): if f.get("type") != "file" or not f.get("name", "").endswith(".md"):
@@ -85,8 +100,14 @@ async def _refresh_collection_corpus(
if not result: if not result:
continue continue
text, sha = result text, sha = result
stem = f["name"][:-len(".md")]
sidecar_text: str | None = None
sidecar_path = sidecar_path_by_slug.get(stem)
if sidecar_path:
sc_result = await gitea.read_file(org, repo, sidecar_path, ref="main")
sidecar_text = sc_result[0] if sc_result else None
try: try:
entry = entry_mod.parse(text) entry, malformed = metadata_mod.read_entry(text, sidecar_text, fallback_slug=stem)
except Exception as parse_err: except Exception as parse_err:
log.warning("refresh_meta_repo: %s/%s: skipping %s: %s", log.warning("refresh_meta_repo: %s/%s: skipping %s: %s",
project_id, collection_id, f["path"], parse_err) project_id, collection_id, f["path"], parse_err)
@@ -95,8 +116,12 @@ async def _refresh_collection_corpus(
log.warning("refresh_meta_repo: %s/%s: skipping %s: missing slug", log.warning("refresh_meta_repo: %s/%s: skipping %s: missing slug",
project_id, collection_id, f["path"]) project_id, collection_id, f["path"])
continue continue
if malformed:
log.warning("refresh_meta_repo: %s/%s: %s has malformed metadata sidecar",
project_id, collection_id, f["path"])
seen_slugs.add(entry.slug) seen_slugs.add(entry.slug)
_upsert_cached_rfc(entry, body_sha=sha, collection_id=collection_id) _upsert_cached_rfc(entry, body_sha=sha, collection_id=collection_id,
metadata_malformed=malformed)
# Entries removed from a collection's rfcs/ — the spec keeps withdrawn entries # Entries removed from a collection's rfcs/ — the spec keeps withdrawn entries
# as historical record (§3), so this fires only for out-of-band deletes; # as historical record (§3), so this fires only for out-of-band deletes;
@@ -112,7 +137,12 @@ async def _refresh_collection_corpus(
project_id, collection_id, missing) project_id, collection_id, missing)
def _upsert_cached_rfc(entry: entry_mod.Entry, body_sha: str, collection_id: str = "default") -> None: def _upsert_cached_rfc(
entry: entry_mod.Entry,
body_sha: str,
collection_id: str = "default",
metadata_malformed: bool = False,
) -> None:
# §6.6: models_json stays NULL when the frontmatter key is absent # §6.6: models_json stays NULL when the frontmatter key is absent
# (inherit operator universe) and '[]' for the explicit opt-out. # (inherit operator universe) and '[]' for the explicit opt-out.
models_json = json.dumps(entry.models) if entry.models is not None else None models_json = json.dumps(entry.models) if entry.models is not None else None
@@ -126,8 +156,8 @@ def _upsert_cached_rfc(entry: entry_mod.Entry, body_sha: str, collection_id: str
graduated_at, graduated_by, owners_json, arbiters_json, tags_json, graduated_at, graduated_by, owners_json, arbiters_json, tags_json,
models_json, funder_login, body, body_sha, models_json, funder_login, body, body_sha,
unreviewed, reviewed_at, reviewed_by, collection_id, unreviewed, reviewed_at, reviewed_by, collection_id,
last_entry_commit_at, updated_at) metadata_malformed, last_entry_commit_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, datetime('now'), datetime('now')) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, datetime('now'), datetime('now'))
ON CONFLICT(collection_id, slug) DO UPDATE SET ON CONFLICT(collection_id, slug) DO UPDATE SET
title = excluded.title, title = excluded.title,
state = excluded.state, state = excluded.state,
@@ -147,6 +177,7 @@ def _upsert_cached_rfc(entry: entry_mod.Entry, body_sha: str, collection_id: str
unreviewed = excluded.unreviewed, unreviewed = excluded.unreviewed,
reviewed_at = excluded.reviewed_at, reviewed_at = excluded.reviewed_at,
reviewed_by = excluded.reviewed_by, reviewed_by = excluded.reviewed_by,
metadata_malformed = excluded.metadata_malformed,
last_entry_commit_at = datetime('now'), last_entry_commit_at = datetime('now'),
updated_at = datetime('now') updated_at = datetime('now')
""", """,
@@ -171,6 +202,7 @@ def _upsert_cached_rfc(entry: entry_mod.Entry, body_sha: str, collection_id: str
entry.reviewed_at, entry.reviewed_at,
entry.reviewed_by, entry.reviewed_by,
collection_id, collection_id,
1 if metadata_malformed else 0,
), ),
) )
+44 -2
View File
@@ -58,6 +58,20 @@ class Entry:
reviewed_at: str | None = None reviewed_at: str | None = None
reviewed_by: str | None = None reviewed_by: str | None = None
body: str = "" body: str = ""
# §22.4a (configurable collection metadata, SLICE-1): frontmatter / sidecar
# keys outside the known set above are preserved here verbatim so they ride
# along untouched through a parse→serialize round-trip and the
# frontmatter→sidecar migration (INV-7). Includes future collection-`fields:`
# schema values, which the engine does not interpret.
extra: dict[str, Any] = field(default_factory=dict)
# Frontmatter keys the Entry models explicitly; everything else is `extra`.
KNOWN_KEYS = {
"slug", "title", "state", "id", "repo", "proposed_by", "proposed_at",
"graduated_at", "graduated_by", "owners", "arbiters", "tags", "models",
"funder", "unreviewed", "reviewed_at", "reviewed_by",
}
def parse(text: str) -> Entry: def parse(text: str) -> Entry:
@@ -66,6 +80,16 @@ def parse(text: str) -> Entry:
raise ValueError("Entry file missing frontmatter") raise ValueError("Entry file missing frontmatter")
fm = yaml.safe_load(match.group(1)) or {} fm = yaml.safe_load(match.group(1)) or {}
body = match.group(2).lstrip("\n") body = match.group(2).lstrip("\n")
return from_frontmatter(fm, body)
def from_frontmatter(fm: dict[str, Any], body: str = "") -> Entry:
"""Build an Entry from an already-parsed metadata mapping + body.
Shared by `parse()` (legacy `.md` frontmatter) and the SLICE-1 dual-read
sidecar path (`metadata.read_entry`), so both produce identical records
(INV-6). `fm` keys outside `KNOWN_KEYS` are preserved on `Entry.extra`.
"""
raw_models = fm.get("models", _ABSENT) raw_models = fm.get("models", _ABSENT)
if raw_models is _ABSENT or raw_models is None: if raw_models is _ABSENT or raw_models is None:
models: list[str] | None = None models: list[str] | None = None
@@ -74,6 +98,7 @@ def parse(text: str) -> Entry:
raw_funder = fm.get("funder") raw_funder = fm.get("funder")
funder = str(raw_funder).strip() if raw_funder else None funder = str(raw_funder).strip() if raw_funder else None
unreviewed = bool(fm.get("unreviewed") or False) unreviewed = bool(fm.get("unreviewed") or False)
extra = {k: v for k, v in fm.items() if k not in KNOWN_KEYS}
return Entry( return Entry(
slug=str(fm.get("slug") or ""), slug=str(fm.get("slug") or ""),
title=str(fm.get("title") or ""), title=str(fm.get("title") or ""),
@@ -93,11 +118,18 @@ def parse(text: str) -> Entry:
reviewed_at=fm.get("reviewed_at") or None, reviewed_at=fm.get("reviewed_at") or None,
reviewed_by=fm.get("reviewed_by") or None, reviewed_by=fm.get("reviewed_by") or None,
body=body, body=body,
extra=extra,
) )
def serialize(entry: Entry) -> str: def to_frontmatter_dict(entry: Entry) -> dict[str, Any]:
"""Emit canonical entry file text — frontmatter then body.""" """The canonical ordered metadata mapping for an entry.
Shared by `serialize()` (which wraps it in `---` fences over the body) and
the SLICE-1 sidecar writer (`metadata.sidecar_yaml`, which emits the same
mapping as a standalone `<slug>.meta.yaml`). Known keys first in canonical
order, then `extra` (INV-7).
"""
fm: dict[str, Any] = { fm: dict[str, Any] = {
"slug": entry.slug, "slug": entry.slug,
"title": entry.title, "title": entry.title,
@@ -129,6 +161,16 @@ def serialize(entry: Entry) -> str:
fm["reviewed_at"] = entry.reviewed_at fm["reviewed_at"] = entry.reviewed_at
if entry.reviewed_by: if entry.reviewed_by:
fm["reviewed_by"] = entry.reviewed_by fm["reviewed_by"] = entry.reviewed_by
# INV-7: forward-compat / unknown keys ride along after the known ones.
for k, v in entry.extra.items():
if k not in fm:
fm[k] = v
return fm
def serialize(entry: Entry) -> str:
"""Emit canonical entry file text — frontmatter then body."""
fm = to_frontmatter_dict(entry)
yaml_text = yaml.safe_dump(fm, sort_keys=False, default_flow_style=False).rstrip() yaml_text = yaml.safe_dump(fm, sort_keys=False, default_flow_style=False).rstrip()
body = entry.body.lstrip("\n") body = entry.body.lstrip("\n")
if body: if body:
+34
View File
@@ -193,6 +193,40 @@ class Gitea:
resp = await self._request("PUT", f"/repos/{owner}/{repo}/contents/{path}", json=body) resp = await self._request("PUT", f"/repos/{owner}/{repo}/contents/{path}", json=body)
return resp.json() return resp.json()
async def change_files(
self,
owner: str,
repo: str,
*,
files: list[dict[str, Any]],
message: str,
branch: str,
author_name: str | None = None,
author_email: str | None = None,
) -> dict:
"""Create/update/delete several files in ONE commit (Gitea ChangeFiles).
Each `files` entry is `{"operation": "create"|"update"|"delete",
"path": str, "content": str (for create/update), "sha": str (required
for update/delete)}`. Plaintext `content` is base64-encoded here.
Backs the §22.4a frontmattersidecar migration's "one commit per
collection" (`metadata_migrate`).
"""
out_files: list[dict[str, Any]] = []
for f in files:
item: dict[str, Any] = {"operation": f["operation"], "path": f["path"]}
if "content" in f and f["content"] is not None:
item["content"] = base64.b64encode(f["content"].encode("utf-8")).decode("ascii")
if f.get("sha"):
item["sha"] = f["sha"]
out_files.append(item)
body: dict[str, Any] = {"message": message, "branch": branch, "files": out_files}
if author_name and author_email:
body["author"] = {"name": author_name, "email": author_email}
body["committer"] = {"name": author_name, "email": author_email}
resp = await self._request("POST", f"/repos/{owner}/{repo}/contents", json=body)
return resp.json()
# ----- Pull requests ----- # ----- Pull requests -----
async def list_pulls(self, owner: str, repo: str, state: str = "open") -> list[dict]: async def list_pulls(self, owner: str, repo: str, state: str = "open") -> list[dict]:
+217
View File
@@ -0,0 +1,217 @@
"""§22.4a configurable collection metadata — sidecar storage + dual-read.
SLICE-1 of docs/design/2026-06-06-configurable-collection-metadata.md.
Entry metadata is collection-configured and stored in a per-entry sidecar,
`<slug>.meta.yaml`, with the `.md` body kept as pure prose (INV-2). This module
is the storage/compat layer:
- the **dual-read** parser (`read_entry`) read the sidecar if present, else
legacy top-of-document frontmatter, with identical resulting records
(INV-6);
- sidecar (de)serialization that preserves unknown / forward-compat keys
(INV-7), reusing `entry`'s canonical field semantics;
- lenient parsing that never hard-fails a read a malformed sidecar surfaces
a flag, not an exception (INV-3).
The collection `fields:` schema and per-field validation are SLICE-2; faceted
filtering and the edit UIs are later slices. This module interprets no field
values it only moves metadata between git and in-memory `Entry` records.
"""
from __future__ import annotations
from typing import Any
import yaml
from . import entry as entry_mod
from .entry import Entry
SIDECAR_SUFFIX = ".meta.yaml"
# ----- filename helpers -----
def sidecar_name(slug: str) -> str:
"""The sidecar filename for an entry whose markdown is `<slug>.md`."""
return f"{slug}{SIDECAR_SUFFIX}"
def is_sidecar(name: str) -> bool:
return name.endswith(SIDECAR_SUFFIX)
def slug_of_sidecar(name: str) -> str:
"""The entry stem for a `<slug>.meta.yaml` filename."""
return name[: -len(SIDECAR_SUFFIX)] if is_sidecar(name) else name
# ----- metadata <-> sidecar -----
def metadata_dict(entry: Entry) -> dict[str, Any]:
"""The full metadata mapping for an entry (known fields + `extra`)."""
return entry_mod.to_frontmatter_dict(entry)
def sidecar_yaml(entry: Entry) -> str:
"""Render an entry's metadata as standalone `<slug>.meta.yaml` text."""
return yaml.safe_dump(
metadata_dict(entry), sort_keys=False, default_flow_style=False
)
def parse_sidecar(text: str) -> tuple[dict[str, Any], bool]:
"""Parse sidecar YAML leniently → `(values, malformed)`.
`malformed` is True when the text is not a YAML mapping (a list, a scalar,
or a YAML syntax error). An empty / whitespace-only sidecar is an empty
mapping, not malformed. Never raises (INV-3).
"""
try:
raw = yaml.safe_load(text)
except yaml.YAMLError:
return {}, True
if raw is None:
return {}, False
if not isinstance(raw, dict):
return {}, True
return raw, False
# ----- frontmatter stripping (INV-2) -----
def strip_frontmatter(md_text: str) -> str:
"""Return the prose body of a `.md`, dropping a leading `---…---` block.
A migrated entry's `.md` is body-only and passes through unchanged. A
not-yet-migrated `.md` still carrying frontmatter yields just its body, so
the dual-read body is the same either way.
"""
match = entry_mod.FRONTMATTER_RE.match(md_text)
if not match:
return md_text
return match.group(2).lstrip("\n")
# ----- dual-read (INV-6) -----
def read_entry(
md_text: str, sidecar_text: str | None, *, fallback_slug: str | None = None
) -> tuple[Entry, bool]:
"""Read an entry from its `.md` and optional sidecar → `(Entry, malformed)`.
- **Sidecar present and well-formed (non-empty):** metadata comes from the
sidecar; the body is the `.md` stripped of any leading frontmatter. The
sidecar is the source of truth (INV-1) and wins over stale `.md`
frontmatter.
- **Sidecar present but malformed:** the entry still loads from the legacy
`.md` frontmatter (if any) and is flagged `malformed` (INV-3).
- **Sidecar present but empty:** it has nothing to override with, so fall
back to the `.md` frontmatter (not flagged).
- **No sidecar:** the legacy path parse the `.md` frontmatter (INV-6).
`fallback_slug` (typically the filename stem) backstops the entry's slug
whenever the metadata source lacks one so a degenerate sidecar never
yields a slug-less record the caller has to silently drop (INV-3).
"""
def _with_slug(entry: Entry) -> Entry:
if not entry.slug and fallback_slug:
entry.slug = fallback_slug
return entry
if sidecar_text is None:
return _with_slug(entry_mod.parse(md_text)), False
values, malformed = parse_sidecar(sidecar_text)
if malformed or not values:
# Malformed or empty sidecar: load from the legacy .md so the entry
# still loads; flag only when the sidecar was actually malformed.
try:
entry = entry_mod.parse(md_text)
except ValueError:
entry = entry_mod.from_frontmatter({}, strip_frontmatter(md_text))
return _with_slug(entry), malformed
body = strip_frontmatter(md_text)
return _with_slug(entry_mod.from_frontmatter(values, body)), False
# ----- frontmatter -> sidecar migration (PUC-5) -----
async def migrate_collection(
gitea: Any,
*,
org: str,
repo: str,
subfolder: str = "",
actor: Any = None,
branch: str = "main",
) -> dict[str, Any]:
"""Migrate a collection's legacy-frontmatter entries to sidecars.
Walks `<subfolder>/rfcs`; for each `<slug>.md` that has legacy frontmatter
and **no** `<slug>.meta.yaml` sibling yet, it stages two file changes
create the sidecar (the entry's metadata, unknown keys preserved, INV-7)
and rewrite the `.md` to body-only (INV-2) and commits all of them in a
single ChangeFiles commit (§6.5: one commit per collection).
Idempotent: an entry that already has a sidecar is skipped; a second run
with nothing left to migrate makes no commit. Returns
`{"migrated": [...], "skipped": [...], "committed": bool}`.
"""
rfcs_dir = f"{subfolder}/rfcs" if subfolder else "rfcs"
listing = await gitea.list_dir(org, repo, rfcs_dir, ref=branch)
names = {f.get("name") for f in listing if f.get("type") == "file"}
ops: list[dict[str, Any]] = []
migrated: list[str] = []
skipped: list[str] = []
for f in listing:
if f.get("type") != "file" or not f.get("name", "").endswith(".md"):
continue
slug = f["name"][:-len(".md")]
if sidecar_name(slug) in names:
skipped.append(slug) # already migrated
continue
result = await gitea.read_file(org, repo, f["path"], ref=branch)
if not result:
continue
text, sha = result
try:
e = entry_mod.parse(text)
except ValueError:
# No frontmatter to lift (e.g. an already-clean body without a
# sidecar) — nothing to migrate; leave it untouched.
skipped.append(slug)
continue
body = strip_frontmatter(text)
new_md = body if (body == "" or body.endswith("\n")) else body + "\n"
ops.append({
"operation": "create",
"path": f"{rfcs_dir}/{sidecar_name(slug)}",
"content": sidecar_yaml(e),
})
ops.append({
"operation": "update",
"path": f["path"],
"content": new_md,
"sha": sha,
})
migrated.append(slug)
committed = False
if ops:
n = len(migrated)
message = f"Migrate {n} entr{'y' if n == 1 else 'ies'} to metadata sidecars (§22.4a)"
kwargs: dict[str, Any] = {}
if actor is not None:
kwargs = {
"author_name": actor.display_name,
"author_email": actor.email or f"{actor.gitea_login}@users.noreply",
}
await gitea.change_files(
org, repo, files=ops, message=message, branch=branch, **kwargs
)
committed = True
return {"migrated": migrated, "skipped": skipped, "committed": committed}
@@ -0,0 +1,8 @@
-- §22.4a SLICE-1 — configurable collection metadata: malformed-sidecar flag.
--
-- The corpus mirror reads an entry's metadata from its `<slug>.meta.yaml`
-- sidecar (dual-read: sidecar-else-legacy-frontmatter). A sidecar that does not
-- parse as a YAML mapping never hard-fails the read (INV-3) — the entry still
-- loads (from the legacy `.md` frontmatter if present) and this derived flag
-- marks it so the catalog can surface it. Additive only — no rebuild. 0 = ok.
ALTER TABLE cached_rfcs ADD COLUMN metadata_malformed INTEGER NOT NULL DEFAULT 0;
+28
View File
@@ -224,6 +224,34 @@ def test_graduate_happy_path_flips_in_place_keeping_body(app_with_fake_gitea):
assert gone not in kinds, f"retired audit row present: {gone}" assert gone not in kinds, f"retired audit row present: {gone}"
def test_graduate_preserves_unknown_frontmatter_keys(app_with_fake_gitea):
"""§22.4a INV-7: a forward-compat / unknown frontmatter key on the
super-draft entry must ride through the graduation rebuild, not be dropped."""
from fastapi.testclient import TestClient
from app import entry as entry_mod
app, fake = app_with_fake_gitea
with TestClient(app) as client:
provision_user_row(user_id=1, login="ben", role="owner")
seed_owned_super_draft(fake, slug="ohm", title="OHM", pitch=PITCH,
owners=["ben"], arbiters=["ben"])
# Inject an unknown key into the seeded entry's frontmatter.
key = ("wiggleverse", "meta", "main", "rfcs/ohm.md")
e = entry_mod.parse(fake.files[key]["content"])
e.extra["priority"] = "P1"
fake.files[key]["content"] = entry_mod.serialize(e)
sign_in_as(client, user_id=1, gitea_login="ben",
display_name="Ben", role="owner", email="ben@test")
r = client.post("/api/rfcs/ohm/graduate?_sync=1",
json={"rfc_id": "RFC-0042", "owners": ["ben"]})
assert r.status_code == 200, r.text
graduated = entry_mod.parse(fake.files[key]["content"])
assert graduated.state == "active"
assert graduated.extra.get("priority") == "P1"
def test_graduate_coexists_with_open_body_edit_pr(app_with_fake_gitea): def test_graduate_coexists_with_open_body_edit_pr(app_with_fake_gitea):
"""§9.8 (meta-only): an open meta-repo body-edit PR no longer blocks """§9.8 (meta-only): an open meta-repo body-edit PR no longer blocks
graduation the body is kept, so they coexist. /check stays graduation the body is kept, so they coexist. /check stays
+199
View File
@@ -0,0 +1,199 @@
"""SLICE-1 unit tests — sidecar metadata: dual-read, unknown-key preservation,
frontmatter stripping, malformed detection.
Pure functions only (no DB / no Gitea). Per
docs/design/2026-06-06-configurable-collection-metadata.md §7.2 (SLICE-1) and
INV-6 (dual-read), INV-7 (unknown keys ride along), INV-2 (clean body).
"""
from __future__ import annotations
import yaml
from app import entry as entry_mod
from app import metadata
LEGACY_MD = """---
slug: view-metrics
title: View today's metrics
state: active
owners:
- ben.stull
tags:
- dashboard
- analytics
priority: P1
owner: hasan
---
This is the prose body.
Second paragraph.
"""
# ---- INV-7: unknown keys ride along ----
def test_parse_preserves_unknown_keys_in_extra():
e = entry_mod.parse(LEGACY_MD)
assert e.extra == {"priority": "P1", "owner": "hasan"}
def test_serialize_round_trip_preserves_unknown_keys():
e = entry_mod.parse(LEGACY_MD)
text = entry_mod.serialize(e)
e2 = entry_mod.parse(text)
assert e2.extra == {"priority": "P1", "owner": "hasan"}
assert e2.tags == ["dashboard", "analytics"]
assert e2.owners == ["ben.stull"]
def test_known_keys_never_leak_into_extra():
e = entry_mod.parse(LEGACY_MD)
for known in ("slug", "title", "state", "owners", "tags"):
assert known not in e.extra
# ---- metadata_dict / sidecar_yaml ----
def test_metadata_dict_merges_known_and_extra():
e = entry_mod.parse(LEGACY_MD)
d = metadata.metadata_dict(e)
assert d["slug"] == "view-metrics"
assert d["title"] == "View today's metrics"
assert d["state"] == "active"
assert d["tags"] == ["dashboard", "analytics"]
# forward-compat keys present
assert d["priority"] == "P1"
assert d["owner"] == "hasan"
def test_sidecar_yaml_is_parseable_and_has_no_frontmatter_fences():
e = entry_mod.parse(LEGACY_MD)
sc = metadata.sidecar_yaml(e)
assert "---" not in sc.splitlines()[0]
loaded = yaml.safe_load(sc)
assert loaded["slug"] == "view-metrics"
assert loaded["priority"] == "P1"
# ---- strip_frontmatter (INV-2) ----
def test_strip_frontmatter_removes_leading_block():
body = metadata.strip_frontmatter(LEGACY_MD)
assert body.startswith("This is the prose body.")
assert "slug:" not in body
assert "priority:" not in body
def test_strip_frontmatter_passthrough_when_no_frontmatter():
plain = "Just a body.\n\nNo frontmatter here.\n"
assert metadata.strip_frontmatter(plain).strip() == plain.strip()
# ---- parse_sidecar (malformed detection, INV-3) ----
def test_parse_sidecar_good():
values, malformed = metadata.parse_sidecar("slug: a\ntitle: A\npriority: P0\n")
assert malformed is False
assert values == {"slug": "a", "title": "A", "priority": "P0"}
def test_parse_sidecar_non_mapping_is_malformed():
values, malformed = metadata.parse_sidecar("- just\n- a\n- list\n")
assert malformed is True
assert values == {}
def test_parse_sidecar_invalid_yaml_is_malformed():
values, malformed = metadata.parse_sidecar("slug: : : not yaml\n bad: [unclosed\n")
assert malformed is True
assert values == {}
def test_parse_sidecar_empty_is_empty_not_malformed():
values, malformed = metadata.parse_sidecar("")
assert malformed is False
assert values == {}
# ---- read_entry dual-read equivalence (INV-6) ----
def test_dual_read_sidecar_matches_legacy():
legacy_entry, legacy_bad = metadata.read_entry(LEGACY_MD, None)
# The migrated form: body-only .md + a sidecar holding the metadata.
migrated_md = metadata.strip_frontmatter(LEGACY_MD)
sidecar_text = metadata.sidecar_yaml(legacy_entry)
sidecar_entry, sidecar_bad = metadata.read_entry(migrated_md, sidecar_text)
assert legacy_bad is False
assert sidecar_bad is False
# Identical resulting records (INV-6).
assert sidecar_entry.slug == legacy_entry.slug
assert sidecar_entry.title == legacy_entry.title
assert sidecar_entry.state == legacy_entry.state
assert sidecar_entry.owners == legacy_entry.owners
assert sidecar_entry.tags == legacy_entry.tags
assert sidecar_entry.extra == legacy_entry.extra
assert sidecar_entry.body.strip() == legacy_entry.body.strip()
def test_read_entry_sidecar_takes_precedence_over_md_frontmatter():
# A not-yet-migrated .md still carrying frontmatter, plus a sidecar that
# disagrees: the sidecar wins for metadata; the body comes from the .md.
md_with_fm = "---\nslug: old\ntitle: Old Title\nstate: super-draft\n---\n\nBody.\n"
sidecar = "slug: new\ntitle: New Title\nstate: active\n"
e, malformed = metadata.read_entry(md_with_fm, sidecar)
assert malformed is False
assert e.title == "New Title"
assert e.state == "active"
assert e.body.strip() == "Body."
def test_read_entry_malformed_sidecar_still_loads_entry():
# INV-3: a malformed sidecar never hard-fails the read. The entry loads
# (from the .md frontmatter if present) and is flagged malformed.
md = "---\nslug: x\ntitle: X\nstate: active\n---\n\nBody.\n"
e, malformed = metadata.read_entry(md, "- not a mapping\n")
assert malformed is True
assert e.slug == "x"
assert e.title == "X"
assert e.body.strip() == "Body."
# ---- dual-read robustness: degenerate sidecars never drop the entry ----
def test_empty_sidecar_falls_back_to_md_frontmatter():
# An empty sidecar has no metadata to override with — keep the .md's.
md = "---\nslug: keep\ntitle: Keep Me\nstate: active\n---\n\nBody.\n"
e, malformed = metadata.read_entry(md, "", fallback_slug="keep")
assert malformed is False
assert e.slug == "keep"
assert e.title == "Keep Me"
def test_malformed_sidecar_on_body_only_md_loads_with_fallback_slug():
# The .md is already body-only (migrated) and the sidecar is corrupt:
# the entry must still load (INV-3), taking its slug from the filename stem.
e, malformed = metadata.read_entry("Just a body.\n", "- a\n- list\n", fallback_slug="foo")
assert malformed is True
assert e.slug == "foo"
def test_slugless_sidecar_uses_fallback_slug():
md = "Body only.\n"
sidecar = "title: No Slug Here\nstate: active\n"
e, malformed = metadata.read_entry(md, sidecar, fallback_slug="bar")
assert malformed is False
assert e.slug == "bar"
assert e.title == "No Slug Here"
# ---- sidecar filename helpers ----
def test_sidecar_filename_helpers():
assert metadata.sidecar_name("view-metrics") == "view-metrics.meta.yaml"
assert metadata.is_sidecar("view-metrics.meta.yaml") is True
assert metadata.is_sidecar("view-metrics.md") is False
assert metadata.slug_of_sidecar("view-metrics.meta.yaml") == "view-metrics"
+121
View File
@@ -0,0 +1,121 @@
"""SLICE-1 integration — the corpus mirror reads sidecars (dual-read) and
derives the malformed flag (PUC-6, INV-3/INV-6).
Per docs/design/2026-06-06-configurable-collection-metadata.md §6.2-6.3.
"""
from __future__ import annotations
import asyncio
from fastapi.testclient import TestClient
from app import cache, db, gitea as gitea_mod
from app.config import load_config
from test_propose_vertical import ( # noqa: F401 (fixtures)
app_with_fake_gitea,
tmp_env,
)
def _refresh():
cfg = load_config()
asyncio.run(cache.refresh_meta_repo(cfg, gitea_mod.Gitea(cfg)))
def _row(slug):
return db.conn().execute(
"SELECT * FROM cached_rfcs WHERE slug = ?", (slug,)
).fetchone()
def test_mirror_reads_metadata_from_sidecar(app_with_fake_gitea):
app, fake = app_with_fake_gitea
with TestClient(app):
# A migrated entry: body-only .md + a sidecar holding the metadata.
fake.files[("wiggleverse", "meta", "main", "rfcs/sidecar-one.md")] = {
"content": "Just the prose body.\n", "sha": "s1"}
fake.files[("wiggleverse", "meta", "main", "rfcs/sidecar-one.meta.yaml")] = {
"content": "slug: sidecar-one\ntitle: From Sidecar\nstate: active\ntags:\n- alpha\n",
"sha": "m1"}
_refresh()
row = _row("sidecar-one")
assert row is not None
assert row["title"] == "From Sidecar"
assert row["state"] == "active"
assert row["body"].strip() == "Just the prose body."
assert row["metadata_malformed"] == 0
def test_malformed_sidecar_flags_but_still_loads(app_with_fake_gitea):
app, fake = app_with_fake_gitea
with TestClient(app):
# .md still has frontmatter; sidecar is malformed (a list, not a map).
fake.files[("wiggleverse", "meta", "main", "rfcs/bad-meta.md")] = {
"content": "---\nslug: bad-meta\ntitle: Legacy Title\nstate: active\n---\n\nBody.\n",
"sha": "b1"}
fake.files[("wiggleverse", "meta", "main", "rfcs/bad-meta.meta.yaml")] = {
"content": "- not\n- a\n- mapping\n", "sha": "b2"}
_refresh()
row = _row("bad-meta")
assert row is not None # INV-3: still loads
assert row["metadata_malformed"] == 1
# Falls back to the legacy .md frontmatter for the metadata.
assert row["title"] == "Legacy Title"
def test_malformed_flag_surfaces_in_catalog_api(app_with_fake_gitea):
app, fake = app_with_fake_gitea
with TestClient(app) as client:
fake.files[("wiggleverse", "meta", "main", "rfcs/flagged.md")] = {
"content": "---\nslug: flagged\ntitle: Flagged\nstate: active\n---\n\nB.\n",
"sha": "f1"}
fake.files[("wiggleverse", "meta", "main", "rfcs/flagged.meta.yaml")] = {
"content": "just a scalar\n", "sha": "f2"}
fake.files[("wiggleverse", "meta", "main", "rfcs/clean.md")] = {
"content": "---\nslug: clean\ntitle: Clean\nstate: active\n---\n\nB.\n",
"sha": "c1"}
_refresh()
items = {i["slug"]: i for i in client.get("/api/rfcs").json()["items"]}
assert items["flagged"]["metadata_malformed"] is True
assert items["clean"]["metadata_malformed"] is False
# And on the detail view.
assert client.get("/api/rfcs/flagged").json()["metadata_malformed"] is True
def test_malformed_sidecar_on_migrated_entry_still_loads_flagged(app_with_fake_gitea):
# INV-3 regression: a migrated (body-only .md) entry whose sidecar is
# corrupt must NOT vanish from the catalog — it loads (slug from the
# filename stem) and is flagged malformed.
app, fake = app_with_fake_gitea
with TestClient(app):
fake.files[("wiggleverse", "meta", "main", "rfcs/orphaned.md")] = {
"content": "Just the body, no frontmatter.\n", "sha": "o1"}
fake.files[("wiggleverse", "meta", "main", "rfcs/orphaned.meta.yaml")] = {
"content": "- corrupt\n- list\n", "sha": "o2"}
_refresh()
row = _row("orphaned")
assert row is not None # did not vanish
assert row["metadata_malformed"] == 1
assert row["body"].strip() == "Just the body, no frontmatter."
def test_legacy_collection_without_sidecars_unchanged(app_with_fake_gitea):
app, fake = app_with_fake_gitea
with TestClient(app):
fake.files[("wiggleverse", "meta", "main", "rfcs/legacy.md")] = {
"content": "---\nslug: legacy\ntitle: Legacy\nstate: super-draft\n---\n\nPitch.\n",
"sha": "l1"}
_refresh()
row = _row("legacy")
assert row is not None
assert row["title"] == "Legacy"
assert row["state"] == "super-draft"
assert row["body"].strip() == "Pitch."
assert row["metadata_malformed"] == 0
+135
View File
@@ -0,0 +1,135 @@
"""SLICE-1 integration — frontmatter→sidecar migration tool (PUC-5).
Per docs/design/2026-06-06-configurable-collection-metadata.md §6.5 / §7.2:
a tool walks a collection; for each entry with legacy frontmatter it writes
`<slug>.meta.yaml` and rewrites `<slug>.md` to the body only one commit per
collection, idempotent, preserving unknown keys (INV-7).
"""
from __future__ import annotations
import asyncio
import yaml
from app import gitea as gitea_mod, metadata
from app.config import load_config
from test_propose_vertical import ( # noqa: F401 (fixtures)
app_with_fake_gitea,
tmp_env,
)
ALPHA_MD = """---
slug: alpha
title: Alpha
state: active
owners:
- ben.stull
tags:
- one
priority: P1
---
Alpha body prose.
"""
BETA_MD = """---
slug: beta
title: Beta
state: super-draft
---
Beta body prose.
"""
def _seed_entries(fake):
fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.md")] = {
"content": ALPHA_MD, "sha": "a0001"}
fake.files[("wiggleverse", "meta", "main", "rfcs/beta.md")] = {
"content": BETA_MD, "sha": "b0001"}
def _run_migration(subfolder=""):
cfg = load_config()
gitea = gitea_mod.Gitea(cfg)
return asyncio.run(
metadata.migrate_collection(
gitea, org="wiggleverse", repo="meta", subfolder=subfolder
)
)
def test_migration_writes_sidecars_and_strips_bodies(app_with_fake_gitea):
_app, fake = app_with_fake_gitea
_seed_entries(fake)
summary = _run_migration()
assert sorted(summary["migrated"]) == ["alpha", "beta"]
# Sidecars now exist.
alpha_sc = fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.meta.yaml")]["content"]
beta_sc = fake.files[("wiggleverse", "meta", "main", "rfcs/beta.meta.yaml")]["content"]
alpha_vals = yaml.safe_load(alpha_sc)
assert alpha_vals["slug"] == "alpha"
assert alpha_vals["title"] == "Alpha"
assert alpha_vals["tags"] == ["one"]
# INV-7: the unknown key rides along into the sidecar.
assert alpha_vals["priority"] == "P1"
# .md bodies are stripped of frontmatter (INV-2).
alpha_md = fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.md")]["content"]
assert "---" not in alpha_md
assert "priority:" not in alpha_md
assert alpha_md.strip() == "Alpha body prose."
assert beta_sc # beta got a sidecar too
def test_migration_is_idempotent(app_with_fake_gitea):
_app, fake = app_with_fake_gitea
_seed_entries(fake)
first = _run_migration()
assert sorted(first["migrated"]) == ["alpha", "beta"]
assert first["committed"] is True
commits_after_first = fake._commit_counter
alpha_md_after_first = fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.md")]["content"]
second = _run_migration()
assert second["migrated"] == []
assert second["committed"] is False
# No new commit; files untouched.
assert fake._commit_counter == commits_after_first
assert fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.md")]["content"] == alpha_md_after_first
def test_migration_one_commit_for_whole_collection(app_with_fake_gitea):
_app, fake = app_with_fake_gitea
_seed_entries(fake)
before = fake._commit_counter
_run_migration()
# Two entries migrated in exactly one commit (ChangeFiles batch).
assert fake._commit_counter == before + 1
def test_migration_dual_read_equivalence_after_migrate(app_with_fake_gitea):
"""An entry reads identically before and after migration (INV-6)."""
_app, fake = app_with_fake_gitea
_seed_entries(fake)
before, _ = metadata.read_entry(ALPHA_MD, None)
_run_migration()
md = fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.md")]["content"]
sc = fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.meta.yaml")]["content"]
after, malformed = metadata.read_entry(md, sc)
assert malformed is False
assert after.slug == before.slug
assert after.title == before.title
assert after.state == before.state
assert after.tags == before.tags
assert after.extra == before.extra
assert after.body.strip() == before.body.strip()
+22
View File
@@ -280,6 +280,28 @@ class FakeGitea:
return httpx.Response(200, json=children) return httpx.Response(200, json=children)
return httpx.Response(404, json={"message": "not found"}) return httpx.Response(404, json={"message": "not found"})
# POST /repos/{owner}/{repo}/contents — ChangeFiles (batch, one commit).
# §22.4a SLICE-1: the frontmatter→sidecar migration writes N files in a
# single commit. Matches the no-path /contents route (the per-path POST
# below needs a /contents/<path> suffix).
m_batch = re.fullmatch(r"/repos/([^/]+)/([^/]+)/contents/?", path)
if method == "POST" and m_batch:
owner, repo = m_batch.groups()
branch = payload["branch"]
sha = self._next_sha()
for f in payload["files"]:
op = f["operation"]
fpath = f["path"]
if op == "delete":
self.files.pop((owner, repo, branch, fpath), None)
else:
content = base64.b64decode(f["content"]).decode()
self.files[(owner, repo, branch, fpath)] = {"content": content, "sha": sha}
br = self.branches[(owner, repo)].setdefault(branch, {})
br["sha"] = sha
br["ts"] = "2026-05-23T00:00:00Z"
return httpx.Response(201, json={"commit": {"sha": sha}})
# POST /repos/{owner}/{repo}/contents/{path} # POST /repos/{owner}/{repo}/contents/{path}
m = re.fullmatch(r"/repos/([^/]+)/([^/]+)/contents/(.+)", path) m = re.fullmatch(r"/repos/([^/]+)/([^/]+)/contents/(.+)", path)
if method == "POST" and m: if method == "POST" and m:
@@ -377,6 +377,16 @@ Amend the binding contract first, then build storage/compat, then schema, then r
#### SLICE-4 — Single-entry metadata edit → completes PUC-1 #### SLICE-4 — Single-entry metadata edit → completes PUC-1
- **Depends on:** SLICE-2 - **Depends on:** SLICE-2
- **DoD:** detail panel renders schema controls; `POST …/meta` validates, direct-commits, re-ingests; scope-role gated (INV-4); lazy-migrates a legacy entry on first edit. - **DoD:** detail panel renders schema controls; `POST …/meta` validates, direct-commits, re-ingests; scope-role gated (INV-4); lazy-migrates a legacy entry on first edit.
- **Carried from SLICE-1 (deferred there):** make the **write paths**
sidecar-aware — every site that today does `entry.parse(<slug>.md)` and
serializes back into the `.md` must read/write metadata via the sidecar so a
migrated (body-only) entry doesn't crash or re-grow frontmatter. The known
sites: graduation + claim + `_read_meta_entry` (`api_graduation.py`),
`mark_entry_reviewed` (`bot.py`), body-edit / accept-change wrappers
(`api_branches.py` `_wrap_body`/`_extract_body`), and the PR-replay wrappers
(`api_prs.py`). Only once these are sidecar-aware should the **operator
trigger** for `metadata.migrate_collection` (the Owner-gated migrate endpoint)
ship.
#### SLICE-5 — Bulk tag/untag → completes PUC-2 #### SLICE-5 — Bulk tag/untag → completes PUC-2
- **Depends on:** SLICE-3, SLICE-4 - **Depends on:** SLICE-3, SLICE-4
@@ -0,0 +1,108 @@
# SLICE-1 plan — sidecar storage + dual-read + migration + malformed flag
Just-in-time implementation plan for **SLICE-1** of
[Configurable Collection Metadata](../2026-06-06-configurable-collection-metadata.md)
(§7.2). Authored at the start of the SLICE-1 coding session (session 0084,
2026-06-07), against the code SLICE-0 (v0.46.2) landed.
## Scope (and non-scope)
**In:** the storage/compat layer only — sidecar files become the source of
truth for entry metadata, with a dual-read parser, an idempotent
frontmatter→sidecar migration tool, and a derived `metadata_malformed` flag.
**Out (later slices):** the `.collection.yaml` `fields:` schema + validation
(SLICE-2), faceted filtering (SLICE-3), the edit/bulk UIs (SLICE-4/5). SLICE-1
maps sidecar values onto the **existing** typed `cached_rfcs` columns; it does
not add per-field schema columns or facet aggregation.
## Invariants honored
- **INV-6 dual-read:** parser reads the sidecar if present, else legacy
top-of-doc frontmatter, with identical resulting in-memory records.
- **INV-7 unknown keys ride along:** preserved through parse→serialize and
through the migration (never dropped).
- **INV-2:** a migrated `.md` body contains no metadata.
- **INV-1:** `cached_rfcs` stays a derived, rebuildable index; the sidecar in
git is the source of truth.
- **INV-3:** bad metadata never hard-fails a read — the entry still loads and
the catalog flags it (`metadata_malformed`).
- **INV-5 / byte-identity:** a collection with no sidecars behaves exactly as
today (legacy frontmatter path); existing entries load identically.
## Components
1. **`app/entry.py` — unknown-key preservation (INV-7).** Add
`extra: dict[str, Any]` to `Entry`. `parse()` collects frontmatter keys
outside the known set into `extra`; `serialize()` re-emits them after the
known keys. Makes frontmatter round-trips lossless.
2. **`app/metadata.py` — new module (sidecar concerns).**
- `SIDECAR_SUFFIX = ".meta.yaml"`; `sidecar_name(slug)`,
`is_sidecar(name)`, `slug_of_sidecar(name)`.
- `metadata_dict(entry) -> dict` — the full metadata mapping (known
emit-rules + `extra`), shared by the sidecar writer and the frontmatter
serializer.
- `sidecar_yaml(entry) -> str` — canonical YAML for a sidecar from
`metadata_dict`.
- `strip_frontmatter(md_text) -> str` — body-only (drops a leading
`---…---` block if present; whole text otherwise).
- `parse_sidecar(text) -> tuple[dict, bool]` — lenient: `(values, malformed)`;
non-mapping / YAML error → `({}, True)`.
- `read_entry(md_text, sidecar_text|None) -> tuple[Entry, bool]` — dual-read:
sidecar present → metadata from sidecar values, body from
`strip_frontmatter(md_text)`, `malformed` from `parse_sidecar`; absent →
`entry.parse(md_text)`, `malformed=False`.
3. **`app/gitea.py``change_files(...)` batch commit.** `POST
/repos/{owner}/{repo}/contents` (Gitea ChangeFiles) with a `files[]` array
of `{operation, path, content(b64), sha?}` — one commit for N files. Backs
the migration's "one commit per collection".
4. **`metadata.migrate_collection(gitea, org, repo, subfolder, actor)`.**
Lists `<subfolder>/rfcs`; for each `<slug>.md` **without** a `<slug>.meta.yaml`
sibling and **with** legacy frontmatter, batch: create the sidecar
(`metadata_dict` → YAML) + update the `.md` to body-only. One ChangeFiles
commit per collection. Idempotent (skip entries already migrated; no-op when
none remain). Returns a summary (`migrated`, `skipped`, `committed`).
> **Deferred (decided mid-slice, after code review):** the **operator
> trigger** for this tool (an Owner-gated endpoint) is held back to SLICE-4.
> The propose/graduate/mark-reviewed/edit write paths still `entry.parse` the
> `.md` directly, so migrating a corpus to body-only `.md`s before those
> paths are sidecar-aware would break them (crash / re-introduce
> frontmatter). SLICE-1 ships the tool as tested groundwork; SLICE-4 makes
> the write paths sidecar-aware (and adds lazy migration) and is where the
> trigger belongs (INV-8: engine write paths unchanged this slice).
5. **`app/cache.py` — dual-read in `_refresh_collection_corpus`.** Build a
sidecar-by-stem map from the dir listing; for each `.md`, read its sidecar
sibling (if any), `metadata.read_entry(...)`, thread `malformed` into
`_upsert_cached_rfc(metadata_malformed=…)`.
6. **`backend/migrations/033_metadata_malformed.sql`** — additive
`ALTER TABLE cached_rfcs ADD COLUMN metadata_malformed INTEGER NOT NULL DEFAULT 0`.
7. **`app/api.py` — surface the flag.** Add `metadata_malformed` (bool) to the
two catalog list dicts and `get_rfc`/`_get_rfc_for_collection`. (Frontend
badge + `?malformed=` filter are SLICE-3.)
## Tests (TDD — write first)
- `test_metadata.py` (unit, pure): dual-read equivalence (sidecar vs legacy →
identical Entry); unknown-key preservation through parse→serialize and
through `metadata_dict`; `strip_frontmatter` (with/without frontmatter);
`parse_sidecar` malformed cases.
- `test_metadata_migration.py` (integration, FakeGitea): migrate a collection
→ sidecars written + `.md` bodies stripped + one commit; **idempotent**
(second run is a no-op); unknown keys preserved in the sidecar.
- extend the cache/propose vertical: a collection with a sidecar mirrors from
the sidecar; a malformed sidecar sets `metadata_malformed` and still loads
the entry (INV-3); a no-sidecar collection is byte-identical to today.
## Release
Minor bump **0.46.2 → 0.47.0** (new functionality: sidecar storage + migration
tool; non-breaking — additive migration 033, dual-read keeps legacy corpora
working, opt-in). §20 CHANGELOG + upgrade-steps: migration 033 auto-applies;
running the migration tool per collection is optional (**MAY**).
+1 -1
View File
@@ -1,7 +1,7 @@
{ {
"name": "rfc-app-frontend", "name": "rfc-app-frontend",
"private": true, "private": true,
"version": "0.46.2", "version": "0.47.0",
"type": "module", "type": "module",
"scripts": { "scripts": {
"dev": "vite", "dev": "vite",