diff --git a/CHANGELOG.md b/CHANGELOG.md index 67937ca..1a5f161 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,57 @@ skip versions are the composition of each intervening adjacent release's steps in order — no A-to-B path is pre-computed beyond that. +## 0.47.0 — 2026-06-07 + +**Minor — metadata sidecars: storage + dual-read + migration tool (§22.4a +SLICE-1).** Entry metadata can now live in a per-entry `.meta.yaml` +**sidecar**, with the `.md` kept as pure prose (INV-2). The corpus mirror reads +the sidecar when present and falls back to legacy top-of-document frontmatter +otherwise (**dual-read**, INV-6), so existing corpora load byte-identically. +This is SLICE-1 of the +[Configurable Collection Metadata](./docs/design/2026-06-06-configurable-collection-metadata.md) +design (§7.2); the collection `fields:` schema + validation (SLICE-2), faceted +filtering (SLICE-3), and the edit UIs (SLICE-4/5) follow. + +Non-breaking and opt-in: a collection with no sidecars behaves exactly as +before, and the §22.13 default collection is untouched (**N=1 sees no change**). + +Added: + +- **Dual-read parser** (`app/metadata.py`) — `read_entry(md, sidecar)` yields + identical records whether metadata comes from a sidecar or legacy + frontmatter (INV-6); a malformed sidecar never hard-fails a read — the entry + still loads and is flagged (INV-3). Unknown / forward-compat keys ride along + untouched through parse→serialize and the migration (INV-7; + `Entry.extra`). +- **Frontmatter→sidecar migration tool** (`metadata.migrate_collection`) — a + deterministic, idempotent tool that lifts a collection's legacy entries into + sidecars + body-only `.md`s in **one commit** (new Gitea `change_files` + batch); a fully-migrated collection is a no-op. The **operator trigger** for + it (an Owner-gated endpoint) is intentionally **deferred to SLICE-4**: the + propose/graduate/mark-reviewed/edit write paths still read `.md` frontmatter + directly, so they must become sidecar-aware before a corpus is migrated in + production. Until then the tool is shippable groundwork, not yet wired to a + production trigger (INV-8: the engine write paths are unchanged this slice). +- **Malformed-metadata flag** — migration `033_metadata_malformed.sql` adds + `cached_rfcs.metadata_malformed` (additive); the corpus mirror derives it and + the catalog + entry-detail APIs surface `metadata_malformed`. A degenerate + sidecar (malformed / empty / slug-less) never drops the entry — it loads with + its slug backstopped from the filename and is flagged (INV-3). +- **INV-7 at graduation** — graduation now carries an entry's unknown / + forward-compat frontmatter keys through the rebuild instead of dropping them. + +### Upgrade steps (0.46.2 → 0.47.0) + +- The framework **MUST** apply migration `033_metadata_malformed.sql` — it runs + automatically at startup (additive column, no rebuild, default `0`). +- **No operator action** otherwise. With no sidecars present (the default after + this upgrade) every corpus stays on the legacy frontmatter path, byte-for-byte + as before. The frontmatter→sidecar migration is **not** operator-triggerable + yet (its endpoint lands in SLICE-4); dual-read makes the storage change + invisible until then. +- No config change. No content change is required. + ## 0.46.2 — 2026-06-07 **Patch — `SPEC.md` §22.4a contract amendment: entry metadata is diff --git a/VERSION b/VERSION index 43c125a..421ab54 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.46.2 +0.47.0 diff --git a/backend/app/api.py b/backend/app/api.py index e0cb986..5cfe89d 100644 --- a/backend/app/api.py +++ b/backend/app/api.py @@ -651,6 +651,7 @@ def make_router( f""" SELECT r.slug, r.title, r.state, r.rfc_id, r.repo, r.owners_json, r.arbiters_json, r.tags_json, + r.metadata_malformed, r.last_main_commit_at, r.last_entry_commit_at, r.updated_at FROM cached_rfcs r JOIN collections c ON c.id = r.collection_id WHERE r.state IN ('super-draft', 'active') @@ -684,6 +685,7 @@ def make_router( "last_active_at": r["last_main_commit_at"] or r["last_entry_commit_at"] or r["updated_at"], "starred_by_me": r["slug"] in starred, "has_open_prs": False, # wired in Slice 2 when per-RFC repos exist + "metadata_malformed": bool(r["metadata_malformed"]), } ) return {"items": items} @@ -744,7 +746,7 @@ def make_router( rows = db.conn().execute( f""" SELECT slug, title, state, rfc_id, repo, - owners_json, arbiters_json, tags_json, + owners_json, arbiters_json, tags_json, metadata_malformed, last_main_commit_at, last_entry_commit_at, updated_at FROM cached_rfcs WHERE state IN ('super-draft', 'active') @@ -775,6 +777,7 @@ def make_router( "last_active_at": r["last_main_commit_at"] or r["last_entry_commit_at"] or r["updated_at"], "starred_by_me": r["slug"] in starred, "has_open_prs": False, + "metadata_malformed": bool(r["metadata_malformed"]), } for r in rows ] @@ -1342,6 +1345,7 @@ def _serialize_rfc(row) -> dict[str, Any]: "arbiters": json.loads(row["arbiters_json"] or "[]"), "tags": json.loads(row["tags_json"] or "[]"), "body": row["body"] or "", + "metadata_malformed": bool(row["metadata_malformed"]), } diff --git a/backend/app/api_graduation.py b/backend/app/api_graduation.py index 74a472a..240e26f 100644 --- a/backend/app/api_graduation.py +++ b/backend/app/api_graduation.py @@ -376,6 +376,9 @@ def make_router( models=super_draft_entry.models, funder=super_draft_entry.funder, body=super_draft_entry.body, + # INV-7 (§22.4a): carry forward-compat / unknown frontmatter keys + # through graduation rather than dropping them on the rebuild. + extra=dict(super_draft_entry.extra), ) graduated_contents = entry_mod.serialize(graduated_entry) diff --git a/backend/app/cache.py b/backend/app/cache.py index aaf9f70..1dbd54b 100644 --- a/backend/app/cache.py +++ b/backend/app/cache.py @@ -27,7 +27,13 @@ import asyncio import json import logging -from . import db, entry as entry_mod, projects as projects_mod, registry as registry_mod +from . import ( + db, + entry as entry_mod, + metadata as metadata_mod, + projects as projects_mod, + registry as registry_mod, +) from .config import Config from .gitea import Gitea, GiteaError @@ -77,6 +83,15 @@ async def _refresh_collection_corpus( project_id, collection_id, rfcs_dir, e) return + # §22.4a SLICE-1: an entry's metadata may live in a `.meta.yaml` + # sidecar (the source of truth) with the `.md` kept as pure prose. Map the + # sidecars surfaced by this listing so each `.md` can dual-read its sibling. + sidecar_path_by_slug = { + metadata_mod.slug_of_sidecar(f["name"]): f["path"] + for f in files + if f.get("type") == "file" and metadata_mod.is_sidecar(f.get("name", "")) + } + seen_slugs: set[str] = set() for f in files: if f.get("type") != "file" or not f.get("name", "").endswith(".md"): @@ -85,8 +100,14 @@ async def _refresh_collection_corpus( if not result: continue text, sha = result + stem = f["name"][:-len(".md")] + sidecar_text: str | None = None + sidecar_path = sidecar_path_by_slug.get(stem) + if sidecar_path: + sc_result = await gitea.read_file(org, repo, sidecar_path, ref="main") + sidecar_text = sc_result[0] if sc_result else None try: - entry = entry_mod.parse(text) + entry, malformed = metadata_mod.read_entry(text, sidecar_text, fallback_slug=stem) except Exception as parse_err: log.warning("refresh_meta_repo: %s/%s: skipping %s: %s", project_id, collection_id, f["path"], parse_err) @@ -95,8 +116,12 @@ async def _refresh_collection_corpus( log.warning("refresh_meta_repo: %s/%s: skipping %s: missing slug", project_id, collection_id, f["path"]) continue + if malformed: + log.warning("refresh_meta_repo: %s/%s: %s has malformed metadata sidecar", + project_id, collection_id, f["path"]) seen_slugs.add(entry.slug) - _upsert_cached_rfc(entry, body_sha=sha, collection_id=collection_id) + _upsert_cached_rfc(entry, body_sha=sha, collection_id=collection_id, + metadata_malformed=malformed) # Entries removed from a collection's rfcs/ — the spec keeps withdrawn entries # as historical record (§3), so this fires only for out-of-band deletes; @@ -112,7 +137,12 @@ async def _refresh_collection_corpus( project_id, collection_id, missing) -def _upsert_cached_rfc(entry: entry_mod.Entry, body_sha: str, collection_id: str = "default") -> None: +def _upsert_cached_rfc( + entry: entry_mod.Entry, + body_sha: str, + collection_id: str = "default", + metadata_malformed: bool = False, +) -> None: # §6.6: models_json stays NULL when the frontmatter key is absent # (inherit operator universe) and '[]' for the explicit opt-out. models_json = json.dumps(entry.models) if entry.models is not None else None @@ -126,8 +156,8 @@ def _upsert_cached_rfc(entry: entry_mod.Entry, body_sha: str, collection_id: str graduated_at, graduated_by, owners_json, arbiters_json, tags_json, models_json, funder_login, body, body_sha, unreviewed, reviewed_at, reviewed_by, collection_id, - last_entry_commit_at, updated_at) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, datetime('now'), datetime('now')) + metadata_malformed, last_entry_commit_at, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, datetime('now'), datetime('now')) ON CONFLICT(collection_id, slug) DO UPDATE SET title = excluded.title, state = excluded.state, @@ -147,6 +177,7 @@ def _upsert_cached_rfc(entry: entry_mod.Entry, body_sha: str, collection_id: str unreviewed = excluded.unreviewed, reviewed_at = excluded.reviewed_at, reviewed_by = excluded.reviewed_by, + metadata_malformed = excluded.metadata_malformed, last_entry_commit_at = datetime('now'), updated_at = datetime('now') """, @@ -171,6 +202,7 @@ def _upsert_cached_rfc(entry: entry_mod.Entry, body_sha: str, collection_id: str entry.reviewed_at, entry.reviewed_by, collection_id, + 1 if metadata_malformed else 0, ), ) diff --git a/backend/app/entry.py b/backend/app/entry.py index 1355bf3..a4ece38 100644 --- a/backend/app/entry.py +++ b/backend/app/entry.py @@ -58,6 +58,20 @@ class Entry: reviewed_at: str | None = None reviewed_by: str | None = None body: str = "" + # §22.4a (configurable collection metadata, SLICE-1): frontmatter / sidecar + # keys outside the known set above are preserved here verbatim so they ride + # along untouched through a parse→serialize round-trip and the + # frontmatter→sidecar migration (INV-7). Includes future collection-`fields:` + # schema values, which the engine does not interpret. + extra: dict[str, Any] = field(default_factory=dict) + + +# Frontmatter keys the Entry models explicitly; everything else is `extra`. +KNOWN_KEYS = { + "slug", "title", "state", "id", "repo", "proposed_by", "proposed_at", + "graduated_at", "graduated_by", "owners", "arbiters", "tags", "models", + "funder", "unreviewed", "reviewed_at", "reviewed_by", +} def parse(text: str) -> Entry: @@ -66,6 +80,16 @@ def parse(text: str) -> Entry: raise ValueError("Entry file missing frontmatter") fm = yaml.safe_load(match.group(1)) or {} body = match.group(2).lstrip("\n") + return from_frontmatter(fm, body) + + +def from_frontmatter(fm: dict[str, Any], body: str = "") -> Entry: + """Build an Entry from an already-parsed metadata mapping + body. + + Shared by `parse()` (legacy `.md` frontmatter) and the SLICE-1 dual-read + sidecar path (`metadata.read_entry`), so both produce identical records + (INV-6). `fm` keys outside `KNOWN_KEYS` are preserved on `Entry.extra`. + """ raw_models = fm.get("models", _ABSENT) if raw_models is _ABSENT or raw_models is None: models: list[str] | None = None @@ -74,6 +98,7 @@ def parse(text: str) -> Entry: raw_funder = fm.get("funder") funder = str(raw_funder).strip() if raw_funder else None unreviewed = bool(fm.get("unreviewed") or False) + extra = {k: v for k, v in fm.items() if k not in KNOWN_KEYS} return Entry( slug=str(fm.get("slug") or ""), title=str(fm.get("title") or ""), @@ -93,11 +118,18 @@ def parse(text: str) -> Entry: reviewed_at=fm.get("reviewed_at") or None, reviewed_by=fm.get("reviewed_by") or None, body=body, + extra=extra, ) -def serialize(entry: Entry) -> str: - """Emit canonical entry file text — frontmatter then body.""" +def to_frontmatter_dict(entry: Entry) -> dict[str, Any]: + """The canonical ordered metadata mapping for an entry. + + Shared by `serialize()` (which wraps it in `---` fences over the body) and + the SLICE-1 sidecar writer (`metadata.sidecar_yaml`, which emits the same + mapping as a standalone `.meta.yaml`). Known keys first in canonical + order, then `extra` (INV-7). + """ fm: dict[str, Any] = { "slug": entry.slug, "title": entry.title, @@ -129,6 +161,16 @@ def serialize(entry: Entry) -> str: fm["reviewed_at"] = entry.reviewed_at if entry.reviewed_by: fm["reviewed_by"] = entry.reviewed_by + # INV-7: forward-compat / unknown keys ride along after the known ones. + for k, v in entry.extra.items(): + if k not in fm: + fm[k] = v + return fm + + +def serialize(entry: Entry) -> str: + """Emit canonical entry file text — frontmatter then body.""" + fm = to_frontmatter_dict(entry) yaml_text = yaml.safe_dump(fm, sort_keys=False, default_flow_style=False).rstrip() body = entry.body.lstrip("\n") if body: diff --git a/backend/app/gitea.py b/backend/app/gitea.py index 4c626a5..0567a1a 100644 --- a/backend/app/gitea.py +++ b/backend/app/gitea.py @@ -193,6 +193,40 @@ class Gitea: resp = await self._request("PUT", f"/repos/{owner}/{repo}/contents/{path}", json=body) return resp.json() + async def change_files( + self, + owner: str, + repo: str, + *, + files: list[dict[str, Any]], + message: str, + branch: str, + author_name: str | None = None, + author_email: str | None = None, + ) -> dict: + """Create/update/delete several files in ONE commit (Gitea ChangeFiles). + + Each `files` entry is `{"operation": "create"|"update"|"delete", + "path": str, "content": str (for create/update), "sha": str (required + for update/delete)}`. Plaintext `content` is base64-encoded here. + Backs the §22.4a frontmatter→sidecar migration's "one commit per + collection" (`metadata_migrate`). + """ + out_files: list[dict[str, Any]] = [] + for f in files: + item: dict[str, Any] = {"operation": f["operation"], "path": f["path"]} + if "content" in f and f["content"] is not None: + item["content"] = base64.b64encode(f["content"].encode("utf-8")).decode("ascii") + if f.get("sha"): + item["sha"] = f["sha"] + out_files.append(item) + body: dict[str, Any] = {"message": message, "branch": branch, "files": out_files} + if author_name and author_email: + body["author"] = {"name": author_name, "email": author_email} + body["committer"] = {"name": author_name, "email": author_email} + resp = await self._request("POST", f"/repos/{owner}/{repo}/contents", json=body) + return resp.json() + # ----- Pull requests ----- async def list_pulls(self, owner: str, repo: str, state: str = "open") -> list[dict]: diff --git a/backend/app/metadata.py b/backend/app/metadata.py new file mode 100644 index 0000000..0ab3d3b --- /dev/null +++ b/backend/app/metadata.py @@ -0,0 +1,217 @@ +"""§22.4a configurable collection metadata — sidecar storage + dual-read. + +SLICE-1 of docs/design/2026-06-06-configurable-collection-metadata.md. + +Entry metadata is collection-configured and stored in a per-entry sidecar, +`.meta.yaml`, with the `.md` body kept as pure prose (INV-2). This module +is the storage/compat layer: + + - the **dual-read** parser (`read_entry`) — read the sidecar if present, else + legacy top-of-document frontmatter, with identical resulting records + (INV-6); + - sidecar (de)serialization that preserves unknown / forward-compat keys + (INV-7), reusing `entry`'s canonical field semantics; + - lenient parsing that never hard-fails a read — a malformed sidecar surfaces + a flag, not an exception (INV-3). + +The collection `fields:` schema and per-field validation are SLICE-2; faceted +filtering and the edit UIs are later slices. This module interprets no field +values — it only moves metadata between git and in-memory `Entry` records. +""" +from __future__ import annotations + +from typing import Any + +import yaml + +from . import entry as entry_mod +from .entry import Entry + +SIDECAR_SUFFIX = ".meta.yaml" + + +# ----- filename helpers ----- + +def sidecar_name(slug: str) -> str: + """The sidecar filename for an entry whose markdown is `.md`.""" + return f"{slug}{SIDECAR_SUFFIX}" + + +def is_sidecar(name: str) -> bool: + return name.endswith(SIDECAR_SUFFIX) + + +def slug_of_sidecar(name: str) -> str: + """The entry stem for a `.meta.yaml` filename.""" + return name[: -len(SIDECAR_SUFFIX)] if is_sidecar(name) else name + + +# ----- metadata <-> sidecar ----- + +def metadata_dict(entry: Entry) -> dict[str, Any]: + """The full metadata mapping for an entry (known fields + `extra`).""" + return entry_mod.to_frontmatter_dict(entry) + + +def sidecar_yaml(entry: Entry) -> str: + """Render an entry's metadata as standalone `.meta.yaml` text.""" + return yaml.safe_dump( + metadata_dict(entry), sort_keys=False, default_flow_style=False + ) + + +def parse_sidecar(text: str) -> tuple[dict[str, Any], bool]: + """Parse sidecar YAML leniently → `(values, malformed)`. + + `malformed` is True when the text is not a YAML mapping (a list, a scalar, + or a YAML syntax error). An empty / whitespace-only sidecar is an empty + mapping, not malformed. Never raises (INV-3). + """ + try: + raw = yaml.safe_load(text) + except yaml.YAMLError: + return {}, True + if raw is None: + return {}, False + if not isinstance(raw, dict): + return {}, True + return raw, False + + +# ----- frontmatter stripping (INV-2) ----- + +def strip_frontmatter(md_text: str) -> str: + """Return the prose body of a `.md`, dropping a leading `---…---` block. + + A migrated entry's `.md` is body-only and passes through unchanged. A + not-yet-migrated `.md` still carrying frontmatter yields just its body, so + the dual-read body is the same either way. + """ + match = entry_mod.FRONTMATTER_RE.match(md_text) + if not match: + return md_text + return match.group(2).lstrip("\n") + + +# ----- dual-read (INV-6) ----- + +def read_entry( + md_text: str, sidecar_text: str | None, *, fallback_slug: str | None = None +) -> tuple[Entry, bool]: + """Read an entry from its `.md` and optional sidecar → `(Entry, malformed)`. + + - **Sidecar present and well-formed (non-empty):** metadata comes from the + sidecar; the body is the `.md` stripped of any leading frontmatter. The + sidecar is the source of truth (INV-1) and wins over stale `.md` + frontmatter. + - **Sidecar present but malformed:** the entry still loads from the legacy + `.md` frontmatter (if any) and is flagged `malformed` (INV-3). + - **Sidecar present but empty:** it has nothing to override with, so fall + back to the `.md` frontmatter (not flagged). + - **No sidecar:** the legacy path — parse the `.md` frontmatter (INV-6). + + `fallback_slug` (typically the filename stem) backstops the entry's slug + whenever the metadata source lacks one — so a degenerate sidecar never + yields a slug-less record the caller has to silently drop (INV-3). + """ + def _with_slug(entry: Entry) -> Entry: + if not entry.slug and fallback_slug: + entry.slug = fallback_slug + return entry + + if sidecar_text is None: + return _with_slug(entry_mod.parse(md_text)), False + + values, malformed = parse_sidecar(sidecar_text) + if malformed or not values: + # Malformed or empty sidecar: load from the legacy .md so the entry + # still loads; flag only when the sidecar was actually malformed. + try: + entry = entry_mod.parse(md_text) + except ValueError: + entry = entry_mod.from_frontmatter({}, strip_frontmatter(md_text)) + return _with_slug(entry), malformed + + body = strip_frontmatter(md_text) + return _with_slug(entry_mod.from_frontmatter(values, body)), False + + +# ----- frontmatter -> sidecar migration (PUC-5) ----- + +async def migrate_collection( + gitea: Any, + *, + org: str, + repo: str, + subfolder: str = "", + actor: Any = None, + branch: str = "main", +) -> dict[str, Any]: + """Migrate a collection's legacy-frontmatter entries to sidecars. + + Walks `/rfcs`; for each `.md` that has legacy frontmatter + and **no** `.meta.yaml` sibling yet, it stages two file changes — + create the sidecar (the entry's metadata, unknown keys preserved, INV-7) + and rewrite the `.md` to body-only (INV-2) — and commits all of them in a + single ChangeFiles commit (§6.5: one commit per collection). + + Idempotent: an entry that already has a sidecar is skipped; a second run + with nothing left to migrate makes no commit. Returns + `{"migrated": [...], "skipped": [...], "committed": bool}`. + """ + rfcs_dir = f"{subfolder}/rfcs" if subfolder else "rfcs" + listing = await gitea.list_dir(org, repo, rfcs_dir, ref=branch) + names = {f.get("name") for f in listing if f.get("type") == "file"} + + ops: list[dict[str, Any]] = [] + migrated: list[str] = [] + skipped: list[str] = [] + for f in listing: + if f.get("type") != "file" or not f.get("name", "").endswith(".md"): + continue + slug = f["name"][:-len(".md")] + if sidecar_name(slug) in names: + skipped.append(slug) # already migrated + continue + result = await gitea.read_file(org, repo, f["path"], ref=branch) + if not result: + continue + text, sha = result + try: + e = entry_mod.parse(text) + except ValueError: + # No frontmatter to lift (e.g. an already-clean body without a + # sidecar) — nothing to migrate; leave it untouched. + skipped.append(slug) + continue + body = strip_frontmatter(text) + new_md = body if (body == "" or body.endswith("\n")) else body + "\n" + ops.append({ + "operation": "create", + "path": f"{rfcs_dir}/{sidecar_name(slug)}", + "content": sidecar_yaml(e), + }) + ops.append({ + "operation": "update", + "path": f["path"], + "content": new_md, + "sha": sha, + }) + migrated.append(slug) + + committed = False + if ops: + n = len(migrated) + message = f"Migrate {n} entr{'y' if n == 1 else 'ies'} to metadata sidecars (§22.4a)" + kwargs: dict[str, Any] = {} + if actor is not None: + kwargs = { + "author_name": actor.display_name, + "author_email": actor.email or f"{actor.gitea_login}@users.noreply", + } + await gitea.change_files( + org, repo, files=ops, message=message, branch=branch, **kwargs + ) + committed = True + + return {"migrated": migrated, "skipped": skipped, "committed": committed} diff --git a/backend/migrations/033_metadata_malformed.sql b/backend/migrations/033_metadata_malformed.sql new file mode 100644 index 0000000..3463e81 --- /dev/null +++ b/backend/migrations/033_metadata_malformed.sql @@ -0,0 +1,8 @@ +-- §22.4a SLICE-1 — configurable collection metadata: malformed-sidecar flag. +-- +-- The corpus mirror reads an entry's metadata from its `.meta.yaml` +-- sidecar (dual-read: sidecar-else-legacy-frontmatter). A sidecar that does not +-- parse as a YAML mapping never hard-fails the read (INV-3) — the entry still +-- loads (from the legacy `.md` frontmatter if present) and this derived flag +-- marks it so the catalog can surface it. Additive only — no rebuild. 0 = ok. +ALTER TABLE cached_rfcs ADD COLUMN metadata_malformed INTEGER NOT NULL DEFAULT 0; diff --git a/backend/tests/test_graduation_vertical.py b/backend/tests/test_graduation_vertical.py index 2cd3321..0706518 100644 --- a/backend/tests/test_graduation_vertical.py +++ b/backend/tests/test_graduation_vertical.py @@ -224,6 +224,34 @@ def test_graduate_happy_path_flips_in_place_keeping_body(app_with_fake_gitea): assert gone not in kinds, f"retired audit row present: {gone}" +def test_graduate_preserves_unknown_frontmatter_keys(app_with_fake_gitea): + """§22.4a INV-7: a forward-compat / unknown frontmatter key on the + super-draft entry must ride through the graduation rebuild, not be dropped.""" + from fastapi.testclient import TestClient + from app import entry as entry_mod + + app, fake = app_with_fake_gitea + with TestClient(app) as client: + provision_user_row(user_id=1, login="ben", role="owner") + seed_owned_super_draft(fake, slug="ohm", title="OHM", pitch=PITCH, + owners=["ben"], arbiters=["ben"]) + # Inject an unknown key into the seeded entry's frontmatter. + key = ("wiggleverse", "meta", "main", "rfcs/ohm.md") + e = entry_mod.parse(fake.files[key]["content"]) + e.extra["priority"] = "P1" + fake.files[key]["content"] = entry_mod.serialize(e) + + sign_in_as(client, user_id=1, gitea_login="ben", + display_name="Ben", role="owner", email="ben@test") + r = client.post("/api/rfcs/ohm/graduate?_sync=1", + json={"rfc_id": "RFC-0042", "owners": ["ben"]}) + assert r.status_code == 200, r.text + + graduated = entry_mod.parse(fake.files[key]["content"]) + assert graduated.state == "active" + assert graduated.extra.get("priority") == "P1" + + def test_graduate_coexists_with_open_body_edit_pr(app_with_fake_gitea): """§9.8 (meta-only): an open meta-repo body-edit PR no longer blocks graduation — the body is kept, so they coexist. /check stays diff --git a/backend/tests/test_metadata.py b/backend/tests/test_metadata.py new file mode 100644 index 0000000..b5e1473 --- /dev/null +++ b/backend/tests/test_metadata.py @@ -0,0 +1,199 @@ +"""SLICE-1 unit tests — sidecar metadata: dual-read, unknown-key preservation, +frontmatter stripping, malformed detection. + +Pure functions only (no DB / no Gitea). Per +docs/design/2026-06-06-configurable-collection-metadata.md §7.2 (SLICE-1) and +INV-6 (dual-read), INV-7 (unknown keys ride along), INV-2 (clean body). +""" +from __future__ import annotations + +import yaml + +from app import entry as entry_mod +from app import metadata + + +LEGACY_MD = """--- +slug: view-metrics +title: View today's metrics +state: active +owners: +- ben.stull +tags: +- dashboard +- analytics +priority: P1 +owner: hasan +--- + +This is the prose body. + +Second paragraph. +""" + + +# ---- INV-7: unknown keys ride along ---- + +def test_parse_preserves_unknown_keys_in_extra(): + e = entry_mod.parse(LEGACY_MD) + assert e.extra == {"priority": "P1", "owner": "hasan"} + + +def test_serialize_round_trip_preserves_unknown_keys(): + e = entry_mod.parse(LEGACY_MD) + text = entry_mod.serialize(e) + e2 = entry_mod.parse(text) + assert e2.extra == {"priority": "P1", "owner": "hasan"} + assert e2.tags == ["dashboard", "analytics"] + assert e2.owners == ["ben.stull"] + + +def test_known_keys_never_leak_into_extra(): + e = entry_mod.parse(LEGACY_MD) + for known in ("slug", "title", "state", "owners", "tags"): + assert known not in e.extra + + +# ---- metadata_dict / sidecar_yaml ---- + +def test_metadata_dict_merges_known_and_extra(): + e = entry_mod.parse(LEGACY_MD) + d = metadata.metadata_dict(e) + assert d["slug"] == "view-metrics" + assert d["title"] == "View today's metrics" + assert d["state"] == "active" + assert d["tags"] == ["dashboard", "analytics"] + # forward-compat keys present + assert d["priority"] == "P1" + assert d["owner"] == "hasan" + + +def test_sidecar_yaml_is_parseable_and_has_no_frontmatter_fences(): + e = entry_mod.parse(LEGACY_MD) + sc = metadata.sidecar_yaml(e) + assert "---" not in sc.splitlines()[0] + loaded = yaml.safe_load(sc) + assert loaded["slug"] == "view-metrics" + assert loaded["priority"] == "P1" + + +# ---- strip_frontmatter (INV-2) ---- + +def test_strip_frontmatter_removes_leading_block(): + body = metadata.strip_frontmatter(LEGACY_MD) + assert body.startswith("This is the prose body.") + assert "slug:" not in body + assert "priority:" not in body + + +def test_strip_frontmatter_passthrough_when_no_frontmatter(): + plain = "Just a body.\n\nNo frontmatter here.\n" + assert metadata.strip_frontmatter(plain).strip() == plain.strip() + + +# ---- parse_sidecar (malformed detection, INV-3) ---- + +def test_parse_sidecar_good(): + values, malformed = metadata.parse_sidecar("slug: a\ntitle: A\npriority: P0\n") + assert malformed is False + assert values == {"slug": "a", "title": "A", "priority": "P0"} + + +def test_parse_sidecar_non_mapping_is_malformed(): + values, malformed = metadata.parse_sidecar("- just\n- a\n- list\n") + assert malformed is True + assert values == {} + + +def test_parse_sidecar_invalid_yaml_is_malformed(): + values, malformed = metadata.parse_sidecar("slug: : : not yaml\n bad: [unclosed\n") + assert malformed is True + assert values == {} + + +def test_parse_sidecar_empty_is_empty_not_malformed(): + values, malformed = metadata.parse_sidecar("") + assert malformed is False + assert values == {} + + +# ---- read_entry dual-read equivalence (INV-6) ---- + +def test_dual_read_sidecar_matches_legacy(): + legacy_entry, legacy_bad = metadata.read_entry(LEGACY_MD, None) + + # The migrated form: body-only .md + a sidecar holding the metadata. + migrated_md = metadata.strip_frontmatter(LEGACY_MD) + sidecar_text = metadata.sidecar_yaml(legacy_entry) + sidecar_entry, sidecar_bad = metadata.read_entry(migrated_md, sidecar_text) + + assert legacy_bad is False + assert sidecar_bad is False + # Identical resulting records (INV-6). + assert sidecar_entry.slug == legacy_entry.slug + assert sidecar_entry.title == legacy_entry.title + assert sidecar_entry.state == legacy_entry.state + assert sidecar_entry.owners == legacy_entry.owners + assert sidecar_entry.tags == legacy_entry.tags + assert sidecar_entry.extra == legacy_entry.extra + assert sidecar_entry.body.strip() == legacy_entry.body.strip() + + +def test_read_entry_sidecar_takes_precedence_over_md_frontmatter(): + # A not-yet-migrated .md still carrying frontmatter, plus a sidecar that + # disagrees: the sidecar wins for metadata; the body comes from the .md. + md_with_fm = "---\nslug: old\ntitle: Old Title\nstate: super-draft\n---\n\nBody.\n" + sidecar = "slug: new\ntitle: New Title\nstate: active\n" + e, malformed = metadata.read_entry(md_with_fm, sidecar) + assert malformed is False + assert e.title == "New Title" + assert e.state == "active" + assert e.body.strip() == "Body." + + +def test_read_entry_malformed_sidecar_still_loads_entry(): + # INV-3: a malformed sidecar never hard-fails the read. The entry loads + # (from the .md frontmatter if present) and is flagged malformed. + md = "---\nslug: x\ntitle: X\nstate: active\n---\n\nBody.\n" + e, malformed = metadata.read_entry(md, "- not a mapping\n") + assert malformed is True + assert e.slug == "x" + assert e.title == "X" + assert e.body.strip() == "Body." + + +# ---- dual-read robustness: degenerate sidecars never drop the entry ---- + +def test_empty_sidecar_falls_back_to_md_frontmatter(): + # An empty sidecar has no metadata to override with — keep the .md's. + md = "---\nslug: keep\ntitle: Keep Me\nstate: active\n---\n\nBody.\n" + e, malformed = metadata.read_entry(md, "", fallback_slug="keep") + assert malformed is False + assert e.slug == "keep" + assert e.title == "Keep Me" + + +def test_malformed_sidecar_on_body_only_md_loads_with_fallback_slug(): + # The .md is already body-only (migrated) and the sidecar is corrupt: + # the entry must still load (INV-3), taking its slug from the filename stem. + e, malformed = metadata.read_entry("Just a body.\n", "- a\n- list\n", fallback_slug="foo") + assert malformed is True + assert e.slug == "foo" + + +def test_slugless_sidecar_uses_fallback_slug(): + md = "Body only.\n" + sidecar = "title: No Slug Here\nstate: active\n" + e, malformed = metadata.read_entry(md, sidecar, fallback_slug="bar") + assert malformed is False + assert e.slug == "bar" + assert e.title == "No Slug Here" + + +# ---- sidecar filename helpers ---- + +def test_sidecar_filename_helpers(): + assert metadata.sidecar_name("view-metrics") == "view-metrics.meta.yaml" + assert metadata.is_sidecar("view-metrics.meta.yaml") is True + assert metadata.is_sidecar("view-metrics.md") is False + assert metadata.slug_of_sidecar("view-metrics.meta.yaml") == "view-metrics" diff --git a/backend/tests/test_metadata_cache.py b/backend/tests/test_metadata_cache.py new file mode 100644 index 0000000..f287208 --- /dev/null +++ b/backend/tests/test_metadata_cache.py @@ -0,0 +1,121 @@ +"""SLICE-1 integration — the corpus mirror reads sidecars (dual-read) and +derives the malformed flag (PUC-6, INV-3/INV-6). + +Per docs/design/2026-06-06-configurable-collection-metadata.md §6.2-6.3. +""" +from __future__ import annotations + +import asyncio + +from fastapi.testclient import TestClient + +from app import cache, db, gitea as gitea_mod +from app.config import load_config + +from test_propose_vertical import ( # noqa: F401 (fixtures) + app_with_fake_gitea, + tmp_env, +) + + +def _refresh(): + cfg = load_config() + asyncio.run(cache.refresh_meta_repo(cfg, gitea_mod.Gitea(cfg))) + + +def _row(slug): + return db.conn().execute( + "SELECT * FROM cached_rfcs WHERE slug = ?", (slug,) + ).fetchone() + + +def test_mirror_reads_metadata_from_sidecar(app_with_fake_gitea): + app, fake = app_with_fake_gitea + with TestClient(app): + # A migrated entry: body-only .md + a sidecar holding the metadata. + fake.files[("wiggleverse", "meta", "main", "rfcs/sidecar-one.md")] = { + "content": "Just the prose body.\n", "sha": "s1"} + fake.files[("wiggleverse", "meta", "main", "rfcs/sidecar-one.meta.yaml")] = { + "content": "slug: sidecar-one\ntitle: From Sidecar\nstate: active\ntags:\n- alpha\n", + "sha": "m1"} + _refresh() + + row = _row("sidecar-one") + assert row is not None + assert row["title"] == "From Sidecar" + assert row["state"] == "active" + assert row["body"].strip() == "Just the prose body." + assert row["metadata_malformed"] == 0 + + +def test_malformed_sidecar_flags_but_still_loads(app_with_fake_gitea): + app, fake = app_with_fake_gitea + with TestClient(app): + # .md still has frontmatter; sidecar is malformed (a list, not a map). + fake.files[("wiggleverse", "meta", "main", "rfcs/bad-meta.md")] = { + "content": "---\nslug: bad-meta\ntitle: Legacy Title\nstate: active\n---\n\nBody.\n", + "sha": "b1"} + fake.files[("wiggleverse", "meta", "main", "rfcs/bad-meta.meta.yaml")] = { + "content": "- not\n- a\n- mapping\n", "sha": "b2"} + _refresh() + + row = _row("bad-meta") + assert row is not None # INV-3: still loads + assert row["metadata_malformed"] == 1 + # Falls back to the legacy .md frontmatter for the metadata. + assert row["title"] == "Legacy Title" + + +def test_malformed_flag_surfaces_in_catalog_api(app_with_fake_gitea): + app, fake = app_with_fake_gitea + with TestClient(app) as client: + fake.files[("wiggleverse", "meta", "main", "rfcs/flagged.md")] = { + "content": "---\nslug: flagged\ntitle: Flagged\nstate: active\n---\n\nB.\n", + "sha": "f1"} + fake.files[("wiggleverse", "meta", "main", "rfcs/flagged.meta.yaml")] = { + "content": "just a scalar\n", "sha": "f2"} + fake.files[("wiggleverse", "meta", "main", "rfcs/clean.md")] = { + "content": "---\nslug: clean\ntitle: Clean\nstate: active\n---\n\nB.\n", + "sha": "c1"} + _refresh() + + items = {i["slug"]: i for i in client.get("/api/rfcs").json()["items"]} + assert items["flagged"]["metadata_malformed"] is True + assert items["clean"]["metadata_malformed"] is False + + # And on the detail view. + assert client.get("/api/rfcs/flagged").json()["metadata_malformed"] is True + + +def test_malformed_sidecar_on_migrated_entry_still_loads_flagged(app_with_fake_gitea): + # INV-3 regression: a migrated (body-only .md) entry whose sidecar is + # corrupt must NOT vanish from the catalog — it loads (slug from the + # filename stem) and is flagged malformed. + app, fake = app_with_fake_gitea + with TestClient(app): + fake.files[("wiggleverse", "meta", "main", "rfcs/orphaned.md")] = { + "content": "Just the body, no frontmatter.\n", "sha": "o1"} + fake.files[("wiggleverse", "meta", "main", "rfcs/orphaned.meta.yaml")] = { + "content": "- corrupt\n- list\n", "sha": "o2"} + _refresh() + + row = _row("orphaned") + assert row is not None # did not vanish + assert row["metadata_malformed"] == 1 + assert row["body"].strip() == "Just the body, no frontmatter." + + +def test_legacy_collection_without_sidecars_unchanged(app_with_fake_gitea): + app, fake = app_with_fake_gitea + with TestClient(app): + fake.files[("wiggleverse", "meta", "main", "rfcs/legacy.md")] = { + "content": "---\nslug: legacy\ntitle: Legacy\nstate: super-draft\n---\n\nPitch.\n", + "sha": "l1"} + _refresh() + + row = _row("legacy") + assert row is not None + assert row["title"] == "Legacy" + assert row["state"] == "super-draft" + assert row["body"].strip() == "Pitch." + assert row["metadata_malformed"] == 0 diff --git a/backend/tests/test_metadata_migration.py b/backend/tests/test_metadata_migration.py new file mode 100644 index 0000000..a3faebc --- /dev/null +++ b/backend/tests/test_metadata_migration.py @@ -0,0 +1,135 @@ +"""SLICE-1 integration — frontmatter→sidecar migration tool (PUC-5). + +Per docs/design/2026-06-06-configurable-collection-metadata.md §6.5 / §7.2: +a tool walks a collection; for each entry with legacy frontmatter it writes +`.meta.yaml` and rewrites `.md` to the body only — one commit per +collection, idempotent, preserving unknown keys (INV-7). +""" +from __future__ import annotations + +import asyncio + +import yaml + +from app import gitea as gitea_mod, metadata +from app.config import load_config + +from test_propose_vertical import ( # noqa: F401 (fixtures) + app_with_fake_gitea, + tmp_env, +) + + +ALPHA_MD = """--- +slug: alpha +title: Alpha +state: active +owners: +- ben.stull +tags: +- one +priority: P1 +--- + +Alpha body prose. +""" + +BETA_MD = """--- +slug: beta +title: Beta +state: super-draft +--- + +Beta body prose. +""" + + +def _seed_entries(fake): + fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.md")] = { + "content": ALPHA_MD, "sha": "a0001"} + fake.files[("wiggleverse", "meta", "main", "rfcs/beta.md")] = { + "content": BETA_MD, "sha": "b0001"} + + +def _run_migration(subfolder=""): + cfg = load_config() + gitea = gitea_mod.Gitea(cfg) + return asyncio.run( + metadata.migrate_collection( + gitea, org="wiggleverse", repo="meta", subfolder=subfolder + ) + ) + + +def test_migration_writes_sidecars_and_strips_bodies(app_with_fake_gitea): + _app, fake = app_with_fake_gitea + _seed_entries(fake) + + summary = _run_migration() + + assert sorted(summary["migrated"]) == ["alpha", "beta"] + + # Sidecars now exist. + alpha_sc = fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.meta.yaml")]["content"] + beta_sc = fake.files[("wiggleverse", "meta", "main", "rfcs/beta.meta.yaml")]["content"] + alpha_vals = yaml.safe_load(alpha_sc) + assert alpha_vals["slug"] == "alpha" + assert alpha_vals["title"] == "Alpha" + assert alpha_vals["tags"] == ["one"] + # INV-7: the unknown key rides along into the sidecar. + assert alpha_vals["priority"] == "P1" + + # .md bodies are stripped of frontmatter (INV-2). + alpha_md = fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.md")]["content"] + assert "---" not in alpha_md + assert "priority:" not in alpha_md + assert alpha_md.strip() == "Alpha body prose." + assert beta_sc # beta got a sidecar too + + +def test_migration_is_idempotent(app_with_fake_gitea): + _app, fake = app_with_fake_gitea + _seed_entries(fake) + + first = _run_migration() + assert sorted(first["migrated"]) == ["alpha", "beta"] + assert first["committed"] is True + + commits_after_first = fake._commit_counter + alpha_md_after_first = fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.md")]["content"] + + second = _run_migration() + assert second["migrated"] == [] + assert second["committed"] is False + # No new commit; files untouched. + assert fake._commit_counter == commits_after_first + assert fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.md")]["content"] == alpha_md_after_first + + +def test_migration_one_commit_for_whole_collection(app_with_fake_gitea): + _app, fake = app_with_fake_gitea + _seed_entries(fake) + before = fake._commit_counter + _run_migration() + # Two entries migrated in exactly one commit (ChangeFiles batch). + assert fake._commit_counter == before + 1 + + +def test_migration_dual_read_equivalence_after_migrate(app_with_fake_gitea): + """An entry reads identically before and after migration (INV-6).""" + _app, fake = app_with_fake_gitea + _seed_entries(fake) + + before, _ = metadata.read_entry(ALPHA_MD, None) + _run_migration() + md = fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.md")]["content"] + sc = fake.files[("wiggleverse", "meta", "main", "rfcs/alpha.meta.yaml")]["content"] + after, malformed = metadata.read_entry(md, sc) + + assert malformed is False + assert after.slug == before.slug + assert after.title == before.title + assert after.state == before.state + assert after.tags == before.tags + assert after.extra == before.extra + assert after.body.strip() == before.body.strip() diff --git a/backend/tests/test_propose_vertical.py b/backend/tests/test_propose_vertical.py index f4bc22c..70503b0 100644 --- a/backend/tests/test_propose_vertical.py +++ b/backend/tests/test_propose_vertical.py @@ -280,6 +280,28 @@ class FakeGitea: return httpx.Response(200, json=children) return httpx.Response(404, json={"message": "not found"}) + # POST /repos/{owner}/{repo}/contents — ChangeFiles (batch, one commit). + # §22.4a SLICE-1: the frontmatter→sidecar migration writes N files in a + # single commit. Matches the no-path /contents route (the per-path POST + # below needs a /contents/ suffix). + m_batch = re.fullmatch(r"/repos/([^/]+)/([^/]+)/contents/?", path) + if method == "POST" and m_batch: + owner, repo = m_batch.groups() + branch = payload["branch"] + sha = self._next_sha() + for f in payload["files"]: + op = f["operation"] + fpath = f["path"] + if op == "delete": + self.files.pop((owner, repo, branch, fpath), None) + else: + content = base64.b64decode(f["content"]).decode() + self.files[(owner, repo, branch, fpath)] = {"content": content, "sha": sha} + br = self.branches[(owner, repo)].setdefault(branch, {}) + br["sha"] = sha + br["ts"] = "2026-05-23T00:00:00Z" + return httpx.Response(201, json={"commit": {"sha": sha}}) + # POST /repos/{owner}/{repo}/contents/{path} m = re.fullmatch(r"/repos/([^/]+)/([^/]+)/contents/(.+)", path) if method == "POST" and m: diff --git a/docs/design/2026-06-06-configurable-collection-metadata.md b/docs/design/2026-06-06-configurable-collection-metadata.md index 87b36f4..dd9d7fc 100644 --- a/docs/design/2026-06-06-configurable-collection-metadata.md +++ b/docs/design/2026-06-06-configurable-collection-metadata.md @@ -377,6 +377,16 @@ Amend the binding contract first, then build storage/compat, then schema, then r #### SLICE-4 — Single-entry metadata edit → completes PUC-1 - **Depends on:** SLICE-2 - **DoD:** detail panel renders schema controls; `POST …/meta` validates, direct-commits, re-ingests; scope-role gated (INV-4); lazy-migrates a legacy entry on first edit. +- **Carried from SLICE-1 (deferred there):** make the **write paths** + sidecar-aware — every site that today does `entry.parse(.md)` and + serializes back into the `.md` must read/write metadata via the sidecar so a + migrated (body-only) entry doesn't crash or re-grow frontmatter. The known + sites: graduation + claim + `_read_meta_entry` (`api_graduation.py`), + `mark_entry_reviewed` (`bot.py`), body-edit / accept-change wrappers + (`api_branches.py` `_wrap_body`/`_extract_body`), and the PR-replay wrappers + (`api_prs.py`). Only once these are sidecar-aware should the **operator + trigger** for `metadata.migrate_collection` (the Owner-gated migrate endpoint) + ship. #### SLICE-5 — Bulk tag/untag → completes PUC-2 - **Depends on:** SLICE-3, SLICE-4 diff --git a/docs/design/plans/2026-06-07-slice1-sidecar-storage.md b/docs/design/plans/2026-06-07-slice1-sidecar-storage.md new file mode 100644 index 0000000..e384e67 --- /dev/null +++ b/docs/design/plans/2026-06-07-slice1-sidecar-storage.md @@ -0,0 +1,108 @@ +# SLICE-1 plan — sidecar storage + dual-read + migration + malformed flag + +Just-in-time implementation plan for **SLICE-1** of +[Configurable Collection Metadata](../2026-06-06-configurable-collection-metadata.md) +(§7.2). Authored at the start of the SLICE-1 coding session (session 0084, +2026-06-07), against the code SLICE-0 (v0.46.2) landed. + +## Scope (and non-scope) + +**In:** the storage/compat layer only — sidecar files become the source of +truth for entry metadata, with a dual-read parser, an idempotent +frontmatter→sidecar migration tool, and a derived `metadata_malformed` flag. + +**Out (later slices):** the `.collection.yaml` `fields:` schema + validation +(SLICE-2), faceted filtering (SLICE-3), the edit/bulk UIs (SLICE-4/5). SLICE-1 +maps sidecar values onto the **existing** typed `cached_rfcs` columns; it does +not add per-field schema columns or facet aggregation. + +## Invariants honored + +- **INV-6 dual-read:** parser reads the sidecar if present, else legacy + top-of-doc frontmatter, with identical resulting in-memory records. +- **INV-7 unknown keys ride along:** preserved through parse→serialize and + through the migration (never dropped). +- **INV-2:** a migrated `.md` body contains no metadata. +- **INV-1:** `cached_rfcs` stays a derived, rebuildable index; the sidecar in + git is the source of truth. +- **INV-3:** bad metadata never hard-fails a read — the entry still loads and + the catalog flags it (`metadata_malformed`). +- **INV-5 / byte-identity:** a collection with no sidecars behaves exactly as + today (legacy frontmatter path); existing entries load identically. + +## Components + +1. **`app/entry.py` — unknown-key preservation (INV-7).** Add + `extra: dict[str, Any]` to `Entry`. `parse()` collects frontmatter keys + outside the known set into `extra`; `serialize()` re-emits them after the + known keys. Makes frontmatter round-trips lossless. + +2. **`app/metadata.py` — new module (sidecar concerns).** + - `SIDECAR_SUFFIX = ".meta.yaml"`; `sidecar_name(slug)`, + `is_sidecar(name)`, `slug_of_sidecar(name)`. + - `metadata_dict(entry) -> dict` — the full metadata mapping (known + emit-rules + `extra`), shared by the sidecar writer and the frontmatter + serializer. + - `sidecar_yaml(entry) -> str` — canonical YAML for a sidecar from + `metadata_dict`. + - `strip_frontmatter(md_text) -> str` — body-only (drops a leading + `---…---` block if present; whole text otherwise). + - `parse_sidecar(text) -> tuple[dict, bool]` — lenient: `(values, malformed)`; + non-mapping / YAML error → `({}, True)`. + - `read_entry(md_text, sidecar_text|None) -> tuple[Entry, bool]` — dual-read: + sidecar present → metadata from sidecar values, body from + `strip_frontmatter(md_text)`, `malformed` from `parse_sidecar`; absent → + `entry.parse(md_text)`, `malformed=False`. + +3. **`app/gitea.py` — `change_files(...)` batch commit.** `POST + /repos/{owner}/{repo}/contents` (Gitea ChangeFiles) with a `files[]` array + of `{operation, path, content(b64), sha?}` — one commit for N files. Backs + the migration's "one commit per collection". + +4. **`metadata.migrate_collection(gitea, org, repo, subfolder, actor)`.** + Lists `/rfcs`; for each `.md` **without** a `.meta.yaml` + sibling and **with** legacy frontmatter, batch: create the sidecar + (`metadata_dict` → YAML) + update the `.md` to body-only. One ChangeFiles + commit per collection. Idempotent (skip entries already migrated; no-op when + none remain). Returns a summary (`migrated`, `skipped`, `committed`). + + > **Deferred (decided mid-slice, after code review):** the **operator + > trigger** for this tool (an Owner-gated endpoint) is held back to SLICE-4. + > The propose/graduate/mark-reviewed/edit write paths still `entry.parse` the + > `.md` directly, so migrating a corpus to body-only `.md`s before those + > paths are sidecar-aware would break them (crash / re-introduce + > frontmatter). SLICE-1 ships the tool as tested groundwork; SLICE-4 makes + > the write paths sidecar-aware (and adds lazy migration) and is where the + > trigger belongs (INV-8: engine write paths unchanged this slice). + +5. **`app/cache.py` — dual-read in `_refresh_collection_corpus`.** Build a + sidecar-by-stem map from the dir listing; for each `.md`, read its sidecar + sibling (if any), `metadata.read_entry(...)`, thread `malformed` into + `_upsert_cached_rfc(metadata_malformed=…)`. + +6. **`backend/migrations/033_metadata_malformed.sql`** — additive + `ALTER TABLE cached_rfcs ADD COLUMN metadata_malformed INTEGER NOT NULL DEFAULT 0`. + +7. **`app/api.py` — surface the flag.** Add `metadata_malformed` (bool) to the + two catalog list dicts and `get_rfc`/`_get_rfc_for_collection`. (Frontend + badge + `?malformed=` filter are SLICE-3.) + +## Tests (TDD — write first) + +- `test_metadata.py` (unit, pure): dual-read equivalence (sidecar vs legacy → + identical Entry); unknown-key preservation through parse→serialize and + through `metadata_dict`; `strip_frontmatter` (with/without frontmatter); + `parse_sidecar` malformed cases. +- `test_metadata_migration.py` (integration, FakeGitea): migrate a collection + → sidecars written + `.md` bodies stripped + one commit; **idempotent** + (second run is a no-op); unknown keys preserved in the sidecar. +- extend the cache/propose vertical: a collection with a sidecar mirrors from + the sidecar; a malformed sidecar sets `metadata_malformed` and still loads + the entry (INV-3); a no-sidecar collection is byte-identical to today. + +## Release + +Minor bump **0.46.2 → 0.47.0** (new functionality: sidecar storage + migration +tool; non-breaking — additive migration 033, dual-read keeps legacy corpora +working, opt-in). §20 CHANGELOG + upgrade-steps: migration 033 auto-applies; +running the migration tool per collection is optional (**MAY**). diff --git a/frontend/package.json b/frontend/package.json index 4e1cc36..519c3ff 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,7 +1,7 @@ { "name": "rfc-app-frontend", "private": true, - "version": "0.46.2", + "version": "0.47.0", "type": "module", "scripts": { "dev": "vite",