From e336e31812b34c8c2ec3c8c080e1d9c3055f0d85 Mon Sep 17 00:00:00 2001 From: Ben Stull Date: Sun, 7 Jun 2026 17:06:04 -0700 Subject: [PATCH] =?UTF-8?q?v0.48.0=20=E2=80=94=20SLICE-2:=20collection=20f?= =?UTF-8?q?ield=20schema=20+=20central=20validation=20(=C2=A722.4a)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Collections declare a `fields:` schema in `.collection.yaml`; entries carry typed metadata (enum/tags/text). New central `metadata_schema` module parses the schema leniently (INV-3) and validates entry values — advisory at read (corpus mirror flags violations as `metadata_malformed` without hard-failing), the enforcement point for the write boundary (edit endpoints land SLICE-4/5). - app/metadata_schema.py: parse_fields (lenient/normalizing) + validate - registry.parse_collection_manifest reads `fields:` into collection config - collections.get_collection unpacks `fields`; served by the collection API - cache._refresh_collection_corpus validates each entry advisory-only Non-breaking, opt-in: no `fields:` → unchanged (INV-5); default `document` collection declares none (N=1 unchanged). No DB migration — schema rides in collections.config_json. SLICE-2 of docs/design/2026-06-06-configurable-collection-metadata.md §7.2. Backend suite green (601 passed). Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 52 ++++++ VERSION | 2 +- backend/app/cache.py | 21 +++ backend/app/collections.py | 22 ++- backend/app/metadata_schema.py | 147 +++++++++++++++++ backend/app/registry.py | 8 + backend/tests/test_collection_helpers.py | 21 +++ backend/tests/test_collection_registry.py | 30 ++++ backend/tests/test_metadata_cache.py | 56 +++++++ backend/tests/test_metadata_schema.py | 149 ++++++++++++++++++ ...26-06-07-slice2-collection-field-schema.md | 109 +++++++++++++ frontend/package.json | 2 +- 12 files changed, 616 insertions(+), 3 deletions(-) create mode 100644 backend/app/metadata_schema.py create mode 100644 backend/tests/test_metadata_schema.py create mode 100644 docs/design/plans/2026-06-07-slice2-collection-field-schema.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a5f161..f24ab6c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,58 @@ skip versions are the composition of each intervening adjacent release's steps in order — no A-to-B path is pre-computed beyond that. +## 0.48.0 — 2026-06-07 + +**Minor — collection field schema + central validation (§22.4a SLICE-2).** A +collection can now declare a small **field schema** in its `.collection.yaml` +(`fields:` block), so its entries carry structured, typed metadata — `priority`, +`tags`, and any custom fields the deployment defines. The schema is mirrored into +the collection record and served on the collection API; a new central validator +checks each entry's stored values against it. This is SLICE-2 of the +[Configurable Collection Metadata](./docs/design/2026-06-06-configurable-collection-metadata.md) +design (§7.2), building on the SLICE-1 sidecars; faceted filtering (SLICE-3) and +the edit UIs (SLICE-4/5) follow. + +Non-breaking and opt-in: a collection with **no `fields:`** behaves exactly as +before (INV-5), and the §22.13 default `document` collection declares none +(**N=1 sees no change**). No DB migration — the normalized schema rides in the +existing `collections.config_json` column. + +Added: + +- **`app/metadata_schema.py`** — the one place that knows a collection's field + shapes. `parse_fields(raw)` normalizes a `.collection.yaml` `fields:` block + **leniently** (INV-3): a bad block or a bad field def (non-mapping, unknown + type, `enum` without a non-empty `values:` list) is skipped with a warning, + never fatal — a typo in one field can't drop the whole collection from the + mirror. `validate(values, fields)` returns advisory `Problem`s (empty = + clean). v1 field types: **`enum`** (single value, controlled by a required + `values:`), **`tags`** (a list; free-form unless `values:` given), **`text`** + (a free string). `ref` / `multi-enum` are deferred (design §2). Keys an entry + carries that the schema does not declare ride along untouched, never flagged + (INV-7). +- **Registry ingest** (`registry.parse_collection_manifest`) reads the `fields:` + block into the collection config (→ `config_json`), beside `enabled_models`. +- **Collection read + API** (`collections.get_collection`) unpacks the schema + and `GET /api/projects/{id}/collections/{cid}` serves it as `fields` (`null` + when unset). +- **Advisory validation at ingest** — the corpus mirror + (`cache._refresh_collection_corpus`) validates each entry against its + collection's schema and flags a violation as `metadata_malformed` **without + blocking the read** (INV-3), OR-ed onto the SLICE-1 sidecar-syntax check. + Write-boundary **enforcement** (a 422 on the metadata-edit endpoints) lands + with those endpoints in SLICE-4/5. + +### Upgrade steps (0.47.0 → 0.48.0) + +- **No migration, no operator action.** A deployment opts in per collection by + adding a `fields:` block to that collection's `.collection.yaml`; until it + does, behavior is byte-for-byte unchanged. +- A collection's field schema is edited in git for v1 (in-app schema management + is deferred, design D8). After editing `.collection.yaml`, the registry mirror + picks the schema up on its next webhook / reconciler sweep. +- No config change. No content change is required. + ## 0.47.0 — 2026-06-07 **Minor — metadata sidecars: storage + dual-read + migration tool (§22.4a diff --git a/VERSION b/VERSION index 421ab54..a758a09 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.47.0 +0.48.0 diff --git a/backend/app/cache.py b/backend/app/cache.py index 1dbd54b..3186005 100644 --- a/backend/app/cache.py +++ b/backend/app/cache.py @@ -28,9 +28,11 @@ import json import logging from . import ( + collections as collections_mod, db, entry as entry_mod, metadata as metadata_mod, + metadata_schema, projects as projects_mod, registry as registry_mod, ) @@ -83,6 +85,13 @@ async def _refresh_collection_corpus( project_id, collection_id, rfcs_dir, e) return + # §22.4a SLICE-2: a collection may declare a metadata field schema. Fetch it + # once for the whole corpus pass; entries whose stored values fail it are + # flagged malformed advisory-only (INV-3) — the read never hard-fails. A + # collection with no schema validates nothing (INV-5, the default unchanged). + col = collections_mod.get_collection(collection_id) + fields_schema = (col or {}).get("fields") or None + # §22.4a SLICE-1: an entry's metadata may live in a `.meta.yaml` # sidecar (the source of truth) with the `.md` kept as pure prose. Map the # sidecars surfaced by this listing so each `.md` can dual-read its sibling. @@ -119,6 +128,18 @@ async def _refresh_collection_corpus( if malformed: log.warning("refresh_meta_repo: %s/%s: %s has malformed metadata sidecar", project_id, collection_id, f["path"]) + # §22.4a SLICE-2: advisory schema validation (INV-3). A schema violation + # flags the entry malformed without blocking the read, OR-ed onto any + # sidecar-syntax malformation above. + if fields_schema: + problems = metadata_schema.validate( + metadata_mod.metadata_dict(entry), fields_schema + ) + if problems: + malformed = True + log.warning("refresh_meta_repo: %s/%s: %s fails its field schema: %s", + project_id, collection_id, f["path"], + "; ".join(p.message for p in problems)) seen_slugs.add(entry.slug) _upsert_cached_rfc(entry, body_sha=sha, collection_id=collection_id, metadata_malformed=malformed) diff --git a/backend/app/collections.py b/backend/app/collections.py index 5fba493..4ee473c 100644 --- a/backend/app/collections.py +++ b/backend/app/collections.py @@ -45,6 +45,21 @@ def _enabled_models_from_config(config_json: str | None) -> list[str] | None: return [str(m) for m in em] if isinstance(em, list) else None +def _fields_from_config(config_json: str | None) -> dict | None: + """§22.4a SLICE-2 per-collection metadata field schema from a `config_json` + blob, or None when the collection declares no `fields:`. The stored value is + already normalized by `metadata_schema.parse_fields` at ingest, so it's + served verbatim.""" + if not config_json: + return None + try: + cfg = json.loads(config_json) + except (json.JSONDecodeError, TypeError): + return None + fields = cfg.get("fields") if isinstance(cfg, dict) else None + return fields if isinstance(fields, dict) and fields else None + + def default_collection_id(project_id: str) -> str: """The id of a project's default (S1: sole) collection. Falls back to the literal 'default' when the project has no collection row yet.""" @@ -102,7 +117,12 @@ def get_collection(collection_id: str) -> dict | None: if row is None: return None out = dict(row) - out["enabled_models"] = _enabled_models_from_config(out.pop("config_json", None)) + config_json = out.pop("config_json", None) + out["enabled_models"] = _enabled_models_from_config(config_json) + # §22.4a SLICE-2: serve the collection's metadata field schema (None when + # the collection declares no `fields:` — INV-5, the default `document` + # collection is unaffected). + out["fields"] = _fields_from_config(config_json) out["entry_noun"] = entry_noun(out["type"]) return out diff --git a/backend/app/metadata_schema.py b/backend/app/metadata_schema.py new file mode 100644 index 0000000..cbf308b --- /dev/null +++ b/backend/app/metadata_schema.py @@ -0,0 +1,147 @@ +"""§22.4a configurable collection metadata — field schema + central validation. + +SLICE-2 of docs/design/2026-06-06-configurable-collection-metadata.md. + +A collection declares a small **field schema** in its `.collection.yaml` +(`fields:` block) so its entries can carry structured metadata — priority, tags, +and any custom fields the deployment defines. This module is the **one place** +that knows a collection's field shapes (modeled on `registry.py`): + + - `parse_fields` — normalize + validate the declared schema, leniently: a bad + block or a bad field def is skipped with a warning, never raised, so a typo + in one field can't nuke the collection mirror (INV-3 spirit). The normalized + schema is a plain, JSON-serializable mapping that rides in + `collections.config_json` (no DB migration) and is served verbatim by the + collection API. + - `validate` — check an entry's stored values against the schema, returning a + list of advisory `Problem`s. Empty list = clean. Used **advisory at read** + (the corpus mirror flags a non-empty result as `metadata_malformed`, INV-3) + and is the enforcement point at the **write** boundary (the metadata-edit + endpoints land in SLICE-4/5). + +Field types (v1): `enum` (single scalar, controlled by a required `values:` +list), `tags` (a list; free-form unless `values:` given), `text` (a free +string). `ref` / `multi-enum` are future (design §2, Q2). Unknown types are +ignored with a warning. Keys an entry carries that the schema does **not** +declare ride along untouched and are never flagged (INV-7). +""" +from __future__ import annotations + +import logging +from dataclasses import dataclass +from typing import Any + +log = logging.getLogger(__name__) + +# §6.3 v1 field types. `ref` (typed cross-entry link) and `multi-enum` are +# deferred (design §2, Q2) — declared with an unknown type they're skipped. +VALID_FIELD_TYPES = {"enum", "tags", "text"} + + +@dataclass +class Problem: + """One advisory schema-validation problem against a declared field. + + `code` is a stable machine token (`not-in-values`, `wrong-type`); `message` + is human-facing (surfaced at the write boundary in SLICE-4/5).""" + field: str + code: str + message: str + + def as_dict(self) -> dict[str, str]: + return {"field": self.field, "code": self.code, "message": self.message} + + +# ----- schema parsing (lenient) ----- + +def parse_fields(raw: Any) -> dict[str, dict]: + """Normalize a `.collection.yaml` `fields:` block → `{name: {type, ...}}`. + + Pure (no I/O). Lenient (INV-3): a non-mapping block yields `{}`; an + individual field def that is not a mapping, has an unknown/missing `type`, or + is an `enum` without a non-empty `values:` list is **skipped with a warning** + — never raised. Order is preserved (facet display order, SLICE-3). The + result is plain dicts so it serializes straight into `config_json` and the + collection API. + """ + if not isinstance(raw, dict): + if raw is not None: + log.warning("metadata_schema: fields block is not a mapping (%s); ignoring", + type(raw).__name__) + return {} + out: dict[str, dict] = {} + for name, spec in raw.items(): + if not isinstance(spec, dict): + log.warning("metadata_schema: field %r def is not a mapping; skipping", name) + continue + ftype = str(spec.get("type") or "").strip() + if ftype not in VALID_FIELD_TYPES: + log.warning("metadata_schema: field %r has unknown type %r; skipping", + name, ftype) + continue + values = spec.get("values") + norm_values: list[str] | None = None + if values is not None: + if not isinstance(values, list): + log.warning("metadata_schema: field %r values is not a list; ignoring", + name) + else: + norm_values = [str(v) for v in values] + if ftype == "enum" and not norm_values: + log.warning("metadata_schema: enum field %r needs a non-empty values " + "list; skipping", name) + continue + field_def: dict[str, Any] = {"type": ftype} + if norm_values is not None: + field_def["values"] = norm_values + label = spec.get("label") + if label: + field_def["label"] = str(label) + out[str(name)] = field_def + return out + + +# ----- value validation (advisory) ----- + +def _is_scalar(v: Any) -> bool: + return isinstance(v, (str, int, float, bool)) + + +def validate(values: dict[str, Any], fields: dict[str, dict]) -> list[Problem]: + """Check an entry's metadata `values` against a collection's field schema. + + Returns advisory `Problem`s (empty = clean). Only **declared** fields are + checked; a field the entry omits is fine (no required fields in v1), and a + key the schema doesn't declare rides along untouched (INV-7). With an empty + schema, everything is clean (INV-5). Never raises (INV-3). + """ + problems: list[Problem] = [] + for name, spec in fields.items(): + if name not in values: + continue + value = values[name] + if value is None: + continue + ftype = spec.get("type") + allowed = spec.get("values") + if ftype == "enum": + if not _is_scalar(value): + problems.append(Problem(name, "wrong-type", + f"{name!r} must be a single value, got {type(value).__name__}")) + elif allowed is not None and str(value) not in allowed: + problems.append(Problem(name, "not-in-values", + f"{name!r} value {value!r} is not one of {allowed}")) + elif ftype == "tags": + if not isinstance(value, list): + problems.append(Problem(name, "wrong-type", + f"{name!r} must be a list, got {type(value).__name__}")) + elif allowed is not None: + for member in value: + if str(member) not in allowed: + problems.append(Problem(name, "not-in-values", + f"{name!r} value {member!r} is not one of {allowed}")) + elif ftype == "text": + if not _is_scalar(value): + problems.append(Problem(name, "wrong-type", + f"{name!r} must be a string, got {type(value).__name__}")) + return problems diff --git a/backend/app/registry.py b/backend/app/registry.py index 046275c..e2504de 100644 --- a/backend/app/registry.py +++ b/backend/app/registry.py @@ -18,6 +18,7 @@ from dataclasses import dataclass, field import yaml from . import db +from . import metadata_schema from .config import Config from .gitea import Gitea @@ -159,6 +160,13 @@ def parse_collection_manifest(text: str) -> CollectionEntry: if not isinstance(em, list): raise RegistryError("collection enabled_models must be a list") cfg["enabled_models"] = [str(m) for m in em] + # §22.4a SLICE-2: the collection's metadata field schema. Parsed leniently — + # a bad field def is skipped with a warning, never fatal (INV-3), so a typo + # in one field can't drop the whole collection from the mirror. Stored only + # when at least one valid field survives. + fields = metadata_schema.parse_fields(raw.get("fields")) + if fields: + cfg["fields"] = fields return CollectionEntry(ctype, vis, initial_state, name, cfg) diff --git a/backend/tests/test_collection_helpers.py b/backend/tests/test_collection_helpers.py index 77c0e02..5930c31 100644 --- a/backend/tests/test_collection_helpers.py +++ b/backend/tests/test_collection_helpers.py @@ -2,6 +2,7 @@ subfolder_of.""" from __future__ import annotations +import json import tempfile from pathlib import Path @@ -62,3 +63,23 @@ def test_get_collection_and_subfolder(): assert collections_mod.subfolder_of("features") == "features" assert collections_mod.subfolder_of("default") == "" assert collections_mod.get_collection("nope") is None + + +# ---- §22.4a SLICE-2: field schema served on the collection ---- + +def test_get_collection_fields_none_when_unset(): + # INV-5: a collection with no `fields:` exposes fields=None (the default + # `document` collection sees zero change). + _db() + _seed() + assert collections_mod.get_collection("features")["fields"] is None + + +def test_get_collection_exposes_field_schema(): + _db() + _seed() + schema = {"priority": {"type": "enum", "values": ["P0", "P1"]}} + db.conn().execute( + "UPDATE collections SET config_json = ? WHERE id = 'features'", + (json.dumps({"fields": schema}),)) + assert collections_mod.get_collection("features")["fields"] == schema diff --git a/backend/tests/test_collection_registry.py b/backend/tests/test_collection_registry.py index fa8a403..9d099d2 100644 --- a/backend/tests/test_collection_registry.py +++ b/backend/tests/test_collection_registry.py @@ -61,6 +61,36 @@ def test_parse_collection_manifest_rejects_bad_visibility(): registry.parse_collection_manifest("type: bdd\nvisibility: nope\n") +# ---- §22.4a SLICE-2: a `fields:` block flows into the collection config ---- + +def test_parse_collection_manifest_reads_field_schema(): + doc = registry.parse_collection_manifest( + "type: bdd\n" + "fields:\n" + " priority:\n" + " type: enum\n" + " values: [P0, P1, P2]\n" + " tags:\n" + " type: tags\n" + ) + assert doc.config["fields"] == { + "priority": {"type": "enum", "values": ["P0", "P1", "P2"]}, + "tags": {"type": "tags"}, + } + + +def test_parse_collection_manifest_lenient_on_bad_field(): + # A bad field def is skipped (INV-3), the manifest still parses, and a + # manifest with no surviving fields carries no `fields` config key at all. + doc = registry.parse_collection_manifest( + "type: bdd\n" + "fields:\n" + " broken:\n" + " type: ref\n" + ) + assert "fields" not in doc.config + + # --- mirror discovery --------------------------------------------------------- diff --git a/backend/tests/test_metadata_cache.py b/backend/tests/test_metadata_cache.py index f287208..a3b9b9d 100644 --- a/backend/tests/test_metadata_cache.py +++ b/backend/tests/test_metadata_cache.py @@ -6,6 +6,7 @@ Per docs/design/2026-06-06-configurable-collection-metadata.md §6.2-6.3. from __future__ import annotations import asyncio +import json from fastapi.testclient import TestClient @@ -105,6 +106,61 @@ def test_malformed_sidecar_on_migrated_entry_still_loads_flagged(app_with_fake_g assert row["body"].strip() == "Just the body, no frontmatter." +def _set_default_fields_schema(schema): + # apply_registry leaves the default collection's config_json untouched, so a + # schema set here survives a corpus refresh (refresh_meta_repo only). + db.conn().execute( + "UPDATE collections SET config_json = ? WHERE id = 'default'", + (json.dumps({"fields": schema}),)) + + +# ---- §22.4a SLICE-2: advisory schema validation at ingest (INV-3) ---- + +def test_schema_violation_flags_malformed(app_with_fake_gitea): + app, fake = app_with_fake_gitea + with TestClient(app): + _set_default_fields_schema( + {"priority": {"type": "enum", "values": ["P0", "P1", "P2"]}}) + fake.files[("wiggleverse", "meta", "main", "rfcs/bad-prio.md")] = { + "content": "---\nslug: bad-prio\ntitle: Bad\nstate: active\npriority: P9\n---\n\nB.\n", + "sha": "bp1"} + fake.files[("wiggleverse", "meta", "main", "rfcs/good-prio.md")] = { + "content": "---\nslug: good-prio\ntitle: Good\nstate: active\npriority: P0\n---\n\nB.\n", + "sha": "gp1"} + _refresh() + + assert _row("bad-prio")["metadata_malformed"] == 1 # INV-3: flagged + assert _row("bad-prio")["title"] == "Bad" # still loads + assert _row("good-prio")["metadata_malformed"] == 0 + + +def test_schema_ignores_undeclared_keys(app_with_fake_gitea): + # INV-7: keys the schema doesn't declare ride along and never flag malformed. + app, fake = app_with_fake_gitea + with TestClient(app): + _set_default_fields_schema( + {"priority": {"type": "enum", "values": ["P0", "P1"]}}) + fake.files[("wiggleverse", "meta", "main", "rfcs/extra-key.md")] = { + "content": "---\nslug: extra-key\ntitle: Extra\nstate: active\nowner: hasan\n---\n\nB.\n", + "sha": "ek1"} + _refresh() + + assert _row("extra-key")["metadata_malformed"] == 0 + + +def test_no_schema_never_flags(app_with_fake_gitea): + # INV-5: a collection with no field schema validates nothing, even when an + # entry carries values that would fail a schema if one existed. + app, fake = app_with_fake_gitea + with TestClient(app): + fake.files[("wiggleverse", "meta", "main", "rfcs/anything.md")] = { + "content": "---\nslug: anything\ntitle: Any\nstate: active\npriority: whatever\n---\n\nB.\n", + "sha": "an1"} + _refresh() + + assert _row("anything")["metadata_malformed"] == 0 + + def test_legacy_collection_without_sidecars_unchanged(app_with_fake_gitea): app, fake = app_with_fake_gitea with TestClient(app): diff --git a/backend/tests/test_metadata_schema.py b/backend/tests/test_metadata_schema.py new file mode 100644 index 0000000..b4166ec --- /dev/null +++ b/backend/tests/test_metadata_schema.py @@ -0,0 +1,149 @@ +"""SLICE-2 unit tests — collection field schema + central validation. + +Pure functions only (no DB / no Gitea). Per +docs/design/2026-06-06-configurable-collection-metadata.md §7.2 (SLICE-2): +`metadata_schema.parse_fields` (lenient schema parsing) and +`metadata_schema.validate` (advisory at read / enforcement point at write). +Honors INV-3 (never hard-fails), INV-5 (no-fields unchanged), INV-7 (undeclared +keys ride along). +""" +from __future__ import annotations + +from app import metadata_schema as ms + + +# ---- parse_fields: normalization + leniency ---- + +def test_parse_fields_each_type(): + raw = { + "priority": {"type": "enum", "values": ["P0", "P1", "P2"], "label": "Priority"}, + "tags": {"type": "tags"}, + "owner": {"type": "text"}, + } + fields = ms.parse_fields(raw) + assert list(fields) == ["priority", "tags", "owner"] # order preserved + assert fields["priority"] == { + "type": "enum", + "values": ["P0", "P1", "P2"], + "label": "Priority", + } + assert fields["tags"] == {"type": "tags"} + assert fields["owner"] == {"type": "text"} + + +def test_parse_fields_controlled_tags_keeps_values(): + fields = ms.parse_fields({"area": {"type": "tags", "values": ["a", "b"]}}) + assert fields["area"] == {"type": "tags", "values": ["a", "b"]} + + +def test_parse_fields_missing_block_is_empty(): + assert ms.parse_fields(None) == {} + assert ms.parse_fields({}) == {} + + +def test_parse_fields_non_mapping_block_skipped(): + assert ms.parse_fields(["not", "a", "mapping"]) == {} + assert ms.parse_fields("nope") == {} + + +def test_parse_fields_enum_without_values_skipped(): + # enum requires a non-empty values list — skipped, not fatal. + assert ms.parse_fields({"p": {"type": "enum"}}) == {} + assert ms.parse_fields({"p": {"type": "enum", "values": []}}) == {} + + +def test_parse_fields_unknown_type_skipped(): + fields = ms.parse_fields( + {"good": {"type": "text"}, "bad": {"type": "ref"}, "huh": {"type": "frob"}} + ) + assert list(fields) == ["good"] + + +def test_parse_fields_non_mapping_def_skipped(): + fields = ms.parse_fields({"good": {"type": "text"}, "bad": "scalar"}) + assert list(fields) == ["good"] + + +def test_parse_fields_values_coerced_to_str_list(): + fields = ms.parse_fields({"p": {"type": "enum", "values": [0, 1, 2]}}) + assert fields["p"]["values"] == ["0", "1", "2"] + + +# ---- validate: advisory problem reporting ---- + +SCHEMA = ms.parse_fields( + { + "priority": {"type": "enum", "values": ["P0", "P1", "P2"]}, + "tags": {"type": "tags"}, + "area": {"type": "tags", "values": ["checkout", "cart"]}, + "owner": {"type": "text"}, + } +) + + +def test_validate_happy(): + values = { + "priority": "P0", + "tags": ["anything", "free"], + "area": ["checkout"], + "owner": "ben", + } + assert ms.validate(values, SCHEMA) == [] + + +def test_validate_absent_fields_ok(): + # A declared field that the entry omits is fine (no required fields in v1). + assert ms.validate({}, SCHEMA) == [] + + +def test_validate_enum_bad_value(): + problems = ms.validate({"priority": "P9"}, SCHEMA) + assert [p.field for p in problems] == ["priority"] + assert problems[0].code == "not-in-values" + + +def test_validate_enum_wrong_type(): + problems = ms.validate({"priority": ["P0"]}, SCHEMA) + assert problems[0].field == "priority" + assert problems[0].code == "wrong-type" + + +def test_validate_tags_free_form_ok(): + assert ms.validate({"tags": ["x", "y", "z"]}, SCHEMA) == [] + + +def test_validate_tags_wrong_type(): + problems = ms.validate({"tags": "notalist"}, SCHEMA) + assert problems[0].field == "tags" + assert problems[0].code == "wrong-type" + + +def test_validate_controlled_tags_bad_member(): + problems = ms.validate({"area": ["checkout", "nope"]}, SCHEMA) + assert problems[0].field == "area" + assert problems[0].code == "not-in-values" + + +def test_validate_text_wrong_type(): + problems = ms.validate({"owner": ["a", "b"]}, SCHEMA) + assert problems[0].field == "owner" + assert problems[0].code == "wrong-type" + + +def test_validate_undeclared_keys_ignored(): + # INV-7: keys outside the schema ride along untouched, never flagged. + assert ms.validate({"random": "value", "slug": "x", "title": "y"}, SCHEMA) == [] + + +def test_validate_empty_schema_no_problems(): + # INV-5: a collection with no fields validates everything as clean. + assert ms.validate({"priority": "anything", "x": 1}, {}) == [] + + +def test_problem_as_dict(): + p = ms.Problem(field="priority", code="not-in-values", message="bad") + assert p.as_dict() == { + "field": "priority", + "code": "not-in-values", + "message": "bad", + } diff --git a/docs/design/plans/2026-06-07-slice2-collection-field-schema.md b/docs/design/plans/2026-06-07-slice2-collection-field-schema.md new file mode 100644 index 0000000..2c67abf --- /dev/null +++ b/docs/design/plans/2026-06-07-slice2-collection-field-schema.md @@ -0,0 +1,109 @@ +# Implementation plan — SLICE-2: collection field schema + central validation + +**Slice:** SLICE-2 of +[`docs/design/2026-06-06-configurable-collection-metadata.md`](../2026-06-06-configurable-collection-metadata.md) +§7.2. **Session:** OHM-0085. **Branch:** `worktree-metadata-slice2-schema`. + +## Goal / Definition of Done (from the design) + +- `.collection.yaml` `fields:` block is parsed and stored. +- `metadata_schema.validate` exists — **advisory at read**, the enforcement + point **at write** (write endpoints land in SLICE-4/5; this slice supplies and + read-wires the function). +- The schema is served via the collection API (`GET …/collections/{id}`). +- A collection with **no `fields:`** behaves exactly as today (INV-5) — the + §22.13 default `document` collection sees zero change. + +## Design decisions (this slice) + +- **Field-def shape** (design §6.3): `fields:` is an ordered mapping + `{name → {type, values?, label?}}`. `type ∈ {enum, tags, text}` (v1). + Order is preserved for facet display (SLICE-3). + - `enum` — single scalar; requires a non-empty `values:` list. + - `tags` — list; `values:` optional (controlled when present, free-form + otherwise). + - `text` — free string scalar. +- **`ref` and `multi-enum` are out of v1** (design §2 future; Q2 leans "later"). + Unknown field types are **ignored with a warning** (design §6.3), not fatal. +- **Lenient schema parsing.** A malformed `fields:` block or an individual bad + field def is skipped with a warning, never raised — a typo in one field must + not nuke the collection mirror (INV-3 spirit). Structural manifest errors + (`type`, `visibility`) keep raising `RegistryError` as before. +- **No DB migration.** The normalized `fields` schema rides in the existing + `collections.config_json` column, exactly like `enabled_models`. +- **`values` source for validation.** An entry's full metadata mapping + (`metadata.metadata_dict(entry)` = `to_frontmatter_dict`) — `tags` come from + the known `Entry.tags`; custom fields (e.g. `priority`) come from + `Entry.extra`. Undeclared keys are forward-compat (INV-7) and never flagged. + +## Tasks + +### 1. `app/metadata_schema.py` (new) — the one place that knows field shapes +- `VALID_FIELD_TYPES = {"enum", "tags", "text"}`. +- `parse_fields(raw) -> dict[str, dict]` — lenient/normalizing. Returns an + ordered mapping of `{name: {"type", "values"?, "label"?}}`. Skips: non-mapping + block; non-mapping field def; unknown/missing type; `enum` without a non-empty + `values:` list. Logs a warning per skip. Pure (no I/O). +- `@dataclass Problem(field, code, message)` + `as_dict()`. +- `validate(values: dict, fields: dict[str, dict]) -> list[Problem]` — for each + **declared** field, validate the entry's value (absent is OK): + - `enum`: scalar ∈ `values:` else `not-in-values`; a list/non-scalar → + `wrong-type`. + - `tags`: must be a list (`wrong-type` otherwise); if controlled, each member + ∈ `values:` else `not-in-values`. + - `text`: must be a scalar string (`wrong-type` otherwise). + Undeclared keys ignored (INV-7). Never raises. +- **Tests** `tests/test_metadata_schema.py`: parse (each type, missing block, + enum-without-values skipped, unknown type skipped, order preserved); validate + (happy, enum bad value, tags uncontrolled-ok, tags controlled-bad, text wrong + type, undeclared key ignored, empty schema → no problems). + +### 2. `registry.py` — parse `fields:` into the collection config +- In `parse_collection_manifest`, after the existing keys: if `raw.get("fields")` + present, `cfg["fields"] = metadata_schema.parse_fields(raw["fields"])` (only + set when non-empty). Flows into `CollectionEntry.config` → + `config_json` via the existing `json.dumps(ce.config)` in + `_upsert_named_collection`. +- **Default collection** (`apply_registry`) currently writes the `collections` + row **without** `config_json`. A default collection's `fields:` would live in + `projects.yaml`? No — the design says `fields:` is a `.collection.yaml` block. + The default collection has no `.collection.yaml`. **Decision:** default- + collection field schemas are out of this slice's happy path (the N=1 default is + `document` with no fields, INV-5). Leave `apply_registry` untouched; only + named collections (with a `.collection.yaml`) carry `fields:`. Documented as a + known limitation (a deployment wanting fields on its primary corpus declares a + named collection — consistent with the design's opt-in story). +- **Tests** extend `tests/test_collection_registry.py`: a manifest with a + `fields:` block round-trips into `get_collection(...)["fields"]`; a manifest + with a bad field def still upserts (lenient). + +### 3. `collections.py` — unpack `fields` on read + serve via API +- Add `_fields_from_config(config_json) -> dict | None` (mirror + `_enabled_models_from_config`). +- `get_collection` sets `out["fields"] = _fields_from_config(config_json)` + (alongside `enabled_models`). `api_collections.get_col` then serves it with no + change. `list_collections` left as-is (facets are SLICE-3). +- **Tests** extend `tests/test_collection_helpers.py`: `get_collection` exposes + `fields`; a no-fields collection → `fields is None` (INV-5). + +### 4. `cache.py` — advisory validation at ingest (INV-3) +- In `_refresh_collection_corpus`, fetch the collection's `fields` schema once + (`collections.get_collection(collection_id)`); for each entry, if a schema is + present, `problems = metadata_schema.validate(metadata.metadata_dict(entry), + fields)` and OR any problems into `metadata_malformed` (warn-log a summary). + No schema → behavior identical to today (INV-5). +- **Tests** `tests/test_metadata_cache.py` (extend): an entry violating an enum + field ingests with `metadata_malformed = 1`; a conforming entry → `0`; a + collection with no schema → `0` regardless of extra keys. + +### 5. Verify · version · ship +- `pytest` full backend suite green (575 baseline + new). +- Bump `VERSION` + `frontend/package.json` → **0.48.0**; CHANGELOG minor entry + (§20) — non-breaking, opt-in, N=1 unchanged; note write-enforcement lands with + SLICE-4/5. +- Commit (cite design §7.2 SLICE-2), PR on Gitea `origin`, merge to `main`. + +## Invariants honored +INV-3 (read never hard-fails — advisory malformed), INV-5 (no-`fields:` +unchanged), INV-7 (undeclared keys ride along, never flagged), INV-8 (additive, +read-mostly; no write-path fork — write enforcement is a later slice). diff --git a/frontend/package.json b/frontend/package.json index 519c3ff..c877233 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,7 +1,7 @@ { "name": "rfc-app-frontend", "private": true, - "version": "0.47.0", + "version": "0.48.0", "type": "module", "scripts": { "dev": "vite",