rfc-app/docs/design/plans/2026-06-07-slice2-collection-field-schema.md

Implementation plan — SLICE-2: collection field schema + central validation

Slice: SLICE-2 of docs/design/2026-06-06-configurable-collection-metadata.md §7.2. Session: OHM-0085. Branch: worktree-metadata-slice2-schema.

Goal / Definition of Done (from the design)

• .collection.yaml fields: block is parsed and stored.
• metadata_schema.validate exists — advisory at read, the enforcement point at write (write endpoints land in SLICE-4/5; this slice supplies and read-wires the function).
• The schema is served via the collection API (GET …/collections/{id}).
• A collection with no fields: behaves exactly as today (INV-5) — the §22.13 default document collection sees zero change.

Design decisions (this slice)

• Field-def shape (design §6.3): fields: is an ordered mapping {name → {type, values?, label?}}. type ∈ {enum, tags, text} (v1). Order is preserved for facet display (SLICE-3).
  • enum — single scalar; requires a non-empty values: list.
  • tags — list; values: optional (controlled when present, free-form otherwise).
  • text — free string scalar.
• ref and multi-enum are out of v1 (design §2 future; Q2 leans "later"). Unknown field types are ignored with a warning (design §6.3), not fatal.
• Lenient schema parsing. A malformed fields: block or an individual bad field def is skipped with a warning, never raised — a typo in one field must not nuke the collection mirror (INV-3 spirit). Structural manifest errors (type, visibility) keep raising RegistryError as before.
• No DB migration. The normalized fields schema rides in the existing collections.config_json column, exactly like enabled_models.
• values source for validation. An entry's full metadata mapping (metadata.metadata_dict(entry) = to_frontmatter_dict) — tags come from the known Entry.tags; custom fields (e.g. priority) come from Entry.extra. Undeclared keys are forward-compat (INV-7) and never flagged.

Tasks

1. app/metadata_schema.py (new) — the one place that knows field shapes

• VALID_FIELD_TYPES = {"enum", "tags", "text"}.
• parse_fields(raw) -> dict[str, dict] — lenient/normalizing. Returns an ordered mapping of {name: {"type", "values"?, "label"?}}. Skips: non-mapping block; non-mapping field def; unknown/missing type; enum without a non-empty values: list. Logs a warning per skip. Pure (no I/O).
• @dataclass Problem(field, code, message) + as_dict().
• validate(values: dict, fields: dict[str, dict]) -> list[Problem] — for each declared field, validate the entry's value (absent is OK):
  • enum: scalar ∈ values: else not-in-values; a list/non-scalar → wrong-type.
  • tags: must be a list (wrong-type otherwise); if controlled, each member ∈ values: else not-in-values.
  • text: must be a scalar string (wrong-type otherwise). Undeclared keys ignored (INV-7). Never raises.
• Tests tests/test_metadata_schema.py: parse (each type, missing block, enum-without-values skipped, unknown type skipped, order preserved); validate (happy, enum bad value, tags uncontrolled-ok, tags controlled-bad, text wrong type, undeclared key ignored, empty schema → no problems).

2. registry.py — parse fields: into the collection config

• In parse_collection_manifest, after the existing keys: if raw.get("fields") present, cfg["fields"] = metadata_schema.parse_fields(raw["fields"]) (only set when non-empty). Flows into CollectionEntry.config → config_json via the existing json.dumps(ce.config) in _upsert_named_collection.
• Default collection (apply_registry) currently writes the collections row without config_json. A default collection's fields: would live in projects.yaml? No — the design says fields: is a .collection.yaml block. The default collection has no .collection.yaml. Decision: default- collection field schemas are out of this slice's happy path (the N=1 default is document with no fields, INV-5). Leave apply_registry untouched; only named collections (with a .collection.yaml) carry fields:. Documented as a known limitation (a deployment wanting fields on its primary corpus declares a named collection — consistent with the design's opt-in story).
• Tests extend tests/test_collection_registry.py: a manifest with a fields: block round-trips into get_collection(...)["fields"]; a manifest with a bad field def still upserts (lenient).

3. collections.py — unpack fields on read + serve via API

• Add _fields_from_config(config_json) -> dict | None (mirror _enabled_models_from_config).
• get_collection sets out["fields"] = _fields_from_config(config_json) (alongside enabled_models). api_collections.get_col then serves it with no change. list_collections left as-is (facets are SLICE-3).
• Tests extend tests/test_collection_helpers.py: get_collection exposes fields; a no-fields collection → fields is None (INV-5).

4. cache.py — advisory validation at ingest (INV-3)

• In _refresh_collection_corpus, fetch the collection's fields schema once (collections.get_collection(collection_id)); for each entry, if a schema is present, problems = metadata_schema.validate(metadata.metadata_dict(entry), fields) and OR any problems into metadata_malformed (warn-log a summary). No schema → behavior identical to today (INV-5).
• Tests tests/test_metadata_cache.py (extend): an entry violating an enum field ingests with metadata_malformed = 1; a conforming entry → 0; a collection with no schema → 0 regardless of extra keys.

5. Verify · version · ship

• pytest full backend suite green (575 baseline + new).
• Bump VERSION + frontend/package.json → 0.48.0; CHANGELOG minor entry (§20) — non-breaking, opt-in, N=1 unchanged; note write-enforcement lands with SLICE-4/5.
• Commit (cite design §7.2 SLICE-2), PR on Gitea origin, merge to main.

Invariants honored

INV-3 (read never hard-fails — advisory malformed), INV-5 (no-fields: unchanged), INV-7 (undeclared keys ride along, never flagged), INV-8 (additive, read-mostly; no write-path fork — write enforcement is a later slice).