Merge pull request 'v0.48.0 — SLICE-2: collection field schema + central validation (§22.4a)' (#32) from worktree-metadata-slice2-schema into main
This commit is contained in:
Ben Stull
@@ -23,6 +23,58 @@ skip versions are the composition of each intervening adjacent
|
||||
release's steps in order — no A-to-B path is pre-computed beyond
|
||||
that.
|
||||
|
||||
## 0.48.0 — 2026-06-07
|
||||
|
||||
**Minor — collection field schema + central validation (§22.4a SLICE-2).** A
|
||||
collection can now declare a small **field schema** in its `.collection.yaml`
|
||||
(`fields:` block), so its entries carry structured, typed metadata — `priority`,
|
||||
`tags`, and any custom fields the deployment defines. The schema is mirrored into
|
||||
the collection record and served on the collection API; a new central validator
|
||||
checks each entry's stored values against it. This is SLICE-2 of the
|
||||
[Configurable Collection Metadata](./docs/design/2026-06-06-configurable-collection-metadata.md)
|
||||
design (§7.2), building on the SLICE-1 sidecars; faceted filtering (SLICE-3) and
|
||||
the edit UIs (SLICE-4/5) follow.
|
||||
|
||||
Non-breaking and opt-in: a collection with **no `fields:`** behaves exactly as
|
||||
before (INV-5), and the §22.13 default `document` collection declares none
|
||||
(**N=1 sees no change**). No DB migration — the normalized schema rides in the
|
||||
existing `collections.config_json` column.
|
||||
|
||||
Added:
|
||||
|
||||
- **`app/metadata_schema.py`** — the one place that knows a collection's field
|
||||
shapes. `parse_fields(raw)` normalizes a `.collection.yaml` `fields:` block
|
||||
**leniently** (INV-3): a bad block or a bad field def (non-mapping, unknown
|
||||
type, `enum` without a non-empty `values:` list) is skipped with a warning,
|
||||
never fatal — a typo in one field can't drop the whole collection from the
|
||||
mirror. `validate(values, fields)` returns advisory `Problem`s (empty =
|
||||
clean). v1 field types: **`enum`** (single value, controlled by a required
|
||||
`values:`), **`tags`** (a list; free-form unless `values:` given), **`text`**
|
||||
(a free string). `ref` / `multi-enum` are deferred (design §2). Keys an entry
|
||||
carries that the schema does not declare ride along untouched, never flagged
|
||||
(INV-7).
|
||||
- **Registry ingest** (`registry.parse_collection_manifest`) reads the `fields:`
|
||||
block into the collection config (→ `config_json`), beside `enabled_models`.
|
||||
- **Collection read + API** (`collections.get_collection`) unpacks the schema
|
||||
and `GET /api/projects/{id}/collections/{cid}` serves it as `fields` (`null`
|
||||
when unset).
|
||||
- **Advisory validation at ingest** — the corpus mirror
|
||||
(`cache._refresh_collection_corpus`) validates each entry against its
|
||||
collection's schema and flags a violation as `metadata_malformed` **without
|
||||
blocking the read** (INV-3), OR-ed onto the SLICE-1 sidecar-syntax check.
|
||||
Write-boundary **enforcement** (a 422 on the metadata-edit endpoints) lands
|
||||
with those endpoints in SLICE-4/5.
|
||||
|
||||
### Upgrade steps (0.47.0 → 0.48.0)
|
||||
|
||||
- **No migration, no operator action.** A deployment opts in per collection by
|
||||
adding a `fields:` block to that collection's `.collection.yaml`; until it
|
||||
does, behavior is byte-for-byte unchanged.
|
||||
- A collection's field schema is edited in git for v1 (in-app schema management
|
||||
is deferred, design D8). After editing `.collection.yaml`, the registry mirror
|
||||
picks the schema up on its next webhook / reconciler sweep.
|
||||
- No config change. No content change is required.
|
||||
|
||||
## 0.47.0 — 2026-06-07
|
||||
|
||||
**Minor — metadata sidecars: storage + dual-read + migration tool (§22.4a
|
||||
|
||||
@@ -28,9 +28,11 @@ import json
|
||||
import logging
|
||||
|
||||
from . import (
|
||||
collections as collections_mod,
|
||||
db,
|
||||
entry as entry_mod,
|
||||
metadata as metadata_mod,
|
||||
metadata_schema,
|
||||
projects as projects_mod,
|
||||
registry as registry_mod,
|
||||
)
|
||||
@@ -83,6 +85,13 @@ async def _refresh_collection_corpus(
|
||||
project_id, collection_id, rfcs_dir, e)
|
||||
return
|
||||
|
||||
# §22.4a SLICE-2: a collection may declare a metadata field schema. Fetch it
|
||||
# once for the whole corpus pass; entries whose stored values fail it are
|
||||
# flagged malformed advisory-only (INV-3) — the read never hard-fails. A
|
||||
# collection with no schema validates nothing (INV-5, the default unchanged).
|
||||
col = collections_mod.get_collection(collection_id)
|
||||
fields_schema = (col or {}).get("fields") or None
|
||||
|
||||
# §22.4a SLICE-1: an entry's metadata may live in a `<slug>.meta.yaml`
|
||||
# sidecar (the source of truth) with the `.md` kept as pure prose. Map the
|
||||
# sidecars surfaced by this listing so each `.md` can dual-read its sibling.
|
||||
@@ -119,6 +128,18 @@ async def _refresh_collection_corpus(
|
||||
if malformed:
|
||||
log.warning("refresh_meta_repo: %s/%s: %s has malformed metadata sidecar",
|
||||
project_id, collection_id, f["path"])
|
||||
# §22.4a SLICE-2: advisory schema validation (INV-3). A schema violation
|
||||
# flags the entry malformed without blocking the read, OR-ed onto any
|
||||
# sidecar-syntax malformation above.
|
||||
if fields_schema:
|
||||
problems = metadata_schema.validate(
|
||||
metadata_mod.metadata_dict(entry), fields_schema
|
||||
)
|
||||
if problems:
|
||||
malformed = True
|
||||
log.warning("refresh_meta_repo: %s/%s: %s fails its field schema: %s",
|
||||
project_id, collection_id, f["path"],
|
||||
"; ".join(p.message for p in problems))
|
||||
seen_slugs.add(entry.slug)
|
||||
_upsert_cached_rfc(entry, body_sha=sha, collection_id=collection_id,
|
||||
metadata_malformed=malformed)
|
||||
|
||||
@@ -45,6 +45,21 @@ def _enabled_models_from_config(config_json: str | None) -> list[str] | None:
|
||||
return [str(m) for m in em] if isinstance(em, list) else None
|
||||
|
||||
|
||||
def _fields_from_config(config_json: str | None) -> dict | None:
|
||||
"""§22.4a SLICE-2 per-collection metadata field schema from a `config_json`
|
||||
blob, or None when the collection declares no `fields:`. The stored value is
|
||||
already normalized by `metadata_schema.parse_fields` at ingest, so it's
|
||||
served verbatim."""
|
||||
if not config_json:
|
||||
return None
|
||||
try:
|
||||
cfg = json.loads(config_json)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
return None
|
||||
fields = cfg.get("fields") if isinstance(cfg, dict) else None
|
||||
return fields if isinstance(fields, dict) and fields else None
|
||||
|
||||
|
||||
def default_collection_id(project_id: str) -> str:
|
||||
"""The id of a project's default (S1: sole) collection. Falls back to the
|
||||
literal 'default' when the project has no collection row yet."""
|
||||
@@ -102,7 +117,12 @@ def get_collection(collection_id: str) -> dict | None:
|
||||
if row is None:
|
||||
return None
|
||||
out = dict(row)
|
||||
out["enabled_models"] = _enabled_models_from_config(out.pop("config_json", None))
|
||||
config_json = out.pop("config_json", None)
|
||||
out["enabled_models"] = _enabled_models_from_config(config_json)
|
||||
# §22.4a SLICE-2: serve the collection's metadata field schema (None when
|
||||
# the collection declares no `fields:` — INV-5, the default `document`
|
||||
# collection is unaffected).
|
||||
out["fields"] = _fields_from_config(config_json)
|
||||
out["entry_noun"] = entry_noun(out["type"])
|
||||
return out
|
||||
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
"""§22.4a configurable collection metadata — field schema + central validation.
|
||||
|
||||
SLICE-2 of docs/design/2026-06-06-configurable-collection-metadata.md.
|
||||
|
||||
A collection declares a small **field schema** in its `.collection.yaml`
|
||||
(`fields:` block) so its entries can carry structured metadata — priority, tags,
|
||||
and any custom fields the deployment defines. This module is the **one place**
|
||||
that knows a collection's field shapes (modeled on `registry.py`):
|
||||
|
||||
- `parse_fields` — normalize + validate the declared schema, leniently: a bad
|
||||
block or a bad field def is skipped with a warning, never raised, so a typo
|
||||
in one field can't nuke the collection mirror (INV-3 spirit). The normalized
|
||||
schema is a plain, JSON-serializable mapping that rides in
|
||||
`collections.config_json` (no DB migration) and is served verbatim by the
|
||||
collection API.
|
||||
- `validate` — check an entry's stored values against the schema, returning a
|
||||
list of advisory `Problem`s. Empty list = clean. Used **advisory at read**
|
||||
(the corpus mirror flags a non-empty result as `metadata_malformed`, INV-3)
|
||||
and is the enforcement point at the **write** boundary (the metadata-edit
|
||||
endpoints land in SLICE-4/5).
|
||||
|
||||
Field types (v1): `enum` (single scalar, controlled by a required `values:`
|
||||
list), `tags` (a list; free-form unless `values:` given), `text` (a free
|
||||
string). `ref` / `multi-enum` are future (design §2, Q2). Unknown types are
|
||||
ignored with a warning. Keys an entry carries that the schema does **not**
|
||||
declare ride along untouched and are never flagged (INV-7).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# §6.3 v1 field types. `ref` (typed cross-entry link) and `multi-enum` are
|
||||
# deferred (design §2, Q2) — declared with an unknown type they're skipped.
|
||||
VALID_FIELD_TYPES = {"enum", "tags", "text"}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Problem:
|
||||
"""One advisory schema-validation problem against a declared field.
|
||||
|
||||
`code` is a stable machine token (`not-in-values`, `wrong-type`); `message`
|
||||
is human-facing (surfaced at the write boundary in SLICE-4/5)."""
|
||||
field: str
|
||||
code: str
|
||||
message: str
|
||||
|
||||
def as_dict(self) -> dict[str, str]:
|
||||
return {"field": self.field, "code": self.code, "message": self.message}
|
||||
|
||||
|
||||
# ----- schema parsing (lenient) -----
|
||||
|
||||
def parse_fields(raw: Any) -> dict[str, dict]:
|
||||
"""Normalize a `.collection.yaml` `fields:` block → `{name: {type, ...}}`.
|
||||
|
||||
Pure (no I/O). Lenient (INV-3): a non-mapping block yields `{}`; an
|
||||
individual field def that is not a mapping, has an unknown/missing `type`, or
|
||||
is an `enum` without a non-empty `values:` list is **skipped with a warning**
|
||||
— never raised. Order is preserved (facet display order, SLICE-3). The
|
||||
result is plain dicts so it serializes straight into `config_json` and the
|
||||
collection API.
|
||||
"""
|
||||
if not isinstance(raw, dict):
|
||||
if raw is not None:
|
||||
log.warning("metadata_schema: fields block is not a mapping (%s); ignoring",
|
||||
type(raw).__name__)
|
||||
return {}
|
||||
out: dict[str, dict] = {}
|
||||
for name, spec in raw.items():
|
||||
if not isinstance(spec, dict):
|
||||
log.warning("metadata_schema: field %r def is not a mapping; skipping", name)
|
||||
continue
|
||||
ftype = str(spec.get("type") or "").strip()
|
||||
if ftype not in VALID_FIELD_TYPES:
|
||||
log.warning("metadata_schema: field %r has unknown type %r; skipping",
|
||||
name, ftype)
|
||||
continue
|
||||
values = spec.get("values")
|
||||
norm_values: list[str] | None = None
|
||||
if values is not None:
|
||||
if not isinstance(values, list):
|
||||
log.warning("metadata_schema: field %r values is not a list; ignoring",
|
||||
name)
|
||||
else:
|
||||
norm_values = [str(v) for v in values]
|
||||
if ftype == "enum" and not norm_values:
|
||||
log.warning("metadata_schema: enum field %r needs a non-empty values "
|
||||
"list; skipping", name)
|
||||
continue
|
||||
field_def: dict[str, Any] = {"type": ftype}
|
||||
if norm_values is not None:
|
||||
field_def["values"] = norm_values
|
||||
label = spec.get("label")
|
||||
if label:
|
||||
field_def["label"] = str(label)
|
||||
out[str(name)] = field_def
|
||||
return out
|
||||
|
||||
|
||||
# ----- value validation (advisory) -----
|
||||
|
||||
def _is_scalar(v: Any) -> bool:
|
||||
return isinstance(v, (str, int, float, bool))
|
||||
|
||||
|
||||
def validate(values: dict[str, Any], fields: dict[str, dict]) -> list[Problem]:
|
||||
"""Check an entry's metadata `values` against a collection's field schema.
|
||||
|
||||
Returns advisory `Problem`s (empty = clean). Only **declared** fields are
|
||||
checked; a field the entry omits is fine (no required fields in v1), and a
|
||||
key the schema doesn't declare rides along untouched (INV-7). With an empty
|
||||
schema, everything is clean (INV-5). Never raises (INV-3).
|
||||
"""
|
||||
problems: list[Problem] = []
|
||||
for name, spec in fields.items():
|
||||
if name not in values:
|
||||
continue
|
||||
value = values[name]
|
||||
if value is None:
|
||||
continue
|
||||
ftype = spec.get("type")
|
||||
allowed = spec.get("values")
|
||||
if ftype == "enum":
|
||||
if not _is_scalar(value):
|
||||
problems.append(Problem(name, "wrong-type",
|
||||
f"{name!r} must be a single value, got {type(value).__name__}"))
|
||||
elif allowed is not None and str(value) not in allowed:
|
||||
problems.append(Problem(name, "not-in-values",
|
||||
f"{name!r} value {value!r} is not one of {allowed}"))
|
||||
elif ftype == "tags":
|
||||
if not isinstance(value, list):
|
||||
problems.append(Problem(name, "wrong-type",
|
||||
f"{name!r} must be a list, got {type(value).__name__}"))
|
||||
elif allowed is not None:
|
||||
for member in value:
|
||||
if str(member) not in allowed:
|
||||
problems.append(Problem(name, "not-in-values",
|
||||
f"{name!r} value {member!r} is not one of {allowed}"))
|
||||
elif ftype == "text":
|
||||
if not _is_scalar(value):
|
||||
problems.append(Problem(name, "wrong-type",
|
||||
f"{name!r} must be a string, got {type(value).__name__}"))
|
||||
return problems
|
||||
@@ -18,6 +18,7 @@ from dataclasses import dataclass, field
|
||||
import yaml
|
||||
|
||||
from . import db
|
||||
from . import metadata_schema
|
||||
from .config import Config
|
||||
from .gitea import Gitea
|
||||
|
||||
@@ -159,6 +160,13 @@ def parse_collection_manifest(text: str) -> CollectionEntry:
|
||||
if not isinstance(em, list):
|
||||
raise RegistryError("collection enabled_models must be a list")
|
||||
cfg["enabled_models"] = [str(m) for m in em]
|
||||
# §22.4a SLICE-2: the collection's metadata field schema. Parsed leniently —
|
||||
# a bad field def is skipped with a warning, never fatal (INV-3), so a typo
|
||||
# in one field can't drop the whole collection from the mirror. Stored only
|
||||
# when at least one valid field survives.
|
||||
fields = metadata_schema.parse_fields(raw.get("fields"))
|
||||
if fields:
|
||||
cfg["fields"] = fields
|
||||
return CollectionEntry(ctype, vis, initial_state, name, cfg)
|
||||
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
subfolder_of."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
@@ -62,3 +63,23 @@ def test_get_collection_and_subfolder():
|
||||
assert collections_mod.subfolder_of("features") == "features"
|
||||
assert collections_mod.subfolder_of("default") == ""
|
||||
assert collections_mod.get_collection("nope") is None
|
||||
|
||||
|
||||
# ---- §22.4a SLICE-2: field schema served on the collection ----
|
||||
|
||||
def test_get_collection_fields_none_when_unset():
|
||||
# INV-5: a collection with no `fields:` exposes fields=None (the default
|
||||
# `document` collection sees zero change).
|
||||
_db()
|
||||
_seed()
|
||||
assert collections_mod.get_collection("features")["fields"] is None
|
||||
|
||||
|
||||
def test_get_collection_exposes_field_schema():
|
||||
_db()
|
||||
_seed()
|
||||
schema = {"priority": {"type": "enum", "values": ["P0", "P1"]}}
|
||||
db.conn().execute(
|
||||
"UPDATE collections SET config_json = ? WHERE id = 'features'",
|
||||
(json.dumps({"fields": schema}),))
|
||||
assert collections_mod.get_collection("features")["fields"] == schema
|
||||
|
||||
@@ -61,6 +61,36 @@ def test_parse_collection_manifest_rejects_bad_visibility():
|
||||
registry.parse_collection_manifest("type: bdd\nvisibility: nope\n")
|
||||
|
||||
|
||||
# ---- §22.4a SLICE-2: a `fields:` block flows into the collection config ----
|
||||
|
||||
def test_parse_collection_manifest_reads_field_schema():
|
||||
doc = registry.parse_collection_manifest(
|
||||
"type: bdd\n"
|
||||
"fields:\n"
|
||||
" priority:\n"
|
||||
" type: enum\n"
|
||||
" values: [P0, P1, P2]\n"
|
||||
" tags:\n"
|
||||
" type: tags\n"
|
||||
)
|
||||
assert doc.config["fields"] == {
|
||||
"priority": {"type": "enum", "values": ["P0", "P1", "P2"]},
|
||||
"tags": {"type": "tags"},
|
||||
}
|
||||
|
||||
|
||||
def test_parse_collection_manifest_lenient_on_bad_field():
|
||||
# A bad field def is skipped (INV-3), the manifest still parses, and a
|
||||
# manifest with no surviving fields carries no `fields` config key at all.
|
||||
doc = registry.parse_collection_manifest(
|
||||
"type: bdd\n"
|
||||
"fields:\n"
|
||||
" broken:\n"
|
||||
" type: ref\n"
|
||||
)
|
||||
assert "fields" not in doc.config
|
||||
|
||||
|
||||
# --- mirror discovery ---------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ Per docs/design/2026-06-06-configurable-collection-metadata.md §6.2-6.3.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
@@ -105,6 +106,61 @@ def test_malformed_sidecar_on_migrated_entry_still_loads_flagged(app_with_fake_g
|
||||
assert row["body"].strip() == "Just the body, no frontmatter."
|
||||
|
||||
|
||||
def _set_default_fields_schema(schema):
|
||||
# apply_registry leaves the default collection's config_json untouched, so a
|
||||
# schema set here survives a corpus refresh (refresh_meta_repo only).
|
||||
db.conn().execute(
|
||||
"UPDATE collections SET config_json = ? WHERE id = 'default'",
|
||||
(json.dumps({"fields": schema}),))
|
||||
|
||||
|
||||
# ---- §22.4a SLICE-2: advisory schema validation at ingest (INV-3) ----
|
||||
|
||||
def test_schema_violation_flags_malformed(app_with_fake_gitea):
|
||||
app, fake = app_with_fake_gitea
|
||||
with TestClient(app):
|
||||
_set_default_fields_schema(
|
||||
{"priority": {"type": "enum", "values": ["P0", "P1", "P2"]}})
|
||||
fake.files[("wiggleverse", "meta", "main", "rfcs/bad-prio.md")] = {
|
||||
"content": "---\nslug: bad-prio\ntitle: Bad\nstate: active\npriority: P9\n---\n\nB.\n",
|
||||
"sha": "bp1"}
|
||||
fake.files[("wiggleverse", "meta", "main", "rfcs/good-prio.md")] = {
|
||||
"content": "---\nslug: good-prio\ntitle: Good\nstate: active\npriority: P0\n---\n\nB.\n",
|
||||
"sha": "gp1"}
|
||||
_refresh()
|
||||
|
||||
assert _row("bad-prio")["metadata_malformed"] == 1 # INV-3: flagged
|
||||
assert _row("bad-prio")["title"] == "Bad" # still loads
|
||||
assert _row("good-prio")["metadata_malformed"] == 0
|
||||
|
||||
|
||||
def test_schema_ignores_undeclared_keys(app_with_fake_gitea):
|
||||
# INV-7: keys the schema doesn't declare ride along and never flag malformed.
|
||||
app, fake = app_with_fake_gitea
|
||||
with TestClient(app):
|
||||
_set_default_fields_schema(
|
||||
{"priority": {"type": "enum", "values": ["P0", "P1"]}})
|
||||
fake.files[("wiggleverse", "meta", "main", "rfcs/extra-key.md")] = {
|
||||
"content": "---\nslug: extra-key\ntitle: Extra\nstate: active\nowner: hasan\n---\n\nB.\n",
|
||||
"sha": "ek1"}
|
||||
_refresh()
|
||||
|
||||
assert _row("extra-key")["metadata_malformed"] == 0
|
||||
|
||||
|
||||
def test_no_schema_never_flags(app_with_fake_gitea):
|
||||
# INV-5: a collection with no field schema validates nothing, even when an
|
||||
# entry carries values that would fail a schema if one existed.
|
||||
app, fake = app_with_fake_gitea
|
||||
with TestClient(app):
|
||||
fake.files[("wiggleverse", "meta", "main", "rfcs/anything.md")] = {
|
||||
"content": "---\nslug: anything\ntitle: Any\nstate: active\npriority: whatever\n---\n\nB.\n",
|
||||
"sha": "an1"}
|
||||
_refresh()
|
||||
|
||||
assert _row("anything")["metadata_malformed"] == 0
|
||||
|
||||
|
||||
def test_legacy_collection_without_sidecars_unchanged(app_with_fake_gitea):
|
||||
app, fake = app_with_fake_gitea
|
||||
with TestClient(app):
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
"""SLICE-2 unit tests — collection field schema + central validation.
|
||||
|
||||
Pure functions only (no DB / no Gitea). Per
|
||||
docs/design/2026-06-06-configurable-collection-metadata.md §7.2 (SLICE-2):
|
||||
`metadata_schema.parse_fields` (lenient schema parsing) and
|
||||
`metadata_schema.validate` (advisory at read / enforcement point at write).
|
||||
Honors INV-3 (never hard-fails), INV-5 (no-fields unchanged), INV-7 (undeclared
|
||||
keys ride along).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from app import metadata_schema as ms
|
||||
|
||||
|
||||
# ---- parse_fields: normalization + leniency ----
|
||||
|
||||
def test_parse_fields_each_type():
|
||||
raw = {
|
||||
"priority": {"type": "enum", "values": ["P0", "P1", "P2"], "label": "Priority"},
|
||||
"tags": {"type": "tags"},
|
||||
"owner": {"type": "text"},
|
||||
}
|
||||
fields = ms.parse_fields(raw)
|
||||
assert list(fields) == ["priority", "tags", "owner"] # order preserved
|
||||
assert fields["priority"] == {
|
||||
"type": "enum",
|
||||
"values": ["P0", "P1", "P2"],
|
||||
"label": "Priority",
|
||||
}
|
||||
assert fields["tags"] == {"type": "tags"}
|
||||
assert fields["owner"] == {"type": "text"}
|
||||
|
||||
|
||||
def test_parse_fields_controlled_tags_keeps_values():
|
||||
fields = ms.parse_fields({"area": {"type": "tags", "values": ["a", "b"]}})
|
||||
assert fields["area"] == {"type": "tags", "values": ["a", "b"]}
|
||||
|
||||
|
||||
def test_parse_fields_missing_block_is_empty():
|
||||
assert ms.parse_fields(None) == {}
|
||||
assert ms.parse_fields({}) == {}
|
||||
|
||||
|
||||
def test_parse_fields_non_mapping_block_skipped():
|
||||
assert ms.parse_fields(["not", "a", "mapping"]) == {}
|
||||
assert ms.parse_fields("nope") == {}
|
||||
|
||||
|
||||
def test_parse_fields_enum_without_values_skipped():
|
||||
# enum requires a non-empty values list — skipped, not fatal.
|
||||
assert ms.parse_fields({"p": {"type": "enum"}}) == {}
|
||||
assert ms.parse_fields({"p": {"type": "enum", "values": []}}) == {}
|
||||
|
||||
|
||||
def test_parse_fields_unknown_type_skipped():
|
||||
fields = ms.parse_fields(
|
||||
{"good": {"type": "text"}, "bad": {"type": "ref"}, "huh": {"type": "frob"}}
|
||||
)
|
||||
assert list(fields) == ["good"]
|
||||
|
||||
|
||||
def test_parse_fields_non_mapping_def_skipped():
|
||||
fields = ms.parse_fields({"good": {"type": "text"}, "bad": "scalar"})
|
||||
assert list(fields) == ["good"]
|
||||
|
||||
|
||||
def test_parse_fields_values_coerced_to_str_list():
|
||||
fields = ms.parse_fields({"p": {"type": "enum", "values": [0, 1, 2]}})
|
||||
assert fields["p"]["values"] == ["0", "1", "2"]
|
||||
|
||||
|
||||
# ---- validate: advisory problem reporting ----
|
||||
|
||||
SCHEMA = ms.parse_fields(
|
||||
{
|
||||
"priority": {"type": "enum", "values": ["P0", "P1", "P2"]},
|
||||
"tags": {"type": "tags"},
|
||||
"area": {"type": "tags", "values": ["checkout", "cart"]},
|
||||
"owner": {"type": "text"},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def test_validate_happy():
|
||||
values = {
|
||||
"priority": "P0",
|
||||
"tags": ["anything", "free"],
|
||||
"area": ["checkout"],
|
||||
"owner": "ben",
|
||||
}
|
||||
assert ms.validate(values, SCHEMA) == []
|
||||
|
||||
|
||||
def test_validate_absent_fields_ok():
|
||||
# A declared field that the entry omits is fine (no required fields in v1).
|
||||
assert ms.validate({}, SCHEMA) == []
|
||||
|
||||
|
||||
def test_validate_enum_bad_value():
|
||||
problems = ms.validate({"priority": "P9"}, SCHEMA)
|
||||
assert [p.field for p in problems] == ["priority"]
|
||||
assert problems[0].code == "not-in-values"
|
||||
|
||||
|
||||
def test_validate_enum_wrong_type():
|
||||
problems = ms.validate({"priority": ["P0"]}, SCHEMA)
|
||||
assert problems[0].field == "priority"
|
||||
assert problems[0].code == "wrong-type"
|
||||
|
||||
|
||||
def test_validate_tags_free_form_ok():
|
||||
assert ms.validate({"tags": ["x", "y", "z"]}, SCHEMA) == []
|
||||
|
||||
|
||||
def test_validate_tags_wrong_type():
|
||||
problems = ms.validate({"tags": "notalist"}, SCHEMA)
|
||||
assert problems[0].field == "tags"
|
||||
assert problems[0].code == "wrong-type"
|
||||
|
||||
|
||||
def test_validate_controlled_tags_bad_member():
|
||||
problems = ms.validate({"area": ["checkout", "nope"]}, SCHEMA)
|
||||
assert problems[0].field == "area"
|
||||
assert problems[0].code == "not-in-values"
|
||||
|
||||
|
||||
def test_validate_text_wrong_type():
|
||||
problems = ms.validate({"owner": ["a", "b"]}, SCHEMA)
|
||||
assert problems[0].field == "owner"
|
||||
assert problems[0].code == "wrong-type"
|
||||
|
||||
|
||||
def test_validate_undeclared_keys_ignored():
|
||||
# INV-7: keys outside the schema ride along untouched, never flagged.
|
||||
assert ms.validate({"random": "value", "slug": "x", "title": "y"}, SCHEMA) == []
|
||||
|
||||
|
||||
def test_validate_empty_schema_no_problems():
|
||||
# INV-5: a collection with no fields validates everything as clean.
|
||||
assert ms.validate({"priority": "anything", "x": 1}, {}) == []
|
||||
|
||||
|
||||
def test_problem_as_dict():
|
||||
p = ms.Problem(field="priority", code="not-in-values", message="bad")
|
||||
assert p.as_dict() == {
|
||||
"field": "priority",
|
||||
"code": "not-in-values",
|
||||
"message": "bad",
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
# Implementation plan — SLICE-2: collection field schema + central validation
|
||||
|
||||
**Slice:** SLICE-2 of
|
||||
[`docs/design/2026-06-06-configurable-collection-metadata.md`](../2026-06-06-configurable-collection-metadata.md)
|
||||
§7.2. **Session:** OHM-0085. **Branch:** `worktree-metadata-slice2-schema`.
|
||||
|
||||
## Goal / Definition of Done (from the design)
|
||||
|
||||
- `.collection.yaml` `fields:` block is parsed and stored.
|
||||
- `metadata_schema.validate` exists — **advisory at read**, the enforcement
|
||||
point **at write** (write endpoints land in SLICE-4/5; this slice supplies and
|
||||
read-wires the function).
|
||||
- The schema is served via the collection API (`GET …/collections/{id}`).
|
||||
- A collection with **no `fields:`** behaves exactly as today (INV-5) — the
|
||||
§22.13 default `document` collection sees zero change.
|
||||
|
||||
## Design decisions (this slice)
|
||||
|
||||
- **Field-def shape** (design §6.3): `fields:` is an ordered mapping
|
||||
`{name → {type, values?, label?}}`. `type ∈ {enum, tags, text}` (v1).
|
||||
Order is preserved for facet display (SLICE-3).
|
||||
- `enum` — single scalar; requires a non-empty `values:` list.
|
||||
- `tags` — list; `values:` optional (controlled when present, free-form
|
||||
otherwise).
|
||||
- `text` — free string scalar.
|
||||
- **`ref` and `multi-enum` are out of v1** (design §2 future; Q2 leans "later").
|
||||
Unknown field types are **ignored with a warning** (design §6.3), not fatal.
|
||||
- **Lenient schema parsing.** A malformed `fields:` block or an individual bad
|
||||
field def is skipped with a warning, never raised — a typo in one field must
|
||||
not nuke the collection mirror (INV-3 spirit). Structural manifest errors
|
||||
(`type`, `visibility`) keep raising `RegistryError` as before.
|
||||
- **No DB migration.** The normalized `fields` schema rides in the existing
|
||||
`collections.config_json` column, exactly like `enabled_models`.
|
||||
- **`values` source for validation.** An entry's full metadata mapping
|
||||
(`metadata.metadata_dict(entry)` = `to_frontmatter_dict`) — `tags` come from
|
||||
the known `Entry.tags`; custom fields (e.g. `priority`) come from
|
||||
`Entry.extra`. Undeclared keys are forward-compat (INV-7) and never flagged.
|
||||
|
||||
## Tasks
|
||||
|
||||
### 1. `app/metadata_schema.py` (new) — the one place that knows field shapes
|
||||
- `VALID_FIELD_TYPES = {"enum", "tags", "text"}`.
|
||||
- `parse_fields(raw) -> dict[str, dict]` — lenient/normalizing. Returns an
|
||||
ordered mapping of `{name: {"type", "values"?, "label"?}}`. Skips: non-mapping
|
||||
block; non-mapping field def; unknown/missing type; `enum` without a non-empty
|
||||
`values:` list. Logs a warning per skip. Pure (no I/O).
|
||||
- `@dataclass Problem(field, code, message)` + `as_dict()`.
|
||||
- `validate(values: dict, fields: dict[str, dict]) -> list[Problem]` — for each
|
||||
**declared** field, validate the entry's value (absent is OK):
|
||||
- `enum`: scalar ∈ `values:` else `not-in-values`; a list/non-scalar →
|
||||
`wrong-type`.
|
||||
- `tags`: must be a list (`wrong-type` otherwise); if controlled, each member
|
||||
∈ `values:` else `not-in-values`.
|
||||
- `text`: must be a scalar string (`wrong-type` otherwise).
|
||||
Undeclared keys ignored (INV-7). Never raises.
|
||||
- **Tests** `tests/test_metadata_schema.py`: parse (each type, missing block,
|
||||
enum-without-values skipped, unknown type skipped, order preserved); validate
|
||||
(happy, enum bad value, tags uncontrolled-ok, tags controlled-bad, text wrong
|
||||
type, undeclared key ignored, empty schema → no problems).
|
||||
|
||||
### 2. `registry.py` — parse `fields:` into the collection config
|
||||
- In `parse_collection_manifest`, after the existing keys: if `raw.get("fields")`
|
||||
present, `cfg["fields"] = metadata_schema.parse_fields(raw["fields"])` (only
|
||||
set when non-empty). Flows into `CollectionEntry.config` →
|
||||
`config_json` via the existing `json.dumps(ce.config)` in
|
||||
`_upsert_named_collection`.
|
||||
- **Default collection** (`apply_registry`) currently writes the `collections`
|
||||
row **without** `config_json`. A default collection's `fields:` would live in
|
||||
`projects.yaml`? No — the design says `fields:` is a `.collection.yaml` block.
|
||||
The default collection has no `.collection.yaml`. **Decision:** default-
|
||||
collection field schemas are out of this slice's happy path (the N=1 default is
|
||||
`document` with no fields, INV-5). Leave `apply_registry` untouched; only
|
||||
named collections (with a `.collection.yaml`) carry `fields:`. Documented as a
|
||||
known limitation (a deployment wanting fields on its primary corpus declares a
|
||||
named collection — consistent with the design's opt-in story).
|
||||
- **Tests** extend `tests/test_collection_registry.py`: a manifest with a
|
||||
`fields:` block round-trips into `get_collection(...)["fields"]`; a manifest
|
||||
with a bad field def still upserts (lenient).
|
||||
|
||||
### 3. `collections.py` — unpack `fields` on read + serve via API
|
||||
- Add `_fields_from_config(config_json) -> dict | None` (mirror
|
||||
`_enabled_models_from_config`).
|
||||
- `get_collection` sets `out["fields"] = _fields_from_config(config_json)`
|
||||
(alongside `enabled_models`). `api_collections.get_col` then serves it with no
|
||||
change. `list_collections` left as-is (facets are SLICE-3).
|
||||
- **Tests** extend `tests/test_collection_helpers.py`: `get_collection` exposes
|
||||
`fields`; a no-fields collection → `fields is None` (INV-5).
|
||||
|
||||
### 4. `cache.py` — advisory validation at ingest (INV-3)
|
||||
- In `_refresh_collection_corpus`, fetch the collection's `fields` schema once
|
||||
(`collections.get_collection(collection_id)`); for each entry, if a schema is
|
||||
present, `problems = metadata_schema.validate(metadata.metadata_dict(entry),
|
||||
fields)` and OR any problems into `metadata_malformed` (warn-log a summary).
|
||||
No schema → behavior identical to today (INV-5).
|
||||
- **Tests** `tests/test_metadata_cache.py` (extend): an entry violating an enum
|
||||
field ingests with `metadata_malformed = 1`; a conforming entry → `0`; a
|
||||
collection with no schema → `0` regardless of extra keys.
|
||||
|
||||
### 5. Verify · version · ship
|
||||
- `pytest` full backend suite green (575 baseline + new).
|
||||
- Bump `VERSION` + `frontend/package.json` → **0.48.0**; CHANGELOG minor entry
|
||||
(§20) — non-breaking, opt-in, N=1 unchanged; note write-enforcement lands with
|
||||
SLICE-4/5.
|
||||
- Commit (cite design §7.2 SLICE-2), PR on Gitea `origin`, merge to `main`.
|
||||
|
||||
## Invariants honored
|
||||
INV-3 (read never hard-fails — advisory malformed), INV-5 (no-`fields:`
|
||||
unchanged), INV-7 (undeclared keys ride along, never flagged), INV-8 (additive,
|
||||
read-mostly; no write-path fork — write enforcement is a later slice).
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "rfc-app-frontend",
|
||||
"private": true,
|
||||
"version": "0.47.0",
|
||||
"version": "0.48.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
|
||||
Reference in New Issue
Block a user