diff --git a/docs/design/2026-06-06-configurable-collection-metadata.md b/docs/design/2026-06-06-configurable-collection-metadata.md new file mode 100644 index 0000000..924154f --- /dev/null +++ b/docs/design/2026-06-06-configurable-collection-metadata.md @@ -0,0 +1,359 @@ +# Solution Design: Configurable Collection Metadata (clean-doc tagging) + +| | | +| --- | --- | +| **Author(s)** | Ben Stull | +| **Reviewers / approvers** | Ben Stull | +| **Status** | `draft` | +| **Version** | v0.1.0 | +| **Source artifacts** | Reference modeled: retired **BDD Release Planner** (`wiggleverse/wiggleverse-ecomm-bdd-release-planner-app`, RETIRED 2026-06-04) · Related spec: [`docs/design/2026-06-05-three-tier-projects-collections.md`](./2026-06-05-three-tier-projects-collections.md) (§22) · Corpus: ecomm Shopify-modeled BDD (`wiggleverse-ecomm-meta/research/shopify`, ~1,238 scenarios) · Supersedes: — | + +**Change log** + +| Date | Version | Change | By | +| --- | --- | --- | --- | +| 2026-06-06 | v0.1.0 | Initial draft from discovery session OHM-0079.0 | Ben Stull | + +--- + +## 1. Executive Summary + +rfc-app gains a generic, **per-collection-configurable metadata system**. A collection declares its fields — `priority`, `tags`, and arbitrary custom fields — in its `.collection.yaml`; each entry's *values* live in a **sidecar** (`.meta.yaml`) so the document body stays pure prose. rfc-app renders schema-driven forms and **faceted left-pane filters**, and supports fast **single + bulk tag/untag** via direct commits for authorized roles. This restores the *corpus-annotation* capability of the retired BDD Release Planner inside the framework, while keeping release *planning* (ordering, ship status, roadmap emission) a downstream concern that reads the sidecars straight from git. + +## 2. Business Context + +The framework hosts RFC standardization for multiple deployments. One deployment hosts the ecomm BDD corpus — ~1,238 Shopify-modeled scenarios, one markdown file per scenario, slugged by feature ID (`DD-FF-NNNN-slug`). A standalone **BDD Release Planner** previously let operators search that corpus, attach metadata (priority P0–P3, owner, status), cluster scenarios into named releases, and emit each release as a roadmap phase. §22 (three-tier projects/collections) absorbed the planner's *corpus hosting* into rfc-app (the corpus now runs as a `bdd` project on the RFC deployment) and the planner was retired — but its *annotation* half (priority/tags on scenarios, filtering, bulk assignment) was never rebuilt. Operators planning work, and downstream tools consuming the corpus, currently have no structured, framework-native way to set or read that metadata. + +## 3. Problem Statement + +Today rfc-app metadata is weak and intrusive: + +- **Tags are free-form strings** with **no filtering** — §7.1 specifies a `Tag:` filter chip that was never implemented. +- **No priority**, and no way for a collection to declare any other structured field. +- **No per-collection schema** — every collection gets the same fixed frontmatter shape; a `bdd` collection cannot say "scenarios have a P0–P3 priority." +- **Metadata pollutes the document body** — a large YAML frontmatter block sits at the top of every doc, mixing rfc-app lifecycle bookkeeping with content metadata. +- **No bulk workflow** — annotating hundreds of scenarios one PR at a time is impractical, so the planner's core gesture has no analog. + +Result: corpus authors can't meaningfully prioritise/tag, downstream consumers have nothing structured to read, and documents aren't clean. + +## 4. Stakeholders / Personas / Actors + +| Actor | Type | Goal in this design | +| --- | --- | --- | +| Corpus contributor | persona | Set priority and tags on a scenario when proposing/curating it | +| Release planner (operator) | persona | Bulk-prioritise/tag many scenarios quickly to shape a plan | +| Collection owner | operator | Declare the metadata fields a collection supports | +| Downstream consumer | system | Read structured per-scenario metadata from the git corpus (e.g. an external release planner) | +| Reader | persona | Read clean scenario docs; filter the catalog by priority/tag | + +## 5. Scope + +- **In scope:** per-collection field schema in `.collection.yaml` (`enum`, `tags`, `text`); per-entry sidecar (`.meta.yaml`) as source of truth with pure-prose doc bodies; schema-derived faceted left-pane filtering with counts; schema-derived metadata form (detail panel); single + bulk tag/untag with direct commit for authorized roles; dual-read compatibility + a one-shot frontmatter→sidecar migration tool + lazy migration on write. +- **Out of scope:** release ordering, ship status, roadmap emission, `RELEASE-PLAN.md` generation — these stay downstream, consuming sidecars from git. In-app management of field definitions / controlled vocabularies; corpus-wide tag rename/merge/delete. Sub-document (per-scenario-within-a-file) grain. A whole-corpus metadata export endpoint. +- **Non-goals:** rebuilding a bespoke "release" entity in rfc-app — releases are not modeled here at all; metadata is the only primitive. + +## 6. Assumptions · Constraints · Dependencies + +- **Assumptions:** git remains the content source of truth and downstream consumers can read the corpus from git; the BDD grain is one markdown file per scenario (already true for the ecomm corpus), so no sub-document parsing is needed. +- **Constraints:** rfc-app is a framework hosting multiple deployments — the upgrade must be **mechanical and non-breaking**, with §20 changelog/upgrade-steps; the hard secrets rule (§6.3) holds; metadata edits must respect scope-role authorization (§22 Part B / S3). +- **Dependencies:** the S3 scope-role resolver (`auth.effective_scope_role`) for edit authorization; the existing git write-through path used by `edit-meta` (§9.5); the §22 collection model (`.collection.yaml`, `cached_rfcs` keyed by `(collection_id, slug)`). + +## 7. Targeted Business Outcomes + +| Outcome | Success metric | Baseline → Target | Guardrail (must not regress) | How / when measured | +| --- | --- | --- | --- | --- | +| Authors can prioritise/tag scenarios | scenarios carrying a priority | 0 → corpus-wide | doc bodies stay prose-clean | corpus inspection after rollout | +| Corpus is filterable | left-pane facet filters available | none → priority+tags+state | filter latency acceptable at ~1.2k entries | manual + perf check | +| Downstream tools can consume metadata | sidecars readable from git | none → all migrated entries | sidecar schema stable | consumer integration | +| Clean docs | docs with no rfc-app frontmatter | 0% → 100% (post-migration) | dual-read keeps legacy working | post-migration inspection | + +## 8. Business Use Cases + +```gherkin +Scenario: BUC-1 — A contributor prioritises a scenario + Given a bdd collection whose schema defines a priority field + When a contributor marks a scenario as P0 + Then the scenario's metadata records priority P0 + And the document body is unchanged prose + +Scenario: BUC-2 — A planner batch-prioritises a set of scenarios + Given a contributor has selected 40 scenarios + When they set priority P1 on the selection + Then all 40 carry priority P1 + And the change lands as a single auditable commit + +Scenario: BUC-3 — A downstream tool consumes priorities + Given scenarios carry priority and tags in their sidecars + When an external release planner reads the corpus from git + Then it can group and order scenarios using that metadata + And rfc-app did not need to model releases at all + +Scenario: BUC-4 — Documents stay clean + Given a migrated collection + When a reader opens a scenario document + Then they see only prose, with no rfc-app metadata block +``` + +- **BUC-1 acceptance:** the scenario's `priority` value is `P0` and its `.md` body is byte-identical to before. +- **BUC-2 acceptance:** 40 sidecars updated; exactly one commit; authorship recorded. +- **BUC-3 acceptance:** sidecars + `.collection.yaml` are sufficient for a consumer to read priority/tags without rfc-app's API. +- **BUC-4 acceptance:** no `---` frontmatter remains in migrated docs. + +## 9. Product Use Cases + +```gherkin +Scenario: PUC-1 — Set priority/tags on a scenario (realizes BUC-1) + Given I am a contributor viewing a scenario whose collection defines priority and tags + When I choose P0 in the priority control and add the tag "checkout" + Then the metadata panel reflects P0 and the checkout tag + And the change is committed directly to the scenario's sidecar + +Scenario: PUC-2 — Bulk tag/untag from the catalog (realizes BUC-2) + Given I have multi-selected several scenarios in the catalog + When I choose "Set priority → P1" from the bulk action bar + Then every selected scenario shows P1 + And the bulk change is one commit + +Scenario: PUC-3 — Filter the catalog by facet (realizes BUC-1/BUC-3) + Given the left pane shows faceted filters generated from the collection schema + When I check Priority P0 and tag "checkout" + Then the catalog shows only scenarios matching both + And each facet value shows its result count + +Scenario: PUC-4 — A collection declares its fields (product-only) + Given a collection owner edits .collection.yaml to add a priority enum field + When the collection is re-ingested + Then the priority filter and the priority form control appear automatically + +Scenario: PUC-5 — Migrate a collection to clean docs (product-only) + Given a collection whose docs still carry top-of-doc frontmatter + When the operator runs the frontmatter→sidecar migration + Then each doc body becomes pure prose and a sidecar holds its metadata + And rfc-app reads the collection identically before and after +``` + +## 10. UX Layout + +### 10.1 Screen: Catalog (left pane) (serves PUC-3) + +- **Purpose:** browse and filter a collection's entries. +- **Layout (top → bottom):** + - **Search:** full-text box (existing). + - **Faceted filter groups** (one per schema field + state): each is a collapsible group showing per-value **result counts** and multi-select checkboxes; `tags`-type fields include a "filter values…" search box to stay usable at 30+ values. (Chosen layout: faceted groups with counts — validated in brainstorming over flat chips.) +- **States:** happy: facets with counts · empty: "no entries match these filters" with a clear-filters action · loading: skeleton facets · error: "couldn't load facets" with retry. + +### 10.2 Screen: Scenario detail — metadata panel (serves PUC-1) + +- **Purpose:** view/edit one entry's metadata. +- **Layout:** a panel rendering one control per schema field — `enum` → single-select; `tags` → removable chips + add-tag input (with existing AI suggest where applicable); `text` → text input. The document body renders below as pure prose; metadata never appears inline in the body. +- **States:** read (role without edit) shows values, no controls · edit (authorized) shows controls · saving: inline spinner · error: field-level validation message (e.g. "P5 is not an allowed priority"). + +### 10.3 Screen: Catalog — bulk action bar (serves PUC-2) + +- **Purpose:** apply a field value to many entries at once. +- **Layout:** selecting ≥1 row reveals a sticky action bar: "*N* selected · Set priority ▾ · Add tag ▾ · Remove tag ▾ · Clear". Each action targets one field; applying commits once. +- **States:** none selected: bar hidden · applying: bar shows progress · partial failure: toast naming entries that failed validation, others applied. + +## 11. Technical Design + +### 11.1 Invariants + +- **INV-1:** The sidecar (`.meta.yaml`) is the source of truth for entry metadata; `cached_rfcs` is a derived index, fully rebuildable from git. +- **INV-2:** A document body (`.md`) never contains rfc-app metadata once migrated; metadata lives only in the sidecar. +- **INV-3:** Reading a collection never hard-fails on bad metadata — an invalid value against the schema surfaces as a warning and the entry still loads. +- **INV-4:** Metadata writes are authorized by scope-role (contributor+ on the collection); content-body edits keep their existing PR-review path. +- **INV-5:** A collection with no `fields:` block behaves exactly as today (free-form `tags` only) — the feature is additive and opt-in per collection. +- **INV-6:** Dual-read holds throughout: parser reads the sidecar if present, else legacy top-of-doc frontmatter, with identical resulting in-memory records. + +### 11.2 High-level architecture + +```mermaid +flowchart LR + subgraph Git[content repo] + CY[.collection.yaml
fields: schema] + MD[slug.md
prose body] + SC[slug.meta.yaml
values] + end + CY --> ING[ingest / parser
validate vs schema] + MD --> ING + SC --> ING + ING --> DB[(cached_rfcs
values + facet counts)] + DB --> API[API: schema · list+filter · facets · edit] + API --> FILT[left-pane faceted filters] + API --> PANEL[detail metadata panel] + API --> BULK[bulk select bar] + PANEL -->|direct commit| SC + BULK -->|1 commit| SC + SC -.read from git.-> CONS[downstream consumers] +``` + +- **ingest/parser** — owns reading `.collection.yaml` schema + sidecars (or legacy frontmatter), validating values, and rebuilding `cached_rfcs`; must never treat the DB as authoritative. +- **API** — owns serving the schema, filtered lists with facet counts, and metadata edits; must never write metadata anywhere but the sidecar in git. + +### 11.3 Data model & ownership + +| Entity | Owned by | Key fields | System of record | +| --- | --- | --- | --- | +| Collection field schema | collection owner | `fields: {name → {type, values?, label}}` in `.collection.yaml` | git | +| Entry metadata values | contributor | sidecar `.meta.yaml`: lifecycle (`slug,title,state,owners,…`) + schema fields (`priority,tags,…`) | git (sidecar) | +| Derived index | ingest | per-entry values + facet aggregations | `cached_rfcs` (SQLite, derived) | + +**Field types (v1):** `enum` (single-select; controlled by required `values:`), `tags` (multi-value; free-form unless `values:` given), `text` (free string). Unknown types are ignored with a warning (forward-compat). + +**Sidecar example:** +```yaml +slug: 01-01-0001-view-today-s-key-performance-metrics-at-a-glance +title: View today's key performance metrics at a glance +state: active +owners: [ben.stull] +priority: P1 +tags: [dashboard, analytics] +owner: hasan +``` + +### 11.4 Interfaces & contracts + +- **`GET /api/projects/

/collections/`** — out: collection incl. `fields` schema. Errors: 404. +- **`GET /api/projects/

/collections//rfcs`** — in: filter params (`?priority=P0&tags=checkout&state=active`, repeatable for multi-value/OR-within-field, AND across fields) · out: entries with metadata values **+ `facets: {field → {value → count}}`**. Errors: 400 on unknown field. +- **`POST /api/projects/

/collections//rfcs//meta`** — in: `{field: value, …}` · out: updated values · effect: validate vs schema → write sidecar → commit directly → re-ingest entry. Errors: 403 (role), 422 (invalid value). +- **`POST /api/projects/

/collections//meta/bulk`** — in: `{slugs: […], op: set|add|remove, field, value}` · out: `{applied: […], rejected: [{slug, reason}]}` · effect: validate → write N sidecars → **one** commit → re-ingest. Errors: 403, 422. + +### 11.5 Per–Product-Use-Case design + +#### PUC-2 — Bulk tag/untag + +```mermaid +sequenceDiagram + actor U as Contributor + participant C as Catalog UI + participant A as API + participant G as Git + participant D as cached_rfcs + U->>C: select rows, "Set priority P1" + C->>A: POST /meta/bulk {slugs, set, priority, P1} + A->>A: authz (scope-role) + validate vs schema + A->>G: write N sidecars, 1 commit + A->>D: re-ingest affected entries + A-->>C: {applied, rejected} + C-->>U: rows show P1; toast on any rejected +``` + +- **Implementation:** reuse the `edit-meta` git write-through, extended to (a) target the sidecar rather than frontmatter and (b) batch N files into one commit. Honors INV-1/INV-4. + +#### PUC-5 — Migration + +- **Implementation:** a tool (framework CLI verb / `tools/` script) walks a collection, and for each entry with legacy frontmatter, writes `.meta.yaml` from the frontmatter and rewrites `.md` to the body only — one commit per collection, idempotent. Dual-read (INV-6) means this can run anytime; lazy migration converts stragglers on their first metadata edit. + +### 11.6 Non-functional requirements & cross-cutting concerns + +- **Security & privacy:** metadata edits gated by `auth.effective_scope_role` (contributor+ on the collection); no secrets in sidecars; git history records authorship. +- **Performance & scale:** facet counts computed from the derived DB; must stay responsive at ~1.2k entries with dozens of tag values (indexed value columns / aggregation query). +- **Availability & resilience:** bad metadata never blocks read (INV-3); a failed re-ingest leaves git authoritative and is recoverable by full rebuild. +- **Observability:** log each metadata commit (actor, field, entry count); warn-log schema validation failures encountered on ingest. +- **Accessibility:** facet groups and form controls keyboard-navigable; checkboxes labelled with value + count. + +### 11.7 Key decisions & alternatives considered + +| Decision | Chosen | Alternatives | Why | +| --- | --- | --- | --- | +| Release modeling | Metadata only; releases downstream | First-class release entity in rfc-app; release-typed tags with behavior | Operator pulled ordering/ship-status out of rfc-app; metadata is the only needed primitive | +| Tag system shape | One generic typed-field system (Approach A) | Releases first-class + simple tags; namespaced facets | Tags/priority/custom are all just fields; one mechanism | +| Metadata storage | Sidecar per entry | Top-of-doc frontmatter (today); end-of-doc block; collection index file; DB-only | Clean docs + git-visible to consumers + locality per scenario | +| Left-pane filtering | Faceted groups with counts | Flat facet chips | Scales to the ~1.2k-scenario, many-tag ecomm corpus | +| Edit governance | Direct commit for authorized roles (bulk = 1 commit) | PR per change | Bulk planning is impractical via PR-per-toggle | +| Mgmt UI | Deferred; edit `.collection.yaml` in git | In-app field/vocab management in v1 | Smallest coherent v1 | + +### 11.8 Testing strategy + +Unit tests for: schema parsing (`fields:` block, all types, missing block); sidecar read/write round-trip; dual-read equivalence (frontmatter vs sidecar produce identical records); schema validation (reject bad enum, accept free-form tag); facet aggregation; bulk op (set/add/remove, single commit, partial-rejection). Two-tier local-Docker→PPE for the API + git write-through. "Tested" = the PUC acceptance scenarios pass plus the migration is proven idempotent and reversible-on-read. + +### 11.9 Failure modes, rollback & flags + +- **Failure mode:** invalid value committed out-of-band → on ingest, warn + load entry with the raw value flagged (INV-3); not surfaced as a filter facet count error. +- **Failure mode:** re-ingest fails after commit → git is authoritative; full rebuild recovers. +- **Migration rollback:** dual-read means an un-migrated or partially-migrated corpus still works; the migration commit is revertible. +- **Feature flag:** the feature is inherently opt-in per collection (INV-5) — no global flag needed; absent a `fields:` block, behavior is unchanged. + +## 12. Delivery Plan + +### 12.1 Approach / strategy + +Build the storage/compat foundation first (sidecars + dual-read + migration) so nothing breaks, then the schema, then read (filtering), then write (single, bulk). Each slice is shippable and non-breaking. + +### 12.2 Slicing plan + +#### SLICE-1 — Sidecar storage + dual-read + migration → completes PUC-5 +- **Depends on:** — +- **DoD:** parser reads sidecar-if-present else legacy frontmatter (INV-6); migration tool splits frontmatter→sidecar idempotently; existing collections load byte-identically; tests green. + +#### SLICE-2 — Collection field schema + validation → completes PUC-4 +- **Depends on:** SLICE-1 +- **DoD:** `.collection.yaml fields:` parsed (`enum`/`tags`/`text`); values validated on read (warn) and on write (reject); schema served via the collection API; no-`fields:` collections unchanged (INV-5). + +#### SLICE-3 — Faceted left-pane filtering (read) → completes PUC-3 +- **Depends on:** SLICE-2 +- **DoD:** list endpoint returns facet counts + honors filter params; left pane renders faceted groups with counts and tag-value search; filters compose (AND across fields). + +#### SLICE-4 — Single-entry metadata edit → completes PUC-1 +- **Depends on:** SLICE-2 +- **DoD:** detail metadata panel renders schema controls; `POST …/meta` validates, direct-commits the sidecar, re-ingests; authorized by scope-role (INV-4); lazy-migrates a legacy entry on first edit. + +#### SLICE-5 — Bulk tag/untag → completes PUC-2 +- **Depends on:** SLICE-3, SLICE-4 +- **DoD:** multi-select + bulk action bar; `POST …/meta/bulk` applies set/add/remove as one commit; partial-rejection reported. + +### 12.3 Rollout / launch plan + +Pre-v1, single production: ship slices in order to the RFC deployment; each minor-version bump carries §20 changelog + upgrade steps. The opt-in-per-collection nature (INV-5) means a deployment adopts it only when it declares a `fields:` block and (optionally) runs the migration. + +### 12.4 Risks & mitigations + +| Risk | L/I | Mitigation | +| --- | --- | --- | +| Frontmatter→sidecar migration corrupts content | L/H | Dual-read; idempotent, revertible migration; body-byte-identity test | +| Doubling file count (sidecars) clutters corpus | M/L | Docs stay clean; sidecars are small/co-located; acceptable for one-file-per-scenario corpora | +| Direct-commit metadata edits bypass review | M/M | Scope-role gate (INV-4); content-body edits still PR'd; full git audit trail | +| Overlap/conflict with §22 S6 "type modules" | M/M | Position as §23, generalizing S6's per-type frontmatter; reconcile at the S6 SPEC merge | +| Facet aggregation slow at scale | L/M | Compute from indexed derived DB; measure at ~1.2k entries | + +## 13. Traceability matrix + +| Business UC | Product UC | Slice | Tests | +| --- | --- | --- | --- | +| BUC-4 | PUC-5 | SLICE-1 | `test_dual_read_equiv`, `test_migration_idempotent` | +| — (product-only) | PUC-4 | SLICE-2 | `test_schema_parse`, `test_validation` | +| BUC-1/BUC-3 | PUC-3 | SLICE-3 | `test_facet_counts`, `test_filter_compose` | +| BUC-1 | PUC-1 | SLICE-4 | `test_single_meta_commit`, `test_authz` | +| BUC-2 | PUC-2 | SLICE-5 | `test_bulk_one_commit`, `test_partial_reject` | +| BUC-3 | (consumer reads git) | — | `test_sidecar_schema_stable` | + +## 14. Open Questions & Decisions log + +**Open** + +| # | Question | Owner | Blocks | +| --- | --- | --- | --- | +| Q1 | Do downstream consumers read sidecars from git, via rfc-app API, or both? (leaning git) | Ben | nothing v1 | +| Q2 | Should a `multi-enum` type (multi-select controlled) ship in v1 or later? | Ben | SLICE-2 scope | +| Q3 | Exact §23 placement / reconciliation with §22 S6 type-modules | Ben | SPEC merge | + +**Resolved** + +| # | Decision | Resolution | Date | +| --- | --- | --- | --- | +| D1 | Release behaviors (ordering, ship status, roadmap emit) | Out of rfc-app; downstream | 2026-06-06 | +| D2 | Tag system shape | Approach A — one generic typed-field system | 2026-06-06 | +| D3 | Metadata grain | Per entry (corpus already one file per scenario) | 2026-06-06 | +| D4 | Schema location | `.collection.yaml` `fields:` block | 2026-06-06 | +| D5 | Value storage | Sidecar `.meta.yaml`; doc body pure prose | 2026-06-06 | +| D6 | Left-pane filtering | Faceted groups with counts | 2026-06-06 | +| D7 | Edit governance | Direct commit for authorized roles; bulk = 1 commit | 2026-06-06 | +| D8 | Management scope | Deferred; edit `.collection.yaml` in git for v1 | 2026-06-06 | + +## 15. Glossary & References + +- **Sidecar** — `.meta.yaml`, the per-entry metadata file that is the source of truth; keeps the `.md` body pure prose. +- **Field schema** — the `fields:` block in `.collection.yaml` declaring a collection's typed metadata fields. +- **Facet** — a schema field surfaced as a left-pane filter group with per-value counts. +- **Downstream consumer** — an external tool (e.g. a release planner) that reads corpus metadata from git; rfc-app does not model releases. +- **References:** retired BDD Release Planner (`wiggleverse-ecomm-bdd-release-planner-app`); §22 three-tier design (`docs/design/2026-06-05-three-tier-projects-collections.md`); SPEC §7.1 (left-pane filter), §9.5 (edit-meta), §20 (versioning), §22 Part B / S3 (scope-role).