From 6f356d3598b1bf12f8e07f204a49dabf3a69dec4 Mon Sep 17 00:00:00 2001 From: Ben Stull Date: Wed, 3 Jun 2026 21:59:20 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20M3-backend=20design=20spec=20(=C2=A722?= =?UTF-8?q?=20registry=20mirror=20+=20data=20spine=20+=20APIs)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brainstormed design for the backend half of §22 slice M3, split at the backend/frontend seam. Covers migration 027 (type/initial_state columns, the 12-table PK rebuilds, default→slug re-stamp), the app/registry.py mirror over REGISTRY_REPO, GET /api/deployment + /api/projects/:id, and the initial_state/unreviewed review semantics. Routing, runtime branding, directory, and switcher are deferred to a later M3-frontend spec. Co-Authored-By: Claude Opus 4.8 --- .../specs/2026-06-03-m3-backend-design.md | 374 ++++++++++++++++++ 1 file changed, 374 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-03-m3-backend-design.md diff --git a/docs/superpowers/specs/2026-06-03-m3-backend-design.md b/docs/superpowers/specs/2026-06-03-m3-backend-design.md new file mode 100644 index 0000000..cf8e35a --- /dev/null +++ b/docs/superpowers/specs/2026-06-03-m3-backend-design.md @@ -0,0 +1,374 @@ +# M3-backend — §22 multi-project: registry mirror + data spine + APIs + +> Design spec for the backend half of §22 slice **M3** ("Registry mirror + +> routing + runtime branding"). The roadmap bundles M3 as one slice; this +> session splits it at the natural backend/frontend seam. **M3-backend** (this +> doc) ships the data spine, the registry mirror, the two runtime-config APIs, +> and the entry-state/review semantics. **M3-frontend** (a separate spec) ships +> `/p//` routing, the 308 redirects, the `VITE_APP_NAME`→runtime-config +> cut, the per-project theme overlay, the deployment directory at `/`, and the +> project switcher — all consuming the APIs defined here. +> +> Section references `§22.x` point at `docs/design/multi-project-spec.md` (the +> draft §22 + slicing plan). The SPEC.md §22 merge itself lands in M7. + +## Status + +- **Date:** 2026-06-03 +- **Slice:** §22 M3 (backend half). M1 + M2 landed and merged to `main`. +- **Version impact:** minor bump, breaking (pre-1.0) — see §8. + +## Goal + +After M3-backend, the framework learns its projects from a git **registry** +(not from `META_REPO`), the `projects` cache table and all slug-bearing tables +are keyed by `(project_id, …)` so a second project can exist without collision, +the default project's identity is re-stamped to its real slug while no `/p/` +URL is yet public, and the runtime exposes deployment + project config over two +new endpoints. The entry-state/review semantics (`initial_state`, `unreviewed`) +ship complete even though OHM (a `document`/`super-draft` project) does not yet +exercise them. + +Non-goals (M3-frontend, later): `/p//` routing, 308 redirects off +`/rfc/`, runtime branding in the UI, theme application, the deployment +directory, the project switcher, the catalog's unreviewed-filter **UI**. + +## Decisions taken in brainstorming + +1. **Scope split** — backend spine first; frontend surfaces are a separate + spec/session. +2. **Review machinery** — build the full `initial_state` / `unreviewed` + plumbing now (parse + columns + mirror + landing logic + mark-reviewed + + catalog filter query side), per the spec's M3 bundle, even though no live + project exercises it yet. +3. **Config cut** — hard cut. `REGISTRY_REPO` required (loud fail if unset), + `META_REPO` removed. Upgrade-steps document the manual registry creation. +4. **Re-stamp** — rewrite `project_id` everywhere: rename `projects.id` + `default` → the config slug and rewrite every child row, folded into the + same create-copy-drop-rename rebuild that adds the `project_id` FK. One + identifier; DB and URL agree. +5. **Mirror structure** — a self-contained `app/registry.py` module (not folded + into `cache.py`), driven by the existing webhook dispatcher + the existing + `Reconciler.sweep()`. + +--- + +## 1. Migration `027_projects_activate.sql` + +Runs while `default` is still the sole project and no `/p/` URL exists — the +safe window for an identity rewrite. One transaction (SQLite DDL is +transactional): the deployment either fully advances to `027` or stays on `026`. + +`DEFAULT_PROJECT_ID` is read at migration time, so it **must be set before the +upgrade deploy** (documented in §8). The slug it names is referred to below as +``; absent the env var, `` stays `default`. + +### 1a. Additive columns + +- `projects`: + - `type TEXT NOT NULL DEFAULT 'document' CHECK (type IN ('document','specification','bdd'))` + - `initial_state TEXT NOT NULL DEFAULT 'super-draft' CHECK (initial_state IN ('super-draft','active'))` +- `cached_rfcs`: + - `unreviewed INTEGER NOT NULL DEFAULT 0` + - `reviewed_at TEXT` + - `reviewed_by TEXT` + +### 1b. New `deployment` singleton table + +Holds deployment-level identity mirrored from the registry's `deployment:` +block (rather than overloading `projects`): + +```sql +CREATE TABLE IF NOT EXISTS deployment ( + id INTEGER PRIMARY KEY CHECK (id = 1), + name TEXT, + tagline TEXT, + registry_sha TEXT, + updated_at TEXT NOT NULL DEFAULT (datetime('now')) +); +INSERT OR IGNORE INTO deployment (id) VALUES (1); +``` + +### 1c. PK / uniqueness rebuilds (the 12 deferred tables) + +Per migration 026's deferred block, create-copy-drop-rename each table to fold +`project_id` into the key and add `project_id … REFERENCES projects(id) ON +DELETE CASCADE`: + +| Table | Key change | +| --- | --- | +| `cached_rfcs` | PK `(slug)` → `(project_id, slug)` | +| `cached_branches` | UNIQUE `(rfc_slug, branch_name)` → `+project_id` | +| `branch_visibility` | UNIQUE `(rfc_slug, branch_name)` → `+project_id` | +| `branch_contribute_grants` | UNIQUE `(rfc_slug, branch_name, grantee_user_id)` → `+project_id` | +| `stars` | UNIQUE `(user_id, rfc_slug)` → `+project_id` | +| `watches` | UNIQUE `(user_id, rfc_slug)` → `+project_id` | +| `pr_seen` | UNIQUE `(user_id, rfc_slug, pr_number)` → `+project_id` | +| `branch_chat_seen` | UNIQUE `(user_id, rfc_slug, branch_name)` → `+project_id` | +| `funder_consents` | PK `(user_id, rfc_slug)` → `+project_id` | +| `rfc_collaborators` | UNIQUE INDEX `(rfc_slug, user_id)` → `+project_id` | +| `contribution_requests` | UNIQUE INDEX `(rfc_slug, requester_user_id) WHERE pending` → `+project_id` | +| `proposed_use_cases` | UNIQUE `(scope, pr_number)` → `+project_id` | + +`cached_prs` is already globally unique (`repo` is the full `org/repo` string, +distinct per project) — **no rebuild**. + +The copy step writes `` in place of `default` for `project_id`, so these +12 tables are re-stamped for free (§1d). + +### 1d. Re-stamp `default` → `` + +- `UPDATE projects SET id = '' WHERE id = 'default'` (only when `` ≠ + `default`). +- The 12 rebuilt tables: re-stamped during their copy (§1c). +- The remaining 7 tables that carry `project_id` but need no key change get a + plain rewrite: `UPDATE SET project_id = '' WHERE project_id = + 'default'` for `threads`, `changes`, `notifications`, `actions`, + `pr_resolution_branches`, `rfc_invitations`, `cached_prs`. + +End state: a single `project_id` value DB-wide. + +### 1e. Notes + +- Migration `016` is absent from the on-disk sequence (`015 → 017`); the runner + already tolerates the gap (the app runs today). **Do not renumber.** New file + is `027`. +- `PRAGMA foreign_keys` is honored going forward; the FK lands on the 12 rebuilt + tables. The other 7 keep app-layer integrity (matching today's posture for + non-rebuilt tables). + +--- + +## 2. Registry format + `app/registry.py` + +### 2a. `projects.yaml` (root of `REGISTRY_REPO`) + +```yaml +deployment: + name: Open Human Model # replaces VITE_APP_NAME (M3-frontend consumes) + tagline: ... +projects: + - id: ohm # url-stable slug, unique in the deployment + name: Open Human Model + type: document # document | specification | bdd — immutable + content_repo: ohm # repo under the deployment's Gitea org + visibility: public # gated | public | unlisted + initial_state: super-draft # optional; defaults from type + enabled_models: [claude, gemini] # optional; falls back to ENABLED_MODELS + theme: { accent: "#5b5bd6" } # optional; M3-frontend consumes +``` + +### 2b. `refresh_registry(config, gitea) -> RegistryResult` + +The config-side analogue of `cache.refresh_meta_repo`. Fetches `projects.yaml` +from `REGISTRY_REPO` at HEAD, parses, **validates**, and upserts: + +- Each project → `projects` row: `id, name, type, content_repo, visibility, + initial_state`, plus `config_json` (JSON blob for `theme`, `enabled_models`), + plus `registry_sha` (commit SHA, provenance). +- The `deployment:` block → the `deployment` singleton (`name`, `tagline`, + `registry_sha`). + +Projects present in the table but absent from the registry are **not** deleted +in M3-backend (archival semantics are out of scope; a removed entry simply +stops being refreshed). This is noted as a known limitation; revisit if/when +project archival is specced. + +### 2c. Validation (loud) + +Per the framework's separation-of-concerns rule, malformed config fails +visibly rather than shipping wrong content silently: + +- Each project requires `id`, `name`, `type`, `content_repo`. +- `type` ∈ {`document`,`specification`,`bdd`}; `visibility` ∈ + {`gated`,`public`,`unlisted`}. +- `initial_state` defaults from `type` when omitted: `document`/`specification` + → `super-draft`, `bdd` → `active`. When present it must be a valid §2.4 + super-draft entry-state value. +- `id` values unique and slug-shaped (`^[a-z0-9][a-z0-9-]*$`). +- **`type` is immutable:** an incoming `type` differing from the stored row's is + rejected (the entry is skipped, the rest proceed; logged loudly). +- **Default-id consistency:** the re-stamped default id (`DEFAULT_PROJECT_ID`, + else `default`) MUST appear as an `id` in the registry, or the registry is + inconsistent with config → surfaced loudly. + +### 2d. Wiring (Option A) + +- **Webhook** (`app/webhooks.py`): add a branch — if the pushed repo + `full_name` matches `REGISTRY_REPO`, call `registry.refresh_registry(...)`. + Same HMAC-verified `/api/webhooks/gitea` dispatcher; no new endpoint. Add + `REGISTRY_REPO` to the set of repos the dispatcher recognizes. +- **Sweep** (`cache.Reconciler.sweep()`): add one `await + registry.refresh_registry(...)` at the top of each pass, so the safety-net + loop keeps `projects` in sync if a webhook is missed. +- **Startup** (`main.py` lifespan): run `refresh_registry` once after + migrations. This **replaces** M1's `seed_default_project` (which is removed). + +### 2e. Failure posture + +- **Startup / first boot:** if `REGISTRY_REPO` is unset/unreachable, or + `projects.yaml` is missing or fails validation, the app **fails loudly** + (refuses to start) — there is no last-known-good to serve. +- **Running deployment:** a malformed `projects.yaml` pushed in a later PR is + logged and **skipped**, leaving the last-good `projects` rows intact — a bad + config PR must not take the deployment down. This mirrors how the corpus + reconciler tolerates a bad content push today. + +--- + +## 3. Config (`app/config.py`) + +- `registry_repo`: **required** — construction fails loudly if unset (matching + the other required vars). Add `registry_repo_full` → `{gitea_org}/{registry_repo}`. +- `meta_repo` and `meta_repo_full`: **removed**. +- `default_project_id`: optional; consumed by migration `027` (re-stamp) and by + `refresh_registry` (the §2c consistency gate). +- `enabled_models`: unchanged — the deployment-level fallback for a project's + optional `enabled_models`. +- `app/projects.py`: `seed_default_project` retired (superseded by the mirror). + `DEFAULT_PROJECT_ID` constant and resolution helpers retained as needed. + +--- + +## 4. APIs — `app/api_deployment.py` + +A new sub-router mounted in `app/api.py`. + +### `GET /api/deployment` + +Returns `{ name, tagline, projects: [...] }`. The deployment `name`/`tagline` +come from the `deployment` singleton. The project list is filtered by caller +visibility (§22.5): + +- `public` projects → visible to everyone (incl. anonymous). +- `gated` projects → only when the caller is a member (`visible_project_ids` + from M2's resolver). +- `unlisted` projects → **omitted entirely** (reachable only by direct id). + +Each item: `{ id, name, type, visibility }` — enough for the M3-frontend +directory + switcher. (Theme is fetched per project.) + +### `GET /api/projects/:id` + +Returns `{ id, name, tagline, type, visibility, initial_state, theme }`. +Guarded by `require_project_readable(user, id)` — 404 for a non-member of a +gated project, reusing the M2 resolver. `unlisted` is readable here by direct +id (it is hidden only from enumeration). + +--- + +## 5. Entry-state & review semantics + +### 5a. Frontmatter (`app/entry.py`) + +Parse three new fields, lenient (default `unreviewed=false`, nulls): +`unreviewed: bool`, `reviewed_at`, `reviewed_by`. Add to the `Entry` +dataclass. These are git-truth (§2 frontmatter) so they survive a cache +rebuild, exactly like `state`. + +### 5b. Cache mirror (`app/cache.py`) + +`_upsert_cached_rfc` writes the three fields into `cached_rfcs` +(`unreviewed`, `reviewed_at`, `reviewed_by`). + +### 5c. Entry-landing path + +When a creating idea-PR merges (§2.4), resolve the project's `initial_state`: + +- `super-draft` → today's behavior unchanged (propose → super-draft → graduate). +- `active` → land the entry `active` with `unreviewed = true`, skipping the §13 + graduate gate. The exact merge-handling site (in the PR-merge reconcile path) + is located during implementation. + +### 5d. Mark-reviewed + +`POST /api/projects/:pid/rfcs/:slug/mark-reviewed`. Authority: +`is_project_superuser` (project_admin or deployment owner/admin) — the same tier +that graduates an entry. Effect: the bot writes `unreviewed: false` + +`reviewed_at`/`reviewed_by` into the entry frontmatter (a git commit, paralleling +graduate), which the mirror then reflects into `cached_rfcs`. Stamps provenance +paralleling `graduated_at`/`graduated_by`. + +### 5e. Catalog filter (query side) + +`GET /api/rfcs` (and the project-scoped form) gains an `unreviewed=true` query +param that filters on the cached column — the owner's worklist. The UI for it +is M3-frontend; only the query side ships here. `unreviewed` applies to +`active` entries only. + +--- + +## 6. Testing + +- **Migration `027`:** seed a `026`-shaped DB with rows under + `project_id='default'`; run `027`; assert: new columns present; the 12 tables + rebuilt with composite keys + FK; all 19 `project_id`-bearing tables + re-stamped to ``; row counts preserved; FK integrity on; idempotent + re-run is a no-op. +- **`registry.py`:** valid `projects.yaml` upserts all fields + `registry_sha`; + each validation failure rejected (bad enum, missing field, dup id, `type` + mutation, missing default id); `initial_state` type-default applied; + startup-strict vs running-tolerant posture. +- **APIs:** `/api/deployment` visibility filtering across + gated/public/unlisted × member/non-member/anonymous; `/api/projects/:id` 404 + gate for gated non-member, 200 for unlisted-by-id. +- **Review flow:** `initial_state: active` lands `unreviewed=true`; + mark-reviewed authority (allow superuser, deny others) + frontmatter write + + mirror reflection; catalog `unreviewed` filter returns the worklist. +- **Two-project isolation:** extend `test_multi_project_authz_vertical.py` with + a genuine **second** registry project sharing a slug with the first, proving + the PK rebuilds isolate them (the core point of M3's activation). + +--- + +## 7. File-touch summary + +**New** +- `backend/migrations/027_projects_activate.sql` +- `backend/app/registry.py` +- `backend/app/api_deployment.py` +- tests: `backend/tests/test_migration_027.py`, `test_registry.py`, + `test_api_deployment.py`, `test_review_flow.py`; extend + `test_multi_project_authz_vertical.py` + +**Modified** +- `backend/app/config.py` (registry_repo required; meta_repo removed; + default_project_id) +- `backend/app/webhooks.py` (registry-repo branch) +- `backend/app/cache.py` (`Reconciler.sweep` registry call; + `_upsert_cached_rfc` review fields) +- `backend/app/entry.py` (frontmatter fields) +- `backend/app/api.py` (mount `api_deployment`; `unreviewed` filter on + `/api/rfcs`) +- `backend/app/main.py` (startup `refresh_registry`; drop `seed_default_project`) +- `backend/app/projects.py` (retire `seed_default_project`) +- `backend/.env.example` (`REGISTRY_REPO`, `DEFAULT_PROJECT_ID`; remove + `META_REPO`) + +--- + +## 8. Versioning & upgrade + +Minor bump, breaking (pre-1.0). `VERSION` + `frontend/package.json#version` move +together (§20). `CHANGELOG.md` gets a breaking entry with an **upgrade-steps** +block: + +1. Create a registry repo under the deployment's Gitea org. +2. Author `projects.yaml`: a `deployment:` block (`name`, `tagline`) and one + `projects:` entry for the existing corpus — `id: `, `name`, `type: + document`, `content_repo: `, `visibility: public`. +3. Set env: `REGISTRY_REPO=`, `DEFAULT_PROJECT_ID=` + (must equal the entry's `id`); remove `META_REPO`. +4. Deploy. Migration `027` runs the rebuilds + re-stamp; `refresh_registry` + reconciles the registry into `projects`. Verify `/api/deployment` returns the + project and `/api/health` is green. + +The SPEC.md §22 merge stays in M7 per the slicing plan; this slice references the +draft at `docs/design/multi-project-spec.md`. + +## Known limitations / deferred + +- **Project archival/deletion** from the registry is not handled (a removed + entry stops refreshing but its rows persist). Defer to a future archival spec. +- All routing, redirects, runtime branding, theme application, directory, and + switcher are **M3-frontend**.