rfc-app/docs/superpowers/specs/2026-06-03-m3-backend-design.md

M3-backend — §22 multi-project: registry mirror + data spine + APIs

Design spec for the backend half of §22 slice M3 ("Registry mirror + routing + runtime branding"). The roadmap bundles M3 as one slice; this session splits it at the natural backend/frontend seam. M3-backend (this doc) ships the data spine, the registry mirror, the two runtime-config APIs, and the entry-state/review semantics. M3-frontend (a separate spec) ships /p/<project>/ routing, the 308 redirects, the VITE_APP_NAME→runtime-config cut, the per-project theme overlay, the deployment directory at /, and the project switcher — all consuming the APIs defined here.

Section references §22.x point at docs/design/multi-project-spec.md (the draft §22 + slicing plan). The SPEC.md §22 merge itself lands in M7.

Status

• Date: 2026-06-03
• Slice: §22 M3 (backend half). M1 + M2 landed and merged to main.
• Version impact: minor bump, breaking (pre-1.0) — see §8.

Goal

After M3-backend, the framework learns its projects from a git registry (not from META_REPO), the projects cache table and all slug-bearing tables are keyed by (project_id, …) so a second project can exist without collision, the default project's identity is re-stamped to its real slug while no /p/ URL is yet public, and the runtime exposes deployment + project config over two new endpoints. The entry-state/review semantics (initial_state, unreviewed) ship complete even though OHM (a document/super-draft project) does not yet exercise them.

Non-goals (M3-frontend, later): /p/<project>/ routing, 308 redirects off /rfc/<slug>, runtime branding in the UI, theme application, the deployment directory, the project switcher, the catalog's unreviewed-filter UI.

Decisions taken in brainstorming

1. Scope split — backend spine first; frontend surfaces are a separate spec/session.
2. Review machinery — build the full initial_state / unreviewed plumbing now (parse + columns + mirror + landing logic + mark-reviewed + catalog filter query side), per the spec's M3 bundle, even though no live project exercises it yet.
3. Config cut — hard cut. REGISTRY_REPO required (loud fail if unset), META_REPO removed. Upgrade-steps document the manual registry creation.
4. Re-stamp — rewrite project_id everywhere: rename projects.id default → the config slug and rewrite every child row, folded into the same create-copy-drop-rename rebuild that adds the project_id FK. One identifier; DB and URL agree.
5. Mirror structure — a self-contained app/registry.py module (not folded into cache.py), driven by the existing webhook dispatcher + the existing Reconciler.sweep().

---

1. Migration 027_projects_activate.sql

Runs while default is still the sole project and no /p/ URL exists — the safe window for an identity rewrite. One transaction (SQLite DDL is transactional): the deployment either fully advances to 027 or stays on 026.

DEFAULT_PROJECT_ID is read at migration time, so it must be set before the upgrade deploy (documented in §8). The slug it names is referred to below as <slug>; absent the env var, <slug> stays default.

1a. Additive columns

• projects:
  • type TEXT NOT NULL DEFAULT 'document' CHECK (type IN ('document','specification','bdd'))
  • initial_state TEXT NOT NULL DEFAULT 'super-draft' CHECK (initial_state IN ('super-draft','active'))
• cached_rfcs:
  • unreviewed INTEGER NOT NULL DEFAULT 0
  • reviewed_at TEXT
  • reviewed_by TEXT

1b. New deployment singleton table

Holds deployment-level identity mirrored from the registry's deployment: block (rather than overloading projects):

CREATE TABLE IF NOT EXISTS deployment (
    id          INTEGER PRIMARY KEY CHECK (id = 1),
    name        TEXT,
    tagline     TEXT,
    registry_sha TEXT,
    updated_at  TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT OR IGNORE INTO deployment (id) VALUES (1);

1c. PK / uniqueness rebuilds (the 12 deferred tables)

Per migration 026's deferred block, create-copy-drop-rename each table to fold project_id into the key and add project_id … REFERENCES projects(id) ON DELETE CASCADE:

Table	Key change
cached_rfcs	PK (slug) → (project_id, slug)
cached_branches	UNIQUE (rfc_slug, branch_name) → +project_id
branch_visibility	UNIQUE (rfc_slug, branch_name) → +project_id
branch_contribute_grants	UNIQUE (rfc_slug, branch_name, grantee_user_id) → +project_id
stars	UNIQUE (user_id, rfc_slug) → +project_id
watches	UNIQUE (user_id, rfc_slug) → +project_id
pr_seen	UNIQUE (user_id, rfc_slug, pr_number) → +project_id
branch_chat_seen	UNIQUE (user_id, rfc_slug, branch_name) → +project_id
funder_consents	PK (user_id, rfc_slug) → +project_id
rfc_collaborators	UNIQUE INDEX (rfc_slug, user_id) → +project_id
contribution_requests	UNIQUE INDEX (rfc_slug, requester_user_id) WHERE pending → +project_id
proposed_use_cases	UNIQUE (scope, pr_number) → +project_id

cached_prs is already globally unique (repo is the full org/repo string, distinct per project) — no rebuild.

The copy step writes <slug> in place of default for project_id, so these 12 tables are re-stamped for free (§1d).

1d. Re-stamp default → <slug>

• UPDATE projects SET id = '<slug>' WHERE id = 'default' (only when <slug> ≠ default).
• The 12 rebuilt tables: re-stamped during their copy (§1c).
• The remaining 7 tables that carry project_id but need no key change get a plain rewrite: UPDATE <t> SET project_id = '<slug>' WHERE project_id = 'default' for threads, changes, notifications, actions, pr_resolution_branches, rfc_invitations, cached_prs.

End state: a single project_id value DB-wide.

1e. Notes

• Migration 016 is absent from the on-disk sequence (015 → 017); the runner already tolerates the gap (the app runs today). Do not renumber. New file is 027.
• PRAGMA foreign_keys is honored going forward; the FK lands on the 12 rebuilt tables. The other 7 keep app-layer integrity (matching today's posture for non-rebuilt tables).

---

2. Registry format + app/registry.py

2a. projects.yaml (root of REGISTRY_REPO)

deployment:
  name: Open Human Model         # replaces VITE_APP_NAME (M3-frontend consumes)
  tagline: ...
projects:
  - id: ohm                      # url-stable slug, unique in the deployment
    name: Open Human Model
    type: document               # document | specification | bdd — immutable
    content_repo: ohm            # repo under the deployment's Gitea org
    visibility: public           # gated | public | unlisted
    initial_state: super-draft   # optional; defaults from type
    enabled_models: [claude, gemini]   # optional; falls back to ENABLED_MODELS
    theme: { accent: "#5b5bd6" }       # optional; M3-frontend consumes

2b. refresh_registry(config, gitea) -> RegistryResult

The config-side analogue of cache.refresh_meta_repo. Fetches projects.yaml from REGISTRY_REPO at HEAD, parses, validates, and upserts:

• Each project → projects row: id, name, type, content_repo, visibility, initial_state, plus config_json (JSON blob for theme, enabled_models), plus registry_sha (commit SHA, provenance).
• The deployment: block → the deployment singleton (name, tagline, registry_sha).

Projects present in the table but absent from the registry are not deleted in M3-backend (archival semantics are out of scope; a removed entry simply stops being refreshed). This is noted as a known limitation; revisit if/when project archival is specced.

2c. Validation (loud)

Per the framework's separation-of-concerns rule, malformed config fails visibly rather than shipping wrong content silently:

• Each project requires id, name, type, content_repo.
• type ∈ {document,specification,bdd}; visibility ∈ {gated,public,unlisted}.
• initial_state defaults from type when omitted: document/specification → super-draft, bdd → active. When present it must be a valid §2.4 super-draft entry-state value.
• id values unique and slug-shaped (^[a-z0-9][a-z0-9-]*$).
• type is immutable: an incoming type differing from the stored row's is rejected (the entry is skipped, the rest proceed; logged loudly).
• Default-id consistency: the re-stamped default id (DEFAULT_PROJECT_ID, else default) MUST appear as an id in the registry, or the registry is inconsistent with config → surfaced loudly.

2d. Wiring (Option A)

• Webhook (app/webhooks.py): add a branch — if the pushed repo full_name matches REGISTRY_REPO, call registry.refresh_registry(...). Same HMAC-verified /api/webhooks/gitea dispatcher; no new endpoint. Add REGISTRY_REPO to the set of repos the dispatcher recognizes.
• Sweep (cache.Reconciler.sweep()): add one await registry.refresh_registry(...) at the top of each pass, so the safety-net loop keeps projects in sync if a webhook is missed.
• Startup (main.py lifespan): run refresh_registry once after migrations. This replaces M1's seed_default_project (which is removed).

2e. Failure posture

• Startup / first boot: if REGISTRY_REPO is unset/unreachable, or projects.yaml is missing or fails validation, the app fails loudly (refuses to start) — there is no last-known-good to serve.
• Running deployment: a malformed projects.yaml pushed in a later PR is logged and skipped, leaving the last-good projects rows intact — a bad config PR must not take the deployment down. This mirrors how the corpus reconciler tolerates a bad content push today.

---

3. Config (app/config.py)

• registry_repo: required — construction fails loudly if unset (matching the other required vars). Add registry_repo_full → {gitea_org}/{registry_repo}.
• meta_repo and meta_repo_full: removed.
• default_project_id: optional; consumed by migration 027 (re-stamp) and by refresh_registry (the §2c consistency gate).
• enabled_models: unchanged — the deployment-level fallback for a project's optional enabled_models.
• app/projects.py: seed_default_project retired (superseded by the mirror). DEFAULT_PROJECT_ID constant and resolution helpers retained as needed.

---

4. APIs — app/api_deployment.py

A new sub-router mounted in app/api.py.

GET /api/deployment

Returns { name, tagline, projects: [...] }. The deployment name/tagline come from the deployment singleton. The project list is filtered by caller visibility (§22.5):

• public projects → visible to everyone (incl. anonymous).
• gated projects → only when the caller is a member (visible_project_ids from M2's resolver).
• unlisted projects → omitted entirely (reachable only by direct id).

Each item: { id, name, type, visibility } — enough for the M3-frontend directory + switcher. (Theme is fetched per project.)

GET /api/projects/:id

Returns { id, name, tagline, type, visibility, initial_state, theme }. Guarded by require_project_readable(user, id) — 404 for a non-member of a gated project, reusing the M2 resolver. unlisted is readable here by direct id (it is hidden only from enumeration).

---

5. Entry-state & review semantics

5a. Frontmatter (app/entry.py)

Parse three new fields, lenient (default unreviewed=false, nulls): unreviewed: bool, reviewed_at, reviewed_by. Add to the Entry dataclass. These are git-truth (§2 frontmatter) so they survive a cache rebuild, exactly like state.

5b. Cache mirror (app/cache.py)

_upsert_cached_rfc writes the three fields into cached_rfcs (unreviewed, reviewed_at, reviewed_by).

5c. Entry-landing path

When a creating idea-PR merges (§2.4), resolve the project's initial_state:

• super-draft → today's behavior unchanged (propose → super-draft → graduate).
• active → land the entry active with unreviewed = true, skipping the §13 graduate gate. The exact merge-handling site (in the PR-merge reconcile path) is located during implementation.

5d. Mark-reviewed

POST /api/projects/:pid/rfcs/:slug/mark-reviewed. Authority: is_project_superuser (project_admin or deployment owner/admin) — the same tier that graduates an entry. Effect: the bot writes unreviewed: false + reviewed_at/reviewed_by into the entry frontmatter (a git commit, paralleling graduate), which the mirror then reflects into cached_rfcs. Stamps provenance paralleling graduated_at/graduated_by.

5e. Catalog filter (query side)

GET /api/rfcs (and the project-scoped form) gains an unreviewed=true query param that filters on the cached column — the owner's worklist. The UI for it is M3-frontend; only the query side ships here. unreviewed applies to active entries only.

---

6. Testing

• Migration 027: seed a 026-shaped DB with rows under project_id='default'; run 027; assert: new columns present; the 12 tables rebuilt with composite keys + FK; all 19 project_id-bearing tables re-stamped to <slug>; row counts preserved; FK integrity on; idempotent re-run is a no-op.
• registry.py: valid projects.yaml upserts all fields + registry_sha; each validation failure rejected (bad enum, missing field, dup id, type mutation, missing default id); initial_state type-default applied; startup-strict vs running-tolerant posture.
• APIs: /api/deployment visibility filtering across gated/public/unlisted × member/non-member/anonymous; /api/projects/:id 404 gate for gated non-member, 200 for unlisted-by-id.
• Review flow: initial_state: active lands unreviewed=true; mark-reviewed authority (allow superuser, deny others) + frontmatter write + mirror reflection; catalog unreviewed filter returns the worklist.
• Two-project isolation: extend test_multi_project_authz_vertical.py with a genuine second registry project sharing a slug with the first, proving the PK rebuilds isolate them (the core point of M3's activation).

---

7. File-touch summary

New

• backend/migrations/027_projects_activate.sql
• backend/app/registry.py
• backend/app/api_deployment.py
• tests: backend/tests/test_migration_027.py, test_registry.py, test_api_deployment.py, test_review_flow.py; extend test_multi_project_authz_vertical.py

Modified

• backend/app/config.py (registry_repo required; meta_repo removed; default_project_id)
• backend/app/webhooks.py (registry-repo branch)
• backend/app/cache.py (Reconciler.sweep registry call; _upsert_cached_rfc review fields)
• backend/app/entry.py (frontmatter fields)
• backend/app/api.py (mount api_deployment; unreviewed filter on /api/rfcs)
• backend/app/main.py (startup refresh_registry; drop seed_default_project)
• backend/app/projects.py (retire seed_default_project)
• backend/.env.example (REGISTRY_REPO, DEFAULT_PROJECT_ID; remove META_REPO)

---

8. Versioning & upgrade

Minor bump, breaking (pre-1.0). VERSION + frontend/package.json#version move together (§20). CHANGELOG.md gets a breaking entry with an upgrade-steps block:

1. Create a registry repo under the deployment's Gitea org.
2. Author projects.yaml: a deployment: block (name, tagline) and one projects: entry for the existing corpus — id: <slug>, name, type: document, content_repo: <old META_REPO value>, visibility: public.
3. Set env: REGISTRY_REPO=<registry repo name>, DEFAULT_PROJECT_ID=<slug> (must equal the entry's id); remove META_REPO.
4. Deploy. Migration 027 runs the rebuilds + re-stamp; refresh_registry reconciles the registry into projects. Verify /api/deployment returns the project and /api/health is green.

The SPEC.md §22 merge stays in M7 per the slicing plan; this slice references the draft at docs/design/multi-project-spec.md.

Known limitations / deferred

• Project archival/deletion from the registry is not handled (a removed entry stops refreshing but its rows persist). Defer to a future archival spec.
• All routing, redirects, runtime branding, theme application, directory, and switcher are M3-frontend.