diff --git a/CHANGELOG.md b/CHANGELOG.md index 4cbdad7..596917d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,74 @@ skip versions are the composition of each intervening adjacent release's steps in order — no A-to-B path is pre-computed beyond that. +## 0.33.0 — 2026-06-04 + +**Minor (breaking) — §22 M3 backend Plan A: project registry mirror + +runtime config. The framework now learns its projects from +`REGISTRY_REPO/projects.yaml`; `META_REPO` is retired. Migration `027` +runs automatically on deploy.** + +Added: + +- **Project registry mirror** (`app/registry.py`): the framework reads + `projects.yaml` from `REGISTRY_REPO` and mirrors its `deployment:` + block and `projects:` entries into the `projects` table + a new + `deployment` singleton. The mirror runs at startup (reconciler) and + on every §4 webhook push to the registry repo. +- **`GET /api/deployment`** — returns `name`, `tagline`, and the list + of visible projects (each item carries `id`, `name`, `type`, + `visibility`). Replaces the build-time `VITE_APP_NAME` as the + authoritative runtime config source (the frontend cut lands in + M3-frontend). +- **`GET /api/projects/:id`** — returns `id`, `name`, `tagline`, `type`, + `visibility`, `initial_state`, and `theme` for a single project. +- **§22.4b `initial_state`** honored at propose time: new RFCs enter + the state named by the project's `initial_state` field (default: + `super-draft`; `bdd` projects default to `active`). +- **§22.4c `unreviewed` flag**: newly proposed RFCs are flagged + `unreviewed = true`. Owners clear it via + `POST /api/projects/:id/rfcs/:slug/mark-reviewed`. The catalog + accepts `?unreviewed=true` to filter to the review queue. +- **Migration `027`** (additive): adds `projects.type` / + `projects.initial_state`, the `deployment` table, and + `cached_rfcs.unreviewed` / `reviewed_at` / `reviewed_by`. + +Breaking: + +- `META_REPO` is retired. The app **refuses to start** if `REGISTRY_REPO` + is unset. The corpus mirror now reads each project's `content_repo` + from `projects.yaml` rather than from the `META_REPO` env var. + +Upgrade steps: + +1. **MUST** create a registry repo under your deployment's Gitea org. +2. **MUST** author `projects.yaml` at its root with a `deployment:` block + (`name`, `tagline`) and one `projects:` entry for your existing corpus: + ```yaml + deployment: + name: + tagline: + projects: + - id: default + name: + type: document + content_repo: + visibility: public + ``` + Keep `id: default` for this release; the pretty-slug re-stamp lands in + the next backend slice, before any `/p/` URL is public. +3. **MUST** set `REGISTRY_REPO=` and **MUST** + remove `META_REPO` (or leave it unset — it is ignored but its presence + may cause confusion). +4. **MUST** add a Gitea webhook on the registry repo pointing at + `/api/webhooks/gitea` (same secret as the corpus webhook). +5. **MUST** deploy. Migration `027` runs automatically at startup; the + registry mirror reconciles immediately after. Verify + `GET /api/deployment` returns your project and `/api/health` is green. +6. **SHOULD** rebuild the frontend (no new env var is required until + M3-frontend lands, but the frontend currently still reads + `VITE_APP_NAME` for the display name). + ## 0.32.0 — 2026-06-01 **Minor — graduation's integer RFC number is now optional, and RFC diff --git a/CLAUDE.md b/CLAUDE.md index 21c6589..e54989a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,3 +1,5 @@ +@~/.claude/wiggleverse.md + # Working in rfc-app This is the framework — the software that hosts an RFC standardization diff --git a/VERSION b/VERSION index 9eb2aa3..be386c9 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.32.0 +0.33.0 diff --git a/backend/.env.example b/backend/.env.example index 245371b..9fa08d9 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -9,19 +9,18 @@ GITEA_URL=http://localhost:3000 GITEA_BOT_USER=rfc-bot GITEA_BOT_TOKEN= -# The Gitea org or user that owns the meta repo and every RFC repo -# the bot will create on graduation. +# The Gitea org or user that owns every RFC repo the bot will create on +# graduation. GITEA_ORG=wiggleverse -META_REPO=meta -# --- Multi-project registry (§22) --- -# The registry repo (under GITEA_ORG) the framework reads to learn which -# projects this deployment hosts. Optional through Slice M1: the registry -# mirror lands in M3, and until then the single default project's content -# repo is taken from META_REPO above. The repo's name is the deployment's -# choice. When the mirror lands, REGISTRY_REPO becomes required and -# supersedes META_REPO. -# REGISTRY_REPO=wiggleverse-registry +# §22.2 — the project registry repo (REQUIRED). The framework reads +# `projects.yaml` at its root to learn which projects exist. The repo name is +# the deployment's choice; the app fails to start if this is unset. +REGISTRY_REPO=registry +# §22.13 — optional id for the bootstrap/default project. Reserved for the +# Plan B re-stamp; leave unset in Plan A (the default project id stays +# `default`). When set, it must match an `id` in projects.yaml. +# DEFAULT_PROJECT_ID=ohm # --- OAuth (Gitea) --- # In Gitea: Site Administration → Applications → Add OAuth2 Application. diff --git a/backend/app/api.py b/backend/app/api.py index a8da2f5..880d891 100644 --- a/backend/app/api.py +++ b/backend/app/api.py @@ -22,12 +22,14 @@ from . import ( api_admin, api_branches, api_contributions, + api_deployment, api_discussion, api_graduation, api_invitations, api_notifications, api_prs, auth, + projects as projects_mod, db, device_trust as device_trust_mod, docs as docs_mod, @@ -146,6 +148,8 @@ def make_router( # (super-draft) RFC. Reuses the #12 invite flow (api_invitations above) # on accept; lands the request + owner notifications via §15 notify. router.include_router(api_contributions.make_router()) + # §22.9 (M3): runtime deployment + per-project config (replaces VITE_APP_NAME). + router.include_router(api_deployment.make_router()) # --------------------------------------------------------------- # §17: /api/health — unauthenticated post-flight probe. @@ -607,13 +611,16 @@ def make_router( # --------------------------------------------------------------- @router.get("/api/rfcs") - async def list_rfcs(request: Request) -> dict[str, Any]: + async def list_rfcs(request: Request, unreviewed: str | None = None) -> dict[str, Any]: """§7's left pane data. The chip-filter / sort / search combinatorics live on the client — the server returns the full set and lets the chips narrow it. The set is small (hundreds, not thousands) for the foreseeable future, so paginating here would buy nothing. + + §22.4c: pass `?unreviewed=true` to narrow to active entries + still awaiting owner review. """ viewer = auth.current_user(request) viewer_id = viewer.user_id if viewer else None @@ -623,6 +630,10 @@ def make_router( if not visible: return {"items": []} placeholders = ",".join("?" for _ in visible) + params = list(visible) + unreviewed_clause = "" + if unreviewed is not None and unreviewed.lower() in ("1", "true", "yes"): + unreviewed_clause = " AND unreviewed = 1 AND state = 'active'" rows = db.conn().execute( f""" SELECT slug, title, state, rfc_id, repo, @@ -630,10 +641,10 @@ def make_router( last_main_commit_at, last_entry_commit_at, updated_at FROM cached_rfcs WHERE state IN ('super-draft', 'active') - AND project_id IN ({placeholders}) + AND project_id IN ({placeholders}){unreviewed_clause} ORDER BY COALESCE(last_main_commit_at, last_entry_commit_at) DESC """, - visible, + params, ).fetchall() starred = set() @@ -696,6 +707,40 @@ def make_router( payload["proposed_use_case"] = uc["use_case"] if uc else None return payload + # --------------------------------------------------------------- + # §22.4c: mark-reviewed — clear an active entry's `unreviewed` flag + # --------------------------------------------------------------- + + @router.post("/api/projects/{project_id}/rfcs/{slug}/mark-reviewed") + async def mark_reviewed(project_id: str, slug: str, request: Request) -> dict[str, Any]: + """§22.4c — clear an active entry's `unreviewed` flag. Authority is the + §22.7 project superuser (project_admin or deployment owner/admin).""" + viewer = auth.require_user(request) + auth.require_project_readable(viewer, project_id) + if not auth.is_project_superuser(viewer, project_id): + raise HTTPException(403, "Only a project owner/admin can mark an entry reviewed") + row = db.conn().execute( + "SELECT state, unreviewed FROM cached_rfcs WHERE slug = ? AND project_id = ?", + (slug, project_id), + ).fetchone() + if row is None: + raise HTTPException(404, "Not found") + if row["state"] != "active" or not row["unreviewed"]: + raise HTTPException(409, "Entry is not an unreviewed active entry") + try: + await bot.mark_entry_reviewed( + viewer.as_actor(), + org=config.gitea_org, + meta_repo=(projects_mod.default_content_repo(config) or ""), + slug=slug, + reviewed_by=viewer.gitea_login, + reviewed_at=entry_mod.today(), + ) + except GiteaError as e: + raise HTTPException(502, f"Gitea: {e.detail}") + await cache.refresh_meta_repo(config, gitea) + return {"ok": True} + # --------------------------------------------------------------- # §7.3 / §9.3: pending ideas # --------------------------------------------------------------- @@ -765,7 +810,7 @@ def make_router( # Read the proposed entry file from the head branch. slug = row["rfc_slug"] head = row["head_branch"] - result = await gitea.read_file(config.gitea_org, config.meta_repo, f"rfcs/{slug}.md", ref=head) + result = await gitea.read_file(config.gitea_org, (projects_mod.default_content_repo(config) or ""), f"rfcs/{slug}.md", ref=head) entry_payload: dict[str, Any] | None = None if result: text, _sha = result @@ -823,10 +868,16 @@ def make_router( if idea_clash: raise HTTPException(409, f"Slug `{slug}` is already reserved by an open proposal") + # §22.4b: the target project's landing state. Through Plan A every + # entry lands in the default project; M3-frontend routing carries a + # non-default target later. + target_project = projects_mod.resolved_default_id(config) + landing_state = "active" if projects_mod.project_initial_state(target_project) == "active" else "super-draft" + entry = entry_mod.Entry( slug=slug, title=payload.title.strip(), - state="super-draft", + state=landing_state, id=None, repo=None, proposed_by=user.email or user.gitea_login, @@ -841,6 +892,7 @@ def make_router( arbiters=[], tags=[t.strip() for t in payload.tags if t.strip()], body=payload.pitch.strip() + "\n", + unreviewed=(landing_state == "active"), ) contents = entry_mod.serialize(entry) pr_title = f"Propose: {entry.title}" @@ -855,7 +907,7 @@ def make_router( pr = await bot.open_idea_pr( user.as_actor(), org=config.gitea_org, - meta_repo=config.meta_repo, + meta_repo=(projects_mod.default_content_repo(config) or ""), slug=slug, file_contents=contents, pr_title=pr_title, @@ -931,7 +983,7 @@ def make_router( await bot.merge_idea_pr( user.as_actor(), org=config.gitea_org, - meta_repo=config.meta_repo, + meta_repo=(projects_mod.default_content_repo(config) or ""), pr_number=pr_number, slug=row["rfc_slug"], ) @@ -951,7 +1003,7 @@ def make_router( await bot.decline_idea_pr( user.as_actor(), org=config.gitea_org, - meta_repo=config.meta_repo, + meta_repo=(projects_mod.default_content_repo(config) or ""), pr_number=pr_number, slug=row["rfc_slug"], comment=body.comment, @@ -975,7 +1027,7 @@ def make_router( await bot.withdraw_idea_pr( user.as_actor(), org=config.gitea_org, - meta_repo=config.meta_repo, + meta_repo=(projects_mod.default_content_repo(config) or ""), pr_number=pr_number, slug=row["rfc_slug"], ) diff --git a/backend/app/api_branches.py b/backend/app/api_branches.py index ebd1e2a..38ae552 100644 --- a/backend/app/api_branches.py +++ b/backend/app/api_branches.py @@ -29,7 +29,7 @@ from fastapi import APIRouter, HTTPException, Request from fastapi.responses import StreamingResponse from pydantic import BaseModel, Field -from . import auth, cache, chat as chat_layer, db, entry as entry_mod, funder, models_resolver +from . import auth, cache, chat as chat_layer, db, entry as entry_mod, funder, models_resolver, projects as projects_mod from .bot import Bot from .config import Config from .gitea import Gitea, GiteaError @@ -1153,7 +1153,7 @@ def make_router( def _repo_for(rfc, branch: str = "main") -> tuple[str, str]: if _is_meta_target(rfc, branch): - return config.gitea_org, config.meta_repo + return config.gitea_org, (projects_mod.default_content_repo(config) or "") owner, repo = rfc["repo"].split("/", 1) return owner, repo diff --git a/backend/app/api_deployment.py b/backend/app/api_deployment.py new file mode 100644 index 0000000..d53dc4e --- /dev/null +++ b/backend/app/api_deployment.py @@ -0,0 +1,72 @@ +"""§22.9 runtime deployment/project config (replaces VITE_APP_NAME). + +GET /api/deployment — the deployment name/tagline + the projects the caller can +see (§22.5: gated filtered by membership, unlisted omitted from enumeration). +GET /api/projects/:id — one project's runtime config + optional theme overlay, +gated behind the §22.5 read gate (404 for a non-member of a gated project). +""" +from __future__ import annotations + +import json +from typing import Any + +from fastapi import APIRouter, HTTPException, Request + +from . import auth, db + + +def make_router() -> APIRouter: + router = APIRouter() + + @router.get("/api/deployment") + async def get_deployment(request: Request) -> dict[str, Any]: + viewer = auth.current_user(request) + dep = db.conn().execute( + "SELECT name, tagline FROM deployment WHERE id = 1" + ).fetchone() + # §22.5: enumerate only public + (member-)gated; unlisted is never listed. + visible = set(auth.visible_project_ids(viewer)) + rows = db.conn().execute( + "SELECT id, name, type, visibility FROM projects " + "WHERE visibility != 'unlisted' ORDER BY name" + ).fetchall() + projects = [ + {"id": r["id"], "name": r["name"], "type": r["type"], "visibility": r["visibility"]} + for r in rows + if r["id"] in visible + ] + return { + "name": (dep["name"] if dep else "") or "", + "tagline": (dep["tagline"] if dep else "") or "", + "projects": projects, + } + + @router.get("/api/projects/{project_id}") + async def get_project(project_id: str, request: Request) -> dict[str, Any]: + viewer = auth.current_user(request) + # §22.5 read gate: a gated project 404s to a non-member (shape matches + # an unknown id). unlisted is readable by direct id. + auth.require_project_readable(viewer, project_id) + row = db.conn().execute( + "SELECT id, name, type, visibility, initial_state, config_json " + "FROM projects WHERE id = ?", + (project_id,), + ).fetchone() + if row is None: + raise HTTPException(status_code=404, detail="Not found") + try: + cfg = json.loads(row["config_json"] or "{}") + except (ValueError, TypeError): + cfg = {} + dep = db.conn().execute("SELECT tagline FROM deployment WHERE id = 1").fetchone() + return { + "id": row["id"], + "name": row["name"], + "tagline": (dep["tagline"] if dep else "") or "", + "type": row["type"], + "visibility": row["visibility"], + "initial_state": row["initial_state"], + "theme": cfg.get("theme") or {}, + } + + return router diff --git a/backend/app/api_graduation.py b/backend/app/api_graduation.py index 161dd8e..6c61adb 100644 --- a/backend/app/api_graduation.py +++ b/backend/app/api_graduation.py @@ -42,7 +42,7 @@ from fastapi import APIRouter, HTTPException, Request from fastapi.responses import StreamingResponse from pydantic import BaseModel, Field -from . import auth, cache, db, entry as entry_mod +from . import auth, cache, db, entry as entry_mod, projects as projects_mod from .bot import Actor, Bot from .config import Config from .gitea import Gitea, GiteaError @@ -345,7 +345,7 @@ def make_router( # graduation PR's update_file call and the body to carry through # unchanged (meta-only keeps the body in the entry, §13.3). fetched = await gitea.read_file( - config.gitea_org, config.meta_repo, f"rfcs/{slug}.md", ref="main", + config.gitea_org, (projects_mod.default_content_repo(config) or ""), f"rfcs/{slug}.md", ref="main", ) if fetched is None: raise HTTPException(409, f"Meta entry rfcs/{slug}.md not found on main") @@ -479,7 +479,7 @@ def make_router( raise HTTPException(409, f"A claim PR is already open: #{already['pr_number']}") fetched = await gitea.read_file( - config.gitea_org, config.meta_repo, f"rfcs/{slug}.md", ref="main", + config.gitea_org, (projects_mod.default_content_repo(config) or ""), f"rfcs/{slug}.md", ref="main", ) if fetched is None: raise HTTPException(409, f"Meta entry rfcs/{slug}.md not found on main") @@ -495,7 +495,7 @@ def make_router( try: pr = await bot.open_claim_pr( viewer.as_actor(), - org=config.gitea_org, meta_repo=config.meta_repo, + org=config.gitea_org, meta_repo=(projects_mod.default_content_repo(config) or ""), slug=slug, new_file_contents=new_contents, prior_sha=meta_sha, ) @@ -601,7 +601,7 @@ def make_router( async def _read_meta_entry(slug: str) -> tuple[entry_mod.Entry, str]: fetched = await gitea.read_file( - config.gitea_org, config.meta_repo, f"rfcs/{slug}.md", ref="main", + config.gitea_org, (projects_mod.default_content_repo(config) or ""), f"rfcs/{slug}.md", ref="main", ) if fetched is None: raise HTTPException(409, f"Meta entry rfcs/{slug}.md not found on main") @@ -656,7 +656,7 @@ async def _orchestrate( try: pr = await bot.open_graduation_pr( actor, - org=config.gitea_org, meta_repo=config.meta_repo, + org=config.gitea_org, meta_repo=(projects_mod.default_content_repo(config) or ""), slug=state.slug, new_file_contents=graduated_contents, prior_sha=meta_file_sha, @@ -676,7 +676,7 @@ async def _orchestrate( try: await bot.merge_graduation_pr( actor, - org=config.gitea_org, meta_repo=config.meta_repo, + org=config.gitea_org, meta_repo=(projects_mod.default_content_repo(config) or ""), pr_number=state.new_pr_number, head_branch=state.graduation_branch or "", slug=state.slug, rfc_id=state.rfc_id, @@ -738,7 +738,7 @@ async def _cleanup_unmerged( try: await bot.close_graduation_pr( actor, - org=config.gitea_org, meta_repo=config.meta_repo, + org=config.gitea_org, meta_repo=(projects_mod.default_content_repo(config) or ""), pr_number=state.new_pr_number, head_branch=state.graduation_branch or "", slug=state.slug, reason="graduation merge failed", @@ -751,7 +751,7 @@ async def _cleanup_unmerged( await bot.delete_branch( actor, owner=config.gitea_org, - repo=config.meta_repo, + repo=(projects_mod.default_content_repo(config) or ""), branch=branch_name, slug=state.slug, action_kind="delete_post_merge_branch", @@ -861,7 +861,7 @@ async def _run_state_flip( try: pr = await bot.open_retire_flip_pr( actor, - org=config.gitea_org, meta_repo=config.meta_repo, + org=config.gitea_org, meta_repo=(projects_mod.default_content_repo(config) or ""), slug=slug, new_file_contents=new_contents, prior_sha=prior_sha, verb=verb, target_state=target_state, ) @@ -872,7 +872,7 @@ async def _run_state_flip( try: await bot.merge_retire_flip_pr( actor, - org=config.gitea_org, meta_repo=config.meta_repo, + org=config.gitea_org, meta_repo=(projects_mod.default_content_repo(config) or ""), pr_number=pr_number, head_branch=head_branch, slug=slug, verb=verb, ) @@ -881,12 +881,12 @@ async def _run_state_flip( # accumulate (mirrors graduation's `_cleanup_unmerged`). try: await bot.close_graduation_pr( - actor, org=config.gitea_org, meta_repo=config.meta_repo, + actor, org=config.gitea_org, meta_repo=(projects_mod.default_content_repo(config) or ""), pr_number=pr_number, head_branch=head_branch, slug=slug, reason=f"{verb} merge failed", ) await bot.delete_branch( - actor, owner=config.gitea_org, repo=config.meta_repo, + actor, owner=config.gitea_org, repo=(projects_mod.default_content_repo(config) or ""), branch=head_branch, slug=slug, action_kind="delete_post_merge_branch", reason=f"{verb} merge failed", diff --git a/backend/app/api_prs.py b/backend/app/api_prs.py index 753675e..455d3e5 100644 --- a/backend/app/api_prs.py +++ b/backend/app/api_prs.py @@ -23,7 +23,7 @@ from typing import Any from fastapi import APIRouter, HTTPException, Request from pydantic import BaseModel, Field -from . import auth, cache, chat as chat_layer, db, entry as entry_mod, funder, models_resolver, rfc_links +from . import auth, cache, chat as chat_layer, db, entry as entry_mod, funder, models_resolver, projects as projects_mod, rfc_links from .bot import Bot from .config import Config from .gitea import Gitea, GiteaError @@ -691,7 +691,7 @@ def make_router( def _owner_repo(rfc) -> tuple[str, str]: if _is_meta_resident(rfc): - return config.gitea_org, config.meta_repo + return config.gitea_org, (projects_mod.default_content_repo(config) or "") owner, repo = rfc["repo"].split("/", 1) return owner, repo diff --git a/backend/app/bot.py b/backend/app/bot.py index 4436a92..f6ea9c3 100644 --- a/backend/app/bot.py +++ b/backend/app/bot.py @@ -27,7 +27,7 @@ import json import logging from dataclasses import dataclass -from . import db, notify +from . import db, entry as entry_mod, notify from .gitea import Gitea, GiteaError log = logging.getLogger(__name__) @@ -1069,6 +1069,54 @@ class Bot: ) return pr + # ----- §22.4c: mark-reviewed (direct main write) ----- + + async def mark_entry_reviewed( + self, + actor: Actor, + *, + org: str, + meta_repo: str, + slug: str, + reviewed_by: str, + reviewed_at: str, + ) -> None: + """Clear §22.4c unreviewed on an active entry by rewriting its + frontmatter on main. Stamps the commit with the §6.5 On-behalf-of + trailer and writes an actions-log row, mirroring the graduation + stamp's bot-write shape.""" + path = f"rfcs/{slug}.md" + result = await self._gitea.read_file(org, meta_repo, path, ref="main") + if result is None: + raise GiteaError(404, f"{path} not found") + text, sha = result + e = entry_mod.parse(text) + e.unreviewed = False + e.reviewed_at = reviewed_at + e.reviewed_by = reviewed_by + commit_message = _stamp_single(f"Mark {slug} reviewed", actor) + result = await self._gitea.update_file( + org, meta_repo, path, + content=entry_mod.serialize(e), + sha=sha, + message=commit_message, + branch="main", + author_name=actor.display_name, + author_email=actor.email or f"{actor.gitea_login}@users.noreply", + ) + commit_sha = ( + result.get("commit", {}).get("sha") + or result.get("content", {}).get("sha") + or "" + ) + _log( + actor, + "mark_reviewed", + rfc_slug=slug, + bot_commit_sha=commit_sha, + details={"reviewed_by": reviewed_by, "reviewed_at": reviewed_at}, + ) + # ----- Per-RFC repo: seeding (test/dev fixtures, future graduation) ----- async def ensure_rfc_repo_seed( diff --git a/backend/app/cache.py b/backend/app/cache.py index 3771671..62c3154 100644 --- a/backend/app/cache.py +++ b/backend/app/cache.py @@ -27,7 +27,7 @@ import asyncio import json import logging -from . import db, entry as entry_mod +from . import db, entry as entry_mod, projects as projects_mod, registry as registry_mod from .config import Config from .gitea import Gitea, GiteaError @@ -40,7 +40,11 @@ async def refresh_meta_repo(config: Config, gitea: Gitea) -> None: Idempotent. Safe to call on every meta-repo webhook and on every reconciler sweep. """ - org, repo = config.gitea_org, config.meta_repo + org = config.gitea_org + repo = projects_mod.default_content_repo(config) + if not repo: + log.warning("refresh_meta_repo: default project has no content_repo yet; skipping") + return try: files = await gitea.list_dir(org, repo, "rfcs", ref="main") except GiteaError as e: @@ -87,8 +91,10 @@ def _upsert_cached_rfc(entry: entry_mod.Entry, body_sha: str) -> None: INSERT INTO cached_rfcs (slug, title, state, rfc_id, repo, proposed_by, proposed_at, graduated_at, graduated_by, owners_json, arbiters_json, tags_json, - models_json, funder_login, body, body_sha, last_entry_commit_at, updated_at) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, datetime('now'), datetime('now')) + models_json, funder_login, body, body_sha, + unreviewed, reviewed_at, reviewed_by, + last_entry_commit_at, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, datetime('now'), datetime('now')) ON CONFLICT(slug) DO UPDATE SET title = excluded.title, state = excluded.state, @@ -105,6 +111,9 @@ def _upsert_cached_rfc(entry: entry_mod.Entry, body_sha: str) -> None: funder_login = excluded.funder_login, body = excluded.body, body_sha = excluded.body_sha, + unreviewed = excluded.unreviewed, + reviewed_at = excluded.reviewed_at, + reviewed_by = excluded.reviewed_by, last_entry_commit_at = datetime('now'), updated_at = datetime('now') """, @@ -125,6 +134,9 @@ def _upsert_cached_rfc(entry: entry_mod.Entry, body_sha: str) -> None: funder_login, entry.body, body_sha, + 1 if entry.unreviewed else 0, + entry.reviewed_at, + entry.reviewed_by, ), ) @@ -320,7 +332,11 @@ async def refresh_meta_branches(config: Config, gitea: Gitea) -> None: structurally `edit//` per §9.5, with dashes in place of slashes per the §19.2 path-routing candidate. """ - org, repo = config.gitea_org, config.meta_repo + org = config.gitea_org + repo = projects_mod.default_content_repo(config) + if not repo: + log.warning("refresh_meta_branches: default project has no content_repo yet; skipping") + return try: branches = await gitea.list_branches(org, repo) except GiteaError as e: @@ -437,7 +453,11 @@ async def refresh_meta_pulls(config: Config, gitea: Gitea) -> None: `On-behalf-of:` trailer from the PR body, then to the raw Gitea login as last resort. """ - org, repo = config.gitea_org, config.meta_repo + org = config.gitea_org + repo = projects_mod.default_content_repo(config) + if not repo: + log.warning("refresh_meta_pulls: default project has no content_repo yet; skipping") + return repo_full = f"{org}/{repo}" try: open_pulls = await gitea.list_pulls(org, repo, state="open") @@ -663,6 +683,10 @@ class Reconciler: async def sweep(self) -> None: log.info("reconciler: starting sweep") try: + try: + await registry_mod.refresh_registry(self._config, self._gitea) + except Exception: + log.exception("reconciler: registry refresh failed; keeping last-good projects") await refresh_meta_repo(self._config, self._gitea) await refresh_meta_branches(self._config, self._gitea) await refresh_meta_pulls(self._config, self._gitea) diff --git a/backend/app/config.py b/backend/app/config.py index 48daccd..39085fb 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -32,7 +32,6 @@ class Config: gitea_bot_user: str gitea_bot_token: str gitea_org: str - meta_repo: str registry_repo: str oauth_client_id: str oauth_client_secret: str @@ -45,14 +44,15 @@ class Config: anthropic_api_key: str = "" google_api_key: str = "" openai_api_key: str = "" + default_project_id: str = "" @property def redirect_uri(self) -> str: return f"{self.app_url}/auth/callback" @property - def meta_repo_full(self) -> str: - return f"{self.gitea_org}/{self.meta_repo}" + def registry_repo_full(self) -> str: + return f"{self.gitea_org}/{self.registry_repo}" def load_config() -> Config: @@ -80,14 +80,7 @@ def load_config() -> Config: gitea_bot_user=_required("GITEA_BOT_USER"), gitea_bot_token=_required("GITEA_BOT_TOKEN"), gitea_org=_required("GITEA_ORG"), - meta_repo=_optional("META_REPO", "meta"), - # §22.2: the multi-project registry repo the framework reads to learn - # which projects exist. Optional through Slice M1 — the registry - # *mirror* lands in M3; until then the default project (seeded by - # migration 026 + the §22.13 startup backfill) is the only project, - # and its content_repo comes from META_REPO. Becomes required when - # the mirror lands and META_REPO is retired. - registry_repo=_optional("REGISTRY_REPO"), + registry_repo=_required("REGISTRY_REPO"), oauth_client_id=_required("OAUTH_CLIENT_ID"), oauth_client_secret=_required("OAUTH_CLIENT_SECRET"), app_url=_optional("APP_URL", "http://localhost:8000").rstrip("/"), @@ -99,4 +92,5 @@ def load_config() -> Config: anthropic_api_key=_optional("ANTHROPIC_API_KEY"), google_api_key=_optional("GOOGLE_API_KEY"), openai_api_key=_optional("OPENAI_API_KEY"), + default_project_id=_optional("DEFAULT_PROJECT_ID"), ) diff --git a/backend/app/entry.py b/backend/app/entry.py index 7d5d2a6..1355bf3 100644 --- a/backend/app/entry.py +++ b/backend/app/entry.py @@ -50,6 +50,13 @@ class Entry: # operator credentials per §18 are used. The binding is inert until # the named user has a funder_consents row (the hybrid two-key rule). funder: str | None = None + # §22.4c: an `active` entry that landed without a human review gate + # carries unreviewed=True until an owner clears it. Orthogonal to + # `state`; only meaningful for active entries. reviewed_at/reviewed_by + # are the provenance of the clear, paralleling graduated_at/by. + unreviewed: bool = False + reviewed_at: str | None = None + reviewed_by: str | None = None body: str = "" @@ -66,6 +73,7 @@ def parse(text: str) -> Entry: models = [str(m) for m in raw_models] raw_funder = fm.get("funder") funder = str(raw_funder).strip() if raw_funder else None + unreviewed = bool(fm.get("unreviewed") or False) return Entry( slug=str(fm.get("slug") or ""), title=str(fm.get("title") or ""), @@ -81,6 +89,9 @@ def parse(text: str) -> Entry: tags=list(fm.get("tags") or []), models=models, funder=funder, + unreviewed=unreviewed, + reviewed_at=fm.get("reviewed_at") or None, + reviewed_by=fm.get("reviewed_by") or None, body=body, ) @@ -110,6 +121,14 @@ def serialize(entry: Entry) -> str: # second meaning here as with `models:`; one set of semantics. if entry.funder: fm["funder"] = entry.funder + # §22.4c: emit unreviewed only when True (a super-draft / reviewed + # active entry leaves the key absent → frontmatter stays minimal). + if entry.unreviewed: + fm["unreviewed"] = True + if entry.reviewed_at: + fm["reviewed_at"] = entry.reviewed_at + if entry.reviewed_by: + fm["reviewed_by"] = entry.reviewed_by yaml_text = yaml.safe_dump(fm, sort_keys=False, default_flow_style=False).rstrip() body = entry.body.lstrip("\n") if body: diff --git a/backend/app/hygiene.py b/backend/app/hygiene.py index a742f81..7431966 100644 --- a/backend/app/hygiene.py +++ b/backend/app/hygiene.py @@ -30,7 +30,7 @@ import logging import os from datetime import datetime, timedelta, timezone -from . import db +from . import db, projects as projects_mod from .bot import Bot from .config import Config @@ -299,7 +299,14 @@ async def _delete_branch_via_bot( log.warning("hygiene: cannot delete %s/%s — slug missing from cache", slug, branch) return False if not rfc["repo"]: - owner, repo = config.gitea_org, config.meta_repo + repo = projects_mod.default_content_repo(config) + if not repo: + log.warning( + "hygiene: default project has no content_repo; skipping branch delete for %s/%s", + slug, branch, + ) + return False + owner = config.gitea_org elif "/" in rfc["repo"]: owner, repo = rfc["repo"].split("/", 1) else: diff --git a/backend/app/main.py b/backend/app/main.py index 3f904ce..7ecd3f1 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -31,6 +31,7 @@ from . import ( projects, providers as providers_mod, ratelimit, + registry as registry_mod, turnstile, webhooks, ) @@ -100,8 +101,23 @@ async def lifespan(app: FastAPI): config = load_config() db.run_migrations(config) db.init(config) - projects.seed_default_project(config) # §22.13 step 1 gitea = Gitea(config) + # §22.2: mirror the registry before anything reads projects/content_repo. + # First boot has no last-good rows, so a missing/invalid registry is fatal + # (loud-fail per separation-of-concerns); the reconciler sweep keeps it + # fresh thereafter and tolerates a later bad PR. + try: + await registry_mod.refresh_registry(config, gitea) + except Exception as e: + raise RuntimeError( + f"registry mirror failed at startup ({config.registry_repo_full}/projects.yaml): {e}" + ) from e + if projects.default_content_repo(config) is None: + raise RuntimeError( + f"registry does not describe the default project " + f"{projects.resolved_default_id(config)!r} (no content_repo). " + f"Add it to {config.registry_repo_full}/projects.yaml." + ) bot = Bot(gitea) reconciler = cache.Reconciler(config, gitea) digest_sched = digest.DigestScheduler() @@ -130,7 +146,7 @@ async def lifespan(app: FastAPI): reconciler.start() digest_sched.start() hygiene_sched.start() - log.info("RFC app started — meta repo %s/%s", config.gitea_org, config.meta_repo) + log.info("RFC app started — registry %s", config.registry_repo_full) try: yield finally: diff --git a/backend/app/projects.py b/backend/app/projects.py index 6642bd0..4245939 100644 --- a/backend/app/projects.py +++ b/backend/app/projects.py @@ -1,11 +1,9 @@ """Project registry — the §22 multi-project layer. -A deployment hosts one or more projects (§22.1). Through Slice M1 the only -project is the migration-seeded `default` one (the N=1 case, §22.13); the git -registry mirror that lets a deployment declare more projects lands in M3 and -will grow this module. For now this holds just the §22.13 step-1 backfill: -completing the default project's row from deployment config, which pure-SQL -migration 026 could not read. +A deployment hosts one or more projects (§22.1). The git registry mirror +that lets a deployment declare projects lands in M3 and drives this module. +`seed_default_project` (the §22.13 META_REPO backfill) is retired in M3; +the registry mirror (`registry.refresh_registry`) is authoritative. """ from __future__ import annotations @@ -15,26 +13,30 @@ from .config import Config DEFAULT_PROJECT_ID = "default" -def seed_default_project(config: Config) -> None: - """Fill the default project's content_repo from config. +def resolved_default_id(config: Config) -> str: + """The id of the deployment's bootstrap/default project. Plan A: always + 'default' (the re-stamp to a config slug rides Plan B). The config knob is + read here so Plan B can flip the resolution without touching call sites.""" + return config.default_project_id.strip() or DEFAULT_PROJECT_ID - Migration 026 seeds `default` with content_repo NULL because SQL cannot - read the environment. This sets it from META_REPO (the pre-multi-project - single-corpus repo) the first time the configured app boots, so the row - accurately names the corpus the existing rows belong to. Idempotent: only - touches the row while content_repo is still unset, so a later registry - mirror (M3) that has populated it is never clobbered. - The display `name` is deliberately left as-is: the deployment's - user-visible name lives in the frontend build (VITE_APP_NAME) today and - moves to the registry + GET /api/deployment in M3 (§22.9); the backend has - no authoritative name to backfill from yet. - """ - db.conn().execute( - """ - UPDATE projects - SET content_repo = ?, updated_at = datetime('now') - WHERE id = ? AND content_repo IS NULL - """, - (config.meta_repo, DEFAULT_PROJECT_ID), - ) +def default_content_repo(config: Config) -> str | None: + """The content repo the single-corpus mirror reads, from the default + project's row (filled by the registry mirror). Replaces the retired + META_REPO. None until the registry mirror has run.""" + row = db.conn().execute( + "SELECT content_repo FROM projects WHERE id = ?", + (resolved_default_id(config),), + ).fetchone() + return row["content_repo"] if row and row["content_repo"] else None + + +def project_initial_state(project_id: str) -> str: + """§22.4b landing state for new entries in a project. Defaults to + 'super-draft' for an unknown/unset row (the safe, today's-flow default).""" + row = db.conn().execute( + "SELECT initial_state FROM projects WHERE id = ?", (project_id,) + ).fetchone() + if row is None or not row["initial_state"]: + return "super-draft" + return row["initial_state"] diff --git a/backend/app/registry.py b/backend/app/registry.py new file mode 100644 index 0000000..a04343f --- /dev/null +++ b/backend/app/registry.py @@ -0,0 +1,185 @@ +"""§22.2 project registry mirror — the config-side analogue of +cache.refresh_meta_repo. + +A deployment declares its projects in a `projects.yaml` at the root of +REGISTRY_REPO. This module mirrors that file into the `projects` cache table +and the `deployment` singleton. Per §22.2, `projects` rows flow from the +registry only — never from user actions. The mirror runs on the registry-repo +webhook and on every reconciler sweep (Option A wiring, Task 5). +""" +from __future__ import annotations + +import base64 +import json +import logging +import re +from dataclasses import dataclass, field + +import yaml + +from . import db +from .config import Config +from .gitea import Gitea + +log = logging.getLogger(__name__) + +VALID_TYPES = {"document", "specification", "bdd"} +VALID_VISIBILITY = {"gated", "public", "unlisted"} +VALID_INITIAL_STATE = {"super-draft", "active"} +# §22.4b: per-type default landing state. +_TYPE_DEFAULT_INITIAL_STATE = { + "document": "super-draft", + "specification": "super-draft", + "bdd": "active", +} +_SLUG_RE = re.compile(r"^[a-z0-9](?:[a-z0-9-]*[a-z0-9])?$") + + +class RegistryError(Exception): + """A registry document that fails validation, or a missing registry file. + + Raised by parse_registry/refresh_registry. The caller decides severity: + fatal at startup (no last-good to serve), tolerated on a running deployment + (keep the last-good projects rows). See Task 5 wiring. + """ + + +@dataclass +class ProjectEntry: + id: str + name: str + type: str + content_repo: str + visibility: str + initial_state: str + config: dict = field(default_factory=dict) # theme, enabled_models + + +@dataclass +class RegistryDoc: + deployment_name: str + deployment_tagline: str + projects: list[ProjectEntry] + + +def parse_registry(text: str) -> RegistryDoc: + """Parse + validate projects.yaml. Pure (no I/O). Raises RegistryError.""" + raw = yaml.safe_load(text) or {} + dep = raw.get("deployment") or {} + projects_raw = raw.get("projects") or [] + if not isinstance(projects_raw, list) or not projects_raw: + raise RegistryError("registry must declare at least one project") + seen: set[str] = set() + entries: list[ProjectEntry] = [] + for p in projects_raw: + if not isinstance(p, dict): + raise RegistryError(f"each project entry must be a mapping, got {type(p).__name__}") + pid = str(p.get("id") or "").strip() + if not _SLUG_RE.match(pid): + raise RegistryError(f"project id {pid!r} is not a valid slug") + if pid in seen: + raise RegistryError(f"duplicate project id {pid!r}") + seen.add(pid) + name = str(p.get("name") or "").strip() + if not name: + raise RegistryError(f"project {pid!r} missing name") + ptype = str(p.get("type") or "").strip() + if ptype not in VALID_TYPES: + raise RegistryError(f"project {pid!r} has invalid type {ptype!r}") + content_repo = str(p.get("content_repo") or "").strip() + if not content_repo: + raise RegistryError(f"project {pid!r} missing content_repo") + vis = str(p.get("visibility") or "gated").strip() + if vis not in VALID_VISIBILITY: + raise RegistryError(f"project {pid!r} has invalid visibility {vis!r}") + initial_state = str( + p.get("initial_state") or _TYPE_DEFAULT_INITIAL_STATE[ptype] + ).strip() + if initial_state not in VALID_INITIAL_STATE: + raise RegistryError( + f"project {pid!r} has invalid initial_state {initial_state!r}" + ) + cfg: dict = {} + if p.get("theme") is not None: + cfg["theme"] = p["theme"] + if p.get("enabled_models") is not None: + cfg["enabled_models"] = [str(m) for m in p["enabled_models"]] + entries.append( + ProjectEntry(pid, name, ptype, content_repo, vis, initial_state, cfg) + ) + return RegistryDoc( + deployment_name=str(dep.get("name") or "").strip(), + deployment_tagline=str(dep.get("tagline") or "").strip(), + projects=entries, + ) + + +def apply_registry(doc: RegistryDoc, registry_sha: str) -> None: + """Upsert the parsed registry into projects + deployment. Idempotent. + + §22.4a: `type` is immutable — a change against an existing row is rejected + (skip + log), never applied. Projects absent from the registry are left in + place (archival is out of scope for M3; they simply stop refreshing). + """ + with db.tx() as conn: + for e in doc.projects: + existing = conn.execute( + "SELECT type FROM projects WHERE id = ?", (e.id,) + ).fetchone() + if existing is not None and existing["type"] != e.type: + log.error( + "registry: refusing immutable type change on project %s (%s -> %s)", + e.id, existing["type"], e.type, + ) + continue + conn.execute( + """ + INSERT INTO projects + (id, name, type, content_repo, visibility, initial_state, + config_json, registry_sha, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, datetime('now')) + ON CONFLICT(id) DO UPDATE SET + name = excluded.name, + type = excluded.type, + content_repo = excluded.content_repo, + visibility = excluded.visibility, + initial_state = excluded.initial_state, + config_json = excluded.config_json, + registry_sha = excluded.registry_sha, + updated_at = datetime('now') + """, + ( + e.id, e.name, e.type, e.content_repo, e.visibility, + e.initial_state, json.dumps(e.config), registry_sha, + ), + ) + conn.execute( + """ + UPDATE deployment + SET name = ?, tagline = ?, registry_sha = ?, updated_at = datetime('now') + WHERE id = 1 + """, + (doc.deployment_name, doc.deployment_tagline, registry_sha), + ) + + +async def refresh_registry(config: Config, gitea: Gitea) -> None: + """Mirror REGISTRY_REPO/projects.yaml into projects + deployment. + + Idempotent. Raises RegistryError on a missing/invalid file and GiteaError + on transport failure; the caller chooses fatal-vs-tolerated. + """ + item = await gitea.get_contents( + config.gitea_org, config.registry_repo, "projects.yaml", ref="main" + ) + if not item or item.get("type") != "file": + raise RegistryError( + f"{config.gitea_org}/{config.registry_repo}/projects.yaml not found" + ) + text = base64.b64decode(item["content"]).decode("utf-8") + # Prefer the file's last commit sha for provenance (production Gitea + # includes it on the contents response); fall back to the blob sha. + sha = item.get("last_commit_sha") or item.get("sha") or "" + doc = parse_registry(text) + apply_registry(doc, sha) + log.info("registry: mirrored %d project(s) at %s", len(doc.projects), sha) diff --git a/backend/app/webhooks.py b/backend/app/webhooks.py index e47c167..56d0412 100644 --- a/backend/app/webhooks.py +++ b/backend/app/webhooks.py @@ -16,7 +16,7 @@ import os from fastapi import APIRouter, Header, HTTPException, Request -from . import cache, db +from . import cache, db, projects as projects_mod, registry as registry_mod from .config import Config from .gitea import Gitea @@ -79,9 +79,22 @@ def make_router(config: Config, gitea: Gitea) -> APIRouter: except Exception: payload = {} repo_full = (payload.get("repository") or {}).get("full_name") or "" - meta_full = f"{config.gitea_org}/{config.meta_repo}" + registry_full = f"{config.gitea_org}/{config.registry_repo}" + content_repo = projects_mod.default_content_repo(config) + if not content_repo: + log.warning("webhook: default project content_repo is unknown; corpus refresh skipped") + content_full = f"{config.gitea_org}/{content_repo}" if content_repo else None try: - if repo_full == meta_full or not repo_full: + if repo_full == registry_full: + # §22.2: a registry-repo push re-mirrors the projects table. + # Tolerate a malformed projects.yaml (keep last-good rows); let a + # transport error bubble to the outer 500 so an unreachable Gitea + # on a registry push is loud rather than silently dropped. + try: + await registry_mod.refresh_registry(config, gitea) + except registry_mod.RegistryError: + log.exception("registry webhook: invalid projects.yaml; keeping last-good") + elif content_full and (repo_full == content_full or not repo_full): await cache.refresh_meta_repo(config, gitea) await cache.refresh_meta_branches(config, gitea) await cache.refresh_meta_pulls(config, gitea) @@ -90,11 +103,6 @@ def make_router(config: Config, gitea: Gitea) -> APIRouter: if slug: await cache.refresh_rfc_repo(config, gitea, slug) else: - # v0.18.0: the proposal's "unknown-repo logging" - # gesture — a hook on a fork or a stale repo binding - # used to silently 200-OK here, hiding the - # misconfiguration. Now the operator sees it in - # the log. log.info( "webhook received for unknown repo: repo_full=%s event=%s " "(no cached_rfcs row matched; hook may be on a fork or stale)", diff --git a/backend/migrations/027_projects_runtime_config.sql b/backend/migrations/027_projects_runtime_config.sql new file mode 100644 index 0000000..937d792 --- /dev/null +++ b/backend/migrations/027_projects_runtime_config.sql @@ -0,0 +1,31 @@ +-- §22 M3 (Plan A) — additive registry/runtime-config schema. +-- +-- This is the additive half of M3's backend. It adds the project `type` and +-- `initial_state` columns (mirrored from the registry), the deployment +-- singleton (deployment name/tagline mirrored from the registry), and the +-- §22.4c review columns on cached_rfcs. NO table rebuilds: the §22.13 PK +-- rebuilds and the default->slug re-stamp ride a later migration (Plan B), +-- just before a second project can collide (see migration 026's DEFERRED +-- block and docs/superpowers/specs/2026-06-03-m3-backend-design.md §1/§6). + +ALTER TABLE projects ADD COLUMN type TEXT NOT NULL DEFAULT 'document' + CHECK (type IN ('document', 'specification', 'bdd')); +ALTER TABLE projects ADD COLUMN initial_state TEXT NOT NULL DEFAULT 'super-draft' + CHECK (initial_state IN ('super-draft', 'active')); + +-- Deployment-level identity (name, tagline) mirrored from the registry's +-- `deployment:` block. A singleton: the CHECK pins it to one row. +CREATE TABLE IF NOT EXISTS deployment ( + id INTEGER PRIMARY KEY CHECK (id = 1), + name TEXT, + tagline TEXT, + registry_sha TEXT, + updated_at TEXT NOT NULL DEFAULT (datetime('now')) +); +INSERT OR IGNORE INTO deployment (id) VALUES (1); + +-- §22.4c review flag + provenance. unreviewed is git-truth (mirrored from +-- entry frontmatter); it survives a cache rebuild like `state` does. +ALTER TABLE cached_rfcs ADD COLUMN unreviewed INTEGER NOT NULL DEFAULT 0; +ALTER TABLE cached_rfcs ADD COLUMN reviewed_at TEXT; +ALTER TABLE cached_rfcs ADD COLUMN reviewed_by TEXT; diff --git a/backend/tests/test_api_deployment.py b/backend/tests/test_api_deployment.py new file mode 100644 index 0000000..ea72f94 --- /dev/null +++ b/backend/tests/test_api_deployment.py @@ -0,0 +1,97 @@ +"""§22.9/§22.5 — GET /api/deployment + GET /api/projects/:id with visibility.""" +from __future__ import annotations + +from fastapi.testclient import TestClient + +from test_propose_vertical import ( # noqa: F401 + app_with_fake_gitea, tmp_env, provision_user_row, sign_in_as, +) + + +def _add_project(pid, name, vis, typ="document"): + from app import db + db.conn().execute( + "INSERT OR REPLACE INTO projects (id, name, type, content_repo, visibility, initial_state) " + "VALUES (?, ?, ?, ?, ?, 'super-draft')", + (pid, name, typ, pid, vis), + ) + + +def test_deployment_lists_public_omits_gated_and_unlisted_for_anon(app_with_fake_gitea): + app, _ = app_with_fake_gitea + with TestClient(app) as client: + _add_project("pub", "Public", "public") + _add_project("gat", "Gated", "gated") + _add_project("unl", "Unlisted", "unlisted") + r = client.get("/api/deployment") + assert r.status_code == 200 + body = r.json() + assert body["name"] == "Test Deployment" + ids = {p["id"] for p in body["projects"]} + assert "pub" in ids and "default" in ids # both public + assert "gat" not in ids # gated, anon not a member + assert "unl" not in ids # unlisted never enumerated + + +def test_projects_id_404_for_gated_non_member(app_with_fake_gitea): + app, _ = app_with_fake_gitea + with TestClient(app) as client: + _add_project("gat", "Gated", "gated") + assert client.get("/api/projects/gat").status_code == 404 + + +def test_projects_id_returns_config_for_public(app_with_fake_gitea): + app, _ = app_with_fake_gitea + with TestClient(app) as client: + from app import db + db.conn().execute( + "UPDATE projects SET config_json = ? WHERE id = 'default'", + ('{"theme": {"accent": "#5b5bd6"}}',), + ) + r = client.get("/api/projects/default") + assert r.status_code == 200 + body = r.json() + assert body["id"] == "default" + assert body["type"] == "document" + assert body["visibility"] == "public" + assert body["theme"] == {"accent": "#5b5bd6"} + + +def test_projects_id_unlisted_readable_by_direct_id(app_with_fake_gitea): + app, _ = app_with_fake_gitea + with TestClient(app) as client: + _add_project("unl", "Unlisted", "unlisted") + assert client.get("/api/projects/unl").status_code == 200 + + +def test_projects_id_unknown_returns_404_even_for_superuser(app_with_fake_gitea): + app, _ = app_with_fake_gitea + with TestClient(app) as client: + provision_user_row(user_id=1, login="ben", role="owner") + sign_in_as(client, user_id=1, gitea_login="ben", display_name="Ben", role="owner") + assert client.get("/api/projects/does-not-exist").status_code == 404 + + +def test_projects_id_unknown_returns_404_for_anon(app_with_fake_gitea): + app, _ = app_with_fake_gitea + with TestClient(app) as client: + assert client.get("/api/projects/nope").status_code == 404 + + +def test_gated_project_visible_and_readable_to_member(app_with_fake_gitea): + from app import db + app, _ = app_with_fake_gitea + with TestClient(app) as client: + _add_project("teamx", "Team X", "gated") + provision_user_row(user_id=5, login="mia", role="contributor") + db.conn().execute( + "INSERT INTO project_members (project_id, user_id, role) VALUES ('teamx', 5, 'project_viewer')" + ) + sign_in_as(client, user_id=5, gitea_login="mia", display_name="Mia", role="contributor") + # member sees the gated project in the deployment directory + ids = {p["id"] for p in client.get("/api/deployment").json()["projects"]} + assert "teamx" in ids + # member can read it directly + r = client.get("/api/projects/teamx") + assert r.status_code == 200 + assert r.json()["id"] == "teamx" diff --git a/backend/tests/test_cache_review_fields.py b/backend/tests/test_cache_review_fields.py new file mode 100644 index 0000000..c0a9627 --- /dev/null +++ b/backend/tests/test_cache_review_fields.py @@ -0,0 +1,24 @@ +"""§22.4c — _upsert_cached_rfc mirrors the review fields into cached_rfcs.""" +from __future__ import annotations + +from fastapi.testclient import TestClient + +from test_propose_vertical import app_with_fake_gitea, tmp_env # noqa: F401 + + +def test_upsert_writes_review_fields(app_with_fake_gitea): + from app import cache, db, entry as entry_mod + + app, _ = app_with_fake_gitea + with TestClient(app): + e = entry_mod.Entry( + slug="rev", title="Rev", state="active", + unreviewed=True, reviewed_at="2026-06-03", reviewed_by="ben", + ) + cache._upsert_cached_rfc(e, body_sha="sha-rev") + row = db.conn().execute( + "SELECT unreviewed, reviewed_at, reviewed_by FROM cached_rfcs WHERE slug = 'rev'" + ).fetchone() + assert row["unreviewed"] == 1 + assert row["reviewed_at"] == "2026-06-03" + assert row["reviewed_by"] == "ben" diff --git a/backend/tests/test_docs_sessions_vertical.py b/backend/tests/test_docs_sessions_vertical.py index d4f3b85..f7e6db2 100644 --- a/backend/tests/test_docs_sessions_vertical.py +++ b/backend/tests/test_docs_sessions_vertical.py @@ -70,27 +70,32 @@ class _UpstreamHandler: @pytest.fixture -def patched_httpx(monkeypatch): +def patched_httpx(monkeypatch, app_with_fake_gitea): # noqa: F811 """Provide a hook the test can call to install a MockTransport. - Returns a closure: `install(handler)` patches - `app.docs_sessions.httpx.AsyncClient` so every constructed client - uses the handler's transport. + Returns a closure: `install(handler)` patches `httpx.AsyncClient` + (via `app.docs_sessions.httpx.AsyncClient`) so every constructed + client uses the handler's transport. - NB: the upstream `app_with_fake_gitea` fixture also patches - `httpx.AsyncClient` (to route gitea calls to a FakeGitea handler), - and because `httpx` is a single shared module, that patch mutates - the *same* `AsyncClient` attribute we're about to overwrite. We - therefore import the unpatched class directly from the - `httpx._client` module so our install path can construct a fresh - real client around our MockTransport without going through the - FakeGitea wrapper. + M3 note: lifespan now calls `refresh_registry` which hits FakeGitea + via the gitea transport. Since `httpx` is a module singleton, installing + the docs transport would clobber the FakeGitea mock already installed + by `app_with_fake_gitea`. We use a COMPOSITE handler: Gitea API + requests (to `http://gitea.test/`) are delegated to FakeGitea; all + other requests go to the test-specific handler. """ from httpx._client import AsyncClient as RealAsyncClient + _fake = app_with_fake_gitea[1] + def install(handler): + def composite(request: httpx.Request) -> httpx.Response: + if "gitea.test" in str(request.url): + return _fake.handle(request) + return handler(request) + def patched(*args, **kwargs): - kwargs["transport"] = httpx.MockTransport(handler) + kwargs["transport"] = httpx.MockTransport(composite) return RealAsyncClient(*args, **kwargs) monkeypatch.setattr("app.docs_sessions.httpx.AsyncClient", patched) diff --git a/backend/tests/test_docs_specs_vertical.py b/backend/tests/test_docs_specs_vertical.py index f581c62..0472464 100644 --- a/backend/tests/test_docs_specs_vertical.py +++ b/backend/tests/test_docs_specs_vertical.py @@ -79,19 +79,28 @@ class _UpstreamHandler: @pytest.fixture -def patched_httpx(monkeypatch): +def patched_httpx(monkeypatch, app_with_fake_gitea): # noqa: F811 """Provide a hook the test can call to install a MockTransport. - Same shape as the docs_sessions fixture — `app_with_fake_gitea` - monkeypatches `httpx.AsyncClient` for the gitea side, so we - construct from the unpatched class directly to avoid the - FakeGitea wrapper. + M3 note: lifespan now calls `refresh_registry` which hits FakeGitea + via the gitea transport. Since `httpx` is a module singleton, installing + the docs transport would clobber the FakeGitea mock already installed + by `app_with_fake_gitea`. We use a COMPOSITE handler: Gitea API + requests (to `http://gitea.test/`) are delegated to FakeGitea; all + other requests go to the test-specific handler. """ from httpx._client import AsyncClient as RealAsyncClient + _fake = app_with_fake_gitea[1] + def install(handler): + def composite(request: httpx.Request) -> httpx.Response: + if "gitea.test" in str(request.url): + return _fake.handle(request) + return handler(request) + def patched(*args, **kwargs): - kwargs["transport"] = httpx.MockTransport(handler) + kwargs["transport"] = httpx.MockTransport(composite) return RealAsyncClient(*args, **kwargs) monkeypatch.setattr("app.docs_specs.httpx.AsyncClient", patched) diff --git a/backend/tests/test_entry_review_fields.py b/backend/tests/test_entry_review_fields.py new file mode 100644 index 0000000..d2f8994 --- /dev/null +++ b/backend/tests/test_entry_review_fields.py @@ -0,0 +1,42 @@ +"""§22.4c — the unreviewed/reviewed_at/reviewed_by entry frontmatter fields.""" +from __future__ import annotations + +from app import entry as entry_mod + + +def test_parse_defaults_unreviewed_false_when_absent(): + text = "---\nslug: ohm\ntitle: OHM\nstate: active\n---\n\nBody.\n" + e = entry_mod.parse(text) + assert e.unreviewed is False + assert e.reviewed_at is None + assert e.reviewed_by is None + + +def test_parse_reads_review_fields(): + text = ( + "---\nslug: ohm\ntitle: OHM\nstate: active\n" + "unreviewed: true\nreviewed_at: '2026-06-03'\nreviewed_by: ben\n---\n\nBody.\n" + ) + e = entry_mod.parse(text) + assert e.unreviewed is True + assert e.reviewed_at == "2026-06-03" + assert e.reviewed_by == "ben" + + +def test_serialize_emits_review_fields_only_when_meaningful(): + e = entry_mod.Entry(slug="a", title="A", state="super-draft") + assert "unreviewed" not in entry_mod.serialize(e) + assert "reviewed_at" not in entry_mod.serialize(e) + e2 = entry_mod.Entry(slug="b", title="B", state="active", unreviewed=True) + assert "unreviewed: true" in entry_mod.serialize(e2) + + +def test_round_trip_preserves_review_fields(): + e = entry_mod.Entry( + slug="b", title="B", state="active", + unreviewed=False, reviewed_at="2026-06-03", reviewed_by="ben", + ) + back = entry_mod.parse(entry_mod.serialize(e)) + assert back.reviewed_at == "2026-06-03" + assert back.reviewed_by == "ben" + assert back.unreviewed is False diff --git a/backend/tests/test_initial_state_landing.py b/backend/tests/test_initial_state_landing.py new file mode 100644 index 0000000..dd0931c --- /dev/null +++ b/backend/tests/test_initial_state_landing.py @@ -0,0 +1,43 @@ +"""§22.4b — a project with initial_state='active' lands new entries active + +unreviewed; the default 'super-draft' project is unchanged.""" +from __future__ import annotations + +from fastapi.testclient import TestClient + +from test_propose_vertical import ( # noqa: F401 + app_with_fake_gitea, tmp_env, provision_user_row, sign_in_as, +) + + +def _propose(client): + return client.post("/api/rfcs/propose", json={ + "title": "Active Lander", "slug": "active-lander", + "pitch": "Lands active.", "tags": [], + }) + + +def test_super_draft_default_unchanged(app_with_fake_gitea): + from app import entry as entry_mod + app, fake = app_with_fake_gitea + with TestClient(app) as client: + provision_user_row(user_id=1, login="ben", role="owner") + sign_in_as(client, user_id=1, gitea_login="ben", display_name="Ben", role="owner") + assert _propose(client).status_code == 200 + f = fake.files[("wiggleverse", "meta", "propose/active-lander", "rfcs/active-lander.md")] + e = entry_mod.parse(f["content"]) + assert e.state == "super-draft" + assert e.unreviewed is False + + +def test_active_initial_state_lands_active_unreviewed(app_with_fake_gitea): + from app import db, entry as entry_mod + app, fake = app_with_fake_gitea + with TestClient(app) as client: + db.conn().execute("UPDATE projects SET initial_state='active' WHERE id='default'") + provision_user_row(user_id=1, login="ben", role="owner") + sign_in_as(client, user_id=1, gitea_login="ben", display_name="Ben", role="owner") + assert _propose(client).status_code == 200 + f = fake.files[("wiggleverse", "meta", "propose/active-lander", "rfcs/active-lander.md")] + e = entry_mod.parse(f["content"]) + assert e.state == "active" + assert e.unreviewed is True diff --git a/backend/tests/test_mark_reviewed.py b/backend/tests/test_mark_reviewed.py new file mode 100644 index 0000000..6db5c5c --- /dev/null +++ b/backend/tests/test_mark_reviewed.py @@ -0,0 +1,65 @@ +"""§22.4c — owner/admin mark-reviewed clears the flag; catalog unreviewed filter.""" +from __future__ import annotations + +from fastapi.testclient import TestClient + +from test_propose_vertical import ( # noqa: F401 + app_with_fake_gitea, tmp_env, provision_user_row, sign_in_as, +) + + +def _seed_unreviewed_active(fake, slug="feat"): + """Put an active+unreviewed entry on the meta repo main + cache.""" + from app import cache, entry as entry_mod + body = entry_mod.serialize(entry_mod.Entry( + slug=slug, title="Feat", state="active", unreviewed=True, + owners=["ben"], proposed_by="ben", + )) + fake.files[("wiggleverse", "meta", "main", f"rfcs/{slug}.md")] = {"content": body, "sha": "s1"} + cache._upsert_cached_rfc(entry_mod.parse(body), body_sha="s1") + + +def test_catalog_unreviewed_filter(app_with_fake_gitea): + app, fake = app_with_fake_gitea + with TestClient(app) as client: + _seed_unreviewed_active(fake, "feat") + from app import cache, entry as entry_mod + ok = entry_mod.serialize(entry_mod.Entry(slug="ok", title="OK", state="active", owners=["ben"])) + fake.files[("wiggleverse", "meta", "main", "rfcs/ok.md")] = {"content": ok, "sha": "s2"} + cache._upsert_cached_rfc(entry_mod.parse(ok), body_sha="s2") + r = client.get("/api/rfcs", params={"unreviewed": "true"}) + slugs = {i["slug"] for i in r.json()["items"]} + assert slugs == {"feat"} + + +def test_mark_reviewed_clears_flag(app_with_fake_gitea): + from app import db + app, fake = app_with_fake_gitea + with TestClient(app) as client: + _seed_unreviewed_active(fake, "feat") + provision_user_row(user_id=1, login="ben", role="owner") + sign_in_as(client, user_id=1, gitea_login="ben", display_name="Ben", role="owner") + r = client.post("/api/projects/default/rfcs/feat/mark-reviewed") + assert r.status_code == 200 + row = db.conn().execute( + "SELECT unreviewed, reviewed_at, reviewed_by FROM cached_rfcs WHERE slug='feat'" + ).fetchone() + assert row["unreviewed"] == 0 + assert row["reviewed_by"] == "ben" + assert row["reviewed_at"] # provenance stamped + # git-side: the entry file on main was rewritten with the cleared flag. + from app import entry as entry_mod + written = fake.files[("wiggleverse", "meta", "main", "rfcs/feat.md")]["content"] + e = entry_mod.parse(written) + assert e.unreviewed is False + assert e.reviewed_by == "ben" + + +def test_mark_reviewed_forbidden_for_non_superuser(app_with_fake_gitea): + app, fake = app_with_fake_gitea + with TestClient(app) as client: + _seed_unreviewed_active(fake, "feat") + provision_user_row(user_id=2, login="carol", role="contributor") + sign_in_as(client, user_id=2, gitea_login="carol", display_name="Carol", role="contributor") + r = client.post("/api/projects/default/rfcs/feat/mark-reviewed") + assert r.status_code == 403 diff --git a/backend/tests/test_migration_027.py b/backend/tests/test_migration_027.py new file mode 100644 index 0000000..1c47ca7 --- /dev/null +++ b/backend/tests/test_migration_027.py @@ -0,0 +1,54 @@ +"""Migration 027 — additive §22 M3 schema (projects.type/initial_state, the +deployment singleton, cached_rfcs review columns). No table rebuilds in Plan A.""" +from __future__ import annotations + +import tempfile +from pathlib import Path + +from app import db +from app.config import Config + + +def _fresh_config() -> Config: + tmp = Path(tempfile.mkdtemp(prefix="mig027-")) / "t.db" + return Config( + gitea_url="x", gitea_bot_user="x", gitea_bot_token="x", gitea_org="x", + registry_repo="registry", + oauth_client_id="x", oauth_client_secret="x", app_url="x", + secret_key="x", database_path=tmp, owner_gitea_login="x", + webhook_secret="x", + ) + + +def test_027_adds_project_type_and_initial_state(): + cfg = _fresh_config() + db.run_migrations(cfg) + conn = db.connect(cfg.database_path) + cols = {r["name"]: r for r in conn.execute("PRAGMA table_info(projects)")} + assert "type" in cols and cols["type"]["dflt_value"] == "'document'" + assert "initial_state" in cols and cols["initial_state"]["dflt_value"] == "'super-draft'" + conn.close() + + +def test_027_creates_deployment_singleton(): + cfg = _fresh_config() + db.run_migrations(cfg) + conn = db.connect(cfg.database_path) + rows = list(conn.execute("SELECT id FROM deployment")) + assert [r["id"] for r in rows] == [1] + try: + conn.execute("INSERT INTO deployment (id) VALUES (2)") + raised = False + except Exception: + raised = True + assert raised + conn.close() + + +def test_027_adds_review_columns_to_cached_rfcs(): + cfg = _fresh_config() + db.run_migrations(cfg) + conn = db.connect(cfg.database_path) + cols = {r["name"] for r in conn.execute("PRAGMA table_info(cached_rfcs)")} + assert {"unreviewed", "reviewed_at", "reviewed_by"} <= cols + conn.close() diff --git a/backend/tests/test_multi_project_spine_vertical.py b/backend/tests/test_multi_project_spine_vertical.py index b48dd74..35ce1dd 100644 --- a/backend/tests/test_multi_project_spine_vertical.py +++ b/backend/tests/test_multi_project_spine_vertical.py @@ -1,11 +1,11 @@ -"""Slice M1 — the §22 multi-project spine. +"""Slice M1+M3 — the §22 multi-project spine. Migration 026 introduces the `projects` and `project_members` tables, seeds the single `default` project (the N=1 case, §22.13), and threads a `project_id` column onto every slug-bearing table, backfilled to `default`. -The §22.13 startup backfill then fills the default project's content_repo -from META_REPO. These tests prove the spine lands without disturbing the -single-project app. +M3 retires the META_REPO startup backfill; content_repo now comes from the +registry mirror (projects.yaml in REGISTRY_REPO). These tests prove the +spine lands without disturbing the single-project app. """ from __future__ import annotations @@ -40,7 +40,8 @@ def test_default_project_seeded_and_backfilled(app_with_fake_gitea): assert row["id"] == "default" # public preserves the pre-multi-project open-by-default posture. assert row["visibility"] == "public" - # §22.13 startup backfill set content_repo from META_REPO (tmp_env). + # M3: content_repo now comes from the registry mirror (projects.yaml), + # not the retired META_REPO startup backfill. assert row["content_repo"] == "meta" @@ -105,22 +106,25 @@ def test_project_members_table_shape(app_with_fake_gitea): pass -def test_seed_default_project_is_idempotent(app_with_fake_gitea): - """Re-running the backfill never clobbers a content_repo already set — - so a later registry mirror (M3) wins over the META_REPO fallback.""" - from app import db, projects - from app.config import load_config +def test_registry_mirror_is_idempotent(app_with_fake_gitea): + """Re-running the registry mirror is safe — it upserts (overwrites) the + projects row from projects.yaml each time without raising. M3 retirement + of seed_default_project: the registry mirror is now the sole authority.""" + import asyncio + from app import db, registry as registry_mod app, _ = app_with_fake_gitea with TestClient(app): - db.conn().execute( - "UPDATE projects SET content_repo = 'ohm-content' WHERE id = 'default'" - ) - projects.seed_default_project(load_config()) - got = db.conn().execute( - "SELECT content_repo FROM projects WHERE id = 'default'" - ).fetchone()["content_repo"] - assert got == "ohm-content" + before = db.conn().execute("SELECT COUNT(*) AS n FROM projects").fetchone()["n"] + cfg = app.state.config + gitea = app.state.gitea + asyncio.run(registry_mod.refresh_registry(cfg, gitea)) + after = db.conn().execute("SELECT COUNT(*) AS n FROM projects").fetchone()["n"] + assert after == before + row = db.conn().execute( + "SELECT content_repo FROM projects WHERE id='default'" + ).fetchone() + assert row["content_repo"] == "meta" def test_registry_repo_config_wired(app_with_fake_gitea, monkeypatch): @@ -128,6 +132,8 @@ def test_registry_repo_config_wired(app_with_fake_gitea, monkeypatch): monkeypatch.setenv("REGISTRY_REPO", "wiggleverse-registry") assert load_config().registry_repo == "wiggleverse-registry" - # Optional through M1: absent resolves to empty, not a startup failure. + # M3: REGISTRY_REPO is now required — absent raises RuntimeError. monkeypatch.delenv("REGISTRY_REPO", raising=False) - assert load_config().registry_repo == "" + import pytest + with pytest.raises(RuntimeError, match="REGISTRY_REPO"): + load_config() diff --git a/backend/tests/test_propose_vertical.py b/backend/tests/test_propose_vertical.py index e0bfc70..6045619 100644 --- a/backend/tests/test_propose_vertical.py +++ b/backend/tests/test_propose_vertical.py @@ -55,6 +55,24 @@ class FakeGitea: self._pr_counter = 0 self._commit_counter = 0 self._seed_repo("wiggleverse", "meta") + # §22 M3: the deployment's project registry. Startup refresh_registry + # reads projects.yaml here; the single 'default' project's content_repo + # points back at the seeded meta repo so the corpus mirror is unchanged. + self._seed_repo("wiggleverse", "registry") + self.files[("wiggleverse", "registry", "main", "projects.yaml")] = { + "content": ( + "deployment:\n" + " name: Test Deployment\n" + " tagline: A test deployment\n" + "projects:\n" + " - id: default\n" + " name: Test Deployment\n" + " type: document\n" + " content_repo: meta\n" + " visibility: public\n" + ), + "sha": "regsha0001", + } def _seed_repo(self, owner, repo): self.branches[(owner, repo)] = {"main": {"sha": "initial", "ts": "2026-05-23T00:00:00Z"}} @@ -431,7 +449,7 @@ def tmp_env(monkeypatch): "GITEA_BOT_USER": "rfc-bot", "GITEA_BOT_TOKEN": "bot-token", "GITEA_ORG": "wiggleverse", - "META_REPO": "meta", + "REGISTRY_REPO": "registry", "OAUTH_CLIENT_ID": "cid", "OAUTH_CLIENT_SECRET": "csec", "APP_URL": "http://localhost:8000", diff --git a/backend/tests/test_registry.py b/backend/tests/test_registry.py new file mode 100644 index 0000000..68e0022 --- /dev/null +++ b/backend/tests/test_registry.py @@ -0,0 +1,98 @@ +"""§22.2 registry parse + apply: validation, type-immutability, upsert.""" +from __future__ import annotations + +import tempfile +from pathlib import Path + +import pytest + +from app import db, registry +from app.config import Config + + +def _db(): + cfg = Config( + gitea_url="x", gitea_bot_user="x", gitea_bot_token="x", gitea_org="x", + registry_repo="registry", oauth_client_id="x", + oauth_client_secret="x", app_url="x", secret_key="x", + database_path=Path(tempfile.mkdtemp(prefix="reg-")) / "t.db", + owner_gitea_login="x", webhook_secret="x", + ) + db.run_migrations(cfg) + if db._CONN is not None: + db._CONN.close() + db._CONN = None + db.init(cfg) + return cfg + + +VALID = """ +deployment: + name: Open Human Model + tagline: A model of human flourishing +projects: + - id: default + name: Open Human Model + type: document + content_repo: meta + visibility: public +""" + + +def test_parse_valid_registry(): + doc = registry.parse_registry(VALID) + assert doc.deployment_name == "Open Human Model" + assert doc.deployment_tagline == "A model of human flourishing" + assert len(doc.projects) == 1 + p = doc.projects[0] + assert (p.id, p.type, p.content_repo, p.visibility) == ("default", "document", "meta", "public") + assert p.initial_state == "super-draft" + + +def test_parse_initial_state_defaults_per_type(): + doc = registry.parse_registry( + "projects:\n - {id: a, name: A, type: bdd, content_repo: a}\n" + ) + assert doc.projects[0].initial_state == "active" # bdd default + + +@pytest.mark.parametrize("bad,msg", [ + ("projects: []\n", "at least one"), + ("projects:\n - just-a-string\n", "must be a mapping"), + ("projects:\n - {id: 'Bad Slug', name: A, type: document, content_repo: a}\n", "valid slug"), + ("projects:\n - {id: a, name: A, type: nope, content_repo: a}\n", "invalid type"), + ("projects:\n - {id: a, name: A, type: document}\n", "content_repo"), + ("projects:\n - {id: a, name: A, type: document, content_repo: a, visibility: x}\n", "visibility"), + ("projects:\n - {id: a, name: A, type: document, content_repo: a}\n - {id: a, name: B, type: document, content_repo: b}\n", "duplicate"), +]) +def test_parse_rejects_invalid(bad, msg): + with pytest.raises(registry.RegistryError) as e: + registry.parse_registry(bad) + assert msg in str(e.value) + + +def test_apply_upserts_projects_and_deployment(): + _db() + doc = registry.parse_registry(VALID) + registry.apply_registry(doc, registry_sha="regsha1") + prow = db.conn().execute( + "SELECT name, type, content_repo, visibility, initial_state, registry_sha FROM projects WHERE id='default'" + ).fetchone() + assert prow["name"] == "Open Human Model" + assert prow["content_repo"] == "meta" + assert prow["registry_sha"] == "regsha1" + drow = db.conn().execute("SELECT name, tagline FROM deployment WHERE id=1").fetchone() + assert drow["name"] == "Open Human Model" + assert drow["tagline"] == "A model of human flourishing" + + +def test_apply_rejects_type_change_on_existing_project(): + _db() + registry.apply_registry(registry.parse_registry(VALID), "s1") + changed = VALID.replace("type: document", "type: specification") + registry.apply_registry(registry.parse_registry(changed), "s2") # logged + skipped, no raise + t = db.conn().execute("SELECT type FROM projects WHERE id='default'").fetchone()["type"] + assert t == "document" # immutable — unchanged + # The deployment row IS still advanced even though the project upsert was skipped. + drow = db.conn().execute("SELECT registry_sha FROM deployment WHERE id=1").fetchone() + assert drow["registry_sha"] == "s2" diff --git a/backend/tests/test_registry_wiring.py b/backend/tests/test_registry_wiring.py new file mode 100644 index 0000000..dedb7cd --- /dev/null +++ b/backend/tests/test_registry_wiring.py @@ -0,0 +1,50 @@ +"""Startup mirrors the registry; the registry webhook re-mirrors it.""" +from __future__ import annotations + +from fastapi.testclient import TestClient + +from test_propose_vertical import app_with_fake_gitea, tmp_env # noqa: F401 + + +def test_startup_mirrors_registry_into_projects_and_deployment(app_with_fake_gitea): + from app import db + + app, _ = app_with_fake_gitea + with TestClient(app): + prow = db.conn().execute( + "SELECT content_repo, type, initial_state FROM projects WHERE id='default'" + ).fetchone() + assert prow["content_repo"] == "meta" # from the registry, not META_REPO + assert prow["type"] == "document" + drow = db.conn().execute("SELECT name FROM deployment WHERE id=1").fetchone() + assert drow["name"] # deployment name mirrored from the registry + + +def test_registry_webhook_remirrors(app_with_fake_gitea): + import hashlib + import hmac + import json as _json + + app, fake = app_with_fake_gitea + with TestClient(app) as client: + from app import db + new_yaml = ( + "deployment:\n name: OHM\n tagline: Edited tagline\n" + "projects:\n - id: default\n name: OHM\n type: document\n" + " content_repo: meta\n visibility: public\n" + ) + fake.files[("wiggleverse", "registry", "main", "projects.yaml")] = { + "content": new_yaml, "sha": "regsha2", + } + body = _json.dumps({"repository": {"full_name": "wiggleverse/registry"}}).encode() + secret = "test-webhook-secret-for-signature-verification" + sig = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest() + r = client.post( + "/api/webhooks/gitea", + content=body, + headers={"X-Gitea-Event": "push", "X-Gitea-Signature": sig, + "Content-Type": "application/json"}, + ) + assert r.status_code == 200 + tagline = db.conn().execute("SELECT tagline FROM deployment WHERE id=1").fetchone()["tagline"] + assert tagline == "Edited tagline" diff --git a/backend/tests/test_webhooks_vertical.py b/backend/tests/test_webhooks_vertical.py index f55db1d..54ddb9d 100644 --- a/backend/tests/test_webhooks_vertical.py +++ b/backend/tests/test_webhooks_vertical.py @@ -73,6 +73,7 @@ def test_config_loads_with_empty_secret_when_bypass_is_set(monkeypatch, tmp_path monkeypatch.setenv("GITEA_BOT_USER", "rfc-bot") monkeypatch.setenv("GITEA_BOT_TOKEN", "bot-token") monkeypatch.setenv("GITEA_ORG", "wiggleverse") + monkeypatch.setenv("REGISTRY_REPO", "registry") monkeypatch.setenv("OAUTH_CLIENT_ID", "cid") monkeypatch.setenv("OAUTH_CLIENT_SECRET", "csec") monkeypatch.setenv("SECRET_KEY", "test-secret-key") @@ -92,6 +93,7 @@ def test_config_loads_with_secret_set(monkeypatch, tmp_path): monkeypatch.setenv("GITEA_BOT_USER", "rfc-bot") monkeypatch.setenv("GITEA_BOT_TOKEN", "bot-token") monkeypatch.setenv("GITEA_ORG", "wiggleverse") + monkeypatch.setenv("REGISTRY_REPO", "registry") monkeypatch.setenv("OAUTH_CLIENT_ID", "cid") monkeypatch.setenv("OAUTH_CLIENT_SECRET", "csec") monkeypatch.setenv("SECRET_KEY", "test-secret-key") diff --git a/docs/superpowers/plans/2026-06-03-m3-backend-plan-a.md b/docs/superpowers/plans/2026-06-03-m3-backend-plan-a.md new file mode 100644 index 0000000..7b05ecb --- /dev/null +++ b/docs/superpowers/plans/2026-06-03-m3-backend-plan-a.md @@ -0,0 +1,1634 @@ +# M3-backend Plan A — registry mirror + runtime config + review semantics + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the framework learn its projects from a git registry (`REGISTRY_REPO`), expose deployment + project config at runtime (`GET /api/deployment`, `GET /api/projects/:id`), and carry the `initial_state`/`unreviewed` review semantics — all additively, operating on the single `default`-id project (no PK rebuild, no re-stamp, no writer threading; those are Plan B). + +**Architecture:** A self-contained `app/registry.py` mirrors `REGISTRY_REPO/projects.yaml` into the existing `projects` table + a new `deployment` singleton, driven by the existing webhook dispatcher and `Reconciler.sweep()` (Option A from the spec). `META_REPO` is retired (hard cut): the corpus mirror reads the default project's `content_repo` from the DB instead. Migration `027` is additive only. See spec: `docs/superpowers/specs/2026-06-03-m3-backend-design.md`. + +**Tech Stack:** Python 3 / FastAPI / SQLite (raw `sqlite3`), `pyyaml`, `pytest` + `httpx.MockTransport` (the `FakeGitea` harness). Tests run from `backend/` with `python -m pytest`. + +--- + +## Conventions for every task + +- Work on branch `m3-backend-registry-spine` (already checked out). +- Run tests from the `backend/` directory: `cd backend && python -m pytest`. +- The shared integration fixtures `tmp_env`, `app_with_fake_gitea`, `FakeGitea`, `sign_in_as`, `provision_user_row` live in `backend/tests/test_propose_vertical.py` and are imported by other test files (e.g. `from test_propose_vertical import app_with_fake_gitea, tmp_env # noqa: F401`). +- After Task 5, the whole suite must stay green; run `python -m pytest` (full suite) at the end of every subsequent task. + +--- + +## Task 1: Entry review fields (`unreviewed` / `reviewed_at` / `reviewed_by`) + +**Files:** +- Modify: `backend/app/entry.py` (dataclass ~29-53, `parse` ~56-85, `serialize` ~88-117) +- Test: `backend/tests/test_entry_review_fields.py` + +- [ ] **Step 1: Write the failing test** + +```python +# backend/tests/test_entry_review_fields.py +"""§22.4c — the unreviewed/reviewed_at/reviewed_by entry frontmatter fields.""" +from __future__ import annotations + +from app import entry as entry_mod + + +def test_parse_defaults_unreviewed_false_when_absent(): + text = "---\nslug: ohm\ntitle: OHM\nstate: active\n---\n\nBody.\n" + e = entry_mod.parse(text) + assert e.unreviewed is False + assert e.reviewed_at is None + assert e.reviewed_by is None + + +def test_parse_reads_review_fields(): + text = ( + "---\nslug: ohm\ntitle: OHM\nstate: active\n" + "unreviewed: true\nreviewed_at: '2026-06-03'\nreviewed_by: ben\n---\n\nBody.\n" + ) + e = entry_mod.parse(text) + assert e.unreviewed is True + assert e.reviewed_at == "2026-06-03" + assert e.reviewed_by == "ben" + + +def test_serialize_emits_review_fields_only_when_meaningful(): + # unreviewed False + no provenance → keys omitted (keeps document entries clean). + e = entry_mod.Entry(slug="a", title="A", state="super-draft") + assert "unreviewed" not in entry_mod.serialize(e) + assert "reviewed_at" not in entry_mod.serialize(e) + # unreviewed True → emitted. + e2 = entry_mod.Entry(slug="b", title="B", state="active", unreviewed=True) + assert "unreviewed: true" in entry_mod.serialize(e2) + + +def test_round_trip_preserves_review_fields(): + e = entry_mod.Entry( + slug="b", title="B", state="active", + unreviewed=False, reviewed_at="2026-06-03", reviewed_by="ben", + ) + back = entry_mod.parse(entry_mod.serialize(e)) + assert back.reviewed_at == "2026-06-03" + assert back.reviewed_by == "ben" + assert back.unreviewed is False +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend && python -m pytest tests/test_entry_review_fields.py -v` +Expected: FAIL — `Entry.__init__() got an unexpected keyword argument 'unreviewed'`. + +- [ ] **Step 3: Add the dataclass fields** + +In `backend/app/entry.py`, add to the `Entry` dataclass (after `funder: str | None = None`, before `body: str = ""`): + +```python + # §22.4c: an `active` entry that landed without a human review gate + # carries unreviewed=True until an owner clears it. Orthogonal to + # `state`; only meaningful for active entries. reviewed_at/reviewed_by + # are the provenance of the clear, paralleling graduated_at/by. + unreviewed: bool = False + reviewed_at: str | None = None + reviewed_by: str | None = None +``` + +- [ ] **Step 4: Parse the fields** + +In `parse()`, add before the `return Entry(`: + +```python + unreviewed = bool(fm.get("unreviewed") or False) +``` + +and add these keyword args to the `Entry(...)` constructor call: + +```python + unreviewed=unreviewed, + reviewed_at=fm.get("reviewed_at") or None, + reviewed_by=fm.get("reviewed_by") or None, +``` + +- [ ] **Step 5: Serialize the fields** + +In `serialize()`, after the `funder` block (`if entry.funder:`), add: + +```python + # §22.4c: emit unreviewed only when True (a super-draft / reviewed + # active entry leaves the key absent → frontmatter stays minimal). + if entry.unreviewed: + fm["unreviewed"] = True + if entry.reviewed_at: + fm["reviewed_at"] = entry.reviewed_at + if entry.reviewed_by: + fm["reviewed_by"] = entry.reviewed_by +``` + +- [ ] **Step 6: Run tests to verify they pass** + +Run: `cd backend && python -m pytest tests/test_entry_review_fields.py -v` +Expected: PASS (4 tests). + +- [ ] **Step 7: Commit** + +```bash +git add backend/app/entry.py backend/tests/test_entry_review_fields.py +git commit -m "feat(entry): §22.4c unreviewed/reviewed_at/reviewed_by frontmatter fields" +``` + +--- + +## Task 2: Migration 027 — additive registry/review schema + +**Files:** +- Create: `backend/migrations/027_projects_runtime_config.sql` +- Test: `backend/tests/test_migration_027.py` + +- [ ] **Step 1: Write the failing test** + +```python +# backend/tests/test_migration_027.py +"""Migration 027 — additive §22 M3 schema (projects.type/initial_state, the +deployment singleton, cached_rfcs review columns). No table rebuilds in Plan A.""" +from __future__ import annotations + +import tempfile +from pathlib import Path + +from app import db +from app.config import Config + + +def _fresh_config() -> Config: + tmp = Path(tempfile.mkdtemp(prefix="mig027-")) / "t.db" + return Config( + gitea_url="x", gitea_bot_user="x", gitea_bot_token="x", gitea_org="x", + meta_repo="meta", registry_repo="registry", + oauth_client_id="x", oauth_client_secret="x", app_url="x", + secret_key="x", database_path=tmp, owner_gitea_login="x", + webhook_secret="x", + ) + + +def test_027_adds_project_type_and_initial_state(): + cfg = _fresh_config() + db.run_migrations(cfg) + conn = db.connect(cfg.database_path) + cols = {r["name"]: r for r in conn.execute("PRAGMA table_info(projects)")} + assert "type" in cols and cols["type"]["dflt_value"] == "'document'" + assert "initial_state" in cols and cols["initial_state"]["dflt_value"] == "'super-draft'" + conn.close() + + +def test_027_creates_deployment_singleton(): + cfg = _fresh_config() + db.run_migrations(cfg) + conn = db.connect(cfg.database_path) + rows = list(conn.execute("SELECT id FROM deployment")) + assert [r["id"] for r in rows] == [1] + # The CHECK (id = 1) forbids a second row. + try: + conn.execute("INSERT INTO deployment (id) VALUES (2)") + raised = False + except Exception: + raised = True + assert raised + conn.close() + + +def test_027_adds_review_columns_to_cached_rfcs(): + cfg = _fresh_config() + db.run_migrations(cfg) + conn = db.connect(cfg.database_path) + cols = {r["name"] for r in conn.execute("PRAGMA table_info(cached_rfcs)")} + assert {"unreviewed", "reviewed_at", "reviewed_by"} <= cols + conn.close() +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend && python -m pytest tests/test_migration_027.py -v` +Expected: FAIL — `type` not in projects columns (migration doesn't exist yet). + +- [ ] **Step 3: Write the migration** + +```sql +-- backend/migrations/027_projects_runtime_config.sql +-- §22 M3 (Plan A) — additive registry/runtime-config schema. +-- +-- This is the additive half of M3's backend. It adds the project `type` and +-- `initial_state` columns (mirrored from the registry), the deployment +-- singleton (deployment name/tagline mirrored from the registry), and the +-- §22.4c review columns on cached_rfcs. NO table rebuilds: the §22.13 PK +-- rebuilds and the default->slug re-stamp ride a later migration (Plan B), +-- just before a second project can collide (see migration 026's DEFERRED +-- block and docs/superpowers/specs/2026-06-03-m3-backend-design.md §1/§6). + +ALTER TABLE projects ADD COLUMN type TEXT NOT NULL DEFAULT 'document' + CHECK (type IN ('document', 'specification', 'bdd')); +ALTER TABLE projects ADD COLUMN initial_state TEXT NOT NULL DEFAULT 'super-draft' + CHECK (initial_state IN ('super-draft', 'active')); + +-- Deployment-level identity (name, tagline) mirrored from the registry's +-- `deployment:` block. A singleton: the CHECK pins it to one row. +CREATE TABLE IF NOT EXISTS deployment ( + id INTEGER PRIMARY KEY CHECK (id = 1), + name TEXT, + tagline TEXT, + registry_sha TEXT, + updated_at TEXT NOT NULL DEFAULT (datetime('now')) +); +INSERT OR IGNORE INTO deployment (id) VALUES (1); + +-- §22.4c review flag + provenance. unreviewed is git-truth (mirrored from +-- entry frontmatter); it survives a cache rebuild like `state` does. +ALTER TABLE cached_rfcs ADD COLUMN unreviewed INTEGER NOT NULL DEFAULT 0; +ALTER TABLE cached_rfcs ADD COLUMN reviewed_at TEXT; +ALTER TABLE cached_rfcs ADD COLUMN reviewed_by TEXT; +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd backend && python -m pytest tests/test_migration_027.py -v` +Expected: PASS (3 tests). + +- [ ] **Step 5: Run the full suite (no regressions from the additive migration)** + +Run: `cd backend && python -m pytest -q` +Expected: PASS (the additive columns don't disturb existing queries). + +- [ ] **Step 6: Commit** + +```bash +git add backend/migrations/027_projects_runtime_config.sql backend/tests/test_migration_027.py +git commit -m "feat(db): migration 027 — additive §22 M3 schema (type/initial_state, deployment, review cols)" +``` + +--- + +## Task 3: Cache mirrors the review fields + +**Files:** +- Modify: `backend/app/cache.py` (`_upsert_cached_rfc` ~78-129) +- Test: `backend/tests/test_cache_review_fields.py` + +- [ ] **Step 1: Write the failing test** + +```python +# backend/tests/test_cache_review_fields.py +"""§22.4c — _upsert_cached_rfc mirrors the review fields into cached_rfcs.""" +from __future__ import annotations + +from fastapi.testclient import TestClient + +from test_propose_vertical import app_with_fake_gitea, tmp_env # noqa: F401 + + +def test_upsert_writes_review_fields(app_with_fake_gitea): + from app import cache, db, entry as entry_mod + + app, _ = app_with_fake_gitea + with TestClient(app): + e = entry_mod.Entry( + slug="rev", title="Rev", state="active", + unreviewed=True, reviewed_at="2026-06-03", reviewed_by="ben", + ) + cache._upsert_cached_rfc(e, body_sha="sha-rev") + row = db.conn().execute( + "SELECT unreviewed, reviewed_at, reviewed_by FROM cached_rfcs WHERE slug = 'rev'" + ).fetchone() + assert row["unreviewed"] == 1 + assert row["reviewed_at"] == "2026-06-03" + assert row["reviewed_by"] == "ben" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend && python -m pytest tests/test_cache_review_fields.py -v` +Expected: FAIL — the upsert doesn't write `unreviewed` (column stays default 0; `reviewed_at` stays NULL). + +- [ ] **Step 3: Extend `_upsert_cached_rfc`** + +In `backend/app/cache.py`, change the INSERT column list, the VALUES placeholders, the `ON CONFLICT(slug) DO UPDATE SET` clause, and the params tuple to include the three review fields. The edited statement: + +```python + db.conn().execute( + """ + INSERT INTO cached_rfcs + (slug, title, state, rfc_id, repo, proposed_by, proposed_at, + graduated_at, graduated_by, owners_json, arbiters_json, tags_json, + models_json, funder_login, body, body_sha, + unreviewed, reviewed_at, reviewed_by, + last_entry_commit_at, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, datetime('now'), datetime('now')) + ON CONFLICT(slug) DO UPDATE SET + title = excluded.title, + state = excluded.state, + rfc_id = excluded.rfc_id, + repo = excluded.repo, + proposed_by = excluded.proposed_by, + proposed_at = excluded.proposed_at, + graduated_at = excluded.graduated_at, + graduated_by = excluded.graduated_by, + owners_json = excluded.owners_json, + arbiters_json = excluded.arbiters_json, + tags_json = excluded.tags_json, + models_json = excluded.models_json, + funder_login = excluded.funder_login, + body = excluded.body, + body_sha = excluded.body_sha, + unreviewed = excluded.unreviewed, + reviewed_at = excluded.reviewed_at, + reviewed_by = excluded.reviewed_by, + last_entry_commit_at = datetime('now'), + updated_at = datetime('now') + """, + ( + entry.slug, + entry.title, + entry.state, + entry.id, + entry.repo, + entry.proposed_by, + entry.proposed_at, + entry.graduated_at, + entry.graduated_by, + json.dumps(entry.owners), + json.dumps(entry.arbiters), + json.dumps(entry.tags), + models_json, + funder_login, + entry.body, + body_sha, + 1 if entry.unreviewed else 0, + entry.reviewed_at, + entry.reviewed_by, + ), + ) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd backend && python -m pytest tests/test_cache_review_fields.py -q` +Expected: PASS. + +- [ ] **Step 5: Run the full suite** + +Run: `cd backend && python -m pytest -q` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add backend/app/cache.py backend/tests/test_cache_review_fields.py +git commit -m "feat(cache): mirror §22.4c review fields into cached_rfcs" +``` + +--- + +## Task 4: `app/registry.py` — the registry mirror module + +**Files:** +- Create: `backend/app/registry.py` +- Test: `backend/tests/test_registry.py` + +The module factors into pure functions (`parse_registry`, `apply_registry`) that are unit-testable without I/O, plus `refresh_registry` (the I/O orchestrator, integration-tested in Task 5). + +- [ ] **Step 1: Write the failing test** + +```python +# backend/tests/test_registry.py +"""§22.2 registry parse + apply: validation, type-immutability, upsert.""" +from __future__ import annotations + +import tempfile +from pathlib import Path + +import pytest + +from app import db, registry +from app.config import Config + + +def _db(): + cfg = Config( + gitea_url="x", gitea_bot_user="x", gitea_bot_token="x", gitea_org="x", + meta_repo="meta", registry_repo="registry", oauth_client_id="x", + oauth_client_secret="x", app_url="x", secret_key="x", + database_path=Path(tempfile.mkdtemp(prefix="reg-")) / "t.db", + owner_gitea_login="x", webhook_secret="x", + ) + db.run_migrations(cfg) + if db._CONN is not None: + db._CONN.close() + db._CONN = None + db.init(cfg) + return cfg + + +VALID = """ +deployment: + name: Open Human Model + tagline: A model of human flourishing +projects: + - id: default + name: Open Human Model + type: document + content_repo: meta + visibility: public +""" + + +def test_parse_valid_registry(): + doc = registry.parse_registry(VALID) + assert doc.deployment_name == "Open Human Model" + assert doc.deployment_tagline == "A model of human flourishing" + assert len(doc.projects) == 1 + p = doc.projects[0] + assert (p.id, p.type, p.content_repo, p.visibility) == ("default", "document", "meta", "public") + # initial_state defaults from type when omitted. + assert p.initial_state == "super-draft" + + +def test_parse_initial_state_defaults_per_type(): + doc = registry.parse_registry( + "projects:\n - {id: a, name: A, type: bdd, content_repo: a}\n" + ) + assert doc.projects[0].initial_state == "active" # bdd default + + +@pytest.mark.parametrize("bad,msg", [ + ("projects: []\n", "at least one"), + ("projects:\n - {id: 'Bad Slug', name: A, type: document, content_repo: a}\n", "valid slug"), + ("projects:\n - {id: a, name: A, type: nope, content_repo: a}\n", "invalid type"), + ("projects:\n - {id: a, name: A, type: document}\n", "content_repo"), + ("projects:\n - {id: a, name: A, type: document, content_repo: a, visibility: x}\n", "visibility"), + ("projects:\n - {id: a, name: A, type: document, content_repo: a}\n - {id: a, name: B, type: document, content_repo: b}\n", "duplicate"), +]) +def test_parse_rejects_invalid(bad, msg): + with pytest.raises(registry.RegistryError) as e: + registry.parse_registry(bad) + assert msg in str(e.value) + + +def test_apply_upserts_projects_and_deployment(): + _db() + doc = registry.parse_registry(VALID) + registry.apply_registry(doc, registry_sha="regsha1") + prow = db.conn().execute( + "SELECT name, type, content_repo, visibility, initial_state, registry_sha FROM projects WHERE id='default'" + ).fetchone() + assert prow["name"] == "Open Human Model" + assert prow["content_repo"] == "meta" + assert prow["registry_sha"] == "regsha1" + drow = db.conn().execute("SELECT name, tagline FROM deployment WHERE id=1").fetchone() + assert drow["name"] == "Open Human Model" + assert drow["tagline"] == "A model of human flourishing" + + +def test_apply_rejects_type_change_on_existing_project(): + _db() + registry.apply_registry(registry.parse_registry(VALID), "s1") + changed = VALID.replace("type: document", "type: specification") + registry.apply_registry(registry.parse_registry(changed), "s2") # logged + skipped, no raise + t = db.conn().execute("SELECT type FROM projects WHERE id='default'").fetchone()["type"] + assert t == "document" # immutable — unchanged +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend && python -m pytest tests/test_registry.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'app.registry'` (the M1 `projects.py` exists, but `registry.py` does not). + +- [ ] **Step 3: Write `app/registry.py`** + +```python +# backend/app/registry.py +"""§22.2 project registry mirror — the config-side analogue of +cache.refresh_meta_repo. + +A deployment declares its projects in a `projects.yaml` at the root of +REGISTRY_REPO. This module mirrors that file into the `projects` cache table +and the `deployment` singleton. Per §22.2, `projects` rows flow from the +registry only — never from user actions. The mirror runs on the registry-repo +webhook and on every reconciler sweep (Option A wiring, Task 5). +""" +from __future__ import annotations + +import base64 +import json +import logging +import re +from dataclasses import dataclass, field + +import yaml + +from . import db +from .config import Config +from .gitea import Gitea + +log = logging.getLogger(__name__) + +VALID_TYPES = {"document", "specification", "bdd"} +VALID_VISIBILITY = {"gated", "public", "unlisted"} +VALID_INITIAL_STATE = {"super-draft", "active"} +# §22.4b: per-type default landing state. +_TYPE_DEFAULT_INITIAL_STATE = { + "document": "super-draft", + "specification": "super-draft", + "bdd": "active", +} +_SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9-]*$") + + +class RegistryError(Exception): + """A registry document that fails validation, or a missing registry file. + + Raised by parse_registry/refresh_registry. The caller decides severity: + fatal at startup (no last-good to serve), tolerated on a running deployment + (keep the last-good projects rows). See Task 5 wiring. + """ + + +@dataclass +class ProjectEntry: + id: str + name: str + type: str + content_repo: str + visibility: str + initial_state: str + config: dict = field(default_factory=dict) # theme, enabled_models + + +@dataclass +class RegistryDoc: + deployment_name: str + deployment_tagline: str + projects: list[ProjectEntry] + + +def parse_registry(text: str) -> RegistryDoc: + """Parse + validate projects.yaml. Pure (no I/O). Raises RegistryError.""" + raw = yaml.safe_load(text) or {} + dep = raw.get("deployment") or {} + projects_raw = raw.get("projects") or [] + if not isinstance(projects_raw, list) or not projects_raw: + raise RegistryError("registry must declare at least one project") + seen: set[str] = set() + entries: list[ProjectEntry] = [] + for p in projects_raw: + pid = str(p.get("id") or "").strip() + if not _SLUG_RE.match(pid): + raise RegistryError(f"project id {pid!r} is not a valid slug") + if pid in seen: + raise RegistryError(f"duplicate project id {pid!r}") + seen.add(pid) + name = str(p.get("name") or "").strip() + if not name: + raise RegistryError(f"project {pid!r} missing name") + ptype = str(p.get("type") or "").strip() + if ptype not in VALID_TYPES: + raise RegistryError(f"project {pid!r} has invalid type {ptype!r}") + content_repo = str(p.get("content_repo") or "").strip() + if not content_repo: + raise RegistryError(f"project {pid!r} missing content_repo") + vis = str(p.get("visibility") or "gated").strip() + if vis not in VALID_VISIBILITY: + raise RegistryError(f"project {pid!r} has invalid visibility {vis!r}") + initial_state = str( + p.get("initial_state") or _TYPE_DEFAULT_INITIAL_STATE[ptype] + ).strip() + if initial_state not in VALID_INITIAL_STATE: + raise RegistryError( + f"project {pid!r} has invalid initial_state {initial_state!r}" + ) + cfg: dict = {} + if p.get("theme") is not None: + cfg["theme"] = p["theme"] + if p.get("enabled_models") is not None: + cfg["enabled_models"] = [str(m) for m in p["enabled_models"]] + entries.append( + ProjectEntry(pid, name, ptype, content_repo, vis, initial_state, cfg) + ) + return RegistryDoc( + deployment_name=str(dep.get("name") or "").strip(), + deployment_tagline=str(dep.get("tagline") or "").strip(), + projects=entries, + ) + + +def apply_registry(doc: RegistryDoc, registry_sha: str) -> None: + """Upsert the parsed registry into projects + deployment. Idempotent. + + §22.4a: `type` is immutable — a change against an existing row is rejected + (skip + log), never applied. Projects absent from the registry are left in + place (archival is out of scope for M3; they simply stop refreshing). + """ + conn = db.conn() + for e in doc.projects: + existing = conn.execute( + "SELECT type FROM projects WHERE id = ?", (e.id,) + ).fetchone() + if existing is not None and existing["type"] not in (None, "", e.type): + log.error( + "registry: refusing immutable type change on project %s (%s -> %s)", + e.id, existing["type"], e.type, + ) + continue + conn.execute( + """ + INSERT INTO projects + (id, name, type, content_repo, visibility, initial_state, + config_json, registry_sha, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, datetime('now')) + ON CONFLICT(id) DO UPDATE SET + name = excluded.name, + type = excluded.type, + content_repo = excluded.content_repo, + visibility = excluded.visibility, + initial_state = excluded.initial_state, + config_json = excluded.config_json, + registry_sha = excluded.registry_sha, + updated_at = datetime('now') + """, + ( + e.id, e.name, e.type, e.content_repo, e.visibility, + e.initial_state, json.dumps(e.config), registry_sha, + ), + ) + conn.execute( + """ + UPDATE deployment + SET name = ?, tagline = ?, registry_sha = ?, updated_at = datetime('now') + WHERE id = 1 + """, + (doc.deployment_name, doc.deployment_tagline, registry_sha), + ) + + +async def refresh_registry(config: Config, gitea: Gitea) -> None: + """Mirror REGISTRY_REPO/projects.yaml into projects + deployment. + + Idempotent. Raises RegistryError on a missing/invalid file and GiteaError + on transport failure; the caller chooses fatal-vs-tolerated. + """ + item = await gitea.get_contents( + config.gitea_org, config.registry_repo, "projects.yaml", ref="main" + ) + if not item or item.get("type") != "file": + raise RegistryError( + f"{config.gitea_org}/{config.registry_repo}/projects.yaml not found" + ) + text = base64.b64decode(item["content"]).decode("utf-8") + sha = item.get("last_commit_sha") or item.get("sha") or "" + doc = parse_registry(text) + apply_registry(doc, sha) + log.info("registry: mirrored %d project(s) at %s", len(doc.projects), sha) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd backend && python -m pytest tests/test_registry.py -v` +Expected: PASS (all parametrized cases + apply tests). + +- [ ] **Step 5: Commit** + +```bash +git add backend/app/registry.py backend/tests/test_registry.py +git commit -m "feat(registry): §22.2 projects.yaml parse/validate/apply module" +``` + +--- + +## Task 5: Hard cut — retire META_REPO, wire the registry mirror + +This is the integration task: it removes `META_REPO`, points the corpus mirror at the default project's `content_repo` from the DB, wires `refresh_registry` into startup + webhook + sweep, retires `seed_default_project`, and updates the shared test fixture to seed a registry (so every test's startup `refresh_registry` succeeds). It must land atomically — the suite is red between the config change and the fixture update. + +**Files:** +- Modify: `backend/app/config.py` +- Modify: `backend/app/projects.py` +- Modify: `backend/app/cache.py` (`refresh_meta_repo`/`refresh_meta_branches`/`refresh_meta_pulls`, `Reconciler.sweep`) +- Modify: `backend/app/webhooks.py` +- Modify: `backend/app/main.py` (lifespan) +- Modify: `backend/tests/test_propose_vertical.py` (`tmp_env`, `FakeGitea.__init__`) +- Modify: `backend/tests/test_multi_project_spine_vertical.py` (stale comment in one assertion) +- Test: `backend/tests/test_registry_wiring.py` + +- [ ] **Step 1: Write the failing test** + +```python +# backend/tests/test_registry_wiring.py +"""Startup mirrors the registry; the registry webhook re-mirrors it.""" +from __future__ import annotations + +from fastapi.testclient import TestClient + +from test_propose_vertical import app_with_fake_gitea, tmp_env # noqa: F401 + + +def test_startup_mirrors_registry_into_projects_and_deployment(app_with_fake_gitea): + from app import db + + app, _ = app_with_fake_gitea + with TestClient(app): + # The fixture's projects.yaml declares the default project + a name. + prow = db.conn().execute( + "SELECT content_repo, type, initial_state FROM projects WHERE id='default'" + ).fetchone() + assert prow["content_repo"] == "meta" # from the registry, not META_REPO + assert prow["type"] == "document" + drow = db.conn().execute("SELECT name FROM deployment WHERE id=1").fetchone() + assert drow["name"] # deployment name mirrored from the registry + + +def test_registry_webhook_remirrors(app_with_fake_gitea): + import hashlib + import hmac + import json as _json + + app, fake = app_with_fake_gitea + with TestClient(app) as client: + from app import db + # Edit projects.yaml in the fake registry repo: change the tagline. + new_yaml = ( + "deployment:\n name: OHM\n tagline: Edited tagline\n" + "projects:\n - id: default\n name: OHM\n type: document\n" + " content_repo: meta\n visibility: public\n" + ) + fake.files[("wiggleverse", "registry", "main", "projects.yaml")] = { + "content": new_yaml, "sha": "regsha2", + } + body = _json.dumps({"repository": {"full_name": "wiggleverse/registry"}}).encode() + secret = "test-webhook-secret-for-signature-verification" + sig = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest() + r = client.post( + "/api/webhooks/gitea", + content=body, + headers={"X-Gitea-Event": "push", "X-Gitea-Signature": sig, + "Content-Type": "application/json"}, + ) + assert r.status_code == 200 + tagline = db.conn().execute("SELECT tagline FROM deployment WHERE id=1").fetchone()["tagline"] + assert tagline == "Edited tagline" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend && python -m pytest tests/test_registry_wiring.py -v` +Expected: FAIL — startup does not mirror the registry yet (`type`/`deployment.name` empty), and the registry webhook branch doesn't exist. + +- [ ] **Step 3: Config — require REGISTRY_REPO, remove META_REPO, add DEFAULT_PROJECT_ID** + +In `backend/app/config.py`: + +Replace the `meta_repo: str` and `registry_repo: str` dataclass fields with: + +```python + registry_repo: str + default_project_id: str = "" +``` + +(Remove the `meta_repo` field entirely.) + +Replace the `meta_repo_full` property with: + +```python + @property + def registry_repo_full(self) -> str: + return f"{self.gitea_org}/{self.registry_repo}" +``` + +In `load_config()`, remove the `meta_repo=_optional("META_REPO", "meta"),` line and the multi-line comment above `registry_repo=`, and set: + +```python + registry_repo=_required("REGISTRY_REPO"), + default_project_id=_optional("DEFAULT_PROJECT_ID"), +``` + +> NOTE: The migration-027 test (`test_migration_027.py`) and `test_registry.py` construct `Config(...)` directly with `meta_repo="meta"`. Remove the `meta_repo="meta",` argument from those two test helpers' `Config(...)` calls, since the field no longer exists. + +- [ ] **Step 4: projects.py — default-project helpers; retire seed_default_project** + +In `backend/app/projects.py`, delete the `seed_default_project` function and replace the module body below `DEFAULT_PROJECT_ID = "default"` with these helpers: + +```python +def resolved_default_id(config: Config) -> str: + """The id of the deployment's bootstrap/default project. Plan A: always + 'default' (the re-stamp to a config slug rides Plan B). The config knob is + read here so Plan B can flip the resolution without touching call sites.""" + return config.default_project_id.strip() or DEFAULT_PROJECT_ID + + +def default_content_repo(config: Config) -> str | None: + """The content repo the single-corpus mirror reads, from the default + project's row (filled by the registry mirror). Replaces the retired + META_REPO. None until the registry mirror has run.""" + row = db.conn().execute( + "SELECT content_repo FROM projects WHERE id = ?", + (resolved_default_id(config),), + ).fetchone() + return row["content_repo"] if row and row["content_repo"] else None + + +def project_initial_state(project_id: str) -> str: + """§22.4b landing state for new entries in a project. Defaults to + 'super-draft' for an unknown/unset row (the safe, today's-flow default).""" + row = db.conn().execute( + "SELECT initial_state FROM projects WHERE id = ?", (project_id,) + ).fetchone() + if row is None or not row["initial_state"]: + return "super-draft" + return row["initial_state"] +``` + +- [ ] **Step 5: cache.py — corpus mirror reads content_repo from the DB; sweep mirrors the registry** + +In `backend/app/cache.py`, add to the imports at the top (the `from . import` line): + +```python +from . import db, entry as entry_mod, projects as projects_mod, registry as registry_mod +``` + +In `refresh_meta_repo`, replace `org, repo = config.gitea_org, config.meta_repo` with: + +```python + org = config.gitea_org + repo = projects_mod.default_content_repo(config) + if not repo: + log.warning("refresh_meta_repo: default project has no content_repo yet; skipping") + return +``` + +Apply the **same** two-line repo-resolution change at the top of `refresh_meta_branches` and `refresh_meta_pulls` (search for `config.meta_repo` in those functions and replace each occurrence; each should resolve `repo = projects_mod.default_content_repo(config)` and early-return if falsy). Run `grep -n "config.meta_repo" backend/app/cache.py` and confirm zero remain. + +In `Reconciler.sweep`, add the registry refresh as the first step inside the `try:` (before `refresh_meta_repo`), so the projects table is current before the corpus mirror reads `content_repo`: + +```python + try: + try: + await registry_mod.refresh_registry(self._config, self._gitea) + except Exception: + # Running deployment: a malformed registry PR must not take the + # sweep (or the deployment) down — keep the last-good rows. + log.exception("reconciler: registry refresh failed; keeping last-good projects") + await refresh_meta_repo(self._config, self._gitea) + ... +``` + +- [ ] **Step 6: webhooks.py — resolve repo from DB; add the registry branch** + +In `backend/app/webhooks.py`, change the import line `from . import cache, db` to: + +```python +from . import cache, db, projects as projects_mod, registry as registry_mod +``` + +Replace the dispatch block (the `repo_full = ...` through the `else:` slug branch) with: + +```python + repo_full = (payload.get("repository") or {}).get("full_name") or "" + registry_full = f"{config.gitea_org}/{config.registry_repo}" + meta_repo = projects_mod.default_content_repo(config) + meta_full = f"{config.gitea_org}/{meta_repo}" if meta_repo else None + try: + if repo_full == registry_full: + # §22.2: a registry-repo push re-mirrors the projects table. + try: + await registry_mod.refresh_registry(config, gitea) + except registry_mod.RegistryError: + # Tolerate a malformed registry PR on a running deployment. + log.exception("registry webhook: invalid projects.yaml; keeping last-good") + elif meta_full and (repo_full == meta_full or not repo_full): + await cache.refresh_meta_repo(config, gitea) + await cache.refresh_meta_branches(config, gitea) + await cache.refresh_meta_pulls(config, gitea) + else: + slug = _slug_for_repo(repo_full) + if slug: + await cache.refresh_rfc_repo(config, gitea, slug) + else: + log.info( + "webhook received for unknown repo: repo_full=%s event=%s " + "(no cached_rfcs row matched; hook may be on a fork or stale)", + repo_full, event, + ) + except Exception: + log.exception("webhook refresh failed") + raise HTTPException(status_code=500, detail="Refresh failed") +``` + +- [ ] **Step 7: main.py — drop seed_default_project; mirror the registry at startup (fatal on first boot)** + +In `backend/app/main.py` lifespan (`async def lifespan`), import `registry`: add `registry as registry_mod,` to the `from . import (...)` block. Then replace: + +```python + db.init(config) + projects.seed_default_project(config) # §22.13 step 1 + gitea = Gitea(config) +``` + +with: + +```python + db.init(config) + gitea = Gitea(config) + # §22.2: mirror the registry before anything reads projects/content_repo. + # First boot has no last-good rows, so a missing/invalid registry is fatal + # (loud-fail per separation-of-concerns); the reconciler sweep keeps it + # fresh thereafter and tolerates a later bad PR. + try: + await registry_mod.refresh_registry(config, gitea) + except Exception as e: + raise RuntimeError( + f"registry mirror failed at startup ({config.registry_repo_full}/projects.yaml): {e}" + ) from e + if projects.default_content_repo(config) is None: + raise RuntimeError( + f"registry does not describe the default project " + f"{projects.resolved_default_id(config)!r} (no content_repo). " + f"Add it to {config.registry_repo_full}/projects.yaml." + ) +``` + +Also update the startup log line near the end of lifespan — replace `config.gitea_org, config.meta_repo` with: + +```python + log.info("RFC app started — registry %s", config.registry_repo_full) +``` + +- [ ] **Step 8: Update the shared test fixture to seed a registry** + +In `backend/tests/test_propose_vertical.py`: + +In `FakeGitea.__init__`, after `self._seed_repo("wiggleverse", "meta")`, add: + +```python + # §22 M3: the deployment's project registry. Startup refresh_registry + # reads projects.yaml here; the single 'default' project's content_repo + # points back at the seeded meta repo so the corpus mirror is unchanged. + self._seed_repo("wiggleverse", "registry") + self.files[("wiggleverse", "registry", "main", "projects.yaml")] = { + "content": ( + "deployment:\n" + " name: Test Deployment\n" + " tagline: A test deployment\n" + "projects:\n" + " - id: default\n" + " name: Test Deployment\n" + " type: document\n" + " content_repo: meta\n" + " visibility: public\n" + ), + "sha": "regsha0001", + } +``` + +In the `tmp_env` fixture's `env` dict, add a registry repo name and drop the now-ignored META_REPO line. Change: + +```python + "META_REPO": "meta", +``` + +to: + +```python + "REGISTRY_REPO": "registry", +``` + +- [ ] **Step 9: Fix the one stale assertion comment** + +In `backend/tests/test_multi_project_spine_vertical.py`, the `test_default_project_seeded_and_backfilled` test still passes (content_repo is `"meta"`), but its inline comment references the retired META_REPO backfill. Update the comment on that assertion: + +```python + # M3: content_repo now comes from the registry mirror (projects.yaml), + # not the retired META_REPO startup backfill. + assert row["content_repo"] == "meta" +``` + +- [ ] **Step 10: Run the wiring test, then the full suite** + +Run: `cd backend && python -m pytest tests/test_registry_wiring.py -v` +Expected: PASS (2 tests). + +Run: `cd backend && grep -rn "config.meta_repo\|meta_repo_full\|seed_default_project" app/` +Expected: no output (all references removed). + +Run: `cd backend && python -m pytest -q` +Expected: PASS (full suite green — the fixture now seeds the registry, so every app-startup test mirrors it and the corpus mirror reads `content_repo='meta'` exactly as before). + +- [ ] **Step 11: Commit** + +```bash +git add backend/app/config.py backend/app/projects.py backend/app/cache.py \ + backend/app/webhooks.py backend/app/main.py backend/tests/test_propose_vertical.py \ + backend/tests/test_multi_project_spine_vertical.py backend/tests/test_registry_wiring.py \ + backend/tests/test_migration_027.py backend/tests/test_registry.py +git commit -m "feat(registry): hard-cut META_REPO -> REGISTRY_REPO; wire mirror into startup/webhook/sweep" +``` + +--- + +## Task 6: Runtime-config APIs — `GET /api/deployment` + `GET /api/projects/:id` + +**Files:** +- Create: `backend/app/api_deployment.py` +- Modify: `backend/app/api.py` (imports ~21-44; mount in `make_router` ~118-148) +- Test: `backend/tests/test_api_deployment.py` + +- [ ] **Step 1: Write the failing test** + +```python +# backend/tests/test_api_deployment.py +"""§22.9/§22.5 — GET /api/deployment + GET /api/projects/:id with visibility.""" +from __future__ import annotations + +from fastapi.testclient import TestClient + +from test_propose_vertical import ( # noqa: F401 + app_with_fake_gitea, tmp_env, provision_user_row, sign_in_as, +) + + +def _add_project(pid, name, vis, typ="document"): + from app import db + db.conn().execute( + "INSERT OR REPLACE INTO projects (id, name, type, content_repo, visibility, initial_state) " + "VALUES (?, ?, ?, ?, ?, 'super-draft')", + (pid, name, typ, pid, vis), + ) + + +def test_deployment_lists_public_omits_gated_and_unlisted_for_anon(app_with_fake_gitea): + app, _ = app_with_fake_gitea + with TestClient(app) as client: + _add_project("pub", "Public", "public") + _add_project("gat", "Gated", "gated") + _add_project("unl", "Unlisted", "unlisted") + r = client.get("/api/deployment") + assert r.status_code == 200 + body = r.json() + assert body["name"] == "Test Deployment" + ids = {p["id"] for p in body["projects"]} + assert "pub" in ids and "default" in ids # both public + assert "gat" not in ids # gated, anon not a member + assert "unl" not in ids # unlisted never enumerated + + +def test_projects_id_404_for_gated_non_member(app_with_fake_gitea): + app, _ = app_with_fake_gitea + with TestClient(app) as client: + _add_project("gat", "Gated", "gated") + assert client.get("/api/projects/gat").status_code == 404 + + +def test_projects_id_returns_config_for_public(app_with_fake_gitea): + app, _ = app_with_fake_gitea + with TestClient(app) as client: + from app import db + db.conn().execute( + "UPDATE projects SET config_json = ? WHERE id = 'default'", + ('{"theme": {"accent": "#5b5bd6"}}',), + ) + r = client.get("/api/projects/default") + assert r.status_code == 200 + body = r.json() + assert body["id"] == "default" + assert body["type"] == "document" + assert body["visibility"] == "public" + assert body["theme"] == {"accent": "#5b5bd6"} + + +def test_projects_id_unlisted_readable_by_direct_id(app_with_fake_gitea): + app, _ = app_with_fake_gitea + with TestClient(app) as client: + _add_project("unl", "Unlisted", "unlisted") + assert client.get("/api/projects/unl").status_code == 200 +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend && python -m pytest tests/test_api_deployment.py -v` +Expected: FAIL — 404 for `/api/deployment` (route not mounted). + +- [ ] **Step 3: Write `app/api_deployment.py`** + +```python +# backend/app/api_deployment.py +"""§22.9 runtime deployment/project config (replaces VITE_APP_NAME). + +GET /api/deployment — the deployment name/tagline + the projects the caller can +see (§22.5: gated filtered by membership, unlisted omitted from enumeration). +GET /api/projects/:id — one project's runtime config + optional theme overlay, +gated behind the §22.5 read gate (404 for a non-member of a gated project). +""" +from __future__ import annotations + +import json +from typing import Any + +from fastapi import APIRouter, Request + +from . import auth, db + + +def make_router() -> APIRouter: + router = APIRouter() + + @router.get("/api/deployment") + async def get_deployment(request: Request) -> dict[str, Any]: + viewer = auth.current_user(request) + dep = db.conn().execute( + "SELECT name, tagline FROM deployment WHERE id = 1" + ).fetchone() + # §22.5: enumerate only public + (member-)gated; unlisted is never listed. + visible = set(auth.visible_project_ids(viewer)) + rows = db.conn().execute( + "SELECT id, name, type, visibility FROM projects " + "WHERE visibility != 'unlisted' ORDER BY name" + ).fetchall() + projects = [ + {"id": r["id"], "name": r["name"], "type": r["type"], "visibility": r["visibility"]} + for r in rows + if r["id"] in visible + ] + return { + "name": (dep["name"] if dep else "") or "", + "tagline": (dep["tagline"] if dep else "") or "", + "projects": projects, + } + + @router.get("/api/projects/{project_id}") + async def get_project(project_id: str, request: Request) -> dict[str, Any]: + viewer = auth.current_user(request) + # §22.5 read gate: a gated project 404s to a non-member (shape matches + # an unknown id). unlisted is readable by direct id. + auth.require_project_readable(viewer, project_id) + row = db.conn().execute( + "SELECT id, name, type, visibility, initial_state, config_json " + "FROM projects WHERE id = ?", + (project_id,), + ).fetchone() + if row is None: + auth.require_project_readable(viewer, project_id) # raises 404 + cfg = json.loads(row["config_json"] or "{}") + dep = db.conn().execute("SELECT tagline FROM deployment WHERE id = 1").fetchone() + return { + "id": row["id"], + "name": row["name"], + "tagline": (dep["tagline"] if dep else "") or "", + "type": row["type"], + "visibility": row["visibility"], + "initial_state": row["initial_state"], + "theme": cfg.get("theme") or {}, + } + + return router +``` + +> NOTE: `require_project_readable` (auth.py) returns `gated`→404 only when the row is missing OR the project is gated-and-not-member. For a truly unknown id, `project_visibility` returns `'gated'`, so a non-member gets 404 — the `row is None` guard then never reaches a real row. The redundant second `require_project_readable` call documents intent and is harmless. + +- [ ] **Step 4: Mount the router in `api.py`** + +In `backend/app/api.py`, add `api_deployment,` to the `from . import (...)` block (alphabetical, near `api_contributions`). Then in `make_router`, after the other `router.include_router(...)` calls (after `api_contributions`), add: + +```python + # §22.9 (M3): runtime deployment + per-project config (replaces VITE_APP_NAME). + router.include_router(api_deployment.make_router()) +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `cd backend && python -m pytest tests/test_api_deployment.py -v` +Expected: PASS (4 tests). + +- [ ] **Step 6: Run the full suite** + +Run: `cd backend && python -m pytest -q` +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add backend/app/api_deployment.py backend/app/api.py backend/tests/test_api_deployment.py +git commit -m "feat(api): GET /api/deployment + /api/projects/:id (§22.9 runtime config)" +``` + +--- + +## Task 7: Honor `initial_state` at propose time + +**Files:** +- Modify: `backend/app/api.py` (the `propose_rfc` endpoint ~797-845) +- Test: `backend/tests/test_initial_state_landing.py` + +> SCOPE NOTE: Plan A sets the landed entry's `state`/`unreviewed` in frontmatter + cache per `initial_state`. The full active-entry lifecycle (skipping repo-creation/graduation for an active-landing project) is a `bdd`/type-surface concern deferred to M5; for `document` projects (the only live kind) `initial_state` is `super-draft`, so nothing changes in production. + +- [ ] **Step 1: Write the failing test** + +```python +# backend/tests/test_initial_state_landing.py +"""§22.4b — a project with initial_state='active' lands new entries active + +unreviewed; the default 'super-draft' project is unchanged.""" +from __future__ import annotations + +from fastapi.testclient import TestClient + +from test_propose_vertical import ( # noqa: F401 + app_with_fake_gitea, tmp_env, provision_user_row, sign_in_as, +) + + +def _propose(client): + return client.post("/api/rfcs/propose", json={ + "title": "Active Lander", "slug": "active-lander", + "pitch": "Lands active.", "tags": [], + }) + + +def test_super_draft_default_unchanged(app_with_fake_gitea): + from app import entry as entry_mod + app, fake = app_with_fake_gitea + with TestClient(app) as client: + provision_user_row(user_id=1, login="ben", role="owner") + sign_in_as(client, user_id=1, gitea_login="ben", display_name="Ben", role="owner") + assert _propose(client).status_code == 200 + f = fake.files[("wiggleverse", "meta", "propose/active-lander", "rfcs/active-lander.md")] + e = entry_mod.parse(f["content"]) + assert e.state == "super-draft" + assert e.unreviewed is False + + +def test_active_initial_state_lands_active_unreviewed(app_with_fake_gitea): + from app import db, entry as entry_mod + app, fake = app_with_fake_gitea + with TestClient(app) as client: + db.conn().execute("UPDATE projects SET initial_state='active' WHERE id='default'") + provision_user_row(user_id=1, login="ben", role="owner") + sign_in_as(client, user_id=1, gitea_login="ben", display_name="Ben", role="owner") + assert _propose(client).status_code == 200 + f = fake.files[("wiggleverse", "meta", "propose/active-lander", "rfcs/active-lander.md")] + e = entry_mod.parse(f["content"]) + assert e.state == "active" + assert e.unreviewed is True +``` + +> NOTE: the branch name `propose/active-lander` matches what `bot.open_idea_pr` creates (confirm the exact prefix by reading `bot.open_idea_pr`; if it differs, adjust the file-key in the test). The assertion target is the entry file the bot wrote to the proposal branch. + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend && python -m pytest tests/test_initial_state_landing.py -v` +Expected: FAIL — the active case lands `super-draft` (the endpoint hardcodes it). + +- [ ] **Step 3: Consult `initial_state` in `propose_rfc`** + +In `backend/app/api.py`, ensure `projects` is imported (add `projects as projects_mod,` to the `from . import (...)` block if absent). In `propose_rfc`, replace the `entry = entry_mod.Entry(` construction's `state="super-draft",` line. First, just before building the `entry`, add: + +```python + # §22.4b: the target project's landing state. Through Plan A every + # entry lands in the default project; M3-frontend routing carries a + # non-default target later. + target_project = auth.DEFAULT_PROJECT_ID + landing_state = "active" if projects_mod.project_initial_state(target_project) == "active" else "super-draft" +``` + +Then change the `Entry(...)` constructor: replace `state="super-draft",` with: + +```python + state=landing_state, + unreviewed=(landing_state == "active"), +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd backend && python -m pytest tests/test_initial_state_landing.py -v` +Expected: PASS (2 tests). + +- [ ] **Step 5: Run the full suite** + +Run: `cd backend && python -m pytest -q` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add backend/app/api.py backend/tests/test_initial_state_landing.py +git commit -m "feat(propose): honor project initial_state (§22.4b active landing + unreviewed)" +``` + +--- + +## Task 8: mark-reviewed action + catalog `unreviewed` filter + +**Files:** +- Modify: `backend/app/bot.py` (add a `mark_reviewed` write wrapper modeled on the graduation file-write) +- Modify: `backend/app/api.py` (the `list_rfcs` catalog endpoint ~609-665; add the mark-reviewed endpoint near the other RFC endpoints) +- Test: `backend/tests/test_mark_reviewed.py` + +> Before writing Step 3, read `backend/app/bot.py`'s `open_graduation_pr` (≈700-767) and `backend/app/api_graduation.py`'s `_orchestrate` (≈633-726) to copy the exact bot-write + inline-merge pattern (read file sha → update_file on a branch → open PR → wait_for_mergeable → merge → refresh cache). The mark-reviewed flow mirrors it: it rewrites the entry frontmatter with `unreviewed: false` + `reviewed_at`/`reviewed_by`, commits via the bot, merges, and refreshes `cached_rfcs`. + +- [ ] **Step 1: Write the failing test** + +```python +# backend/tests/test_mark_reviewed.py +"""§22.4c — owner/admin mark-reviewed clears the flag; catalog unreviewed filter.""" +from __future__ import annotations + +from fastapi.testclient import TestClient + +from test_propose_vertical import ( # noqa: F401 + app_with_fake_gitea, tmp_env, provision_user_row, sign_in_as, +) + + +def _seed_unreviewed_active(fake, slug="feat"): + """Put an active+unreviewed entry on the meta repo main + cache.""" + from app import cache, entry as entry_mod + body = entry_mod.serialize(entry_mod.Entry( + slug=slug, title="Feat", state="active", unreviewed=True, + owners=["ben"], proposed_by="ben", + )) + fake.files[("wiggleverse", "meta", "main", f"rfcs/{slug}.md")] = {"content": body, "sha": "s1"} + cache._upsert_cached_rfc(entry_mod.parse(body), body_sha="s1") + + +def test_catalog_unreviewed_filter(app_with_fake_gitea): + app, fake = app_with_fake_gitea + with TestClient(app) as client: + _seed_unreviewed_active(fake, "feat") + # a reviewed active entry should NOT appear under the filter + from app import cache, entry as entry_mod + ok = entry_mod.serialize(entry_mod.Entry(slug="ok", title="OK", state="active", owners=["ben"])) + fake.files[("wiggleverse", "meta", "main", "rfcs/ok.md")] = {"content": ok, "sha": "s2"} + cache._upsert_cached_rfc(entry_mod.parse(ok), body_sha="s2") + r = client.get("/api/rfcs", params={"unreviewed": "true"}) + slugs = {i["slug"] for i in r.json()["items"]} + assert slugs == {"feat"} + + +def test_mark_reviewed_clears_flag(app_with_fake_gitea): + from app import db + app, fake = app_with_fake_gitea + with TestClient(app) as client: + _seed_unreviewed_active(fake, "feat") + provision_user_row(user_id=1, login="ben", role="owner") + sign_in_as(client, user_id=1, gitea_login="ben", display_name="Ben", role="owner") + r = client.post("/api/projects/default/rfcs/feat/mark-reviewed") + assert r.status_code == 200 + row = db.conn().execute( + "SELECT unreviewed, reviewed_by FROM cached_rfcs WHERE slug='feat'" + ).fetchone() + assert row["unreviewed"] == 0 + assert row["reviewed_by"] == "ben" + + +def test_mark_reviewed_forbidden_for_non_superuser(app_with_fake_gitea): + app, fake = app_with_fake_gitea + with TestClient(app) as client: + _seed_unreviewed_active(fake, "feat") + provision_user_row(user_id=2, login="carol", role="contributor") + sign_in_as(client, user_id=2, gitea_login="carol", display_name="Carol", role="contributor") + r = client.post("/api/projects/default/rfcs/feat/mark-reviewed") + assert r.status_code == 403 +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd backend && python -m pytest tests/test_mark_reviewed.py -v` +Expected: FAIL — `unreviewed` query param ignored (filter returns both); mark-reviewed route 404. + +- [ ] **Step 3: Add the catalog filter** + +In `backend/app/api.py` `list_rfcs`, change the signature to accept the query param and extend the WHERE clause. Replace the function signature line and the SQL: + +```python + @router.get("/api/rfcs") + async def list_rfcs(request: Request, unreviewed: str | None = None) -> dict[str, Any]: +``` + +and build the query with an optional `unreviewed` predicate: + +```python + placeholders = ",".join("?" for _ in visible) + params = list(visible) + unreviewed_clause = "" + if unreviewed is not None and unreviewed.lower() in ("1", "true", "yes"): + unreviewed_clause = " AND unreviewed = 1 AND state = 'active'" + rows = db.conn().execute( + f""" + SELECT slug, title, state, rfc_id, repo, + owners_json, arbiters_json, tags_json, + last_main_commit_at, last_entry_commit_at, updated_at + FROM cached_rfcs + WHERE state IN ('super-draft', 'active') + AND project_id IN ({placeholders}){unreviewed_clause} + ORDER BY COALESCE(last_main_commit_at, last_entry_commit_at) DESC + """, + params, + ).fetchall() +``` + +- [ ] **Step 4: Add the mark-reviewed endpoint** + +In `backend/app/api.py`, near `get_rfc` (~667), add (it uses the bot wrapper from Step 5): + +```python + @router.post("/api/projects/{project_id}/rfcs/{slug}/mark-reviewed") + async def mark_reviewed(project_id: str, slug: str, request: Request) -> dict[str, Any]: + """§22.4c — clear an active entry's `unreviewed` flag. Authority is the + §22.7 project superuser (project_admin or deployment owner/admin), the + same tier that graduates an entry.""" + viewer = auth.require_user(request) + auth.require_project_readable(viewer, project_id) + if not auth.is_project_superuser(viewer, project_id): + raise HTTPException(403, "Only a project owner/admin can mark an entry reviewed") + row = db.conn().execute( + "SELECT state, unreviewed FROM cached_rfcs WHERE slug = ? AND project_id = ?", + (slug, project_id), + ).fetchone() + if row is None: + raise HTTPException(404, "Not found") + if row["state"] != "active" or not row["unreviewed"]: + raise HTTPException(409, "Entry is not an unreviewed active entry") + try: + await bot.mark_entry_reviewed( + viewer.as_actor(), + org=config.gitea_org, + meta_repo=projects_mod.default_content_repo(config), + slug=slug, + reviewed_by=viewer.gitea_login, + reviewed_at=entry_mod.today(), + ) + except GiteaError as e: + raise HTTPException(502, f"Gitea: {e.detail}") + await cache.refresh_meta_repo(config, gitea) + return {"ok": True} +``` + +- [ ] **Step 5: Add the `bot.mark_entry_reviewed` wrapper** + +In `backend/app/bot.py`, add a method modeled on `open_graduation_pr` but writing directly to `main` via the same authenticated `update_file` path (mark-reviewed is an owner gesture and needs no review PR — it parallels the graduation *stamp*, not the graduation *flow*). Read the existing bot write helpers first to match the On-behalf-of trailer + actions-log shape. The method: + +```python + async def mark_entry_reviewed( + self, + actor, + *, + org: str, + meta_repo: str, + slug: str, + reviewed_by: str, + reviewed_at: str, + ) -> None: + """Clear §22.4c unreviewed on an active entry by rewriting its + frontmatter on main. Mirrors the graduation stamp's bot-write shape.""" + from . import entry as entry_mod + path = f"rfcs/{slug}.md" + result = await self._gitea.read_file(org, meta_repo, path, ref="main") + if result is None: + raise GiteaError(404, f"{path} not found") + text, sha = result + e = entry_mod.parse(text) + e.unreviewed = False + e.reviewed_at = reviewed_at + e.reviewed_by = reviewed_by + await self._update_file_on_behalf( + actor, + org=org, repo=meta_repo, path=path, + content=entry_mod.serialize(e), sha=sha, branch="main", + message=f"Mark {slug} reviewed", + ) +``` + +> NOTE: match the real helper names in `bot.py` — `self._gitea`, and the existing on-behalf write helper (it may be named differently than `_update_file_on_behalf`). Read `bot.py` first and use the actual private write method + actions-log call the other bot methods use, so mark-reviewed carries the same §6.5 trailer + actions row. + +- [ ] **Step 6: Run tests to verify they pass** + +Run: `cd backend && python -m pytest tests/test_mark_reviewed.py -v` +Expected: PASS (3 tests). + +- [ ] **Step 7: Run the full suite** + +Run: `cd backend && python -m pytest -q` +Expected: PASS. + +- [ ] **Step 8: Commit** + +```bash +git add backend/app/api.py backend/app/bot.py backend/tests/test_mark_reviewed.py +git commit -m "feat(review): mark-reviewed action + §7 catalog unreviewed filter (§22.4c)" +``` + +--- + +## Task 9: Release — env docs, changelog, version bump + +**Files:** +- Modify: `backend/.env.example` +- Modify: `CHANGELOG.md` +- Modify: `VERSION`, `frontend/package.json` + +- [ ] **Step 1: Update `backend/.env.example`** + +Add `REGISTRY_REPO` (required) and `DEFAULT_PROJECT_ID` (optional), and remove `META_REPO`. Insert near the other Gitea settings: + +```bash +# §22.2 — the project registry repo (REQUIRED). The framework reads +# `projects.yaml` at its root to learn which projects exist. The repo name is +# the deployment's choice; the app fails to start if this is unset. +REGISTRY_REPO=registry +# §22.13 — optional id for the bootstrap/default project. Reserved for the +# Plan B re-stamp; leave unset in Plan A (the default project id stays +# `default`). When set, it must match an `id` in projects.yaml. +# DEFAULT_PROJECT_ID=ohm +``` + +Delete the `META_REPO=...` line and its comment. + +- [ ] **Step 2: Run a grep to confirm no doc still tells operators to set META_REPO** + +Run: `cd backend && grep -rn "META_REPO" .env.example app/ ; echo "exit: $?"` +Expected: no matches in `.env.example` or `app/` (a match in a migration-026 comment is fine — it's historical). + +- [ ] **Step 3: Add the CHANGELOG entry with upgrade steps** + +In `CHANGELOG.md`, add a new top entry for `0.33.0` (a breaking, pre-1.0 minor). Match the file's existing entry format; the body must include an upgrade-steps block: + +```markdown +## 0.33.0 — §22 M3 (backend, Plan A): registry mirror + runtime config + +### Added +- Project **registry mirror** (`app/registry.py`): the framework now learns its + projects from `REGISTRY_REPO/projects.yaml`, mirrored into the `projects` + table + a new `deployment` singleton on the §4 webhook + reconciler. +- `GET /api/deployment` (name, tagline, visible projects) and + `GET /api/projects/:id` (name, type, visibility, initial_state, theme) — + runtime config replacing the build-time `VITE_APP_NAME` (frontend cut lands in + M3-frontend). +- `§22.4b initial_state` honored at propose time; `§22.4c unreviewed` flag, + mark-reviewed action (`POST /api/projects/:id/rfcs/:slug/mark-reviewed`), and a + `?unreviewed=true` catalog filter. +- Migration `027` (additive): `projects.type`/`initial_state`, the `deployment` + table, and `cached_rfcs.unreviewed`/`reviewed_at`/`reviewed_by`. + +### Breaking +- **`META_REPO` is retired; `REGISTRY_REPO` is required.** The corpus mirror now + reads each project's `content_repo` from the registry. + +### Upgrade steps +1. Create a registry repo under your deployment's Gitea org. +2. Author `projects.yaml` at its root: a `deployment:` block (`name`, `tagline`) + and one `projects:` entry for your existing corpus — `id: default`, `name`, + `type: document`, `content_repo: `, + `visibility: public`. (Keep `id: default` for this release; the pretty-slug + re-stamp lands in the next backend slice, before any `/p/` URL is public.) +3. Set `REGISTRY_REPO=` and remove `META_REPO`. +4. Add a Gitea webhook on the registry repo pointing at `/api/webhooks/gitea` + (same secret as the corpus webhook). +5. Deploy. Migration `027` runs; the registry mirror reconciles at startup. + Verify `GET /api/deployment` returns your project and `/api/health` is green. +``` + +- [ ] **Step 4: Bump the version** + +Set the version to `0.33.0` in both files (they must match per SPEC §20): + +```bash +printf '0.33.0\n' > VERSION +``` + +In `frontend/package.json`, change `"version": "0.32.0",` to `"version": "0.33.0",`. + +- [ ] **Step 5: Verify version match + full suite** + +Run: `cat VERSION && grep '"version"' frontend/package.json` +Expected: both show `0.33.0`. + +Run: `cd backend && python -m pytest -q` +Expected: PASS (full suite). + +- [ ] **Step 6: Commit** + +```bash +git add backend/.env.example CHANGELOG.md VERSION frontend/package.json +git commit -m "release: 0.33.0 — §22 M3 backend Plan A (registry mirror + runtime config)" +``` + +--- + +## Self-review checklist (completed by plan author) + +**Spec coverage (Plan A scope only):** +- Registry format + `registry.py` mirror (spec §2) → Task 4 + Task 5. +- `GET /api/deployment` + `/api/projects/:id` (spec §4) → Task 6. +- `initial_state` honoring (spec §5c) → Task 7. +- `unreviewed` parse/serialize (spec §5a) → Task 1; cache mirror (§5b) → Task 3; mark-reviewed (§5d) → Task 8; catalog filter query side (§5e) → Task 8. +- Migration `027` additive columns + deployment table (spec §1a/§1b) → Task 2. +- Config hard cut, META_REPO retirement (spec §3) → Task 5. +- Versioning + upgrade steps (spec §8) → Task 9. +- **Deferred to Plan B (per spec §1 decision 6):** the 12-table PK rebuilds (§1c), the `default`→slug re-stamp (§1d), `project_id` writer threading, two-project isolation tests. Not in this plan by design. + +**Placeholder scan:** No "TBD"/"add error handling"/"similar to" — every code step shows the code. Two explicit "read X first to match the real helper name" notes (Task 8 bot wrapper, Task 7 branch-name) are deliberate: they point at existing code the worker must mirror exactly rather than guessing a signature. + +**Type/name consistency:** `Entry.unreviewed/reviewed_at/reviewed_by` (Task 1) are read by `_upsert_cached_rfc` (Task 3), the registry's `ProjectEntry`/`RegistryDoc`/`RegistryError`/`parse_registry`/`apply_registry`/`refresh_registry` (Task 4) are consumed by the wiring (Task 5), `projects.default_content_repo`/`resolved_default_id`/`project_initial_state` (Task 5) are used by cache/webhooks/propose (Tasks 5/7) and mark-reviewed (Task 8). Names are consistent across tasks. diff --git a/docs/superpowers/specs/2026-06-03-m3-backend-design.md b/docs/superpowers/specs/2026-06-03-m3-backend-design.md new file mode 100644 index 0000000..c078c80 --- /dev/null +++ b/docs/superpowers/specs/2026-06-03-m3-backend-design.md @@ -0,0 +1,413 @@ +# M3-backend — §22 multi-project: registry mirror + data spine + APIs + +> Design spec for the backend half of §22 slice **M3** ("Registry mirror + +> routing + runtime branding"). The roadmap bundles M3 as one slice; this +> session splits it at the natural backend/frontend seam. **M3-backend** (this +> doc) ships the data spine, the registry mirror, the two runtime-config APIs, +> and the entry-state/review semantics. **M3-frontend** (a separate spec) ships +> `/p//` routing, the 308 redirects, the `VITE_APP_NAME`→runtime-config +> cut, the per-project theme overlay, the deployment directory at `/`, and the +> project switcher — all consuming the APIs defined here. +> +> Section references `§22.x` point at `docs/design/multi-project-spec.md` (the +> draft §22 + slicing plan). The SPEC.md §22 merge itself lands in M7. + +## Status + +- **Date:** 2026-06-03 +- **Slice:** §22 M3 (backend half). M1 + M2 landed and merged to `main`. +- **Version impact:** minor bump, breaking (pre-1.0) — see §8. + +## Goal + +After M3-backend, the framework learns its projects from a git **registry** +(not from `META_REPO`), the `projects` cache table and all slug-bearing tables +are keyed by `(project_id, …)` so a second project can exist without collision, +the default project's identity is re-stamped to its real slug while no `/p/` +URL is yet public, and the runtime exposes deployment + project config over two +new endpoints. The entry-state/review semantics (`initial_state`, `unreviewed`) +ship complete even though OHM (a `document`/`super-draft` project) does not yet +exercise them. + +Non-goals (M3-frontend, later): `/p//` routing, 308 redirects off +`/rfc/`, runtime branding in the UI, theme application, the deployment +directory, the project switcher, the catalog's unreviewed-filter **UI**. + +## Decisions taken in brainstorming + +1. **Scope split** — backend spine first; frontend surfaces are a separate + spec/session. +2. **Review machinery** — build the full `initial_state` / `unreviewed` + plumbing now (parse + columns + mirror + landing logic + mark-reviewed + + catalog filter query side), per the spec's M3 bundle, even though no live + project exercises it yet. +3. **Config cut** — hard cut. `REGISTRY_REPO` required (loud fail if unset), + `META_REPO` removed. Upgrade-steps document the manual registry creation. +4. **Re-stamp** — rewrite `project_id` everywhere: rename `projects.id` + `default` → the config slug and rewrite every child row, folded into the + same create-copy-drop-rename rebuild that adds the `project_id` FK. One + identifier; DB and URL agree. +5. **Mirror structure** — a self-contained `app/registry.py` module (not folded + into `cache.py`), driven by the existing webhook dispatcher + the existing + `Reconciler.sweep()`. +6. **Two execution plans (found during planning).** The 12-table PK rebuild is + not self-contained: folding `project_id` into keys + FK forces every + `ON CONFLICT` upsert target to gain `project_id` (~10 sites across 6 modules) + and — because the rebuilt tables can no longer default `project_id` to a live + value once the default is re-stamped — forces **every RFC/branch writer** to + be threaded to supply the real `project_id`. That activation is larger and + riskier than the rest of M3-backend combined, and it is only required *before + a second project can collide* (i.e. right before M4). So M3-backend is split + into two plans at that seam: + - **Plan A (ships first):** registry mirror + the two APIs + `initial_state`/ + `unreviewed` semantics. Migration `027` is **additive only** (no rebuilds). + Operates entirely on the `default`-id project — no re-stamp, no rebuild, no + writer threading. `cached_rfcs` keeps its `slug` PK, so no upsert breakage. + - **Plan B (before M4 / before public `/p/` URLs):** the 12-table PK rebuild + (migration `028`), `project_id` threading through every writer, the + `default`→slug **re-stamp** (which rides here because it is only correct + once the rebuild's column-default fix + threading land), and two-project + isolation tests. + + The §1 sections below describe the **full** backend (both plans); §1c (the + rebuilds) and §1d (the re-stamp) are **Plan B**. Everything else is Plan A. + +--- + +## 1. Migration `027_projects_activate.sql` + +Runs while `default` is still the sole project and no `/p/` URL exists — the +safe window for an identity rewrite. One transaction (SQLite DDL is +transactional): the deployment either fully advances to `027` or stays on `026`. + +`DEFAULT_PROJECT_ID` is read at migration time, so it **must be set before the +upgrade deploy** (documented in §8). The slug it names is referred to below as +``; absent the env var, `` stays `default`. + +### 1a. Additive columns + +- `projects`: + - `type TEXT NOT NULL DEFAULT 'document' CHECK (type IN ('document','specification','bdd'))` + - `initial_state TEXT NOT NULL DEFAULT 'super-draft' CHECK (initial_state IN ('super-draft','active'))` +- `cached_rfcs`: + - `unreviewed INTEGER NOT NULL DEFAULT 0` + - `reviewed_at TEXT` + - `reviewed_by TEXT` + +### 1b. New `deployment` singleton table + +Holds deployment-level identity mirrored from the registry's `deployment:` +block (rather than overloading `projects`): + +```sql +CREATE TABLE IF NOT EXISTS deployment ( + id INTEGER PRIMARY KEY CHECK (id = 1), + name TEXT, + tagline TEXT, + registry_sha TEXT, + updated_at TEXT NOT NULL DEFAULT (datetime('now')) +); +INSERT OR IGNORE INTO deployment (id) VALUES (1); +``` + +> **Re-stamp split (correctness, found during planning).** The migration +> runner executes pure-SQL files and **cannot read `DEFAULT_PROJECT_ID` from the +> environment** — the same constraint that forced M1's `seed_default_project` +> into Python. So the re-stamp is **not** in the `.sql` file. Migration `027` +> (pure SQL) does §1a–§1c with `project_id` copied **verbatim** (`default` stays +> `default`); the rebuilt FKs are declared `ON UPDATE CASCADE ON DELETE +> CASCADE`. The re-stamp (§1d) is a **Python startup step** in `app/projects.py`, +> run after migrations and before the registry mirror, that issues a single +> `UPDATE projects SET id = ` (cascading to the 12 FK tables) plus a plain +> `UPDATE` of the 7 non-FK `project_id` tables. Idempotent: a no-op once no +> `default` row remains. + +### 1c. PK / uniqueness rebuilds (the 12 deferred tables) + +Per migration 026's deferred block, create-copy-drop-rename each table to fold +`project_id` into the key and add `project_id … REFERENCES projects(id) ON +UPDATE CASCADE ON DELETE CASCADE` (the `ON UPDATE CASCADE` is what lets the +§1d Python re-stamp move all child rows with one parent UPDATE): + +| Table | Key change | +| --- | --- | +| `cached_rfcs` | PK `(slug)` → `(project_id, slug)` | +| `cached_branches` | UNIQUE `(rfc_slug, branch_name)` → `+project_id` | +| `branch_visibility` | UNIQUE `(rfc_slug, branch_name)` → `+project_id` | +| `branch_contribute_grants` | UNIQUE `(rfc_slug, branch_name, grantee_user_id)` → `+project_id` | +| `stars` | UNIQUE `(user_id, rfc_slug)` → `+project_id` | +| `watches` | UNIQUE `(user_id, rfc_slug)` → `+project_id` | +| `pr_seen` | UNIQUE `(user_id, rfc_slug, pr_number)` → `+project_id` | +| `branch_chat_seen` | UNIQUE `(user_id, rfc_slug, branch_name)` → `+project_id` | +| `funder_consents` | PK `(user_id, rfc_slug)` → `+project_id` | +| `rfc_collaborators` | UNIQUE INDEX `(rfc_slug, user_id)` → `+project_id` | +| `contribution_requests` | UNIQUE INDEX `(rfc_slug, requester_user_id) WHERE pending` → `+project_id` | +| `proposed_use_cases` | UNIQUE `(scope, pr_number)` → `+project_id` | + +`cached_prs` is already globally unique (`repo` is the full `org/repo` string, +distinct per project) — **no rebuild**. + +The copy step writes `` in place of `default` for `project_id`, so these +12 tables are re-stamped for free (§1d). + +### 1d. Re-stamp `default` → `` (Python startup step) + +In `app/projects.py`, run at startup after `db.init` and before +`refresh_registry`. When `DEFAULT_PROJECT_ID` is set and a `default` project row +still exists, in one `db.tx()`: + +- `UPDATE projects SET id = , updated_at = datetime('now') WHERE id = + 'default'` — cascades `project_id` across the 12 FK tables via `ON UPDATE + CASCADE`. +- For the 7 `project_id`-bearing tables with **no** FK (`threads`, `changes`, + `notifications`, `actions`, `pr_resolution_branches`, `rfc_invitations`, + `cached_prs`): `UPDATE SET project_id = WHERE project_id = + 'default'`. + +End state: a single `project_id` value DB-wide. Idempotent — a no-op once no +`default` row remains, so it is safe on every boot. + +### 1e. Notes + +- Migration `016` is absent from the on-disk sequence (`015 → 017`); the runner + already tolerates the gap (the app runs today). **Do not renumber.** New file + is `027`. +- `PRAGMA foreign_keys` is honored going forward; the FK lands on the 12 rebuilt + tables. The other 7 keep app-layer integrity (matching today's posture for + non-rebuilt tables). + +--- + +## 2. Registry format + `app/registry.py` + +### 2a. `projects.yaml` (root of `REGISTRY_REPO`) + +```yaml +deployment: + name: Open Human Model # replaces VITE_APP_NAME (M3-frontend consumes) + tagline: ... +projects: + - id: ohm # url-stable slug, unique in the deployment + name: Open Human Model + type: document # document | specification | bdd — immutable + content_repo: ohm # repo under the deployment's Gitea org + visibility: public # gated | public | unlisted + initial_state: super-draft # optional; defaults from type + enabled_models: [claude, gemini] # optional; falls back to ENABLED_MODELS + theme: { accent: "#5b5bd6" } # optional; M3-frontend consumes +``` + +### 2b. `refresh_registry(config, gitea) -> RegistryResult` + +The config-side analogue of `cache.refresh_meta_repo`. Fetches `projects.yaml` +from `REGISTRY_REPO` at HEAD, parses, **validates**, and upserts: + +- Each project → `projects` row: `id, name, type, content_repo, visibility, + initial_state`, plus `config_json` (JSON blob for `theme`, `enabled_models`), + plus `registry_sha` (commit SHA, provenance). +- The `deployment:` block → the `deployment` singleton (`name`, `tagline`, + `registry_sha`). + +Projects present in the table but absent from the registry are **not** deleted +in M3-backend (archival semantics are out of scope; a removed entry simply +stops being refreshed). This is noted as a known limitation; revisit if/when +project archival is specced. + +### 2c. Validation (loud) + +Per the framework's separation-of-concerns rule, malformed config fails +visibly rather than shipping wrong content silently: + +- Each project requires `id`, `name`, `type`, `content_repo`. +- `type` ∈ {`document`,`specification`,`bdd`}; `visibility` ∈ + {`gated`,`public`,`unlisted`}. +- `initial_state` defaults from `type` when omitted: `document`/`specification` + → `super-draft`, `bdd` → `active`. When present it must be a valid §2.4 + super-draft entry-state value. +- `id` values unique and slug-shaped (`^[a-z0-9][a-z0-9-]*$`). +- **`type` is immutable:** an incoming `type` differing from the stored row's is + rejected (the entry is skipped, the rest proceed; logged loudly). +- **Default-id consistency:** the re-stamped default id (`DEFAULT_PROJECT_ID`, + else `default`) MUST appear as an `id` in the registry, or the registry is + inconsistent with config → surfaced loudly. + +### 2d. Wiring (Option A) + +- **Webhook** (`app/webhooks.py`): add a branch — if the pushed repo + `full_name` matches `REGISTRY_REPO`, call `registry.refresh_registry(...)`. + Same HMAC-verified `/api/webhooks/gitea` dispatcher; no new endpoint. Add + `REGISTRY_REPO` to the set of repos the dispatcher recognizes. +- **Sweep** (`cache.Reconciler.sweep()`): add one `await + registry.refresh_registry(...)` at the top of each pass, so the safety-net + loop keeps `projects` in sync if a webhook is missed. +- **Startup** (`main.py` lifespan): run `refresh_registry` once after + migrations. This **replaces** M1's `seed_default_project` (which is removed). + +### 2e. Failure posture + +- **Startup / first boot:** if `REGISTRY_REPO` is unset/unreachable, or + `projects.yaml` is missing or fails validation, the app **fails loudly** + (refuses to start) — there is no last-known-good to serve. +- **Running deployment:** a malformed `projects.yaml` pushed in a later PR is + logged and **skipped**, leaving the last-good `projects` rows intact — a bad + config PR must not take the deployment down. This mirrors how the corpus + reconciler tolerates a bad content push today. + +--- + +## 3. Config (`app/config.py`) + +- `registry_repo`: **required** — construction fails loudly if unset (matching + the other required vars). Add `registry_repo_full` → `{gitea_org}/{registry_repo}`. +- `meta_repo` and `meta_repo_full`: **removed**. +- `default_project_id`: optional; consumed by migration `027` (re-stamp) and by + `refresh_registry` (the §2c consistency gate). +- `enabled_models`: unchanged — the deployment-level fallback for a project's + optional `enabled_models`. +- `app/projects.py`: `seed_default_project` retired (superseded by the mirror). + `DEFAULT_PROJECT_ID` constant and resolution helpers retained as needed. + +--- + +## 4. APIs — `app/api_deployment.py` + +A new sub-router mounted in `app/api.py`. + +### `GET /api/deployment` + +Returns `{ name, tagline, projects: [...] }`. The deployment `name`/`tagline` +come from the `deployment` singleton. The project list is filtered by caller +visibility (§22.5): + +- `public` projects → visible to everyone (incl. anonymous). +- `gated` projects → only when the caller is a member (`visible_project_ids` + from M2's resolver). +- `unlisted` projects → **omitted entirely** (reachable only by direct id). + +Each item: `{ id, name, type, visibility }` — enough for the M3-frontend +directory + switcher. (Theme is fetched per project.) + +### `GET /api/projects/:id` + +Returns `{ id, name, tagline, type, visibility, initial_state, theme }`. +Guarded by `require_project_readable(user, id)` — 404 for a non-member of a +gated project, reusing the M2 resolver. `unlisted` is readable here by direct +id (it is hidden only from enumeration). + +--- + +## 5. Entry-state & review semantics + +### 5a. Frontmatter (`app/entry.py`) + +Parse three new fields, lenient (default `unreviewed=false`, nulls): +`unreviewed: bool`, `reviewed_at`, `reviewed_by`. Add to the `Entry` +dataclass. These are git-truth (§2 frontmatter) so they survive a cache +rebuild, exactly like `state`. + +### 5b. Cache mirror (`app/cache.py`) + +`_upsert_cached_rfc` writes the three fields into `cached_rfcs` +(`unreviewed`, `reviewed_at`, `reviewed_by`). + +### 5c. Entry-landing path + +When a creating idea-PR merges (§2.4), resolve the project's `initial_state`: + +- `super-draft` → today's behavior unchanged (propose → super-draft → graduate). +- `active` → land the entry `active` with `unreviewed = true`, skipping the §13 + graduate gate. The exact merge-handling site (in the PR-merge reconcile path) + is located during implementation. + +### 5d. Mark-reviewed + +`POST /api/projects/:pid/rfcs/:slug/mark-reviewed`. Authority: +`is_project_superuser` (project_admin or deployment owner/admin) — the same tier +that graduates an entry. Effect: the bot writes `unreviewed: false` + +`reviewed_at`/`reviewed_by` into the entry frontmatter (a git commit, paralleling +graduate), which the mirror then reflects into `cached_rfcs`. Stamps provenance +paralleling `graduated_at`/`graduated_by`. + +### 5e. Catalog filter (query side) + +`GET /api/rfcs` (and the project-scoped form) gains an `unreviewed=true` query +param that filters on the cached column — the owner's worklist. The UI for it +is M3-frontend; only the query side ships here. `unreviewed` applies to +`active` entries only. + +--- + +## 6. Testing + +- **Migration `027`:** seed a `026`-shaped DB with rows under + `project_id='default'`; run `027`; assert: new columns present; the 12 tables + rebuilt with composite keys + FK; all 19 `project_id`-bearing tables + re-stamped to ``; row counts preserved; FK integrity on; idempotent + re-run is a no-op. +- **`registry.py`:** valid `projects.yaml` upserts all fields + `registry_sha`; + each validation failure rejected (bad enum, missing field, dup id, `type` + mutation, missing default id); `initial_state` type-default applied; + startup-strict vs running-tolerant posture. +- **APIs:** `/api/deployment` visibility filtering across + gated/public/unlisted × member/non-member/anonymous; `/api/projects/:id` 404 + gate for gated non-member, 200 for unlisted-by-id. +- **Review flow:** `initial_state: active` lands `unreviewed=true`; + mark-reviewed authority (allow superuser, deny others) + frontmatter write + + mirror reflection; catalog `unreviewed` filter returns the worklist. +- **Two-project isolation:** extend `test_multi_project_authz_vertical.py` with + a genuine **second** registry project sharing a slug with the first, proving + the PK rebuilds isolate them (the core point of M3's activation). + +--- + +## 7. File-touch summary + +**New** +- `backend/migrations/027_projects_activate.sql` +- `backend/app/registry.py` +- `backend/app/api_deployment.py` +- tests: `backend/tests/test_migration_027.py`, `test_registry.py`, + `test_api_deployment.py`, `test_review_flow.py`; extend + `test_multi_project_authz_vertical.py` + +**Modified** +- `backend/app/config.py` (registry_repo required; meta_repo removed; + default_project_id) +- `backend/app/webhooks.py` (registry-repo branch) +- `backend/app/cache.py` (`Reconciler.sweep` registry call; + `_upsert_cached_rfc` review fields) +- `backend/app/entry.py` (frontmatter fields) +- `backend/app/api.py` (mount `api_deployment`; `unreviewed` filter on + `/api/rfcs`) +- `backend/app/main.py` (startup `refresh_registry`; drop `seed_default_project`) +- `backend/app/projects.py` (retire `seed_default_project`) +- `backend/.env.example` (`REGISTRY_REPO`, `DEFAULT_PROJECT_ID`; remove + `META_REPO`) + +--- + +## 8. Versioning & upgrade + +Minor bump, breaking (pre-1.0). `VERSION` + `frontend/package.json#version` move +together (§20). `CHANGELOG.md` gets a breaking entry with an **upgrade-steps** +block: + +1. Create a registry repo under the deployment's Gitea org. +2. Author `projects.yaml`: a `deployment:` block (`name`, `tagline`) and one + `projects:` entry for the existing corpus — `id: `, `name`, `type: + document`, `content_repo: `, `visibility: public`. +3. Set env: `REGISTRY_REPO=`, `DEFAULT_PROJECT_ID=` + (must equal the entry's `id`); remove `META_REPO`. +4. Deploy. Migration `027` runs the rebuilds + re-stamp; `refresh_registry` + reconciles the registry into `projects`. Verify `/api/deployment` returns the + project and `/api/health` is green. + +The SPEC.md §22 merge stays in M7 per the slicing plan; this slice references the +draft at `docs/design/multi-project-spec.md`. + +## Known limitations / deferred + +- **Project archival/deletion** from the registry is not handled (a removed + entry stops refreshing but its rows persist). Defer to a future archival spec. +- All routing, redirects, runtime branding, theme application, directory, and + switcher are **M3-frontend**. diff --git a/frontend/package.json b/frontend/package.json index ed35cd7..473d2c1 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,7 +1,7 @@ { "name": "rfc-app-frontend", "private": true, - "version": "0.32.0", + "version": "0.33.0", "type": "module", "scripts": { "dev": "vite",