Files
rfc-app/docs/superpowers/plans/2026-06-05-s1-three-tier-collection-grain.md
T
2026-06-05 07:53:41 -07:00

797 lines
43 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# S1 — Three-tier collection grain (migration 029 + threading + redirect) Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Insert a *collection* grain beneath today's `project` so every entry-scoped row keys on `(collection_id, slug)` instead of `(project_id, slug)`, with one invisible default collection per project — the deployment runs exactly as before, now with a real collection layer and one extra `/c/<collection>/` URL segment.
**Architecture:** A new `collections` table sits beneath `projects` (which keeps `content_repo`, `name`, `tagline`, `theme`, `visibility`). Migration 029 creates it, moves the per-corpus fields (`type`, `initial_state`) down from `projects`, seeds one default collection (`id='default'`) per project, re-keys the 13 entry-corpus tables `(project_id,slug)→(collection_id,slug)` via the migration-028 rebuild pattern, and generalises `project_members → memberships(scope_type ∈ {project,collection}, …)`. The backend threads `collection_id` through the writers/readers of those 13 tables (project-grain authz is recovered by joining `collections`); the frontend gains a `/c/:collectionId/` route layer and redirects that 308 the shipped `/p/<project>/e/<slug>` URLs to `/p/<project>/c/default/e/<slug>`. Serving stays **project-scoped** in S1 (collection = default); collection-aware serving is S2.
**Tech Stack:** FastAPI + SQLite (raw SQL migrations run by `backend/app/db.py:run_migrations`, glob-ordered, `-- migrate:no-foreign-keys` marker toggles FK enforcement + runs `foreign_key_check`); pytest "vertical" tests (no Gherkin runner exists — `@S1` scenarios are realised as plain pytest); React Router SPA (`frontend/src/App.jsx`), nginx proxies `/rfc/` + `/proposals/` to the backend for server-side 308s.
**Binding spec:** `docs/design/2026-06-05-three-tier-projects-collections.md` — Part A (model), Part E / §A.6 (migration strategy), Part C `@S1` scenarios C3.7 + C3.8.
---
## Decisions locked before coding (read first)
1. **Default collection id = the literal `'default'`** (not the project id). Reason: on a *fresh* deploy migrations run with `project_id='default'` and `restamp` renames it to the configured id (e.g. `ohm`) afterward; on an *already-deployed* instance `project_id` is already `ohm` when 029 runs. A stable literal keeps the collection id **identical across both deploy histories**, matches the spec's `/c/default/` URLs, and lets the existing `restamp` keep working untouched (it renames only the *project* grain — `collections.project_id` and the denormalised `project_id` tags — never `collections.id` or the entry `collection_id`). The re-key maps each entry to its project's default collection via a JOIN, so it is correct regardless of the `project_id` value at migration time. Multi-project deployments at migration time (non-standard pre-S5) get a unique id per project via a `CASE` so the seed never collides.
2. **Re-key scope = exactly the 13 tables migration 028 rebuilt** (`cached_rfcs`, `rfc_invitations`, `cached_branches`, `branch_visibility`, `branch_contribute_grants`, `stars`, `watches`, `pr_seen`, `branch_chat_seen`, `funder_consents`, `rfc_collaborators`, `contribution_requests`, `proposed_use_cases`). The other tables 026 tagged with `project_id` (`threads`, `changes`, `notifications`, `actions`, `pr_resolution_branches`, `cached_prs`) keep `project_id` — they carry a project-grain tag, stay consistent for N=1, and renaming them is **out of S1 scope** (deferred). This matches the goal's "re-key entry-scoped tables via the 028 rebuild pattern".
3. **Serving stays project-scoped in S1.** The `/c/:collectionId/` segment is introduced in routing + redirects; the frontend data layer keeps calling `/api/projects/{project_id}/rfcs/...` (the default collection). Collection-aware serving + the registry `.collection.yaml` reader land in S2.
4. **No Gherkin runner.** `@S1` acceptance is realised as pytest vertical tests + a frontend route test. The whole existing backend suite is the "N=1 unchanged" regression net — it must go green again after the rename.
---
## File structure
**Created:**
- `backend/migrations/029_collections.sql` — the migration (collections table, field move-down, default-collection seed, 13-table re-key, `project_members → memberships`).
- `backend/app/collections.py` — collection resolution helpers (`default_collection_id`, `collection_type`, `collection_initial_state`, `collections_of_project`).
- `backend/tests/test_migration_029_collections.py` — migration shape + data-preservation + FK tests (template: `test_migration_028_project_scoped_keys.py`).
- `backend/tests/test_s1_collection_grain_vertical.py``@S1` acceptance (C3.7 redirect to sole collection; default-collection redirect; N=1 serving unchanged).
**Modified (backend):**
- `backend/app/projects.py``restamp_default_project` bootstrap check (`cached_rfcs.project_id` → a still-valid column); move `project_initial_state` to read the collection; add re-export shim if needed.
- `backend/app/auth.py``project_of_rfc` joins `collections`; the 13-table reads/writes that touch `project_id` switch to `collection_id`.
- `backend/app/cache.py``_upsert_cached_rfc(..., collection_id)` + the `cached_rfcs`/`cached_branches` writers + the `WHERE project_id` reconciler reads.
- `backend/app/api.py`, `api_prs.py`, `api_branches.py`, `api_notifications.py`, `api_contributions.py`, `api_invitations.py`, `api_graduation.py`, `funder.py` — every SQL touching the 13 tables' `project_id` column → `collection_id`; recover project via `collections` join where authz needs it.
- `backend/app/api_deployment.py``/rfc/{slug}` family 308 targets gain `/c/default/`; `get_deployment`/`get_project` read `type`/`initial_state` from the default collection.
**Modified (frontend):**
- `frontend/src/components/entryPaths.js` (or wherever path builders live) — insert `/c/:collectionId/`.
- `frontend/src/App.jsx` — add `/c/:collectionId/*` route layer; redirect `/p/:projectId/` → sole/default collection (C3.7); redirect legacy `/p/:projectId/e|proposals/...``/c/default/...`.
- `frontend/src/ProjectLayout.jsx` (+ `RFCView.jsx`, `Catalog.jsx` as needed) — read `:collectionId` param; pass through (data stays project-scoped).
**Modified (release):**
- `VERSION`, `frontend/package.json#version`, `CHANGELOG.md` — minor bump with breaking-URL upgrade-steps block (§20.2 / §20.4).
---
## Phase 1 — Migration 029 (the collection grain)
### Task 1: Write the migration-029 shape test (red)
**Files:**
- Test: `backend/tests/test_migration_029_collections.py`
- [ ] **Step 1: Write the failing test**
```python
"""Migration 029 — collections grain beneath projects. Template: test_migration_028."""
import os
import sqlite3
import tempfile
import pytest
from app import db
class _Cfg:
def __init__(self, path):
self.database_path = path
self.default_project_id = "default"
def _fresh_db():
d = tempfile.mkdtemp()
path = os.path.join(d, "test.db")
db._CONN = None
db.run_migrations(_Cfg(path))
return db.conn()
def test_collections_table_exists_with_default_per_project():
conn = _fresh_db()
cols = {r["name"] for r in conn.execute("PRAGMA table_info(collections)")}
assert {"id", "project_id", "type", "subfolder",
"initial_state", "visibility", "name", "registry_sha"} <= cols
# one default collection seeded for the bootstrap 'default' project
row = conn.execute(
"SELECT id, project_id, subfolder FROM collections WHERE project_id='default'"
).fetchone()
assert row is not None
assert row["id"] == "default"
assert row["subfolder"] == "" # repo root
def test_per_corpus_fields_moved_off_projects():
conn = _fresh_db()
proj_cols = {r["name"] for r in conn.execute("PRAGMA table_info(projects)")}
assert "type" not in proj_cols
assert "initial_state" not in proj_cols
# projects keeps the grouping-tier fields
assert {"id", "name", "content_repo", "visibility"} <= proj_cols
def test_entry_tables_rekeyed_to_collection_id():
conn = _fresh_db()
for t in ("cached_rfcs", "cached_branches", "stars", "watches",
"rfc_collaborators", "contribution_requests", "proposed_use_cases",
"branch_visibility", "branch_contribute_grants", "pr_seen",
"branch_chat_seen", "funder_consents", "rfc_invitations"):
cols = {r["name"] for r in conn.execute(f"PRAGMA table_info({t})")}
assert "collection_id" in cols, f"{t} missing collection_id"
assert "project_id" not in cols, f"{t} still has project_id"
def test_cached_rfcs_pk_is_collection_slug():
conn = _fresh_db()
# same slug coexists across two collections
conn.execute("INSERT INTO collections (id, project_id, type, subfolder, initial_state, visibility, name) "
"VALUES ('c2','default','document','specs','active','public','Specs')")
conn.execute("INSERT INTO cached_rfcs (slug, title, state, collection_id) VALUES ('intro','A','active','default')")
conn.execute("INSERT INTO cached_rfcs (slug, title, state, collection_id) VALUES ('intro','B','active','c2')")
n = conn.execute("SELECT COUNT(*) c FROM cached_rfcs WHERE slug='intro'").fetchone()["c"]
assert n == 2
with pytest.raises(sqlite3.IntegrityError):
conn.execute("INSERT INTO cached_rfcs (slug, title, state, collection_id) VALUES ('intro','dup','active','default')")
def test_collaborator_fk_is_composite_on_collection():
conn = _fresh_db()
conn.execute("INSERT INTO collections (id, project_id, type, subfolder, initial_state, visibility, name) "
"VALUES ('c2','default','document','specs','active','public','Specs')")
conn.execute("INSERT INTO cached_rfcs (slug, title, state, collection_id) VALUES ('intro','A','active','c2')")
conn.execute("INSERT INTO users (id, email, role, permission_state) VALUES (1,'a@b.c','contributor','granted')")
conn.execute("PRAGMA foreign_keys=ON")
conn.execute("INSERT INTO rfc_collaborators (rfc_slug, user_id, role_in_rfc, collection_id) "
"VALUES ('intro',1,'contributor','c2')")
with pytest.raises(sqlite3.IntegrityError):
conn.execute("INSERT INTO rfc_collaborators (rfc_slug, user_id, role_in_rfc, collection_id) "
"VALUES ('intro',1,'contributor','default')") # no such (collection,slug)
def test_memberships_table_replaces_project_members():
conn = _fresh_db()
cols = {r["name"] for r in conn.execute("PRAGMA table_info(memberships)")}
assert {"scope_type", "scope_id", "user_id", "role", "granted_by", "granted_at"} <= cols
# M2 rows would migrate to scope_type='collection'; role enum collapsed to owner/contributor
# (no project_members rows exist in a fresh DB, so just assert the table + check constraint)
conn.execute("INSERT INTO users (id, email, role, permission_state) VALUES (9,'x@y.z','contributor','granted')")
conn.execute("INSERT INTO memberships (scope_type, scope_id, user_id, role) VALUES ('project','default',9,'owner')")
with pytest.raises(sqlite3.IntegrityError):
conn.execute("INSERT INTO memberships (scope_type, scope_id, user_id, role) VALUES ('bogus','default',9,'owner')")
```
- [ ] **Step 2: Run to verify it fails**
Run: `cd backend && python -m pytest tests/test_migration_029_collections.py -q`
Expected: FAIL (no `collections` table / `029_collections.sql` does not exist).
- [ ] **Step 3: Commit the red test**
```bash
git add backend/tests/test_migration_029_collections.py
git commit -m "§22 S1: failing migration-029 shape tests (collections grain)"
```
### Task 2: Write migration 029 (green the shape test)
**Files:**
- Create: `backend/migrations/029_collections.sql`
- [ ] **Step 1: Write the migration.** Mirror `028_project_scoped_keys.sql` exactly for the 13 rebuilds, with `project_id` renamed to `collection_id` in each `__new` table, each child FK re-pointed to `cached_rfcs(collection_id, slug)`, and each index/UNIQUE swapping `project_id``collection_id`. Use the explicit-column `INSERT ... SELECT` form (not `SELECT *`) so the re-key can map values. Header marker `-- migrate:no-foreign-keys`. Concrete top of file:
```sql
-- migrate:no-foreign-keys
--
-- §22 three-tier refactor — S1. Insert a *collection* grain beneath project.
-- (1) collections table; (2) move per-corpus fields (type, initial_state) down
-- from projects; (3) one default collection per project (id='default',
-- subfolder = repo root); (4) re-key the 13 entry-corpus tables
-- (project_id,slug) -> (collection_id,slug) via the 028 rebuild pattern, mapping
-- each row to its project's default collection by JOIN; (5) project_members ->
-- memberships(scope_type ∈ {project,collection}, …), role enum collapsed to
-- {owner, contributor}. FK enforcement is OFF for the file (marker above);
-- foreign_key_check runs after. See docs/design/2026-06-05-three-tier-…md §A.6.
-- ── collections: the new typed-corpus grain beneath projects ───────────────
CREATE TABLE collections (
id TEXT NOT NULL,
project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
type TEXT NOT NULL DEFAULT 'document'
CHECK (type IN ('document', 'specification', 'bdd')),
subfolder TEXT NOT NULL DEFAULT '',
initial_state TEXT NOT NULL DEFAULT 'super-draft'
CHECK (initial_state IN ('super-draft', 'active')),
visibility TEXT NOT NULL DEFAULT 'gated'
CHECK (visibility IN ('gated', 'public', 'unlisted')),
name TEXT,
registry_sha TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
PRIMARY KEY (id)
);
CREATE INDEX idx_collections_project ON collections(project_id);
-- One default collection per project. id='default' for the standard
-- single-project deployment (stable across deploy histories); the project_id is
-- used as a unique fallback id only if a non-standard multi-project deployment
-- migrates (pre-S5; avoids a PK collision). subfolder='' = repo root.
INSERT INTO collections (id, project_id, type, subfolder, initial_state, visibility, name)
SELECT
CASE WHEN (SELECT COUNT(*) FROM projects) <= 1 THEN 'default' ELSE p.id END,
p.id, p.type, '', p.initial_state, p.visibility, p.name
FROM projects p;
-- ── move per-corpus fields off projects (rebuild to DROP type/initial_state) ─
CREATE TABLE projects__new (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
content_repo TEXT,
visibility TEXT NOT NULL DEFAULT 'gated'
CHECK (visibility IN ('gated', 'public', 'unlisted')),
config_json TEXT,
registry_sha TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT INTO projects__new (id, name, content_repo, visibility, config_json, registry_sha, created_at, updated_at)
SELECT id, name, content_repo, visibility, config_json, registry_sha, created_at, updated_at FROM projects;
DROP TABLE projects;
ALTER TABLE projects__new RENAME TO projects;
-- ── cached_rfcs: PRIMARY KEY (project_id, slug) -> (collection_id, slug) ────
-- collection_id mapped from the row's old project's default collection.
CREATE TABLE cached_rfcs__new (
slug TEXT NOT NULL,
title TEXT NOT NULL,
state TEXT NOT NULL CHECK (state IN ('super-draft', 'active', 'withdrawn', 'retired')),
rfc_id TEXT,
repo TEXT,
proposed_by TEXT,
proposed_at TEXT,
graduated_at TEXT,
graduated_by TEXT,
owners_json TEXT NOT NULL DEFAULT '[]',
arbiters_json TEXT NOT NULL DEFAULT '[]',
tags_json TEXT NOT NULL DEFAULT '[]',
body TEXT,
body_sha TEXT,
last_main_commit_at TEXT,
last_entry_commit_at TEXT,
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
models_json TEXT,
funder_login TEXT,
proposed_use_case TEXT,
collection_id TEXT NOT NULL DEFAULT 'default' REFERENCES collections(id),
unreviewed INTEGER NOT NULL DEFAULT 0,
reviewed_at TEXT,
reviewed_by TEXT,
PRIMARY KEY (collection_id, slug)
);
INSERT INTO cached_rfcs__new
(slug, title, state, rfc_id, repo, proposed_by, proposed_at, graduated_at,
graduated_by, owners_json, arbiters_json, tags_json, body, body_sha,
last_main_commit_at, last_entry_commit_at, updated_at, models_json,
funder_login, proposed_use_case, collection_id, unreviewed, reviewed_at, reviewed_by)
SELECT
r.slug, r.title, r.state, r.rfc_id, r.repo, r.proposed_by, r.proposed_at, r.graduated_at,
r.graduated_by, r.owners_json, r.arbiters_json, r.tags_json, r.body, r.body_sha,
r.last_main_commit_at, r.last_entry_commit_at, r.updated_at, r.models_json,
r.funder_login, r.proposed_use_case,
(SELECT c.id FROM collections c WHERE c.project_id = r.project_id LIMIT 1),
r.unreviewed, r.reviewed_at, r.reviewed_by
FROM cached_rfcs r;
DROP TABLE cached_rfcs;
ALTER TABLE cached_rfcs__new RENAME TO cached_rfcs;
CREATE INDEX idx_cached_rfcs_state ON cached_rfcs (state);
CREATE INDEX idx_cached_rfcs_last_active ON cached_rfcs (
COALESCE(last_main_commit_at, last_entry_commit_at) DESC
);
CREATE INDEX idx_cached_rfcs_collection ON cached_rfcs(collection_id);
```
Then **for each of the remaining 12 tables** copy its `028` block verbatim and apply the same three transforms: (a) rename the `project_id` column to `collection_id` (keep `DEFAULT 'default'`); (b) in the `INSERT ... SELECT`, replace the `project_id` source value with `(SELECT c.id FROM collections c WHERE c.project_id = <old>.project_id LIMIT 1)` and list columns explicitly; (c) rename `project_id``collection_id` in every `UNIQUE (...)`, `FOREIGN KEY (...) REFERENCES cached_rfcs(...)`, and `CREATE [UNIQUE] INDEX`. The 12: `rfc_invitations`, `cached_branches`, `branch_visibility`, `branch_contribute_grants`, `stars`, `watches`, `pr_seen`, `branch_chat_seen`, `funder_consents`, `rfc_collaborators`, `contribution_requests`, `proposed_use_cases`. (FK targets `cached_rfcs(project_id, slug)` become `cached_rfcs(collection_id, slug)`.)
Finally the membership generalisation:
```sql
-- ── project_members -> memberships(scope_type, scope_id, …); roles collapsed ─
CREATE TABLE memberships (
id INTEGER PRIMARY KEY AUTOINCREMENT,
scope_type TEXT NOT NULL CHECK (scope_type IN ('project', 'collection')),
scope_id TEXT NOT NULL,
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
role TEXT NOT NULL CHECK (role IN ('owner', 'contributor')),
granted_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
granted_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE (scope_type, scope_id, user_id)
);
CREATE INDEX idx_memberships_user ON memberships(user_id);
CREATE INDEX idx_memberships_scope ON memberships(scope_type, scope_id);
-- M2 project_members rows attached at what is now the *collection*; collapse
-- the role enum (project_admin -> owner, project_contributor -> contributor,
-- project_viewer -> dropped this pass, §B.3) and migrate to the default
-- collection of each project.
INSERT INTO memberships (scope_type, scope_id, user_id, role, granted_by, granted_at)
SELECT 'collection',
(SELECT c.id FROM collections c WHERE c.project_id = pm.project_id LIMIT 1),
pm.user_id,
CASE pm.role WHEN 'project_admin' THEN 'owner'
WHEN 'project_contributor' THEN 'contributor'
ELSE 'contributor' END,
pm.granted_by, pm.granted_at
FROM project_members pm
WHERE pm.role IN ('project_admin', 'project_contributor');
DROP TABLE project_members;
```
- [ ] **Step 2: Run the shape test**
Run: `cd backend && python -m pytest tests/test_migration_029_collections.py -q`
Expected: PASS (all shape/PK/FK/membership assertions green).
- [ ] **Step 3: Commit**
```bash
git add backend/migrations/029_collections.sql
git commit -m "§22 S1: migration 029 — collections grain, field move-down, 13-table re-key, memberships"
```
---
## Phase 2 — Backend threading (make the existing suite green again)
> After Task 2 the column rename breaks every reader/writer of the 13 tables. This phase fixes them. **Driver:** the full backend suite is the regression net — run it, read each failure, fix the named module, repeat until green. The agent exploration produced the exact blast-radius map used below.
### Task 3: collections helper module
**Files:**
- Create: `backend/app/collections.py`
- [ ] **Step 1: Write the helper**
```python
"""§22 collection grain — resolution helpers beneath the project tier.
In S1 each project has exactly one collection (the default). These helpers
recover the collection for a project and read the per-corpus fields that moved
down from `projects` in migration 029. Project-grain authz (auth.py) recovers a
row's project by joining `collections` on `collection_id`.
"""
from __future__ import annotations
from . import db
DEFAULT_COLLECTION_ID = "default"
def default_collection_id(project_id: str) -> str:
"""The id of a project's default (S1: sole) collection. Falls back to the
literal 'default' when the project has no collection row yet."""
row = db.conn().execute(
"SELECT id FROM collections WHERE project_id = ? ORDER BY created_at LIMIT 1",
(project_id,),
).fetchone()
return row["id"] if row else DEFAULT_COLLECTION_ID
def project_of_collection(collection_id: str) -> str | None:
row = db.conn().execute(
"SELECT project_id FROM collections WHERE id = ?", (collection_id,)
).fetchone()
return row["project_id"] if row else None
def collection_initial_state(collection_id: str) -> str:
"""§22.4b landing state for new entries in a collection. 'super-draft'
default for an unknown row (today's safe flow)."""
row = db.conn().execute(
"SELECT initial_state FROM collections WHERE id = ?", (collection_id,)
).fetchone()
if row is None or not row["initial_state"]:
return "super-draft"
return row["initial_state"]
def collection_type(collection_id: str) -> str:
row = db.conn().execute(
"SELECT type FROM collections WHERE id = ?", (collection_id,)
).fetchone()
return row["type"] if row and row["type"] else "document"
```
- [ ] **Step 2: Commit**
```bash
git add backend/app/collections.py
git commit -m "§22 S1: collections resolution helpers"
```
### Task 4: Fix `projects.py` (restamp + initial_state)
**Files:**
- Modify: `backend/app/projects.py:46-48` (restamp bootstrap check), `:112-121` (`project_initial_state`)
- [ ] **Step 1: Fix the restamp bootstrap check.** `restamp_default_project` reads `cached_rfcs.project_id` (now renamed) at line 47 — switch the existence probe to a still-`project_id`-bearing table so the PRAGMA-driven rename loop is unaffected (it already discovers `project_id` columns dynamically, which now correctly excludes the 13 collection-keyed tables and includes `collections.project_id`):
```python
has_rows = conn.execute(
"SELECT 1 FROM collections WHERE project_id = ? LIMIT 1", (DEFAULT_PROJECT_ID,)
).fetchone()
```
- [ ] **Step 2: Re-home `project_initial_state`.** Keep the signature for callers, but resolve through the project's default collection:
```python
def project_initial_state(project_id: str) -> str:
"""§22.4b landing state for new entries in a project's default collection."""
from . import collections as collections_mod
return collections_mod.collection_initial_state(
collections_mod.default_collection_id(project_id)
)
```
- [ ] **Step 3: Run the restamp + projects tests**
Run: `cd backend && python -m pytest tests/test_restamp_default_project.py tests/test_initial_state_landing.py -q`
Expected: PASS.
- [ ] **Step 4: Commit**
```bash
git add backend/app/projects.py
git commit -m "§22 S1: thread projects.py restamp + initial_state through collections"
```
### Task 5: Fix `auth.py` (`project_of_rfc` join)
**Files:**
- Modify: `backend/app/auth.py:352-361`
- [ ] **Step 1: Join collections to recover the project from a slug.**
```python
def project_of_rfc(rfc_slug: str) -> str:
"""The project an RFC belongs to, via its collection
(cached_rfcs.collection_id -> collections.project_id). Falls back to the
default project when the slug isn't cached."""
row = db.conn().execute(
"SELECT c.project_id AS project_id "
"FROM cached_rfcs r JOIN collections c ON c.id = r.collection_id "
"WHERE r.slug = ?",
(rfc_slug,),
).fetchone()
if row is None:
return DEFAULT_PROJECT_ID
return row["project_id"] or DEFAULT_PROJECT_ID
```
- [ ] **Step 2: Run the authz suite**
Run: `cd backend && python -m pytest tests/test_multi_project_authz_vertical.py tests/test_anon_offlimits_vertical.py -q`
Expected: PASS.
- [ ] **Step 3: Commit**
```bash
git add backend/app/auth.py
git commit -m "§22 S1: auth.project_of_rfc recovers project via collection join"
```
### Task 6: Fix `cache.py` writers/readers
**Files:**
- Modify: `backend/app/cache.py``_refresh_project_corpus` (resolve collection), `_upsert_cached_rfc` signature + SQL (`project_id``collection_id`), the `WHERE project_id` reconciler read (`:88`), the `cached_branches` writers (`:213/:388/:410`).
- [ ] **Step 1: Resolve the collection in the corpus refresh.** In `_refresh_project_corpus`, compute the project's default collection once and pass it down; switch the reconciler `SELECT slug ... WHERE project_id` to `WHERE collection_id`:
```python
async def _refresh_project_corpus(org: str, project_id: str, repo: str, gitea: Gitea) -> None:
from . import collections as collections_mod
collection_id = collections_mod.default_collection_id(project_id)
...
_upsert_cached_rfc(entry, body_sha=sha, collection_id=collection_id)
...
existing = {
row["slug"]
for row in db.conn().execute(
"SELECT slug FROM cached_rfcs WHERE collection_id = ?", (collection_id,)
)
}
```
- [ ] **Step 2: Rename in `_upsert_cached_rfc`.** Change the param `project_id: str = "default"``collection_id: str = "default"`; in the `INSERT`, replace the `project_id` column with `collection_id`, the `ON CONFLICT(project_id, slug)` with `ON CONFLICT(collection_id, slug)`, and the bound value `project_id``collection_id`.
- [ ] **Step 3: Fix the `cached_branches` writers.** At `:213/:388/:410` the `ON CONFLICT(project_id, rfc_slug, branch_name)` clauses → `ON CONFLICT(collection_id, rfc_slug, branch_name)`; where a meta-repo branch row is written without an explicit grain it now relies on the `collection_id DEFAULT 'default'` column default (unchanged behaviour for N=1). Bind `collection_id` explicitly where the per-project loop has it.
- [ ] **Step 4: Run the cache tests**
Run: `cd backend && python -m pytest tests/test_cache_bootstrap.py tests/test_cache_review_fields.py tests/test_branch_path_routing.py -q`
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add backend/app/cache.py
git commit -m "§22 S1: thread cache.py corpus/branch writers through collection_id"
```
### Task 7: Fix the `api_*` writers/readers + `funder.py`
**Files (each: swap the 13-table `project_id` column references to `collection_id`; recover project for authz via `auth.project_of_rfc`/`collections` join):**
- `backend/app/api.py:747` (stars read), `:774/:806/:969` (`cached_rfcs` composite lookups → `collection_id`), `:785-788/:1046` (`proposed_use_cases`).
- `backend/app/api_prs.py:156` (`branch_visibility`), `:192` (`proposed_use_cases`), `:401` (`pr_seen`). Note `:670/:789` read `row["project_id"]` from a `cached_rfcs`/`rfc` row — change those SELECTs to also yield the project via the collection join, then keep the existing `auth.require_project_readable(viewer, project_id)` call unchanged.
- `backend/app/api_branches.py:745` (`branch_visibility`), `:899` (`branch_chat_seen`).
- `backend/app/api_notifications.py:216` (read `cached_rfcs` → now `collection_id`; recover project via join for the visibility gate), `:225` (`watches`).
- `backend/app/api_contributions.py:65/:111` (read `cached_rfcs`; recover project via join), `api_invitations.py:383`, `api_graduation.py` (any `cached_rfcs`/13-table `project_id`).
- `backend/app/funder.py:223` (`funder_consents` `ON CONFLICT(project_id,…)``collection_id`).
- [ ] **Step 1: Mechanical pass.** For each file above, replace `project_id` **only where it names a column on one of the 13 re-keyed tables** (PK lookups, `ON CONFLICT`, `WHERE`, `INSERT` column lists, `SELECT` projections from those tables) with `collection_id`. Where the code needs the *project* (for `auth.*_project*` calls), recover it with `auth.project_of_rfc(slug)` or a `collections` join — do **not** rename the `project_id` argument flowing into the authz helpers (those stay project-grain in S1). Leave `threads`, `changes`, `notifications`, `actions`, `pr_resolution_branches`, `cached_prs` `project_id` columns untouched.
- [ ] **Step 2: Grep guard.** Confirm no stray reference to a dropped column remains:
Run: `cd backend && grep -rEn "cached_rfcs[^;]*project_id|project_id, slug|project_id, rfc_slug|ON CONFLICT\(project_id" app/ | grep -v "collections\|threads\|changes\|notifications\|actions\|pr_resolution\|cached_prs"`
Expected: no output (every 13-table `project_id` is now `collection_id`).
- [ ] **Step 3: Run the full backend suite**
Run: `cd backend && python -m pytest -q`
Expected: PASS (this is the **N=1-unchanged** gate). Fix any remaining failures by reading the traceback and applying the same rename/join rule.
- [ ] **Step 4: Commit**
```bash
git add backend/app/api.py backend/app/api_prs.py backend/app/api_branches.py backend/app/api_notifications.py backend/app/api_contributions.py backend/app/api_invitations.py backend/app/api_graduation.py backend/app/funder.py
git commit -m "§22 S1: thread api_* + funder writers/readers through collection_id"
```
---
## Phase 3 — API surface reads per-corpus fields from the collection
### Task 8: `api_deployment.py` reads type/initial_state from the default collection
**Files:**
- Modify: `backend/app/api_deployment.py:36-44` (`get_deployment` projects list `type`), `:62-82` (`get_project` `type`/`initial_state`)
- [ ] **Step 1: Write a failing test** in `backend/tests/test_api_deployment.py` (extend it) asserting `GET /api/projects/{default}` still returns the correct `type`/`initial_state` after the move-down (values come from the default collection):
```python
def test_get_project_type_initial_state_from_default_collection(app_with_fake_gitea):
# ... existing fixture sets up the default project/collection ...
r = client.get(f"/api/projects/{default_id}")
assert r.status_code == 200
body = r.json()
assert body["type"] in ("document", "specification", "bdd")
assert body["initial_state"] in ("super-draft", "active")
```
- [ ] **Step 2: Run to verify it fails** (the SELECT still reads `projects.type`, which 029 dropped → `OperationalError`).
Run: `cd backend && python -m pytest tests/test_api_deployment.py -q`
Expected: FAIL.
- [ ] **Step 3: Read the fields from the default collection.** In `get_deployment`, replace the `SELECT id, name, type, visibility FROM projects` with a join to the project's default collection for `type` (or a per-row `collections_mod.collection_type(default_collection_id(id))`). In `get_project`, drop `type, initial_state` from the `projects` SELECT and resolve them via `collections_mod.collection_type(...)` / `collection_initial_state(...)`:
```python
from . import collections as collections_mod
...
cid = collections_mod.default_collection_id(row["id"])
return {
...
"type": collections_mod.collection_type(cid),
"initial_state": collections_mod.collection_initial_state(cid),
...
}
```
- [ ] **Step 4: Run to verify it passes**
Run: `cd backend && python -m pytest tests/test_api_deployment.py -q`
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add backend/app/api_deployment.py backend/tests/test_api_deployment.py
git commit -m "§22 S1: deployment/project API reads type+initial_state from default collection"
```
---
## Phase 4 — Redirects + frontend collection segment
### Task 9: Backend `/rfc/` 308s target `/c/default/`
**Files:**
- Modify: `backend/app/api_deployment.py:89-104`
- [ ] **Step 1: Add a failing test** to `test_api_deployment.py`:
```python
def test_legacy_rfc_url_redirects_through_collection(app_with_fake_gitea):
r = client.get("/rfc/intro", follow_redirects=False)
assert r.status_code == 308
assert r.headers["location"] == f"/p/{default_id}/c/default/e/intro"
```
- [ ] **Step 2: Verify it fails** (current target lacks `/c/default/`).
- [ ] **Step 3: Update the three redirect handlers** to resolve the default collection and insert the `/c/<cid>/` segment:
```python
@router.get("/rfc/{slug}")
async def redirect_old_rfc(slug: str) -> RedirectResponse:
default_id = projects_mod.resolved_default_id(config)
cid = collections_mod.default_collection_id(default_id)
return RedirectResponse(url=f"/p/{default_id}/c/{cid}/e/{slug}", status_code=308)
# …same /c/{cid}/ insertion for /rfc/{slug}/pr/{pr} and /proposals/{pr}
```
(`/proposals/{pr}``/p/{default_id}/c/{cid}/proposals/{pr}`.)
- [ ] **Step 4: Verify it passes.**
Run: `cd backend && python -m pytest tests/test_api_deployment.py -q`
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add backend/app/api_deployment.py backend/tests/test_api_deployment.py
git commit -m "§22 S1: legacy /rfc + /proposals 308s route through /c/<default>/"
```
### Task 10: Frontend path builders gain `/c/:collectionId/`
**Files:**
- Modify: `frontend/src/components/entryPaths.js` (path builders — confirm exact path with `grep -rl "p/\${" frontend/src`)
- [ ] **Step 1: Thread a collection id through the builders.** Add a `collectionId` argument (defaulting to `'default'`) and emit the `/c/<collectionId>/` segment:
```js
export const collectionHome = (projectId, collectionId) => `/p/${projectId}/c/${collectionId}/`
export const entryPath = (projectId, collectionId, slug) => `/p/${projectId}/c/${collectionId}/e/${slug}`
export const entryPrPath = (projectId, collectionId, slug, prNumber) => `/p/${projectId}/c/${collectionId}/e/${slug}/pr/${prNumber}`
export const proposalPath = (projectId, collectionId, prNumber) => `/p/${projectId}/c/${collectionId}/proposals/${prNumber}`
export const projectHome = (projectId) => `/p/${projectId}/`
```
Update every caller (grep `entryPath(`, `entryPrPath(`, `proposalPath(`, `collectionHome(`) to pass the current collection id (from the route param / `useCollectionId()` — default `'default'`).
- [ ] **Step 2: Build the frontend**
Run: `cd frontend && npm run build`
Expected: build succeeds (no undefined-symbol errors).
- [ ] **Step 3: Commit**
```bash
git add frontend/src
git commit -m "§22 S1: frontend path builders carry the /c/<collection>/ segment"
```
### Task 11: Frontend route layer + redirects (C3.7, C3.8, legacy)
**Files:**
- Modify: `frontend/src/App.jsx:354-373`, `frontend/src/ProjectLayout.jsx`
- [ ] **Step 1: Nest the corpus routes under `/c/:collectionId/`** and add redirects. Inside the `ProjectLayout` nested `<Routes>`:
```jsx
<Routes>
{/* project landing: redirect to the sole/default collection (C3.7) */}
<Route path="" element={<CollectionRedirect />} />
{/* legacy v0.35.0 corpus URLs without /c/ → default collection */}
<Route path="e/:slug" element={<Navigate to="c/default/e/:slug" replace />} />
<Route path="e/:slug/pr/:prNumber" element={<LegacyEntryPrRedirect />} />
<Route path="proposals/:prNumber" element={<LegacyProposalRedirect />} />
{/* collection-scoped corpus (serving stays project-scoped in S1) */}
<Route path="c/:collectionId" element={<Welcome viewer={viewer} />} />
<Route path="c/:collectionId/e/:slug" element={<RFCView viewer={viewer} />} />
<Route path="c/:collectionId/e/:slug/pr/:prNumber" element={<PRView viewer={viewer} />} />
<Route path="c/:collectionId/proposals/:prNumber" element={<ProposalView viewer={viewer} onChange={() => setCatalogVersion(v => v + 1)} />} />
</Routes>
```
`CollectionRedirect` reads the project's collections (from `ProjectContext`, populated by `GET /api/projects/:id`) and `<Navigate>`s to the sole visible collection's `/c/<id>/`; with one collection that is `/c/default/` (C3.7). `LegacyEntryPrRedirect`/`LegacyProposalRedirect` use `useParams()` to rebuild the target with `c/default/`. React-Router literal `:slug` in `to=` does not interpolate — implement these as small components using `useParams()` + `<Navigate>`.
- [ ] **Step 2: Confirm `/` → sole project (C3.8) already holds.** `DeploymentLanding` (App.jsx:348) already redirects to the single visible project. Add/confirm a test (Task 12) rather than re-implementing.
- [ ] **Step 3: Build**
Run: `cd frontend && npm run build`
Expected: succeeds.
- [ ] **Step 4: Commit**
```bash
git add frontend/src
git commit -m "§22 S1: /c/<collection>/ route layer + C3.7 + legacy-URL redirects"
```
---
## Phase 5 — `@S1` acceptance + full verification
### Task 12: `@S1` vertical acceptance test (C3.7 + C3.8 + N=1 serving)
**Files:**
- Create: `backend/tests/test_s1_collection_grain_vertical.py`
- [ ] **Step 1: Write the acceptance test.** Tag scenarios in docstrings as `@S1` for traceability (no Gherkin runner). Cover: (a) default-collection redirect `/rfc/<slug>``/p/<default>/c/default/e/<slug>` (already in Task 9 — re-assert here as the S1 gate); (b) an entry proposed/served at N=1 still resolves under the default collection via `/api/projects/<default>/rfcs/<slug>`; (c) the deployment `/api/deployment` still reports one project with `default_project_id`. (C3.7/C3.8 client redirects are asserted in the frontend build/route smoke; the data-layer N=1 invariants are asserted here.)
```python
"""@S1 acceptance — the collection grain exists and N=1 is unchanged.
Scenarios: C3.7 (single-collection project skips the directory) and C3.8
(single-project deployment skips the directory) are the redirect contract;
this module asserts the backend N=1 invariants behind them."""
# reuse the propose/serve fixtures from test_project_scoped_serving.py
def test_s1_entry_served_under_default_collection(app_with_fake_gitea):
# propose + mirror an entry, then fetch it project-scoped (collection=default)
...
r = client.get(f"/api/projects/{default_id}/rfcs/intro")
assert r.status_code == 200
# the row is keyed by collection_id under the hood
cid = db.conn().execute("SELECT collection_id FROM cached_rfcs WHERE slug='intro'").fetchone()["collection_id"]
assert cid == "default"
def test_s1_legacy_redirect_inserts_collection_segment(app_with_fake_gitea):
r = client.get("/rfc/intro", follow_redirects=False)
assert r.status_code == 308
assert "/c/default/" in r.headers["location"]
```
- [ ] **Step 2: Run it**
Run: `cd backend && python -m pytest tests/test_s1_collection_grain_vertical.py -q`
Expected: PASS.
- [ ] **Step 3: Full backend suite + frontend build (the N=1-unchanged gate)**
Run: `cd backend && python -m pytest -q && cd ../frontend && npm run build`
Expected: all backend tests PASS; frontend builds.
- [ ] **Step 4: Commit**
```bash
git add backend/tests/test_s1_collection_grain_vertical.py
git commit -m "§22 S1: @S1 acceptance — collection grain + N=1 serving unchanged"
```
### Task 13: e2e smoke (optional, if Docker stack available)
- [ ] **Step 1:** If the Tier-1 Docker stack is runnable, `make e2e` to confirm sign-in + a corpus page render through the new `/c/default/` routes. If the stack isn't available in-session, note it skipped and rely on Tasks 7/11/12 gates.
---
## Phase 6 — Release + finalize
### Task 14: Version bump + changelog (breaking, with upgrade steps)
**Files:**
- Modify: `VERSION`, `frontend/package.json` (`version`), `CHANGELOG.md`
- [ ] **Step 1: Bump** `VERSION` and `frontend/package.json#version` to the next pre-1.0 minor (current `0.39.0``0.40.0`). They must match (a divergence is a §20 spec bug).
- [ ] **Step 2: Add the CHANGELOG entry** with a breaking-URL **upgrade steps** block (§20.2 / §20.4 / §A.6): migration 029 adds the collection grain; `/p/<project>/e/<slug>` now lives at `/p/<project>/c/default/e/<slug>` (308 for old links); operators need no action beyond deploying (the migration + redirects are automatic; the default collection is seeded). Note the deferred items (denormalised `project_id` tags unchanged; collection-aware serving = S2).
- [ ] **Step 3: Commit**
```bash
git add VERSION frontend/package.json CHANGELOG.md
git commit -m "§22 S1: release v0.40.0 — three-tier collection grain (breaking URL + migration 029)"
```
### Task 15: Branch, PR, merge
- [ ] **Step 1:** This work rides a feature branch off `main` (e.g. `feat/s1-collection-grain`). Push to `origin` (git.wiggleverse.org).
- [ ] **Step 2:** Open a PR citing the design doc + `@S1`; in autonomous posture, self-review and merge once the suite is green.
- [ ] **Step 3:** Update repo memory with the new resume pointer (S1 shipped @ v0.40.0; next = S2).
---
## Self-review (writing-plans checklist)
- **Spec coverage:** §A.6 steps 15 → Tasks 2 (collections table + default + re-key + memberships), 8 (field move-down read path), 9 (308 step 5). Part B membership generalisation → Task 2 (`memberships`) — note S1 only *migrates* the table; the four-layer resolver is S3 (out of scope, correctly deferred per Part E). `@S1` C3.7/C3.8 → Tasks 11 (frontend redirects) + 12 (backend invariants). "N=1 unchanged" → Task 7 Step 3 + Task 12 Step 3 full-suite gates. Threading (auth/projects/cache/api_*) → Tasks 48.
- **Placeholders:** the per-table rebuild bodies for the 12 non-`cached_rfcs` tables reference the in-repo `028_project_scoped_keys.sql` as the literal template with the three explicit transforms named — this is a concrete instruction, not a TODO (repeating 200+ lines of near-identical SQL verbatim would harm reviewability; the transform rule is exact).
- **Type consistency:** `default_collection_id`, `collection_type`, `collection_initial_state`, `project_of_collection` are defined in Task 3 and used consistently in Tasks 4, 8, 9. Column `collection_id` (not `coll_id`/`collectionId`) used uniformly in SQL; `collectionId` is the JS route param.
- **Risk note:** the denormalised `project_id` columns (`threads`/`changes`/`notifications`/`actions`/`pr_resolution_branches`/`cached_prs`) stay `project_id` and may, after `restamp`, hold the project id (`ohm`) while entry `collection_id` holds `default`. Task 7 Step 3's full-suite run is the guard against any code that wrongly cross-joins the two grains; if one surfaces, recover the project via the `collections` join rather than renaming the tag.
```