benstull/rfc-app
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
677c5eb72fdc6554ab1e85ceb35a652f65d8584c
rfc-app/docs/superpowers/plans/2026-06-05-s1-three-tier-collection-grain.md
T
Ben Stull 08bdea8539 §22 S1 plan: three-tier collection grain (migration 029 + threading + redirect) 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 07:53:41 -07:00

43 KiB
Raw Blame History

S1 — Three-tier collection grain (migration 029 + threading + redirect) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Insert a collection grain beneath today's project so every entry-scoped row keys on (collection_id, slug) instead of (project_id, slug), with one invisible default collection per project — the deployment runs exactly as before, now with a real collection layer and one extra /c/<collection>/ URL segment.

Architecture: A new collections table sits beneath projects (which keeps content_repo, name, tagline, theme, visibility). Migration 029 creates it, moves the per-corpus fields (type, initial_state) down from projects, seeds one default collection (id='default') per project, re-keys the 13 entry-corpus tables (project_id,slug)→(collection_id,slug) via the migration-028 rebuild pattern, and generalises project_members → memberships(scope_type ∈ {project,collection}, …). The backend threads collection_id through the writers/readers of those 13 tables (project-grain authz is recovered by joining collections); the frontend gains a /c/:collectionId/ route layer and redirects that 308 the shipped /p/<project>/e/<slug> URLs to /p/<project>/c/default/e/<slug>. Serving stays project-scoped in S1 (collection = default); collection-aware serving is S2.

Tech Stack: FastAPI + SQLite (raw SQL migrations run by backend/app/db.py:run_migrations, glob-ordered, -- migrate:no-foreign-keys marker toggles FK enforcement + runs foreign_key_check); pytest "vertical" tests (no Gherkin runner exists — @S1 scenarios are realised as plain pytest); React Router SPA (frontend/src/App.jsx), nginx proxies /rfc/ + /proposals/ to the backend for server-side 308s.

Binding spec: docs/design/2026-06-05-three-tier-projects-collections.md — Part A (model), Part E / §A.6 (migration strategy), Part C @S1 scenarios C3.7 + C3.8.

Decisions locked before coding (read first)

  1. Default collection id = the literal 'default' (not the project id). Reason: on a fresh deploy migrations run with project_id='default' and restamp renames it to the configured id (e.g. ohm) afterward; on an already-deployed instance project_id is already ohm when 029 runs. A stable literal keeps the collection id identical across both deploy histories, matches the spec's /c/default/ URLs, and lets the existing restamp keep working untouched (it renames only the project grain — collections.project_id and the denormalised project_id tags — never collections.id or the entry collection_id). The re-key maps each entry to its project's default collection via a JOIN, so it is correct regardless of the project_id value at migration time. Multi-project deployments at migration time (non-standard pre-S5) get a unique id per project via a CASE so the seed never collides.

  2. Re-key scope = exactly the 13 tables migration 028 rebuilt (cached_rfcs, rfc_invitations, cached_branches, branch_visibility, branch_contribute_grants, stars, watches, pr_seen, branch_chat_seen, funder_consents, rfc_collaborators, contribution_requests, proposed_use_cases). The other tables 026 tagged with project_id (threads, changes, notifications, actions, pr_resolution_branches, cached_prs) keep project_id — they carry a project-grain tag, stay consistent for N=1, and renaming them is out of S1 scope (deferred). This matches the goal's "re-key entry-scoped tables via the 028 rebuild pattern".

  3. Serving stays project-scoped in S1. The /c/:collectionId/ segment is introduced in routing + redirects; the frontend data layer keeps calling /api/projects/{project_id}/rfcs/... (the default collection). Collection-aware serving + the registry .collection.yaml reader land in S2.

  4. No Gherkin runner. @S1 acceptance is realised as pytest vertical tests + a frontend route test. The whole existing backend suite is the "N=1 unchanged" regression net — it must go green again after the rename.

File structure

Created:

  • backend/migrations/029_collections.sql — the migration (collections table, field move-down, default-collection seed, 13-table re-key, project_members → memberships).
  • backend/app/collections.py — collection resolution helpers (default_collection_id, collection_type, collection_initial_state, collections_of_project).
  • backend/tests/test_migration_029_collections.py — migration shape + data-preservation + FK tests (template: test_migration_028_project_scoped_keys.py).
  • backend/tests/test_s1_collection_grain_vertical.py — @S1 acceptance (C3.7 redirect to sole collection; default-collection redirect; N=1 serving unchanged).

Modified (backend):

  • backend/app/projects.py — restamp_default_project bootstrap check (cached_rfcs.project_id → a still-valid column); move project_initial_state to read the collection; add re-export shim if needed.
  • backend/app/auth.py — project_of_rfc joins collections; the 13-table reads/writes that touch project_id switch to collection_id.
  • backend/app/cache.py — _upsert_cached_rfc(..., collection_id) + the cached_rfcs/cached_branches writers + the WHERE project_id reconciler reads.
  • backend/app/api.py, api_prs.py, api_branches.py, api_notifications.py, api_contributions.py, api_invitations.py, api_graduation.py, funder.py — every SQL touching the 13 tables' project_id column → collection_id; recover project via collections join where authz needs it.
  • backend/app/api_deployment.py — /rfc/{slug} family 308 targets gain /c/default/; get_deployment/get_project read type/initial_state from the default collection.

Modified (frontend):

  • frontend/src/components/entryPaths.js (or wherever path builders live) — insert /c/:collectionId/.
  • frontend/src/App.jsx — add /c/:collectionId/* route layer; redirect /p/:projectId/ → sole/default collection (C3.7); redirect legacy /p/:projectId/e|proposals/... → /c/default/....
  • frontend/src/ProjectLayout.jsx (+ RFCView.jsx, Catalog.jsx as needed) — read :collectionId param; pass through (data stays project-scoped).

Modified (release):

  • VERSION, frontend/package.json#version, CHANGELOG.md — minor bump with breaking-URL upgrade-steps block (§20.2 / §20.4).

Phase 1 — Migration 029 (the collection grain)

Task 1: Write the migration-029 shape test (red)

Files:

  • Test: backend/tests/test_migration_029_collections.py

  • Step 1: Write the failing test

"""Migration 029 — collections grain beneath projects. Template: test_migration_028."""
import os
import sqlite3
import tempfile

import pytest

from app import db


class _Cfg:
    def __init__(self, path):
        self.database_path = path
        self.default_project_id = "default"


def _fresh_db():
    d = tempfile.mkdtemp()
    path = os.path.join(d, "test.db")
    db._CONN = None
    db.run_migrations(_Cfg(path))
    return db.conn()


def test_collections_table_exists_with_default_per_project():
    conn = _fresh_db()
    cols = {r["name"] for r in conn.execute("PRAGMA table_info(collections)")}
    assert {"id", "project_id", "type", "subfolder",
            "initial_state", "visibility", "name", "registry_sha"} <= cols
    # one default collection seeded for the bootstrap 'default' project
    row = conn.execute(
        "SELECT id, project_id, subfolder FROM collections WHERE project_id='default'"
    ).fetchone()
    assert row is not None
    assert row["id"] == "default"
    assert row["subfolder"] == ""  # repo root


def test_per_corpus_fields_moved_off_projects():
    conn = _fresh_db()
    proj_cols = {r["name"] for r in conn.execute("PRAGMA table_info(projects)")}
    assert "type" not in proj_cols
    assert "initial_state" not in proj_cols
    # projects keeps the grouping-tier fields
    assert {"id", "name", "content_repo", "visibility"} <= proj_cols


def test_entry_tables_rekeyed_to_collection_id():
    conn = _fresh_db()
    for t in ("cached_rfcs", "cached_branches", "stars", "watches",
              "rfc_collaborators", "contribution_requests", "proposed_use_cases",
              "branch_visibility", "branch_contribute_grants", "pr_seen",
              "branch_chat_seen", "funder_consents", "rfc_invitations"):
        cols = {r["name"] for r in conn.execute(f"PRAGMA table_info({t})")}
        assert "collection_id" in cols, f"{t} missing collection_id"
        assert "project_id" not in cols, f"{t} still has project_id"


def test_cached_rfcs_pk_is_collection_slug():
    conn = _fresh_db()
    # same slug coexists across two collections
    conn.execute("INSERT INTO collections (id, project_id, type, subfolder, initial_state, visibility, name) "
                 "VALUES ('c2','default','document','specs','active','public','Specs')")
    conn.execute("INSERT INTO cached_rfcs (slug, title, state, collection_id) VALUES ('intro','A','active','default')")
    conn.execute("INSERT INTO cached_rfcs (slug, title, state, collection_id) VALUES ('intro','B','active','c2')")
    n = conn.execute("SELECT COUNT(*) c FROM cached_rfcs WHERE slug='intro'").fetchone()["c"]
    assert n == 2
    with pytest.raises(sqlite3.IntegrityError):
        conn.execute("INSERT INTO cached_rfcs (slug, title, state, collection_id) VALUES ('intro','dup','active','default')")


def test_collaborator_fk_is_composite_on_collection():
    conn = _fresh_db()
    conn.execute("INSERT INTO collections (id, project_id, type, subfolder, initial_state, visibility, name) "
                 "VALUES ('c2','default','document','specs','active','public','Specs')")
    conn.execute("INSERT INTO cached_rfcs (slug, title, state, collection_id) VALUES ('intro','A','active','c2')")
    conn.execute("INSERT INTO users (id, email, role, permission_state) VALUES (1,'a@b.c','contributor','granted')")
    conn.execute("PRAGMA foreign_keys=ON")
    conn.execute("INSERT INTO rfc_collaborators (rfc_slug, user_id, role_in_rfc, collection_id) "
                 "VALUES ('intro',1,'contributor','c2')")
    with pytest.raises(sqlite3.IntegrityError):
        conn.execute("INSERT INTO rfc_collaborators (rfc_slug, user_id, role_in_rfc, collection_id) "
                     "VALUES ('intro',1,'contributor','default')")  # no such (collection,slug)


def test_memberships_table_replaces_project_members():
    conn = _fresh_db()
    cols = {r["name"] for r in conn.execute("PRAGMA table_info(memberships)")}
    assert {"scope_type", "scope_id", "user_id", "role", "granted_by", "granted_at"} <= cols
    # M2 rows would migrate to scope_type='collection'; role enum collapsed to owner/contributor
    # (no project_members rows exist in a fresh DB, so just assert the table + check constraint)
    conn.execute("INSERT INTO users (id, email, role, permission_state) VALUES (9,'x@y.z','contributor','granted')")
    conn.execute("INSERT INTO memberships (scope_type, scope_id, user_id, role) VALUES ('project','default',9,'owner')")
    with pytest.raises(sqlite3.IntegrityError):
        conn.execute("INSERT INTO memberships (scope_type, scope_id, user_id, role) VALUES ('bogus','default',9,'owner')")
  • Step 2: Run to verify it fails

Run: cd backend && python -m pytest tests/test_migration_029_collections.py -q Expected: FAIL (no collections table / 029_collections.sql does not exist).

  • Step 3: Commit the red test
git add backend/tests/test_migration_029_collections.py
git commit -m "§22 S1: failing migration-029 shape tests (collections grain)"

Task 2: Write migration 029 (green the shape test)

Files:

  • Create: backend/migrations/029_collections.sql

  • Step 1: Write the migration. Mirror 028_project_scoped_keys.sql exactly for the 13 rebuilds, with project_id renamed to collection_id in each __new table, each child FK re-pointed to cached_rfcs(collection_id, slug), and each index/UNIQUE swapping project_id→collection_id. Use the explicit-column INSERT ... SELECT form (not SELECT *) so the re-key can map values. Header marker -- migrate:no-foreign-keys. Concrete top of file:

-- migrate:no-foreign-keys
--
-- §22 three-tier refactor — S1. Insert a *collection* grain beneath project.
-- (1) collections table; (2) move per-corpus fields (type, initial_state) down
-- from projects; (3) one default collection per project (id='default',
-- subfolder = repo root); (4) re-key the 13 entry-corpus tables
-- (project_id,slug) -> (collection_id,slug) via the 028 rebuild pattern, mapping
-- each row to its project's default collection by JOIN; (5) project_members ->
-- memberships(scope_type ∈ {project,collection}, …), role enum collapsed to
-- {owner, contributor}. FK enforcement is OFF for the file (marker above);
-- foreign_key_check runs after. See docs/design/2026-06-05-three-tier-…md §A.6.

-- ── collections: the new typed-corpus grain beneath projects ───────────────
CREATE TABLE collections (
    id            TEXT NOT NULL,
    project_id    TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
    type          TEXT NOT NULL DEFAULT 'document'
                      CHECK (type IN ('document', 'specification', 'bdd')),
    subfolder     TEXT NOT NULL DEFAULT '',
    initial_state TEXT NOT NULL DEFAULT 'super-draft'
                      CHECK (initial_state IN ('super-draft', 'active')),
    visibility    TEXT NOT NULL DEFAULT 'gated'
                      CHECK (visibility IN ('gated', 'public', 'unlisted')),
    name          TEXT,
    registry_sha  TEXT,
    created_at    TEXT NOT NULL DEFAULT (datetime('now')),
    updated_at    TEXT NOT NULL DEFAULT (datetime('now')),
    PRIMARY KEY (id)
);
CREATE INDEX idx_collections_project ON collections(project_id);

-- One default collection per project. id='default' for the standard
-- single-project deployment (stable across deploy histories); the project_id is
-- used as a unique fallback id only if a non-standard multi-project deployment
-- migrates (pre-S5; avoids a PK collision). subfolder='' = repo root.
INSERT INTO collections (id, project_id, type, subfolder, initial_state, visibility, name)
SELECT
    CASE WHEN (SELECT COUNT(*) FROM projects) <= 1 THEN 'default' ELSE p.id END,
    p.id, p.type, '', p.initial_state, p.visibility, p.name
FROM projects p;

-- ── move per-corpus fields off projects (rebuild to DROP type/initial_state) ─
CREATE TABLE projects__new (
    id            TEXT PRIMARY KEY,
    name          TEXT NOT NULL,
    content_repo  TEXT,
    visibility    TEXT NOT NULL DEFAULT 'gated'
                      CHECK (visibility IN ('gated', 'public', 'unlisted')),
    config_json   TEXT,
    registry_sha  TEXT,
    created_at    TEXT NOT NULL DEFAULT (datetime('now')),
    updated_at    TEXT NOT NULL DEFAULT (datetime('now'))
);
INSERT INTO projects__new (id, name, content_repo, visibility, config_json, registry_sha, created_at, updated_at)
SELECT id, name, content_repo, visibility, config_json, registry_sha, created_at, updated_at FROM projects;
DROP TABLE projects;
ALTER TABLE projects__new RENAME TO projects;

-- ── cached_rfcs: PRIMARY KEY (project_id, slug) -> (collection_id, slug) ────
-- collection_id mapped from the row's old project's default collection.
CREATE TABLE cached_rfcs__new (
    slug                TEXT NOT NULL,
    title               TEXT NOT NULL,
    state               TEXT NOT NULL CHECK (state IN ('super-draft', 'active', 'withdrawn', 'retired')),
    rfc_id              TEXT,
    repo                TEXT,
    proposed_by         TEXT,
    proposed_at         TEXT,
    graduated_at        TEXT,
    graduated_by        TEXT,
    owners_json         TEXT NOT NULL DEFAULT '[]',
    arbiters_json       TEXT NOT NULL DEFAULT '[]',
    tags_json           TEXT NOT NULL DEFAULT '[]',
    body                TEXT,
    body_sha            TEXT,
    last_main_commit_at TEXT,
    last_entry_commit_at TEXT,
    updated_at          TEXT NOT NULL DEFAULT (datetime('now')),
    models_json         TEXT,
    funder_login        TEXT,
    proposed_use_case   TEXT,
    collection_id       TEXT NOT NULL DEFAULT 'default' REFERENCES collections(id),
    unreviewed          INTEGER NOT NULL DEFAULT 0,
    reviewed_at         TEXT,
    reviewed_by         TEXT,
    PRIMARY KEY (collection_id, slug)
);
INSERT INTO cached_rfcs__new
    (slug, title, state, rfc_id, repo, proposed_by, proposed_at, graduated_at,
     graduated_by, owners_json, arbiters_json, tags_json, body, body_sha,
     last_main_commit_at, last_entry_commit_at, updated_at, models_json,
     funder_login, proposed_use_case, collection_id, unreviewed, reviewed_at, reviewed_by)
SELECT
    r.slug, r.title, r.state, r.rfc_id, r.repo, r.proposed_by, r.proposed_at, r.graduated_at,
    r.graduated_by, r.owners_json, r.arbiters_json, r.tags_json, r.body, r.body_sha,
    r.last_main_commit_at, r.last_entry_commit_at, r.updated_at, r.models_json,
    r.funder_login, r.proposed_use_case,
    (SELECT c.id FROM collections c WHERE c.project_id = r.project_id LIMIT 1),
    r.unreviewed, r.reviewed_at, r.reviewed_by
FROM cached_rfcs r;
DROP TABLE cached_rfcs;
ALTER TABLE cached_rfcs__new RENAME TO cached_rfcs;
CREATE INDEX idx_cached_rfcs_state ON cached_rfcs (state);
CREATE INDEX idx_cached_rfcs_last_active ON cached_rfcs (
    COALESCE(last_main_commit_at, last_entry_commit_at) DESC
);
CREATE INDEX idx_cached_rfcs_collection ON cached_rfcs(collection_id);

Then for each of the remaining 12 tables copy its 028 block verbatim and apply the same three transforms: (a) rename the project_id column to collection_id (keep DEFAULT 'default'); (b) in the INSERT ... SELECT, replace the project_id source value with (SELECT c.id FROM collections c WHERE c.project_id = <old>.project_id LIMIT 1) and list columns explicitly; (c) rename project_id → collection_id in every UNIQUE (...), FOREIGN KEY (...) REFERENCES cached_rfcs(...), and CREATE [UNIQUE] INDEX. The 12: rfc_invitations, cached_branches, branch_visibility, branch_contribute_grants, stars, watches, pr_seen, branch_chat_seen, funder_consents, rfc_collaborators, contribution_requests, proposed_use_cases. (FK targets cached_rfcs(project_id, slug) become cached_rfcs(collection_id, slug).)

Finally the membership generalisation:

-- ── project_members -> memberships(scope_type, scope_id, …); roles collapsed ─
CREATE TABLE memberships (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    scope_type  TEXT NOT NULL CHECK (scope_type IN ('project', 'collection')),
    scope_id    TEXT NOT NULL,
    user_id     INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    role        TEXT NOT NULL CHECK (role IN ('owner', 'contributor')),
    granted_by  INTEGER REFERENCES users(id) ON DELETE SET NULL,
    granted_at  TEXT NOT NULL DEFAULT (datetime('now')),
    UNIQUE (scope_type, scope_id, user_id)
);
CREATE INDEX idx_memberships_user ON memberships(user_id);
CREATE INDEX idx_memberships_scope ON memberships(scope_type, scope_id);

-- M2 project_members rows attached at what is now the *collection*; collapse
-- the role enum (project_admin -> owner, project_contributor -> contributor,
-- project_viewer -> dropped this pass, §B.3) and migrate to the default
-- collection of each project.
INSERT INTO memberships (scope_type, scope_id, user_id, role, granted_by, granted_at)
SELECT 'collection',
       (SELECT c.id FROM collections c WHERE c.project_id = pm.project_id LIMIT 1),
       pm.user_id,
       CASE pm.role WHEN 'project_admin' THEN 'owner'
                    WHEN 'project_contributor' THEN 'contributor'
                    ELSE 'contributor' END,
       pm.granted_by, pm.granted_at
FROM project_members pm
WHERE pm.role IN ('project_admin', 'project_contributor');
DROP TABLE project_members;
  • Step 2: Run the shape test

Run: cd backend && python -m pytest tests/test_migration_029_collections.py -q Expected: PASS (all shape/PK/FK/membership assertions green).

  • Step 3: Commit
git add backend/migrations/029_collections.sql
git commit -m "§22 S1: migration 029 — collections grain, field move-down, 13-table re-key, memberships"

Phase 2 — Backend threading (make the existing suite green again)

After Task 2 the column rename breaks every reader/writer of the 13 tables. This phase fixes them. Driver: the full backend suite is the regression net — run it, read each failure, fix the named module, repeat until green. The agent exploration produced the exact blast-radius map used below.

Task 3: collections helper module

Files:

  • Create: backend/app/collections.py

  • Step 1: Write the helper

"""§22 collection grain — resolution helpers beneath the project tier.

In S1 each project has exactly one collection (the default). These helpers
recover the collection for a project and read the per-corpus fields that moved
down from `projects` in migration 029. Project-grain authz (auth.py) recovers a
row's project by joining `collections` on `collection_id`.
"""
from __future__ import annotations

from . import db

DEFAULT_COLLECTION_ID = "default"


def default_collection_id(project_id: str) -> str:
    """The id of a project's default (S1: sole) collection. Falls back to the
    literal 'default' when the project has no collection row yet."""
    row = db.conn().execute(
        "SELECT id FROM collections WHERE project_id = ? ORDER BY created_at LIMIT 1",
        (project_id,),
    ).fetchone()
    return row["id"] if row else DEFAULT_COLLECTION_ID


def project_of_collection(collection_id: str) -> str | None:
    row = db.conn().execute(
        "SELECT project_id FROM collections WHERE id = ?", (collection_id,)
    ).fetchone()
    return row["project_id"] if row else None


def collection_initial_state(collection_id: str) -> str:
    """§22.4b landing state for new entries in a collection. 'super-draft'
    default for an unknown row (today's safe flow)."""
    row = db.conn().execute(
        "SELECT initial_state FROM collections WHERE id = ?", (collection_id,)
    ).fetchone()
    if row is None or not row["initial_state"]:
        return "super-draft"
    return row["initial_state"]


def collection_type(collection_id: str) -> str:
    row = db.conn().execute(
        "SELECT type FROM collections WHERE id = ?", (collection_id,)
    ).fetchone()
    return row["type"] if row and row["type"] else "document"
  • Step 2: Commit
git add backend/app/collections.py
git commit -m "§22 S1: collections resolution helpers"

Task 4: Fix projects.py (restamp + initial_state)

Files:

  • Modify: backend/app/projects.py:46-48 (restamp bootstrap check), :112-121 (project_initial_state)

  • Step 1: Fix the restamp bootstrap check. restamp_default_project reads cached_rfcs.project_id (now renamed) at line 47 — switch the existence probe to a still-project_id-bearing table so the PRAGMA-driven rename loop is unaffected (it already discovers project_id columns dynamically, which now correctly excludes the 13 collection-keyed tables and includes collections.project_id):

    has_rows = conn.execute(
        "SELECT 1 FROM collections WHERE project_id = ? LIMIT 1", (DEFAULT_PROJECT_ID,)
    ).fetchone()
  • Step 2: Re-home project_initial_state. Keep the signature for callers, but resolve through the project's default collection:
def project_initial_state(project_id: str) -> str:
    """§22.4b landing state for new entries in a project's default collection."""
    from . import collections as collections_mod
    return collections_mod.collection_initial_state(
        collections_mod.default_collection_id(project_id)
    )
  • Step 3: Run the restamp + projects tests

Run: cd backend && python -m pytest tests/test_restamp_default_project.py tests/test_initial_state_landing.py -q Expected: PASS.

  • Step 4: Commit
git add backend/app/projects.py
git commit -m "§22 S1: thread projects.py restamp + initial_state through collections"

Task 5: Fix auth.py (project_of_rfc join)

Files:

  • Modify: backend/app/auth.py:352-361

  • Step 1: Join collections to recover the project from a slug.

def project_of_rfc(rfc_slug: str) -> str:
    """The project an RFC belongs to, via its collection
    (cached_rfcs.collection_id -> collections.project_id). Falls back to the
    default project when the slug isn't cached."""
    row = db.conn().execute(
        "SELECT c.project_id AS project_id "
        "FROM cached_rfcs r JOIN collections c ON c.id = r.collection_id "
        "WHERE r.slug = ?",
        (rfc_slug,),
    ).fetchone()
    if row is None:
        return DEFAULT_PROJECT_ID
    return row["project_id"] or DEFAULT_PROJECT_ID
  • Step 2: Run the authz suite

Run: cd backend && python -m pytest tests/test_multi_project_authz_vertical.py tests/test_anon_offlimits_vertical.py -q Expected: PASS.

  • Step 3: Commit
git add backend/app/auth.py
git commit -m "§22 S1: auth.project_of_rfc recovers project via collection join"

Task 6: Fix cache.py writers/readers

Files:

  • Modify: backend/app/cache.py — _refresh_project_corpus (resolve collection), _upsert_cached_rfc signature + SQL (project_id→collection_id), the WHERE project_id reconciler read (:88), the cached_branches writers (:213/:388/:410).

  • Step 1: Resolve the collection in the corpus refresh. In _refresh_project_corpus, compute the project's default collection once and pass it down; switch the reconciler SELECT slug ... WHERE project_id to WHERE collection_id:

async def _refresh_project_corpus(org: str, project_id: str, repo: str, gitea: Gitea) -> None:
    from . import collections as collections_mod
    collection_id = collections_mod.default_collection_id(project_id)
    ...
        _upsert_cached_rfc(entry, body_sha=sha, collection_id=collection_id)
    ...
    existing = {
        row["slug"]
        for row in db.conn().execute(
            "SELECT slug FROM cached_rfcs WHERE collection_id = ?", (collection_id,)
        )
    }

  • Step 2: Rename in _upsert_cached_rfc. Change the param project_id: str = "default" → collection_id: str = "default"; in the INSERT, replace the project_id column with collection_id, the ON CONFLICT(project_id, slug) with ON CONFLICT(collection_id, slug), and the bound value project_id → collection_id.

  • Step 3: Fix the cached_branches writers. At :213/:388/:410 the ON CONFLICT(project_id, rfc_slug, branch_name) clauses → ON CONFLICT(collection_id, rfc_slug, branch_name); where a meta-repo branch row is written without an explicit grain it now relies on the collection_id DEFAULT 'default' column default (unchanged behaviour for N=1). Bind collection_id explicitly where the per-project loop has it.

  • Step 4: Run the cache tests

Run: cd backend && python -m pytest tests/test_cache_bootstrap.py tests/test_cache_review_fields.py tests/test_branch_path_routing.py -q Expected: PASS.

  • Step 5: Commit
git add backend/app/cache.py
git commit -m "§22 S1: thread cache.py corpus/branch writers through collection_id"

Task 7: Fix the api_* writers/readers + funder.py

Files (each: swap the 13-table project_id column references to collection_id; recover project for authz via auth.project_of_rfc/collections join):

  • backend/app/api.py:747 (stars read), :774/:806/:969 (cached_rfcs composite lookups → collection_id), :785-788/:1046 (proposed_use_cases).

  • backend/app/api_prs.py:156 (branch_visibility), :192 (proposed_use_cases), :401 (pr_seen). Note :670/:789 read row["project_id"] from a cached_rfcs/rfc row — change those SELECTs to also yield the project via the collection join, then keep the existing auth.require_project_readable(viewer, project_id) call unchanged.

  • backend/app/api_branches.py:745 (branch_visibility), :899 (branch_chat_seen).

  • backend/app/api_notifications.py:216 (read cached_rfcs → now collection_id; recover project via join for the visibility gate), :225 (watches).

  • backend/app/api_contributions.py:65/:111 (read cached_rfcs; recover project via join), api_invitations.py:383, api_graduation.py (any cached_rfcs/13-table project_id).

  • backend/app/funder.py:223 (funder_consents ON CONFLICT(project_id,…) → collection_id).

  • Step 1: Mechanical pass. For each file above, replace project_id only where it names a column on one of the 13 re-keyed tables (PK lookups, ON CONFLICT, WHERE, INSERT column lists, SELECT projections from those tables) with collection_id. Where the code needs the project (for auth.*_project* calls), recover it with auth.project_of_rfc(slug) or a collections join — do not rename the project_id argument flowing into the authz helpers (those stay project-grain in S1). Leave threads, changes, notifications, actions, pr_resolution_branches, cached_prs project_id columns untouched.

  • Step 2: Grep guard. Confirm no stray reference to a dropped column remains:

Run: cd backend && grep -rEn "cached_rfcs[^;]*project_id|project_id, slug|project_id, rfc_slug|ON CONFLICT\(project_id" app/ | grep -v "collections\|threads\|changes\|notifications\|actions\|pr_resolution\|cached_prs" Expected: no output (every 13-table project_id is now collection_id).

  • Step 3: Run the full backend suite

Run: cd backend && python -m pytest -q Expected: PASS (this is the N=1-unchanged gate). Fix any remaining failures by reading the traceback and applying the same rename/join rule.

  • Step 4: Commit
git add backend/app/api.py backend/app/api_prs.py backend/app/api_branches.py backend/app/api_notifications.py backend/app/api_contributions.py backend/app/api_invitations.py backend/app/api_graduation.py backend/app/funder.py
git commit -m "§22 S1: thread api_* + funder writers/readers through collection_id"

Phase 3 — API surface reads per-corpus fields from the collection

Task 8: api_deployment.py reads type/initial_state from the default collection

Files:

  • Modify: backend/app/api_deployment.py:36-44 (get_deployment projects list type), :62-82 (get_project type/initial_state)

  • Step 1: Write a failing test in backend/tests/test_api_deployment.py (extend it) asserting GET /api/projects/{default} still returns the correct type/initial_state after the move-down (values come from the default collection):

def test_get_project_type_initial_state_from_default_collection(app_with_fake_gitea):
    # ... existing fixture sets up the default project/collection ...
    r = client.get(f"/api/projects/{default_id}")
    assert r.status_code == 200
    body = r.json()
    assert body["type"] in ("document", "specification", "bdd")
    assert body["initial_state"] in ("super-draft", "active")
  • Step 2: Run to verify it fails (the SELECT still reads projects.type, which 029 dropped → OperationalError).

Run: cd backend && python -m pytest tests/test_api_deployment.py -q Expected: FAIL.

  • Step 3: Read the fields from the default collection. In get_deployment, replace the SELECT id, name, type, visibility FROM projects with a join to the project's default collection for type (or a per-row collections_mod.collection_type(default_collection_id(id))). In get_project, drop type, initial_state from the projects SELECT and resolve them via collections_mod.collection_type(...) / collection_initial_state(...):
from . import collections as collections_mod
...
    cid = collections_mod.default_collection_id(row["id"])
    return {
        ...
        "type": collections_mod.collection_type(cid),
        "initial_state": collections_mod.collection_initial_state(cid),
        ...
    }
  • Step 4: Run to verify it passes

Run: cd backend && python -m pytest tests/test_api_deployment.py -q Expected: PASS.

  • Step 5: Commit
git add backend/app/api_deployment.py backend/tests/test_api_deployment.py
git commit -m "§22 S1: deployment/project API reads type+initial_state from default collection"

Phase 4 — Redirects + frontend collection segment

Task 9: Backend /rfc/ 308s target /c/default/

Files:

  • Modify: backend/app/api_deployment.py:89-104

  • Step 1: Add a failing test to test_api_deployment.py:

def test_legacy_rfc_url_redirects_through_collection(app_with_fake_gitea):
    r = client.get("/rfc/intro", follow_redirects=False)
    assert r.status_code == 308
    assert r.headers["location"] == f"/p/{default_id}/c/default/e/intro"

  • Step 2: Verify it fails (current target lacks /c/default/).

  • Step 3: Update the three redirect handlers to resolve the default collection and insert the /c/<cid>/ segment:

    @router.get("/rfc/{slug}")
    async def redirect_old_rfc(slug: str) -> RedirectResponse:
        default_id = projects_mod.resolved_default_id(config)
        cid = collections_mod.default_collection_id(default_id)
        return RedirectResponse(url=f"/p/{default_id}/c/{cid}/e/{slug}", status_code=308)
    # …same /c/{cid}/ insertion for /rfc/{slug}/pr/{pr} and /proposals/{pr}

(/proposals/{pr} → /p/{default_id}/c/{cid}/proposals/{pr}.)

  • Step 4: Verify it passes.

Run: cd backend && python -m pytest tests/test_api_deployment.py -q Expected: PASS.

  • Step 5: Commit
git add backend/app/api_deployment.py backend/tests/test_api_deployment.py
git commit -m "§22 S1: legacy /rfc + /proposals 308s route through /c/<default>/"

Task 10: Frontend path builders gain /c/:collectionId/

Files:

  • Modify: frontend/src/components/entryPaths.js (path builders — confirm exact path with grep -rl "p/\${" frontend/src)

  • Step 1: Thread a collection id through the builders. Add a collectionId argument (defaulting to 'default') and emit the /c/<collectionId>/ segment:

export const collectionHome = (projectId, collectionId) => `/p/${projectId}/c/${collectionId}/`
export const entryPath = (projectId, collectionId, slug) => `/p/${projectId}/c/${collectionId}/e/${slug}`
export const entryPrPath = (projectId, collectionId, slug, prNumber) => `/p/${projectId}/c/${collectionId}/e/${slug}/pr/${prNumber}`
export const proposalPath = (projectId, collectionId, prNumber) => `/p/${projectId}/c/${collectionId}/proposals/${prNumber}`
export const projectHome = (projectId) => `/p/${projectId}/`

Update every caller (grep entryPath(, entryPrPath(, proposalPath(, collectionHome() to pass the current collection id (from the route param / useCollectionId() — default 'default').

  • Step 2: Build the frontend

Run: cd frontend && npm run build Expected: build succeeds (no undefined-symbol errors).

  • Step 3: Commit
git add frontend/src
git commit -m "§22 S1: frontend path builders carry the /c/<collection>/ segment"

Task 11: Frontend route layer + redirects (C3.7, C3.8, legacy)

Files:

  • Modify: frontend/src/App.jsx:354-373, frontend/src/ProjectLayout.jsx

  • Step 1: Nest the corpus routes under /c/:collectionId/ and add redirects. Inside the ProjectLayout nested <Routes>:

<Routes>
  {/* project landing: redirect to the sole/default collection (C3.7) */}
  <Route path="" element={<CollectionRedirect />} />
  {/* legacy v0.35.0 corpus URLs without /c/ → default collection */}
  <Route path="e/:slug" element={<Navigate to="c/default/e/:slug" replace />} />
  <Route path="e/:slug/pr/:prNumber" element={<LegacyEntryPrRedirect />} />
  <Route path="proposals/:prNumber" element={<LegacyProposalRedirect />} />
  {/* collection-scoped corpus (serving stays project-scoped in S1) */}
  <Route path="c/:collectionId" element={<Welcome viewer={viewer} />} />
  <Route path="c/:collectionId/e/:slug" element={<RFCView viewer={viewer} />} />
  <Route path="c/:collectionId/e/:slug/pr/:prNumber" element={<PRView viewer={viewer} />} />
  <Route path="c/:collectionId/proposals/:prNumber" element={<ProposalView viewer={viewer} onChange={() => setCatalogVersion(v => v + 1)} />} />
</Routes>

CollectionRedirect reads the project's collections (from ProjectContext, populated by GET /api/projects/:id) and <Navigate>s to the sole visible collection's /c/<id>/; with one collection that is /c/default/ (C3.7). LegacyEntryPrRedirect/LegacyProposalRedirect use useParams() to rebuild the target with c/default/. React-Router literal :slug in to= does not interpolate — implement these as small components using useParams() + <Navigate>.

  • Step 2: Confirm / → sole project (C3.8) already holds. DeploymentLanding (App.jsx:348) already redirects to the single visible project. Add/confirm a test (Task 12) rather than re-implementing.

  • Step 3: Build

Run: cd frontend && npm run build Expected: succeeds.

  • Step 4: Commit
git add frontend/src
git commit -m "§22 S1: /c/<collection>/ route layer + C3.7 + legacy-URL redirects"

Phase 5 — @S1 acceptance + full verification

Task 12: @S1 vertical acceptance test (C3.7 + C3.8 + N=1 serving)

Files:

  • Create: backend/tests/test_s1_collection_grain_vertical.py

  • Step 1: Write the acceptance test. Tag scenarios in docstrings as @S1 for traceability (no Gherkin runner). Cover: (a) default-collection redirect /rfc/<slug> → /p/<default>/c/default/e/<slug> (already in Task 9 — re-assert here as the S1 gate); (b) an entry proposed/served at N=1 still resolves under the default collection via /api/projects/<default>/rfcs/<slug>; (c) the deployment /api/deployment still reports one project with default_project_id. (C3.7/C3.8 client redirects are asserted in the frontend build/route smoke; the data-layer N=1 invariants are asserted here.)

"""@S1 acceptance — the collection grain exists and N=1 is unchanged.
Scenarios: C3.7 (single-collection project skips the directory) and C3.8
(single-project deployment skips the directory) are the redirect contract;
this module asserts the backend N=1 invariants behind them."""
# reuse the propose/serve fixtures from test_project_scoped_serving.py
def test_s1_entry_served_under_default_collection(app_with_fake_gitea):
    # propose + mirror an entry, then fetch it project-scoped (collection=default)
    ...
    r = client.get(f"/api/projects/{default_id}/rfcs/intro")
    assert r.status_code == 200
    # the row is keyed by collection_id under the hood
    cid = db.conn().execute("SELECT collection_id FROM cached_rfcs WHERE slug='intro'").fetchone()["collection_id"]
    assert cid == "default"

def test_s1_legacy_redirect_inserts_collection_segment(app_with_fake_gitea):
    r = client.get("/rfc/intro", follow_redirects=False)
    assert r.status_code == 308
    assert "/c/default/" in r.headers["location"]
  • Step 2: Run it

Run: cd backend && python -m pytest tests/test_s1_collection_grain_vertical.py -q Expected: PASS.

  • Step 3: Full backend suite + frontend build (the N=1-unchanged gate)

Run: cd backend && python -m pytest -q && cd ../frontend && npm run build Expected: all backend tests PASS; frontend builds.

  • Step 4: Commit
git add backend/tests/test_s1_collection_grain_vertical.py
git commit -m "§22 S1: @S1 acceptance — collection grain + N=1 serving unchanged"

Task 13: e2e smoke (optional, if Docker stack available)

  • Step 1: If the Tier-1 Docker stack is runnable, make e2e to confirm sign-in + a corpus page render through the new /c/default/ routes. If the stack isn't available in-session, note it skipped and rely on Tasks 7/11/12 gates.

Phase 6 — Release + finalize

Task 14: Version bump + changelog (breaking, with upgrade steps)

Files:

  • Modify: VERSION, frontend/package.json (version), CHANGELOG.md

  • Step 1: Bump VERSION and frontend/package.json#version to the next pre-1.0 minor (current 0.39.0 → 0.40.0). They must match (a divergence is a §20 spec bug).

  • Step 2: Add the CHANGELOG entry with a breaking-URL upgrade steps block (§20.2 / §20.4 / §A.6): migration 029 adds the collection grain; /p/<project>/e/<slug> now lives at /p/<project>/c/default/e/<slug> (308 for old links); operators need no action beyond deploying (the migration + redirects are automatic; the default collection is seeded). Note the deferred items (denormalised project_id tags unchanged; collection-aware serving = S2).

  • Step 3: Commit

git add VERSION frontend/package.json CHANGELOG.md
git commit -m "§22 S1: release v0.40.0 — three-tier collection grain (breaking URL + migration 029)"

Task 15: Branch, PR, merge

  • Step 1: This work rides a feature branch off main (e.g. feat/s1-collection-grain). Push to origin (git.wiggleverse.org).
  • Step 2: Open a PR citing the design doc + @S1; in autonomous posture, self-review and merge once the suite is green.
  • Step 3: Update repo memory with the new resume pointer (S1 shipped @ v0.40.0; next = S2).

Self-review (writing-plans checklist)

  • Spec coverage: §A.6 steps 1–5 → Tasks 2 (collections table + default + re-key + memberships), 8 (field move-down read path), 9 (308 step 5). Part B membership generalisation → Task 2 (memberships) — note S1 only migrates the table; the four-layer resolver is S3 (out of scope, correctly deferred per Part E). @S1 C3.7/C3.8 → Tasks 11 (frontend redirects) + 12 (backend invariants). "N=1 unchanged" → Task 7 Step 3 + Task 12 Step 3 full-suite gates. Threading (auth/projects/cache/api_*) → Tasks 4–8.
  • Placeholders: the per-table rebuild bodies for the 12 non-cached_rfcs tables reference the in-repo 028_project_scoped_keys.sql as the literal template with the three explicit transforms named — this is a concrete instruction, not a TODO (repeating 200+ lines of near-identical SQL verbatim would harm reviewability; the transform rule is exact).
  • Type consistency: default_collection_id, collection_type, collection_initial_state, project_of_collection are defined in Task 3 and used consistently in Tasks 4, 8, 9. Column collection_id (not coll_id/collectionId) used uniformly in SQL; collectionId is the JS route param.
  • Risk note: the denormalised project_id columns (threads/changes/notifications/actions/pr_resolution_branches/cached_prs) stay project_id and may, after restamp, hold the project id (ohm) while entry collection_id holds default. Task 7 Step 3's full-suite run is the guard against any code that wrongly cross-joins the two grains; if one surfaces, recover the project via the collections join rather than renaming the tag.
Reference in New Issue View Git Blame Copy Permalink