Pipeline & provenance

Where this data comes from

SA Policy Space claims to be a research tool. Research tools earn that word by showing their work. This page is that work — every step from raw PMG committee record to what you see on an idea page, plus the scripts that run the pipeline and the tests that catch drift.

What's in the database right now

171

Policy ideas

Committees tracked

3,099

Meetings indexed

779

Idea↔meeting links

533

Meetings (linked only)

3,810

Documents (linked)

Snapshot generated 2026-05-11 · latest PMG meeting indexed 2026-05-08 · earliest 1998-03-15.

The pipeline

1
PMG ingest
pmg.org.za
scripts/update_meetings.py pulls new committee meetings and associated documents from the Parliamentary Monitoring Group API for every tracked committee (scripts/committee_catalog.py). Weekly cron at 06:00 UTC Monday.
2
Score & classify
build_pmg_idea_review.py
Each meeting is scored against the current idea set by token overlap + binding-constraint match + committee match. Meetings that don't match any existing idea are checked for new-idea signal (keyword density + committee fit).
3
Auto-apply + human review
auto_link_and_seed.py
Meetings above the auto-link threshold are linked to ideas directly in Supabase. Medium-confidence matches are written to a migration SQL and opened as a PR for a human (Laurence) to review. Residual ambiguous items go into a GitHub issue queue. See docs/PMG_AUTO_PIPELINE.md.
4
Snapshot → SQLite
dump_supabase_snapshot.py
Supabase is dumped to data/pmg_snapshot.json (committed), covering all policy ideas, plans, comparisons, and the subset of meetings + documents linked to ideas. The snapshot seeds dev.sqlite3 for local dev and CI, so local/prod parity is enforced.
5
Drift-guard CI
__tests__/data/readme-claims.test.ts
Every CI run asserts that public-facing counts (README, layout metadata, OG image, OpenAPI) match the committed snapshot. If Supabase grows past what the site claims, the build fails and forces reconciliation.
6
Briefing
draft_weekly_briefing.py
Same pipeline feeds the weekly editorial briefing (draft auto-generated from the snapshot diff, human-edited, published as MDX at /briefings/<week>).

Downloads

pmg_snapshot.json

Full Supabase dump — all 171 ideas, 53 plans, 65 international comparisons, 778 idea↔meeting links, plus all linked meetings and documents. Committed to the repo; refreshes weekly after the Monday PMG sync. ~1.6 MB.

Raw JSON →

OpenAPI 3.0 spec (openapi.json)

Contract for the versioned public API at /api/v1/. Inputs are zod-validated; every response uses the { version, data, meta } envelope.

OpenAPI →

Raw JSON data files

Reform packages, stakeholder stances, fiscal estimates, indicator series, international comparisons — all committed and versioned.

Browse on GitHub →

How to audit

Every claim on the site is traceable to a source. Here's how to verify the most-common categories:

Meeting link. Every idea page lists the PMG meetings that mention it, with the PMG URL. Click through and read the committee record directly.
Counts in this site's copy. The __tests__/data/readme-claims.test.ts suite asserts README / metadata / OpenAPI counts match the committed snapshot. If any of them drift past a threshold, the build fails — check recent Actions runs.
SQLite ↔ Supabase parity. Eight golden queries in __tests__/parity/backend-parity.test.ts run against both backends in CI and assert the results match. Catches the "query-behaviour divergence" class of bug before it ships.
Reproducing the snapshot locally. With Supabase read credentials, python scripts/dump_supabase_snapshot.py rebuilds the snapshot from scratch. The diff against the committed copy should be empty (mod timestamps).

Citation

APA

Wilse-Samson, L. (2026). SA Policy Space: A Database of South African Reform Ideas. NYU Wagner School of Public Policy. https://sa-policy-space.vercel.app

When citing a specific idea or briefing, include the URL and the snapshot date shown on the page footer. Each issue of Briefings carries Article JSON-LD and a stable permalink for citation.

The pipeline

PMG ingest

pmg.org.za

scripts/update_meetings.py pulls new committee meetings and associated documents from the Parliamentary Monitoring Group API for every tracked committee (scripts/committee_catalog.py). Weekly cron at 06:00 UTC Monday.

Score & classify

build_pmg_idea_review.py

Each meeting is scored against the current idea set by token overlap + binding-constraint match + committee match. Meetings that don't match any existing idea are checked for new-idea signal (keyword density + committee fit).

Auto-apply + human review

auto_link_and_seed.py

Meetings above the auto-link threshold are linked to ideas directly in Supabase. Medium-confidence matches are written to a migration SQL and opened as a PR for a human (Laurence) to review. Residual ambiguous items go into a GitHub issue queue. See docs/PMG_AUTO_PIPELINE.md.

Snapshot → SQLite

dump_supabase_snapshot.py

Supabase is dumped to data/pmg_snapshot.json (committed), covering all policy ideas, plans, comparisons, and the subset of meetings + documents linked to ideas. The snapshot seeds dev.sqlite3 for local dev and CI, so local/prod parity is enforced.

Drift-guard CI

__tests__/data/readme-claims.test.ts

Every CI run asserts that public-facing counts (README, layout metadata, OG image, OpenAPI) match the committed snapshot. If Supabase grows past what the site claims, the build fails and forces reconciliation.

Briefing

draft_weekly_briefing.py

Same pipeline feeds the weekly editorial briefing (draft auto-generated from the snapshot diff, human-edited, published as MDX at /briefings/<week>).

How to audit

Every claim on the site is traceable to a source. Here's how to verify the most-common categories:

Meeting link. Every idea page lists the PMG meetings that mention it, with the PMG URL. Click through and read the committee record directly.
Counts in this site's copy. The __tests__/data/readme-claims.test.ts suite asserts README / metadata / OpenAPI counts match the committed snapshot. If any of them drift past a threshold, the build fails — check recent Actions runs.
SQLite ↔ Supabase parity. Eight golden queries in __tests__/parity/backend-parity.test.ts run against both backends in CI and assert the results match. Catches the "query-behaviour divergence" class of bug before it ships.
Reproducing the snapshot locally. With Supabase read credentials, python scripts/dump_supabase_snapshot.py rebuilds the snapshot from scratch. The diff against the committed copy should be empty (mod timestamps).

Citation

APA

Wilse-Samson, L. (2026). SA Policy Space: A Database of South African Reform Ideas. NYU Wagner School of Public Policy. https://sa-policy-space.vercel.app

When citing a specific idea or briefing, include the URL and the snapshot date shown on the page footer. Each issue of Briefings carries Article JSON-LD and a stable permalink for citation.