Pipeline & provenance
SA Policy Space claims to be a research tool. Research tools earn that word by showing their work. This page is that work — every step from raw PMG committee record to what you see on an idea page, plus the scripts that run the pipeline and the tests that catch drift.
Snapshot generated 2026-08-24 · latest PMG meeting indexed 2026-08-21 · earliest 1998-03-15.
The totals above hide an uneven picture, so here it is per committee. Two different things can make a row look thin, and they mean opposite things for how you read the site. A stale ingest is a gap in the record — no meeting from that committee has entered the corpus in over 6 months, either because the committee no longer sits (the 7th Parliament dissolved and merged several) or because our ingest is not reaching it. Our silence about it is not evidence that the committee has been silent. A linking gap is the opposite: the meetings are here and current, but no policy idea has been linked to a recent one. That is a backlog on our side, and those meetings are searchable in the corpus regardless.
| Committee | Corpus meetings | Idea-linked | Ideas sourced |
|---|---|---|---|
| Basic EducationCurrent | 1532021-11-09 → 2026-08-21 | 88latest 2026-08-04 | 10 |
| Mineral ResourcesLinks lagging | 1512021-11-09 → 2026-08-21 | 26latest 2024-10-29 | 8 |
| Employment and LabourLinks lagging | 672024-07-09 → 2026-08-19 | 3latest 2024-10-23 | 10 |
| Higher Education and TrainingCurrent | 5852009-05-27 → 2026-08-19 | 63latest 2026-06-10 | 8 |
| Human SettlementsCurrent | 1482021-08-05 → 2026-08-19 | 3latest 2026-06-03 | 0 |
| PoliceLinks lagging | 582024-07-10 → 2026-08-19 | 4latest 2024-10-30 | 8 |
| Small Business DevelopmentCurrent | 1282021-11-10 → 2026-08-19 | 82latest 2026-06-24 | 10 |
| Trade, Industry and CompetitionLinks lagging | 2262021-11-09 → 2026-08-19 | 51latest 2024-09-04 | 20 |
| Communications and Digital TechnologiesLinks lagging | 1042024-07-09 → 2026-08-18 | 2latest 2024-12-04 | 3 |
| Finance Standing CommitteeLinks lagging | 2412021-11-09 → 2026-08-18 | 24latest 2024-08-21 | 8 |
| Home AffairsNo links yet | 722024-07-09 → 2026-08-18 | 0no links | 2 |
| Science, Technology and InnovationCurrent | 552024-07-10 → 2026-08-18 | 25latest 2026-08-11 | 10 |
| Water and SanitationCurrent | 682024-07-09 → 2026-08-18 | 45latest 2026-08-04 | 11 |
| EnergyLinks lagging | 792024-07-10 → 2026-08-12 | 14latest 2024-12-04 | 15 |
| HealthCurrent | 7591998-03-15 → 2026-08-12 | 82latest 2026-03-04 | 10 |
| Public Works and InfrastructureCurrent | 1292021-11-10 → 2026-08-12 | 96latest 2026-08-12 | 8 |
| Forestry, Fisheries and the EnvironmentCurrent | 622024-07-09 → 2026-08-11 | 18latest 2026-08-11 | 7 |
| Justice and Constitutional DevelopmentLinks lagging | 702024-07-09 → 2026-08-11 | 3latest 2024-10-29 | 6 |
| TourismNo links yet | 492024-07-10 → 2026-08-11 | 0no links | 0 |
| TransportLinks lagging | 1482021-11-09 → 2026-08-11 | 87latest 2024-10-29 | 11 |
| Finance Select Committee (NCOP)Current | 1002021-11-09 → 2026-08-05 | 30latest 2026-08-05 | 6 |
| Public EnterprisesIngest stale | 642021-11-10 → 2024-03-27 | 13latest 2022-06-01 | 8 |
| Human Settlements, Water and SanitationIngest stale | 5851999-03-08 → 2021-07-27 | 0no links | 11 |
Computed over the full meetings table when the snapshot was dumped (2026-08-24); staleness is measured against that date, not against your clock. "Ideas sourced" counts policy ideas whose recorded source committee names this one, matched the way the auto-linker matches — an idea sourced to a merged department (“Mineral Resources and Energy”) counts under both successors, so the column sums above 171. Rows are keyed on the committee a meeting was ingested under, while committee pages are keyed on the committee an idea is sourced to — the two label sets do not line up one-for-one.
scripts/update_meetings.py pulls new committee meetings and associated documents from the Parliamentary Monitoring Group API for every tracked committee (scripts/committee_catalog.py). Weekly cron at 06:00 UTC Monday.
Each meeting is scored against the current idea set by token overlap + binding-constraint match + committee match. Meetings that don't match any existing idea are checked for new-idea signal (keyword density + committee fit).
Meetings above the auto-link threshold are linked to ideas directly in Supabase. Medium-confidence matches are written to a migration SQL and opened as a PR for a human (Laurence) to review. Residual ambiguous items go into a GitHub issue queue. See docs/PMG_AUTO_PIPELINE.md.
Supabase is dumped to data/pmg_snapshot.json (committed), covering all policy ideas, plans, comparisons, and the subset of meetings + documents linked to ideas. The snapshot seeds dev.sqlite3 for local dev and CI, so local/prod parity is enforced.
Every CI run asserts that public-facing counts (README, layout metadata, OG image, OpenAPI) match the committed snapshot. If Supabase grows past what the site claims, the build fails and forces reconciliation.
Same pipeline feeds the weekly editorial briefing (draft auto-generated from the snapshot diff, human-edited, published as MDX at /briefings/<week>).
/api/v1/. Inputs are zod-validated; every response uses the { version, data, meta } envelope.Every claim on the site is traceable to a source. Here's how to verify the most-common categories:
__tests__/data/readme-claims.test.ts suite asserts README / metadata / OpenAPI counts match the committed snapshot. If any of them drift past a threshold, the build fails — check recent Actions runs.__tests__/parity/backend-parity.test.ts run against both backends in CI and assert the results match. Catches the "query-behaviour divergence" class of bug before it ships.python scripts/dump_supabase_snapshot.py rebuilds the snapshot from scratch. The diff against the committed copy should be empty (mod timestamps).APA
Wilse-Samson, L. (2026). SA Policy Space: A Database of South African Reform Ideas. https://sa-policy-space.vercel.app
When citing a specific idea or briefing, include the URL and the snapshot date shown on the page footer. Each issue of Briefings carries Article JSON-LD and a stable permalink for citation.