Skip to the content.

Scripts — reproducing and validating the published datasets

The published release (export/higher_education_cost_cpi_sectors_v1.*, ..._tuition_longrun_v1.*, ..._spending_mix_v1.*) is rebuildable from raw sources with a single command, and every headline figure in the four published charts is re-checked by a pre-publish validation gate.

Use the project virtual environment for both:

.venv/bin/python scripts/reproduce.py        # raw → export
.venv/bin/python scripts/validate_charts.py  # pre-publish data gate

reproduce.py — raw → export

python scripts/reproduce.py

Runs the project’s canonical pipeline end to end — the numbered notebooks 01-ingest → 02-clean → 03-prepare — which contain the exact ingest, cleaning, and prepare/export logic (the reusable code lives in ../src/). Running the notebooks is how this project is built, so the reproduced export/ matches the published datasets.

Stage Notebook What it does Output
1. Ingest 01-ingest Ensures the raw BLS CPI-U sector series, NCES Digest 330.10 tuition table, Delta Cost Project zip (~115 MB), and IPEDS finance/tuition files are in data/raw/ (downloads any that are missing), and loads them into DuckDB as the *_raw tables data/raw/ + DuckDB raw tables
2. Clean 02-clean Builds the interim *_clean tables in DuckDB (cpi_sectors_clean, tuition_longrun_clean, spending_mix_clean, the finance spine) — all cleaning happens in DuckDB SQL DuckDB *_clean tables + data/interim/ Parquet
3. Prepare 03-prepare Builds the chart-ready chart_* tables and packages the sellable datasets (CSV + Excel + Parquet + codebook) export/*.{csv,xlsx,parquet} + codebooks

Each stage first regenerates its .ipynb from the matching scripts/_build_nb_*.py nbformat builder, then executes it in place with nbconvert, so a run always executes the current pipeline code (not a stale checked-in notebook).

Prerequisites

Options

python scripts/reproduce.py --no-download        # fail (don't fetch) if data/raw/ is empty
python scripts/reproduce.py --stages 02 03       # re-run only clean + prepare (raw already present)
python scripts/reproduce.py --build-only         # (re)generate the notebooks from scripts/, don't execute

validate_charts.py — pre-publish data gate

python scripts/validate_charts.py   # prints "safe to publish" and exits 0 on success

Re-derives, straight from the project DuckDB, what each of the four published charts shows and asserts the exact headline figures — tuition the steepest sector (index ≈ 1,476 vs All Items ≈ 443), tuition out-ran general inflation by ≈ +233%, public 4-year real tuition ≈ $2,442 → ≈ $9,872 (~4×), and the honest spending-mix split (instruction 48.8% → 43.1%, admin flat ≈ 15%, non-instructional 32.5% → 37.4%). It also confirms each published export/*.csv still matches its DuckDB source table (no drift) and that the social/web chart sets stay in parity. Exit code 0 = safe to publish; non-zero names the failing check. Run it after reproduce.py and before any release.

Scope

This entrypoint reproduces the three published datasets that back the four release charts. The project also builds an institution-level combined dataset (higher_education_cost_v1) and additional exploration tables (admin-vs-faculty headcount, per-state admin, etc.); those run through the same 01→02→03 notebook pipeline and are not separately curated for the public release.