Data Sources — higher-education-cost
All data used in this project is from authoritative sources. Crowd-edited references (Wikipedia, etc.) are not used as primary sources.
Document every data source here before ingesting it. Include enough detail that someone else could independently locate and verify the original data.
Source Template
Copy and fill in for each source:
[Source Name]
- Publisher: [Agency, organization, or author]
- URL: [Direct link to the file or page]
-
Format: [CSV JSON HTML table ZIP PDF hand-curated] -
License: [Public domain CC0 CC-BY proprietary etc.] - Fields used: [Column names or description of what was extracted]
- Coverage: [Geographic scope, date range, or other relevant bounds]
- Notes: [Any caveats, known issues, or methodology notes]
- Retrieved: [YYYY-MM-DD]
Sources
The URLs and structure below were confirmed live during
01-ingeston 2026-09-30. Two IPEDS collection years (2011-12 and 2021-22) were pulled to support the over-time story;01-ingest.ipynbnarrates every download.
IPEDS — Integrated Postsecondary Education Data System
- Publisher: National Center for Education Statistics (NCES), U.S. Department of Education
- URL: Complete data files at https://nces.ed.gov/ipeds/datacenter/DataFiles.aspx — per-survey, per-year zipped CSVs at the stable pattern
https://nces.ed.gov/ipeds/datacenter/data/<SURVEY_FILE>.zip - Format: ZIP of CSV (one per survey component, per collection year)
- License: Public domain (U.S. federal government work)
- Fields used / files pulled (2011-12 and 2021-22):
HD<year>— Institutional Characteristics header (stateSTABBR, sector, level, Carnegie class); keys other tables viaUNITIDIC<year>_AY— Charges: published tuition & required fees (sticker price), academic-yearF<yy><yy>_F1A— Finance GASB (public institutions): expenses by functionF<yy><yy>_F2— Finance FASB (private nonprofit): expenses by functionEFFY<year>— 12-month unduplicated enrollment (for per-FTE normalization)
- Coverage: U.S. postsecondary institutions; this ingest pulls collection years 2011-12 and 2021-22 (a full decade)
- Notes: The Finance survey splits into GASB (public,
*_F1A) and FASB (private nonprofit,*_F2) files with different accounting standards — expense-function categories do NOT map 1:1 and are kept as separate raw tables (ipeds_finance_gasb_*,ipeds_finance_fasb_*); they must be harmonized before any admin-vs-classroom comparison. IPEDS survey definitions have also shifted over the decade, so 2011-12 and 2021-22 are treated as separate series requiring harmonization. IPEDS CSVs read aslatin-1(occasional non-UTF-8 bytes in text fields) and withindex_col=False(see IC2012 note). - File-variant note (
IC2012_AY): the baseIC2012_AY.zip→ic2012_ay.csvis the correct file (no_rvrevised variant needed), but its header has 268 names while every data row has 269 fields — the rows carry a trailing comma / extra unnamed empty field. Pandas’ default parser responds tofields > headerby promoting column 0 to the row index, which shifted every labeled column one position left and leftUNITIDand all tuition values holding the interleavedX-prefixed imputation flags (literal"R"). Reading withindex_col=FalsekeepsUNITIDas data and drops the trailing empty field, parsing the file correctly.IC2022_AYhas matching 235-name/235-field rows and is unaffected (the fix is a no-op for it). The raw ZIP on disk is kept verbatim; only the parse was corrected. - Retrieved: 2026-09-30
College Scorecard
- Publisher: U.S. Department of Education
- URL: Data page https://collegescorecard.ed.gov/data/ — bulk “Most Recent Institution-Level Data” ZIP at
https://ed-public-download.scorecard.network/downloads/Most-Recent-Cohorts-Institution_05192025.zip - Format: CSV inside a bulk ZIP
- License: Public domain (U.S. federal government work)
- Fields used: Net price (
NPT4_*), cost of attendance (COSTT4_*), completion rate (C150_*), median post-enrollment earnings (MD_EARN_*), by institution - Coverage: U.S. Title IV institutions; most-recent cohort snapshot (cost/net-price fields robust from ~2009)
- Notes: The historical bulk host
ed-public-download.app.cloud.gov/downloads/CollegeScorecard_Raw_Data.zipnow returns 404; the current hosted-public-download.scorecard.networkwas resolved from the data.ed.gov CKAN package listing (college-scorecard-all-data-files). This ingest uses the single-CSV “Most Recent Institution-Level Data” file (latest cohort per institution), not the full multi-year raw archive (College_Scorecard_Raw_Data_05192025.zip, documented in config for later use). Net price ≠ sticker price — report both explicitly. - Retrieved: 2026-09-30
BLS Consumer Price Index (CPI-U)
- Publisher: U.S. Bureau of Labor Statistics
- URL: BLS public timeseries API
https://api.bls.gov/publicAPI/v2/timeseries/data/(see Notes on the flat-file host) - Format: JSON (API) → saved as long-format CSV in
data/raw/cpi_u/ - License: Public domain (U.S. federal government work)
- Fields used:
CUUR0000SA0(CPI-U All Items, U.S. city average, NSA — real-dollar deflator);CUUR0000SEEB01(CPI-U college tuition & fees subindex, to compare tuition inflation vs general inflation). Pulled 2000–2024, monthly + M13 annual averages. - Coverage: U.S., monthly, 1913–present (All Items); this ingest pulls 2000–2024
- Notes: The flat-file host
https://download.bls.gov/pub/time.series/cu/returns HTTP 403 (Akamai block page) to automated GETs — confirmed 2026-09-30 even with a full browser header set — so CPI-U is pulled from the BLS public API instead. The API is keyless for modest use (25 requests/day, ≤10 years per request; the 2000–2024 span is fetched in two windows); an optionalBLS_API_KEYin the gitignored.envraises the limits but is not required. CPI-U reference base is 1982-84=100; NSA (do not mix with seasonally adjusted). Nominal tuition growth is misleading; inflation adjustment is required for the “cost over time” story. - Retrieved: 2026-09-30
BLS CPI-U sector subindices (REFRAME — “big education” relative inflation)
- Publisher: U.S. Bureau of Labor Statistics
- URL: BLS public timeseries API
https://api.bls.gov/publicAPI/v2/timeseries/data/ - Format: JSON (API) → saved as long-format CSV in
data/raw/cpi_u/cpi_sectors_raw.csv; DuckDB tablecpi_sectors_raw - License: Public domain (U.S. federal government work)
- Fields used / series pulled (all CPI-U, U.S. city average, NOT seasonally adjusted —
CUUR0000…prefix):CUUR0000SA0— All Items (general-inflation baseline; 1913)CUUR0000SEEB01— College tuition & fees (⭐ starts Dec 1978 — the binding constraint that sets the 1979 common window)CUUR0000SAM— Medical care, all (1935)CUUR0000SAM2— Medical care services (big-healthcare line; 1956)CUUR0000SEMF01— Prescription drugs (big-pharma line; ~1947)CUUR0000SETB01— Gasoline, all types (big-oil line; 1976) — the CUUR unadjusted ID, never theCUUSseasonally-adjusted twinCUUR0000SA0E— Energy, aggregate (1957)CUUR0000SAH1— Shelter / housing (1953)CUUR0000SEMD— Hospital & related services (optional; starts Dec 1996 — late-window detail only, NOT the headline)
- Coverage: U.S., annual averages (M13), pulled 1979–2025 in five ≤10-year windows (all series batched per call)
- Notes: This is the data behind the project’s reframed headline — “has higher-education cost outpaced general inflation and the other ‘big’ sectors (big pharma/healthcare, big oil/energy) + housing?”. Each sector is re-indexed to 100 at a common base year 1979 in
03-prepareand plotted as a multi-line “which line is steepest” chart; college tuition is reliably the steepest. Common window = 1979 because the college-tuition subindex begins Dec 1978 (every other sector reaches further back and is truncated to it); hospital services (1996) is excluded from the headline. NSA only — never mix withCUUS…/CUSR…seasonally-adjusted series. CPI subindices embed BLS hedonic quality adjustment and periodic weight updates; the tuition subindex tracks published (sticker-ish) price movement, not net-of-grant-aid price (consistent with the project’s sticker-vs-net rule). Prior art (cited, not copied): the AEI / Mark J. Perry “Chart of the Century” is exactly this chart over a ~2000 window — we extend it with a longer 1979 window and our own “big education” framing, and reproduce none of its image or text. - Retrieved: 2026-10 (reframe ingest).
IPEDS 12-month FTE (EFIA) — re-ingested in 02-clean
- Publisher: NCES, U.S. Dept. of Education. URL:
https://nces.ed.gov/ipeds/datacenter/data/EFIA<year>.zip. License: public domain. - Fields used:
FTEUG(reported 12-month undergraduate FTE) +FTEGD(reported 12-month graduate FTE); one row perUNITID. Why re-ingested: the originally-loaded EFFY file carries 12-month headcount only — no FTE and no full-time/part-time split — so there is no viable FTE approximation. EFIA is IPEDS’s own authoritative 12-month FTE, required for the per-FTE metric. Lands asipeds_fte_2012_raw/ipeds_fte_2022_raw;fte_cleancomputesfte_total = FTEUG + FTEGD. - Retrieved: 2026-09-30 (re-ingest).
Supplemental (validation / context)
- SHEEO State Higher Education Finance (SHEF) — state appropriations & public funding per FTE over time (https://shef.sheeo.org/) — adds the “who pays” (state support vs tuition) dimension.
Reframe-expansion sources (second wave — ingested 2026-10)
Four more authoritative U.S. federal / public-domain sources, added to answer the
reframe’s Suspicion-1 “how dramatic is the long tuition climb?” and Suspicion-2
“is the driver administration?” questions. Each re-enters at 01-ingest.
A. NCES Digest of Education Statistics — Table 330.10 (long tuition history)
- Publisher: National Center for Education Statistics (NCES), U.S. Dept. of Education
- URL:
https://nces.ed.gov/programs/digest/d24/tables/dt24_330.10.asp(data pulled from the table’s companion Excel.../tables/xls/tabn330.10.xlsx; the newest edition carrying the full 1963-64 series is auto-selected at build — d24/d23/d22) - Format: Excel (
.xlsx) companion of the Digest HTML table → DuckDBnces_tuition_longrun_raw - License: Public domain (U.S. federal government work)
- Fields used: Average undergraduate tuition & fees / room / board, by control (all/public/private), level (all/4-year/2-year), dollar basis (current + constant), selected academic years 1963-64 → 2023-24. Parsed tidy-long:
academic_year, year_start, control, level, dollar_basis, base_year, cost_component, value. - Coverage: U.S. national averages, full-time students, degree-granting institutions, 1963-64 → latest
- Notes: HTML
read_htmlcannot parse the multi-level table, so the clean companion Excel is used. Constant-dollar columns are NCES’s own CPI-U All-Items adjustment (d24 base = 2023-24). National averages, not per-institution; published (sticker) price, not net-of-grant-aid; the institution universe and room/board reporting basis shifted across Digest editions; the table NUMBER drifts by edition (345/349/320/330.10). Anchors confirmed: public 4-yr tuition&fees ≈ $243 (1963-64) → ≈ $9,750 (2022-23). - Retrieved: 2026-10
B. NCES Delta Cost Project Database 1987-2012 (harmonized spending-by-function)
- Publisher: NCES (IPEDS Analytics: Delta Cost Project)
- URL: landing
https://nces.ed.gov/ipeds/use-the-data/delta-cost-project-finance-data; CSV ziphttps://nces.ed.gov/ipeds/deltacostproject/download/IPEDS_Analytics_DCP_87_12_CSV.zip(~115 MB, two CSVs:delta_public_87_99.csv,delta_public_00_12.csv) - Format: ZIP of two CSVs (needed-column subset loaded) → DuckDB
delta_cost_raw - License: Public domain (U.S. federal government work)
- Fields used:
unitid, academicyear, control, sector, fte12mn(constructed 12-mo FTE),fte_count, the CFI-method function totalsinstruction01 / research01 / pubserv01 / acadsupp01 / studserv01 / instsupp01(instsupp01 = institutional support = admin),cpi_scalar_2012. control: 1=public, 2=private-nonprofit, 3=for-profit. - Coverage: U.S. public + private-nonprofit (+for-profit) institutions, academic years 1986-87 → 2011-12
- Notes: NCES-harmonized expense-by-function + FTE across the Common-Form / FASB / GASB / Aligned-Form reporting-standard changes, with Delta’s own constructed FTE. Delta is a DERIVED / imputed product — cite Jaquette & Parra, “The Problem with the Delta Cost Project Database.” It ends 2011-12 (a seam to the raw-IPEDS 2013+ finance tail at 2012). Prefer spending SHARES over absolute per-FTE where imputation risk is higher. Loaded verbatim; for-profit filtering + per-FTE/SHARE computation happen in 02-clean.
- Retrieved: 2026-10
C. Raw IPEDS Finance tail 2013-2021 (extend the spending mix to the present)
- Publisher: NCES, U.S. Dept. of Education
- URL:
https://nces.ed.gov/ipeds/datacenter/data/<FILE>.zip—F1516_F1A/F1516_F2/EFIA2016(FY2016) andF1819_F1A/F1819_F2/EFIA2019(FY2019) - Format: ZIP of CSV → DuckDB
ipeds_finance_gasb_<yr>_raw,ipeds_finance_fasb_<yr>_raw,ipeds_fte_<yr>_raw - License: Public domain (U.S. federal government work)
- Fields used: GASB
F1C011…F1C071+ FASBF2E011…F2E061expense-by-function totals (same crosswalk as the finance spine), EFIAFTEUG+FTEGD12-month FTE. Each year’s function-total codes are asserted present before parsing. - Coverage: U.S. public (GASB) + private-nonprofit (FASB), FY2016 and FY2019 (mid-points between the project’s existing 2012 and 2022)
- Notes: All post-Aligned-Form (2008-10), so the modern
F1C/F2Ecodes hold (confirmed per year). Harmonized onto Delta’s category definitions in 02-clean so the national spending-mix series stacks cleanly across the 2012 seam. GASB isolates O&M-of-plant while FASB distributes it (O&M excluded from the admin/classroom numerators). - Retrieved: 2026-10
D. IPEDS Human Resources S<year>_OC (admin vs faculty headcount)
- Publisher: NCES / IPEDS Human Resources component
- URL:
https://nces.ed.gov/ipeds/datacenter/data/S<year>_OC.zip— S2013_OC, S2016_OC, S2019_OC, S2022_OC - Format: ZIP of CSV → DuckDB
ipeds_hr_<yr>_raw - License: Public domain (U.S. federal government work)
- Fields used:
STAFFCAT(occupation × full/part-time),FTPT(1 = all staff),HRTOTLT(grand-total headcount). Admin = STAFFCAT 1300 (Management); faculty = STAFFCAT 1210 (Instructional staff), both at FTPT=1. Denominator = EFIA 12-month FTE (reused from Source C / pulled here). - Coverage: U.S. institutions, collection years 2012-13 → 2021-22 (2013, 2016, 2019, 2022)
- Notes: The most intuitive “bloat” proxy. ⭐ HARD series break at 2012-13: IPEDS replaced the old primary-function / EEO occupational-activity categories with 2010-SOC-based categories effective the 2012-13 collection, so this series is clean only from 2012-13 forward; NO pre-2012 crosswalk is attempted. Admin/faculty aggregation happens in 02-clean.
- Retrieved: 2026-10
⭐ Honest-framing note (Suspicion 2). The spending-mix data (Delta + IPEDS tail) shows non-instructional spending (student services + academic support
- institutional support) grew as a share of core spending while instruction’s share fell — but institutional support (admin) ALONE is roughly flat. So the defensible finding is “non-instructional spending outgrew instruction,” not “administrators alone exploded” (consistent with Kelchen’s and ACTA’s IPEDS analyses). On the HEADCOUNT proxy (IPEDS HR), administrator headcount did outpace faculty and enrollment 2013→present — a complementary, more quotable cut, not a contradiction (headcount ≠ share of dollars). Charts carry this nuance explicitly.
⭐ “Admin” is a definitional choice (chart 4 caveat — carried to the README). Chart 4’s “admin” is IPEDS institutional support only, a specific accounting category. Different definitions of administration yield different “bloat” magnitudes, and the choice matters: microdata analyses of administrator headcount — e.g. FIRE / Kay & Honeycutt, 2025 — find MORE administrative growth than the IPEDS spending-share accounting shows. So “admin spending share is roughly flat (~15%)” is true for this accounting measure and does NOT contradict headcount-based findings that administrators grew faster than faculty; the two measure different things (share of dollars vs count of people). Chart 4 and the README state this explicitly so the slopegraph isn’t misread as “admin exploded.”
Plain-English definitions of the four core functional categories (for the README and codebook): instruction = teaching activity (faculty salaries, academic departments, instructional operations); institutional support (= “admin”) = executive management, general administration, finance, HR, legal, and public relations; academic support = libraries, academic computing, museums, deans’ offices, curriculum development, and other support of the instruction/research mission; student services = admissions, the registrar, financial-aid administration, counseling, career services, and student activities.
Harmonization & series-break caveats (settled in 02-clean)
Carried forward into chart footnotes and the _sources methodology /
series_breaks columns for the harmonized tables.
- ⭐ GASB expenses-by-function live in the
F1Cblock, notF1E. The harmonization design draft mapped GASB functions toF1E01…F1E07; the IPEDS F1A data dictionary (both collection years) shows those are the scholarships / grants block (Pell grants, state grants, …), not expenses by function. The dictionary-confirmed GASB function block isF1C:F1C011instruction,F1C021research,F1C031public service,F1C051academic support,F1C061student services,F1C071institutional support (= admin),F1C191total expenses. FASB usesF2E011…F2E061(+F2E131total — notF2E071, which is a function line). “Admin” = institutional support only (F1C071/F2E061). - Academic support is NOT GASB-sparse. The design flagged GASB academic
support as sparse (≈236/2,004) and excluded it from cross-sector comparison —
but that was the wrong column (
F1E04= local-government grants). The correctF1C051is well-populated in both sectors (≈1,875/2,004 in 2012; 1,825/1,936 in 2022), on par with FASBF2E041. The “sparse” rationale is therefore defunct; whether to use academic support in public-vs-private charts is decided in03-prepare. - GASB vs FASB operation-and-maintenance (O&M) of plant. GASB isolates O&M as
its own function; FASB distributes it into the other functions. The harmonized
functions are comparable only to within that allocation difference; O&M is
deliberately excluded from the admin-vs-classroom numerators, and per-standard
raw totals (GASB
F1C191/ FASBF2E131) are never compared across standards or used as a share denominator (the share denominator, built in03-prepare, is the in-pipeline sum of the harmonized function totals). - IPEDS 2011-12 vs 2021-22 is a series break, not a continuous panel. Finance subcomponent column sets changed across the decade (GASB 322→288 cols, FASB 394→340); only the stable function-total codes are used. Institutions enter and exit between the two years (HD 7,735 → 6,256), so over-time aggregates note the changing institution set.
- CPI-U re-pulled through 2024. The first ingest truncated
cpi_u_rawat 2019 (keyless BLS API ≤10 years/request); 02-clean re-pullsCUUR0000SA0+CUUR0000SEEB01across three ≤10-year windows so an M13 annual-average exists for 2022. Spending is deflated to constant 2022 dollars with All-Items only; the tuition subindex is for the tuition-vs-general-inflation comparison and never deflates spending. - ⭐ Energy and gasoline are volatile (charts 1 & 2 caveat). Oil-linked CPI
sub-indices (gasoline
SETB01, energySA0E) swing sharply year to year, so a long-window cumulative comparison understates their short-run spikes — gasoline can sit far above general inflation in any given year even though it lagged All Items across the full 1979→present window (which is what charts 1 & 2 show). The long-run cumulative framing is the honest comparison for a “which cost rose most over decades” question, but it is not a claim that energy prices are low in any particular year. - ⭐ 2025 figures are preliminary (data-vintage note). The CPI sector series (charts 1 & 2) run into mid/late 2025, and 2025 annual averages are still subject to minor revision by BLS. The headline comparisons (tuition ≈ 3× All Items; ≈ +233% out-performance) are robust to that revision, but the exact 2025 index values may shift slightly in later vintages.
Notes on Data Quality
- All source files are saved verbatim to
data/raw/and never modified. - Discrepancies between sources should be noted here and resolved explicitly.
Source Provenance in DuckDB
Every table in data/project.duckdb has a corresponding entry in the
_sources metadata table:
SELECT * FROM _sources;