Demographics catalog
The free-government demographic cache at data/demographics/ — pulled once, reused across every model and every version. This is the per-source reference.
Models enrich property records (First American parcels, BuildZoom permits) with free nationwide public-government context — income, vacancy, broadband, affordability, hazard, climate — joined on zero-padded FIPS string keys. The cache is not build-specific; there is no auto-refresh, so deleting it costs hours of bandwidth + dev time to re-pull. The plain-language permit↔property match is on Data sources; this is the source-by-source catalog underneath it.
MANIFEST.jsonCounts vary by what is measured: the README "Complete Inventory" reports 44 source families / 134 main parquets / 29,136,543 rows / 1.73 GB; the Wave-10 directory audit counts 52 source dirs (196 parquets incl. raw/sub-tables). The family-vs-directory distinction is the gap; both reported as-is. Cache-wide vintage range 2010–2025.
The eight load-bearing tiers
These eight families carry the primary socioeconomic, count, leverage, affordability, employment and infrastructure signal. All grains joined on zero-padded FIPS strings (Join keys below). Row counts + Maricopa 04013 anchors are from the per-wave QA audits (2026-05-27/28). Safe T0 ≥ = hard floor below which the source is future leakage (CLAUDE.md rule #2; pub-lag math in Vintage-safety below).
| Source | Grain | Join key | Vintage | Safe T0 ≥ | What it provides · anchors |
|---|---|---|---|---|---|
| ACS 5-yr (Census) | Block-Group · Tract · ZCTA · County | geoid (12/11/5/5) | 2019–2023 live (center ≈2021) 2018–2022 backfill (center ≈2020) | 2025-Q1 live 2024-Q1 backfill | 139 curated variables / 30 tables: median HH income, home value, gross rent, tenure, year built, education, mobility, commute. BG = finest socioeconomic context. Maricopa MHI $85,518 (2023) / $80,675 (2022). |
| Decennial 2020 (Census) | Block · Block-Group · County | geoid_bg · fips5 (15/12/5) | April-1 2020 PIT | 2021-Q3 | 100%-count population + housing-unit + vacancy. Freshest exact vacancy. Block file 8.13M rows. Maricopa county pop 4,420,568, hu_vacant 169,248. |
| IRS SOI (IRS) | USPS ZIP (not ZCTA) | zip5 | 2018–2022 (5 vintages) | 2025-Q1 (TY2022) | Income-distribution by ZIP — 5 AGI brackets per ZIP (multi-row). Mortgage-interest density / leverage proxy. Column is zip5 not zipcode. |
| HUD CHAS / FMR / IL (HUD) | Tract · County · ZIP | county_fips · geoid · zip5 | CHAS 2018–2022 FMR/IL FY25 (forward-looking) | CHAS 2025-Q1 FMR/IL already safe | Cost-burden / affordability bands (CHAS, 23 sub-tables), Fair Market Rents (FMR), AMI Income Limits (IL), plus Small-Area FMR by ZIP. Maricopa 2BR FMR $1,950. |
| BLS QCEW (BLS) | County × NAICS · State | county_fips · state_fips | 2024 annual | 2025-Q3 | Employer-side wage / employment cycle by industry. 264,923 county rows (audited). |
| FCC BDC (FCC) | County · Place · CBSA · CongDist · State · Tribal · National | county_fips | Dec 2024 collection | 2025-Q1 | Broadband availability by technology / speed — infrastructure premium. County file 366,510 rows. Plus a mobile (BDC-mobile) family. Maricopa 100% any-tech, 92% gigabit. |
| HMDA (CFPB) | County | county_fips | 2022 + 2023 | 2024-Q4 (2023) | Mortgage-origination signal (mortgage applications). 152,065 (2022) + 141,940 (2023) county rows; county_fips 5-digit string, Maricopa present both years. README inventory lists it alongside the eight tiers |
Additional families present (documented)
The cache holds many more free-government families beyond the eight above; listed here are only those with a spec/audit in notes/demographics/. Row counts from the README auto-inventory + per-wave audits; some sources carry documented gaps (flagged).
| Family | Grain | Join key | Vintage | What it provides · notes |
|---|---|---|---|---|
| BEA regional income | County | county_fips | 2019–2023 | Per-capita / personal income by county. 2019 has 3,113 rows vs 3,114 later (1-county delta, cosmetic). |
| CDC SVI · PLACES · WONDER | County · Tract | county_fips · geoid | SVI 2022 PLACES 2024 WONDER 2020–2023 | Social Vulnerability Index, local health prevalence, mortality. PLACES Maricopa 993 tracts vs SVI 1,009 (PLACES suppresses low-pop). |
| EPA EJSCREEN · Walkability · AQS | Block-Group · County | geoid · geoid20 · county_fips | EJ/Walk 2024 AQS 2020–2024 | Environmental-justice indicators, walkability index, air quality. AQS sparse (~1,030 monitored counties/yr; absent = no monitor, not clean air). |
| FEMA NRI · NFHL | County · Tract | county_fips · tract_geoid · fips5 | NRI 2024 | National Risk Index (natural-hazard risk) + flood-hazard layer. Use county_fips/tract_geoid, not legacy STCOFIPS. |
| NOAA storms · normals | County · Event | county_fips | storms 2015–2024 normals 1991–2020 | Storm-event history (943,100 events audited) + 30-yr climate normals. 61% of storm events were zone-based, crosswalked to counties; 154 normals counties via nearest-neighbor (has_station=False). |
| USDA RUCC · RUCA · SNAP · Food Atlas · FAR | County · Tract · ZIP | fips5 · tract_geoid · zip5 | RUCA 2010 FAR 2020 rest 2023 | Rural-urban continuum, commuting areas, SNAP participation, food-access atlas, frontier codes. |
| FHFA HPI | County · CBSA · State · ZIP3 | fips_code · cbsa · state_fips · zip3 | time-series | House Price Index. County annual 106,252 rows, 2,795 unique FIPS, Maricopa present. |
| Housing market — Zillow · Redfin · NAR EHS | County · ZIP · MSA · State | geoid · fips_code · cbsa · state_fips | time-series | ZHVI / ZORI / sale-price / velocity / inventory (Zillow 8,677,707 rows, 7 files), weekly market tracker (Redfin), existing-home-sales (NAR). See open issues below. |
Other documented dirs the audits also cover: Census secondary (CBP, ZBP, PEP, HVS, QWI, AHS, LODES), IRS Migration + County, IRS/ACS migration flows, SAIPE, US Drought Monitor, Tigerline (geocoding shapes), HUD ZIP crosswalk, BLM PAD-US, WRLURI, NHTSA FARS, FDIC SOD, NCES CCD, USGS NLCD, MS USBF, USPS vacancy, SBA loans, energy/minerals, DOE LEAD, FBI UCR, FRED macro.
Join keys
All keys are zero-padded strings, never integers (CLAUDE.md rule #1; 04013 not 4013). Across the 44 families the geo-key column travels under 10 different names; the canonical ones:
| Key | Format | Example | Used by |
|---|---|---|---|
| state_fips | 2-digit | 04 | all sources |
| county_fips / fips5 | 5-digit | 04013 | ACS · Decennial · HUD · BLS · FCC · HMDA · BEA · CDC · EPA · FEMA · property records |
| tract_geoid | 11-digit | 04013010101 | ACS · HUD CHAS · CDC · FEMA NRI tract |
| bg_geoid / geoid | 12-digit | 040130101011 | ACS · Decennial · FCC · EPA EJSCREEN/Walkability |
| block_geoid | 15-digit | 040130101011001 | Decennial · FCC |
| zcta | 5-digit | 85003 | ACS only |
| zip5 / zip_usps | 5-digit | 85003 | IRS SOI · HUD SAFMR · Census ZBP · USDA |
Vintage-safety + join gotchas (no future leak)
Features must be computable from the T0 snapshot alone (CLAUDE.md rule #2). Each source publishes with a lag, giving a hard T0 floor below which it is future leakage. Below the floor → fall back to prior vintage or NaN. Safe-T0 floors (from README):
| Source | Snapshot | Pub lag | Safe T0 ≥ |
|---|---|---|---|
| ACS 5-yr 2019–2023 | rolling avg ~2021 | ~14mo | 2025-Q1 |
| Decennial 2020 | April-1 2020 PIT | ~18mo | 2021-Q3 |
| IRS SOI 2022 | tax year 2022 | ~24mo | 2025-Q1 |
| HUD CHAS 2018–2022 | ACS derivative | ~36mo | 2025-Q1 |
| BLS QCEW 2024 | calendar 2024 | ~6mo | 2025-Q3 |
| FCC BDC Dec 2024 | Dec 2024 collection | ~4mo | 2025-Q1 |
| HMDA 2023 | calendar 2023 | ~9mo | 2024-Q4 |
Four gotchas to apply before any join
| Gotcha | Why it bites | Fix |
|---|---|---|
| ZCTA ≠ USPS ZIP | ZCTA is a Census polygon approximation of a mail ZIP, not 1:1. ~3–5% of property records have a USPS ZIP with no matching ZCTA; ~2% of ZCTAs cross county lines. | Bridge via HUD ZIP↔ZCTA crosswalk. Prefer block-group / tract joins when lat/lon is available; fall back to ZCTA only when you have just a ZIP. |
ACS sentinel -666666688 | Census encodes "estimate not available" as this numeric sentinel; parquets retain it raw (NaN-normalization covers only textual nulls). Example: BG 040130101021 has b19013_001_e = -666666688. | Filter before use: df['mhi'].where(df['mhi'] > 0). |
FCC county_fips INTEGER recast | VALIDATION_REPORT flagged 12 FCC parquets storing county_fips as INTEGER. County files now fixed to zero-padded VARCHAR; non-county grains carry it 100%-null (correct — no county key there). Remaining FAIL: fcc_bdc_mobile provider_summary provider_id as int64. | County files already fixed. provider_id is an FCC identifier not a FIPS, so cast-on-read suffices. |
| Prefixed / multi-row keys | HUD CHAS geoid is summary-level-prefixed (1400000US04013010102); HUD FMR county FIPS is the first 5 chars of a 10-digit fips; IRS SOI is multi-row per ZIP (5 AGI brackets); Tigerline tract county_fips is 3-digit. | CHAS: SUBSTRING(geoid, 10) or LIKE. SOI: aggregate or join on (zip5, agi_bracket). Tigerline: geoid[:5]. |
Known gaps & status
The cache is curated, not finished. Honest open items from the QA audits:
Validation across the cache: 115 PASS · 6 WARN · 13 FAIL (VALIDATION_REPORT totals). The 13 FAILs = the 12 FCC INTEGER-FIPS files (since recast) + the empty Redfin ZIP.
| Item | Status | Detail |
|---|---|---|
| Redfin ZIP weekly | FAIL — empty | 0-row parquet; ETL never wrote data. Raw TSV (1.5 GB) exists; re-ingest required. |
Redfin county fips_code | FAIL — 99.8% null | Only 4 FIPS populated; region_name + state_fips + cbsa usable as workaround pending FIPS re-derive. |
| USGS NLCD county | FAIL — partial | 2,287 of ~3,143 counties (856 missing); re-pull required. |
| MS USBF | FAIL — stub | 16-row AZ+DC stub; national pull never completed. |
| ACS BG 34 dead columns | WARN | 34 cols 100% NULL at BG (B07001/B08006/B17001/B19083/B25097 — not published at BG). No data loss; schema bloat only. |
| FBI UCR · FRED macro | WARN — empty dirs | Spec + script exist, zero parquets. planned |
Tier-F (local market context) wiring into the REI feature builder is in progress. Three model features — listing_duration_months, months_since_prev_sale, mortgage_age_months — remain under leakage audit and are unvalidated under audit.
Where this connects
| What | Where |
|---|---|
| Summary card + deep-link (property-record side on same page) | Data sources → US demographics cache |
| Geocoding helper (lat/lon → block-group geoid) | src/new_model/geocoding.py |
| Full per-source specs | notes/demographics/<source>_spec.md |
| Cross-source manifest | data/demographics/MANIFEST.json |
Rendered from notes/demographics/ (README + DATA_CATALOG + specs + audits).