---
source_html: notes/investigations/ars/artifacts/scorecard.html
synthesized: 2026-06-15
what: ARS address-standardization QA scorecard — 104 synthetic addresses scored against Smarty us-street ground truth, generated 2026-06-15. Includes 13 regression cases from ars_defects_13.csv.
---

## Bottom line

91/104 addresses PASS (87.5%); average field score 97.5% across addresses. ARS sync latency 16.08s. Smarty us-street is ground truth; a row passes only if all five scored fields (delivery line · city · state · ZIP · FIPS) match. The 13 regression cases from `ars_defects_13.csv` **all 13/13 reproduce as failures** (delivery-line divergence). All numbers from `sync_compare.json`; nothing hand-typed.

## Per-field agreement

| Field | Match | % |
|---|---|---|
| Delivery line | 91/104 | 87.5% |
| City | 104/104 | 100.0% |
| State | 104/104 | 100.0% |
| ZIP | 104/104 | 100.0% |
| FIPS | 104/104 | 100.0% |

## Pass rate by ARS resolution source

ARS resolves each address via Athena → its own cache → Smarty passthrough. The first two are 8020 data (the real test).

| Source | n | Pass |
|---|---|---|
| smarty | 63 | 50/63 |
| athena | 33 | 33/33 |
| cache | 5 | 5/5 |
| null | 3 | 3/3 |

## Regression defects (ars_defects_13.csv)

13 known ARS standardization defects, street+zip input only. ARS pulled the correct source internally (`src=smarty` on all 13) then mangled the delivery line on output — the defect is in the ARS standardization layer, not the upstream source.

| Case | ARS delivery line | Smarty (truth) | Verdict |
|---|---|---|---|
| `defect_a_ct_18` | `660` | `660 MULBERRY CT` | **PARTIAL** (80%) |
| `defect_a_ct_78` | `2640` | `2640 SAPPHIRE CT` | **PARTIAL** (80%) |
| `defect_a_ct_102` | `640` | `640 BEIGE CT` | **PARTIAL** (80%) |
| `defect_b_ord_390` | `123 1ST` | `123 1ST ST` | **PARTIAL** (80%) |
| `defect_b_ord_397` | `258 7TH` | `258 7TH ST` | **PARTIAL** (80%) |
| `defect_b_ord_399` | `741 9TH` | `741 9TH ST` | **PARTIAL** (80%) |
| `defect_b_ord_401` | `963 4TH` | `963 4TH ST` | **PARTIAL** (80%) |
| `defect_b_ord_403` | `357 8TH` | `357 8TH ST` | **PARTIAL** (80%) |
| `defect_c_unit_249` | `CIR APT 7 33101` | `900 CHESTNUT CIR APT 7` | **PARTIAL** (80%) |
| `defect_c_unit_270` | `800 SEQUOIA CT` | `800 SEQUOIA CT UNIT E` | **PARTIAL** (80%) |
| `defect_d_long_439` | `12345 INTERCONTINENTAL AIRPORT EXPY` | `12345 INTERCONTINENTAL AIRPORT EXPRESSWAY FRONTAGE RD` | **PARTIAL** (80%) |
| `defect_d_long_448` | `9173 SW DESERT MT` | `9173 SW DESERT MOUNTAIN FOOTHILLS PKWY` | **PARTIAL** (80%) |
| `defect_d_multicity_453` | `800 EL` | `800 EL CAMINO REAL` | **PARTIAL** (80%) |

**13/13 caught.** `defect_c_unit_270` (`800 Sequoia Ct Unit E` → ARS drops `Unit E`) now FAILS: the secondary-unit scoring rule was tightened so a unit is excused only when the input never carried one (a phantom Smarty fabricated, e.g. the `# L` from a malformed `L A` city). When the input carried a unit and ARS dropped it, that is lost data and counts as a real delivery-line mismatch. See `tests/test_unit_scoring.py`.

## Caveats — passes that do not confirm deliverability

- **over-match** (1): ARS resolved a should-not-resolve address as match-mailable while Smarty did not confirm DPV: `nonexistent`. A real ARS over-reach, tracked separately.
- **weak DPV** (8): edge address forms where Smarty's own DPV is unconfirmed: `zip_missing_empire, rural_route, dl_half, fips_sanjuan_pr, dl_grid_utah, dl_dir_suffix, dl_us_highway, dl_county_road`. Standardization agrees; deliverability is not vouched for.