CDC_Mortality_UnderlyingCause_County

[goutamabrol]

Summary

PR checklist - link

Adds the CDC_Mortality_UnderlyingCause_County import under statvar_imports/us_cdc/county_mortality_underlyingCause/ for CDC WONDER county-level mortality data (1999–2024).

• Data Source: Covers all US counties across 50 states + DC from 1999 to 2024 (1,909,796 total observations across 3,098 counties):
  • 1999–2017 (Static Historical): Pre-downloaded from CDC WONDER Database D76 (1,342,838 rows) and stored in GCS (gs://unresolved_mcf/cdc/UnderlyingCause/county_mortality_underlyingCause/input_files/UnderlyingCauseofDeath_County_1999_2017.csv).
  • 2018–Present (Live): Downloaded live from CDC WONDER Underlying Cause of Death (Database D158, 566,958 rows for 2018–2024).
• Coverage: Maps 123 active StatVars (107 common causes + 16 rare causes) across 137 mapped causes in the ICD-10 113 Cause List, and fully supersedes legacy US_Texas_Diabetes across all 12 years (2011–2022, with 79.54% exact matches and 97.32% agreement within ±2 deaths after accounting for 15 shifted county FIPS mappings in the legacy PV map, expanded to all 50 states + DC and 1999–2024).
• Downloader (download.py): Copies the frozen 1999–2017 CSV from GCS and automates live 2018–present D158 downloads with session agreements, dynamic year discovery, 2-year/1-year query partitioning (75k-row cap), and 31-min silent backoff on HTTP 429.
• Mappings: Dynamic 5-digit FIPS zero-padding (geoId/{CountyCode1}) in PV map with no hardcoded county lookup tables.
• Automation (manifest.json): Configured for Cloud Batch execution with download.py, stat_var_processor.py, and monthly cron.
• Validation (validation_config.json): Enforces zero lint errors (LINT_ERROR_COUNT), 0.1% historical deletion threshold (DELETED_RECORDS_PERCENT), rolling date freshness (SQL_VALIDATOR for the 107 common causes), and floor date checks for the 16 rare causes.
• Testing: 18/18 unit tests pass locally (download_test.py), and end-to-end Cloud Staging (datcom-import-test) completed with all validations PASSED.

Cloud Batch job (test) - link

Validation - link

CRA Report - link

[gemini-code-assist]

Code Review

This pull request introduces a new automated importer for CDC WONDER County-Level Mortality Data, including a download script, metadata, mapping files, unit tests, and validation configurations. The reviewer feedback highlights two important improvements for the download script: first, making the is_state_downloaded check more robust to ensure all intermediate years are covered rather than just the first and last; second, modifying the downloader to log errors and continue processing other states instead of immediately terminating the script with logging.fatal when a single state fails.