US_UrbanSchool_Algebra1

[goutamabrol]

Summary

Refactors the US_UrbanSchool_Algebra1 (statvar_imports/school_algebra1) pipeline to separate raw data downloading from preprocessing, standardizes ncesid and YEAR generation across CRDC survey years (2009–2024), and updates the manifest and test fixtures.

PR Checklist doc - link

Changes

• download_script.py: Decoupled in-place file mutation from downloading so raw CRDC files (.csv and .xlsx) in input_files/ remain intact; added dynamic survey year range detection and skipping for existing downloads.
• preprocess_data.py: Added a dedicated preprocessing script that reads raw files from input_files/, constructs zero-padded 12-digit ncesid values (LEAID + SCHID or COMBOKEY), extracts the 4-digit survey completion YEAR, filters to columns mapped in Algebra1_pvmap.csv, and writes clean CSVs to processed_files/.
• Algebra1_metadata.csv: Removed unused unit and scalingFactor columns from output_columns (observationDate,observationAbout,variableMeasured,value).
• manifest.json:
  • Added preprocess_data.py to scripts and updated stat_var_processor.py to read processed_files/*.csv and output counters to counters/algebra1_counters.csv.
  • Added counters/*.csv and validation_config.json to source_files.
  • Added "node_mcf": "output/*.mcf" to import_inputs.
  • Updated resource_limits to default values (cpu: 8, memory: 32, disk: 100).
• test_data/ & README.md: Regenerated input_test_data.csv and sample_output.csv to match the zero-padded ncesid and 4-column output schema, and updated README.md to document the 3-step workflow and 2009–2024 coverage.

Cloud Batch job (test) - link

Validation - link

CRA = link

[gemini-code-assist]

Code Review

This pull request refactors the school algebra 1 data import workflow by introducing a dedicated preprocessing step (preprocess_data.py), updating the download script to skip already downloaded files, and updating the manifest, metadata, and test data accordingly. Feedback on the changes suggests optimizing the survey year generation in download_script.py to use a step of 2 for biennial years, which avoids unnecessary download attempts for non-existent odd years.