HARPSPOL: background scatter destroys science products at every tested extraction_height #39

Background scatter destroys HARPSPOL science products at every tested setting

Splitting this out of #38 as requested.

Running the scatter step for HARPSPOL removes most or all of the stellar flux from the extracted spectra. This is not a tuning problem: three extraction_height values spanning a factor of ~30 all fail, differing only in degree.

Measurements

Same observation throughout (HD 136504, night 2015-06-03, calibrations and science reduced identically apart from the scatter setting):

The trend is monotonic — a larger height masks more of the order, so less stellar flux leaks into the background fit — but even the best case keeps only 2.3 % of the blue flux and 24 % of the red, and leaves 21 orders unnormalisable. Blue goes negative for the two smaller values.

Note 0.6 and 10 land close together because estimate_background_scatter reads values below 2 as a fraction of order spacing, and 0.6 × order spacing on HARPS is itself about ten pixels. 20 is genuinely a different setting and still fails.

Across a full month (17 sequences, 4 calibration nights, 2015-06) at the inherited 0.6, the number of orders with no usable continuum rose from 6 to 1066.

The damage scales with target brightness

Over that month, the change in extracted intensity ranged from −3.4× on α Cen (the brightest) to −325× on CPD-246168 (among the faintest). Faint targets are hit hardest.

Where I think this comes from

BackgroundScatter fits the model on the LAMP,LAMP,TUN flat (id_scatter), and the result is a 5×5 array of 2D polynomial coefficients. Following it to the point of use:

ScienceExtraction.run(scatter=...)
  -> extract_to_arrays(..., scatter=scatter)
    -> extract(..., scatter=scatter)
      -> calc_scatter_correction:  polyval2d(x, y, scatter)

bias_scaling and norm_scaling are applied, but by CalibrationStep.calibrate → combine_calibrate, i.e. to the input frames the model is fitted to. Nothing between the fit and calc_scatter_correction rescales the output to the science exposure level.

So a model fitted to a bright lamp exposure is subtracted at full lamp amplitude from a much fainter stellar frame. That would produce exactly the brightness-dependent over-subtraction above, and would explain why no choice of extraction_height helps: the parameter changes how much flux contaminates the fit, not the amplitude of what is subtracted.

I have not confirmed this is the mechanism — it is the hypothesis the measurements point at, and the scaling trace above is the evidence for it.

Meanwhile

#38 drops the scatter entry from HARPSPOL's get_expected_values alongside its settings block, so the step is not reachable for this instrument until this is understood. Happy to test a fix against the same month.