Skip to content

Add fir_filter step and FIR highpass and bandpass designs - #4855

Open
galenlynch wants to merge 4 commits into
SpikeInterface:mainfrom
galenlynch:feat/fir-filter
Open

galenlynch wants to merge 4 commits into
SpikeInterface:mainfrom
galenlynch:feat/fir-filter

Conversation

@galenlynch

Copy link
Copy Markdown
Contributor

This PR adds a step that applies any FIR kernel, and separate functions that design the default kernels, as suggested on #4563:

  • fir_filter(recording, taps) applies a FIR kernel by overlap-save, either one kernel for all channels or one per channel group. Its output doesn't depend on chunking.
  • design_fir_highpass(freq_min, fs) and design_fir_bandpass(freq_min, freq_max, fs) design Kaiser-window kernels: −6 dB at the cutoffs, a 175 Hz transition at the low edge (and 2 kHz at the bandpass's high edge), and at least 45 dB down beyond. The default highpass has 463 taps and the default bandpass 503.

Best merged after #4853: without it, phase_shift followed by this filter runs an FFT of a slow length.


Speed. One thread, 1 s chunks of 384 channels, i9-12900K:

filter_time

get_traces, int16 in and float32 out, best of 7; MS/s counts channel-samples. The IIR is sosfiltfilt, which computes in float64.

Saving 30 s through the filter keeps most of the gain and uses less RAM. Input and output are in RAM, so the time covers reading, filtering and writing but not the disk (median of 3; RAM is the peak above baseline):

1 job 8 jobs
highpass, IIR 67 MS/s, 0.24 GB 273 MS/s, 2.19 GB
highpass, FIR 163 MS/s, 0.05 GB 548 MS/s, 0.88 GB
bandpass, IIR 60 MS/s, 0.24 GB 262 MS/s, 2.46 GB
bandpass, FIR 131 MS/s, 0.05 GB 488 MS/s, 0.88 GB

In a chain of phase_shift, the highpass and median common_reference, the FIR makes saving 1.15–1.17 times faster with main's phase_shift (given #4853), and 1.27–1.36 times faster with #4563's.


Response. The defaults aim to be no worse than the default Butterworth. On 383 channels of a Neuropixels 2.0 recording, compared with the probe's noise spec (69 nV/√Hz):

  • Below 150 Hz the FIR passes more than the Butterworth, but less than 2.5% of the probe's own noise.
  • Between 150 and 300 Hz it passes 13% less (highpass) and 5% less (bandpass).
  • In the passband it is flat to 0.7%, where the Butterworth is up to 5.3% low (highpass, above 400 Hz) and 3.8% low (bandpass, 400 Hz to 4.5 kHz).
  • Between 6 and 9 kHz the bandpass passes 30% less than the Butterworth.
fir_spectrum fir_spectrum_bandpass

Median spectrum over channels, relative to the probe's noise spec, for the 300 Hz highpass (top) and 300–6000 Hz bandpass (bottom).

Precision. A FIR has no poles, so float32 rounding errors don't build up, even at low cutoffs:

highpass cutoff FIR in float32 sosfiltfilt in float32
300 Hz 5e-7 5e-6
10 Hz 3e-7 5e-3

rms error against a float64 reference, relative to the signal; 10 s of 32 channels.

Seams. Each output sample depends only on the len(taps) - 1 input samples around it, and the step reads exactly those, so chunked output equals whole-signal output up to float rounding. The PR adds check_chunk_invariance(recording, chunk_size, atol) to core.testing to check this for any preprocessor:

import numpy as np
from spikeinterface import NumpyRecording
from spikeinterface.core.testing import check_chunk_invariance
from spikeinterface.preprocessing import design_fir_highpass, fir_filter, highpass_filter

rec = NumpyRecording(np.fromfile("raw.dat", dtype=np.int16).reshape(-1, 384), 30_000.0)
iir = highpass_filter(rec, freq_min=300.0, dtype="float32")
fir = fir_filter(rec, design_fir_highpass(300.0, 30_000.0), dtype="float32")
for r in (iir, fir):
    print(check_chunk_invariance(r, chunk_size=30_000, atol=np.inf))  # largest seam error
Numbers

Largest seam error with 1 s chunks, on 10 s of a Neuropixels 2.0 probe (384 int16 channels, 2.34 µV per count, float32 output):

IIR FIR
highpass 0.0082 µV 0.00027 µV
bandpass 0.0093 µV 0.00019 µV

Shorter kernels miss the design goal. A 400 Hz transition (221 taps), for example, passes 2.2 times the Butterworth's power between 150 and 300 Hz.

Details
  • Alignment. Output sample n is sum_j taps[j] * x[n + center - j]. The default center makes a symmetric kernel zero-phase, and center=0 makes the filter causal.
  • Edges. The input is zero beyond the recording's edges, or reflected with add_reflect_padding=True, as in filter.
  • Integer output is rounded.
  • Offsets. offset_to_uV is scaled by the kernel's DC gain.
  • Serialization. The taps are serialized, because for this step they are the parameters.
  • Dependencies. Only scipy.fft and scipy.signal.

A follow-up PR adds method="fir" to highpass_filter and bandpass_filter, for users who don't want to design a kernel.

fir_filter(recording, taps) applies any FIR kernel, shared or per
channel, by overlap-save. Its margin is the kernel's support, so its
output does not depend on chunking. Channels are filtered in blocks
sized to keep each block's buffers in the L2 cache. design_fir_bandpass
and design_fir_highpass design Kaiser-window kernels from a stated spec:
-6 dB at the cutoffs, a given transition width at each edge, and a given
stopband attenuation. Keeping design separate from application lets
kernels be inspected or combined before use.

The defaults (175 Hz lower transition, 45 dB stopband) pass at most
0.03% of the probe's specified noise power below 150 Hz, and no more
than the default Butterworth between 150 and 300 Hz, on every channel of
a Neuropixels 2.0 recording. A 400 Hz lower transition passed 2.4 times
the Butterworth's power between 150 and 300 Hz.

check_chunk_invariance in core.testing compares chunked reads with one
whole read, so any preprocessor's seam error can be tested directly.
The core test jobs run without scipy and collect every test module, so a
module-level scipy import failed collection there.
@galenlynch

Copy link
Copy Markdown
Contributor Author

The test failures came from a module-level import scipy in test_fir_filter.py, which broke collection in the jobs without scipy; f795fbf moves it into the tests that use it.

"Load fixtures from 0.105.1" fails for a reason outside this PR. In .github/scripts/serialization/objects.py, the branch for versions after 0.105.0 expects set_probe to return a new recording:

else:
    rec_with_probe = rec.set_probe(probe)  # new API returns a new recording; portable across versions

In 0.105.1, set_probe sets the probe in place and returns None, so serialize_objects.py fails with 'NoneType' object has no attribute 'save'. 0.105.1 is the first release to take that branch. Something like this should fix it:

else:
    rec.set_probe(probe)
    rec_with_probe = rec

The job runs only for PRs that touch core, which this one does through core/testing.py.

@alejoe91 alejoe91 added preprocessing Related to preprocessing module performance Performance issues/improvements labels Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Performance issues/improvements preprocessing Related to preprocessing module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants