Skip to content

Skip the float64 copy of pandas and Polars input - #786

Merged
mrjbq7 merged 3 commits into
devfrom
astype-copy-false
Oct 29, 2026
Merged

mrjbq7 merged 3 commits into
devfrom
astype-copy-false

Conversation

@mario4tier

@mario4tier mario4tier commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

to_numpy() already returns float64 for a float64 Series, and astype(float) then copied it again on every call, in the Function, abstract and stream APIs. With copy=False, astype converts only input that is not float64 yet, so results are unchanged.

Median time per call, before and after, for pandas float64 into ATR. Two source builds per host (dev and this branch, both with TA-Lib C 0.8.3), 3 processes each, Python 3.12, numpy 2.5.3, pandas 3.1.0, polars 1.44.2:

API rows i7-13700K Ryzen 7 PRO 8840U
Function 10,000 21.9 → 18.0 µs, 18.0% faster 28.3 → 23.0 µs, 18.8% faster
Function 100,000 182 → 109 µs, 39.9% faster 202 → 127 µs, 37.2% faster
abstract 10,000 54.2 → 49.5 µs, 8.7% faster 64.1 → 55.7 µs, 13.0% faster
abstract 100,000 224 → 151 µs, 32.4% faster 259 → 161 µs, 37.7% faster
stream open 10,000 13.9 → 9.9 µs, 28.5% faster 16.3 → 12.1 µs, 25.4% faster
stream open 100,000 145 → 79 µs, 45.2% faster 154 → 94 µs, 38.8% faster

Opening a stream on a function that reads little history gains most: ADD at 100,000 rows is 47.6 → 1.5 µs, 96.8% faster. Polars gains more than pandas; nullable Float64, Arrow-backed and 2-D-built DataFrame columns gain too. Outputs were identical in every case, including leading NaN or null.

Two things change:

  • An int64 Series still needs the conversion and now also pays for the keyword argument: up to 3.3% slower at 1,000 rows (stream open of ADD, 2.46 → 2.54 µs on the i7). NumPy input is unaffected.
  • The kernels run without the GIL, so a thread writing to the same pandas Series during a call is now seen by the kernel, as it already is with NumPy input; the copy used to hide it. This matters most on free-threaded Python.

astype(float) copied every float64 Series again after to_numpy();
with copy=False, only input that is not already float64 is converted.
@mrjbq7

mrjbq7 commented Oct 29, 2026

Copy link
Copy Markdown
Member

Nice!

@mrjbq7
mrjbq7 merged commit 649655a into dev Oct 29, 2026
7 checks passed
@mrjbq7

mrjbq7 commented Oct 29, 2026

Copy link
Copy Markdown
Member

Oh oops I thought this was to main branch!

@mrjbq7

mrjbq7 commented Oct 29, 2026

Copy link
Copy Markdown
Member

But I liked it, so I cherry-picked it to main.

@mario4tier

Copy link
Copy Markdown
Member Author

No problem for the merge to dev. Glad that this is heading in the next release ... one way or another 😄

@mario4tier
mario4tier deleted the astype-copy-false branch October 29, 2026 04:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants