Repository navigation
Add sort_paths_par, a parallel external sort for .paths streams - #168
Merged
Merged
Conversation
Same signature and contract as sort_paths (sorted, deduplicated .paths output, same memory floor, path limit, run-file naming and cleanup), built on the ideas of the bigsort/sortbin tool: - The input is inflated straight into segment arenas, so paths are never copied. Background workers index each segment into 16-byte keys (an 8-byte big-endian prefix plus offset/length), sort and deduplicate them. - When the budget fills, sorted segments are merged into one run file in the background while decoding continues. - The final merge is cut into ranges by sampled splitters; each range is merged by one worker and deflated by one worker, primed with the previous range's last 32 KiB, and the raw deflate blocks are concatenated into a single zlib stream (as pigz does). Output decodes identically to sort_paths'; the compressed bytes differ. - Segment arenas, key arrays, merge buffers and deflate streams are recycled, which keeps memory near the budget across spills. A stream that ends partway through a path is an error, where sort_paths silently drops the partial path. On 109M paths (1.47 GB uncompressed), sort_paths takes 120 s at a 1 GiB budget, of which ~70 s is single-threaded deflate of the output. sort_paths_par takes 7.4 s at 1 GiB (931 MB peak RSS), 9.1 s at 256 MiB (293 MB) and 8.1 s fully in memory, on 64 threads. tests/paths_sort_par.rs runs sort_paths' tests against it, plus a differential test against sort_paths across budgets that force spills and intermediate merges, checked with a strict inflate that verifies adler32.
Owner
|
LGTM! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Same as
sort_paths, but uses multithreading for compression and sorting.16GB budget:
1GB budget:
