Add a checksum: option to Zstd.compress and Zstd::StreamingCompress

[reidmorrison]

Summary

Adds a checksum: keyword to Zstd.compress and Zstd::StreamingCompress.new. With checksum: true, each frame ends with the content checksum that the zstd format defines (the low 4 bytes of the XXH64 hash of the uncompressed data), which libzstd verifies when it decompresses. The default is false, so nothing changes unless it is set.

compressed = Zstd.compress(data, checksum: true)

stream = Zstd::StreamingCompress.new(level: 5, checksum: true)

Motivation

IOStreams is a Ruby library that streams files through compression, encryption and storage (local files, S3, SFTP) one block at a time. On MRI it writes .zst files with Zstd::StreamingCompress. The zstd command line tool writes the content checksum by default, and zstd-jni, which IOStreams uses on JRuby, can enable it, but zstd-ruby has no way to set ZSTD_c_checksumFlag. So the .zst files IOStreams writes on MRI carry no checksum, and corruption that leaves a frame structurally valid decompresses into wrong data without an error.

Flipping one random bit in a frame 1,500 times, and decompressing each copy with Zstd::StreamingDecompress:

Data	checksum: false	checksum: true
1 MB of CSV-like text	932 return wrong data, 568 raise	all 1,500 raise
200 KB of random bytes	1,499 return wrong data, 1 raises	all 1,500 raise

Change

set_compress_params in common.h, which Zstd.compress and Zstd::StreamingCompress#initialize share, accepts checksum: next to level: and dict:, and sets ZSTD_c_checksumFlag. It follows the existing keywords:

• nil, or leaving it out, means the default, false, which is also libzstd's default.
• A value other than true, false or nil raises ArgumentError, after freeing the context as the dict: check does.
• It is set before dict:, whose Zstd::CDict branch returns early. The checksum is a frame parameter, which a CDict does not override, so it works with either kind of dictionary.

Decompression needs no change: libzstd verifies the checksum whenever a frame has one, so Zstd.decompress, Zstd::StreamingDecompress and Zstd::StreamReader raise Restored data doesn't match checksum when it does not match. The streaming classes check it at the end of the frame, after returning that frame's data, which the README notes.

Zstd::StreamWriter only passes level: on to StreamingCompress (not dict: either), so I left it unchanged.

Compatibility

Without checksum: true, the output is byte for byte the same as before. With it, the frame is the same apart from the checksum flag in the frame header descriptor and the 4 checksum bytes at its end, which every zstd decoder supports. The checksum matches the one the zstd command line tool writes for the same content, and zstd -t accepts the frames.

Overlap with #162

#162 changes set_compress_params so that it no longer frees the context before raising, because its callers free it instead. This PR follows the convention on main, and frees the context before raising ArgumentError for an invalid checksum:. Git merges common.h cleanly either way (only the specs conflict), so whichever of the two is merged second has to drop that ZSTD_freeCCtx(ctx): a build with both crashes on Zstd.compress('x', checksum: 1). If #162 is merged first, I will rebase this PR onto it.

Tests

spec/zstd-ruby_spec.rb and spec/zstd-ruby-streaming-compress_spec.rb check that:

• by default, and with checksum: false or nil, no checksum is written and the output is unchanged;
• checksum: true sets the flag and adds 4 bytes, which for 'abc' are the low 4 bytes of its XXH64 hash, as the zstd command line tool writes them;
• it works with a String dictionary, a Zstd::CDict, and across StreamingCompress#flush;
• a flipped bit in incompressible content decompresses into wrong data without a checksum, and raises with one, from both Zstd.decompress and Zstd::StreamingDecompress;
• any other value raises ArgumentError.

bundle exec rake passes with 100 examples, 0 failures (91 on main), on Ruby 3.2, 3.4 and 4.0 on macOS (arm64), with no new compiler warnings.

🤖 Generated with Claude Code