Fix StreamingDecompress#decompress dropping the frames after an empty or skippable frame

[reidmorrison]

Summary

Zstd::StreamingDecompress#decompress stops as soon as a frame that produced no output ends, and silently drops the rest of its input. Empty and skippable frames produce no output, so every frame after one is lost:

skippable = [0x184D2A50, 4].pack('VV') + 'meta'

Zstd::StreamingDecompress.new.decompress(skippable + Zstd.compress('abc'))
# => ""     (before)
# => "abc"  (after)

Zstd::StreamingDecompress.new.decompress(Zstd.compress('') + Zstd.compress('abc'))
# => ""     (before)
# => "abc"  (after)

pzstd, the parallel compressor that ships with zstd, writes a skippable frame before each frame. For a file written by pzstd (400 KB of content), decompress returns 0 bytes when it is given the whole file, and raises Unknown frame descriptor when it is given 64 KiB chunks, as the second chunk starts in the middle of the frame that was dropped. Zstd.decompress, decompress_with_pos and the zstd command line tool decode it correctly.

I found it while adding StreamingDecompress#frame_complete? (#165) for IOStreams, a streaming file library that reads .zst files with Zstd::StreamingDecompress.

Cause

A regression from #116, in v2.0.1. Up to v2.0.0 the loop was while (input.pos < input.size). #116 first stopped it after any frame (do ... while (input.pos < input.size && ret > 0)), then changed that to the current break when ret == 0 && output.pos == 0, which still stops after a frame without output. Built from the tags, v2.0.0 returns "abc" for both examples above, and v2.0.1 returns "".

Fix

Remove the break, with a comment so that it does not come back. The loop still ends once all of the input is consumed: each call to ZSTD_decompressStream consumes input, flushes output, or returns an error after libzstd sees no progress over several calls, and the early return it takes on a truncated frame header consumes the rest of the input. I checked 8,026 truncated inputs (39 files written by the zstd command line tool and by this gem, cut at many points, passed whole and in 64 KiB chunks): decompress returns a prefix of the content for each, and it raises for 200 random inputs.

Compatibility

Only input with a frame after an empty or skippable frame, in the same call, changes: decompress now returns those frames' data instead of dropping it. All other input returns the same as before.

Tests

spec/zstd-ruby-streaming-decompress_spec.rb adds specs that pass several frames to decompress in one call: two frames, frames after empty frames, and frames after skippable frames, also split within the frame after the first skippable frame. The last two fail on main.

bundle exec rake passes with 94 examples, 0 failures (91 on main), on Ruby 3.2, 3.4 and 4.0 on macOS (arm64), with no new compiler warnings.

🤖 Generated with Claude Code