Skip to content

fix: decode split multibyte UTF-8 chunks in readStream with StringDecoder - #185

Open
kanchingyg wants to merge 1 commit into
tinylibs:mainfrom
kanchingyg:fix/utf8-split-chunks
Open

kanchingyg wants to merge 1 commit into
tinylibs:mainfrom
kanchingyg:fix/utf8-split-chunks

Conversation

@kanchingyg

Copy link
Copy Markdown

Fixes #183

readStream in src/main.ts converts each stream chunk with chunk.toString(). When a subprocess writes UTF-8 output and a 2-, 3-, or 4-byte character is split across two Buffer chunks, each half is decoded in isolation and replaced with U+FFFD.

This change uses StringDecoder('utf8') from node:string_decoder in readStream (decoder.write(chunk) per chunk followed by decoder.end()) without mutating the stream encoding on this.process.stdout.

Added an async test in src/test/main_test.ts that synchronizes two stdout writes split mid-character (buf.subarray(0, 3) then buf.subarray(3)) and verifies the decoded output matches 'ñé€🚀'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

readStream decodes each chunk separately — multibyte UTF-8 characters split across chunks become U+FFFD

1 participant