fix: preserve Unicode escapes in Java literals

[exKAZUu]

Why

The literal cases in tree-sitter/tree-sitter-java#210 fail on published 3.0.3: repeated u markers in string escapes, and quotes escaped through a Unicode-encoded backslash.

Requirements

• required: Review eligible upstream improvements and open dedicated, nonduplicate PRs, ignoring backward compatibility.

• chosen: Recognize repeated Unicode escape markers and Unicode-encoded backslashes in string and character literals, since these concrete upstream cases are valid Java and fail on the published parser.

• chosen: Extend the existing literal tokens without changing public node kinds; full Unicode preprocessing of the source is outside this focused literal repair.

• chosen: Consume Unicode character escapes atomically and pair encoded backslashes, because an overlapping raw escape fallback can consume a prefix and let a later encoded backslash swallow following declarations.

• chosen: Consume a Unicode-encoded backslash followed by a complete Unicode escape as one string escape token: encoded newline, quote, and tab bodies need the same complete highlight ranges as encoded backslashes; preserve raw-backslash eligibility. Consume mixed raw/encoded octal digits through the Java length limit (three digits only when starting with 0–3), because stopping at the first encoded digit truncates a single escape highlight.

• chosen: Preserve quote-bounded malformed or incomplete Unicode character literals as permissive character tokens, so editing them does not swallow following declarations. The fallback must end at the closing quote and cannot overlap complete encoded backslashes; a raw Unicode-prefix fallback regresses the existing paired-backslash controls.

• chosen: After a proven malformed Unicode prefix, use a quote-bounded tail that still consumes complete Unicode escapes and encoded/raw backslash pairs atomically. Keeping the normal encoded-backslash quote escape in that tail swallows following fields; treating every backslash as raw instead breaks seven paired-backslash controls.

Customer Summary

Literals such as "\uuu0041", "\u005c"", and '\u005c'' parse without losing following declarations.

Technical Summary

String escape tokens recognize repeated u markers, encoded backslashes, and bounded octal bodies. Raw and encoded octal escapes follow the Java digit limit: "\411" yields an escape \41 followed by the string fragment 1. The character token consumes Unicode escapes atomically and pairs encoded backslashes, preventing an overlapping raw escape from swallowing following declarations. A quote-bounded fallback retains malformed or incomplete Unicode character content without consuming neighboring declarations. The grammar retains raw source ranges and existing literal/escape node kinds. Java's Unicode translation precedes literal parsing, as specified in JLS §3.3.

Testing

Canonical generation, all 151 native and Wasm corpus cases, all 68 public tests across 11 files, and bun run verify pass. All PR CI checks, including both macOS runners, pass. Escape-body and mixed octal controls check complete source ranges, shipped highlight captures, and following declarations. Malformed Unicode prefixes followed by encoded backslashes reproduce field loss before the repair. A 675-case fragment matrix preserves every previously successful declaration boundary and fixes seven failures against the preceding head; all 360 incremental edits match fresh parse trees.

Published 3.0.3 reproduces the three original upstream failures. The initial grammar repair also passed 2,000 incremental fuzz iterations.

Notes

The grammar still accepts some invalid escape bodies such as \A, while applying the octal digit limit; it does not implement a complete source-wide Unicode translation phase.