Repository navigation
fix: escape control characters in grammar string literals - #681
Open
OsamaAnsar wants to merge 1 commit into
Open
OsamaAnsar wants to merge 1 commit into
OsamaAnsar wants to merge 1 commit into
Conversation
The grammar for `enum`, `const` and object property names escaped only backslash, tab, carriage return, line feed and double quote. Other control characters, such as backspace, form feed and U+0000 - U+001F, were put into the grammar literal as raw characters, so the model would have to emit a raw control character that is invalid in JSON (and which the string rule of the grammar forbids elsewhere). Escape them like `JSON.stringify` does: `\b`, `\f`, and `\u00XX` for the rest.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of change
GbnfStringValue(used forenum/conststring values and object property names) escaped only\, tab, CR, LF and"when building the grammar literal. Backspace, form feed and the other control characters U+0000 - U+001F were put into the literal raw. The model would then have to emit a raw control character, which is invalid inside a JSON string (andGbnfInsideStringCharexcludes\x00-\x1Ffor ordinary strings), soJSON.parseof the output fails.This escapes them the way
JSON.stringifydoes:\b,\f, and\u00XXfor the rest. Characters that are valid in JSON strings (DEL, non-ASCII, U+2028) are unchanged.Before / after (real output of
getGbnfGrammarForGbnfJsonSchema, control characters shown as<0xNN>):Tests: new standalone
test/standalone/utils/GbnfStringValue.test.ts(no native addon or model needed). It checks the escaped form for\b,\f,\u0000,\u0001,\u001f, that all 32 control characters match theJSON.stringifyform and round-trip throughJSON.parse, that the existing escapes are unchanged, and the full schema grammar forenum,constand property names. With only the source fix reverted 4 of the 6 tests fail; with the fix all 6 pass.tsc --noEmitand eslint onsrc/utils/gbnfJsonandtest/standalone/utilspass.Not tested: the native addon could not be built in my environment, so the grammar was checked only as text (not compiled by llama.cpp, no sampling with a model), and
LlamaGrammar.test.tsand the model-dependent tests were not run.Pull-Request Checklist
masterbranchnpm run formatto apply eslint formattingnpm run testpasses with this change - the new tests,tsc --noEmitand eslint pass; tests that need the native addon / models were not runFixes #0000- N/A, no existing issue found