Skip to content

fix: escape control characters in grammar string literals - #681

Open
OsamaAnsar wants to merge 1 commit into
withcatai:masterfrom
OsamaAnsar:fix/gbnf-string-value-control-chars
Open

OsamaAnsar wants to merge 1 commit into
withcatai:masterfrom
OsamaAnsar:fix/gbnf-string-value-control-chars

Conversation

@OsamaAnsar

Copy link
Copy Markdown

Description of change

GbnfStringValue (used for enum / const string values and object property names) escaped only \, tab, CR, LF and " when building the grammar literal. Backspace, form feed and the other control characters U+0000 - U+001F were put into the literal raw. The model would then have to emit a raw control character, which is invalid inside a JSON string (and GbnfInsideStringChar excludes \x00-\x1F for ordinary strings), so JSON.parse of the output fails.

This escapes them the way JSON.stringify does: \b, \f, and \u00XX for the rest. Characters that are valid in JSON strings (DEL, non-ASCII, U+2028) are unchanged.

Before / after (real output of getGbnfGrammarForGbnfJsonSchema, control characters shown as <0xNN>):

{"enum": ["a<BS>b"]}      before: val0 ::= "\"a<0x08>b\""      after: val0 ::= "\"a\bb\""
{"enum": ["a<FF>b"]}      before: val0 ::= "\"a<0x0c>b\""      after: val0 ::= "\"a\fb\""
{"enum": ["a<0x01>b"]}    before: val0 ::= "\"a<0x01>b\""      after: val0 ::= "\"a\u0001b\""
{"const": "x<0x1f>"}      before: "\"x<0x1f>\""                after: "\"x\u001f\""
object property "k<BS>"   before: "\"k<0x08>\""                after: "\"k\b\""

Tests: new standalone test/standalone/utils/GbnfStringValue.test.ts (no native addon or model needed). It checks the escaped form for \b, \f, \u0000, \u0001, \u001f, that all 32 control characters match the JSON.stringify form and round-trip through JSON.parse, that the existing escapes are unchanged, and the full schema grammar for enum, const and property names. With only the source fix reverted 4 of the 6 tests fail; with the fix all 6 pass. tsc --noEmit and eslint on src/utils/gbnfJson and test/standalone/utils pass.

Not tested: the native addon could not be built in my environment, so the grammar was checked only as text (not compiled by llama.cpp, no sampling with a model), and LlamaGrammar.test.ts and the model-dependent tests were not run.

Pull-Request Checklist

  • Code is up-to-date with the master branch
  • npm run format to apply eslint formatting
  • npm run test passes with this change - the new tests, tsc --noEmit and eslint pass; tests that need the native addon / models were not run
  • This pull request links relevant issues as Fixes #0000 - N/A, no existing issue found
  • There are new or updated unit tests validating the change
  • Documentation has been updated to reflect this change - N/A, no API or usage change
  • The new commits and pull request title follow conventions explained in pull request guidelines (PRs that do not follow this convention will not be merged)

The grammar for `enum`, `const` and object property names escaped only backslash, tab, carriage
return, line feed and double quote. Other control characters, such as backspace, form feed and
U+0000 - U+001F, were put into the grammar literal as raw characters, so the model would have to
emit a raw control character that is invalid in JSON (and which the string rule of the grammar
forbids elsewhere).

Escape them like `JSON.stringify` does: `\b`, `\f`, and `\u00XX` for the rest.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant