Skip to content

fix(server): preserve delimiters in string metadata overrides - #2380

Draft
allenflux wants to merge 2 commits into
abetlen:mainfrom
allenflux:allenflux/fix-server-kv-string-delimiters
Draft

allenflux wants to merge 2 commits into
abetlen:mainfrom
allenflux:allenflux/fix-server-kv-string-delimiters

Conversation

@allenflux

Copy link
Copy Markdown

String metadata overrides such as tokenizer.chat_template=str:user: or tokenizer.chat_template=str:{{ messages[0]['role'] == 'user' }} currently fail during server model setup with ValueError: too many values to unpack. The parser splits the entire value at each = and :, including delimiters inside the string.

Split only once at the key/value and type/value boundaries so the string reaches the model constructor unchanged. Add regression tests for colons, equals signs, empty values, Unicode, and existing numeric/boolean overrides, plus a changelog entry.

Validation on macOS arm64, Python 3.12.11, with the repository's pinned native library built with GPU backends disabled:

  • Original source: six string regression cases fail; two control cases pass.
  • make test with PYTEST_ADDOPTS="tests/test_server_model.py tests/test_llama_chat_format.py tests/test_llama_grammar.py tests/test_llama_speculative.py": 14 passed.
  • make lint: Ruff checks and formatting pass across llama_cpp and tests.

The new tests use real server settings and loader code, mocking only model construction. No model weights were downloaded; the full model/inference suite was not run. The native dependency was built unchanged.

AI assistance was used for investigation, implementation, and validation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant