bug: getEmbeddingFor() never prepends BOS for UGM-vocab models, even when the GGUF sets add_bos_token = true (breaks bge-m3 CLS pooling) #683

Summary

For models whose vocabulary type is ugm (e.g. BAAI/bge-m3, an XLM-RoBERTa model converted with tokenizer.ggml.model = "t5"), LlamaEmbeddingContext.getEmbeddingFor(text) appends EOS (</s>) but never prepends BOS (<s>), even though the GGUF has tokenizer.ggml.add_bos_token = true and model.tokens.shouldPrependBosToken is true.

bge-m3 uses CLS pooling (bert.pooling_type = CLS), so the first token's output is the sentence embedding. Without <s>, the first word of the text gets pooled instead, and the embedding drifts away from the reference.

llama.cpp's own tokenizer does add BOS for UGM when add_bos is set (llama-vocab.cpp, case LLAMA_VOCAB_TYPE_UGM: if (add_special && add_bos) output.push_back(special_bos_id);), so node-llama-cpp currently diverges from llama.cpp here.

Cause

src/utils/tokenizerUtils.ts → resolveBeginningTokenToPrepend() returns null for ugm unconditionally, before checking tokens.shouldPrependBosToken:

if (vocabularyType === LlamaVocabularyType.ugm)
    return null;

This looks like it was written for T5 models (whose GGUFs have add_bos_token = false), but XLM-R-based embedding models like bge-m3 are also converted as UGM and do need BOS.

Reproduction

• node-llama-cpp 3.22.2 (also 3.20.0), macOS arm64 (Metal), Node 26
• Model: hf:ggml-org/bge-m3-Q8_0-GGUF/bge-m3-q8_0.gguf (same result with an F16 bge-m3 GGUF)

import {getLlama, resolveModelFile} from "node-llama-cpp";

const modelPath = await resolveModelFile("hf:ggml-org/bge-m3-Q8_0-GGUF/bge-m3-q8_0.gguf");
const llama = await getLlama();
const model = await llama.loadModel({modelPath});
// batchSize = contextSize to keep the batch-split issue (#674) out of the picture
const ctx = await model.createEmbeddingContext({contextSize: 2048, batchSize: 2048});

const cos = (a, b) => {
    let d = 0, x = 0, y = 0;
    for (let i = 0; i < a.length; i++) { d += a[i] * b[i]; x += a[i] ** 2; y += b[i] ** 2; }
    return d / Math.sqrt(x * y);
};
const vec = async (input) => Array.from((await ctx.getEmbeddingFor(input)).vector);

console.log(model.vocabularyType, model.tokens.shouldPrependBosToken, model.tokens.shouldAppendEosToken);

const text = "The quick brown fox jumps over the lazy dog. ".repeat(12);
const tokens = model.tokenize(text);
console.log(cos(await vec(text), await vec([model.tokens.bos, ...tokens, model.tokens.eos])));
console.log(cos(await vec(text), await vec([...tokens, model.tokens.eos])));

Output:

ugm true true
0.9891   <- string input vs. [<s>, ...tokens, </s>]
1.0000   <- string input is exactly [...tokens, </s>] (no BOS)

The gap is much larger on real documents. On Japanese Markdown chunks of 670–830 tokens (same context, batchSize: 2048), cos(string, [<s>, ...tokens, </s>]) was 0.52–0.62. Passing [<s>, ...tokens] (and letting getEmbeddingFor append </s>) gives 1.0000 in every case.

[<s>, ...tokens, </s>] is the correct reference here:

• Its token IDs match the official BAAI/bge-m3 tokenizer.json.
• Its vectors match Ollama's bge-m3 (/api/embed) at cosine ≈ 1.00.

Impact

We hit this through qmd, which calls getEmbeddingFor(string). We ran a 60-query retrieval test over ~3,450 chunks with bge-m3 Q8_0:

We did not measure the retrieval effect of the BOS bug on its own, without #674.

Suggested fix

In resolveBeginningTokenToPrepend(), let ugm fall through to the shouldPrependBosToken check (i.e. drop the ugm → null branch). This is what llama.cpp's tokenizer does. As far as I can tell, T5 GGUFs are converted with add_bos_token = false, so their behavior should stay the same (I have not checked every T5 conversion).

// before
if (vocabularyType === LlamaVocabularyType.ugm)
    return null;

// after: remove the branch above, so UGM falls through to
if (tokens.shouldPrependBosToken)
    return tokens.bos;

As a workaround, users can call getEmbeddingFor([model.tokens.bos, ...model.tokenize(text)]).

Related, but a separate problem: #674 / #678 (non-causal embeddings silently diverge when the input is longer than batchSize). bge-m3 hits both. This issue is only about the missing BOS.