[g-andrade]

With bidirectional attention, transformers halves the Gemma 3 sliding window (__post_init__), so EmbeddingGemma (sliding_window: 512) attends to ±256 tokens. Before this proposed change, bumblebee used ±511.

This is the window fix from #475, which was closed because of its size.

google/embeddinggemma-300m was checked against sentence-transformers on 110 texts of up to 1,353 tokens. Before, the minimum cosine was 0.987 on inputs over 512 tokens. Now it's 1.000000 on all of them.

Test reference values

from transformers import Gemma3TextConfig, Gemma3TextModel
import torch

repo = "bumblebee-testing/tiny-random-Gemma3TextModel"

config_dict = Gemma3TextConfig.from_pretrained(repo).to_dict()
config_dict.update(use_bidirectional_attention=True, sliding_window=4)
config = Gemma3TextConfig(**config_dict)

model = Gemma3TextModel.from_pretrained(repo, config=config)

inputs = {
    "input_ids": torch.tensor([[10, 20, 30, 40, 50, 60, 70, 80, 0, 0]]),
    "attention_mask": torch.tensor([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0]]),
}

outputs = model(**inputs)
print(outputs.last_hidden_state[:, 1:4, 1:4])

🤖 Generated with Claude Code