All pull requests

Pull requests

• Check the content hash when caching is disabled

  #609·Linxiushen opened Sep 28, 2026·

  ·

• Clean up the temporary file if writing to the cache fails

  #607·Linxiushen opened Sep 21, 2026·

  ·

• feat: add explicit encoding cache release

  #605·mehrdadxzaker opened Sep 12, 2026·

  ·
  3

• Replace transmute-based hash_current_thread with a TLS-backed atomic counter

  #603·Berat121 opened Sep 10, 2026·

  ·

• Speed up educational BPE training

  #600·Marchematics opened Aug 31, 2026·

  ·
  1

  5

• Handle Data Gym vocab files without trailing newline

  #598·tandede opened Aug 18, 2026·

  ·
  1

• Validate base64 tokens when loading BPE files

  #592·weivwang opened Aug 12, 2026·

  ·

• Add get_token_length methods to more cheaply calculate token lengths

  #591·thomasaarholt opened Aug 11, 2026·

  ·

• Add DisallowedSpecialTokenError (ValueError subclass) for disallowed special tokens (#290)

  #590·JayYarlagadda opened Aug 7, 2026·

  ·
  1

• Fix remaining panics from #400 in encode_ordinary/_encode_bytes/encode_with_unstable

  #587·yanmo42 opened Jul 30, 2026·

  ·

• fix: Unable to route GET requests through proxy

  #584·thiagobarbosa opened Jul 23, 2026·

  ·
  1

• Avoid panic when byte_pair_encode is given an empty piece

  #583·devYRPauli opened Jul 20, 2026·

  ·
  1

• Handle surrogate pairs in encode_with_unstable

  #581·devYRPauli opened Jul 16, 2026·

  ·
  1

  7

• Add CI support for Windows ARM64

  #580·chinazhangchao opened Jul 14, 2026·

  ·
  5

• Return ValueError instead of panicking on duplicate mergeable_ranks

  #579·devYRPauli opened Jul 13, 2026·

  ·
  1

• fix: apply surrogate-pair fixup in encode_with_unstable

  #575·steps-re opened Jul 8, 2026·

  ·
  1

• fix: reject encoders missing single-byte tokens to prevent reachable panic (#568)

  #570·yen0304 opened Jun 14, 2026·

  ·
  1

• fix(model): add exact mapping for gpt-4.5

  #569·yen0304 opened Jun 12, 2026·

  ·

• fix(py): make encode_with_unstable handle surrogates like encode (#541)

  #567·MasterOfLogic1 opened Jun 7, 2026·

  ·
  1

• perf: speed up batch tokenization and decoding hot paths

  #561·eonr opened May 27, 2026·

  ·

• Fix CoreBPE::new type inference failure

  #558·fallintoplace opened May 24, 2026·

  ·
  1

• fix(py): make encode_with_unstable handle surrogates like encode (#541)

  #556·LeSingh1 opened May 24, 2026·

  ·
  1

• Repair surrogates in encode_with_unstable (Fixes #541)

  #553·jbbqqf opened May 23, 2026·

  ·
  1

• Add lexer fast path for o200k_base, cl100k_base, and gpt-2 pretokenization patterns; custom regexes fall back to fancy_regex

  #552·augustasio opened May 22, 2026·

  ·

• Add 2-byte pair lookup table to skip hashmap for initial BPE merge scan

  #551·augustasio opened May 22, 2026·

  ·