Issues

• LongBench v2: questions whose API call failed are dropped and result.py divides by the rows written, so non-random failures raise the reported accuracy

  THUDM/LongBench#144 · Filter by author shaurya416 opened on Sep 23, 2026

• Would an EvalPort-format export of LongBench v2 be welcome?

  THUDM/LongBench#143 · Filter by author adhabnr-ux opened on Aug 22, 2026

• Qwen3-235B-A22B-Thinking-2507测试的配置问题

  THUDM/LongBench#139 · Filter by author YutingLi0606 opened on Mar 1, 2026

• Inquiry about Model Evaluation and Leaderboard Inclusion

  THUDM/LongBench#136 · Filter by author zhuangzhong10 opened on Feb 3, 2026

• PDF Source Content

  THUDM/LongBench#135 · Filter by author LeonEricsson opened on Nov 24, 2025

• 为什么推理的时候推多次总有100多条样本 答案为空

  THUDM/LongBench#133 · Filter by author llllllaz opened on Sep 28, 2025
  3

• Add evaluation script for memory-augmented models (A-Mem, Mem0, etc.) on LongBench & LongBench v2

  THUDM/LongBench#132 · Filter by author xcc-zach opened on Aug 7, 2025
  1

• Adapting pred.py to non-chat models.

  THUDM/LongBench#131 · Filter by author DragonEntropy opened on Aug 6, 2025

• Important context information missing for some Qasper tasks in LongBench v1

  THUDM/LongBench#130 · Filter by author VisionTheta opened on Jun 24, 2025

• o3?

  THUDM/LongBench#129 · Filter by author OverLordGoldDragon opened on Jun 6, 2025

• 数据集问题：ID 66f39ac5821e116aacb2da81 的答案无法在 context 中找到

  enhancement
  Bug

  THUDM/LongBench#128 · Filter by author xcc-zach opened on May 13, 2025
  1

• 怎么测试qwen2.5系列模型

  THUDM/LongBench#127 · Filter by author leileilin opened on May 12, 2025
  3

• [error need help] error to run longbench V1 following the document

  THUDM/LongBench#125 · Filter by author gaowayne opened on Apr 26, 2025

• [bugs and help need] I followed main readme page guideline, there is connection error and model running error

  THUDM/LongBench#124 · Filter by author gaowayne opened on Apr 26, 2025
  4

• How you evaluate reasoning models like QwQ-32B, since the response time and token length is very long?

  THUDM/LongBench#123 · Filter by author txchen-USTC opened on Apr 21, 2025
  5

• Reproducibility Question

  THUDM/LongBench#119 · Filter by author qinchuanhui opened on Apr 3, 2025

• A bug that has been mentioned many times: 'NoneType' object is not callable

  THUDM/LongBench#118 · Filter by author zZzZ9zZ9 opened on Apr 1, 2025

• Created a markdown version on GitHub

  documentation
  THUDM/LongBench#117 · Filter by author itsnamgyu opened on Mar 26, 2025
  1

• Do models really answer directly in the non-CoT setting?

  THUDM/LongBench#116 · Filter by author itsnamgyu opened on Mar 23, 2025
  1

• Confused about many shot learning problems (eg #340)

  THUDM/LongBench#115 · Filter by author Naqu6 opened on Mar 22, 2025

• Are agentic systems in scope for your leaderboard?

  THUDM/LongBench#114 · Filter by author L3Gaunt opened on Mar 5, 2025
  1

• Evaluation Setup for Reasoning Model

  THUDM/LongBench#113 · Filter by author YizhaoGao opened on Feb 25, 2025
  4

• Longbench评测使用模型以及参数设置问题

  THUDM/LongBench#112 · Filter by author Xiaoqi-Zhou916 opened on Feb 17, 2025
  1

• Reproducibility Issues

  THUDM/LongBench#111 · Filter by author Hisham-Cohere opened on Feb 7, 2025
  1

• The difference between longbench v2 and helmet?

  THUDM/LongBench#110 · Filter by author leileilin opened on Feb 6, 2025