Xiaoxing Hu†, Wenxi Yao*, Chaoyuan Shi*, Ziqi Ye*, Xiaorong Guo, Peixuan Li, Zhuoran Sun, Ziyang Gong, Peiyuan Zhang, Mingxin Liu, Shuran Ma, Hongbo Liu, Xiaomin Yu, Yu Tian, Wentao Wang, Zhenjie Yang, Jian Hu, Yunhang Shen, Haoyu Cao, and Xue Yang✉
† Project Lead · * Core Contributions · ✉ Corresponding Author
If EvaOmni is useful for your research, please consider starring this repository and citing our work！ ⭐⭐⭐
EvaOmni is an evidence-grounded benchmark for evaluating audio-visual reasoning in omni-modal large language models. Existing multiple-choice benchmarks often score only whether a model selects the correct option. A model can therefore obtain credit from textual, logical, or single-modality shortcuts without identifying the evidence that supports its answer. EvaOmni requires a model to return both an answer and timestamped audio and visual evidence. The benchmark reports answer accuracy together with evidence coverage and temporal grounding, making it possible to measure whether a correct answer is supported by the relevant moments in both modalities.
Shortcut evaluation across benchmarks, with examples of text-only, audio-only, and visual-only shortcuts.
- Evidence-grounded evaluation: models predict an answer together with audio and visual evidence intervals.
- Audio-visual reasoning: questions are designed to require complementary information from the two modalities.
- Shortcut screening: construction includes text-only, audio-only, and visual-only checks.
- Fine-grained taxonomy: 1,027 questions from 392 videos cover 15 capabilities across 3 reasoning levels.
- Grounded metrics: Coverage Grounded Score (C.G.) measures evidence coverage, while Bottleneck Grounded Score (B.G.) measures the weaker of the two modality-localization scores.
- Transparent evaluation: this initial release includes a runnable metric evaluator, a prediction schema, and a worked example. Provider-specific inference adapters and the original prompt templates are planned.
Annotation pipeline and a grounded question example with timestamped audio and visual evidence.
| Item | Value |
|---|---|
| Videos | 392 |
| Question-answer pairs | 1,027 |
| Fine-grained capabilities | 15 |
| Reasoning levels | 3 |
| Video duration | 1–12 minutes |
| Evaluation frame rate | 1.0 FPS |
| Evaluation resolution | 480 pixels on the longer side |
Capability taxonomy and dataset statistics, including video duration, token length, evidence, and video categories.
- Level 1 — Retrieval and Grounding: temporal alignment, event binding, and entity binding.
- Level 2 — Basic Understanding: causal reasoning, emotional intelligence, intent understanding, atmosphere understanding, implicit event reasoning, semantic disambiguation, temporal understanding, spatial understanding, and conditional counting.
- Level 3 — Complex Reasoning: multi-hop reasoning, long-range reasoning, and multi-evidence chains.
The current manuscript evaluates 7 open-weight and 17 proprietary Omni LLMs, with a human expert reference baseline. The table below reports Acc., C.G., and B.G. for all three capability levels and overall. All values are percentages and higher is better. Within each model group, bold indicates the best results and underlining indicates the second-best results. These are manuscript results, not outputs generated by this starter release.
| Model | Level 1 R.&G. | Level 2 B.U. | Level 3 C.R. | Overall | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Acc. | C.G. | B.G. | Acc. | C.G. | B.G. | Acc. | C.G. | B.G. | Acc. | C.G. | B.G. | |
| Open-Weight Omni LLMs | ||||||||||||
| Qwen2.5-Omni-7B | 16.6 | 1.2 | 0.6 | 17.4 | 0.5 | 0.0 | 13.7 | 0.3 | 0.0 | 16.4 | 0.8 | 0.3 |
| Qwen3-Omni-Instruct-30B-A3B | 28.6 | 5.1 | 0.6 | 22.4 | 3.4 | 0.4 | 22.2 | 2.4 | 0.1 | 25.2 | 4.0 | 0.4 |
| Qwen3-Omni-Thinking-30B-A3B | 26.3 | 11.4 | 4.8 | 24.4 | 6.2 | 1.2 | 21.6 | 4.1 | 0.6 | 24.8 | 8.3 | 2.8 |
| Nemotron3-Nano-Omni-30B-A3B | 30.7 | 15.0 | 5.3 | 22.1 | 13.1 | 3.9 | 10.5 | 4.7 | 0.7 | 24.2 | 12.6 | 4.0 |
| MiMo-V2.5-310B-A15B | 65.4 | 61.3 | 30.8 | 52.3 | 42.3 | 15.1 | 45.1 | 29.6 | 6.7 | 56.8 | 48.4 | 20.5 |
| MiMo-V2.6-Flash-310B-A15B | 61.3 | 55.3 | 29.0 | 48.6 | 41.4 | 14.5 | 45.8 | 28.6 | 5.9 | 53.6 | 45.3 | 19.3 |
| MiMo-V2.6-Pro-1T-A42B | 60.1 | 51.3 | 28.8 | 44.3 | 35.2 | 12.0 | 49.7 | 26.6 | 5.0 | 51.8 | 40.7 | 18.1 |
| Proprietary Omni LLMs | ||||||||||||
| Doubao-Seed-2.0-Lite | 54.1 | 36.4 | 10.8 | 54.1 | 36.1 | 11.7 | 62.7 | 32.2 | 5.9 | 55.4 | 35.6 | 10.4 |
| Doubao-Seed-2.1-Lite | 78.8 | 74.8 | 38.0 | 67.0 | 61.5 | 22.6 | 59.5 | 42.7 | 9.4 | 70.9 | 64.3 | 27.2 |
| Qwen3.5-Omni-Flash | 47.5 | 41.9 | 15.5 | 30.9 | 24.0 | 7.2 | 29.4 | 15.7 | 2.7 | 37.7 | 30.4 | 10.0 |
| Qwen3.5-Omni-Plus | 58.1 | 52.0 | 22.1 | 42.3 | 26.0 | 8.6 | 42.5 | 19.8 | 3.3 | 49.0 | 36.0 | 13.5 |
| Qwen3.8-Omni-Flash | 65.0 | 53.6 | 19.3 | 52.3 | 41.4 | 11.8 | 56.2 | 35.9 | 7.8 | 58.2 | 45.7 | 14.4 |
| Gemini2.5-Flash | 59.2 | 55.2 | 26.7 | 39.3 | 31.4 | 11.8 | 41.8 | 24.6 | 4.0 | 48.1 | 40.4 | 16.9 |
| Gemini2.5-Flash-Lite | 53.0 | 47.2 | 22.1 | 30.9 | 22.4 | 8.0 | 27.5 | 14.8 | 2.2 | 39.7 | 31.8 | 13.1 |
| Gemini2.5-Pro | 76.5 | 73.8 | 42.3 | 65.0 | 55.0 | 22.5 | 72.5 | 45.8 | 9.4 | 71.0 | 61.6 | 28.9 |
| Gemini3.1-Flash-Lite | 59.4 | 52.8 | 24.2 | 36.6 | 30.7 | 10.5 | 32.0 | 19.0 | 3.0 | 45.6 | 38.3 | 15.2 |
| Gemini3.1-Flash-Lite-Preview | 59.7 | 53.6 | 25.2 | 39.3 | 31.1 | 10.3 | 38.6 | 23.2 | 3.3 | 47.8 | 39.4 | 15.6 |
| Gemini3.1-Pro-Preview | 74.7 | 72.8 | 41.5 | 68.6 | 59.8 | 24.8 | 76.5 | 48.7 | 11.2 | 72.3 | 63.7 | 29.8 |
| Gemini3.5-Flash | 64.1 | 59.4 | 28.8 | 69.5 | 61.4 | 27.4 | 77.8 | 47.8 | 11.3 | 68.5 | 58.6 | 25.6 |
| Gemini3.5-Flash-Lite | 60.6 | 55.6 | 26.5 | 38.2 | 27.9 | 10.7 | 42.5 | 24.2 | 3.5 | 48.3 | 39.1 | 16.3 |
| Gemini3.6-Flash | 80.9 | 78.0 | 44.7 | 68.4 | 61.3 | 26.1 | 73.2 | 44.2 | 10.6 | 74.4 | 65.8 | 31.7 |
| Gemini3.7-Flash | 85.3 | 84.0 | 49.1 | 73.6 | 65.3 | 29.2 | 79.1 | 49.1 | 12.3 | 79.4 | 70.8 | 35.1 |
| Gemini3.8-Flash | 83.4 | 81.9 | 48.4 | 73.9 | 63.3 | 27.4 | 74.5 | 43.1 | 9.5 | 78.0 | 68.2 | 33.6 |
| Gemini4-Argon | 67.1 | 55.9 | 29.3 | 52.0 | 33.7 | 13.0 | 72.5 | 33.5 | 7.5 | 61.4 | 43.0 | 19.1 |
| Reference baseline | ||||||||||||
| Human Expert | 97.8 | 97.8 | 95.8 | 96.5 | 94.8 | 93.2 | 94.4 | 93.8 | 93.0 | 96.8 | 96.0 | 94.3 |
Gemini3.7-Flash achieves 79.4% accuracy but 35.1% B.G. overall. This gap illustrates why answer correctness and evidence localization should be assessed together. Humans may pause and replay videos in the reported baseline; viewing conditions differ from model inference.
Our dataset is hosted on Hugging Face and ModelScope respectively. Use either ModelScope or Hugging Face:
Download command:
pip install modelscope
modelscope download --dataset VisionXLab/EvaOmni-Dataset --local_dir ./dataor
pip install huggingface_hub
huggingface-cli download --repo-type dataset --resume-download VisionXLab/EvaOmni-Dataset --local-dir ./data
Note
Use resized_video to reproduce the paper's results, or rawvideo (original videos) for custom preprocessing.
Clone the repository.
git clone https://github.com/VisionXLab/EvaOmni.git
cd EvaOmniDownload the dataset using either command above. The resulting directory structure is:
data/
├── test_file.json
├── rawvideo/
│ ├── 001.mp4
│ ├── 002.mp4
│ └── ...
└── resized_video/
├── 001.mp4
├── 002.mp4
└── ...
Set model_name, base_url, and api_key in scripts/eval_example.py. Implement call_api() using your provider's SDK or HTTP API, ensuring that both video and audio are supplied, then remove the unconditional API-adapter placeholder error in main(). With Python 3.10+, run the following command from the repository root:
python scripts/eval_example.py \
--annotations data/test_file.json \
--video-dir data/resized_video \
--output-dir outputs/my_model \
--workers 4 \
--batch-size 16 \
--retries 3The script displays progress and saves each completed result. Rerun the same command to resume: successful questions are skipped, and failed questions are retried. Raw responses, predictions, and logs are saved in the output directory; after evaluation, summary.json and grounded_scores.json report accuracy and grounded metrics. To check the pipeline without calling an API, add --mock --limit 8 and use a separate output directory such as outputs/mock_example.
@article{hu2026evaomni,
title={EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning},
author={Hu, Xiaoxing and Yao, Wenxi and Shi, Chaoyuan and Ye, Ziqi and Guo, Xiaorong and Li, Peixuan and Sun, Zhuoran and Gong, Ziyang and Zhang, Peiyuan and Liu, Mingxin and Ma, Shuran and Liu, Hongbo and Yu, Xiaomin and Tian, Yu and Wang, Wentao and Yang, Zhenjie and Hu, Jian and Shen, Yunhang and Cao, Haoyu and Yang, Xue},
journal={arXiv preprint arXiv:2610.28444},
year={2026}
}Repository code is released under the MIT License. For dataset annotation and media terms, refer to the dataset hosting pages and docs/data_license.md.
We thank all the annotators, reviewers, and open-source communities whose work made this benchmark possible. Please open an issue if you find a broken link, an annotation problem, or an evaluation mismatch.