Can LLMs Access their World Knowledge for Event Prediction?

Paper: https://arxiv.org/abs/2610.19021

Recent work has questioned whether LLMs can use pretrained world knowledge for temporal event prediction. This project studies event forecasting as an interface problem: grounding events with natural-language descriptions and using symbolic labels to separate event semantics from the prediction space. It introduces DescribeEvent, a benchmark of about 150K events across seven datasets and six domains, and evaluates when capable LLMs benefit from descriptions and symbolic labels compared with description-free variants and non-LLM baselines.

Datasets

Hugging Face: https://huggingface.co/DescribeEvents

Each folder is an experiment entry for a different dataset.

• amazon_review_events/
• earthquake_region_events/
• gdelt_news_events/
• github_repo_events/
• github_user_events/
• nba_quarter_events/
• wikipedia_edit_events/

Each dataset is downloaded automatically from Hugging Face on the first run and cached under .cache/ in the repository root.

Prerequisites

Python 3.10+.

pip install -r requirements.txt

Running Experiments

Model and decoding parameters are set in each dataset's config.py.

Each driver runs four experimental setups:

1. Semantic event type with description
2. Semantic event type without description
3. Symbolic event type without description
4. Symbolic event type with description

We use OpenRouter for closed-source models, which makes it easy to switch between them.

# OpenRouter API
export OPENROUTER_API_KEY=<your-openrouter-key>
bash <dataset>/run_openrouter.sh

For open-source models, we run inference locally with vLLM. Local vLLM experiments use TENSOR_PARALLEL_SIZE = 2 by default. Our experiments were run on 2 NVIDIA RTX A6000 48GB GPUs.

# Local vLLM
CUDA_VISIBLE_DEVICES=0,1 bash <dataset>/run_vllm.sh

Outputs

Experiment outputs are written to <dataset>/results_<model>/ (created automatically).

Citation

If you find this work useful, please cite:

@article{wu2026llms,
  title   = {Can {LLM}s Access their World Knowledge for Event Prediction?},
  author  = {Wu, Zijun and Zhang, Zhitian and Hajimirsadeghi, Hossein and Dvornik, Nikita and Pashevich, Alexander},
  journal = {arXiv preprint arXiv:2610.19021},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.19021}
}

License

This project is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license. See LICENSE.