beam

beam ranks methods from the score tables of a benchmark and reports how much the ranking depends on the metric scaling, the weighting, the aggregation and the datasets. The scale type, direction, range and allowed transformations of each metric are stored in a metric card.

Documentation: how-tos, vignettes, and explanations.

Install

Python package:

python3.12 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,docs]"

[docs] installs Jupyter so Quarto can execute the Python code chunks in the vignettes. [io] installs pandas for the CSV adapter. [dev] covers the test suite.

R package:

library("remotes")
remotes::install_github("imallona/beam", subdir = "r/beam")
rbeam::install_beam_python()

Heterogeneity diagnostics (optional, needs R)

The ranking and its sensitivity diagnostics are pure Python. The heterogeneity diagnostics (Bradley-Terry trees, mixed-effects models, Plackett-Luce models, variance decomposition, network meta-analysis) call Rscript and need lme4, glmmTMB, psychotree, partykit, PlackettLuce, qvcalc, meta, netmeta and jsonlite. The conda recipe puts Python and R in one environment so that Rscript is on the path:

mamba env create -f envs/heterogeneity.yml
conda activate beam-heterogeneity
pip install -e ".[dev]"

From R, install them once with rbeam::install_beam_heterogeneity_deps(). The availability checks (beam.heterogeneity.r_available() and the related functions) report whether the toolchain is in place.

Usage

From a CSV to an HTML report.

Shell:

beam validate scores.csv
beam rank scores.csv --weights entropy --method topsis --out result.json --report report.html
beam report result.json --out report.html
beam metric show ari
beam heterogeneity scores.csv --model bradley-terry-tree --features features.csv --out tree.json
beam run beam.yaml

Python:

import beam
from beam.cards import Registry
from beam.config import run_config

beam.load_scores("scores.csv")
result = beam.rank("scores.csv", weights="entropy", method="topsis")
beam.report(result, "report.html")
print(Registry().get("ari"))
run_config("beam.yaml")

R:

library(rbeam)

result <- beam_rank("scores.csv", weights = "entropy", method = "topsis")
beam_validate("scores.csv")
beam_report(result, "report.html")
beam_metric_show("ari")
beam_run("beam.yaml")

Generated files

• Documentation site: vignettes, how-tos, explanations, and the Python API reference.
• Ontology release: docs/beam.owl.ttl (OWL) and docs/beam.skos.ttl (SKOS), regenerated from the cards on each release.

License

• Code: GPL-3.0-or-later (LICENSE).
• Metric cards under src/beam/metrics/: CC-BY-4.0 (src/beam/metrics/LICENSE.md).

Citation

Mallona, Izaskun (2026). beam: Benchmark Evaluation And Metrics. Version 0.3.1. https://github.com/imallona/beam. ORCID 0000-0002-2853-7526.

@software{mallona_beam_2026,
  author  = {Mallona, Izaskun},
  title   = {beam: Benchmark Evaluation And Metrics},
  version = {0.3.1},
  year    = {2026},
  url     = {https://github.com/imallona/beam},
  license = {GPL-3.0-or-later}
}

Contact

Izaskun Mallona, izaskun.mallona.work@gmail.com.

Related reading

• Commonly used software tools produce conflicting and overly-optimistic AUPRC values
• Performance Evaluation in Machine Learning: The Good, The Bad, The Ugly and The Way Forward
• Measurement theory and paleobiology

Started

21st Feb 2025