From ee6f09be339c66dfe087f46fb6dfc2f687a0798c Mon Sep 17 00:00:00 2001 From: Alex Kholodniak Date: Sun, 11 Oct 2026 23:29:18 +0300 Subject: [PATCH] Label the benchmarked build as 0.2.1 The environment line said 0.1.0, copied from stale install metadata in the benchmark venv. The measured revision 8e7aa7d has the same library and binding code as the 0.2.1 release. --- BENCHMARKS.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/BENCHMARKS.md b/BENCHMARKS.md index cd4bd5a..4072d00 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -64,7 +64,9 @@ memory costs.[^rss] ## Environment and results Run on 2026-10-11: Apple M1 Pro, 8 logical CPUs, macOS 14.5 arm64, Python 3.12.11. -Local `rouge-score-rs` 0.1.0 (Git revision `8e7aa7d`) was built with maturin 1.15.0 in release mode. +Local `rouge-score-rs` was built from Git revision `8e7aa7d` with maturin 1.15.0 in release mode; its library and +binding code is identical to the 0.2.1 release (the package list in the JSON shows a stale 0.1.0 from the +environment's install metadata). Comparison versions: `rouge-score` 0.1.2, `rouge-rust` 0.1.12, `rouge` 1.0.1, Evaluate 0.4.6, datasets 5.1.0, NumPy 2.5.4, NLTK 3.10.3 and PyArrow 26.0.0. All installed versions and the library Git revision are in @@ -148,7 +150,7 @@ to this benchmark. TorchMetrics is inspected at 1.8.2 and is not timed or instal | Package | Stemming | rougeLsum | Custom tokenizer | Multiple references | Bootstrap aggregation | Parity with rouge-score | Parallelism | Licence | |---|---|---|---|---|---|---|---|---|---| | rouge-score 0.1.2 [^google] | Porter | Yes; newlines by default | Object with `tokenize` | `score_multi`, best F1 per metric | `BootstrapAggregator` | Reference | Serial scoring; caller can distribute pairs | Apache-2.0 | -| rouge-score-rs 0.1.0 [^rs] | Compatible Porter | Yes; newlines by default | Same API, plus native `unicode-v1` and `character-v1` profiles | Same API | Compatible `BootstrapAggregator` | Exact on all tested full/subset configurations and Evaluate aggregates | Rayon batch; GIL released; custom tokenizer/sentence-splitting fallback is serial | Apache-2.0 | +| rouge-score-rs 0.2.1 [^rs] | Compatible Porter | Yes; newlines by default | Same API, plus native `unicode-v1` and `character-v1` profiles | Same API | Compatible `BootstrapAggregator` | Exact on all tested full/subset configurations and Evaluate aggregates | Rayon batch; GIL released; custom tokenizer/sentence-splitting fallback is serial | Apache-2.0 | | rouge-rust 0.1.12 [^rust] | No | No | No | No | No | Exact on the tested 1/2/L subset without stemming; not full API parity | Rayon batch; GIL released | MIT | | rouge 1.0.1 (pltrdy) [^pltrdy] | No | No compatible `rougeLsum`; its `rouge-l` is summary-level | No | No | No; optional arithmetic mean | Different: tokenization, counting and F1 smoothing | Serial Python | Apache-2.0 | | TorchMetrics ROUGEScore 1.8.2 [^torch] | Porter | Yes; NLTK sentence splitting | Callable tokenizer and normalizer | Best F1 or average | No in ROUGEScore; corpus arithmetic mean | Not bit-identical by default: tensor dtype, sentence splitting and aggregation differ | Python scoring loops; Metric supports distributed state synchronization | Apache-2.0 |