corese-benchmark

Benchmark framework for comparing RDF triple stores: Corese, Apache Jena, and Eclipse RDF4J.

Description

This repository benchmarks Java-based RDF triple stores on loading and SPARQL query performance. Goals:

• Compare Corese against Apache Jena and RDF4J
• Detect regressions between corese.local (current branch) and corese.4.x.x (stable)
• Compare different versions of Corese (including corese-next)
• Produce JSON + CSV reports consumed by an interactive web dashboard

Principles:

• Java 21 harness with a rigorous measurement protocol (warmup + 3 repetitions + median)
• Memory measured as delta before/after loading with forced GC
• Automatic JSON + CSV output
• Run history for trend tracking
• Python orchestration for dataset download, build, and deployment

---

Dashboard

The interactive benchmark dashboard is published at: https://corese-stack.github.io/corese-benchmark/

To view it locally, open site/index.html in a browser.

---

How to Run

Prerequisites

• Java 21+
• Conda (for Python orchestration)

conda env create -f python-utils/environment.yml
conda activate benchmarkenv

---

A) Detect a Corese regression (corese.local)

Main use case: compare your local branch against the stable release.

1. Build corese-core locally

Inside the corese-core repository:

./gradlew shadowJar

2. Run the benchmark

If corese-core is in the sibling directory (../corese-core/), the JAR is detected automatically:

conda activate benchmarkenv
cd corese-benchmark
python python-utils/workflow.py --engines corese.local,corese.4.6.5,jena.6.2.0,rdf4j.5.3.2

If corese-core is elsewhere, use --coreseLocalPath:

python python-utils/workflow.py \
  --engines corese.local,corese.4.6.5 \
  --coreseLocalPath C:\path\to\corese-core

3. Read the result

Open site/index.html. The corese.local card shows:

• +X% improvement (green)
• −X% regression (red)

relative to corese.4.6.5 as the fixed reference (100/100).

---

B) Compare multiple engines

python python-utils/workflow.py --engines corese.4.6.5,jena.6.2.0,rdf4j.5.3.2

Default dataset: dbpedia. To change:

python python-utils/workflow.py --engines corese.4.6.5,jena.6.2.0 --dataset bowlogna

---

C) Compare Corese versions

One build per Corese version:

python python-utils/workflow.py --coreseVersions 4.6.5,4.6.3

---

D) Compare Corese commits

Clones, checks out, and compiles each commit from GitHub:

python python-utils/workflow.py --coreseCommits a17f3d6,b089a03

---

E) Benchmark with SPARQL queries

For datasets that have associated queries (e.g. Bowlogna):

python python-utils/workflow.py --engines corese.local,corese.4.6.5 --dataset bowlogna

The SPARQL Queries section appears in the dashboard with:

• Per-engine latency cards: avg, p50, p95, p99
• Median time per query and per engine
• Color-coded comparison table (green = best, orange = ×5–19, red = ×20+)

---

Run the Java harness directly

# Build
./gradlew clean build -PcoreseVersion=4.6.5 -PjenaVersion=6.2.0 -Prdf4jVersion=5.3.2

# Run (without queries)
./gradlew runBenchmark --args="input/dbpedia/unzip out/json/report.json corese.4.6.5,jena.6.2.0 dbpedia"

# Run (with queries)
./gradlew runBenchmark --args="input/dbpedia/unzip out/json/report.json corese.4.6.5 dbpedia queries.txt"

---

Outputs

Each run automatically generates:

File	Description
out/json/<timestamp>_<engines>.json	Full report (loaded by the dashboard)
out/csv/<timestamp>_<engines>.csv	Same data in CSV format (Excel, etc.)
site/data/benchmark-report.json	Copy of the latest report (current dashboard)
site/data/benchmark-report.csv	Copy of the latest CSV (download button)
site/data/history/<timestamp>.json	History entry (last 30 runs kept)
site/data/history-data.js	Inline history (trend chart)

The historical trend chart appears in the dashboard once 2 or more runs are available.

---

Measurement Protocol

For each file and each engine:

1. Warmup — load → clear → GC (result discarded)
2. 3 repetitions:
   • Forced GC + memory snapshot before
   • Load file
   • Forced GC + memory snapshot after
   • Record time and memory delta
3. Keep the median of the 3 repetitions

This protocol ensures reproducible results by eliminating JIT compilation bias and GC timing noise.

Note on clear(): RDF4J performs a full repository reinitialisation between repetitions (rather than just conn.clear()) to release internal SAIL memory structures and avoid negative memory delta artefacts caused by delayed GC.

---

Metrics

Metric	Description
loadingTimeMs	Median loading time over 3 repetitions (ms)
throughputTriplesPerSec	Triples loaded per second
memoryDeltaMb	Memory consumed by loading (after − before GC, MB)
cpuPercent	JVM CPU usage during loading (via OperatingSystemMXBean)
repetitionsMs	Raw values of the 3 repetitions (for transparency)

---

Datasets

DBpedia sample

5 files sampled from the DBpedia 3.5.1 dump:

• category_labels_en.nt
• disambiguations_en.nt
• homepages_en.nt
• persondata_en.nt
• specific_mappingbased_properties_en.nt

python python-utils/workflow.py --dataset dbpedia --engines corese.4.6.5,jena.6.2.0,rdf4j.5.3.2

Bowlogna

Synthetic dataset describing student/university relations. Includes associated SPARQL queries for latency benchmarking. Source: DICE Research / HOBBIT

Scale	Departments	Triples (approx.)
small	1	~1.2 M
medium	5	~6 M
large	10	~12 M

python python-utils/workflow.py --dataset bowlogna --scales small --engines corese.4.6.5,jena.6.2.0,rdf4j.5.3.2

BSBM

Berlin SPARQL Benchmark — synthetic e-commerce dataset. Generated with bsbmtools 0.2.

Scale	Products	Triples (approx.)
small	1,000	~50 K
medium	50,000	~1 M
large	500,000	~10 M

python python-utils/workflow.py --dataset bsbm --scales small,medium --engines corese.4.6.5,jena.6.2.0,rdf4j.5.3.2

LUBM

Lehigh University Benchmark — university domain with RDFS reasoning potential. Generated with UBA 1.7.

Scale	Universities	Triples (approx.)
small	1	~100 K
medium	10	~1 M
large	100	~10 M

python python-utils/workflow.py --dataset lubm --scales small --engines corese.4.6.5,jena.6.2.0,rdf4j.5.3.2

---

Engines

Engine	Default version	Notes
Corese	4.6.5	Use corese.local to benchmark a local branch
Corese 5 (next)	local JAR	corese.next — uses the new Repositories/RepositoryConnection API. Requires a Corese 5 JAR in libs/ and --coreseLocalPath or sibling corese-core/.
Apache Jena	6.2.0
Eclipse RDF4J	5.3.2	Malformed IRIs are skipped with a WARN log (DBpedia homepages_en.nt contains ~300 invalid IPv4 URIs)

---