[hassiebp]

What does this PR do?

Adds an optional experiment_id parameter to Langfuse.run_experiment, DatasetClient.run_experiment and RunnerContext.run_experiment. All calls that pass the same id record their items and scores in one experiment. This lets pipelines that split items across workers, or resume after an interruption, build a single experiment. Before this, that only worked for Langfuse dataset items, through the id derived from run_name. For local data, every call got a random id.

Semantics:

• A supplied id always wins, for local data and dataset items alike. It is used as-is for langfuse.experiment.id on every item, for dataset_run_id on run-level scores, and (URL-encoded) in experiment_url. Dataset items still record their dataset and item ids. Silently ignoring an explicit argument on dataset data would be more surprising, and the server has no experiment entity to conflict with: it only groups events by the id string.
• Omitted: unchanged. Dataset items get the derived id; local data gets a random one.
• Empty or whitespace-only id: raises ValueError before any item runs. The server drops blank experiment ids.

The docstring explains how to create a reusable id:

• Langfuse.create_trace_id() for a random id that callers store and share.
• Langfuse.create_trace_id(seed="...") for a deterministic id that every worker derives independently. The JS SDK's createTraceId(seed) returns the same value, since both hash the seed with SHA-256 and take the first 16 bytes as hex.

It also notes that run_evaluators only see the items of their own call, so each call adds its own run-level scores.

The JS SDK is getting the same parameter as experimentId, with the same semantics, in a parallel PR.

Type of change

• Bug fix
• New feature
• Breaking change
• Refactor
• Documentation update
• Tooling, CI, or repo maintenance

Verification

uv run --frozen ruff check .                      # All checks passed!
uv run --frozen ruff format --check .             # 575 files already formatted
uv run --frozen mypy langfuse --no-error-summary  # clean
uv run --frozen pytest -n auto --dist worksteal tests/unit   # 940 passed, 2 skipped

• The new tests in TestCallerSuppliedExperimentId cover:
  • two local runs sharing one experiment (spans plus run scores)
  • a supplied id overriding the derived dataset id
  • URL encoding of the id
  • a call from inside a running event loop
  • rejection of blank ids
  • forwarding of the id by DatasetClient and RunnerContext
• Removing the "supplied id wins" guard fails 4 of the new tests.
• End-to-end run against a local Langfuse (web 4.63.0): two separate worker processes each derived create_trace_id(seed=...) → e5bd20a8dacf6fc3f5982073d88d782b and ran 2 and 1 local items.
  • GET /api/public/experiments returned a single experiment with that id and itemCount: 3.
  • GET /api/public/experiment-items returned 3 items.
  • ClickHouse held 3 item scores and 2 run-level scores (one per call) under the id.
  • The 32-hex format passes through ingestion (String column), score datasetRunId (z.string(), no lookup) and the experiments UI routes (z.string()).
• I did not open the UI page visually.

Checklist

• I self-reviewed the diff using code_review.md.
• I added or updated tests for behavior changes.
• I updated docs, examples, or .env.template if needed.
• I did not hand-edit generated files; if generated files changed, I used the upstream regeneration path.
• I did not commit secrets or credentials.

  

Confidence Score: 5/5

The PR appears safe to merge; no actionable issues were found.

Summary

Adds optional experiment_id support to Langfuse.run_experiment, DatasetClient.run_experiment, and RunnerContext.run_experiment.

• Experiment calls can share a caller-chosen ID.

No actionable issues found. Tests were inspected, not executed.

_{Reviews (1) · Last reviewed commit: "feat(experiments): accept a caller-suppl..." · Reviewed by Greptile}

[github-actions]

@claude review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-20T08:20:32.416681Z	2d0ddcf	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[claude]

Claude Code Review

No review was started: this request came from a bot account. Manual reviews can only be requested by someone with write access to this repository. Ask a maintainer to comment @claude review, or have your automation post the comment from a user account with write access.

_{Tip: disable this comment in your organization's Code Review settings.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2d0ddcf1e6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Update ExperimentResult's experiment-ID contract

When callers supply this argument, the public ExperimentResult reference remains contradictory: langfuse/experiment.py:332-335 still states that dataset IDs are always derived and local IDs are always random, although the returned ID can now be caller-supplied in either case. Update that generated-reference docstring so users relying on the result type see the new contract.

AGENTS.md reference: AGENTS.md:L130-L134

Useful? React with 👍 / 👎.

[claude]

LGTM. This adds an opt-in experiment_id parameter that is passed through cleanly, and nothing changes when it is left out.

What was reviewed:

• How experiment_id is passed from Langfuse.run_experiment, DatasetClient.run_experiment and RunnerContext.run_experiment down to _run_experiment_async. All of these signatures are keyword-only (*), so putting the parameter before _dataset_version cannot break positional callers.
• The supplied id replaces _create_experiment_id only when it is given. The same value is then used for span attributes, dataset_run_id on run-level scores, and the result. Blank or non-string ids raise an error before any item runs.
• The experiment URL now escapes the id with urllib.parse.quote(..., safe=''). I also looked at the earlier candidate about a missing length check on the id and ruled it out: the value is caller-controlled and its own docs say to create it with create_trace_id.

Extended reasoning...

The change touches the experiment runner in langfuse/_client/client.py, two pass-through wrappers (datasets.py, experiment.py) and the unit tests. It adds one optional keyword-only parameter, and the default code path is the same as before. It touches no auth, crypto or permission code. The only CODEOWNERS rule covers .github/, which this PR does not change. The change is small and self-contained, comes with dedicated tests, and the bug hunt found no issues.