Repository navigation
ci(release): release the experiment action with a major tag - #126
Merged
Merged
Conversation
A workflow that used the experiment action at main ran each change on the main branch at once. Release Please now gives the action releases such as experiment-action-v1.0.0. After each release, a new job moves experiment-action-v1 to the release commit, but never back to an earlier release. The READMEs tell users to use experiment-action-v1.
aikins01
added a commit
that referenced
this pull request
Oct 30, 2026
* feat(sdk): add runExperiment and score runExperiment gets a dataset version page by page and makes the experiment. It runs the task on each item, with concurrency and runs for each item. It sends the outputs and the local scores in batches of at most 100. Then it completes the experiment, waits for the server evaluators, and returns the summary. Each item run has its own trace. Each span of the run has td.experiment.id and td.dataset.item_id. In CI mode, runExperiment records the GitHub Actions run, and it rejects when the check is not successful. score sends a score on a trace, a span, or a session to /v1/scores. It never rejects. It sends its errors to onError. getPrompt and the new calls now use one function for the key, the base URL, and the Retry-After header. * fix(sdk): keep each min score and print scores as the python sdk does minScoresFromEnv now makes its object from pairs. Before, the name __proto__ set the prototype of the object, and its mark went away with no error. A number score now prints as Math.round(value * 100) / 100, a rule that the Python SDK can copy. The tests now also cover the experiment attributes on log records and the request order that the code holds. * feat(sdk-python): add run_experiment and score run_experiment and run_experiment_async are the Python twins of runExperiment, with the same options, defaults, errors, and requests. The sync call runs the items in threads, each in a copy of the context of the caller. The async call runs them as asyncio tasks. The two read the dataset page by page and post the outputs in batches of at most 100. score and score_async send a score to /v1/scores. They never raise and send their errors to on_error. The new _api module holds the key and base URL, one JSON request with its retry rules, the redirect guard, and the Retry-After parser. The prompt code now uses it too. The checks count lengths in UTF-16 units, as the API does. * feat(sdk): add the experiment action for pull requests actions/experiment is a composite action. It runs the eval script of the user, a .ts, .js, or .py file or a command, with the dataset and the min-scores inputs in the variables that the SDKs read. Then it writes the result to the job summary and to one comment on the pull request, and it fails the job when the script exits with an error. The inputs go to the scripts through env, never into a run line. The results file has mode 0600, and the comment escapes the names. A token that cannot write the comment gives a warning. CI now runs shellcheck on the action script. * docs(sdk): document experiments, scores, and the experiment action The READMEs of the two core SDKs get the Experiments and the Scores sections. They tell which data goes to telemetry.dev when output capture is off: the outputs, the task errors, the scorer reasons, and the metadata. The conformance document gets the experiment attributes and the wire contract of the two SDKs: the requests, the retry rules, the limits, the score rules, CI mode, and the variables of the action. * fix(sdk): complete the experiment after a run stops send() called fetch as a method of the Api object. Workers and browsers reject a fetch call that has a different `this` object. Thus score and runExperiment did not operate there without a `fetch` option. send() calls fetch as a function, not as a method. When a run stops after runExperiment makes the experiment, it sends the complete request before it rejects. Examples are an item post or a page fetch that is not successful, and more than 10,000 items. The server then completes the experiment with its item runs, or sets the status to failed when it has none. An error of that request goes to onError, and runExperiment rejects with the first error. The create request tries again only after a 429. The server sends a 503 when its database work is longer than its time limit. That 503 can come after the commit, and a retry then makes a second experiment. Each run of an item gets its own copy of the item. Thus a task that changes its input does not change the next run. The SDK cuts a pull request title without half of a surrogate pair, because Postgres rejects a lone surrogate in jsonb. A score field that can be null can also be missing, as in the Python SDK. A result without checkStatus stays an invalid_response. score() without options reports invalid_score and does not reject. New tests do checks of the 4 MiB batch limit, the 10,000-item limit, the 30-minute wait, and the retry after a timeout or a network error. * fix(sdk-python): complete the experiment after a run stops An output that JSON cannot hold stopped the full run outside the guard of its item. An example is a dict that holds itself. An exception with a __str__ that raises did the same. The SDK sends None as the output, as the TypeScript SDK does, and the name of the type as the error. A lone surrogate in an output or an error becomes U+FFFD. The request body is the JSON of JSON.stringify, which escapes a lone surrogate in a label. When a run stops after run_experiment makes the experiment, it sends the complete request before it raises. This also occurs after Ctrl-C and after a cancellation. An error of that request goes to on_error. An error of an item run stops all workers. The create request tries again only after a 429, as in the TypeScript SDK. Each run of an item gets a deep copy of the item. The requests for experiments, scores, and prompts send the User-Agent telemetry-dev-python/<version>, because Cloudflare rejects Python-urllib. New tests do checks of the 4 MiB batch limit, the 10,000-item limit, the 30-minute wait, and the retry after a timeout or a dropped connection. * fix(sdk): use the tsx of the project and change only our comment The action ran a .ts script with npx --yes tsx. npx then downloads the newest tsx from npm, in a step that has the API key. The action uses the tsx in node_modules/.bin of the working directory or of a directory above it. If it finds no tsx, it stops with a clear error. report.mjs used each comment that started with its marker, thus it tried to change a pasted copy. It changes only a comment of its token. For a user token, that is a comment of its login. The GITHUB_TOKEN and app tokens cannot read /user, and they write as a bot. The README tells that the key must be for all environments or for experiment. It tells that the capture settings do not apply to item outputs, and that a pull request from a fork gets no API key. The example does not run the job for those pull requests. * docs(sdk): tell how a stopped experiment run completes The conformance document gives the complete request after a run stops, and the create retry only after a 429. It also gives the copy of the item for each run and the outputs that JSON cannot hold. It gives the Python User-Agent, the fetch call as a function, the fields that a result must have, and the cut of the title. * docs(sdk): give api.telemetry.dev as the default base url The base-url input of the experiment action and the baseUrl comment in api.ts gave the old default, https://ingest.telemetry.dev. The default base URL of the SDKs changed to https://api.telemetry.dev in #109. * feat(sdk): record the prompt on model calls outside withPrompt compile() keeps each text that it makes, with the prompt. For a template with a variable, compile() also keeps the variables. It keeps a maximum of 1000 texts and 2,000,000 characters, and it removes the text that it kept first. It does not keep a text from a template that has only variable tags. promptAttributesFor (TypeScript) and prompt_attributes_for (Python) find a kept text in a request. They examine 6 levels and 500 values, and they do not call getters or properties. They give the td.prompt.* attributes of the first text from a template with a variable. If the request has no such text, they give the name, version, and label of the first text from a template with no variable. In a withPrompt or with_prompt scope, they give no attributes. They do not throw an error. The 11 provider integrations send the request of the app to the helper before the span starts. They do this also when input capture is off. The integrations are OpenAI chat and responses, Anthropic messages, Bedrock Converse and InvokeModel, Google GenAI generateContent, LiteLLM completion and responses, and OpenRouter chat and responses. Bedrock embedding models do not get the attributes. The spans for images, audio, video, and Realtime do not examine the request. Each integration reads the helper from the core module when it runs. Thus it can operate with a core that does not have the helper. As a result, the minimum Python core versions do not change. In Python, fork() waits for the lock of the kept texts. Thus a child process does not get a lock that a different thread holds. Refs #97 * fix(sdk): find the prompt by message identity, not by text The compile() function of a chat prompt tags each message object that it makes with the prompt and the variables of that call. TypeScript uses a non-enumerable symbol property, and Python uses a dict subclass with a slot. The keys and the JSON of a message do not change. The promptAttributesFor and prompt_attributes_for functions read the tag of the newest message of the last 1000, and they compare no text. Thus a message from a different task, rollout version, turn, or call does not get the wrong prompt, and a copy gets no prompt. We removed the text registry, its limits, its lock, and its fork hooks. As a result, compile() keeps no variables alive. The integrations give only the request messages to the core: OpenAI messages and input, Anthropic messages, Bedrock Converse messages, Google GenAI contents, LiteLLM messages and input, and OpenRouter messages and input. Bedrock InvokeModel sends a serialized body, thus it does not look for a prompt. Python Anthropic reads the list that the app gave, not its own copy. Its messages.stream() decides at __enter__, where its span starts. A span with input capture off removes td.prompt.variables from the attributes that the app gives. The SDKs do not record text prompts outside withPrompt and with_prompt. * fix(sdk): name the newest compile, not the last tagged message A compile tag holds a serial number that increases with each chat compile. promptAttributesFor and prompt_attributes_for give the tag with the highest number. Thus a message that a placeholder adds from an older compile of a different prompt does not win over the compile that made the list. They look at the first 1000 and the last 1000 messages. A template that ends with a placeholder has its own messages at the start. The TypeScript compile() keeps a copy of the top level of the variables, as Python does. Thus a later change to the object of the app does not change the variables that a message or a withPrompt span records. The Python compiled message keeps the constructor of dict. Thus type(message)(message) and dataclasses.asdict make a copy with all of the keys and without the tag. A model-call span with a parent context in withPrompt or with_prompt removes the td.prompt.* attributes that the app gives. Thus the span does not mix two prompts when the app calls promptAttributesFor outside the scope and gives an explicit parent. The READMEs and docs/sdk-conformance.md tell that the Python messages are a dict subclass, which yaml.safe_dump, marshal, and checks of the exact type reject. They also tell that a tagged message keeps the variables of its compile in memory. We corrected the docstring of prompt_attributes_for: with input capture on, it calls the mask of the app. Refs #97 * fix(sdk): keep the variables of recent compiles only A compiled message kept the variables of its compile, with the lists in them. Thus a chat that keeps a compiled message of each turn in a new history list for each turn kept the history list of every turn. Memory increased with the square of the turn count: 49 MB in TypeScript and 38 MB in Python after 2000 turns, also with input capture off. compile() can get a message of an older compile in a variable, or in a list in a variable. If the older compile is 100 or more compiles older, it discards its variables. The newest compile identifies the prompt of a request, thus a call still records the variables of its own compile. A recent compile keeps its variables when a newer compile gets its messages. Thus a judge compile that gets the messages before their call does not remove them. After 2000 turns, the SDKs keep 3.7 MB (TypeScript) and 4.4 MB (Python) with input capture on, and 0.7 MB and 1.3 MB with it off. These numbers include the history of the app. If input capture is off in the client, compile() keeps no variables in the tag, because promptAttributesFor and prompt_attributes_for cannot record them. withPrompt and with_prompt keep the variables of the last compile of each prompt object, as before. compile() also copies each list in the variables. Thus a change to the history of the app after compile() does not change the variables that a call records. compile() does not copy the objects in a list, and the READMEs and docs/sdk-conformance.md tell this. A span that gets its prompt from withPrompt or with_prompt discards the td.prompt.* attributes of the app at start, and also in update, end, updateActiveSpan, and update_current_span. Refs #97 * fix(sdk): give the base url to getPrompt in the prompt tests The tests of this branch gave apiUrl and api_url to getPrompt and get_prompt. The main branch removed these options in #101. Thus the tests give baseUrl, base_url, and newBaseUrl. Refs #97 * fix(sdk): keep the variables of the last two chat turns A compile removed the variables of a compile 100 or more compiles before it, when it got a message of that compile. A chat that cuts its history to less than approximately 100 turns never gave such a message. Each tag kept a copy of the window, and the messages in the window kept the tags of previous turns. Thus every turn kept its variables: 2000 of 2000 turns in TypeScript and Python. The count also included the compiles of other requests. Thus 99 other compiles before a guard compile, or a guard compile at each step of an agent loop, removed the variables of a request before its model call. With this change, when a compile C gets a message of a compile B in a variable, each compile A whose message B got removes its variables. B keeps its own. The rule does not count compiles, thus the compiles of other requests do not change it. A chain of tags with variables is at most two tags long. The repros of the review keep the variables of 2 of 2000 turns in the two SDKs. This is true for one list that the app cuts in place to 20 messages, for a new list of the last 20 or 198 messages, and for a history with no window. With no window, the SDK keeps two copies of the history list, not approximately 100. The tests do the same check for 300 turns, also for a tuple (Python) and for one message that is not in a list. A guard compile after 150 other compiles, and a guard compile at each of 150 agent steps, keep the variables of the request. In Python, _drop_history reads the variables of a tag one time, because a compile in another thread can remove them. The READMEs and docs/sdk-conformance.md tell the new rule. They again recommend copies for compiled messages that the app keeps but does not give to compile(): a copy keeps no variables. Refs #97 * fix(sdk): keep compiled variables for 5 minutes, not two turns With the last commit, a compile C removed the variables of a compile A when C got a message of a compile B, and B got a message of A. An inspection of this rule found four errors: - A chat that keeps the messages of a compile with no compiled messages in its variables kept the variables of each turn. Examples are a question prompt in a placeholder and a reply prompt for each answer. - Other code removed the variables of a shared message. A system message from startup had no variables after two turns of a chat that got it. - Two compiles before a call removed the variables of the request: a guard and then a judge of the guard, or turns of one session that add their question to the history before their calls. - The docs told the reader that a call with only messages of previous turns records no variables. But the call records the variables of the newest of them. A rule that looks at which compile got which message has one of these errors. A shared system message and a message of a chat history have the same shape. Thus a tag keeps its variables for 5 minutes after its compile. A call keeps them for 5 more minutes when the newest compile of the call is that compile. Other compiles do not change this time. After this time, a call records the prompt without td.prompt.variables. Each compile and each call removes the variables of the expired tags, also when the app keeps their messages. The SDKs find these tags by the minute of their time, through a weak reference. Thus the reference does not keep a tag in memory, and a clock change does not stop the minutes after it. In TypeScript, a runtime without WeakRef (Cloudflare Workers before 2025-05-05) keeps no variables in the tag. At one turn in 2 minutes, five chat shapes keep the variables of 3 of 300 turns in the two SDKs. A loop with no time between its compiles keeps all of their variables for 5 minutes. The shared system message, the judge of a guard, and the turns that share a history keep their variables. The tests do these checks, also with threads in Python. The READMEs, docs/sdk-conformance.md, and the docstrings tell the new rule. Refs #97 * fix(sdk): measure the variables time of compiled messages in elapsed time In #117, the prompt cache and the Retry-After wait moved to performance.now(), so that a change of the system clock does not move them. But a compile tag used Date.now() for the 5 minutes that it keeps its variables. Thus a clock that moved back an hour kept the variables of each compile for one more hour. A clock that moved forward removed the variables before 5 minutes. The Python tags use monotonic(). With this change, compile() and promptAttributesFor read performance.now(). The test of the 5 minutes moves the system clock back an hour, and the tag continues to remove the variables after 5 minutes. docs/sdk-conformance.md tells that the SDKs measure this time in elapsed time. Refs #97 * docs(sdk): tell that a call with no compiled message records its input In #117, the READMEs tell that a model call outside withPrompt and with_prompt records the compiled messages as its input when input capture is on. This branch replaced that sentence. Its text told that the SDK does not record a copy of a compiled message. The READMEs now tell that such a call gets no td.prompt.* attributes. With input capture on, the call records the messages as its input. Refs #97 * fix(cli): break the sign-in lock only when its holder has exited A lock now records its holder's pid and host and is broken on the same machine only once that process has exited. Every removal, a release or a break, claims a marker named after the lock's token first, so one command can't remove a lock another just took, and markers left by a crashed command are cleared the same way. Locks from CLI 0.1.0 or another machine keep the 3 second age rule. * docs(sdk): name the read prompts access of the prompt api key (#103) The TypeScript and Python READMEs said that the key of getPrompt and get_prompt "needs the read scope for prompts". The API keys page of telemetry.dev shows this scope as the Read prompts choice in the Access of a key. The READMEs and the apiKey doc comment now name Read prompts and the prompts:read scope. Refs telemetry-dev/telemetry.dev#512 * fix(sdk-python): fetch prompts again in a forked process (#105) * fix(sdk-python): reset prompt fetches in a forked process A process that os.fork() makes kept the pending fetch events of the parent, but not the fetch threads. In the child, a call with no copy waited 10 s and then got a timeout error. A kept copy never refreshed. A cache lock that a different thread held at the fork stayed locked in the child, and each get_prompt call in the child waited for all time. Fork hooks now hold the cache lock during fork(). In the child, the hooks remove the pending fetch events and release the lock. Thus a call that must fetch starts a new fetch. The child keeps the copies of the parent. Closes #98 * fix(sdk-python): make the prompt cache fork hooks safe A pending fetch keeps the id of the process whose thread runs the fetch. In a different process, a call ignores this fetch and starts a new fetch. Thus a fork without fork hooks also gets a new fetch, for example a fork in C code. The child hook does not reset the entries. The cache lock is reentrant. Thus a fork in the thread that holds the lock does not wait for that thread. The before hook records in a thread local if it got the lock. The after hooks release the lock only in that condition. The module registers the hooks one time in each process, also when importlib.reload runs the module again. The hooks find the lock when they run. The lock test examines the time of the fork and the errors of the holder thread. The fork must return after the holder releases the lock. * fix(sdk-python): replace a held prompt cache lock in a forked child If the before fork hook did not get the cache lock, a thread of the parent can hold the lock in the child. A signal can stop the wait of the before hook. uWSGI with py-call-osafterfork runs only the child hook. In these two conditions, a get_prompt call in the child waited for all time. The child hook replaces the lock in such a child. The code reads the lock from the module each time. The child hook removes the fetches of the parent again. Before this change, the child found these fetches only through the process ID. If the child got the ID of the process that started a fetch, the child waited for that fetch, and its calls got timeout errors. The ID check stays for a fork that runs no hooks. A call keeps its fetch event in a local variable. Thus a fork in the call, as in a signal handler, does not let the call continue with no event and no result. The capture server reads its delay before it records a request. Thus a test that finds the request and sets the delay to 0 does not cancel the wait of that request. The signal test stops its timer. Thus a failure of the test does not stop pytest with SIGALRM. in_fork finds the child through its process ID. Thus a fork function that forks two times does not send a child back into pytest. * fix(sdk-python): use base_url in the prompt fork tests The version_of helper of the fork tests gave the api_url option to get_prompt. #101 replaced api_url with base_url. Thus pyright found an error after the rebase on #101. * fix(sdk-python): correct the prompt fork claims and fork in the test The README and docs/sdk-conformance.md said that a forked process does not wait for a fetch that did not end before the fork. That is not true for one case. If a signal handler forks while a call waits for a fetch, the call continues in the child. No thread of the child completes that fetch. Thus the call waits until 10 seconds after the start of the fetch, and then fails with the timeout code. The two documents now name this case, and the code does not change. The comment at the process ID check in _load, and the message of the commit "replace a held prompt cache lock in a forked child", said that the check handles a fork that runs no hooks. The check removes only the copied fetch. Such a fork can also copy a lock that a thread of the parent holds. Then a call in the child waits for the lock with no limit. The comment now says this, and the comment at the hook registration now says that only a fork that runs the hooks waits for the lock. The signal row of the owner test of the lock replacement called the fork hooks directly. The row now forks with os.fork() through in_fork. A different thread holds the cache lock, and SIGALRM stops the before hook. The child must get the prompt. Thus the row needs os.fork() to run the after hooks when a before hook raises. The row fails when the child hook does not replace the lock. It also fails when an after hook releases a lock that the before hook did not get. If no signal stops the before hook, the holder thread fails the row after 5 s. The fork function of the row stops its timer also when the fork raises. If the child loses the thread local _forking, the child hook replaces the lock and does not release it. The child gets a free lock in the two cases, thus no test can see a difference. * chore: release main (#125) * feat(cli): report errors as one JSON line with a stable code under --json (#127) With --json, a failure writes one JSON line to stderr with a code from a documented list, a hint, retryable, and the exit code, and nothing to stdout. Color and the spinner are off under --json, and arguments after -- are not options. Network errors are found anywhere in the cause chain, so a refused connection says telemetry.dev couldn't be reached and a connect timeout is a retryable timeout. * feat(cli): summarize a trace's failures and repeated calls, and suggest next commands (#128) trace lists the failed spans where failures began, with their path, and the operations that ran more than once with their wall time counted once for overlapping calls. traces, sessions, and trace suggest the likely next command on stderr in human output, keeping the project and passing option values as --name=value. * test(cli): give each cli test 30 seconds, since each one starts the cli as a process (#130) A test that starts the CLI seven times took 5,031 ms on a CI runner, just past vitest's 5 second default, and failed main after #128. * ci(release): release the experiment action with a major tag (#126) A workflow that used the experiment action at main ran each change on the main branch at once. Release Please now gives the action releases such as experiment-action-v1.0.0. After each release, a new job moves experiment-action-v1 to the release commit, but never back to an earlier release. The READMEs tell users to use experiment-action-v1. * chore: ignore the experiment action changelog in the formatter (#132) Release Please writes actions/experiment/CHANGELOG.md in its own format, and the check of the release PR stopped on that file. The formatter already ignores the changelogs of the npm packages for the same reason. * chore: release main (#129) Co-authored-by: Ephraim Duncan <55143799+ephraimduncan@users.noreply.github.com> * feat(cli): add --json to login and logout (#131) login --json prints the sign-in link and returns without opening a browser or waiting; login --finish --json prints the sign-in. Sign-in network errors now report unreachable or timeout, and a failed project lookup after sign-in no longer fails login. * feat(cli): map telemetry.dev's tool error codes to cli error codes (#134) A tool error's code in _meta now gives project_not_found, access_denied, payloads_not_allowed, invalid_cursor, trace_not_found, span_not_found, or server_error, with their own exit codes and hints. Errors without a known code stay tool_error. * chore: release main (#133) --------- Co-authored-by: ephraimduncan <ephraimduncan68@gmail.com> Co-authored-by: Ephraim Duncan <55143799+ephraimduncan@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Workflows that use the experiment action at
@mainrun each change to the action as soon as it merges. This PR gives the action versioned releases and a major tag,experiment-action-v1, which workflows can use in place of@main.Release Please now holds
actions/experimentas asimplepackage, with the componentexperiment-actionand the initial version 1.0.0. The action has no package to publish, so the discover step ignores it. A newtag-experiment-actionjob runs after Release Please. When the push is the commit of anexperiment-action-vX.Y.Zrelease, the job pointsexperiment-action-vXat that commit through the Git refs API.The job moves a major tag only forward on
main. Thus a rerun of an earlier release leaves the tag at the later release. Afeat!:orfix!:commit on the action startsexperiment-action-v2, andexperiment-action-v1stays at the last 1.x release. The job finds the release by its tag, not by the outputs of Release Please, so a rerun after Release Please failed partway still moves the tag.The job uses
GITHUB_TOKENwithcontents: write, and GitHub does not give this token theworkflowspermission. Reports from other repositories show that GitHub then rejects a tag on a commit when no branch head has the same.github/workflowsfiles. The job runs on the release commit, which is normally the head ofmain, so the rule does not stop the usual release.RELEASING.mdgives the command to move the tag by hand if a workflow change merges before the job runs.Merge order
After this PR merges, Release Please opens a release PR with
experiment-action1.0.0, becausefeat(sdk): add the experiment action for pull requestsis afterbootstrap-sha. The tagexperiment-action-v1exists only after that release PR merges. Until then, the README example names a tag that does not exist, so merge the release PR soon after this one. https://github.com/telemetry-dev/telemetry.dev/pull/570 changes the step on the experiment page of telemetry.dev, and it must merge after the tag exists.I ran
bash .github/scripts/release_test.shwith bash 5.2 in an Ubuntu 24.04 container,shellcheck .github/scripts/*.sh actions/experiment/*.sh,actionlint, andvp check. The release tests now cover a push with no release of the action, the first release, a patch release, a rerun of an earlier release, and a new major version.