Repository navigation
feat(sdk): record the prompt of compiled messages outside withPrompt - #112
Merged
Merged
Conversation
ephraimduncan
force-pushed
the
feat/sdk/prompt-outside-scope
branch
from
October 30, 2026 00:26
0b16f09 to
1fcedd7
Compare
compile() keeps each text that it makes, with the prompt. For a template with a variable, compile() also keeps the variables. It keeps a maximum of 1000 texts and 2,000,000 characters, and it removes the text that it kept first. It does not keep a text from a template that has only variable tags. promptAttributesFor (TypeScript) and prompt_attributes_for (Python) find a kept text in a request. They examine 6 levels and 500 values, and they do not call getters or properties. They give the td.prompt.* attributes of the first text from a template with a variable. If the request has no such text, they give the name, version, and label of the first text from a template with no variable. In a withPrompt or with_prompt scope, they give no attributes. They do not throw an error. The 11 provider integrations send the request of the app to the helper before the span starts. They do this also when input capture is off. The integrations are OpenAI chat and responses, Anthropic messages, Bedrock Converse and InvokeModel, Google GenAI generateContent, LiteLLM completion and responses, and OpenRouter chat and responses. Bedrock embedding models do not get the attributes. The spans for images, audio, video, and Realtime do not examine the request. Each integration reads the helper from the core module when it runs. Thus it can operate with a core that does not have the helper. As a result, the minimum Python core versions do not change. In Python, fork() waits for the lock of the kept texts. Thus a child process does not get a lock that a different thread holds. Refs #97
The compile() function of a chat prompt tags each message object that it makes with the prompt and the variables of that call. TypeScript uses a non-enumerable symbol property, and Python uses a dict subclass with a slot. The keys and the JSON of a message do not change. The promptAttributesFor and prompt_attributes_for functions read the tag of the newest message of the last 1000, and they compare no text. Thus a message from a different task, rollout version, turn, or call does not get the wrong prompt, and a copy gets no prompt. We removed the text registry, its limits, its lock, and its fork hooks. As a result, compile() keeps no variables alive. The integrations give only the request messages to the core: OpenAI messages and input, Anthropic messages, Bedrock Converse messages, Google GenAI contents, LiteLLM messages and input, and OpenRouter messages and input. Bedrock InvokeModel sends a serialized body, thus it does not look for a prompt. Python Anthropic reads the list that the app gave, not its own copy. Its messages.stream() decides at __enter__, where its span starts. A span with input capture off removes td.prompt.variables from the attributes that the app gives. The SDKs do not record text prompts outside withPrompt and with_prompt.
A compile tag holds a serial number that increases with each chat compile. promptAttributesFor and prompt_attributes_for give the tag with the highest number. Thus a message that a placeholder adds from an older compile of a different prompt does not win over the compile that made the list. They look at the first 1000 and the last 1000 messages. A template that ends with a placeholder has its own messages at the start. The TypeScript compile() keeps a copy of the top level of the variables, as Python does. Thus a later change to the object of the app does not change the variables that a message or a withPrompt span records. The Python compiled message keeps the constructor of dict. Thus type(message)(message) and dataclasses.asdict make a copy with all of the keys and without the tag. A model-call span with a parent context in withPrompt or with_prompt removes the td.prompt.* attributes that the app gives. Thus the span does not mix two prompts when the app calls promptAttributesFor outside the scope and gives an explicit parent. The READMEs and docs/sdk-conformance.md tell that the Python messages are a dict subclass, which yaml.safe_dump, marshal, and checks of the exact type reject. They also tell that a tagged message keeps the variables of its compile in memory. We corrected the docstring of prompt_attributes_for: with input capture on, it calls the mask of the app. Refs #97
A compiled message kept the variables of its compile, with the lists in them. Thus a chat that keeps a compiled message of each turn in a new history list for each turn kept the history list of every turn. Memory increased with the square of the turn count: 49 MB in TypeScript and 38 MB in Python after 2000 turns, also with input capture off. compile() can get a message of an older compile in a variable, or in a list in a variable. If the older compile is 100 or more compiles older, it discards its variables. The newest compile identifies the prompt of a request, thus a call still records the variables of its own compile. A recent compile keeps its variables when a newer compile gets its messages. Thus a judge compile that gets the messages before their call does not remove them. After 2000 turns, the SDKs keep 3.7 MB (TypeScript) and 4.4 MB (Python) with input capture on, and 0.7 MB and 1.3 MB with it off. These numbers include the history of the app. If input capture is off in the client, compile() keeps no variables in the tag, because promptAttributesFor and prompt_attributes_for cannot record them. withPrompt and with_prompt keep the variables of the last compile of each prompt object, as before. compile() also copies each list in the variables. Thus a change to the history of the app after compile() does not change the variables that a call records. compile() does not copy the objects in a list, and the READMEs and docs/sdk-conformance.md tell this. A span that gets its prompt from withPrompt or with_prompt discards the td.prompt.* attributes of the app at start, and also in update, end, updateActiveSpan, and update_current_span. Refs #97
A compile removed the variables of a compile 100 or more compiles before it, when it got a message of that compile. A chat that cuts its history to less than approximately 100 turns never gave such a message. Each tag kept a copy of the window, and the messages in the window kept the tags of previous turns. Thus every turn kept its variables: 2000 of 2000 turns in TypeScript and Python. The count also included the compiles of other requests. Thus 99 other compiles before a guard compile, or a guard compile at each step of an agent loop, removed the variables of a request before its model call. With this change, when a compile C gets a message of a compile B in a variable, each compile A whose message B got removes its variables. B keeps its own. The rule does not count compiles, thus the compiles of other requests do not change it. A chain of tags with variables is at most two tags long. The repros of the review keep the variables of 2 of 2000 turns in the two SDKs. This is true for one list that the app cuts in place to 20 messages, for a new list of the last 20 or 198 messages, and for a history with no window. With no window, the SDK keeps two copies of the history list, not approximately 100. The tests do the same check for 300 turns, also for a tuple (Python) and for one message that is not in a list. A guard compile after 150 other compiles, and a guard compile at each of 150 agent steps, keep the variables of the request. In Python, _drop_history reads the variables of a tag one time, because a compile in another thread can remove them. The READMEs and docs/sdk-conformance.md tell the new rule. They again recommend copies for compiled messages that the app keeps but does not give to compile(): a copy keeps no variables. Refs #97
With the last commit, a compile C removed the variables of a compile A when C got a message of a compile B, and B got a message of A. An inspection of this rule found four errors: - A chat that keeps the messages of a compile with no compiled messages in its variables kept the variables of each turn. Examples are a question prompt in a placeholder and a reply prompt for each answer. - Other code removed the variables of a shared message. A system message from startup had no variables after two turns of a chat that got it. - Two compiles before a call removed the variables of the request: a guard and then a judge of the guard, or turns of one session that add their question to the history before their calls. - The docs told the reader that a call with only messages of previous turns records no variables. But the call records the variables of the newest of them. A rule that looks at which compile got which message has one of these errors. A shared system message and a message of a chat history have the same shape. Thus a tag keeps its variables for 5 minutes after its compile. A call keeps them for 5 more minutes when the newest compile of the call is that compile. Other compiles do not change this time. After this time, a call records the prompt without td.prompt.variables. Each compile and each call removes the variables of the expired tags, also when the app keeps their messages. The SDKs find these tags by the minute of their time, through a weak reference. Thus the reference does not keep a tag in memory, and a clock change does not stop the minutes after it. In TypeScript, a runtime without WeakRef (Cloudflare Workers before 2025-05-05) keeps no variables in the tag. At one turn in 2 minutes, five chat shapes keep the variables of 3 of 300 turns in the two SDKs. A loop with no time between its compiles keeps all of their variables for 5 minutes. The shared system message, the judge of a guard, and the turns that share a history keep their variables. The tests do these checks, also with threads in Python. The READMEs, docs/sdk-conformance.md, and the docstrings tell the new rule. Refs #97
…time In #117, the prompt cache and the Retry-After wait moved to performance.now(), so that a change of the system clock does not move them. But a compile tag used Date.now() for the 5 minutes that it keeps its variables. Thus a clock that moved back an hour kept the variables of each compile for one more hour. A clock that moved forward removed the variables before 5 minutes. The Python tags use monotonic(). With this change, compile() and promptAttributesFor read performance.now(). The test of the 5 minutes moves the system clock back an hour, and the tag continues to remove the variables after 5 minutes. docs/sdk-conformance.md tells that the SDKs measure this time in elapsed time. Refs #97
In #117, the READMEs tell that a model call outside withPrompt and with_prompt records the compiled messages as its input when input capture is on. This branch replaced that sentence. Its text told that the SDK does not record a copy of a compiled message. The READMEs now tell that such a call gets no td.prompt.* attributes. With input capture on, the call records the messages as its input. Refs #97
ephraimduncan
force-pushed
the
feat/sdk/prompt-outside-scope
branch
from
October 30, 2026 01:18
1fcedd7 to
0c2f2b6
Compare
…de-scope # Conflicts: # docs/sdk-conformance.md # sdks/python/src/telemetry_dev/_prompts.py # sdks/python/tests/test_prompts.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A model call outside
withPrompt/with_promptnow records the prompt of the compiled messages that it sends. Before this change, a span recordedtd.prompt.*only when the call ran in a scope. Thus an app that gave the messages ofcompile()directly to its provider client showed no prompt on its model calls.Fixes #97
compile()of a chat prompt tags each message object that it makes with the prompt, a copy of its variables, and a serial number. In TypeScript the tag is a non-enumerable symbol property. In Python each message is an instance of a privatedictsubclass with the tag in a slot. In the two SDKs, the keys and the JSON of a message do not change. Before a provider integration starts its span, it gives the request messages topromptAttributesFor/prompt_attributes_for, also when input capture is off. The newest compile among the messages wins, and awithPrompt/with_promptscope wins over the tags.The integrations that do this are OpenAI chat completions and responses, Anthropic messages, Bedrock Converse, Google GenAI
generateContent, OpenRouter chat and responses, and LiteLLM completion and responses (Python). Bedrock InvokeModel sends a serialized body, so it records a prompt only in a scope. An integration reads the function from the core when it runs, so a new integration with an older core records the prompt only in a scope.What has no tag
A copy of a message has no tag: a spread,
structuredClone, or a JSON round trip in TypeScript, anddict(),copy.deepcopy, pickle, or a JSON round trip in Python. A new message with the same text has no tag either, so a message of a different call or prompt version never gets the prompt attributes. With input capture on, a call still records such messages as its input. A text prompt compiles to a string, which cannot carry a tag. For these cases,withPrompt/with_promptcontinues to record the prompt.In Python, a compiled message is a
dict, but not of the exact type. Thustype(message) is dictis false, andyaml.safe_dumpandmarshalreject a compiled message. An app that does this must copy the messages withdict(message)first.Variables
With input capture on when
compile()runs, a tag keeps the variables of its compile for 5 minutes of elapsed time, so a change of the system clock does not move this time. A call keeps them for 5 more minutes when the newest compile of the call is that compile. Other compiles do not change this time. After it, a call records the prompt withouttd.prompt.variables, and the next compile or call removes the variables from the tag, also when the app keeps the message. Thus a chat history keeps the variables of the compiles of its last 5 minutes, for a history of any length or depth. A call with only messages of earlier compiles records the variables of the newest of them while they are less than 5 minutes old. With input capture off, or in a TypeScript runtime withoutWeakRef(Cloudflare Workers before 2025-05-05), the tag keeps no variables.Tests:
packages/sdk/tests/prompts.test.ts("promptAttributesFor names the compile of a message object; a copy or a string names nothing", "an unrelated message with the text of a compiled message is not tagged", "two tasks with one text and different other variables each record their own", "a rollout with two versions of the same messages records the version of each call", "the newest compiled message wins, also after 170 messages of history", "a message that a placeholder brings from an older compile does not win", "promptAttributesFor looks at the first and the last 1000 messages", "compile records its variables, not later changes of the object or of its lists", "the variables of a compile are freed with its messages", "when %s, compiled messages keep no variables", "a chat history in %s keeps the variables of the last 5 minutes", "a request keeps its variables until its call: %s", "a compile keeps its variables for 5 minutes after it or after the last call that names it", "withPrompt wins over a compiled message of another prompt, through a parent and in updates", "td.prompt.variables follows the input capture of the client and of the span", "promptAttributesFor reports an error and gives undefined"), the "outside withPrompt records the prompt" tables ofpackages/anthropic/tests/anthropic.test.ts,packages/bedrock/tests/bedrock.test.ts,packages/google-genai/tests/google-genai.test.ts,packages/openai/tests/openai.test.ts, andpackages/openrouter/tests/openrouter.test.ts,sdks/python/tests/test_prompts.py(the Python twins of the core tests above, "test_a_compiled_message_rebuilds_like_a_dict_and_the_rebuilt_one_has_no_tag", and "test_prompt_attributes_for_runs_no_code_of_the_app"), and the "outside with_prompt" tests of the Python packages for Anthropic (with "messages_stream takes the prompt from the context where its span starts", sync and async), Bedrock, Google GenAI, LiteLLM, OpenAI, and OpenRouter.Follow-ups: #114.