From 339eb0c01c5754191eb7d1b5ec37db21ebd28855 Mon Sep 17 00:00:00 2001 From: ephraimduncan Date: Thu, 29 Oct 2026 14:18:40 +0000 Subject: [PATCH 1/9] feat(sdk): record the prompt on model calls outside withPrompt compile() keeps each text that it makes, with the prompt. For a template with a variable, compile() also keeps the variables. It keeps a maximum of 1000 texts and 2,000,000 characters, and it removes the text that it kept first. It does not keep a text from a template that has only variable tags. promptAttributesFor (TypeScript) and prompt_attributes_for (Python) find a kept text in a request. They examine 6 levels and 500 values, and they do not call getters or properties. They give the td.prompt.* attributes of the first text from a template with a variable. If the request has no such text, they give the name, version, and label of the first text from a template with no variable. In a withPrompt or with_prompt scope, they give no attributes. They do not throw an error. The 11 provider integrations send the request of the app to the helper before the span starts. They do this also when input capture is off. The integrations are OpenAI chat and responses, Anthropic messages, Bedrock Converse and InvokeModel, Google GenAI generateContent, LiteLLM completion and responses, and OpenRouter chat and responses. Bedrock embedding models do not get the attributes. The spans for images, audio, video, and Realtime do not examine the request. Each integration reads the helper from the core module when it runs. Thus it can operate with a core that does not have the helper. As a result, the minimum Python core versions do not change. In Python, fork() waits for the lock of the kept texts. Thus a child process does not get a lock that a different thread holds. Refs #97 --- docs/sdk-conformance.md | 28 +- packages/anthropic/src/index.ts | 16 +- packages/anthropic/tests/anthropic.test.ts | 74 ++++- packages/bedrock/src/converse.ts | 2 + packages/bedrock/src/internal.ts | 10 + packages/bedrock/src/invoke-model.ts | 5 +- packages/bedrock/tests/bedrock.test.ts | 111 ++++++- packages/google-genai/src/index.ts | 16 +- .../google-genai/tests/google-genai.test.ts | 81 ++++- packages/openai/src/index.ts | 20 +- packages/openai/tests/openai.test.ts | 112 ++++++- packages/openrouter/src/index.ts | 13 + packages/openrouter/tests/openrouter.test.ts | 93 ++++++ packages/sdk/README.md | 14 +- packages/sdk/src/index.ts | 1 + packages/sdk/src/prompts.ts | 152 +++++++++- packages/sdk/tests/prompts.test.ts | 228 ++++++++++++++ .../src/telemetry_dev_anthropic/__init__.py | 12 +- sdks/python-anthropic/tests/test_anthropic.py | 50 ++++ .../src/telemetry_dev_bedrock/_fields.py | 9 +- .../telemetry_dev_bedrock/_invoke_model.py | 4 +- .../src/telemetry_dev_bedrock/_registry.py | 10 +- sdks/python-bedrock/tests/test_bedrock.py | 87 ++++++ .../telemetry_dev_google_genai/__init__.py | 8 +- .../tests/test_google_genai.py | 56 ++++ .../src/telemetry_dev_litellm/__init__.py | 12 +- sdks/python-litellm/tests/test_litellm.py | 64 ++++ .../src/telemetry_dev_openai/__init__.py | 16 +- sdks/python-openai/tests/test_openai.py | 66 ++++ .../src/telemetry_dev_openrouter/__init__.py | 9 + .../tests/test_openrouter.py | 67 +++++ sdks/python/README.md | 18 +- sdks/python/src/telemetry_dev/__init__.py | 2 + sdks/python/src/telemetry_dev/_prompts.py | 192 +++++++++++- sdks/python/tests/test_prompts.py | 283 +++++++++++++++++- 35 files changed, 1898 insertions(+), 43 deletions(-) diff --git a/docs/sdk-conformance.md b/docs/sdk-conformance.md index 09583dc..efe4a7a 100644 --- a/docs/sdk-conformance.md +++ b/docs/sdk-conformance.md @@ -41,9 +41,10 @@ repository, not this SDK workspace. Paths below refer to that product repository | error | span status ERROR + `error.type` + `exception` event (`exception.type/message/stacktrace`, `log.severity_number: 17`) | | | `log()` | OTLP log record to `/v1/logs` (always; never span events) | severity debug=5/info=9/warn=13/error=17; trace-correlated via active context | | `withPrompt(prompt, fn)` / `with with_prompt(prompt):` → `prompt.name` | `td.prompt.name` | on model-call spans that start in scope: operation `chat`, `text_completion`, or `generate_content` | +| outside a scope: a text that `prompt.compile()` made, in the request of a provider integration | `td.prompt.name`, `td.prompt.version`, `td.prompt.label` | on the `chat` spans of the provider integrations listed in [Prompts](#prompts); the scope wins | | `prompt.version` | `td.prompt.version` (int) | omitted for a fallback prompt (`version` null / None) | | `prompt.label` | `td.prompt.label` | only for a fetch by label; a fetch without label or version is the `production` label | -| variables of the last `prompt.compile()` before `withPrompt` / `with_prompt` starts, or in its scope | `td.prompt.variables` (JSON object) | mask → JSON stringify → truncate; gated by the effective captureInput of the span | +| variables of the last `prompt.compile()` before `withPrompt` / `with_prompt` starts, or in its scope; outside a scope, of the `compile()` that made the text, when its template has a variable | `td.prompt.variables` (JSON object) | mask → JSON stringify → truncate; gated by the effective captureInput of the span | `SpanHandle.serializeContent(value, key)` / `SpanHandle.serialize_content(value, key)` returns what `input` / `output` would emit for `value` under `key`: mask → JSON stringify → truncate. It @@ -178,6 +179,31 @@ cannot tell a concurrent task from an awaited helper, so in TypeScript all spans its last `compile()`; concurrent tasks that compile their own variables call `withPrompt` in each task. +Outside a scope, the SDKs find the prompt from the text of the request: + +- `compile()` keeps each text that it makes (each message of a chat prompt, the text of a text + prompt), with the prompt and the variables. It keeps at most 1000 texts and 2,000,000 + characters, and drops the oldest first. A text that `compile()` makes again becomes the newest. + It does not keep a text whose template has only variable tags, for example `{{ question }}`: + that text is input of the app, and an unrelated request can hold it too. +- `promptAttributesFor(request)` / `prompt_attributes_for(request)` look for a kept text in + `request`: strings in arrays and plain objects (Python: `list`, `tuple`, and `dict` but not their + subclasses, and the fields that an object stores in `__dict__` or `__pydantic_extra__`), 6 + levels deep and 500 values in all. They run no getter or property. A string matches only when it is equal to a kept text. +- The first text from a template with a variable gives the name, version, label, and variables + of its `compile()`. Without one, the first text from a template with no variable gives the + name, version, and label, but no variables: the same static text comes from each `compile()`. +- In a `withPrompt` / `with_prompt` scope they give nothing, thus the span gets the prompt of the + scope. They never throw. +- The provider integrations call them with the request of the app before the span starts, also + when input capture is off: OpenAI chat completions and responses, Anthropic messages, Amazon + Bedrock Converse and InvokeModel (not embedding models), Google GenAI `generateContent`, + LiteLLM completion and responses (Python), and OpenRouter chat and responses. The other spans + of the integrations (images, audio, video, and OpenAI Realtime) do not look in the request. +- A provider integration reads the function from the core module when it runs, thus it also + operates with an older core that does not have it. With such a core, the prompt is recorded + only in a scope. + ## The shared scenario Both e2e runners emit exactly this (under diff --git a/packages/anthropic/src/index.ts b/packages/anthropic/src/index.ts index d9cb1f6..4e5b057 100644 --- a/packages/anthropic/src/index.ts +++ b/packages/anthropic/src/index.ts @@ -5,6 +5,7 @@ import { type SpanHandle, type StartSpanOptions, } from "@telemetry-dev/sdk"; +import * as telemetrySdk from "@telemetry-dev/sdk"; import { Stream } from "@anthropic-ai/sdk/core/streaming"; import { Messages as BetaMessages } from "@anthropic-ai/sdk/resources/beta/messages"; import { Messages } from "@anthropic-ai/sdk/resources/messages"; @@ -147,6 +148,16 @@ function toolDefinitions(tools: T): Record | undefined { : undefined; } +// Cores older than promptAttributesFor record a prompt only inside withPrompt. A namespace read +// keeps them loadable without the export. +function promptAttributes(request: unknown): SpanFields["attributes"] { + const read = ( + telemetrySdk as { promptAttributesFor?: (request: unknown) => SpanFields["attributes"] } + ).promptAttributesFor; + + return typeof read === "function" ? read(request) : undefined; +} + function messagesRequest(body: ValueRecord): RequestMapping { const model = readString(body.model); @@ -161,7 +172,10 @@ function messagesRequest(body: ValueRecord): RequestMapping { type: "generation", model, input, - attributes: toolDefinitions(body.tools), + attributes: { + ...toolDefinitions(body.tools), + ...promptAttributes([body.system, body.messages]), + }, systemInstructions: body.system, temperature: readNumber(body.temperature), topP: readNumber(body.top_p), diff --git a/packages/anthropic/tests/anthropic.test.ts b/packages/anthropic/tests/anthropic.test.ts index e53d040..c8cab08 100644 --- a/packages/anthropic/tests/anthropic.test.ts +++ b/packages/anthropic/tests/anthropic.test.ts @@ -4,7 +4,14 @@ import { type PushMetricExporter, } from "@opentelemetry/sdk-metrics"; import { InMemorySpanExporter, type ReadableSpan } from "@opentelemetry/sdk-trace-base"; -import { flush, init, shutdown, type TelemetryOptions } from "@telemetry-dev/sdk"; +import { + type ChatMessage, + flush, + getPrompt, + init, + shutdown, + type TelemetryOptions, +} from "@telemetry-dev/sdk"; import * as telemetrySdk from "@telemetry-dev/sdk"; import Anthropic from "@anthropic-ai/sdk"; import { createRequire } from "node:module"; @@ -274,6 +281,71 @@ afterEach(async () => { vi.restoreAllMocks(); }); +test.each([ + { field: "system", olderCore: false }, + { field: "messages", olderCore: false }, + { field: "messages", olderCore: true }, +])( + "messages.create outside withPrompt records the prompt of a compiled text in $field, older core $olderCore", + async ({ field, olderCore }) => { + if (olderCore) { + vi.spyOn( + telemetrySdk as { promptAttributesFor: unknown }, + "promptAttributesFor", + "get", + ).mockReturnValue(undefined); + } + + const prompt = await getPrompt("support/reply", { + apiKey: "td_live_a", + apiUrl: "https://prompts.anthropic.test", + fetch: async () => + Response.json({ + name: "support/reply", + type: "chat", + label: "production", + version: 12, + messages: [ + { role: "system", content: "You answer for the {{team}} desk." }, + { role: "user", content: "Where is order {{order}}?" }, + ], + text: null, + model: { provider: "anthropic", name: "claude-sonnet-4-6" }, + parameters: {}, + tools: [], + responseFormat: { type: "text" }, + rollout: null, + }), + }); + const [system, user] = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const spans = setupSpans(); + const client = clientWith(createFakeFetch(jsonResponse(messagePayload())).fetch); + + await client.messages.create({ + model: "claude-sonnet-4-6", + max_tokens: 64, + system: field === "system" ? system!.content : "Be terse.", + messages: [{ role: "user", content: field === "messages" ? user!.content : "Hi" }], + }); + + const { attributes } = await exportedSpan(spans); + expect(attributes["gen_ai.input.messages"]).toBeDefined(); + + if (olderCore) { + expect(attributes["td.prompt.name"]).toBeUndefined(); + + return; + } + + expect(attributes).toMatchObject({ + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + "td.prompt.variables": '{"team":"billing","order":"#4"}', + }); + }, +); + test("messages.create maps request, response, usage, finish reason, provider, and sampling attributes", async () => { const spans = setupSpans(); const messages = [{ role: "user" as const, content: "Say hello" }]; diff --git a/packages/bedrock/src/converse.ts b/packages/bedrock/src/converse.ts index 074d1b1..e552c6d 100644 --- a/packages/bedrock/src/converse.ts +++ b/packages/bedrock/src/converse.ts @@ -9,6 +9,7 @@ import { numberValue, omitUndefined, parseJson, + promptAttributes, PROVIDER, stringValue, StreamRetention, @@ -34,6 +35,7 @@ export function converseRequestFields(input: JsonRecord): SpanFields { model: stringValue(input.modelId), input: normalizeMessages(input.messages), systemInstructions: normalizeSystem(input.system), + attributes: promptAttributes([input.system, input.messages]), temperature: numberValue(inference.temperature), topP: numberValue(inference.topP), maxTokens: numberValue(inference.maxTokens), diff --git a/packages/bedrock/src/internal.ts b/packages/bedrock/src/internal.ts index 104f595..7d82cc8 100644 --- a/packages/bedrock/src/internal.ts +++ b/packages/bedrock/src/internal.ts @@ -192,6 +192,16 @@ export function captureMaskedBeforeStart(): boolean { } } +// Cores older than promptAttributesFor record a prompt only inside withPrompt. A namespace read +// keeps them loadable without the export. +export function promptAttributes(request: unknown): SpanFields["attributes"] { + const read = ( + telemetrySdk as { promptAttributesFor?: (request: unknown) => SpanFields["attributes"] } + ).promptAttributesFor; + + return typeof read === "function" ? read(request) : undefined; +} + // The decision is made before the span starts so samplers see the same attributes as before. function startRequestSpan(name: string, options: StartSpanOptions): SpanHandle { const withheld = options.input !== undefined && isCut(options) && captureMaskedBeforeStart(); diff --git a/packages/bedrock/src/invoke-model.ts b/packages/bedrock/src/invoke-model.ts index d1e1f11..d2532d2 100644 --- a/packages/bedrock/src/invoke-model.ts +++ b/packages/bedrock/src/invoke-model.ts @@ -9,6 +9,7 @@ import { numberValue, omitUndefined, parseJson, + promptAttributes, PROVIDER, STREAM_CAPTURE_MAX_ITEMS, stringValue, @@ -35,12 +36,14 @@ export function invokeModelSpanName(input: JsonRecord): string { export function invokeModelRequestFields(input: JsonRecord): SpanFields { const model = stringValue(input.modelId); const body = parseBody(input.body, input.contentType); + const embedding = isEmbeddingModel(model); return omitUndefined({ provider: PROVIDER, model, input: body, - outputType: isEmbeddingModel(model) ? "embedding" : undefined, + outputType: embedding ? "embedding" : undefined, + attributes: embedding ? undefined : promptAttributes(body), ...nativeSamplingFields(model, body), }); } diff --git a/packages/bedrock/tests/bedrock.test.ts b/packages/bedrock/tests/bedrock.test.ts index 8e94b8f..d12bf19 100644 --- a/packages/bedrock/tests/bedrock.test.ts +++ b/packages/bedrock/tests/bedrock.test.ts @@ -9,7 +9,8 @@ import { } from "@aws-sdk/client-bedrock-runtime"; import { afterEach, expect, test, vi } from "vitest"; -import type { MaskFn } from "@telemetry-dev/sdk"; +import { type ChatMessage, getPrompt, type MaskFn } from "@telemetry-dev/sdk"; +import * as telemetrySdk from "@telemetry-dev/sdk"; import { instrumentBedrock, uninstrumentBedrock, wrapBedrock } from "../src/index.ts"; import { @@ -423,6 +424,114 @@ test("ConverseStream records stream errors with partial output", async () => { expect(span.attributes["telemetry.dev.capture.truncated"]).toBe(true); }); +test.each([false, true])( + "Converse and InvokeModel outside withPrompt record the prompt of a compiled text; embeddings do not; older core %s", + async (olderCore) => { + const spans = setup(); + if (olderCore) { + vi.spyOn( + telemetrySdk as { promptAttributesFor: unknown }, + "promptAttributesFor", + "get", + ).mockReturnValue(undefined); + } + const prompt = await getPrompt("support/reply", { + apiKey: "td_live_a", + apiUrl: "https://prompts.bedrock.test", + fetch: async () => + Response.json({ + name: "support/reply", + type: "chat", + label: "production", + version: 12, + messages: [ + { role: "system", content: "You answer for the {{team}} desk." }, + { role: "user", content: "Where is order {{order}}?" }, + ], + text: null, + model: { provider: "amazon-bedrock", name: "anthropic.claude-3-haiku" }, + parameters: {}, + tools: [], + responseFormat: { type: "text" }, + rollout: null, + }), + }); + const [system, user] = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const converseReply = { + output: { message: { role: "assistant", content: [{ text: "Soon." }] } }, + stopReason: "end_turn", + }; + + const client = wrapBedrock( + new FakeClient([ + converseReply, + converseReply, + { + body: bytes({ content: [{ type: "text", text: "Soon." }], stop_reason: "end_turn" }), + contentType: "application/json", + }, + { + body: bytes({ embedding: [0.1, 0.2], inputTextTokenCount: 6 }), + contentType: "application/json", + }, + ]), + ); + + await client.send( + new ConverseCommand({ + modelId: "anthropic.claude-3-haiku", + system: [{ text: system!.content }], + messages: [{ role: "user", content: [{ text: "Hi" }] }], + }), + ); + await client.send( + new ConverseCommand({ + modelId: "anthropic.claude-3-haiku", + messages: [{ role: "user", content: [{ text: user!.content }] }], + }), + ); + await client.send( + new InvokeModelCommand({ + modelId: "anthropic.claude-3-haiku", + contentType: "application/json", + body: bytes({ + anthropic_version: "bedrock-2023-05-31", + max_tokens: 64, + messages: [{ role: "user", content: user!.content }], + }), + }), + ); + await client.send( + new InvokeModelCommand({ + modelId: "amazon.titan-embed-text-v2:0", + contentType: "application/json", + body: bytes({ inputText: user!.content }), + }), + ); + + const [systemSpan, messagesSpan, invokeSpan, embedSpan] = spans.getFinishedSpans(); + + for (const span of [systemSpan, messagesSpan, invokeSpan]) { + expect(span!.attributes["gen_ai.request.model"]).toBe("anthropic.claude-3-haiku"); + expect( + Object.entries(span!.attributes).filter(([key]) => key.startsWith("td.prompt.")), + ).toEqual( + olderCore + ? [] + : [ + ["td.prompt.name", "support/reply"], + ["td.prompt.version", 12], + ["td.prompt.label", "production"], + ["td.prompt.variables", '{"team":"billing","order":"#4"}'], + ], + ); + } + + expect(embedSpan!.attributes["gen_ai.operation.name"]).toBe("embeddings"); + expect(embedSpan!.attributes["td.prompt.name"]).toBeUndefined(); + }, +); + test("InvokeModel captures provider-native bodies and embedding spans", async () => { const spans = setup(); diff --git a/packages/google-genai/src/index.ts b/packages/google-genai/src/index.ts index 96ffef5..2272e46 100644 --- a/packages/google-genai/src/index.ts +++ b/packages/google-genai/src/index.ts @@ -567,7 +567,11 @@ function responseAttributes(response: UnknownRecord, metadataContent: boolean) { function generateRequestFields(params: UnknownRecord): RequestMapping { const model = readString(params.model); const config = asRecord(params.config); - const attributes = requestAttributesFromConfig(config); + + const attributes = { + ...requestAttributesFromConfig(config), + ...promptAttributes([config?.systemInstruction, params.contents]), + }; const input = captureEnabled("input") ? normalizeContentInput(params.contents) : undefined; const systemInstructions = captureEnabled("input") @@ -940,6 +944,16 @@ function captureMaskedBeforeStart(): boolean { } } +// Cores older than promptAttributesFor record a prompt only inside withPrompt. A namespace read +// keeps them loadable without the export. +function promptAttributes(request: unknown): SpanFields["attributes"] { + const read = ( + telemetrySdk as { promptAttributesFor?: (request: unknown) => SpanFields["attributes"] } + ).promptAttributesFor; + + return typeof read === "function" ? read(request) : undefined; +} + // The decision is made before the span starts so samplers see the same attributes as before. function startRequestSpan(request: RequestMapping, provider: string): SpanHandle { const fields = { ...request.fields, provider }; diff --git a/packages/google-genai/tests/google-genai.test.ts b/packages/google-genai/tests/google-genai.test.ts index d1d291a..e7417ae 100644 --- a/packages/google-genai/tests/google-genai.test.ts +++ b/packages/google-genai/tests/google-genai.test.ts @@ -10,7 +10,8 @@ import { type PushMetricExporter, type ResourceMetrics, } from "@opentelemetry/sdk-metrics"; -import { flush, init, shutdown } from "@telemetry-dev/sdk"; +import { type ChatMessage, flush, getPrompt, init, shutdown } from "@telemetry-dev/sdk"; +import * as telemetrySdk from "@telemetry-dev/sdk"; import { FunctionCallingConfigMode, GoogleGenAI, @@ -234,6 +235,84 @@ afterEach(async () => { vi.restoreAllMocks(); }); +test.each([ + { field: "systemInstruction", captureInput: true, olderCore: false }, + { field: "contents", captureInput: false, olderCore: false }, + { field: "contents", captureInput: true, olderCore: true }, +])( + "generateContent outside withPrompt records the prompt of a compiled text in $field, input capture $captureInput, older core $olderCore", + async ({ field, captureInput, olderCore }) => { + if (olderCore) { + vi.spyOn( + telemetrySdk as { promptAttributesFor: unknown }, + "promptAttributesFor", + "get", + ).mockReturnValue(undefined); + } + + const prompt = await getPrompt("support/reply", { + apiKey: "td_live_a", + apiUrl: "https://prompts.google-genai.test", + fetch: async () => + Response.json({ + name: "support/reply", + type: "chat", + label: "production", + version: 12, + messages: [ + { role: "system", content: "You answer for the {{team}} desk." }, + { role: "user", content: "Where is order {{order}}?" }, + ], + text: null, + model: { provider: "gcp.gemini", name: "gemini-2.5-flash" }, + parameters: {}, + tools: [], + responseFormat: { type: "text" }, + rollout: null, + }), + }); + const [system, user] = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const spans = setupSpans({ captureInput }); + vi.stubGlobal( + "fetch", + createFakeFetch( + jsonResponse({ + candidates: [ + { content: { role: "model", parts: [{ text: "Soon." }] }, finishReason: "STOP" }, + ], + }), + ).fetch, + ); + + await clientWith().models.generateContent({ + model: "gemini-2.5-flash", + contents: [{ role: "user", parts: [{ text: field === "contents" ? user!.content : "Hi" }] }], + config: { + temperature: 0.2, + systemInstruction: field === "systemInstruction" ? system!.content : "Be terse.", + }, + }); + + const { attributes } = await exportedSpan(spans); + expect(attributes["gen_ai.request.temperature"]).toBe(0.2); + + if (olderCore) { + expect(attributes["td.prompt.name"]).toBeUndefined(); + + return; + } + + expect(attributes).toMatchObject({ + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + }); + expect(attributes["td.prompt.variables"]).toBe( + captureInput ? '{"team":"billing","order":"#4"}' : undefined, + ); + }, +); + test("generateContent maps request, response, usage, finish reason, provider, and sampling attributes", async () => { const spans = setupSpans(); diff --git a/packages/openai/src/index.ts b/packages/openai/src/index.ts index 3512476..c62671c 100644 --- a/packages/openai/src/index.ts +++ b/packages/openai/src/index.ts @@ -156,6 +156,7 @@ function stopSequences(value: T): string[] | undefined { function chatRequest(body: JsonRecord): RequestMapping { const model = readString(body.model); const capture = captureEnabled("input") ? sanitizeResponsesCapture(body.messages) : undefined; + const prompt = promptAttributes(body.messages); return { name: `chat ${model ?? "unknown"}`, @@ -170,7 +171,9 @@ function chatRequest(body: JsonRecord): RequestMapping { seed: readNumber(body.seed), frequencyPenalty: readNumber(body.frequency_penalty), presencePenalty: readNumber(body.presence_penalty), - attributes: capture?.truncated ? { "telemetry.dev.capture.truncated": true } : undefined, + attributes: capture?.truncated + ? { ...prompt, "telemetry.dev.capture.truncated": true } + : prompt, }, }; } @@ -243,6 +246,7 @@ function chatResponse(response: T): SpanFields { function responsesRequest(body: JsonRecord): RequestMapping { const model = readString(body.model); const capture = captureEnabled("input") ? sanitizeResponsesCapture(body.input) : undefined; + const prompt = promptAttributes([body.instructions, body.input]); return { name: `chat ${model ?? "unknown"}`, @@ -254,7 +258,9 @@ function responsesRequest(body: JsonRecord): RequestMapping { temperature: readNumber(body.temperature), topP: readNumber(body.top_p), maxTokens: readNumber(body.max_output_tokens), - attributes: capture?.truncated ? { "telemetry.dev.capture.truncated": true } : undefined, + attributes: capture?.truncated + ? { ...prompt, "telemetry.dev.capture.truncated": true } + : prompt, }, }; } @@ -1626,6 +1632,16 @@ function captureMaskedBeforeStart(): boolean { } } +// Cores older than promptAttributesFor record a prompt only inside withPrompt. A namespace read +// keeps them loadable without the export. +function promptAttributes(request: unknown): SpanFields["attributes"] { + const read = ( + telemetrySdk as { promptAttributesFor?: (request: unknown) => SpanFields["attributes"] } + ).promptAttributesFor; + + return typeof read === "function" ? read(request) : undefined; +} + // The decision is made before the span starts so samplers see the same attributes as before. function startRequestSpan(name: string, fields: StartSpanOptions): SpanHandle { const cut = diff --git a/packages/openai/tests/openai.test.ts b/packages/openai/tests/openai.test.ts index 4467aa3..06518ee 100644 --- a/packages/openai/tests/openai.test.ts +++ b/packages/openai/tests/openai.test.ts @@ -8,7 +8,14 @@ import { InMemoryMetricExporter, type PushMetricExporter, } from "@opentelemetry/sdk-metrics"; -import { flush, init, shutdown, type TelemetryOptions } from "@telemetry-dev/sdk"; +import { + type ChatMessage, + flush, + getPrompt, + init, + shutdown, + type TelemetryOptions, +} from "@telemetry-dev/sdk"; import * as telemetrySdk from "@telemetry-dev/sdk"; import { EventEmitter } from "node:events"; import { createRequire } from "node:module"; @@ -189,7 +196,7 @@ function setupSpans( metricExporter?: PushMetricExporter, options: Pick< TelemetryOptions, - "captureOutput" | "mask" | "maxAttributeLength" | "onError" | "sampler" + "captureInput" | "captureOutput" | "mask" | "maxAttributeLength" | "onError" | "sampler" > = {}, ): InMemorySpanExporter { const spanExporter = new InMemorySpanExporter(); @@ -2887,6 +2894,107 @@ test("chat streams preserve caller-requested usage chunks", async () => { expect(span.attributes["gen_ai.usage.total_tokens"]).toBe(7); }); +test.each([ + { api: "chat completions", captureInput: false, olderCore: false }, + { api: "responses instructions", captureInput: true, olderCore: false }, + { api: "responses input", captureInput: true, olderCore: false }, + { api: "chat completions", captureInput: true, olderCore: true }, +])( + "$api outside withPrompt record the prompt of compiled messages, input capture $captureInput, older core $olderCore", + async ({ api, captureInput, olderCore }) => { + if (olderCore) { + vi.spyOn( + telemetrySdk as { promptAttributesFor: unknown }, + "promptAttributesFor", + "get", + ).mockReturnValue(undefined); + } + + const prompt = await getPrompt("support/reply", { + apiKey: "td_live_a", + apiUrl: "https://prompts.openai.test", + fetch: async () => + Response.json({ + name: "support/reply", + type: "chat", + label: "production", + version: 12, + messages: [ + { role: "system", content: "You answer for the {{team}} desk." }, + { role: "user", content: "Where is order {{order}}?" }, + ], + text: null, + model: { provider: "openai", name: "gpt-4o" }, + parameters: {}, + tools: [], + responseFormat: { type: "text" }, + rollout: null, + }), + }); + const [system, user] = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const spans = setupSpans(undefined, { captureInput }); + const client = clientWith( + createFakeFetch( + jsonResponse( + api.startsWith("responses") + ? { + id: "resp_1", + object: "response", + status: "completed", + model: "gpt-4.1", + output: [], + usage: { input_tokens: 1, output_tokens: 1, total_tokens: 2 }, + } + : { + id: "chatcmpl_1", + object: "chat.completion", + created: 1, + model: "gpt-4o", + choices: [ + { + index: 0, + message: { role: "assistant", content: "Soon." }, + finish_reason: "stop", + }, + ], + usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 }, + }, + ), + ).fetch, + ); + + if (api.startsWith("responses")) { + const inInput = api === "responses input"; + + await client.responses.create({ + model: "gpt-4.1", + instructions: inInput ? "Be terse." : system!.content, + input: [{ role: "user", content: inInput ? user!.content : "Hi" }], + }); + } else { + await client.chat.completions.create({ model: "gpt-4o", messages: [system!, user!] }); + } + + const { attributes } = await exportedSpan(spans); + expect(attributes["gen_ai.input.messages"] !== undefined).toBe(captureInput); + + if (olderCore) { + expect(attributes["td.prompt.name"]).toBeUndefined(); + + return; + } + + expect(attributes).toMatchObject({ + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + }); + expect(attributes["td.prompt.variables"]).toBe( + captureInput ? '{"team":"billing","order":"#4"}' : undefined, + ); + }, +); + test("responses create maps instructions to system instructions", async () => { const spans = setupSpans(); diff --git a/packages/openrouter/src/index.ts b/packages/openrouter/src/index.ts index bfb0076..e0d696e 100644 --- a/packages/openrouter/src/index.ts +++ b/packages/openrouter/src/index.ts @@ -4,6 +4,7 @@ import { type SpanHandle, type StartSpanOptions, } from "@telemetry-dev/sdk"; +import * as telemetrySdk from "@telemetry-dev/sdk"; import { activeContext } from "@telemetry-dev/otel"; import { Chat } from "@openrouter/sdk/sdk/chat.js"; import { Embeddings } from "@openrouter/sdk/sdk/embeddings.js"; @@ -384,6 +385,16 @@ function usageCost(usage: T): number | undefined { return cost ?? upstreamCost; } +// Cores older than promptAttributesFor record a prompt only inside withPrompt. A namespace read +// keeps them loadable without the export. +function promptAttributes(request: unknown): SpanFields["attributes"] { + const read = ( + telemetrySdk as { promptAttributesFor?: (request: unknown) => SpanFields["attributes"] } + ).promptAttributesFor; + + return typeof read === "function" ? read(request) : undefined; +} + function chatRequest(request: JsonRecord): RequestMapping { const body = asRecord(request.chatRequest) ?? {}; const model = readString(body.model); @@ -403,6 +414,7 @@ function chatRequest(request: JsonRecord): RequestMapping { seed: readNumber(body.seed), frequencyPenalty: readNumber(body.frequencyPenalty), presencePenalty: readNumber(body.presencePenalty), + attributes: promptAttributes(body.messages), }, }; } @@ -472,6 +484,7 @@ function responsesRequest(request: JsonRecord): RequestMapping { maxTokens: readNumber(body.maxOutputTokens), frequencyPenalty: readNumber(body.frequencyPenalty), presencePenalty: readNumber(body.presencePenalty), + attributes: promptAttributes([body.instructions, body.input]), }, }; } diff --git a/packages/openrouter/tests/openrouter.test.ts b/packages/openrouter/tests/openrouter.test.ts index 2dc9baf..dc67157 100644 --- a/packages/openrouter/tests/openrouter.test.ts +++ b/packages/openrouter/tests/openrouter.test.ts @@ -5,13 +5,16 @@ import { Embeddings } from "@openrouter/sdk/sdk/embeddings.js"; import { EventStream } from "@openrouter/sdk/lib/event-streams.js"; import { Responses } from "@openrouter/sdk/sdk/responses.js"; import { + type ChatMessage, flush, + getPrompt, init, type MaskFn, shutdown, startActiveSpan, type TelemetryOptions, } from "@telemetry-dev/sdk"; +import * as telemetrySdk from "@telemetry-dev/sdk"; import { afterEach, expect, test, vi } from "vitest"; import { instrumentOpenRouter, uninstrumentOpenRouter, wrapOpenRouter } from "../src/index.ts"; @@ -308,6 +311,96 @@ afterEach(async () => { vi.restoreAllMocks(); }); +test.each([ + { api: "chat", olderCore: false }, + { api: "streamed responses instructions", olderCore: false }, + { api: "responses input", olderCore: false }, + { api: "chat", olderCore: true }, +])( + "$api outside withPrompt record the prompt of compiled messages, older core $olderCore", + async ({ api, olderCore }) => { + if (olderCore) { + vi.spyOn( + telemetrySdk as { promptAttributesFor: unknown }, + "promptAttributesFor", + "get", + ).mockReturnValue(undefined); + } + + const prompt = await getPrompt("support/reply", { + apiKey: "td_live_a", + apiUrl: "https://prompts.openrouter.test", + fetch: async () => + Response.json({ + name: "support/reply", + type: "chat", + label: "production", + version: 12, + messages: [ + { role: "system", content: "You answer for the {{team}} desk." }, + { role: "user", content: "Where is order {{order}}?" }, + ], + text: null, + model: { provider: "openrouter", name: "openai/gpt-4o" }, + parameters: {}, + tools: [], + responseFormat: { type: "text" }, + rollout: null, + }), + }); + const [system, user] = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const spans = setupSpans(); + + if (api === "chat") { + const client = clientWith(createFakeFetcher(jsonResponse(chatBody("chat_1"))).fetcher); + await client.chat.send({ + chatRequest: { model: "openai/gpt-4o", messages: [system!, user!] }, + }); + } else if (api === "responses input") { + const client = clientWith(createFakeFetcher(jsonResponse(responsesBody("resp_1"))).fetcher); + await client.responses.send({ + responsesRequest: { + model: "openai/gpt-4o", + input: [{ type: "message", role: "user", content: user!.content }], + }, + }); + } else { + const client = clientWith( + createFakeFetcher( + sseResponse([{ type: "response.completed", response: responsesBody("resp_1") }]), + ).fetcher, + ); + const stream = await client.responses.send({ + responsesRequest: { + model: "openai/gpt-4o", + instructions: system!.content, + input: "Hi", + stream: true, + }, + }); + + if (!(stream instanceof ReadableStream)) throw new Error("expected a readable stream"); + await collectStream(stream); + } + + const { attributes } = await exportedSpan(spans); + expect(attributes["gen_ai.request.model"]).toBe("openai/gpt-4o"); + + if (olderCore) { + expect(attributes["td.prompt.name"]).toBeUndefined(); + + return; + } + + expect(attributes).toMatchObject({ + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + "td.prompt.variables": '{"team":"billing","order":"#4"}', + }); + }, +); + test("chat maps request, response, usage, primary cost, provider, and sampling fields", async () => { const spans = setupSpans(); diff --git a/packages/sdk/README.md b/packages/sdk/README.md index afd5ef1..00cd298 100644 --- a/packages/sdk/README.md +++ b/packages/sdk/README.md @@ -179,8 +179,18 @@ compiles its own. Model calls are the spans of type `generation` (operations `ch `text_completion`, and `generate_content`), including the spans of the provider integrations that use this SDK: OpenAI, Anthropic, Amazon Bedrock, Google GenAI, and OpenRouter. Spans from the AI SDK and TanStack AI integrations, and from other OpenTelemetry tracers, do not get the attributes. -A model call outside `withPrompt` does not get the attributes, even when it sends messages from -`compile()`; with input capture on, the call still records those messages as its input. + +Outside `withPrompt`, those provider integrations find the prompt in the request: +`compile()` keeps the texts that it makes (the last 1000, at most 2,000,000 characters), and a +chat call whose request holds one of them gets `td.prompt.name`, `td.prompt.version`, and +`td.prompt.label`, also when input capture is off. A text from a template with a variable also +gives `td.prompt.variables`, and it wins over a text from a template with none. A message of only +a variable tag, for example `{{ question }}`, is not kept, because its text is your user's input. +The text in the request must be equal to the compiled text. `withPrompt` wins over this. The +integrations do it for OpenAI chat completions and responses, Anthropic messages, Bedrock Converse +and InvokeModel, Google GenAI `generateContent`, and OpenRouter chat and responses, but not for +images, audio, video, or OpenAI Realtime. For your own spans, `promptAttributesFor(request)` +gives the attributes to pass as `attributes`; it gives nothing inside `withPrompt`. ## API diff --git a/packages/sdk/src/index.ts b/packages/sdk/src/index.ts index 2fc3343..18d8946 100644 --- a/packages/sdk/src/index.ts +++ b/packages/sdk/src/index.ts @@ -54,6 +54,7 @@ export { type GetPromptOptions, type GetTextPromptOptions, type Prompt, + promptAttributesFor, PromptError, type PromptMessage, type PromptModel, diff --git a/packages/sdk/src/prompts.ts b/packages/sdk/src/prompts.ts index 2e27b17..afd76c2 100644 --- a/packages/sdk/src/prompts.ts +++ b/packages/sdk/src/prompts.ts @@ -1,4 +1,9 @@ -import { type Attributes, type Context, createContextKey } from "@opentelemetry/api"; +import { + type Attributes, + type AttributeValue, + type Context, + createContextKey, +} from "@opentelemetry/api"; import { activeContext, DEFAULT_BASE_URL, @@ -92,6 +97,8 @@ export interface ChatPrompt extends PromptBase { /** * Fills in the variables and replaces each placeholder message with the messages in its * variable. `withPrompt` records the variables of the last call before it starts or inside it. + * Each text is also kept with its variables (the last 1000 texts), so that a model call outside + * `withPrompt` that sends it can name the prompt; see `promptAttributesFor`. */ compile(variables?: PromptVariables): (ChatMessage | M)[]; } @@ -102,7 +109,8 @@ export interface TextPrompt extends PromptBase { readonly text: string; /** * Fills in the variables. `withPrompt` records the variables of the last call before it starts - * or inside it. + * or inside it. The text is also kept with its variables (the last 1000 texts), so that a model + * call outside `withPrompt` that sends it can name the prompt; see `promptAttributesFor`. */ compile(variables?: PromptVariables): string; } @@ -193,6 +201,13 @@ interface PromptScope { variables: object | undefined; } +/** The prompt that `compile()` made a text from. */ +interface CompiledText { + readonly prompt: Prompt; + /** Only for a template with a variable: the text holds the values, so they belong to it. */ + readonly variables: object | undefined; +} + const FETCH_TIMEOUT_MS = 10_000; /** The longest Retry-After that getPrompt honors, so that a bad header cannot stop fetches. */ const MAX_RETRY_AFTER_MS = 3_600_000; @@ -204,11 +219,19 @@ const HTTP_DATE = /^[A-Z][a-z]{2}, \d{2} [A-Z][a-z]{2} \d{4} \d{2}:\d{2}:\d{2} G const VARIABLE = /\{\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*\}\}/g; const MODEL_CALL_OPERATIONS = new Set(["chat", "text_completion", "generate_content"]); const PROMPT_KEY = createContextKey("telemetry.dev prompt"); +const MAX_COMPILED_TEXTS = 1_000; +const MAX_COMPILED_CHARACTERS = 2_000_000; +const MAX_SEARCH_DEPTH = 6; +const MAX_SEARCH_NODES = 500; const cache = new Map(); const compiledVariables = new WeakMap(); /** The Retry-After of the response that failed with this error, in milliseconds. */ const retryAfter = new WeakMap(); +// The texts of the last compile() calls, oldest first, for model calls outside withPrompt. +const compiledTexts = new Map(); +let compiledCharacters = 0; +let longestCompiledText = 0; /** * Fetches a prompt version from telemetry.dev and keeps it. When the copy is older than @@ -344,19 +367,97 @@ export function promptAttributes( | undefined; if (!scope) return undefined; - const { prompt, variables } = scope; + return attributesOf(scope.prompt, scope.variables, capture); +} + +/** + * The td.prompt.* attributes for a model call outside `withPrompt`, from the texts that + * `compile()` made: the first text in `request` from a template with a variable, else the first + * text from a template with none. Only the first gives `td.prompt.variables`, because only its text + * holds the values. Gives undefined inside `withPrompt`, whose prompt the span gets, and when no + * text matches. It reads own data properties of arrays and plain objects, 6 levels and 500 values + * deep, and runs no getter. It never throws. + */ +export function promptAttributesFor(request: unknown): Record | undefined { + const config = currentClient().core?.config; + + try { + if (compiledTexts.size === 0 || activeContext().getValue(PROMPT_KEY) !== undefined) { + return undefined; + } + + const found = findCompiledText(request); + + if (!found) return undefined; + + return attributesOf(found.prompt, found.variables, config) as Record; + } catch (error) { + reportError(config?.onError, error); + + return undefined; + } +} + +function attributesOf( + prompt: Prompt, + variables: object | undefined, + capture: FieldCaptureConfig | undefined, +): Attributes { return omitUndefined({ "td.prompt.name": prompt.name, "td.prompt.version": prompt.version ?? undefined, "td.prompt.label": prompt.label ?? undefined, "td.prompt.variables": - variables !== undefined && capture.captureInput + variables !== undefined && capture?.captureInput ? prepareCaptureValue("td.prompt.variables", variables, capture) : undefined, }); } +function findCompiledText(request: unknown): CompiledText | undefined { + let firstStatic: CompiledText | undefined; + let budget = MAX_SEARCH_NODES; + const stack: [value: unknown, depth: number][] = [[request, 0]]; + + while (stack.length > 0 && budget > 0) { + budget -= 1; + const [value, depth] = stack.pop()!; + + if (typeof value === "string") { + const found = value.length <= longestCompiledText ? compiledTexts.get(value) : undefined; + + if (found?.variables !== undefined) return found; + firstStatic ??= found; + } else if (depth < MAX_SEARCH_DEPTH) { + const children = ownValues(value, budget); + + for (let index = children.length - 1; index >= 0; index -= 1) { + stack.push([children[index], depth + 1]); + } + } + } + + return firstStatic; +} + +/** At most `limit` own data values of an array or a plain object, in order. */ +function ownValues(value: unknown, limit: number): unknown[] { + if (typeof value !== "object" || value === null) return []; + let keys: string[]; + + if (Array.isArray(value)) { + keys = Array.from({ length: Math.min(value.length, limit) }, (_, index) => String(index)); + } else { + const prototype: unknown = Object.getPrototypeOf(value); + + if (prototype !== Object.prototype && prototype !== null) return []; + keys = Object.keys(value).slice(0, limit); + } + + return keys.map((key) => Object.getOwnPropertyDescriptor(value, key)?.value); +} + function checkOptions(name: string, options: GetPromptOptions, type: string): void { if (typeof name !== "string" || name.length === 0) { throw new TypeError("name must be a non-empty string"); @@ -548,9 +649,12 @@ function createPrompt(input: { if (scope?.prompt === prompt) scope.variables = variables; + const compileText = (template: string) => + remember(fill(template, variables), template, prompt, variables); + return type === "chat" - ? compileMessages(content.messages, variables) - : fill(content.text ?? "", variables); + ? compileMessages(content.messages, variables, compileText) + : compileText(content.text ?? ""); }, }) as Prompt; @@ -560,10 +664,11 @@ function createPrompt(input: { function compileMessages( messages: readonly PromptMessage[], variables: PromptVariables, + compileText: (template: string) => string, ): unknown[] { return messages.flatMap((message) => { if (message.role !== "placeholder") { - return [{ role: message.role, content: fill(message.content, variables) }]; + return [{ role: message.role, content: compileText(message.content) }]; } const value = Object.hasOwn(variables, message.name) ? variables[message.name] : undefined; @@ -578,6 +683,39 @@ function compileMessages( }); } +/** + * Keeps `text` for promptAttributesFor, as the newest entry. A template of only variable tags is + * skipped: its text is input of the app, which an unrelated request can hold too. + */ +function remember(text: string, template: string, prompt: Prompt, variables: object): string { + const fixedText = template.replace(VARIABLE, ""); + + if (fixedText.trim() === "" || text.length > MAX_COMPILED_CHARACTERS) return text; + + if (compiledTexts.delete(text)) compiledCharacters -= text.length; + compiledTexts.set(text, { prompt, variables: fixedText === template ? undefined : variables }); + compiledCharacters += text.length; + longestCompiledText = Math.max(longestCompiledText, text.length); + let droppedLongest = false; + + while (compiledTexts.size > MAX_COMPILED_TEXTS || compiledCharacters > MAX_COMPILED_CHARACTERS) { + const oldest = compiledTexts.keys().next().value!; + compiledTexts.delete(oldest); + compiledCharacters -= oldest.length; + droppedLongest ||= oldest.length === longestCompiledText; + } + + if (droppedLongest) { + longestCompiledText = 0; + + for (const kept of compiledTexts.keys()) { + longestCompiledText = Math.max(longestCompiledText, kept.length); + } + } + + return text; +} + function fill(text: string, variables: PromptVariables): string { return text.replace(VARIABLE, (_tag, name: string) => { const value = Object.hasOwn(variables, name) ? variables[name] : undefined; diff --git a/packages/sdk/tests/prompts.test.ts b/packages/sdk/tests/prompts.test.ts index 6048d16..372150e 100644 --- a/packages/sdk/tests/prompts.test.ts +++ b/packages/sdk/tests/prompts.test.ts @@ -4,8 +4,11 @@ import type { AddressInfo } from "node:net"; import { afterEach, beforeEach, expect, test, vi } from "vitest"; import { + type ChatMessage, flush, getPrompt, + type Prompt, + promptAttributesFor, propagateAttributes, PromptError, shutdown, @@ -729,6 +732,231 @@ test("variables are omitted when input capture is off; a version fetch has no la expect(off.spans.getFinishedSpans()[0]!.attributes["td.prompt.variables"]).toBeUndefined(); }); +async function promptOf(overrides: Record): Promise { + const { fetch } = fakeFetch(reply(overrides)); + + return getPrompt("support/reply", { + apiKey: "td_live_a", + apiUrl: newApiUrl(), + fetch, + type: overrides.type as "chat", + }); +} + +test("promptAttributesFor finds a compiled text in the request shapes of the providers", async () => { + setup(); + const prompt = await promptOf({ + messages: [ + { role: "system", content: "Answer as the {{team}} desk." }, + { role: "user", content: "Order {{order}}: where is it?" }, + ], + }); + const compiled = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const [system, user] = compiled.map((message) => message.content); + + const requests = { + same: compiled, + spread: compiled.map((message) => ({ ...message })), + json: JSON.parse(JSON.stringify(compiled)) as unknown, + anthropic: [system, compiled.slice(1)], + bedrock: [[{ text: system }], [{ role: "user", content: [{ text: user }] }]], + google: [{ parts: [{ text: system }] }, [{ role: "user", parts: [{ text: user }] }]], + }; + + for (const request of Object.values(requests)) { + expect(promptAttributesFor(request)).toEqual({ + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + "td.prompt.variables": '{"team":"billing","order":"#4"}', + }); + } + + expect( + promptAttributesFor([{ role: "user", content: "Order #5: where is it?" }]), + ).toBeUndefined(); +}); + +test("a static text gives no variables, and a template of only a variable tag is not kept", async () => { + setup(); + const chat = await promptOf({ + messages: [ + { role: "system", content: "You are the refunds agent." }, + { role: "user", content: "{{ question }}" }, + ], + }); + const compiled = chat.compile({ question: "Where is my refund?" }); + + expect(promptAttributesFor(compiled)).toEqual({ + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + }); + expect(promptAttributesFor([{ role: "user", content: "Where is my refund?" }])).toBeUndefined(); + + const text = await promptOf({ type: "text", messages: [], text: "Summarize {{doc}} in a line." }); + const summary = text.compile({ doc: "the refund policy" }) as string; + + expect(promptAttributesFor(summary)).toMatchObject({ + "td.prompt.variables": '{"doc":"the refund policy"}', + }); +}); + +test("a text with variables wins over a static text, and the first one wins over a later one", async () => { + setup(); + const refunds = await promptOf({ + name: "refunds/system", + messages: [{ role: "system", content: "You are the refunds agent, v2." }], + }); + const order = await promptOf({ + name: "orders/question", + messages: [{ role: "user", content: "Where is order {{order}}?" }], + }); + const language = await promptOf({ + name: "style/language", + messages: [{ role: "system", content: "Reply in {{language}}." }], + }); + + const system = refunds.compile() as ChatMessage[]; + const question = order.compile({ order: "#4" }) as ChatMessage[]; + const style = language.compile({ language: "French" }) as ChatMessage[]; + + expect(promptAttributesFor([...system, ...question])).toMatchObject({ + "td.prompt.name": "orders/question", + "td.prompt.variables": '{"order":"#4"}', + }); + expect(promptAttributesFor([...system, ...style, ...question])).toMatchObject({ + "td.prompt.name": "style/language", + "td.prompt.variables": '{"language":"French"}', + }); +}); + +test("each compile keeps its own values, so two tasks with one prompt get their own variables", async () => { + setup(); + const prompt = await promptOf({ + messages: [{ role: "user", content: "Track parcel {{parcel}}." }], + }); + + const first = prompt.compile({ parcel: "A-1" }); + const second = prompt.compile({ parcel: "B-2" }); + + expect(promptAttributesFor(first)?.["td.prompt.variables"]).toBe('{"parcel":"A-1"}'); + expect(promptAttributesFor(second)?.["td.prompt.variables"]).toBe('{"parcel":"B-2"}'); +}); + +test("withPrompt wins over a compiled text of another prompt", async () => { + const { spans } = setup(); + const scoped = await promptOf({ name: "scoped/prompt" }); + const other = await promptOf({ + name: "other/prompt", + messages: [{ role: "user", content: "Cancel order {{order}}." }], + }); + const compiled = other.compile({ order: "#9" }); + + withPrompt(scoped, () => { + startSpan("chat", { type: "generation", attributes: promptAttributesFor(compiled) }).end(); + }); + await flush(); + + expect(spans.getFinishedSpans()[0]!.attributes["td.prompt.name"]).toBe("scoped/prompt"); +}); + +test("promptAttributesFor omits variables when input capture is off and masks them when on", async () => { + const prompt = await promptOf({ + messages: [{ role: "user", content: "My card is {{card}}." }], + }); + const compiled = prompt.compile({ card: "4111 1111" }); + + setup({ captureInput: false }); + expect(promptAttributesFor(compiled)).toEqual({ + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + }); + + await shutdown(); + setup({ + mask: (value, { key }) => (key === "td.prompt.variables" ? { card: "[masked]" } : value), + }); + expect(promptAttributesFor(compiled)?.["td.prompt.variables"]).toBe('{"card":"[masked]"}'); +}); + +test("the kept texts stop at 1000 and at 2,000,000 characters, oldest out first", async () => { + setup(); + const ticket = await promptOf({ type: "text", messages: [], text: "Ticket {{n}} is open." }); + + for (let n = 0; n <= 1000; n += 1) ticket.compile({ n }); + + expect(promptAttributesFor("Ticket 0 is open.")).toBeUndefined(); + expect(promptAttributesFor("Ticket 1 is open.")).toMatchObject({ + "td.prompt.variables": '{"n":1}', + }); + + const note = await promptOf({ type: "text", messages: [], text: "Note: {{body}}" }); + const notes = ["a", "b", "c", "d", "e"].map( + (letter) => note.compile({ body: letter.repeat(500_000 - "Note: ".length) }) as string, + ); + + expect(promptAttributesFor(notes[0])).toBeUndefined(); + expect(promptAttributesFor(notes[1])?.["td.prompt.name"]).toBe("support/reply"); + + // A text over the character limit is not kept, and it does not push the others out. + note.compile({ body: "f".repeat(2_000_001) }); + expect(promptAttributesFor(notes[1])?.["td.prompt.name"]).toBe("support/reply"); + + // A text compiled again counts once. + for (let n = 0; n < 5; n += 1) note.compile({ body: "e".repeat(500_000 - "Note: ".length) }); + expect(promptAttributesFor(notes[4])?.["td.prompt.name"]).toBe("support/reply"); +}); + +test("promptAttributesFor looks 6 levels and 500 values deep", async () => { + setup(); + const prompt = await promptOf({ + messages: [{ role: "user", content: "Find invoice {{invoice}}." }], + }); + const [{ content }] = prompt.compile({ invoice: "INV-7" }) as ChatMessage[]; + const nest = (value: unknown, levels: number): unknown => + levels === 0 ? value : nest([value], levels - 1); + + expect(promptAttributesFor(nest(content, 6))).toBeDefined(); + expect(promptAttributesFor(nest(content, 7))).toBeUndefined(); + + // The request array, 498 strings, and the text are 500 values. + expect(promptAttributesFor([...Array(498).fill("filler"), content])).toBeDefined(); + expect(promptAttributesFor([...Array(499).fill("filler"), content])).toBeUndefined(); +}); + +test("promptAttributesFor runs no getter and reports a trap that throws", async () => { + const errors: Error[] = []; + setup({ onError: (error) => errors.push(error) }); + const prompt = await promptOf({ + messages: [{ role: "user", content: "Refund order {{order}}." }], + }); + const [{ content }] = prompt.compile({ order: "#3" }) as ChatMessage[]; + let reads = 0; + + const getter = { + role: "user", + get content() { + reads += 1; + + return content; + }, + }; + + expect(promptAttributesFor([getter])).toBeUndefined(); + expect(reads).toBe(0); + + const trap = new Proxy([], { + get() { + throw new Error("trap"); + }, + }); + + expect(promptAttributesFor([trap, content])).toBeUndefined(); + expect(errors.map((error) => error.message)).toEqual(["trap"]); +}); + test("a fallback prompt records its name and label but no version", async () => { vi.spyOn(console, "error").mockImplementation(() => {}); const { spans } = setup(); diff --git a/sdks/python-anthropic/src/telemetry_dev_anthropic/__init__.py b/sdks/python-anthropic/src/telemetry_dev_anthropic/__init__.py index d842e15..956b634 100644 --- a/sdks/python-anthropic/src/telemetry_dev_anthropic/__init__.py +++ b/sdks/python-anthropic/src/telemetry_dev_anthropic/__init__.py @@ -244,6 +244,12 @@ def _offered_tools(tools: Any) -> dict[str, str] | None: return {"gen_ai.tool.definitions": json_dumps(definitions)} if definitions else None +def _prompt_attributes(request: object) -> dict[str, Any]: + # Older cores have no prompt_attributes_for and record a prompt only inside with_prompt. + read = getattr(telemetry_dev, "prompt_attributes_for", None) + return cast("dict[str, Any]", read(request)) if callable(read) else {} + + def _messages_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: params = _sent_params(params) model = _string(params.get("model")) @@ -264,7 +270,11 @@ def _messages_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: "type": "generation", "model": model, "input": input_value, - "attributes": _offered_tools(tools), + "attributes": { + **(_offered_tools(tools) or {}), + **_prompt_attributes([params.get("system"), params.get("messages")]), + } + or None, "system_instructions": _native(params.get("system")), "temperature": _number(params.get("temperature")), "top_p": _number(params.get("top_p")), diff --git a/sdks/python-anthropic/tests/test_anthropic.py b/sdks/python-anthropic/tests/test_anthropic.py index 3911ffb..8ffbc67 100644 --- a/sdks/python-anthropic/tests/test_anthropic.py +++ b/sdks/python-anthropic/tests/test_anthropic.py @@ -421,6 +421,56 @@ def handler(request: httpx.Request) -> httpx.Response: assert a["gen_ai.usage.cache_read.input_tokens"] == 1000 +SUPPORT_PROMPT = telemetry_dev.ChatPrompt( + name="support/reply", + version=12, + label="production", + messages=[ + {"role": "system", "content": "You answer for the {{team}} desk."}, + {"role": "user", "content": "Where is order {{order}}?"}, + ], + model=None, + parameters={}, + tools=[], + response_format={"type": "text"}, + is_fallback=False, +) + + +@pytest.mark.parametrize( + ("field", "capture_input", "older_core"), + [("system", False, False), ("messages", True, False), ("messages", True, True)], +) +def test_create_outside_with_prompt_records_the_prompt_of_a_compiled_text( + make: Any, monkeypatch: pytest.MonkeyPatch, field: str, capture_input: bool, older_core: bool +) -> None: + memory = make(capture_input=capture_input) + if older_core: + monkeypatch.delattr(telemetry_dev, "prompt_attributes_for") + system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") + client = wrapped_sync_client(lambda _request: json_response(message_payload())) + + client.messages.create( + model="claude-sonnet-4-6", + max_tokens=64, + system=system["content"] if field == "system" else "Be terse.", + messages=[{"role": "user", "content": user["content"] if field == "messages" else "Hi"}], + ) + + a = attrs(only_span(memory)) + assert a["gen_ai.request.model"] == "claude-sonnet-4-6" + assert {key: value for key, value in a.items() if key.startswith("td.prompt.")} == ( + {} + if older_core + else { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + **({"td.prompt.variables": '{"team":"billing","order":"#4"}'} if capture_input else {}), + } + ) + + def test_create_maps_native_messages_system_params_usage_finish_and_provider( memory: SimpleNamespace, ) -> None: diff --git a/sdks/python-bedrock/src/telemetry_dev_bedrock/_fields.py b/sdks/python-bedrock/src/telemetry_dev_bedrock/_fields.py index 1017467..04af28b 100644 --- a/sdks/python-bedrock/src/telemetry_dev_bedrock/_fields.py +++ b/sdks/python-bedrock/src/telemetry_dev_bedrock/_fields.py @@ -1,7 +1,8 @@ from __future__ import annotations -from typing import Any +from typing import Any, cast +import telemetry_dev from telemetry_dev._serialize import json_dumps PROVIDER = "amazon-bedrock" @@ -11,6 +12,12 @@ def clean(fields: dict[str, Any]) -> dict[str, Any]: return {key: value for key, value in fields.items() if value is not None} +def prompt_attributes(request: object) -> dict[str, Any]: + # Older cores have no prompt_attributes_for and record a prompt only inside with_prompt. + read = getattr(telemetry_dev, "prompt_attributes_for", None) + return cast("dict[str, Any]", read(request)) if callable(read) else {} + + def metadata_fields(response: dict[str, Any] | BaseException | None) -> dict[str, Any]: metadata: dict[str, Any] | None = None if isinstance(response, dict) and isinstance(response.get("ResponseMetadata"), dict): diff --git a/sdks/python-bedrock/src/telemetry_dev_bedrock/_invoke_model.py b/sdks/python-bedrock/src/telemetry_dev_bedrock/_invoke_model.py index d22e50b..04a1162 100644 --- a/sdks/python-bedrock/src/telemetry_dev_bedrock/_invoke_model.py +++ b/sdks/python-bedrock/src/telemetry_dev_bedrock/_invoke_model.py @@ -4,7 +4,7 @@ import math from typing import Any -from ._fields import clean +from ._fields import clean, prompt_attributes _USAGE_ITERATION_LIMIT = 1000 _CLAUDE_USAGE_KEYS = ( @@ -93,6 +93,8 @@ def request_fields(params: dict[str, Any]) -> dict[str, Any]: fields = {"provider": "amazon-bedrock", "model": model, "input": body} if is_embedding_model(model): fields["output_type"] = "embedding" + else: + fields["attributes"] = prompt_attributes(body) or None fields.update(sampling_fields(model, body)) return clean(fields) diff --git a/sdks/python-bedrock/src/telemetry_dev_bedrock/_registry.py b/sdks/python-bedrock/src/telemetry_dev_bedrock/_registry.py index 6d09848..0b68201 100644 --- a/sdks/python-bedrock/src/telemetry_dev_bedrock/_registry.py +++ b/sdks/python-bedrock/src/telemetry_dev_bedrock/_registry.py @@ -6,7 +6,14 @@ import telemetry_dev -from ._fields import PROVIDER, clean, merge_fields, metadata_fields, usage_from_converse +from ._fields import ( + PROVIDER, + clean, + merge_fields, + metadata_fields, + prompt_attributes, + usage_from_converse, +) from ._invoke_model import is_embedding_model from ._invoke_model import request_fields as invoke_request_fields from ._invoke_model import response_fields as invoke_response_fields @@ -50,6 +57,7 @@ def _converse_request(params: dict[str, Any]) -> dict[str, Any]: "model": _model(params), "input": normalize_messages(params.get("messages")), "system_instructions": normalize_content_list(params.get("system")) or None, + "attributes": prompt_attributes([params.get("system"), params.get("messages")]) or None, "temperature": inference.get("temperature"), "top_p": inference.get("topP"), "max_tokens": inference.get("maxTokens"), diff --git a/sdks/python-bedrock/tests/test_bedrock.py b/sdks/python-bedrock/tests/test_bedrock.py index 3b6acac..8919186 100644 --- a/sdks/python-bedrock/tests/test_bedrock.py +++ b/sdks/python-bedrock/tests/test_bedrock.py @@ -114,6 +114,93 @@ def _json_attr(span: Any, key: str) -> Any: return json.loads(value) +SUPPORT_PROMPT = telemetry_dev.ChatPrompt( + name="support/reply", + version=12, + label="production", + messages=[ + {"role": "system", "content": "You answer for the {{team}} desk."}, + {"role": "user", "content": "Where is order {{order}}?"}, + ], + model=None, + parameters={}, + tools=[], + response_format={"type": "text"}, + is_fallback=False, +) + + +@pytest.mark.parametrize("older_core", [False, True]) +def test_converse_and_invoke_model_outside_with_prompt_record_the_prompt_of_a_compiled_text( + memory: SimpleNamespace, monkeypatch: pytest.MonkeyPatch, older_core: bool +) -> None: + if older_core: + monkeypatch.delattr(telemetry_dev, "prompt_attributes_for") + system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") + converse_reply = { + "output": {"message": {"role": "assistant", "content": [{"text": "Soon."}]}}, + "stopReason": "end_turn", + } + answer = _body({"content": [{"type": "text", "text": "Soon."}], "stop_reason": "end_turn"}) + embedding = _body({"embedding": [0.1, 0.2], "inputTextTokenCount": 6}) + client = _client("bedrock-runtime") + _stub_api_call( + client, + [ + converse_reply, + converse_reply, + { + "body": StreamingBody(io.BytesIO(answer), len(answer)), + "contentType": "application/json", + }, + { + "body": StreamingBody(io.BytesIO(embedding), len(embedding)), + "contentType": "application/json", + }, + ], + monkeypatch, + ) + wrapped = cast(Any, wrap_bedrock(client)) + + wrapped.converse( + modelId="anthropic.claude-3-haiku", + system=[{"text": system["content"]}], + messages=[{"role": "user", "content": [{"text": "Hi"}]}], + ) + wrapped.converse( + modelId="anthropic.claude-3-haiku", + messages=[{"role": "user", "content": [{"text": user["content"]}]}], + ) + wrapped.invoke_model( + modelId="anthropic.claude-3-haiku", + contentType="application/json", + body=_body({"max_tokens": 64, "messages": [{"role": "user", "content": user["content"]}]}), + )["body"].read() + wrapped.invoke_model( + modelId="amazon.titan-embed-text-v2:0", + contentType="application/json", + body=_body({"inputText": user["content"]}), + )["body"].read() + + system_span, messages_span, invoke_span, embed_span = ( + dict(span.attributes or {}) for span in memory.span_exporter.get_finished_spans() + ) + for span in (system_span, messages_span, invoke_span): + assert span["gen_ai.request.model"] == "anthropic.claude-3-haiku" + assert {key: value for key, value in span.items() if key.startswith("td.prompt.")} == ( + {} + if older_core + else { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + "td.prompt.variables": '{"team":"billing","order":"#4"}', + } + ) + assert embed_span["gen_ai.operation.name"] == "embeddings" + assert "td.prompt.name" not in embed_span + + def test_converse_happy_path(memory: SimpleNamespace, monkeypatch: pytest.MonkeyPatch) -> None: client = _client("bedrock-runtime") request = { diff --git a/sdks/python-google-genai/src/telemetry_dev_google_genai/__init__.py b/sdks/python-google-genai/src/telemetry_dev_google_genai/__init__.py index 5782425..776c9de 100644 --- a/sdks/python-google-genai/src/telemetry_dev_google_genai/__init__.py +++ b/sdks/python-google-genai/src/telemetry_dev_google_genai/__init__.py @@ -353,6 +353,12 @@ def _request_attributes(config: Any) -> dict[str, Any]: return attrs +def _prompt_attributes(request: object) -> dict[str, Any]: + # Older cores have no prompt_attributes_for and record a prompt only inside with_prompt. + read = getattr(telemetry_dev, "prompt_attributes_for", None) + return cast("dict[str, Any]", read(request)) if callable(read) else {} + + def _generate_request_fields(model: Any, contents: Any, config: Any) -> tuple[str, dict[str, Any]]: model_name = _string(model) or "unknown" fields: dict[str, Any] = { @@ -383,7 +389,7 @@ def _generate_request_fields(model: Any, contents: Any, config: Any) -> tuple[st output_type = _output_type(config) if output_type is not None: fields["output_type"] = output_type - attrs = _request_attributes(config) + attrs = {**_request_attributes(config), **_prompt_attributes([system_instruction, contents])} if attrs: fields["attributes"] = attrs return f"chat {model_name}", fields diff --git a/sdks/python-google-genai/tests/test_google_genai.py b/sdks/python-google-genai/tests/test_google_genai.py index f045e68..c74c08c 100644 --- a/sdks/python-google-genai/tests/test_google_genai.py +++ b/sdks/python-google-genai/tests/test_google_genai.py @@ -656,6 +656,62 @@ async def run() -> Any: assert attrs(span)["google_genai.response.operation_done"] is True +SUPPORT_PROMPT = telemetry_dev.ChatPrompt( + name="support/reply", + version=12, + label="production", + messages=[ + {"role": "system", "content": "You answer for the {{team}} desk."}, + {"role": "user", "content": "Where is order {{order}}?"}, + ], + model=None, + parameters={}, + tools=[], + response_format={"type": "text"}, + is_fallback=False, +) + + +@pytest.mark.parametrize( + ("field", "capture_input", "older_core"), + [("system_instruction", True, False), ("contents", False, False), ("contents", True, True)], +) +def test_generate_content_outside_with_prompt_records_the_prompt_of_a_compiled_text( + make: Any, monkeypatch: pytest.MonkeyPatch, field: str, capture_input: bool, older_core: bool +) -> None: + memory = make(capture_input=capture_input) + if older_core: + monkeypatch.delattr(telemetry_dev, "prompt_attributes_for") + system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") + client, _transport = client_with_transport([happy_payload()]) + + wrap_google_genai(client).models.generate_content( + model="gemini-2.5-flash", + contents=[ + types.Content( + role="user", + parts=[types.Part(text=user["content"] if field == "contents" else "Hi")], + ) + ], + config=types.GenerateContentConfig( + system_instruction=system["content"] if field == "system_instruction" else "Be terse." + ), + ) + + a = attrs(only_span(memory)) + assert a["gen_ai.request.model"] == "gemini-2.5-flash" + assert {key: value for key, value in a.items() if key.startswith("td.prompt.")} == ( + {} + if older_core + else { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + **({"td.prompt.variables": '{"team":"billing","order":"#4"}'} if capture_input else {}), + } + ) + + @pytest.mark.parametrize("async_mode", [False, True], ids=["sync", "async"]) def test_generate_content_maps_request_response_usage_and_sampling( memory: SimpleNamespace, async_mode: bool diff --git a/sdks/python-litellm/src/telemetry_dev_litellm/__init__.py b/sdks/python-litellm/src/telemetry_dev_litellm/__init__.py index e149a00..ee18732 100644 --- a/sdks/python-litellm/src/telemetry_dev_litellm/__init__.py +++ b/sdks/python-litellm/src/telemetry_dev_litellm/__init__.py @@ -455,6 +455,12 @@ def _output_type(response_format: Any) -> str | None: return kind +def _prompt_attributes(request: object) -> dict[str, Any]: + # Older cores have no prompt_attributes_for and record a prompt only inside with_prompt. + read = getattr(telemetry_dev, "prompt_attributes_for", None) + return cast("dict[str, Any]", read(request)) if callable(read) else {} + + def _completion_request( args: tuple[Any, ...], kwargs: Mapping[str, Any] ) -> tuple[str, dict[str, Any]]: @@ -478,6 +484,7 @@ def _completion_request( "presence_penalty": _number(kwargs.get("presence_penalty")), "output_type": _output_type(kwargs.get("response_format")), "metadata": _request_metadata(kwargs), + "attributes": _prompt_attributes(_arg(args, kwargs, "messages", 1)) or None, }, ) @@ -545,7 +552,10 @@ def _responses_request( input_budget: telemetry_dev.CaptureBudget | None = None if capture_input: input_value, input_budget = _bounded_responses_native(_arg(args, kwargs, "input", 0)) - attributes: dict[str, Any] = {"gen_ai.operation.name": "chat"} + attributes: dict[str, Any] = { + "gen_ai.operation.name": "chat", + **_prompt_attributes([kwargs.get("instructions"), _arg(args, kwargs, "input", 0)]), + } if input_budget is not None and input_budget.truncated: attributes["telemetry.dev.capture.truncated"] = True return ( diff --git a/sdks/python-litellm/tests/test_litellm.py b/sdks/python-litellm/tests/test_litellm.py index f7b08a1..ae34865 100644 --- a/sdks/python-litellm/tests/test_litellm.py +++ b/sdks/python-litellm/tests/test_litellm.py @@ -57,6 +57,70 @@ def finished_spans(env: SimpleNamespace) -> list[ReadableSpan]: return list(env.span_exporter.get_finished_spans()) +SUPPORT_PROMPT = telemetry_dev.ChatPrompt( + name="support/reply", + version=12, + label="production", + messages=[ + {"role": "system", "content": "You answer for the {{team}} desk."}, + {"role": "user", "content": "Where is order {{order}}?"}, + ], + model=None, + parameters={}, + tools=[], + response_format={"type": "text"}, + is_fallback=False, +) + + +@pytest.mark.parametrize( + ("api", "capture_input", "older_core"), + [ + ("completion", False, False), + ("responses instructions", True, False), + ("responses input", True, False), + ("completion", True, True), + ], +) +def test_calls_outside_with_prompt_record_the_prompt_of_compiled_messages( + make: Any, monkeypatch: pytest.MonkeyPatch, api: str, capture_input: bool, older_core: bool +) -> None: + memory = make(capture_input=capture_input) + if older_core: + monkeypatch.delattr(telemetry_dev, "prompt_attributes_for") + system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") + + if api == "completion": + telemetry_dev_litellm.completion( + model="gpt-4o-mini", messages=[system, user], mock_response="Soon." + ) + else: + + def fake_responses(*_args: Any, **_kwargs: Any) -> SimpleNamespace: + return responses_result() + + monkeypatch.setattr(litellm, "responses", fake_responses) + in_input = api == "responses input" + telemetry_dev_litellm.responses( + [{"role": "user", "content": user["content"] if in_input else "Hi"}], + "openai/gpt-4.1-mini", + instructions="Be terse." if in_input else system["content"], + ) + + a = attrs(only_span(memory)) + assert a["gen_ai.request.model"] == ("gpt-4o-mini" if api == "completion" else "gpt-4.1-mini") + assert {key: value for key, value in a.items() if key.startswith("td.prompt.")} == ( + {} + if older_core + else { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + **({"td.prompt.variables": '{"team":"billing","order":"#4"}'} if capture_input else {}), + } + ) + + def test_completion_maps_model_messages_usage_finish_provider_and_sampling( memory: SimpleNamespace, ) -> None: diff --git a/sdks/python-openai/src/telemetry_dev_openai/__init__.py b/sdks/python-openai/src/telemetry_dev_openai/__init__.py index ee02a97..eb84699 100644 --- a/sdks/python-openai/src/telemetry_dev_openai/__init__.py +++ b/sdks/python-openai/src/telemetry_dev_openai/__init__.py @@ -349,9 +349,18 @@ def _capture_text(value: str, budget: telemetry_dev.CaptureBudget) -> str: return value +def _prompt_attributes(request: object) -> dict[str, Any]: + # Older cores have no prompt_attributes_for and record a prompt only inside with_prompt. + read = getattr(telemetry_dev, "prompt_attributes_for", None) + return cast("dict[str, Any]", read(request)) if callable(read) else {} + + def _chat_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: model = _string(params.get("model")) + attributes = _prompt_attributes(params.get("messages")) captured, truncated = _bounded_responses_capture(params.get("messages"), "input") + if truncated: + attributes["telemetry.dev.capture.truncated"] = True return ( f"chat {model or 'unknown'}", { @@ -366,7 +375,7 @@ def _chat_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: "seed": _number(params.get("seed")), "frequency_penalty": _number(params.get("frequency_penalty")), "presence_penalty": _number(params.get("presence_penalty")), - "attributes": {"telemetry.dev.capture.truncated": True} if truncated else None, + "attributes": attributes or None, }, ) @@ -432,7 +441,10 @@ def _chat_response(response: Any) -> dict[str, Any]: def _responses_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: model = _string(params.get("model")) + attributes = _prompt_attributes([params.get("instructions"), params.get("input")]) captured, truncated = _bounded_responses_capture(params.get("input"), "input") + if truncated: + attributes["telemetry.dev.capture.truncated"] = True return ( f"chat {model or 'unknown'}", { @@ -443,7 +455,7 @@ def _responses_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: "temperature": _number(params.get("temperature")), "top_p": _number(params.get("top_p")), "max_tokens": _number(params.get("max_output_tokens")), - "attributes": {"telemetry.dev.capture.truncated": True} if truncated else None, + "attributes": attributes or None, }, ) diff --git a/sdks/python-openai/tests/test_openai.py b/sdks/python-openai/tests/test_openai.py index 238c7e6..6f71978 100644 --- a/sdks/python-openai/tests/test_openai.py +++ b/sdks/python-openai/tests/test_openai.py @@ -1206,6 +1206,72 @@ def text(self) -> str: assert a["gen_ai.usage.text.output_tokens"] == 2 +SUPPORT_PROMPT = telemetry_dev.ChatPrompt( + name="support/reply", + version=12, + label="production", + messages=[ + {"role": "system", "content": "You answer for the {{team}} desk."}, + {"role": "user", "content": "Where is order {{order}}?"}, + ], + model=None, + parameters={}, + tools=[], + response_format={"type": "text"}, + is_fallback=False, +) + + +def prompt_attrs(span: ReadableSpan) -> dict[str, object]: + return {key: value for key, value in attrs(span).items() if key.startswith("td.prompt.")} + + +@pytest.mark.parametrize( + ("api", "capture_input", "older_core"), + [ + ("chat", False, False), + ("responses instructions", True, False), + ("responses input", True, False), + ("chat", True, True), + ], +) +def test_calls_outside_with_prompt_record_the_prompt_of_compiled_messages( + make: Any, monkeypatch: pytest.MonkeyPatch, api: str, capture_input: bool, older_core: bool +) -> None: + memory = make(capture_input=capture_input) + if older_core: + monkeypatch.delattr(telemetry_dev, "prompt_attributes_for") + system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") + + def handler(_request: httpx.Request) -> httpx.Response: + return json_response(chat_completion() if api == "chat" else response_payload()) + + client = wrap_openai(sync_client(handler)) + if api == "chat": + cast(Any, client.chat.completions.create)(model="gpt-4o", messages=[system, user]) + else: + in_input = api == "responses input" + cast(Any, client.responses.create)( + model="gpt-4.1", + instructions="Be terse." if in_input else system["content"], + input=[{"role": "user", "content": user["content"] if in_input else "Hi"}], + ) + + span = only_span(memory) + assert attrs(span)["gen_ai.request.model"] == ("gpt-4o" if api == "chat" else "gpt-4.1") + assert prompt_attrs(span) == ( + {} + if older_core + else { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + **({"td.prompt.variables": '{"team":"billing","order":"#4"}'} if capture_input else {}), + } + ) + assert ("gen_ai.input.messages" in attrs(span)) is capture_input + + def test_chat_completion_maps_native_messages_usage_finish_provider_and_sampling( memory: SimpleNamespace, ) -> None: diff --git a/sdks/python-openrouter/src/telemetry_dev_openrouter/__init__.py b/sdks/python-openrouter/src/telemetry_dev_openrouter/__init__.py index 016014c..4af3fb7 100644 --- a/sdks/python-openrouter/src/telemetry_dev_openrouter/__init__.py +++ b/sdks/python-openrouter/src/telemetry_dev_openrouter/__init__.py @@ -174,6 +174,12 @@ def _cost_usd(raw_usage: Any) -> float | None: return cost if cost is not None else upstream_cost +def _prompt_attributes(request: object) -> dict[str, Any]: + # Older cores have no prompt_attributes_for and record a prompt only inside with_prompt. + read = getattr(telemetry_dev, "prompt_attributes_for", None) + return cast("dict[str, Any]", read(request)) if callable(read) else {} + + def _chat_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: model = _string(params.get("model")) max_completion_tokens = _number(params.get("max_completion_tokens")) @@ -193,6 +199,7 @@ def _chat_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: "seed": _number(params.get("seed")), "frequency_penalty": _number(params.get("frequency_penalty")), "presence_penalty": _number(params.get("presence_penalty")), + "attributes": _prompt_attributes(params.get("messages")) or None, }, ) @@ -260,6 +267,8 @@ def _responses_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: "max_tokens": _number(params.get("max_output_tokens")), "frequency_penalty": _number(params.get("frequency_penalty")), "presence_penalty": _number(params.get("presence_penalty")), + "attributes": _prompt_attributes([params.get("instructions"), params.get("input")]) + or None, }, ) diff --git a/sdks/python-openrouter/tests/test_openrouter.py b/sdks/python-openrouter/tests/test_openrouter.py index 1efe3a5..01456c2 100644 --- a/sdks/python-openrouter/tests/test_openrouter.py +++ b/sdks/python-openrouter/tests/test_openrouter.py @@ -221,6 +221,73 @@ def embeddings_payload() -> dict[str, Any]: } +SUPPORT_PROMPT = telemetry_dev.ChatPrompt( + name="support/reply", + version=12, + label="production", + messages=[ + {"role": "system", "content": "You answer for the {{team}} desk."}, + {"role": "user", "content": "Where is order {{order}}?"}, + ], + model=None, + parameters={}, + tools=[], + response_format={"type": "text"}, + is_fallback=False, +) + + +@pytest.mark.parametrize( + ("api", "capture_input", "older_core"), + [ + ("chat", False, False), + ("responses instructions", True, False), + ("responses input", True, False), + ("chat", True, True), + ], +) +def test_calls_outside_with_prompt_record_the_prompt_of_compiled_messages( + make: Any, monkeypatch: pytest.MonkeyPatch, api: str, capture_input: bool, older_core: bool +) -> None: + memory = make(capture_input=capture_input) + if older_core: + monkeypatch.delattr(telemetry_dev, "prompt_attributes_for") + system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") + + def handler(_request: httpx.Request) -> httpx.Response: + return json_response(chat_payload() if api == "chat" else responses_payload()) + + client = wrap_open_router(sync_client(handler)) + if api == "chat": + client.chat.send( + model="openai/gpt-4o-mini", + messages=[ + ChatSystemMessage(role="system", content=system["content"]), + ChatUserMessage(role="user", content=user["content"]), + ], + ) + else: + in_input = api == "responses input" + client.responses.send( + model="openai/gpt-4o-mini", + input=[EasyInputMessage(role="user", content=user["content"] if in_input else "Hi")], + instructions="Be terse." if in_input else system["content"], + ) + + a = attrs(only_span(memory)) + assert a["gen_ai.request.model"] == "openai/gpt-4o-mini" + assert {key: value for key, value in a.items() if key.startswith("td.prompt.")} == ( + {} + if older_core + else { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + **({"td.prompt.variables": '{"team":"billing","order":"#4"}'} if capture_input else {}), + } + ) + + def test_chat_send_maps_messages_usage_cost_and_sampling(memory: SimpleNamespace) -> None: def handler(request: httpx.Request) -> httpx.Response: return json_response(chat_payload()) diff --git a/sdks/python/README.md b/sdks/python/README.md index b785d10..dfcd2b8 100644 --- a/sdks/python/README.md +++ b/sdks/python/README.md @@ -251,9 +251,21 @@ ran in it, or before it started in the code that started it; with none, it recor for code that has not compiled in its own task. Model calls are the spans of type `generation` (operations `chat`, `text_completion`, and `generate_content`), including the spans of the provider integrations: OpenAI, Anthropic, Amazon Bedrock, Google GenAI, LiteLLM, and OpenRouter. Spans from -other OpenTelemetry tracers do not get the attributes. A model call outside `with_prompt` does not -get the attributes, even when it sends messages from `compile()`; with input capture on, the call -still records those messages as its input. +other OpenTelemetry tracers do not get the attributes. + +Outside `with_prompt`, those provider integrations find the prompt in the request: `compile()` +keeps the texts that it makes (the last 1000, at most 2,000,000 characters), and a chat call +whose request holds one of them gets `td.prompt.name`, `td.prompt.version`, and +`td.prompt.label`, also when input capture is off. A text from a template with a variable also +gives `td.prompt.variables`, and it wins over a text from a template with none. A message of only +a variable tag, for example `{{ question }}`, is not kept, because its text is your user's input. +The text in the request must be equal to the compiled text. `with_prompt` wins over this. The +integrations do it for OpenAI chat completions and responses, Anthropic messages, Bedrock +Converse and InvokeModel, Google GenAI `generate_content`, LiteLLM completion and responses, and +OpenRouter chat and responses, but not for images, audio, video, or OpenAI Realtime. With an +older `telemetry-dev` that has no `prompt_attributes_for`, a provider integration records the +prompt only in `with_prompt`. For your own spans, `prompt_attributes_for(request)` gives the +attributes to pass as `attributes`; it gives `{}` inside `with_prompt`. ## Auto-metrics diff --git a/sdks/python/src/telemetry_dev/__init__.py b/sdks/python/src/telemetry_dev/__init__.py index 9ed7aab..b0bd718 100644 --- a/sdks/python/src/telemetry_dev/__init__.py +++ b/sdks/python/src/telemetry_dev/__init__.py @@ -27,6 +27,7 @@ TextPrompt, get_prompt, get_prompt_async, + prompt_attributes_for, with_prompt, ) from ._semconv import LogLevel, SpanType @@ -67,6 +68,7 @@ "init", "log", "observe", + "prompt_attributes_for", "propagate_attributes", "shutdown", "start_span", diff --git a/sdks/python/src/telemetry_dev/_prompts.py b/sdks/python/src/telemetry_dev/_prompts.py index 9673740..43b863e 100644 --- a/sdks/python/src/telemetry_dev/_prompts.py +++ b/sdks/python/src/telemetry_dev/_prompts.py @@ -4,11 +4,14 @@ import contextlib import copy import hashlib +import itertools import json +import os import random import re import socket import threading +import types import urllib.error import urllib.parse import urllib.request @@ -50,6 +53,10 @@ _VARIABLE = re.compile(r"\{\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*\}\}") _MODEL_CALL_OPERATIONS = frozenset({"chat", "text_completion", "generate_content"}) _PROMPT_KEY = otel_context.create_key("telemetry_dev.prompt") +_MAX_COMPILED_TEXTS = 1000 +_MAX_COMPILED_CHARACTERS = 2_000_000 +_MAX_SEARCH_DEPTH = 6 +_MAX_SEARCH_NODES = 500 class PromptModel(TypedDict): @@ -98,14 +105,15 @@ def compile( ) -> list[dict[str, Any]]: """Fill in the variables and replace each placeholder message with the list of messages in its variable. ``with_prompt`` records the variables of the last call before it starts - or inside it.""" + or inside it. Each text is also kept with its variables (the last 1000 texts), so that a + model call outside ``with_prompt`` that sends it can name the prompt; see + ``prompt_attributes_for``.""" values = _record(self, variables, kwargs) compiled: list[dict[str, Any]] = [] for message in self.messages: if message.get("role") != "placeholder": - compiled.append( - {"role": message["role"], "content": _fill(str(message["content"]), values)} - ) + content = _compile_text(self, str(message["content"]), values) + compiled.append({"role": message["role"], "content": content}) continue name = str(message["name"]) value = values.get(name) @@ -141,8 +149,10 @@ def messages(self) -> list[dict[str, Any]]: def compile(self, variables: Mapping[str, Any] | None = None, /, **kwargs: Any) -> str: """Fill in the variables. ``with_prompt`` records the variables of the last call before it - starts or inside it.""" - return _fill(self.text, _record(self, variables, kwargs)) + starts or inside it. The text is also kept with its variables (the last 1000 texts), so + that a model call outside ``with_prompt`` that sends it can name the prompt; see + ``prompt_attributes_for``.""" + return _compile_text(self, self.text, _record(self, variables, kwargs)) Prompt: TypeAlias = ChatPrompt | TextPrompt @@ -282,16 +292,45 @@ def prompt_attributes( scope = otel_context.get_value(_PROMPT_KEY, context) or otel_context.get_value(_PROMPT_KEY) if not isinstance(scope, _PromptScope): return {} - prompt = scope.prompt + task = _task_variables.get() + variables = task[1] if task is not None and task[0] is scope else scope.variables + return _attributes_of(scope.prompt, variables, client if capture_input else None) + + +def prompt_attributes_for(request: object) -> dict[str, AttributeValue]: + """The td.prompt.* attributes for a model call outside ``with_prompt``, from the texts that + ``compile()`` made: the first text in ``request`` from a template with a variable, else the + first text from a template with none. Only the first gives ``td.prompt.variables``, because + only its text holds the values. Gives ``{}`` inside ``with_prompt``, whose prompt the span + gets, and when no text matches. It reads lists, tuples, dicts, and the fields that objects + store (pydantic models), 6 levels and 500 values deep, and runs no property. It never + raises.""" + client = get_client() + try: + if isinstance(otel_context.get_value(_PROMPT_KEY), _PromptScope): + return {} + found = _compiled_texts.find(request) + if found is None: + return {} + capture = client if client is not None and client.capture_input else None + return _attributes_of(found.prompt, found.variables, capture) + except Exception as exc: + if client is not None: + client.report("prompt_attributes_for failed", exc) + return {} + + +def _attributes_of( + prompt: Prompt, variables: dict[str, Any] | None, capture: Client | None +) -> dict[str, AttributeValue]: + """The attributes of ``prompt``; ``capture`` is the client when input capture is on.""" attrs: dict[str, AttributeValue] = {ATTR_PROMPT_NAME: prompt.name} if prompt.version is not None: attrs[ATTR_PROMPT_VERSION] = prompt.version if prompt.label is not None: attrs[ATTR_PROMPT_LABEL] = prompt.label - task = _task_variables.get() - variables = task[1] if task is not None and task[0] is scope else scope.variables - if variables is not None and capture_input: - serialized = client.serialize(variables, ATTR_PROMPT_VARIABLES) + if variables is not None and capture is not None: + serialized = capture.serialize(variables, ATTR_PROMPT_VARIABLES) if serialized is not None: attrs[ATTR_PROMPT_VARIABLES] = serialized return attrs @@ -404,6 +443,137 @@ def _record( return values +@dataclass(frozen=True) +class _CompiledText: + """The prompt that ``compile()`` made a text from.""" + + prompt: Prompt + variables: dict[str, Any] | None + """Only for a template with a variable: the text holds the values, so they belong to it.""" + + +class _CompiledTexts: + """The texts of the last ``compile()`` calls, oldest first, for model calls outside + ``with_prompt``.""" + + def __init__(self) -> None: + self.texts: OrderedDict[str, _CompiledText] = OrderedDict() + self.characters = 0 + self.longest = 0 + self.lock = threading.Lock() + + def remember(self, text: str, template: str, prompt: Prompt, values: dict[str, Any]) -> None: + """Keep ``text`` as the newest entry. A template of only variable tags is skipped: its + text is input of the app, which an unrelated request can hold too.""" + fixed_text = _VARIABLE.sub("", template) + if not fixed_text.strip() or len(text) > _MAX_COMPILED_CHARACTERS: + return + compiled = _CompiledText(prompt, None if fixed_text == template else values) + with self.lock: + if self.texts.pop(text, None) is not None: + self.characters -= len(text) + self.texts[text] = compiled + self.characters += len(text) + self.longest = max(self.longest, len(text)) + dropped_longest = False + while ( + len(self.texts) > _MAX_COMPILED_TEXTS or self.characters > _MAX_COMPILED_CHARACTERS + ): + oldest, _ = self.texts.popitem(last=False) + self.characters -= len(oldest) + dropped_longest = dropped_longest or len(oldest) == self.longest + if dropped_longest: + self.longest = max(map(len, self.texts), default=0) + + def find(self, request: object) -> _CompiledText | None: + first_static: _CompiledText | None = None + budget = _MAX_SEARCH_NODES + stack: list[tuple[object, int]] = [(request, 0)] + with self.lock: + while stack and budget > 0: + budget -= 1 + value, depth = stack.pop() + if type(value) is str: + found = self.texts.get(value) if len(value) <= self.longest else None + if found is not None and found.variables is not None: + return found + first_static = first_static or found + elif depth < _MAX_SEARCH_DEPTH: + children = _stored_values(value, budget) + stack.extend((child, depth + 1) for child in reversed(children)) + return first_static + + +_compiled_texts = _CompiledTexts() + + +def _lock_compiled_texts_before_fork() -> None: + _compiled_texts.lock.acquire() + + +def _unlock_compiled_texts_after_fork() -> None: + _compiled_texts.lock.release() + + +# fork() waits for the lock: the child has only the thread that forked, so a lock that another +# thread held at the fork would stay held there for all time. +if hasattr(os, "register_at_fork"): # absent on Windows + os.register_at_fork( + before=_lock_compiled_texts_before_fork, + after_in_parent=_unlock_compiled_texts_after_fork, + after_in_child=_unlock_compiled_texts_after_fork, + ) + + +def _compile_text(prompt: Prompt, template: str, values: dict[str, Any]) -> str: + text = _fill(template, values) + _compiled_texts.remember(text, template, prompt, values) + return text + + +_TYPE_MRO = type.__dict__["__mro__"] +_TYPE_NAMESPACE = type.__dict__["__dict__"] + + +def _stored_values(value: object, limit: int) -> list[object]: + """At most ``limit`` values that ``value`` stores, in order: the items of a list or tuple, the + values of a dict, or the fields of another object in ``__dict__`` and ``__pydantic_extra__``. + Subclasses of list, tuple, and dict are skipped and fields are read from C-level storage, so + no override or property of the app runs.""" + cls = type(value) + if cls is list or cls is tuple: + return list(cast("list[object] | tuple[object, ...]", value)[:limit]) + if cls is dict: + return list(itertools.islice(cast("dict[object, object]", value).values(), limit)) + values: list[object] = [] + for name in ("__dict__", "__pydantic_extra__"): + stored = _stored(value, name) + if type(stored) is dict: + fields = cast("dict[str, object]", stored) + values.extend(itertools.islice(fields.values(), limit - len(values))) + return values + + +def _stored(value: object, name: str) -> object: + cls = type(value) + for klass in _TYPE_MRO.__get__(cls): + namespace = _TYPE_NAMESPACE.__get__(klass) + if name not in namespace: + continue + descriptor: object = namespace[name] + kind = type(descriptor) + storage = kind is types.MemberDescriptorType or ( + name == "__dict__" and kind is types.GetSetDescriptorType + ) + if not storage or cast(Any, descriptor).__name__ != name: + return None + try: + return cast(Any, descriptor).__get__(value, cls) + except (AttributeError, TypeError): + return None + return None + + def _fill(text: str, values: Mapping[str, Any]) -> str: def replace(match: re.Match[str]) -> str: value = values.get(match.group(1)) diff --git a/sdks/python/tests/test_prompts.py b/sdks/python/tests/test_prompts.py index 892d676..9fd8656 100644 --- a/sdks/python/tests/test_prompts.py +++ b/sdks/python/tests/test_prompts.py @@ -3,6 +3,9 @@ import asyncio import contextvars import itertools +import json +import os +import signal import socket import threading import time @@ -24,8 +27,10 @@ from telemetry_dev import ( ChatPrompt, PromptError, + TextPrompt, get_prompt, get_prompt_async, + prompt_attributes_for, propagate_attributes, start_span, with_prompt, @@ -76,9 +81,42 @@ def clock(monkeypatch: pytest.MonkeyPatch) -> list[float]: @pytest.fixture(autouse=True) -def empty_cache() -> None: +def empty_cache(monkeypatch: pytest.MonkeyPatch) -> None: # A fetch that runs on after its test must not answer a later test on a reused port. prompts_module._cache.clear() # pyright: ignore[reportPrivateUsage] + monkeypatch.setattr( + prompts_module, + "_compiled_texts", + prompts_module._CompiledTexts(), # pyright: ignore[reportPrivateUsage] + ) + + +def chat_prompt(*messages: dict[str, Any], name: str = "support/reply") -> ChatPrompt: + return ChatPrompt( + name=name, + version=12, + label="production", + messages=list(messages), + model=None, + parameters={}, + tools=[], + response_format={"type": "text"}, + is_fallback=False, + ) + + +def text_prompt(text: str) -> TextPrompt: + return TextPrompt( + name="support/reply", + version=12, + label="production", + text=text, + model=None, + parameters={}, + tools=[], + response_format={"type": "text"}, + is_fallback=False, + ) def query_of(path: str) -> dict[str, str]: @@ -793,6 +831,249 @@ def mask(value: Any, context: telemetry_dev.MaskContext) -> Any: assert "td.prompt.variables" not in (span.attributes or {}) +class Part: + def __init__(self, text: str) -> None: + self.text = text + + +def test_prompt_attributes_for_finds_a_compiled_text_in_the_request_shapes_of_the_providers( + make: MakeClient, +) -> None: + make() + prompt = chat_prompt( + {"role": "system", "content": "Answer as the {{team}} desk."}, + {"role": "user", "content": "Order {{order}}: where is it?"}, + ) + compiled = prompt.compile(team="billing", order="#4") + system, user = (message["content"] for message in compiled) + + requests = { + "same": compiled, + "copy": [dict(message) for message in compiled], + "json": json.loads(json.dumps(compiled)), + "anthropic": [system, compiled[1:]], + "bedrock": [[{"text": system}], [{"role": "user", "content": [{"text": user}]}]], + "objects": (Part(system), [SimpleNamespace(role="user", parts=[Part(user)])]), + } + for request in requests.values(): + assert prompt_attributes_for(request) == { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + "td.prompt.variables": '{"team":"billing","order":"#4"}', + } + + assert prompt_attributes_for([{"role": "user", "content": "Order #5: where is it?"}]) == {} + + +def test_a_static_text_gives_no_variables_and_a_template_of_only_a_variable_tag_is_not_kept( + make: MakeClient, +) -> None: + make() + chat = chat_prompt( + {"role": "system", "content": "You are the refunds agent."}, + {"role": "user", "content": "{{ question }}"}, + ) + compiled = chat.compile(question="Where is my refund?") + + assert prompt_attributes_for(compiled) == { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + } + assert prompt_attributes_for([{"role": "user", "content": "Where is my refund?"}]) == {} + + summary = text_prompt("Summarize {{doc}} in a line.").compile(doc="the refund policy") + assert prompt_attributes_for(summary)["td.prompt.variables"] == '{"doc":"the refund policy"}' + + +def test_a_text_with_variables_wins_over_a_static_text_and_the_first_one_wins_over_a_later_one( + make: MakeClient, +) -> None: + make() + system = chat_prompt( + {"role": "system", "content": "You are the refunds agent, v2."}, name="refunds/system" + ).compile() + question = chat_prompt( + {"role": "user", "content": "Where is order {{order}}?"}, name="orders/question" + ).compile(order="#4") + style = chat_prompt( + {"role": "system", "content": "Reply in {{language}}."}, name="style/language" + ).compile(language="French") + + first = prompt_attributes_for([*system, *question]) + assert (first["td.prompt.name"], first["td.prompt.variables"]) == ( + "orders/question", + '{"order":"#4"}', + ) + second = prompt_attributes_for([*system, *style, *question]) + assert (second["td.prompt.name"], second["td.prompt.variables"]) == ( + "style/language", + '{"language":"French"}', + ) + + +def test_each_compile_keeps_its_own_values_so_two_threads_with_one_prompt_get_their_own( + make: MakeClient, +) -> None: + make() + prompt = chat_prompt({"role": "user", "content": "Track parcel {{parcel}}."}) + + first = prompt.compile(parcel="A-1") + second = prompt.compile(parcel="B-2") + + assert prompt_attributes_for(first)["td.prompt.variables"] == '{"parcel":"A-1"}' + assert prompt_attributes_for(second)["td.prompt.variables"] == '{"parcel":"B-2"}' + + +def test_with_prompt_wins_over_a_compiled_text_of_another_prompt(make: MakeClient) -> None: + memory = make() + scoped = chat_prompt({"role": "user", "content": "Hi"}, name="scoped/prompt") + compiled = chat_prompt( + {"role": "user", "content": "Cancel order {{order}}."}, name="other/prompt" + ).compile(order="#9") + + with with_prompt(scoped): + start_span("chat", type="generation", attributes=prompt_attributes_for(compiled)).end() + + [span] = memory.span_exporter.get_finished_spans() + assert (span.attributes or {})["td.prompt.name"] == "scoped/prompt" + + +def test_prompt_attributes_for_omits_variables_when_input_capture_is_off_and_masks_them( + make: MakeClient, +) -> None: + compiled = chat_prompt({"role": "user", "content": "My card is {{card}}."}).compile( + card="4111 1111" + ) + + make(capture_input=False) + assert prompt_attributes_for(compiled) == { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + } + + telemetry_dev.shutdown() + + def mask(value: Any, context: telemetry_dev.MaskContext) -> Any: + return {"card": "[masked]"} if context.key == "td.prompt.variables" else value + + make(mask=mask) + assert prompt_attributes_for(compiled)["td.prompt.variables"] == '{"card":"[masked]"}' + + +def test_the_kept_texts_stop_at_1000_and_at_2000000_characters_oldest_out_first( + make: MakeClient, +) -> None: + make() + ticket = text_prompt("Ticket {{n}} is open.") + for n in range(1001): + ticket.compile(n=n) + + assert prompt_attributes_for("Ticket 0 is open.") == {} + assert prompt_attributes_for("Ticket 1 is open.")["td.prompt.variables"] == '{"n":1}' + + note = text_prompt("Note: {{body}}") + notes = [note.compile(body=letter * (500_000 - len("Note: "))) for letter in "abcde"] + + assert prompt_attributes_for(notes[0]) == {} + assert prompt_attributes_for(notes[1])["td.prompt.name"] == "support/reply" + + # A text over the character limit is not kept, and it does not push the others out. + note.compile(body="f" * 2_000_001) + assert prompt_attributes_for(notes[1])["td.prompt.name"] == "support/reply" + + # A text compiled again counts once. + for _ in range(5): + note.compile(body="e" * (500_000 - len("Note: "))) + assert prompt_attributes_for(notes[4])["td.prompt.name"] == "support/reply" + + +def test_prompt_attributes_for_looks_6_levels_and_500_values_deep(make: MakeClient) -> None: + make() + [message] = chat_prompt({"role": "user", "content": "Find invoice {{invoice}}."}).compile( + invoice="INV-7" + ) + content = message["content"] + + def nest(value: object, levels: int) -> object: + return value if levels == 0 else nest([value], levels - 1) + + assert prompt_attributes_for(nest(content, 6)) != {} + assert prompt_attributes_for(nest(content, 7)) == {} + + # The request list, 498 strings, and the text are 500 values. + assert prompt_attributes_for([*["filler"] * 498, content]) != {} + assert prompt_attributes_for([*["filler"] * 499, content]) == {} + + +def test_prompt_attributes_for_runs_no_property_or_override_of_the_app(make: MakeClient) -> None: + make() + [message] = chat_prompt({"role": "user", "content": "Refund order {{order}}."}).compile( + order="#3" + ) + content = message["content"] + reads: list[str] = [] + + class Message: + @property + def content(self) -> str: + reads.append("property") + return content + + class Messages(list[object]): + def __getitem__(self, index: Any) -> Any: + reads.append("__getitem__") + return super().__getitem__(index) + + def __iter__(self) -> Any: + reads.append("__iter__") + return super().__iter__() + + assert prompt_attributes_for([Message(), Messages([content])]) == {} + assert reads == [] + + +@pytest.mark.skipif(not hasattr(os, "fork"), reason="needs os.fork") +# Python 3.12+ warns on fork() in a process with threads. +@pytest.mark.filterwarnings("ignore:This process .* is multi-threaded:DeprecationWarning") +def test_a_fork_while_a_thread_keeps_a_compiled_text_leaves_compile_usable_in_the_child() -> None: + prompt = text_prompt("Ship order {{order}}.") + texts = prompts_module._compiled_texts # pyright: ignore[reportPrivateUsage] + holding, release = threading.Event(), threading.Event() + + def hold() -> None: + with texts.lock: + holding.set() + release.wait(0.2) + + holder = threading.Thread(target=hold) + holder.start() + holding.wait() + read_end, write_end = os.pipe() + pid = os.fork() + if pid == 0: + try: + os.close(read_end) + os.write(write_end, prompt.compile(order="#1").encode()) + finally: + os._exit(0) + os.close(write_end) + release.set() + holder.join() + + deadline = time.monotonic() + 5 + while os.waitpid(pid, os.WNOHANG) == (0, 0): + if time.monotonic() > deadline: + os.kill(pid, signal.SIGKILL) + os.waitpid(pid, 0) + pytest.fail("compile() in the child waited for a lock held at the fork") + time.sleep(0.01) + with os.fdopen(read_end, "rb") as child: + assert child.read() == b"Ship order #1." + + def test_a_fallback_prompt_records_its_name_and_label_but_no_version( server: CaptureServer, make: MakeClient ) -> None: From ae47605555ef9f5dfed1d6f762ac56823e692edb Mon Sep 17 00:00:00 2001 From: ephraimduncan Date: Thu, 29 Oct 2026 15:31:55 +0000 Subject: [PATCH 2/9] fix(sdk): find the prompt by message identity, not by text The compile() function of a chat prompt tags each message object that it makes with the prompt and the variables of that call. TypeScript uses a non-enumerable symbol property, and Python uses a dict subclass with a slot. The keys and the JSON of a message do not change. The promptAttributesFor and prompt_attributes_for functions read the tag of the newest message of the last 1000, and they compare no text. Thus a message from a different task, rollout version, turn, or call does not get the wrong prompt, and a copy gets no prompt. We removed the text registry, its limits, its lock, and its fork hooks. As a result, compile() keeps no variables alive. The integrations give only the request messages to the core: OpenAI messages and input, Anthropic messages, Bedrock Converse messages, Google GenAI contents, LiteLLM messages and input, and OpenRouter messages and input. Bedrock InvokeModel sends a serialized body, thus it does not look for a prompt. Python Anthropic reads the list that the app gave, not its own copy. Its messages.stream() decides at __enter__, where its span starts. A span with input capture off removes td.prompt.variables from the attributes that the app gives. The SDKs do not record text prompts outside withPrompt and with_prompt. --- docs/sdk-conformance.md | 54 +-- packages/anthropic/src/index.ts | 8 +- packages/anthropic/tests/anthropic.test.ts | 17 +- packages/bedrock/src/converse.ts | 2 +- packages/bedrock/src/internal.ts | 6 +- packages/bedrock/src/invoke-model.ts | 5 +- packages/bedrock/tests/bedrock.test.ts | 104 +++--- packages/google-genai/src/index.ts | 8 +- .../google-genai/tests/google-genai.test.ts | 42 +-- packages/openai/src/index.ts | 8 +- packages/openai/tests/openai.test.ts | 31 +- packages/openrouter/src/index.ts | 8 +- packages/openrouter/tests/openrouter.test.ts | 36 +- packages/sdk/README.md | 26 +- packages/sdk/src/attrs.ts | 3 + packages/sdk/src/prompts.ts | 150 +++----- packages/sdk/tests/prompts.test.ts | 272 +++++++-------- .../src/telemetry_dev_anthropic/__init__.py | 55 ++- sdks/python-anthropic/tests/test_anthropic.py | 84 ++++- .../src/telemetry_dev_bedrock/_fields.py | 4 +- .../telemetry_dev_bedrock/_invoke_model.py | 4 +- .../src/telemetry_dev_bedrock/_registry.py | 2 +- sdks/python-bedrock/tests/test_bedrock.py | 104 +++--- .../telemetry_dev_google_genai/__init__.py | 6 +- .../tests/test_google_genai.py | 43 ++- .../src/telemetry_dev_litellm/__init__.py | 6 +- sdks/python-litellm/tests/test_litellm.py | 33 +- .../src/telemetry_dev_openai/__init__.py | 6 +- sdks/python-openai/tests/test_openai.py | 38 +- .../src/telemetry_dev_openrouter/__init__.py | 7 +- .../tests/test_openrouter.py | 37 +- sdks/python/README.md | 30 +- sdks/python/src/telemetry_dev/_prompts.py | 189 +++------- sdks/python/src/telemetry_dev/_spans.py | 4 + sdks/python/tests/test_prompts.py | 328 +++++++++--------- 35 files changed, 822 insertions(+), 938 deletions(-) diff --git a/docs/sdk-conformance.md b/docs/sdk-conformance.md index efe4a7a..610c0c1 100644 --- a/docs/sdk-conformance.md +++ b/docs/sdk-conformance.md @@ -41,10 +41,10 @@ repository, not this SDK workspace. Paths below refer to that product repository | error | span status ERROR + `error.type` + `exception` event (`exception.type/message/stacktrace`, `log.severity_number: 17`) | | | `log()` | OTLP log record to `/v1/logs` (always; never span events) | severity debug=5/info=9/warn=13/error=17; trace-correlated via active context | | `withPrompt(prompt, fn)` / `with with_prompt(prompt):` → `prompt.name` | `td.prompt.name` | on model-call spans that start in scope: operation `chat`, `text_completion`, or `generate_content` | -| outside a scope: a text that `prompt.compile()` made, in the request of a provider integration | `td.prompt.name`, `td.prompt.version`, `td.prompt.label` | on the `chat` spans of the provider integrations listed in [Prompts](#prompts); the scope wins | +| outside a scope: a message object that `compile()` of a chat prompt returned, in the request messages of a provider integration | `td.prompt.name`, `td.prompt.version`, `td.prompt.label` | on the `chat` spans of the provider integrations listed in [Prompts](#prompts); the scope wins | | `prompt.version` | `td.prompt.version` (int) | omitted for a fallback prompt (`version` null / None) | | `prompt.label` | `td.prompt.label` | only for a fetch by label; a fetch without label or version is the `production` label | -| variables of the last `prompt.compile()` before `withPrompt` / `with_prompt` starts, or in its scope; outside a scope, of the `compile()` that made the text, when its template has a variable | `td.prompt.variables` (JSON object) | mask → JSON stringify → truncate; gated by the effective captureInput of the span | +| variables of the last `prompt.compile()` before `withPrompt` / `with_prompt` starts, or in its scope; outside a scope, of the `compile()` that returned the message | `td.prompt.variables` (JSON object) | mask → JSON stringify → truncate; gated by the effective captureInput of the span | `SpanHandle.serializeContent(value, key)` / `SpanHandle.serialize_content(value, key)` returns what `input` / `output` would emit for `value` under `key`: mask → JSON stringify → truncate. It @@ -179,27 +179,39 @@ cannot tell a concurrent task from an awaited helper, so in TypeScript all spans its last `compile()`; concurrent tasks that compile their own variables call `withPrompt` in each task. -Outside a scope, the SDKs find the prompt from the text of the request: - -- `compile()` keeps each text that it makes (each message of a chat prompt, the text of a text - prompt), with the prompt and the variables. It keeps at most 1000 texts and 2,000,000 - characters, and drops the oldest first. A text that `compile()` makes again becomes the newest. - It does not keep a text whose template has only variable tags, for example `{{ question }}`: - that text is input of the app, and an unrelated request can hold it too. -- `promptAttributesFor(request)` / `prompt_attributes_for(request)` look for a kept text in - `request`: strings in arrays and plain objects (Python: `list`, `tuple`, and `dict` but not their - subclasses, and the fields that an object stores in `__dict__` or `__pydantic_extra__`), 6 - levels deep and 500 values in all. They run no getter or property. A string matches only when it is equal to a kept text. -- The first text from a template with a variable gives the name, version, label, and variables - of its `compile()`. Without one, the first text from a template with no variable gives the - name, version, and label, but no variables: the same static text comes from each `compile()`. +Outside a scope, the SDKs find the prompt from the message objects of the request, not from +their text: + +- `compile()` of a chat prompt tags each message object that it makes with the prompt and the + variables of that call. It does not tag the messages of a placeholder variable: the app passes + those. TypeScript: a non-enumerable symbol property. Python: the messages are instances of a + `dict` subclass that holds the tag in a slot. The keys and the JSON of a message do not change. +- A copy has no tag: a spread, `structuredClone`, or a JSON round trip (TypeScript); `dict()`, + `.copy()`, `copy.copy`, `copy.deepcopy`, pickle, or a JSON round trip (Python). A new message + with the same text has no tag either, thus a message from another call, task, or prompt version + is never recorded. +- `promptAttributesFor(messages)` / `prompt_attributes_for(messages)` take a list of messages or + one message (Python: a `list` or `tuple`, not a subclass, so no code of the app runs). They + look at the last 1000 items, from the end, and give the name, version, label, and variables of + the newest tagged message. They do not look inside a message. - In a `withPrompt` / `with_prompt` scope they give nothing, thus the span gets the prompt of the scope. They never throw. -- The provider integrations call them with the request of the app before the span starts, also - when input capture is off: OpenAI chat completions and responses, Anthropic messages, Amazon - Bedrock Converse and InvokeModel (not embedding models), Google GenAI `generateContent`, - LiteLLM completion and responses (Python), and OpenRouter chat and responses. The other spans - of the integrations (images, audio, video, and OpenAI Realtime) do not look in the request. +- They give `td.prompt.variables` only when input capture is on in the client. A span whose + effective captureInput is off drops `td.prompt.variables` from its attributes, also when the app + passes the result of these functions to its own span. +- `compile()` of a text prompt returns a string, which cannot carry a tag. Outside a scope, the + SDKs do not record a text prompt: use `withPrompt` / `with_prompt`. +- The provider integrations call them with the request messages before the span starts, also + when input capture is off: OpenAI chat completions (`messages`) and responses (`input`), + Anthropic messages (`messages`; the `system` string has no tag), Amazon Bedrock Converse + (`messages`), Google GenAI `generateContent` (`contents`), LiteLLM completion (`messages`) and + responses (`input`) (Python), and OpenRouter chat (`messages`) and responses (`input`). Bedrock + Converse and Google GenAI take another message shape: a compiled message that the app changes + in place keeps its tag, and a new message does not. Bedrock InvokeModel takes a serialized body + and is not recorded outside a scope. The other spans of the integrations (embeddings, images, + audio, video, and OpenAI Realtime) do not look at the request. +- Python Anthropic `messages.stream()` starts its span at `__enter__`. It decides there, with the + messages that `.stream()` received, thus a scope around `__enter__` wins. - A provider integration reads the function from the core module when it runs, thus it also operates with an older core that does not have it. With such a core, the prompt is recorded only in a scope. diff --git a/packages/anthropic/src/index.ts b/packages/anthropic/src/index.ts index 4e5b057..1082844 100644 --- a/packages/anthropic/src/index.ts +++ b/packages/anthropic/src/index.ts @@ -150,12 +150,12 @@ function toolDefinitions(tools: T): Record | undefined { // Cores older than promptAttributesFor record a prompt only inside withPrompt. A namespace read // keeps them loadable without the export. -function promptAttributes(request: unknown): SpanFields["attributes"] { +function promptAttributes(messages: unknown): SpanFields["attributes"] { const read = ( - telemetrySdk as { promptAttributesFor?: (request: unknown) => SpanFields["attributes"] } + telemetrySdk as { promptAttributesFor?: (messages: unknown) => SpanFields["attributes"] } ).promptAttributesFor; - return typeof read === "function" ? read(request) : undefined; + return typeof read === "function" ? read(messages) : undefined; } function messagesRequest(body: ValueRecord): RequestMapping { @@ -174,7 +174,7 @@ function messagesRequest(body: ValueRecord): RequestMapping { input, attributes: { ...toolDefinitions(body.tools), - ...promptAttributes([body.system, body.messages]), + ...promptAttributes(body.messages), }, systemInstructions: body.system, temperature: readNumber(body.temperature), diff --git a/packages/anthropic/tests/anthropic.test.ts b/packages/anthropic/tests/anthropic.test.ts index c8cab08..12eeae9 100644 --- a/packages/anthropic/tests/anthropic.test.ts +++ b/packages/anthropic/tests/anthropic.test.ts @@ -282,12 +282,12 @@ afterEach(async () => { }); test.each([ - { field: "system", olderCore: false }, - { field: "messages", olderCore: false }, - { field: "messages", olderCore: true }, + { messages: "compiled", olderCore: false }, + { messages: "copied", olderCore: false }, + { messages: "compiled", olderCore: true }, ])( - "messages.create outside withPrompt records the prompt of a compiled text in $field, older core $olderCore", - async ({ field, olderCore }) => { + "messages.create outside withPrompt records the prompt of $messages messages, older core $olderCore", + async ({ messages: kind, olderCore }) => { if (olderCore) { vi.spyOn( telemetrySdk as { promptAttributesFor: unknown }, @@ -318,20 +318,21 @@ test.each([ }), }); const [system, user] = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const message = user as { role: "user"; content: string }; const spans = setupSpans(); const client = clientWith(createFakeFetch(jsonResponse(messagePayload())).fetch); await client.messages.create({ model: "claude-sonnet-4-6", max_tokens: 64, - system: field === "system" ? system!.content : "Be terse.", - messages: [{ role: "user", content: field === "messages" ? user!.content : "Hi" }], + system: system!.content, + messages: [kind === "compiled" ? message : { ...message }], }); const { attributes } = await exportedSpan(spans); expect(attributes["gen_ai.input.messages"]).toBeDefined(); - if (olderCore) { + if (olderCore || kind === "copied") { expect(attributes["td.prompt.name"]).toBeUndefined(); return; diff --git a/packages/bedrock/src/converse.ts b/packages/bedrock/src/converse.ts index e552c6d..fd9c8e1 100644 --- a/packages/bedrock/src/converse.ts +++ b/packages/bedrock/src/converse.ts @@ -35,7 +35,7 @@ export function converseRequestFields(input: JsonRecord): SpanFields { model: stringValue(input.modelId), input: normalizeMessages(input.messages), systemInstructions: normalizeSystem(input.system), - attributes: promptAttributes([input.system, input.messages]), + attributes: promptAttributes(input.messages), temperature: numberValue(inference.temperature), topP: numberValue(inference.topP), maxTokens: numberValue(inference.maxTokens), diff --git a/packages/bedrock/src/internal.ts b/packages/bedrock/src/internal.ts index 7d82cc8..2a89e01 100644 --- a/packages/bedrock/src/internal.ts +++ b/packages/bedrock/src/internal.ts @@ -194,12 +194,12 @@ export function captureMaskedBeforeStart(): boolean { // Cores older than promptAttributesFor record a prompt only inside withPrompt. A namespace read // keeps them loadable without the export. -export function promptAttributes(request: unknown): SpanFields["attributes"] { +export function promptAttributes(messages: unknown): SpanFields["attributes"] { const read = ( - telemetrySdk as { promptAttributesFor?: (request: unknown) => SpanFields["attributes"] } + telemetrySdk as { promptAttributesFor?: (messages: unknown) => SpanFields["attributes"] } ).promptAttributesFor; - return typeof read === "function" ? read(request) : undefined; + return typeof read === "function" ? read(messages) : undefined; } // The decision is made before the span starts so samplers see the same attributes as before. diff --git a/packages/bedrock/src/invoke-model.ts b/packages/bedrock/src/invoke-model.ts index d2532d2..d1e1f11 100644 --- a/packages/bedrock/src/invoke-model.ts +++ b/packages/bedrock/src/invoke-model.ts @@ -9,7 +9,6 @@ import { numberValue, omitUndefined, parseJson, - promptAttributes, PROVIDER, STREAM_CAPTURE_MAX_ITEMS, stringValue, @@ -36,14 +35,12 @@ export function invokeModelSpanName(input: JsonRecord): string { export function invokeModelRequestFields(input: JsonRecord): SpanFields { const model = stringValue(input.modelId); const body = parseBody(input.body, input.contentType); - const embedding = isEmbeddingModel(model); return omitUndefined({ provider: PROVIDER, model, input: body, - outputType: embedding ? "embedding" : undefined, - attributes: embedding ? undefined : promptAttributes(body), + outputType: isEmbeddingModel(model) ? "embedding" : undefined, ...nativeSamplingFields(model, body), }); } diff --git a/packages/bedrock/tests/bedrock.test.ts b/packages/bedrock/tests/bedrock.test.ts index d12bf19..6098bea 100644 --- a/packages/bedrock/tests/bedrock.test.ts +++ b/packages/bedrock/tests/bedrock.test.ts @@ -424,10 +424,15 @@ test("ConverseStream records stream errors with partial output", async () => { expect(span.attributes["telemetry.dev.capture.truncated"]).toBe(true); }); -test.each([false, true])( - "Converse and InvokeModel outside withPrompt record the prompt of a compiled text; embeddings do not; older core %s", - async (olderCore) => { +test.each([ + { messages: "changed in place", olderCore: false }, + { messages: "new", olderCore: false }, + { messages: "changed in place", olderCore: true }, +])( + "Converse outside withPrompt records the prompt of $messages messages, older core $olderCore", + async ({ messages: kind, olderCore }) => { const spans = setup(); + if (olderCore) { vi.spyOn( telemetrySdk as { promptAttributesFor: unknown }, @@ -435,6 +440,7 @@ test.each([false, true])( "get", ).mockReturnValue(undefined); } + const prompt = await getPrompt("support/reply", { apiKey: "td_live_a", apiUrl: "https://prompts.bedrock.test", @@ -457,78 +463,48 @@ test.each([false, true])( }), }); const [system, user] = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; - const converseReply = { - output: { message: { role: "assistant", content: [{ text: "Soon." }] } }, - stopReason: "end_turn", - }; + const blocks = [{ text: user!.content }]; + // Converse takes content blocks: a message changed in place keeps its tag, a new one has none. + const message = + kind === "new" + ? { role: "user", content: blocks } + : Object.assign(user!, { content: blocks }); const client = wrapBedrock( - new FakeClient([ - converseReply, - converseReply, - { - body: bytes({ content: [{ type: "text", text: "Soon." }], stop_reason: "end_turn" }), - contentType: "application/json", - }, - { - body: bytes({ embedding: [0.1, 0.2], inputTextTokenCount: 6 }), - contentType: "application/json", - }, - ]), + new BedrockRuntimeClient({ + region: "us-east-1", + credentials: { accessKeyId: "test", secretAccessKey: "test" }, + maxAttempts: 1, + requestHandler: jsonRequestHandler("prompt-1", { + output: { message: { role: "assistant", content: [{ text: "Soon." }] } }, + stopReason: "end_turn", + usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 }, + }), + }), ); await client.send( new ConverseCommand({ modelId: "anthropic.claude-3-haiku", system: [{ text: system!.content }], - messages: [{ role: "user", content: [{ text: "Hi" }] }], - }), - ); - await client.send( - new ConverseCommand({ - modelId: "anthropic.claude-3-haiku", - messages: [{ role: "user", content: [{ text: user!.content }] }], - }), - ); - await client.send( - new InvokeModelCommand({ - modelId: "anthropic.claude-3-haiku", - contentType: "application/json", - body: bytes({ - anthropic_version: "bedrock-2023-05-31", - max_tokens: 64, - messages: [{ role: "user", content: user!.content }], - }), - }), - ); - await client.send( - new InvokeModelCommand({ - modelId: "amazon.titan-embed-text-v2:0", - contentType: "application/json", - body: bytes({ inputText: user!.content }), + messages: [message] as ConverseCommandInput["messages"], }), ); - const [systemSpan, messagesSpan, invokeSpan, embedSpan] = spans.getFinishedSpans(); - - for (const span of [systemSpan, messagesSpan, invokeSpan]) { - expect(span!.attributes["gen_ai.request.model"]).toBe("anthropic.claude-3-haiku"); - expect( - Object.entries(span!.attributes).filter(([key]) => key.startsWith("td.prompt.")), - ).toEqual( - olderCore - ? [] - : [ - ["td.prompt.name", "support/reply"], - ["td.prompt.version", 12], - ["td.prompt.label", "production"], - ["td.prompt.variables", '{"team":"billing","order":"#4"}'], - ], - ); - } - - expect(embedSpan!.attributes["gen_ai.operation.name"]).toBe("embeddings"); - expect(embedSpan!.attributes["td.prompt.name"]).toBeUndefined(); + const [span] = spans.getFinishedSpans(); + expect(span!.attributes["gen_ai.response.id"]).toBe("prompt-1"); + expect( + Object.entries(span!.attributes).filter(([key]) => key.startsWith("td.prompt.")), + ).toEqual( + olderCore || kind === "new" + ? [] + : [ + ["td.prompt.name", "support/reply"], + ["td.prompt.version", 12], + ["td.prompt.label", "production"], + ["td.prompt.variables", '{"team":"billing","order":"#4"}'], + ], + ); }, ); diff --git a/packages/google-genai/src/index.ts b/packages/google-genai/src/index.ts index 2272e46..341d5e8 100644 --- a/packages/google-genai/src/index.ts +++ b/packages/google-genai/src/index.ts @@ -570,7 +570,7 @@ function generateRequestFields(params: UnknownRecord): RequestMapping { const attributes = { ...requestAttributesFromConfig(config), - ...promptAttributes([config?.systemInstruction, params.contents]), + ...promptAttributes(params.contents), }; const input = captureEnabled("input") ? normalizeContentInput(params.contents) : undefined; @@ -946,12 +946,12 @@ function captureMaskedBeforeStart(): boolean { // Cores older than promptAttributesFor record a prompt only inside withPrompt. A namespace read // keeps them loadable without the export. -function promptAttributes(request: unknown): SpanFields["attributes"] { +function promptAttributes(messages: unknown): SpanFields["attributes"] { const read = ( - telemetrySdk as { promptAttributesFor?: (request: unknown) => SpanFields["attributes"] } + telemetrySdk as { promptAttributesFor?: (messages: unknown) => SpanFields["attributes"] } ).promptAttributesFor; - return typeof read === "function" ? read(request) : undefined; + return typeof read === "function" ? read(messages) : undefined; } // The decision is made before the span starts so samplers see the same attributes as before. diff --git a/packages/google-genai/tests/google-genai.test.ts b/packages/google-genai/tests/google-genai.test.ts index e7417ae..e7d5ec6 100644 --- a/packages/google-genai/tests/google-genai.test.ts +++ b/packages/google-genai/tests/google-genai.test.ts @@ -236,12 +236,13 @@ afterEach(async () => { }); test.each([ - { field: "systemInstruction", captureInput: true, olderCore: false }, - { field: "contents", captureInput: false, olderCore: false }, - { field: "contents", captureInput: true, olderCore: true }, + { contents: "changed in place", captureInput: true, olderCore: false }, + { contents: "changed in place", captureInput: false, olderCore: false }, + { contents: "new", captureInput: true, olderCore: false }, + { contents: "changed in place", captureInput: true, olderCore: true }, ])( - "generateContent outside withPrompt records the prompt of a compiled text in $field, input capture $captureInput, older core $olderCore", - async ({ field, captureInput, olderCore }) => { + "generateContent outside withPrompt records the prompt of $contents contents, input capture $captureInput, older core $olderCore", + async ({ contents: kind, captureInput, olderCore }) => { if (olderCore) { vi.spyOn( telemetrySdk as { promptAttributesFor: unknown }, @@ -272,31 +273,32 @@ test.each([ }), }); const [system, user] = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const parts = [{ text: user!.content }]; + // Gemini takes parts: a message changed in place keeps its tag, a new one has none. + const content = kind === "new" ? { role: "user", parts } : Object.assign(user!, { parts }); const spans = setupSpans({ captureInput }); - vi.stubGlobal( - "fetch", - createFakeFetch( - jsonResponse({ - candidates: [ - { content: { role: "model", parts: [{ text: "Soon." }] }, finishReason: "STOP" }, - ], - }), - ).fetch, + const fake = createFakeFetch( + jsonResponse({ + candidates: [ + { content: { role: "model", parts: [{ text: "Soon." }] }, finishReason: "STOP" }, + ], + }), ); + vi.stubGlobal("fetch", fake.fetch); await clientWith().models.generateContent({ model: "gemini-2.5-flash", - contents: [{ role: "user", parts: [{ text: field === "contents" ? user!.content : "Hi" }] }], - config: { - temperature: 0.2, - systemInstruction: field === "systemInstruction" ? system!.content : "Be terse.", - }, + contents: [content], + config: { temperature: 0.2, systemInstruction: system!.content }, }); const { attributes } = await exportedSpan(spans); expect(attributes["gen_ai.request.temperature"]).toBe(0.2); + expect(fake.requests[0]!.body).toMatchObject({ + contents: [{ role: "user", parts: [{ text: "Where is order #4?" }] }], + }); - if (olderCore) { + if (olderCore || kind === "new") { expect(attributes["td.prompt.name"]).toBeUndefined(); return; diff --git a/packages/openai/src/index.ts b/packages/openai/src/index.ts index c62671c..9b86e73 100644 --- a/packages/openai/src/index.ts +++ b/packages/openai/src/index.ts @@ -246,7 +246,7 @@ function chatResponse(response: T): SpanFields { function responsesRequest(body: JsonRecord): RequestMapping { const model = readString(body.model); const capture = captureEnabled("input") ? sanitizeResponsesCapture(body.input) : undefined; - const prompt = promptAttributes([body.instructions, body.input]); + const prompt = promptAttributes(body.input); return { name: `chat ${model ?? "unknown"}`, @@ -1634,12 +1634,12 @@ function captureMaskedBeforeStart(): boolean { // Cores older than promptAttributesFor record a prompt only inside withPrompt. A namespace read // keeps them loadable without the export. -function promptAttributes(request: unknown): SpanFields["attributes"] { +function promptAttributes(messages: unknown): SpanFields["attributes"] { const read = ( - telemetrySdk as { promptAttributesFor?: (request: unknown) => SpanFields["attributes"] } + telemetrySdk as { promptAttributesFor?: (messages: unknown) => SpanFields["attributes"] } ).promptAttributesFor; - return typeof read === "function" ? read(request) : undefined; + return typeof read === "function" ? read(messages) : undefined; } // The decision is made before the span starts so samplers see the same attributes as before. diff --git a/packages/openai/tests/openai.test.ts b/packages/openai/tests/openai.test.ts index 06518ee..4437d75 100644 --- a/packages/openai/tests/openai.test.ts +++ b/packages/openai/tests/openai.test.ts @@ -2895,13 +2895,13 @@ test("chat streams preserve caller-requested usage chunks", async () => { }); test.each([ - { api: "chat completions", captureInput: false, olderCore: false }, - { api: "responses instructions", captureInput: true, olderCore: false }, - { api: "responses input", captureInput: true, olderCore: false }, - { api: "chat completions", captureInput: true, olderCore: true }, + { api: "chat completions", messages: "compiled", captureInput: false, olderCore: false }, + { api: "responses", messages: "compiled", captureInput: true, olderCore: false }, + { api: "chat completions", messages: "copied", captureInput: true, olderCore: false }, + { api: "chat completions", messages: "compiled", captureInput: true, olderCore: true }, ])( - "$api outside withPrompt record the prompt of compiled messages, input capture $captureInput, older core $olderCore", - async ({ api, captureInput, olderCore }) => { + "$api outside withPrompt records the prompt of $messages messages, input capture $captureInput, older core $olderCore", + async ({ api, messages: kind, captureInput, olderCore }) => { if (olderCore) { vi.spyOn( telemetrySdk as { promptAttributesFor: unknown }, @@ -2931,12 +2931,13 @@ test.each([ rollout: null, }), }); - const [system, user] = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const compiled = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const messages = kind === "compiled" ? compiled : compiled.map((message) => ({ ...message })); const spans = setupSpans(undefined, { captureInput }); const client = clientWith( createFakeFetch( jsonResponse( - api.startsWith("responses") + api === "responses" ? { id: "resp_1", object: "response", @@ -2963,22 +2964,16 @@ test.each([ ).fetch, ); - if (api.startsWith("responses")) { - const inInput = api === "responses input"; - - await client.responses.create({ - model: "gpt-4.1", - instructions: inInput ? "Be terse." : system!.content, - input: [{ role: "user", content: inInput ? user!.content : "Hi" }], - }); + if (api === "responses") { + await client.responses.create({ model: "gpt-4.1", input: messages }); } else { - await client.chat.completions.create({ model: "gpt-4o", messages: [system!, user!] }); + await client.chat.completions.create({ model: "gpt-4o", messages }); } const { attributes } = await exportedSpan(spans); expect(attributes["gen_ai.input.messages"] !== undefined).toBe(captureInput); - if (olderCore) { + if (olderCore || kind === "copied") { expect(attributes["td.prompt.name"]).toBeUndefined(); return; diff --git a/packages/openrouter/src/index.ts b/packages/openrouter/src/index.ts index e0d696e..74e2c76 100644 --- a/packages/openrouter/src/index.ts +++ b/packages/openrouter/src/index.ts @@ -387,12 +387,12 @@ function usageCost(usage: T): number | undefined { // Cores older than promptAttributesFor record a prompt only inside withPrompt. A namespace read // keeps them loadable without the export. -function promptAttributes(request: unknown): SpanFields["attributes"] { +function promptAttributes(messages: unknown): SpanFields["attributes"] { const read = ( - telemetrySdk as { promptAttributesFor?: (request: unknown) => SpanFields["attributes"] } + telemetrySdk as { promptAttributesFor?: (messages: unknown) => SpanFields["attributes"] } ).promptAttributesFor; - return typeof read === "function" ? read(request) : undefined; + return typeof read === "function" ? read(messages) : undefined; } function chatRequest(request: JsonRecord): RequestMapping { @@ -484,7 +484,7 @@ function responsesRequest(request: JsonRecord): RequestMapping { maxTokens: readNumber(body.maxOutputTokens), frequencyPenalty: readNumber(body.frequencyPenalty), presencePenalty: readNumber(body.presencePenalty), - attributes: promptAttributes([body.instructions, body.input]), + attributes: promptAttributes(body.input), }, }; } diff --git a/packages/openrouter/tests/openrouter.test.ts b/packages/openrouter/tests/openrouter.test.ts index dc67157..3f5cb68 100644 --- a/packages/openrouter/tests/openrouter.test.ts +++ b/packages/openrouter/tests/openrouter.test.ts @@ -312,13 +312,13 @@ afterEach(async () => { }); test.each([ - { api: "chat", olderCore: false }, - { api: "streamed responses instructions", olderCore: false }, - { api: "responses input", olderCore: false }, - { api: "chat", olderCore: true }, + { api: "chat", messages: "compiled", olderCore: false }, + { api: "streamed responses", messages: "compiled", olderCore: false }, + { api: "chat", messages: "copied", olderCore: false }, + { api: "chat", messages: "compiled", olderCore: true }, ])( - "$api outside withPrompt record the prompt of compiled messages, older core $olderCore", - async ({ api, olderCore }) => { + "$api outside withPrompt records the prompt of $messages messages, older core $olderCore", + async ({ api, messages: kind, olderCore }) => { if (olderCore) { vi.spyOn( telemetrySdk as { promptAttributesFor: unknown }, @@ -348,22 +348,13 @@ test.each([ rollout: null, }), }); - const [system, user] = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const compiled = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const messages = kind === "compiled" ? compiled : compiled.map((message) => ({ ...message })); const spans = setupSpans(); if (api === "chat") { const client = clientWith(createFakeFetcher(jsonResponse(chatBody("chat_1"))).fetcher); - await client.chat.send({ - chatRequest: { model: "openai/gpt-4o", messages: [system!, user!] }, - }); - } else if (api === "responses input") { - const client = clientWith(createFakeFetcher(jsonResponse(responsesBody("resp_1"))).fetcher); - await client.responses.send({ - responsesRequest: { - model: "openai/gpt-4o", - input: [{ type: "message", role: "user", content: user!.content }], - }, - }); + await client.chat.send({ chatRequest: { model: "openai/gpt-4o", messages } }); } else { const client = clientWith( createFakeFetcher( @@ -371,12 +362,7 @@ test.each([ ).fetcher, ); const stream = await client.responses.send({ - responsesRequest: { - model: "openai/gpt-4o", - instructions: system!.content, - input: "Hi", - stream: true, - }, + responsesRequest: { model: "openai/gpt-4o", input: messages, stream: true }, }); if (!(stream instanceof ReadableStream)) throw new Error("expected a readable stream"); @@ -386,7 +372,7 @@ test.each([ const { attributes } = await exportedSpan(spans); expect(attributes["gen_ai.request.model"]).toBe("openai/gpt-4o"); - if (olderCore) { + if (olderCore || kind === "copied") { expect(attributes["td.prompt.name"]).toBeUndefined(); return; diff --git a/packages/sdk/README.md b/packages/sdk/README.md index 00cd298..b73d74c 100644 --- a/packages/sdk/README.md +++ b/packages/sdk/README.md @@ -180,17 +180,21 @@ compiles its own. Model calls are the spans of type `generation` (operations `ch use this SDK: OpenAI, Anthropic, Amazon Bedrock, Google GenAI, and OpenRouter. Spans from the AI SDK and TanStack AI integrations, and from other OpenTelemetry tracers, do not get the attributes. -Outside `withPrompt`, those provider integrations find the prompt in the request: -`compile()` keeps the texts that it makes (the last 1000, at most 2,000,000 characters), and a -chat call whose request holds one of them gets `td.prompt.name`, `td.prompt.version`, and -`td.prompt.label`, also when input capture is off. A text from a template with a variable also -gives `td.prompt.variables`, and it wins over a text from a template with none. A message of only -a variable tag, for example `{{ question }}`, is not kept, because its text is your user's input. -The text in the request must be equal to the compiled text. `withPrompt` wins over this. The -integrations do it for OpenAI chat completions and responses, Anthropic messages, Bedrock Converse -and InvokeModel, Google GenAI `generateContent`, and OpenRouter chat and responses, but not for -images, audio, video, or OpenAI Realtime. For your own spans, `promptAttributesFor(request)` -gives the attributes to pass as `attributes`; it gives nothing inside `withPrompt`. +Outside `withPrompt`, those provider integrations record the prompt of the message objects that +`compile()` returned. `compile()` of a chat prompt tags each message that it makes with its prompt +and variables, in a non-enumerable symbol property: the keys and the JSON of the message do not +change. A chat call that sends a tagged message gets `td.prompt.name`, `td.prompt.version`, and +`td.prompt.label` of the newest one (of the last 1000 messages), also when input capture is off, +and `td.prompt.variables` when it is on. A copy of a message (a spread, `structuredClone`, or +JSON) and a new message with the same text have no tag, so the SDK does not record them. Bedrock +Converse and Google GenAI take another message shape: change the compiled messages in place, or +use `withPrompt`. A text prompt compiles to a string, which cannot carry a tag: use `withPrompt`. +`withPrompt` wins over the tag. The integrations do it for OpenAI chat completions and responses, +Anthropic messages, Bedrock Converse, Google GenAI `generateContent`, and OpenRouter chat and +responses, but not for Bedrock InvokeModel (its body is bytes), embeddings, images, audio, video, +or OpenAI Realtime. For your own spans, `promptAttributesFor(messages)` gives the attributes to +pass as `attributes`; it gives nothing inside `withPrompt`, and a span with `captureInput: false` +drops `td.prompt.variables`. ## API diff --git a/packages/sdk/src/attrs.ts b/packages/sdk/src/attrs.ts index 75afcb4..6dbf34f 100644 --- a/packages/sdk/src/attrs.ts +++ b/packages/sdk/src/attrs.ts @@ -179,6 +179,9 @@ export function fieldsToAttributes( if (fields.attributes) { Object.assign(attrs, omitUndefined(fields.attributes)); + + // The variables are input, also when the app passes them from promptAttributesFor(). + if (!cfg.captureInput) delete attrs["td.prompt.variables"]; } return attrs; diff --git a/packages/sdk/src/prompts.ts b/packages/sdk/src/prompts.ts index afd76c2..48cfe27 100644 --- a/packages/sdk/src/prompts.ts +++ b/packages/sdk/src/prompts.ts @@ -97,8 +97,8 @@ export interface ChatPrompt extends PromptBase { /** * Fills in the variables and replaces each placeholder message with the messages in its * variable. `withPrompt` records the variables of the last call before it starts or inside it. - * Each text is also kept with its variables (the last 1000 texts), so that a model call outside - * `withPrompt` that sends it can name the prompt; see `promptAttributesFor`. + * It tags each message object that it makes with this prompt and these variables, so that a + * model call outside `withPrompt` that sends the message records them; see `promptAttributesFor`. */ compile(variables?: PromptVariables): (ChatMessage | M)[]; } @@ -109,8 +109,8 @@ export interface TextPrompt extends PromptBase { readonly text: string; /** * Fills in the variables. `withPrompt` records the variables of the last call before it starts - * or inside it. The text is also kept with its variables (the last 1000 texts), so that a model - * call outside `withPrompt` that sends it can name the prompt; see `promptAttributesFor`. + * or inside it. A string cannot carry a tag, so a model call outside `withPrompt` does not + * record a text prompt. */ compile(variables?: PromptVariables): string; } @@ -201,11 +201,10 @@ interface PromptScope { variables: object | undefined; } -/** The prompt that `compile()` made a text from. */ -interface CompiledText { +/** The `compile()` call that made a chat message. */ +interface CompileTag { readonly prompt: Prompt; - /** Only for a template with a variable: the text holds the values, so they belong to it. */ - readonly variables: object | undefined; + readonly variables: PromptVariables; } const FETCH_TIMEOUT_MS = 10_000; @@ -219,19 +218,15 @@ const HTTP_DATE = /^[A-Z][a-z]{2}, \d{2} [A-Z][a-z]{2} \d{4} \d{2}:\d{2}:\d{2} G const VARIABLE = /\{\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*\}\}/g; const MODEL_CALL_OPERATIONS = new Set(["chat", "text_completion", "generate_content"]); const PROMPT_KEY = createContextKey("telemetry.dev prompt"); -const MAX_COMPILED_TEXTS = 1_000; -const MAX_COMPILED_CHARACTERS = 2_000_000; -const MAX_SEARCH_DEPTH = 6; -const MAX_SEARCH_NODES = 500; +// Not enumerable, so JSON.stringify, spreads, and structuredClone leave the tag out of a copy. +const COMPILED_BY = Symbol("telemetry.dev compiled by"); +/** How many messages promptAttributesFor looks at, from the end. */ +const MAX_TAG_LOOKUP = 1_000; const cache = new Map(); const compiledVariables = new WeakMap(); /** The Retry-After of the response that failed with this error, in milliseconds. */ const retryAfter = new WeakMap(); -// The texts of the last compile() calls, oldest first, for model calls outside withPrompt. -const compiledTexts = new Map(); -let compiledCharacters = 0; -let longestCompiledText = 0; /** * Fetches a prompt version from telemetry.dev and keeps it. When the copy is older than @@ -372,26 +367,24 @@ export function promptAttributes( } /** - * The td.prompt.* attributes for a model call outside `withPrompt`, from the texts that - * `compile()` made: the first text in `request` from a template with a variable, else the first - * text from a template with none. Only the first gives `td.prompt.variables`, because only its text - * holds the values. Gives undefined inside `withPrompt`, whose prompt the span gets, and when no - * text matches. It reads own data properties of arrays and plain objects, 6 levels and 500 values - * deep, and runs no getter. It never throws. + * The td.prompt.* attributes for a model call outside `withPrompt` that sends `messages` (a list + * of messages, or one message): those of the newest message, of the last 1000, that `compile()` + * of a chat prompt returned. The tag is on the message object, so a copy of it gives nothing. Gives + * undefined inside `withPrompt`, whose prompt the span gets. `td.prompt.variables` needs input + * capture on in the client, and the span drops it when its own input capture is off. It never + * throws. */ -export function promptAttributesFor(request: unknown): Record | undefined { +export function promptAttributesFor(messages: unknown): Record | undefined { const config = currentClient().core?.config; try { - if (compiledTexts.size === 0 || activeContext().getValue(PROMPT_KEY) !== undefined) { - return undefined; - } + if (activeContext().getValue(PROMPT_KEY) !== undefined) return undefined; - const found = findCompiledText(request); + const tag = newestTag(messages); - if (!found) return undefined; + if (!tag) return undefined; - return attributesOf(found.prompt, found.variables, config) as Record; + return attributesOf(tag.prompt, tag.variables, config) as Record; } catch (error) { reportError(config?.onError, error); @@ -415,47 +408,23 @@ function attributesOf( }); } -function findCompiledText(request: unknown): CompiledText | undefined { - let firstStatic: CompiledText | undefined; - let budget = MAX_SEARCH_NODES; - const stack: [value: unknown, depth: number][] = [[request, 0]]; - - while (stack.length > 0 && budget > 0) { - budget -= 1; - const [value, depth] = stack.pop()!; - - if (typeof value === "string") { - const found = value.length <= longestCompiledText ? compiledTexts.get(value) : undefined; +function newestTag(messages: unknown): CompileTag | undefined { + if (!Array.isArray(messages)) return tagOf(messages); + const stop = Math.max(0, messages.length - MAX_TAG_LOOKUP); - if (found?.variables !== undefined) return found; - firstStatic ??= found; - } else if (depth < MAX_SEARCH_DEPTH) { - const children = ownValues(value, budget); + for (let index = messages.length - 1; index >= stop; index -= 1) { + const tag = tagOf(messages[index]); - for (let index = children.length - 1; index >= 0; index -= 1) { - stack.push([children[index], depth + 1]); - } - } + if (tag) return tag; } - return firstStatic; + return undefined; } -/** At most `limit` own data values of an array or a plain object, in order. */ -function ownValues(value: unknown, limit: number): unknown[] { - if (typeof value !== "object" || value === null) return []; - let keys: string[]; - - if (Array.isArray(value)) { - keys = Array.from({ length: Math.min(value.length, limit) }, (_, index) => String(index)); - } else { - const prototype: unknown = Object.getPrototypeOf(value); - - if (prototype !== Object.prototype && prototype !== null) return []; - keys = Object.keys(value).slice(0, limit); - } - - return keys.map((key) => Object.getOwnPropertyDescriptor(value, key)?.value); +function tagOf(message: unknown): CompileTag | undefined { + return typeof message === "object" && message !== null + ? (message as { [COMPILED_BY]?: CompileTag })[COMPILED_BY] + : undefined; } function checkOptions(name: string, options: GetPromptOptions, type: string): void { @@ -649,26 +618,24 @@ function createPrompt(input: { if (scope?.prompt === prompt) scope.variables = variables; - const compileText = (template: string) => - remember(fill(template, variables), template, prompt, variables); - return type === "chat" - ? compileMessages(content.messages, variables, compileText) - : compileText(content.text ?? ""); + ? compileMessages(content.messages, Object.freeze({ prompt, variables })) + : fill(content.text ?? "", variables); }, }) as Prompt; return prompt; } -function compileMessages( - messages: readonly PromptMessage[], - variables: PromptVariables, - compileText: (template: string) => string, -): unknown[] { +function compileMessages(messages: readonly PromptMessage[], tag: CompileTag): unknown[] { + const { variables } = tag; + return messages.flatMap((message) => { if (message.role !== "placeholder") { - return [{ role: message.role, content: compileText(message.content) }]; + const compiled = { role: message.role, content: fill(message.content, variables) }; + Object.defineProperty(compiled, COMPILED_BY, { value: tag }); + + return [compiled]; } const value = Object.hasOwn(variables, message.name) ? variables[message.name] : undefined; @@ -683,39 +650,6 @@ function compileMessages( }); } -/** - * Keeps `text` for promptAttributesFor, as the newest entry. A template of only variable tags is - * skipped: its text is input of the app, which an unrelated request can hold too. - */ -function remember(text: string, template: string, prompt: Prompt, variables: object): string { - const fixedText = template.replace(VARIABLE, ""); - - if (fixedText.trim() === "" || text.length > MAX_COMPILED_CHARACTERS) return text; - - if (compiledTexts.delete(text)) compiledCharacters -= text.length; - compiledTexts.set(text, { prompt, variables: fixedText === template ? undefined : variables }); - compiledCharacters += text.length; - longestCompiledText = Math.max(longestCompiledText, text.length); - let droppedLongest = false; - - while (compiledTexts.size > MAX_COMPILED_TEXTS || compiledCharacters > MAX_COMPILED_CHARACTERS) { - const oldest = compiledTexts.keys().next().value!; - compiledTexts.delete(oldest); - compiledCharacters -= oldest.length; - droppedLongest ||= oldest.length === longestCompiledText; - } - - if (droppedLongest) { - longestCompiledText = 0; - - for (const kept of compiledTexts.keys()) { - longestCompiledText = Math.max(longestCompiledText, kept.length); - } - } - - return text; -} - function fill(text: string, variables: PromptVariables): string { return text.replace(VARIABLE, (_tag, name: string) => { const value = Object.hasOwn(variables, name) ? variables[name] : undefined; diff --git a/packages/sdk/tests/prompts.test.ts b/packages/sdk/tests/prompts.test.ts index 372150e..8e68644 100644 --- a/packages/sdk/tests/prompts.test.ts +++ b/packages/sdk/tests/prompts.test.ts @@ -743,108 +743,154 @@ async function promptOf(overrides: Record): Promise { }); } -test("promptAttributesFor finds a compiled text in the request shapes of the providers", async () => { +const DESK_PROMPT = { + messages: [ + { role: "system", content: "Answer as the {{team}} desk." }, + { role: "user", content: "Order {{order}}: where is it?" }, + ], +}; + +test("promptAttributesFor names the compile of a message object; a copy or a string names nothing", async () => { setup(); - const prompt = await promptOf({ - messages: [ - { role: "system", content: "Answer as the {{team}} desk." }, - { role: "user", content: "Order {{order}}: where is it?" }, - ], - }); + const prompt = await promptOf(DESK_PROMPT); const compiled = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; - const [system, user] = compiled.map((message) => message.content); - - const requests = { - same: compiled, - spread: compiled.map((message) => ({ ...message })), - json: JSON.parse(JSON.stringify(compiled)) as unknown, - anthropic: [system, compiled.slice(1)], - bedrock: [[{ text: system }], [{ role: "user", content: [{ text: user }] }]], - google: [{ parts: [{ text: system }] }, [{ role: "user", parts: [{ text: user }] }]], + const attributes = { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + "td.prompt.variables": '{"team":"billing","order":"#4"}', }; - for (const request of Object.values(requests)) { - expect(promptAttributesFor(request)).toEqual({ - "td.prompt.name": "support/reply", - "td.prompt.version": 12, - "td.prompt.label": "production", - "td.prompt.variables": '{"team":"billing","order":"#4"}', - }); + expect(promptAttributesFor(compiled)).toEqual(attributes); + expect(promptAttributesFor(compiled[0])).toEqual(attributes); + expect(JSON.stringify(compiled[0])).toBe( + '{"role":"system","content":"Answer as the billing desk."}', + ); + + for (const copy of [ + compiled.map((message) => ({ ...message })), + JSON.parse(JSON.stringify(compiled)) as unknown, + structuredClone(compiled), + compiled.map((message) => message.content), + ]) { + expect(promptAttributesFor(copy)).toBeUndefined(); } - expect( - promptAttributesFor([{ role: "user", content: "Order #5: where is it?" }]), - ).toBeUndefined(); + const text = await promptOf({ type: "text", messages: [], text: "Summarize {{doc}}." }); + expect(promptAttributesFor([text.compile({ doc: "the policy" })])).toBeUndefined(); }); -test("a static text gives no variables, and a template of only a variable tag is not kept", async () => { +test("an unrelated message with the text of a compiled message is not tagged", async () => { setup(); - const chat = await promptOf({ - messages: [ - { role: "system", content: "You are the refunds agent." }, - { role: "user", content: "{{ question }}" }, - ], + const prompt = await promptOf({ + messages: [{ role: "system", content: "You are a helpful assistant." }], }); - const compiled = chat.compile({ question: "Where is my refund?" }); + prompt.compile(); - expect(promptAttributesFor(compiled)).toEqual({ - "td.prompt.name": "support/reply", - "td.prompt.version": 12, - "td.prompt.label": "production", - }); - expect(promptAttributesFor([{ role: "user", content: "Where is my refund?" }])).toBeUndefined(); + expect( + promptAttributesFor([ + { role: "system", content: "You are a helpful assistant." }, + { role: "user", content: "What is 2 + 2?" }, + ]), + ).toBeUndefined(); +}); - const text = await promptOf({ type: "text", messages: [], text: "Summarize {{doc}} in a line." }); - const summary = text.compile({ doc: "the refund policy" }) as string; +test("two tasks with one text and different other variables each record their own", async () => { + const { spans } = setup(); + const prompt = await promptOf(DESK_PROMPT); - expect(promptAttributesFor(summary)).toMatchObject({ - "td.prompt.variables": '{"doc":"the refund policy"}', + // Both compiles make the system text "Answer as the billing desk.", and each task sends only it. + const [systemA] = prompt.compile({ team: "billing", order: "#A" }) as ChatMessage[]; + const [systemB] = prompt.compile({ team: "billing", order: "#B" }) as ChatMessage[]; + startSpan("a", { type: "generation", attributes: promptAttributesFor([systemA]) }).end(); + startSpan("b", { type: "generation", attributes: promptAttributesFor([systemB]) }).end(); + await flush(); + + expect( + Object.fromEntries( + spans.getFinishedSpans().map((span) => [span.name, span.attributes["td.prompt.variables"]]), + ), + ).toEqual({ + a: '{"team":"billing","order":"#A"}', + b: '{"team":"billing","order":"#B"}', }); }); -test("a text with variables wins over a static text, and the first one wins over a later one", async () => { +test("a rollout with two versions of the same messages records the version of each call", async () => { setup(); - const refunds = await promptOf({ - name: "refunds/system", - messages: [{ role: "system", content: "You are the refunds agent, v2." }], - }); - const order = await promptOf({ - name: "orders/question", - messages: [{ role: "user", content: "Where is order {{order}}?" }], - }); - const language = await promptOf({ - name: "style/language", - messages: [{ role: "system", content: "Reply in {{language}}." }], - }); + const { fetch } = fakeFetch( + reply({ + ...DESK_PROMPT, + rollout: { + percent: 16, + version: 13, + ...DESK_PROMPT, + text: null, + model: { provider: "openai", name: "gpt-4o-mini" }, + parameters: {}, + tools: [], + responseFormat: { type: "text" }, + }, + }), + ); + const options = { apiKey: "td_live_a", apiUrl: newApiUrl(), fetch }; + vi.spyOn(Math, "random").mockReturnValueOnce(0.159).mockReturnValueOnce(0.16); - const system = refunds.compile() as ChatMessage[]; - const question = order.compile({ order: "#4" }) as ChatMessage[]; - const style = language.compile({ language: "French" }) as ChatMessage[]; + const rollout = await getPrompt("support/reply", options); + const production = await getPrompt("support/reply", options); + const fromRollout = rollout.compile({ team: "billing", order: "#4" }); + const fromProduction = production.compile({ team: "billing", order: "#4" }); - expect(promptAttributesFor([...system, ...question])).toMatchObject({ - "td.prompt.name": "orders/question", - "td.prompt.variables": '{"order":"#4"}', - }); - expect(promptAttributesFor([...system, ...style, ...question])).toMatchObject({ - "td.prompt.name": "style/language", - "td.prompt.variables": '{"language":"French"}', - }); + expect(promptAttributesFor(fromRollout)?.["td.prompt.version"]).toBe(13); + expect(promptAttributesFor(fromProduction)?.["td.prompt.version"]).toBe(12); }); -test("each compile keeps its own values, so two tasks with one prompt get their own variables", async () => { +test("the newest compiled message wins, also after 170 messages of history", async () => { setup(); const prompt = await promptOf({ - messages: [{ role: "user", content: "Track parcel {{parcel}}." }], + messages: [ + { role: "system", content: "Answer as the {{team}} desk." }, + { role: "placeholder", name: "history" }, + { role: "user", content: "{{question}}" }, + ], }); + const history = Array.from({ length: 170 }, (_, index) => ({ + role: index % 2 === 0 ? ("user" as const) : ("assistant" as const), + content: `message ${index}`, + })); + + const firstTurn = prompt.compile({ + team: "billing", + question: "Where is #4?", + history, + }) as ChatMessage[]; + const answer = { role: "assistant", content: "On its way." }; + const secondTurn = prompt.compile({ + team: "returns", + question: "Can I return it?", + }) as ChatMessage[]; + const chat = [...firstTurn, answer, ...secondTurn.slice(1)]; + + expect(promptAttributesFor(firstTurn)?.["td.prompt.variables"]).toContain('"team":"billing"'); + expect(promptAttributesFor([firstTurn[0], ...history])?.["td.prompt.variables"]).toContain( + '"team":"billing"', + ); + expect(promptAttributesFor(chat)?.["td.prompt.variables"]).toBe( + '{"team":"returns","question":"Can I return it?"}', + ); +}); - const first = prompt.compile({ parcel: "A-1" }); - const second = prompt.compile({ parcel: "B-2" }); +test("promptAttributesFor looks at the last 1000 messages", async () => { + setup(); + const prompt = await promptOf(DESK_PROMPT); + const [system] = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; + const filler = (count: number) => Array.from({ length: count }, () => ({ role: "user" })); - expect(promptAttributesFor(first)?.["td.prompt.variables"]).toBe('{"parcel":"A-1"}'); - expect(promptAttributesFor(second)?.["td.prompt.variables"]).toBe('{"parcel":"B-2"}'); + expect(promptAttributesFor([system, ...filler(999)])).toBeDefined(); + expect(promptAttributesFor([system, ...filler(1000)])).toBeUndefined(); }); -test("withPrompt wins over a compiled text of another prompt", async () => { +test("withPrompt wins over a compiled message of another prompt", async () => { const { spans } = setup(); const scoped = await promptOf({ name: "scoped/prompt" }); const other = await promptOf({ @@ -861,7 +907,7 @@ test("withPrompt wins over a compiled text of another prompt", async () => { expect(spans.getFinishedSpans()[0]!.attributes["td.prompt.name"]).toBe("scoped/prompt"); }); -test("promptAttributesFor omits variables when input capture is off and masks them when on", async () => { +test("td.prompt.variables follows the input capture of the client and of the span", async () => { const prompt = await promptOf({ messages: [{ role: "user", content: "My card is {{card}}." }], }); @@ -875,77 +921,25 @@ test("promptAttributesFor omits variables when input capture is off and masks th }); await shutdown(); - setup({ + const { spans } = setup({ mask: (value, { key }) => (key === "td.prompt.variables" ? { card: "[masked]" } : value), }); - expect(promptAttributesFor(compiled)?.["td.prompt.variables"]).toBe('{"card":"[masked]"}'); -}); - -test("the kept texts stop at 1000 and at 2,000,000 characters, oldest out first", async () => { - setup(); - const ticket = await promptOf({ type: "text", messages: [], text: "Ticket {{n}} is open." }); - - for (let n = 0; n <= 1000; n += 1) ticket.compile({ n }); - - expect(promptAttributesFor("Ticket 0 is open.")).toBeUndefined(); - expect(promptAttributesFor("Ticket 1 is open.")).toMatchObject({ - "td.prompt.variables": '{"n":1}', - }); - - const note = await promptOf({ type: "text", messages: [], text: "Note: {{body}}" }); - const notes = ["a", "b", "c", "d", "e"].map( - (letter) => note.compile({ body: letter.repeat(500_000 - "Note: ".length) }) as string, - ); - - expect(promptAttributesFor(notes[0])).toBeUndefined(); - expect(promptAttributesFor(notes[1])?.["td.prompt.name"]).toBe("support/reply"); - - // A text over the character limit is not kept, and it does not push the others out. - note.compile({ body: "f".repeat(2_000_001) }); - expect(promptAttributesFor(notes[1])?.["td.prompt.name"]).toBe("support/reply"); - - // A text compiled again counts once. - for (let n = 0; n < 5; n += 1) note.compile({ body: "e".repeat(500_000 - "Note: ".length) }); - expect(promptAttributesFor(notes[4])?.["td.prompt.name"]).toBe("support/reply"); -}); - -test("promptAttributesFor looks 6 levels and 500 values deep", async () => { - setup(); - const prompt = await promptOf({ - messages: [{ role: "user", content: "Find invoice {{invoice}}." }], - }); - const [{ content }] = prompt.compile({ invoice: "INV-7" }) as ChatMessage[]; - const nest = (value: unknown, levels: number): unknown => - levels === 0 ? value : nest([value], levels - 1); + const attributes = promptAttributesFor(compiled); + expect(attributes?.["td.prompt.variables"]).toBe('{"card":"[masked]"}'); - expect(promptAttributesFor(nest(content, 6))).toBeDefined(); - expect(promptAttributesFor(nest(content, 7))).toBeUndefined(); + startSpan("private", { type: "generation", captureInput: false, attributes }).end(); + startSpan("updated", { type: "generation", captureInput: false }).update({ attributes }).end(); + await flush(); - // The request array, 498 strings, and the text are 500 values. - expect(promptAttributesFor([...Array(498).fill("filler"), content])).toBeDefined(); - expect(promptAttributesFor([...Array(499).fill("filler"), content])).toBeUndefined(); + for (const span of spans.getFinishedSpans()) { + expect(span.attributes["td.prompt.name"]).toBe("support/reply"); + expect(span.attributes["td.prompt.variables"]).toBeUndefined(); + } }); -test("promptAttributesFor runs no getter and reports a trap that throws", async () => { +test("promptAttributesFor reports an error and gives undefined", async () => { const errors: Error[] = []; setup({ onError: (error) => errors.push(error) }); - const prompt = await promptOf({ - messages: [{ role: "user", content: "Refund order {{order}}." }], - }); - const [{ content }] = prompt.compile({ order: "#3" }) as ChatMessage[]; - let reads = 0; - - const getter = { - role: "user", - get content() { - reads += 1; - - return content; - }, - }; - - expect(promptAttributesFor([getter])).toBeUndefined(); - expect(reads).toBe(0); const trap = new Proxy([], { get() { @@ -953,7 +947,7 @@ test("promptAttributesFor runs no getter and reports a trap that throws", async }, }); - expect(promptAttributesFor([trap, content])).toBeUndefined(); + expect(promptAttributesFor(trap)).toBeUndefined(); expect(errors.map((error) => error.message)).toEqual(["trap"]); }); diff --git a/sdks/python-anthropic/src/telemetry_dev_anthropic/__init__.py b/sdks/python-anthropic/src/telemetry_dev_anthropic/__init__.py index 956b634..d815bd7 100644 --- a/sdks/python-anthropic/src/telemetry_dev_anthropic/__init__.py +++ b/sdks/python-anthropic/src/telemetry_dev_anthropic/__init__.py @@ -244,10 +244,16 @@ def _offered_tools(tools: Any) -> dict[str, str] | None: return {"gen_ai.tool.definitions": json_dumps(definitions)} if definitions else None -def _prompt_attributes(request: object) -> dict[str, Any]: +def _add_prompt(fields: Mapping[str, Any], messages: object) -> dict[str, Any]: + """``fields`` plus the prompt tag of ``messages``, the list that the app passed: the copy that + ``_normalize_request_params`` makes has no tag. Call it where the span starts, so that a + ``with_prompt`` scope there wins.""" # Older cores have no prompt_attributes_for and record a prompt only inside with_prompt. read = getattr(telemetry_dev, "prompt_attributes_for", None) - return cast("dict[str, Any]", read(request)) if callable(read) else {} + prompt = cast("dict[str, Any]", read(messages)) if callable(read) else {} + if not prompt: + return dict(fields) + return {**fields, "attributes": {**(fields.get("attributes") or {}), **prompt}} def _messages_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: @@ -270,11 +276,7 @@ def _messages_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: "type": "generation", "model": model, "input": input_value, - "attributes": { - **(_offered_tools(tools) or {}), - **_prompt_attributes([params.get("system"), params.get("messages")]), - } - or None, + "attributes": _offered_tools(tools), "system_instructions": _native(params.get("system")), "temperature": _number(params.get("temperature")), "top_p": _number(params.get("top_p")), @@ -1462,16 +1464,21 @@ def __getattr__(self, name: str) -> Any: class _InstrumentedMessageStreamManager: - def __init__(self, inner: Any, name: str, fields: Mapping[str, Any], provider: str) -> None: + def __init__( + self, inner: Any, name: str, fields: Mapping[str, Any], messages: object, provider: str + ) -> None: self._inner = inner self._name = name self._fields = fields + self._messages = messages self._provider = provider self._proxy: _InstrumentedStream | None = None def __enter__(self) -> Any: handle = telemetry_dev.start_span( - self._name, provider=self._provider, **_clean_fields(self._fields) + self._name, + provider=self._provider, + **_clean_fields(_add_prompt(self._fields, self._messages)), ) end = _end_once(handle) started_at = time.perf_counter() @@ -1501,16 +1508,21 @@ def __getattr__(self, name: str) -> Any: class _InstrumentedAsyncMessageStreamManager: - def __init__(self, inner: Any, name: str, fields: Mapping[str, Any], provider: str) -> None: + def __init__( + self, inner: Any, name: str, fields: Mapping[str, Any], messages: object, provider: str + ) -> None: self._inner = inner self._name = name self._fields = fields + self._messages = messages self._provider = provider self._proxy: _InstrumentedAsyncStream | None = None async def __aenter__(self) -> Any: handle = telemetry_dev.start_span( - self._name, provider=self._provider, **_clean_fields(self._fields) + self._name, + provider=self._provider, + **_clean_fields(_add_prompt(self._fields, self._messages)), ) end = _end_once(handle) started_at = time.perf_counter() @@ -1543,9 +1555,10 @@ def __getattr__(self, name: str) -> Any: def _start_span( - params: Mapping[str, Any], provider: str + params: Mapping[str, Any], messages: object, provider: str ) -> tuple[telemetry_dev.SpanHandle, Callable[..., None], float]: name, fields = _messages_request(params) + fields = _add_prompt(fields, messages) handle = telemetry_dev.start_span(name, provider=provider, **_clean_fields(fields)) return handle, _end_once(handle), time.perf_counter() @@ -1558,7 +1571,9 @@ def wrapper(*args: Any, **kwargs: Any) -> Any: if _raw_response_requested(call_kwargs): return original(*args, **call_kwargs) streaming = call_kwargs.get("stream") is True - handle, end, started_at = _start_span(call_kwargs, provider(resource)) + handle, end, started_at = _start_span( + call_kwargs, _sent_params(kwargs).get("messages"), provider(resource) + ) try: result = original(*args, **call_kwargs) except BaseException as exc: @@ -1582,7 +1597,9 @@ async def wrapper(*args: Any, **kwargs: Any) -> Any: if _raw_response_requested(call_kwargs): return await original(*args, **call_kwargs) streaming = call_kwargs.get("stream") is True - handle, end, started_at = _start_span(call_kwargs, provider(resource)) + handle, end, started_at = _start_span( + call_kwargs, _sent_params(kwargs).get("messages"), provider(resource) + ) try: result = await original(*args, **call_kwargs) except BaseException as exc: @@ -1606,8 +1623,11 @@ def wrapper(*args: Any, **kwargs: Any) -> Any: resource = args[0] if args else None call_kwargs = _normalize_request_params(kwargs) name, fields = _messages_request(call_kwargs) + messages = _sent_params(kwargs).get("messages") manager = original(*args, **call_kwargs) - return _InstrumentedMessageStreamManager(manager, name, fields, provider(resource)) + return _InstrumentedMessageStreamManager( + manager, name, fields, messages, provider(resource) + ) setattr(wrapper, _WRAPPED_ATTR, True) setattr(wrapper, _ORIGINAL_ATTR, original) @@ -1622,8 +1642,11 @@ def wrapper(*args: Any, **kwargs: Any) -> Any: resource = args[0] if args else None call_kwargs = _normalize_request_params(kwargs) name, fields = _messages_request(call_kwargs) + messages = _sent_params(kwargs).get("messages") manager = original(*args, **call_kwargs) - return _InstrumentedAsyncMessageStreamManager(manager, name, fields, provider(resource)) + return _InstrumentedAsyncMessageStreamManager( + manager, name, fields, messages, provider(resource) + ) setattr(wrapper, _WRAPPED_ATTR, True) setattr(wrapper, _ORIGINAL_ATTR, original) diff --git a/sdks/python-anthropic/tests/test_anthropic.py b/sdks/python-anthropic/tests/test_anthropic.py index 8ffbc67..c22c555 100644 --- a/sdks/python-anthropic/tests/test_anthropic.py +++ b/sdks/python-anthropic/tests/test_anthropic.py @@ -437,12 +437,29 @@ def handler(request: httpx.Request) -> httpx.Response: ) +SCOPED_PROMPT = telemetry_dev.ChatPrompt( + name="scoped/prompt", + version=3, + label="production", + messages=[{"role": "user", "content": "Hi"}], + model=None, + parameters={}, + tools=[], + response_format={"type": "text"}, + is_fallback=False, +) + + @pytest.mark.parametrize( - ("field", "capture_input", "older_core"), - [("system", False, False), ("messages", True, False), ("messages", True, True)], + ("messages", "capture_input", "older_core"), + [("compiled", False, False), ("copied", True, False), ("compiled", True, True)], ) -def test_create_outside_with_prompt_records_the_prompt_of_a_compiled_text( - make: Any, monkeypatch: pytest.MonkeyPatch, field: str, capture_input: bool, older_core: bool +def test_create_outside_with_prompt_records_the_prompt_of_compiled_messages( + make: Any, + monkeypatch: pytest.MonkeyPatch, + messages: str, + capture_input: bool, + older_core: bool, ) -> None: memory = make(capture_input=capture_input) if older_core: @@ -453,15 +470,15 @@ def test_create_outside_with_prompt_records_the_prompt_of_a_compiled_text( client.messages.create( model="claude-sonnet-4-6", max_tokens=64, - system=system["content"] if field == "system" else "Be terse.", - messages=[{"role": "user", "content": user["content"] if field == "messages" else "Hi"}], + system=system["content"], + messages=[user if messages == "compiled" else dict(user)], ) a = attrs(only_span(memory)) assert a["gen_ai.request.model"] == "claude-sonnet-4-6" assert {key: value for key, value in a.items() if key.startswith("td.prompt.")} == ( {} - if older_core + if older_core or messages == "copied" else { "td.prompt.name": "support/reply", "td.prompt.version": 12, @@ -471,6 +488,59 @@ def test_create_outside_with_prompt_records_the_prompt_of_a_compiled_text( ) +@pytest.mark.parametrize("scope_around", ["stream()", "__enter__"]) +def test_messages_stream_takes_the_prompt_from_the_context_where_its_span_starts( + memory: SimpleNamespace, scope_around: str +) -> None: + _system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") + client = wrapped_sync_client(lambda _request: named_sse_response(stream_events())) + request: dict[str, Any] = {"model": "claude-sonnet-4-6", "max_tokens": 64, "messages": [user]} + + if scope_around == "stream()": + with telemetry_dev.with_prompt(SCOPED_PROMPT): + manager = client.messages.stream(**request) + with manager as stream: + "".join(stream.text_stream) + else: + manager = client.messages.stream(**request) + with telemetry_dev.with_prompt(SCOPED_PROMPT), manager as stream: + "".join(stream.text_stream) + + # The span starts at __enter__, so the scope there wins, and without one the message names it. + assert attrs(only_span(memory))["td.prompt.name"] == ( + "support/reply" if scope_around == "stream()" else "scoped/prompt" + ) + + +@pytest.mark.parametrize("scope_around", ["stream()", "__aenter__"]) +async def test_async_messages_stream_takes_the_prompt_from_the_context_where_its_span_starts( + memory: SimpleNamespace, scope_around: str +) -> None: + _system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") + + async def handler(_request: httpx.Request) -> httpx.Response: + return named_sse_response(stream_events()) + + client = wrapped_async_client(handler) + request: dict[str, Any] = {"model": "claude-sonnet-4-6", "max_tokens": 64, "messages": [user]} + + if scope_around == "stream()": + with telemetry_dev.with_prompt(SCOPED_PROMPT): + manager = client.messages.stream(**request) + async with manager as stream: + [text async for text in stream.text_stream] + else: + manager = client.messages.stream(**request) + with telemetry_dev.with_prompt(SCOPED_PROMPT): + async with manager as stream: + [text async for text in stream.text_stream] + await client.close() + + assert attrs(only_span(memory))["td.prompt.name"] == ( + "support/reply" if scope_around == "stream()" else "scoped/prompt" + ) + + def test_create_maps_native_messages_system_params_usage_finish_and_provider( memory: SimpleNamespace, ) -> None: diff --git a/sdks/python-bedrock/src/telemetry_dev_bedrock/_fields.py b/sdks/python-bedrock/src/telemetry_dev_bedrock/_fields.py index 04af28b..f6fc07d 100644 --- a/sdks/python-bedrock/src/telemetry_dev_bedrock/_fields.py +++ b/sdks/python-bedrock/src/telemetry_dev_bedrock/_fields.py @@ -12,10 +12,10 @@ def clean(fields: dict[str, Any]) -> dict[str, Any]: return {key: value for key, value in fields.items() if value is not None} -def prompt_attributes(request: object) -> dict[str, Any]: +def prompt_attributes(messages: object) -> dict[str, Any]: # Older cores have no prompt_attributes_for and record a prompt only inside with_prompt. read = getattr(telemetry_dev, "prompt_attributes_for", None) - return cast("dict[str, Any]", read(request)) if callable(read) else {} + return cast("dict[str, Any]", read(messages)) if callable(read) else {} def metadata_fields(response: dict[str, Any] | BaseException | None) -> dict[str, Any]: diff --git a/sdks/python-bedrock/src/telemetry_dev_bedrock/_invoke_model.py b/sdks/python-bedrock/src/telemetry_dev_bedrock/_invoke_model.py index 04a1162..d22e50b 100644 --- a/sdks/python-bedrock/src/telemetry_dev_bedrock/_invoke_model.py +++ b/sdks/python-bedrock/src/telemetry_dev_bedrock/_invoke_model.py @@ -4,7 +4,7 @@ import math from typing import Any -from ._fields import clean, prompt_attributes +from ._fields import clean _USAGE_ITERATION_LIMIT = 1000 _CLAUDE_USAGE_KEYS = ( @@ -93,8 +93,6 @@ def request_fields(params: dict[str, Any]) -> dict[str, Any]: fields = {"provider": "amazon-bedrock", "model": model, "input": body} if is_embedding_model(model): fields["output_type"] = "embedding" - else: - fields["attributes"] = prompt_attributes(body) or None fields.update(sampling_fields(model, body)) return clean(fields) diff --git a/sdks/python-bedrock/src/telemetry_dev_bedrock/_registry.py b/sdks/python-bedrock/src/telemetry_dev_bedrock/_registry.py index 0b68201..d27304d 100644 --- a/sdks/python-bedrock/src/telemetry_dev_bedrock/_registry.py +++ b/sdks/python-bedrock/src/telemetry_dev_bedrock/_registry.py @@ -57,7 +57,7 @@ def _converse_request(params: dict[str, Any]) -> dict[str, Any]: "model": _model(params), "input": normalize_messages(params.get("messages")), "system_instructions": normalize_content_list(params.get("system")) or None, - "attributes": prompt_attributes([params.get("system"), params.get("messages")]) or None, + "attributes": prompt_attributes(params.get("messages")) or None, "temperature": inference.get("temperature"), "top_p": inference.get("topP"), "max_tokens": inference.get("maxTokens"), diff --git a/sdks/python-bedrock/tests/test_bedrock.py b/sdks/python-bedrock/tests/test_bedrock.py index 8919186..505d4d6 100644 --- a/sdks/python-bedrock/tests/test_bedrock.py +++ b/sdks/python-bedrock/tests/test_bedrock.py @@ -14,6 +14,7 @@ from botocore.client import BaseClient from botocore.exceptions import ClientError, EventStreamError from botocore.response import StreamingBody +from botocore.stub import Stubber import telemetry_dev_bedrock._streams as bedrock_streams from telemetry_dev_bedrock import instrument_bedrock, uninstrument_bedrock, wrap_bedrock @@ -130,75 +131,56 @@ def _json_attr(span: Any, key: str) -> Any: ) -@pytest.mark.parametrize("older_core", [False, True]) -def test_converse_and_invoke_model_outside_with_prompt_record_the_prompt_of_a_compiled_text( - memory: SimpleNamespace, monkeypatch: pytest.MonkeyPatch, older_core: bool +@pytest.mark.parametrize( + ("messages", "older_core"), + [("changed in place", False), ("new", False), ("changed in place", True)], +) +def test_converse_outside_with_prompt_records_the_prompt_of_compiled_messages( + memory: SimpleNamespace, monkeypatch: pytest.MonkeyPatch, messages: str, older_core: bool ) -> None: if older_core: monkeypatch.delattr(telemetry_dev, "prompt_attributes_for") system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") - converse_reply = { - "output": {"message": {"role": "assistant", "content": [{"text": "Soon."}]}}, - "stopReason": "end_turn", - } - answer = _body({"content": [{"type": "text", "text": "Soon."}], "stop_reason": "end_turn"}) - embedding = _body({"embedding": [0.1, 0.2], "inputTextTokenCount": 6}) + blocks = [{"text": user["content"]}] + # Converse takes content blocks: a message changed in place keeps its tag, a new one has none. + if messages == "new": + message = {"role": "user", "content": blocks} + else: + user["content"] = blocks + message = user client = _client("bedrock-runtime") - _stub_api_call( - client, - [ - converse_reply, - converse_reply, - { - "body": StreamingBody(io.BytesIO(answer), len(answer)), - "contentType": "application/json", - }, - { - "body": StreamingBody(io.BytesIO(embedding), len(embedding)), - "contentType": "application/json", - }, - ], - monkeypatch, + # The stubber runs botocore's validation of the request. + stubber = Stubber(client) + stubber.add_response( + "converse", + { + "output": {"message": {"role": "assistant", "content": [{"text": "Soon."}]}}, + "stopReason": "end_turn", + "usage": {"inputTokens": 1, "outputTokens": 1, "totalTokens": 2}, + "metrics": {"latencyMs": 1}, + }, ) - wrapped = cast(Any, wrap_bedrock(client)) - wrapped.converse( - modelId="anthropic.claude-3-haiku", - system=[{"text": system["content"]}], - messages=[{"role": "user", "content": [{"text": "Hi"}]}], - ) - wrapped.converse( - modelId="anthropic.claude-3-haiku", - messages=[{"role": "user", "content": [{"text": user["content"]}]}], - ) - wrapped.invoke_model( - modelId="anthropic.claude-3-haiku", - contentType="application/json", - body=_body({"max_tokens": 64, "messages": [{"role": "user", "content": user["content"]}]}), - )["body"].read() - wrapped.invoke_model( - modelId="amazon.titan-embed-text-v2:0", - contentType="application/json", - body=_body({"inputText": user["content"]}), - )["body"].read() - - system_span, messages_span, invoke_span, embed_span = ( - dict(span.attributes or {}) for span in memory.span_exporter.get_finished_spans() - ) - for span in (system_span, messages_span, invoke_span): - assert span["gen_ai.request.model"] == "anthropic.claude-3-haiku" - assert {key: value for key, value in span.items() if key.startswith("td.prompt.")} == ( - {} - if older_core - else { - "td.prompt.name": "support/reply", - "td.prompt.version": 12, - "td.prompt.label": "production", - "td.prompt.variables": '{"team":"billing","order":"#4"}', - } + with stubber: + cast(Any, wrap_bedrock(client)).converse( + modelId="anthropic.claude-3-haiku", + system=[{"text": system["content"]}], + messages=[message], ) - assert embed_span["gen_ai.operation.name"] == "embeddings" - assert "td.prompt.name" not in embed_span + + [span] = memory.span_exporter.get_finished_spans() + a = dict(span.attributes or {}) + assert a["gen_ai.request.model"] == "anthropic.claude-3-haiku" + assert {key: value for key, value in a.items() if key.startswith("td.prompt.")} == ( + {} + if older_core or messages == "new" + else { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + "td.prompt.variables": '{"team":"billing","order":"#4"}', + } + ) def test_converse_happy_path(memory: SimpleNamespace, monkeypatch: pytest.MonkeyPatch) -> None: diff --git a/sdks/python-google-genai/src/telemetry_dev_google_genai/__init__.py b/sdks/python-google-genai/src/telemetry_dev_google_genai/__init__.py index 776c9de..8fc4b34 100644 --- a/sdks/python-google-genai/src/telemetry_dev_google_genai/__init__.py +++ b/sdks/python-google-genai/src/telemetry_dev_google_genai/__init__.py @@ -353,10 +353,10 @@ def _request_attributes(config: Any) -> dict[str, Any]: return attrs -def _prompt_attributes(request: object) -> dict[str, Any]: +def _prompt_attributes(messages: object) -> dict[str, Any]: # Older cores have no prompt_attributes_for and record a prompt only inside with_prompt. read = getattr(telemetry_dev, "prompt_attributes_for", None) - return cast("dict[str, Any]", read(request)) if callable(read) else {} + return cast("dict[str, Any]", read(messages)) if callable(read) else {} def _generate_request_fields(model: Any, contents: Any, config: Any) -> tuple[str, dict[str, Any]]: @@ -389,7 +389,7 @@ def _generate_request_fields(model: Any, contents: Any, config: Any) -> tuple[st output_type = _output_type(config) if output_type is not None: fields["output_type"] = output_type - attrs = {**_request_attributes(config), **_prompt_attributes([system_instruction, contents])} + attrs = {**_request_attributes(config), **_prompt_attributes(contents)} if attrs: fields["attributes"] = attrs return f"chat {model_name}", fields diff --git a/sdks/python-google-genai/tests/test_google_genai.py b/sdks/python-google-genai/tests/test_google_genai.py index c74c08c..e397763 100644 --- a/sdks/python-google-genai/tests/test_google_genai.py +++ b/sdks/python-google-genai/tests/test_google_genai.py @@ -673,36 +673,49 @@ async def run() -> Any: @pytest.mark.parametrize( - ("field", "capture_input", "older_core"), - [("system_instruction", True, False), ("contents", False, False), ("contents", True, True)], + ("contents", "capture_input", "older_core"), + [ + ("changed in place", True, False), + ("changed in place", False, False), + ("new", True, False), + ("changed in place", True, True), + ], ) -def test_generate_content_outside_with_prompt_records_the_prompt_of_a_compiled_text( - make: Any, monkeypatch: pytest.MonkeyPatch, field: str, capture_input: bool, older_core: bool +def test_generate_content_outside_with_prompt_records_the_prompt_of_compiled_messages( + make: Any, + monkeypatch: pytest.MonkeyPatch, + contents: str, + capture_input: bool, + older_core: bool, ) -> None: memory = make(capture_input=capture_input) if older_core: monkeypatch.delattr(telemetry_dev, "prompt_attributes_for") system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") - client, _transport = client_with_transport([happy_payload()]) + parts = [{"text": user["content"]}] + # Gemini takes parts: a message changed in place keeps its tag, a new one has none. + if contents == "new": + content: dict[str, Any] = {"role": "user", "parts": parts} + else: + del user["content"] + user["parts"] = parts + content = user + client, transport = client_with_transport([happy_payload()]) wrap_google_genai(client).models.generate_content( model="gemini-2.5-flash", - contents=[ - types.Content( - role="user", - parts=[types.Part(text=user["content"] if field == "contents" else "Hi")], - ) - ], - config=types.GenerateContentConfig( - system_instruction=system["content"] if field == "system_instruction" else "Be terse." - ), + contents=[content], + config=types.GenerateContentConfig(system_instruction=system["content"]), ) + assert cast(Any, transport.calls[0])["contents"] == [ + {"role": "user", "parts": [{"text": "Where is order #4?"}]} + ] a = attrs(only_span(memory)) assert a["gen_ai.request.model"] == "gemini-2.5-flash" assert {key: value for key, value in a.items() if key.startswith("td.prompt.")} == ( {} - if older_core + if older_core or contents == "new" else { "td.prompt.name": "support/reply", "td.prompt.version": 12, diff --git a/sdks/python-litellm/src/telemetry_dev_litellm/__init__.py b/sdks/python-litellm/src/telemetry_dev_litellm/__init__.py index ee18732..1a975f4 100644 --- a/sdks/python-litellm/src/telemetry_dev_litellm/__init__.py +++ b/sdks/python-litellm/src/telemetry_dev_litellm/__init__.py @@ -455,10 +455,10 @@ def _output_type(response_format: Any) -> str | None: return kind -def _prompt_attributes(request: object) -> dict[str, Any]: +def _prompt_attributes(messages: object) -> dict[str, Any]: # Older cores have no prompt_attributes_for and record a prompt only inside with_prompt. read = getattr(telemetry_dev, "prompt_attributes_for", None) - return cast("dict[str, Any]", read(request)) if callable(read) else {} + return cast("dict[str, Any]", read(messages)) if callable(read) else {} def _completion_request( @@ -554,7 +554,7 @@ def _responses_request( input_value, input_budget = _bounded_responses_native(_arg(args, kwargs, "input", 0)) attributes: dict[str, Any] = { "gen_ai.operation.name": "chat", - **_prompt_attributes([kwargs.get("instructions"), _arg(args, kwargs, "input", 0)]), + **_prompt_attributes(_arg(args, kwargs, "input", 0)), } if input_budget is not None and input_budget.truncated: attributes["telemetry.dev.capture.truncated"] = True diff --git a/sdks/python-litellm/tests/test_litellm.py b/sdks/python-litellm/tests/test_litellm.py index ae34865..a4c8dd1 100644 --- a/sdks/python-litellm/tests/test_litellm.py +++ b/sdks/python-litellm/tests/test_litellm.py @@ -74,44 +74,43 @@ def finished_spans(env: SimpleNamespace) -> list[ReadableSpan]: @pytest.mark.parametrize( - ("api", "capture_input", "older_core"), + ("api", "messages", "capture_input", "older_core"), [ - ("completion", False, False), - ("responses instructions", True, False), - ("responses input", True, False), - ("completion", True, True), + ("completion", "compiled", False, False), + ("responses", "compiled", True, False), + ("completion", "copied", True, False), + ("completion", "compiled", True, True), ], ) def test_calls_outside_with_prompt_record_the_prompt_of_compiled_messages( - make: Any, monkeypatch: pytest.MonkeyPatch, api: str, capture_input: bool, older_core: bool + make: Any, + monkeypatch: pytest.MonkeyPatch, + api: str, + messages: str, + capture_input: bool, + older_core: bool, ) -> None: memory = make(capture_input=capture_input) if older_core: monkeypatch.delattr(telemetry_dev, "prompt_attributes_for") - system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") + compiled = SUPPORT_PROMPT.compile(team="billing", order="#4") + sent = compiled if messages == "compiled" else [dict(message) for message in compiled] if api == "completion": - telemetry_dev_litellm.completion( - model="gpt-4o-mini", messages=[system, user], mock_response="Soon." - ) + telemetry_dev_litellm.completion(model="gpt-4o-mini", messages=sent, mock_response="Soon.") else: def fake_responses(*_args: Any, **_kwargs: Any) -> SimpleNamespace: return responses_result() monkeypatch.setattr(litellm, "responses", fake_responses) - in_input = api == "responses input" - telemetry_dev_litellm.responses( - [{"role": "user", "content": user["content"] if in_input else "Hi"}], - "openai/gpt-4.1-mini", - instructions="Be terse." if in_input else system["content"], - ) + telemetry_dev_litellm.responses(sent, "openai/gpt-4.1-mini") a = attrs(only_span(memory)) assert a["gen_ai.request.model"] == ("gpt-4o-mini" if api == "completion" else "gpt-4.1-mini") assert {key: value for key, value in a.items() if key.startswith("td.prompt.")} == ( {} - if older_core + if older_core or messages == "copied" else { "td.prompt.name": "support/reply", "td.prompt.version": 12, diff --git a/sdks/python-openai/src/telemetry_dev_openai/__init__.py b/sdks/python-openai/src/telemetry_dev_openai/__init__.py index eb84699..fdad482 100644 --- a/sdks/python-openai/src/telemetry_dev_openai/__init__.py +++ b/sdks/python-openai/src/telemetry_dev_openai/__init__.py @@ -349,10 +349,10 @@ def _capture_text(value: str, budget: telemetry_dev.CaptureBudget) -> str: return value -def _prompt_attributes(request: object) -> dict[str, Any]: +def _prompt_attributes(messages: object) -> dict[str, Any]: # Older cores have no prompt_attributes_for and record a prompt only inside with_prompt. read = getattr(telemetry_dev, "prompt_attributes_for", None) - return cast("dict[str, Any]", read(request)) if callable(read) else {} + return cast("dict[str, Any]", read(messages)) if callable(read) else {} def _chat_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: @@ -441,7 +441,7 @@ def _chat_response(response: Any) -> dict[str, Any]: def _responses_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: model = _string(params.get("model")) - attributes = _prompt_attributes([params.get("instructions"), params.get("input")]) + attributes = _prompt_attributes(params.get("input")) captured, truncated = _bounded_responses_capture(params.get("input"), "input") if truncated: attributes["telemetry.dev.capture.truncated"] = True diff --git a/sdks/python-openai/tests/test_openai.py b/sdks/python-openai/tests/test_openai.py index 6f71978..1ad3a36 100644 --- a/sdks/python-openai/tests/test_openai.py +++ b/sdks/python-openai/tests/test_openai.py @@ -1227,41 +1227,43 @@ def prompt_attrs(span: ReadableSpan) -> dict[str, object]: @pytest.mark.parametrize( - ("api", "capture_input", "older_core"), + ("api", "messages", "capture_input", "older_core"), [ - ("chat", False, False), - ("responses instructions", True, False), - ("responses input", True, False), - ("chat", True, True), + ("chat", "compiled", False, False), + ("responses", "compiled", True, False), + ("chat", "copied", True, False), + ("chat", "compiled", True, True), ], ) def test_calls_outside_with_prompt_record_the_prompt_of_compiled_messages( - make: Any, monkeypatch: pytest.MonkeyPatch, api: str, capture_input: bool, older_core: bool + make: Any, + monkeypatch: pytest.MonkeyPatch, + api: str, + messages: str, + capture_input: bool, + older_core: bool, ) -> None: memory = make(capture_input=capture_input) if older_core: monkeypatch.delattr(telemetry_dev, "prompt_attributes_for") - system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") + compiled = SUPPORT_PROMPT.compile(team="billing", order="#4") + sent = compiled if messages == "compiled" else [dict(message) for message in compiled] def handler(_request: httpx.Request) -> httpx.Response: return json_response(chat_completion() if api == "chat" else response_payload()) client = wrap_openai(sync_client(handler)) if api == "chat": - cast(Any, client.chat.completions.create)(model="gpt-4o", messages=[system, user]) + cast(Any, client.chat.completions.create)(model="gpt-4o", messages=sent) else: - in_input = api == "responses input" - cast(Any, client.responses.create)( - model="gpt-4.1", - instructions="Be terse." if in_input else system["content"], - input=[{"role": "user", "content": user["content"] if in_input else "Hi"}], - ) + cast(Any, client.responses.create)(model="gpt-4.1", input=sent) span = only_span(memory) - assert attrs(span)["gen_ai.request.model"] == ("gpt-4o" if api == "chat" else "gpt-4.1") - assert prompt_attrs(span) == ( + a = attrs(span) + assert a["gen_ai.request.model"] == ("gpt-4o" if api == "chat" else "gpt-4.1") + assert {key: value for key, value in a.items() if key.startswith("td.prompt.")} == ( {} - if older_core + if older_core or messages == "copied" else { "td.prompt.name": "support/reply", "td.prompt.version": 12, @@ -1269,7 +1271,7 @@ def handler(_request: httpx.Request) -> httpx.Response: **({"td.prompt.variables": '{"team":"billing","order":"#4"}'} if capture_input else {}), } ) - assert ("gen_ai.input.messages" in attrs(span)) is capture_input + assert ("gen_ai.input.messages" in a) is capture_input def test_chat_completion_maps_native_messages_usage_finish_provider_and_sampling( diff --git a/sdks/python-openrouter/src/telemetry_dev_openrouter/__init__.py b/sdks/python-openrouter/src/telemetry_dev_openrouter/__init__.py index 4af3fb7..a5056d3 100644 --- a/sdks/python-openrouter/src/telemetry_dev_openrouter/__init__.py +++ b/sdks/python-openrouter/src/telemetry_dev_openrouter/__init__.py @@ -174,10 +174,10 @@ def _cost_usd(raw_usage: Any) -> float | None: return cost if cost is not None else upstream_cost -def _prompt_attributes(request: object) -> dict[str, Any]: +def _prompt_attributes(messages: object) -> dict[str, Any]: # Older cores have no prompt_attributes_for and record a prompt only inside with_prompt. read = getattr(telemetry_dev, "prompt_attributes_for", None) - return cast("dict[str, Any]", read(request)) if callable(read) else {} + return cast("dict[str, Any]", read(messages)) if callable(read) else {} def _chat_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: @@ -267,8 +267,7 @@ def _responses_request(params: Mapping[str, Any]) -> tuple[str, dict[str, Any]]: "max_tokens": _number(params.get("max_output_tokens")), "frequency_penalty": _number(params.get("frequency_penalty")), "presence_penalty": _number(params.get("presence_penalty")), - "attributes": _prompt_attributes([params.get("instructions"), params.get("input")]) - or None, + "attributes": _prompt_attributes(params.get("input")) or None, }, ) diff --git a/sdks/python-openrouter/tests/test_openrouter.py b/sdks/python-openrouter/tests/test_openrouter.py index 01456c2..400c4f5 100644 --- a/sdks/python-openrouter/tests/test_openrouter.py +++ b/sdks/python-openrouter/tests/test_openrouter.py @@ -238,47 +238,42 @@ def embeddings_payload() -> dict[str, Any]: @pytest.mark.parametrize( - ("api", "capture_input", "older_core"), + ("api", "messages", "capture_input", "older_core"), [ - ("chat", False, False), - ("responses instructions", True, False), - ("responses input", True, False), - ("chat", True, True), + ("chat", "compiled", False, False), + ("responses", "compiled", True, False), + ("chat", "copied", True, False), + ("chat", "compiled", True, True), ], ) def test_calls_outside_with_prompt_record_the_prompt_of_compiled_messages( - make: Any, monkeypatch: pytest.MonkeyPatch, api: str, capture_input: bool, older_core: bool + make: Any, + monkeypatch: pytest.MonkeyPatch, + api: str, + messages: str, + capture_input: bool, + older_core: bool, ) -> None: memory = make(capture_input=capture_input) if older_core: monkeypatch.delattr(telemetry_dev, "prompt_attributes_for") - system, user = SUPPORT_PROMPT.compile(team="billing", order="#4") + compiled = SUPPORT_PROMPT.compile(team="billing", order="#4") + sent = compiled if messages == "compiled" else [dict(message) for message in compiled] def handler(_request: httpx.Request) -> httpx.Response: return json_response(chat_payload() if api == "chat" else responses_payload()) client = wrap_open_router(sync_client(handler)) if api == "chat": - client.chat.send( - model="openai/gpt-4o-mini", - messages=[ - ChatSystemMessage(role="system", content=system["content"]), - ChatUserMessage(role="user", content=user["content"]), - ], - ) + client.chat.send(model="openai/gpt-4o-mini", messages=cast(Any, sent)) else: - in_input = api == "responses input" - client.responses.send( - model="openai/gpt-4o-mini", - input=[EasyInputMessage(role="user", content=user["content"] if in_input else "Hi")], - instructions="Be terse." if in_input else system["content"], - ) + client.responses.send(model="openai/gpt-4o-mini", input=cast(Any, sent)) a = attrs(only_span(memory)) assert a["gen_ai.request.model"] == "openai/gpt-4o-mini" assert {key: value for key, value in a.items() if key.startswith("td.prompt.")} == ( {} - if older_core + if older_core or messages == "copied" else { "td.prompt.name": "support/reply", "td.prompt.version": 12, diff --git a/sdks/python/README.md b/sdks/python/README.md index dfcd2b8..e39a537 100644 --- a/sdks/python/README.md +++ b/sdks/python/README.md @@ -253,19 +253,23 @@ for code that has not compiled in its own task. Model calls are the spans of typ integrations: OpenAI, Anthropic, Amazon Bedrock, Google GenAI, LiteLLM, and OpenRouter. Spans from other OpenTelemetry tracers do not get the attributes. -Outside `with_prompt`, those provider integrations find the prompt in the request: `compile()` -keeps the texts that it makes (the last 1000, at most 2,000,000 characters), and a chat call -whose request holds one of them gets `td.prompt.name`, `td.prompt.version`, and -`td.prompt.label`, also when input capture is off. A text from a template with a variable also -gives `td.prompt.variables`, and it wins over a text from a template with none. A message of only -a variable tag, for example `{{ question }}`, is not kept, because its text is your user's input. -The text in the request must be equal to the compiled text. `with_prompt` wins over this. The -integrations do it for OpenAI chat completions and responses, Anthropic messages, Bedrock -Converse and InvokeModel, Google GenAI `generate_content`, LiteLLM completion and responses, and -OpenRouter chat and responses, but not for images, audio, video, or OpenAI Realtime. With an -older `telemetry-dev` that has no `prompt_attributes_for`, a provider integration records the -prompt only in `with_prompt`. For your own spans, `prompt_attributes_for(request)` gives the -attributes to pass as `attributes`; it gives `{}` inside `with_prompt`. +Outside `with_prompt`, those provider integrations record the prompt of the message dicts that +`compile()` returned. `compile()` of a chat prompt returns instances of a `dict` subclass that +also name its prompt and variables: the keys and the JSON of the message do not change. A chat +call that sends such a message gets `td.prompt.name`, `td.prompt.version`, and `td.prompt.label` +of the newest one (of the last 1000 messages), also when input capture is off, and +`td.prompt.variables` when it is on. A copy of a message (`dict()`, `.copy()`, `copy.copy`, +`copy.deepcopy`, pickle, or JSON) and a new message with the same text do not name the prompt, so +the SDK does not record them. Bedrock Converse and Google GenAI take another message shape: change +the compiled dicts in place, or use `with_prompt`. A text prompt compiles to a string, which +cannot name its prompt: use `with_prompt`. `with_prompt` wins over the message. The integrations +do it for OpenAI chat completions and responses, Anthropic messages (for `messages.stream()`, at +`__enter__`), Bedrock Converse, Google GenAI `generate_content`, LiteLLM completion and responses, +and OpenRouter chat and responses, but not for Bedrock InvokeModel (its body is bytes), embeddings, +images, audio, video, or OpenAI Realtime. With an older `telemetry-dev` that has no +`prompt_attributes_for`, a provider integration records the prompt only in `with_prompt`. For your +own spans, `prompt_attributes_for(messages)` gives the attributes to pass as `attributes`; it gives +`{}` inside `with_prompt`, and a span with `capture_input=False` drops `td.prompt.variables`. ## Auto-metrics diff --git a/sdks/python/src/telemetry_dev/_prompts.py b/sdks/python/src/telemetry_dev/_prompts.py index 43b863e..cb230fd 100644 --- a/sdks/python/src/telemetry_dev/_prompts.py +++ b/sdks/python/src/telemetry_dev/_prompts.py @@ -4,14 +4,11 @@ import contextlib import copy import hashlib -import itertools import json -import os import random import re import socket import threading -import types import urllib.error import urllib.parse import urllib.request @@ -53,10 +50,8 @@ _VARIABLE = re.compile(r"\{\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*\}\}") _MODEL_CALL_OPERATIONS = frozenset({"chat", "text_completion", "generate_content"}) _PROMPT_KEY = otel_context.create_key("telemetry_dev.prompt") -_MAX_COMPILED_TEXTS = 1000 -_MAX_COMPILED_CHARACTERS = 2_000_000 -_MAX_SEARCH_DEPTH = 6 -_MAX_SEARCH_NODES = 500 +# How many messages prompt_attributes_for looks at, from the end. +_MAX_TAG_LOOKUP = 1000 class PromptModel(TypedDict): @@ -105,15 +100,16 @@ def compile( ) -> list[dict[str, Any]]: """Fill in the variables and replace each placeholder message with the list of messages in its variable. ``with_prompt`` records the variables of the last call before it starts - or inside it. Each text is also kept with its variables (the last 1000 texts), so that a - model call outside ``with_prompt`` that sends it can name the prompt; see - ``prompt_attributes_for``.""" + or inside it. It tags each message dict that it makes with this prompt and these + variables, so that a model call outside ``with_prompt`` that sends the message records + them; see ``prompt_attributes_for``.""" values = _record(self, variables, kwargs) + tag = _CompileTag(self, values) compiled: list[dict[str, Any]] = [] for message in self.messages: if message.get("role") != "placeholder": - content = _compile_text(self, str(message["content"]), values) - compiled.append({"role": message["role"], "content": content}) + content = _fill(str(message["content"]), values) + compiled.append(_CompiledMessage(tag, role=message["role"], content=content)) continue name = str(message["name"]) value = values.get(name) @@ -149,10 +145,9 @@ def messages(self) -> list[dict[str, Any]]: def compile(self, variables: Mapping[str, Any] | None = None, /, **kwargs: Any) -> str: """Fill in the variables. ``with_prompt`` records the variables of the last call before it - starts or inside it. The text is also kept with its variables (the last 1000 texts), so - that a model call outside ``with_prompt`` that sends it can name the prompt; see - ``prompt_attributes_for``.""" - return _compile_text(self, self.text, _record(self, variables, kwargs)) + starts or inside it. A string cannot carry a tag, so a model call outside ``with_prompt`` + does not record a text prompt.""" + return _fill(self.text, _record(self, variables, kwargs)) Prompt: TypeAlias = ChatPrompt | TextPrompt @@ -297,23 +292,22 @@ def prompt_attributes( return _attributes_of(scope.prompt, variables, client if capture_input else None) -def prompt_attributes_for(request: object) -> dict[str, AttributeValue]: - """The td.prompt.* attributes for a model call outside ``with_prompt``, from the texts that - ``compile()`` made: the first text in ``request`` from a template with a variable, else the - first text from a template with none. Only the first gives ``td.prompt.variables``, because - only its text holds the values. Gives ``{}`` inside ``with_prompt``, whose prompt the span - gets, and when no text matches. It reads lists, tuples, dicts, and the fields that objects - store (pydantic models), 6 levels and 500 values deep, and runs no property. It never - raises.""" +def prompt_attributes_for(messages: object) -> dict[str, AttributeValue]: + """The td.prompt.* attributes for a model call outside ``with_prompt`` that sends ``messages`` + (a list or tuple of messages, or one message): those of the newest message, of the last 1000, + that ``compile()`` of a chat prompt returned. The tag is on the dict that ``compile()`` made, so + a copy of it gives ``{}``. Gives ``{}`` inside ``with_prompt``, whose prompt the span gets. + ``td.prompt.variables`` needs input capture on in the client, and the span drops it when its + own input capture is off. It runs no code of the app and never raises.""" client = get_client() try: if isinstance(otel_context.get_value(_PROMPT_KEY), _PromptScope): return {} - found = _compiled_texts.find(request) - if found is None: + tag = _newest_tag(messages) + if tag is None: return {} capture = client if client is not None and client.capture_input else None - return _attributes_of(found.prompt, found.variables, capture) + return _attributes_of(tag.prompt, tag.variables, capture) except Exception as exc: if client is not None: client.report("prompt_attributes_for failed", exc) @@ -444,133 +438,36 @@ def _record( @dataclass(frozen=True) -class _CompiledText: - """The prompt that ``compile()`` made a text from.""" +class _CompileTag: + """The ``compile()`` call that made a chat message.""" - prompt: Prompt - variables: dict[str, Any] | None - """Only for a template with a variable: the text holds the values, so they belong to it.""" + prompt: ChatPrompt + variables: dict[str, Any] -class _CompiledTexts: - """The texts of the last ``compile()`` calls, oldest first, for model calls outside - ``with_prompt``.""" +class _CompiledMessage(dict[str, Any]): + """A message that ``compile()`` made: a dict with the tag in a slot. A copy, pickle, or + ``copy.deepcopy`` of it is a plain dict without the tag.""" - def __init__(self) -> None: - self.texts: OrderedDict[str, _CompiledText] = OrderedDict() - self.characters = 0 - self.longest = 0 - self.lock = threading.Lock() - - def remember(self, text: str, template: str, prompt: Prompt, values: dict[str, Any]) -> None: - """Keep ``text`` as the newest entry. A template of only variable tags is skipped: its - text is input of the app, which an unrelated request can hold too.""" - fixed_text = _VARIABLE.sub("", template) - if not fixed_text.strip() or len(text) > _MAX_COMPILED_CHARACTERS: - return - compiled = _CompiledText(prompt, None if fixed_text == template else values) - with self.lock: - if self.texts.pop(text, None) is not None: - self.characters -= len(text) - self.texts[text] = compiled - self.characters += len(text) - self.longest = max(self.longest, len(text)) - dropped_longest = False - while ( - len(self.texts) > _MAX_COMPILED_TEXTS or self.characters > _MAX_COMPILED_CHARACTERS - ): - oldest, _ = self.texts.popitem(last=False) - self.characters -= len(oldest) - dropped_longest = dropped_longest or len(oldest) == self.longest - if dropped_longest: - self.longest = max(map(len, self.texts), default=0) - - def find(self, request: object) -> _CompiledText | None: - first_static: _CompiledText | None = None - budget = _MAX_SEARCH_NODES - stack: list[tuple[object, int]] = [(request, 0)] - with self.lock: - while stack and budget > 0: - budget -= 1 - value, depth = stack.pop() - if type(value) is str: - found = self.texts.get(value) if len(value) <= self.longest else None - if found is not None and found.variables is not None: - return found - first_static = first_static or found - elif depth < _MAX_SEARCH_DEPTH: - children = _stored_values(value, budget) - stack.extend((child, depth + 1) for child in reversed(children)) - return first_static - - -_compiled_texts = _CompiledTexts() - - -def _lock_compiled_texts_before_fork() -> None: - _compiled_texts.lock.acquire() - - -def _unlock_compiled_texts_after_fork() -> None: - _compiled_texts.lock.release() - - -# fork() waits for the lock: the child has only the thread that forked, so a lock that another -# thread held at the fork would stay held there for all time. -if hasattr(os, "register_at_fork"): # absent on Windows - os.register_at_fork( - before=_lock_compiled_texts_before_fork, - after_in_parent=_unlock_compiled_texts_after_fork, - after_in_child=_unlock_compiled_texts_after_fork, - ) + __slots__ = ("compiled_by",) + def __init__(self, compiled_by: _CompileTag, **fields: Any) -> None: + super().__init__(**fields) + self.compiled_by = compiled_by -def _compile_text(prompt: Prompt, template: str, values: dict[str, Any]) -> str: - text = _fill(template, values) - _compiled_texts.remember(text, template, prompt, values) - return text - - -_TYPE_MRO = type.__dict__["__mro__"] -_TYPE_NAMESPACE = type.__dict__["__dict__"] - - -def _stored_values(value: object, limit: int) -> list[object]: - """At most ``limit`` values that ``value`` stores, in order: the items of a list or tuple, the - values of a dict, or the fields of another object in ``__dict__`` and ``__pydantic_extra__``. - Subclasses of list, tuple, and dict are skipped and fields are read from C-level storage, so - no override or property of the app runs.""" - cls = type(value) - if cls is list or cls is tuple: - return list(cast("list[object] | tuple[object, ...]", value)[:limit]) - if cls is dict: - return list(itertools.islice(cast("dict[object, object]", value).values(), limit)) - values: list[object] = [] - for name in ("__dict__", "__pydantic_extra__"): - stored = _stored(value, name) - if type(stored) is dict: - fields = cast("dict[str, object]", stored) - values.extend(itertools.islice(fields.values(), limit - len(values))) - return values + def __reduce__(self) -> tuple[type[dict[str, Any]], tuple[dict[str, Any]]]: + return dict, (dict(self),) -def _stored(value: object, name: str) -> object: - cls = type(value) - for klass in _TYPE_MRO.__get__(cls): - namespace = _TYPE_NAMESPACE.__get__(klass) - if name not in namespace: - continue - descriptor: object = namespace[name] - kind = type(descriptor) - storage = kind is types.MemberDescriptorType or ( - name == "__dict__" and kind is types.GetSetDescriptorType - ) - if not storage or cast(Any, descriptor).__name__ != name: - return None - try: - return cast(Any, descriptor).__get__(value, cls) - except (AttributeError, TypeError): - return None +def _newest_tag(messages: object) -> _CompileTag | None: + # Exact types: a subclass of list or tuple could run code of the app on each read. + if type(messages) is _CompiledMessage: + return messages.compiled_by + if type(messages) is not list and type(messages) is not tuple: + return None + for message in reversed(cast("list[object]", messages)[-_MAX_TAG_LOOKUP:]): + if type(message) is _CompiledMessage: + return message.compiled_by return None diff --git a/sdks/python/src/telemetry_dev/_spans.py b/sdks/python/src/telemetry_dev/_spans.py index 0ac1b26..d66f086 100644 --- a/sdks/python/src/telemetry_dev/_spans.py +++ b/sdks/python/src/telemetry_dev/_spans.py @@ -38,6 +38,7 @@ ATTR_FINISH_REASONS, ATTR_OPERATION, ATTR_OUTPUT_TYPE, + ATTR_PROMPT_VARIABLES, ATTR_PROVIDER, ATTR_REQUEST_MODEL, ATTR_RESPONSE_ID, @@ -279,6 +280,9 @@ def _fields_to_attributes( raw = fields.get("attributes") if raw is not None: for key, value in raw.items(): + # The variables are input, also when the app passes them from prompt_attributes_for(). + if key == ATTR_PROMPT_VARIABLES and not state.capture_input: + continue attr = coerce_attr_value( value, max_len=sys.maxsize, diff --git a/sdks/python/tests/test_prompts.py b/sdks/python/tests/test_prompts.py index 9fd8656..29297e3 100644 --- a/sdks/python/tests/test_prompts.py +++ b/sdks/python/tests/test_prompts.py @@ -2,15 +2,17 @@ import asyncio import contextvars +import copy +import gc import itertools import json -import os -import signal +import pickle import socket import threading import time import urllib.error import urllib.request +import weakref from collections.abc import Callable, Generator from datetime import datetime, timedelta, timezone from email.utils import format_datetime @@ -81,14 +83,9 @@ def clock(monkeypatch: pytest.MonkeyPatch) -> list[float]: @pytest.fixture(autouse=True) -def empty_cache(monkeypatch: pytest.MonkeyPatch) -> None: +def empty_cache() -> None: # A fetch that runs on after its test must not answer a later test on a reused port. prompts_module._cache.clear() # pyright: ignore[reportPrivateUsage] - monkeypatch.setattr( - prompts_module, - "_compiled_texts", - prompts_module._CompiledTexts(), # pyright: ignore[reportPrivateUsage] - ) def chat_prompt(*messages: dict[str, Any], name: str = "support/reply") -> ChatPrompt: @@ -831,102 +828,153 @@ def mask(value: Any, context: telemetry_dev.MaskContext) -> Any: assert "td.prompt.variables" not in (span.attributes or {}) -class Part: - def __init__(self, text: str) -> None: - self.text = text +DESK_MESSAGES = [ + {"role": "system", "content": "Answer as the {{team}} desk."}, + {"role": "user", "content": "Order {{order}}: where is it?"}, +] -def test_prompt_attributes_for_finds_a_compiled_text_in_the_request_shapes_of_the_providers( +def test_prompt_attributes_for_names_the_compile_of_a_message_dict_and_a_copy_names_nothing( make: MakeClient, ) -> None: make() - prompt = chat_prompt( - {"role": "system", "content": "Answer as the {{team}} desk."}, - {"role": "user", "content": "Order {{order}}: where is it?"}, - ) - compiled = prompt.compile(team="billing", order="#4") - system, user = (message["content"] for message in compiled) - - requests = { - "same": compiled, - "copy": [dict(message) for message in compiled], - "json": json.loads(json.dumps(compiled)), - "anthropic": [system, compiled[1:]], - "bedrock": [[{"text": system}], [{"role": "user", "content": [{"text": user}]}]], - "objects": (Part(system), [SimpleNamespace(role="user", parts=[Part(user)])]), + compiled = chat_prompt(*DESK_MESSAGES).compile(team="billing", order="#4") + attributes = { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + "td.prompt.variables": '{"team":"billing","order":"#4"}', } - for request in requests.values(): - assert prompt_attributes_for(request) == { - "td.prompt.name": "support/reply", - "td.prompt.version": 12, - "td.prompt.label": "production", - "td.prompt.variables": '{"team":"billing","order":"#4"}', - } - assert prompt_attributes_for([{"role": "user", "content": "Order #5: where is it?"}]) == {} + assert prompt_attributes_for(compiled) == attributes + assert prompt_attributes_for(tuple(compiled)) == attributes + assert prompt_attributes_for(compiled[0]) == attributes + assert json.dumps(compiled[0]) == '{"role": "system", "content": "Answer as the billing desk."}' + + copies: list[object] = [ + [dict(message) for message in compiled], + [message.copy() for message in compiled], + [copy.copy(message) for message in compiled], + copy.deepcopy(compiled), + pickle.loads(pickle.dumps(compiled)), + json.loads(json.dumps(compiled)), + [message["content"] for message in compiled], + ] + for messages in copies: + assert prompt_attributes_for(messages) == {} + + summary = text_prompt("Summarize {{doc}}.").compile(doc="the policy") + assert prompt_attributes_for([summary]) == {} -def test_a_static_text_gives_no_variables_and_a_template_of_only_a_variable_tag_is_not_kept( +def test_an_unrelated_message_with_the_text_of_a_compiled_message_is_not_tagged( make: MakeClient, ) -> None: make() - chat = chat_prompt( - {"role": "system", "content": "You are the refunds agent."}, - {"role": "user", "content": "{{ question }}"}, - ) - compiled = chat.compile(question="Where is my refund?") + chat_prompt({"role": "system", "content": "You are a helpful assistant."}).compile() - assert prompt_attributes_for(compiled) == { - "td.prompt.name": "support/reply", - "td.prompt.version": 12, - "td.prompt.label": "production", + unrelated = [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What is 2 + 2?"}, + ] + assert prompt_attributes_for(unrelated) == {} + + +def test_two_tasks_with_one_text_and_different_other_variables_each_record_their_own( + make: MakeClient, +) -> None: + memory = make() + prompt = chat_prompt(*DESK_MESSAGES) + + # Both compiles make the system text "Answer as the billing desk.", and each task sends only it. + system_a = prompt.compile(team="billing", order="#A")[0] + system_b = prompt.compile(team="billing", order="#B")[0] + start_span("a", type="generation", attributes=prompt_attributes_for([system_a])).end() + start_span("b", type="generation", attributes=prompt_attributes_for([system_b])).end() + + assert { + span.name: (span.attributes or {})["td.prompt.variables"] + for span in memory.span_exporter.get_finished_spans() + } == { + "a": '{"team":"billing","order":"#A"}', + "b": '{"team":"billing","order":"#B"}', } - assert prompt_attributes_for([{"role": "user", "content": "Where is my refund?"}]) == {} - summary = text_prompt("Summarize {{doc}} in a line.").compile(doc="the refund policy") - assert prompt_attributes_for(summary)["td.prompt.variables"] == '{"doc":"the refund policy"}' +def test_a_rollout_with_two_versions_of_the_same_messages_records_the_version_of_each_call( + server: CaptureServer, make: MakeClient, monkeypatch: pytest.MonkeyPatch +) -> None: + make() + server.replies = [ + ( + 200, + reply( + messages=DESK_MESSAGES, + rollout={ + "percent": 16, + "version": 13, + "messages": DESK_MESSAGES, + "text": None, + "model": {"provider": "openai", "name": "gpt-4o-mini"}, + "parameters": {}, + "tools": [], + "responseFormat": {"type": "text"}, + }, + ), + ) + ] + draws = iter([15, 16]) + + def draw(_stop: int) -> int: + return next(draws) + + monkeypatch.setattr(prompts_module.random, "randrange", draw) + rollout = get_prompt("support/reply", api_key="td_live_a", api_url=server.url) + production = get_prompt("support/reply", api_key="td_live_a", api_url=server.url) + from_rollout = rollout.compile(team="billing", order="#4") + from_production = production.compile(team="billing", order="#4") + + assert prompt_attributes_for(from_rollout)["td.prompt.version"] == 13 + assert prompt_attributes_for(from_production)["td.prompt.version"] == 12 -def test_a_text_with_variables_wins_over_a_static_text_and_the_first_one_wins_over_a_later_one( + +def test_the_newest_compiled_message_wins_also_after_170_messages_of_history( make: MakeClient, ) -> None: make() - system = chat_prompt( - {"role": "system", "content": "You are the refunds agent, v2."}, name="refunds/system" - ).compile() - question = chat_prompt( - {"role": "user", "content": "Where is order {{order}}?"}, name="orders/question" - ).compile(order="#4") - style = chat_prompt( - {"role": "system", "content": "Reply in {{language}}."}, name="style/language" - ).compile(language="French") - - first = prompt_attributes_for([*system, *question]) - assert (first["td.prompt.name"], first["td.prompt.variables"]) == ( - "orders/question", - '{"order":"#4"}', + prompt = chat_prompt( + {"role": "system", "content": "Answer as the {{team}} desk."}, + {"role": "placeholder", "name": "history"}, + {"role": "user", "content": "{{question}}"}, ) - second = prompt_attributes_for([*system, *style, *question]) - assert (second["td.prompt.name"], second["td.prompt.variables"]) == ( - "style/language", - '{"language":"French"}', + history = [ + {"role": "user" if index % 2 == 0 else "assistant", "content": f"message {index}"} + for index in range(170) + ] + + first_turn = prompt.compile(team="billing", question="Where is #4?", history=history) + second_turn = prompt.compile(team="returns", question="Can I return it?") + chat = [*first_turn, {"role": "assistant", "content": "On its way."}, *second_turn[1:]] + + assert '"team":"billing"' in str(prompt_attributes_for(first_turn)["td.prompt.variables"]) + assert '"team":"billing"' in str( + prompt_attributes_for([first_turn[0], *history])["td.prompt.variables"] + ) + assert ( + prompt_attributes_for(chat)["td.prompt.variables"] + == '{"team":"returns","question":"Can I return it?"}' ) -def test_each_compile_keeps_its_own_values_so_two_threads_with_one_prompt_get_their_own( - make: MakeClient, -) -> None: +def test_prompt_attributes_for_looks_at_the_last_1000_messages(make: MakeClient) -> None: make() - prompt = chat_prompt({"role": "user", "content": "Track parcel {{parcel}}."}) - - first = prompt.compile(parcel="A-1") - second = prompt.compile(parcel="B-2") + system = chat_prompt(*DESK_MESSAGES).compile(team="billing", order="#4")[0] - assert prompt_attributes_for(first)["td.prompt.variables"] == '{"parcel":"A-1"}' - assert prompt_attributes_for(second)["td.prompt.variables"] == '{"parcel":"B-2"}' + assert prompt_attributes_for([system, *[{"role": "user"}] * 999]) != {} + assert prompt_attributes_for([system, *[{"role": "user"}] * 1000]) == {} -def test_with_prompt_wins_over_a_compiled_text_of_another_prompt(make: MakeClient) -> None: +def test_with_prompt_wins_over_a_compiled_message_of_another_prompt(make: MakeClient) -> None: memory = make() scoped = chat_prompt({"role": "user", "content": "Hi"}, name="scoped/prompt") compiled = chat_prompt( @@ -940,7 +988,7 @@ def test_with_prompt_wins_over_a_compiled_text_of_another_prompt(make: MakeClien assert (span.attributes or {})["td.prompt.name"] == "scoped/prompt" -def test_prompt_attributes_for_omits_variables_when_input_capture_is_off_and_masks_them( +def test_prompt_variables_follow_the_input_capture_of_the_client_and_of_the_span( make: MakeClient, ) -> None: compiled = chat_prompt({"role": "user", "content": "My card is {{card}}."}).compile( @@ -959,69 +1007,26 @@ def test_prompt_attributes_for_omits_variables_when_input_capture_is_off_and_mas def mask(value: Any, context: telemetry_dev.MaskContext) -> Any: return {"card": "[masked]"} if context.key == "td.prompt.variables" else value - make(mask=mask) - assert prompt_attributes_for(compiled)["td.prompt.variables"] == '{"card":"[masked]"}' - - -def test_the_kept_texts_stop_at_1000_and_at_2000000_characters_oldest_out_first( - make: MakeClient, -) -> None: - make() - ticket = text_prompt("Ticket {{n}} is open.") - for n in range(1001): - ticket.compile(n=n) - - assert prompt_attributes_for("Ticket 0 is open.") == {} - assert prompt_attributes_for("Ticket 1 is open.")["td.prompt.variables"] == '{"n":1}' - - note = text_prompt("Note: {{body}}") - notes = [note.compile(body=letter * (500_000 - len("Note: "))) for letter in "abcde"] - - assert prompt_attributes_for(notes[0]) == {} - assert prompt_attributes_for(notes[1])["td.prompt.name"] == "support/reply" - - # A text over the character limit is not kept, and it does not push the others out. - note.compile(body="f" * 2_000_001) - assert prompt_attributes_for(notes[1])["td.prompt.name"] == "support/reply" - - # A text compiled again counts once. - for _ in range(5): - note.compile(body="e" * (500_000 - len("Note: "))) - assert prompt_attributes_for(notes[4])["td.prompt.name"] == "support/reply" - - -def test_prompt_attributes_for_looks_6_levels_and_500_values_deep(make: MakeClient) -> None: - make() - [message] = chat_prompt({"role": "user", "content": "Find invoice {{invoice}}."}).compile( - invoice="INV-7" - ) - content = message["content"] - - def nest(value: object, levels: int) -> object: - return value if levels == 0 else nest([value], levels - 1) + memory = make(mask=mask) + attributes = prompt_attributes_for(compiled) + assert attributes["td.prompt.variables"] == '{"card":"[masked]"}' - assert prompt_attributes_for(nest(content, 6)) != {} - assert prompt_attributes_for(nest(content, 7)) == {} + start_span("private", type="generation", capture_input=False, attributes=attributes).end() + with start_span("updated", type="generation", capture_input=False): + telemetry_dev.update_current_span(attributes=attributes) - # The request list, 498 strings, and the text are 500 values. - assert prompt_attributes_for([*["filler"] * 498, content]) != {} - assert prompt_attributes_for([*["filler"] * 499, content]) == {} + for span in memory.span_exporter.get_finished_spans(): + assert (span.attributes or {})["td.prompt.name"] == "support/reply" + assert "td.prompt.variables" not in (span.attributes or {}) -def test_prompt_attributes_for_runs_no_property_or_override_of_the_app(make: MakeClient) -> None: +def test_prompt_attributes_for_runs_no_code_of_the_app(make: MakeClient) -> None: make() [message] = chat_prompt({"role": "user", "content": "Refund order {{order}}."}).compile( order="#3" ) - content = message["content"] reads: list[str] = [] - class Message: - @property - def content(self) -> str: - reads.append("property") - return content - class Messages(list[object]): def __getitem__(self, index: Any) -> Any: reads.append("__getitem__") @@ -1031,47 +1036,36 @@ def __iter__(self) -> Any: reads.append("__iter__") return super().__iter__() - assert prompt_attributes_for([Message(), Messages([content])]) == {} + def __reversed__(self) -> Any: + reads.append("__reversed__") + return super().__reversed__() + + assert prompt_attributes_for(Messages([message])) == {} assert reads == [] -@pytest.mark.skipif(not hasattr(os, "fork"), reason="needs os.fork") -# Python 3.12+ warns on fork() in a process with threads. -@pytest.mark.filterwarnings("ignore:This process .* is multi-threaded:DeprecationWarning") -def test_a_fork_while_a_thread_keeps_a_compiled_text_leaves_compile_usable_in_the_child() -> None: - prompt = text_prompt("Ship order {{order}}.") - texts = prompts_module._compiled_texts # pyright: ignore[reportPrivateUsage] - holding, release = threading.Event(), threading.Event() - - def hold() -> None: - with texts.lock: - holding.set() - release.wait(0.2) - - holder = threading.Thread(target=hold) - holder.start() - holding.wait() - read_end, write_end = os.pipe() - pid = os.fork() - if pid == 0: - try: - os.close(read_end) - os.write(write_end, prompt.compile(order="#1").encode()) - finally: - os._exit(0) - os.close(write_end) - release.set() - holder.join() +def test_the_variables_of_a_compile_are_freed_with_its_messages(make: MakeClient) -> None: + make() + prompt = chat_prompt( + {"role": "system", "content": "Answer as the {{team}} desk."}, + {"role": "placeholder", "name": "history"}, + ) - deadline = time.monotonic() + 5 - while os.waitpid(pid, os.WNOHANG) == (0, 0): - if time.monotonic() > deadline: - os.kill(pid, signal.SIGKILL) - os.waitpid(pid, 0) - pytest.fail("compile() in the child waited for a lock held at the fork") - time.sleep(0.01) - with os.fdopen(read_end, "rb") as child: - assert child.read() == b"Ship order #1." + class History(list[dict[str, str]]): + """A list that a weak reference can point to.""" + + history = History([{"role": "user", "content": "Hi"}]) + freed = weakref.ref(history) + messages = prompt.compile(team="billing", history=history) + del history + # with_prompt keeps the variables of the last compile of a prompt object. + prompt.compile(team="returns") + gc.collect() + assert freed() is not None + + del messages + gc.collect() + assert freed() is None def test_a_fallback_prompt_records_its_name_and_label_but_no_version( From a6b98fe916d5ed4b37bddd7972b07507182f2d2d Mon Sep 17 00:00:00 2001 From: ephraimduncan Date: Thu, 29 Oct 2026 16:20:12 +0000 Subject: [PATCH 3/9] fix(sdk): name the newest compile, not the last tagged message A compile tag holds a serial number that increases with each chat compile. promptAttributesFor and prompt_attributes_for give the tag with the highest number. Thus a message that a placeholder adds from an older compile of a different prompt does not win over the compile that made the list. They look at the first 1000 and the last 1000 messages. A template that ends with a placeholder has its own messages at the start. The TypeScript compile() keeps a copy of the top level of the variables, as Python does. Thus a later change to the object of the app does not change the variables that a message or a withPrompt span records. The Python compiled message keeps the constructor of dict. Thus type(message)(message) and dataclasses.asdict make a copy with all of the keys and without the tag. A model-call span with a parent context in withPrompt or with_prompt removes the td.prompt.* attributes that the app gives. Thus the span does not mix two prompts when the app calls promptAttributesFor outside the scope and gives an explicit parent. The READMEs and docs/sdk-conformance.md tell that the Python messages are a dict subclass, which yaml.safe_dump, marshal, and checks of the exact type reject. They also tell that a tagged message keeps the variables of its compile in memory. We corrected the docstring of prompt_attributes_for: with input capture on, it calls the mask of the app. Refs #97 --- docs/sdk-conformance.md | 47 ++++--- packages/sdk/README.md | 36 ++++-- packages/sdk/src/prompts.ts | 51 +++++--- packages/sdk/src/span.ts | 11 +- packages/sdk/tests/prompts.test.ts | 148 ++++++++++++++++++++-- sdks/python-openai/tests/test_openai.py | 4 - sdks/python/README.md | 46 ++++--- sdks/python/src/telemetry_dev/_prompts.py | 72 +++++++---- sdks/python/src/telemetry_dev/_spans.py | 12 +- sdks/python/tests/test_prompts.py | 123 ++++++++++++++++-- 10 files changed, 433 insertions(+), 117 deletions(-) diff --git a/docs/sdk-conformance.md b/docs/sdk-conformance.md index 610c0c1..e3e6fec 100644 --- a/docs/sdk-conformance.md +++ b/docs/sdk-conformance.md @@ -44,7 +44,7 @@ repository, not this SDK workspace. Paths below refer to that product repository | outside a scope: a message object that `compile()` of a chat prompt returned, in the request messages of a provider integration | `td.prompt.name`, `td.prompt.version`, `td.prompt.label` | on the `chat` spans of the provider integrations listed in [Prompts](#prompts); the scope wins | | `prompt.version` | `td.prompt.version` (int) | omitted for a fallback prompt (`version` null / None) | | `prompt.label` | `td.prompt.label` | only for a fetch by label; a fetch without label or version is the `production` label | -| variables of the last `prompt.compile()` before `withPrompt` / `with_prompt` starts, or in its scope; outside a scope, of the `compile()` that returned the message | `td.prompt.variables` (JSON object) | mask → JSON stringify → truncate; gated by the effective captureInput of the span | +| variables of the last `prompt.compile()` before `withPrompt` / `with_prompt` starts, or in its scope; outside a scope, of the newest `compile()` that made a message of the request | `td.prompt.variables` (JSON object) | mask → JSON stringify → truncate; gated by the effective captureInput of the span | `SpanHandle.serializeContent(value, key)` / `SpanHandle.serialize_content(value, key)` returns what `input` / `output` would emit for `value` under `key`: mask → JSON stringify → truncate. It @@ -182,23 +182,38 @@ task. Outside a scope, the SDKs find the prompt from the message objects of the request, not from their text: -- `compile()` of a chat prompt tags each message object that it makes with the prompt and the - variables of that call. It does not tag the messages of a placeholder variable: the app passes - those. TypeScript: a non-enumerable symbol property. Python: the messages are instances of a - `dict` subclass that holds the tag in a slot. The keys and the JSON of a message do not change. +- `compile()` of a chat prompt tags each message object that it makes with the prompt, a copy of + the top level of the variables of that call, and a serial number that grows with each chat + compile in the process. It does not tag the messages of a placeholder variable: the app passes + those, and they keep the tag of their own compile. TypeScript: a non-enumerable symbol + property; the keys and the JSON of a message do not change. Python: the messages are instances + of a private `dict` subclass that holds the tag in a slot. `isinstance(message, dict)` is true + and the keys and the JSON do not change, but a check of the exact type + (`type(message) is dict`), `yaml.safe_dump`, and `marshal` reject it. Its constructor is the + one of `dict`. - A copy has no tag: a spread, `structuredClone`, or a JSON round trip (TypeScript); `dict()`, - `.copy()`, `copy.copy`, `copy.deepcopy`, pickle, or a JSON round trip (Python). A new message - with the same text has no tag either, thus a message from another call, task, or prompt version - is never recorded. + `.copy()`, `copy.copy`, `copy.deepcopy`, pickle, a JSON round trip, `type(message)(message)`, or + `dataclasses.asdict` (Python). A new message with the same text has no tag either, thus a + message from another call, task, or prompt version is never recorded. - `promptAttributesFor(messages)` / `prompt_attributes_for(messages)` take a list of messages or - one message (Python: a `list` or `tuple`, not a subclass, so no code of the app runs). They - look at the last 1000 items, from the end, and give the name, version, label, and variables of - the newest tagged message. They do not look inside a message. -- In a `withPrompt` / `with_prompt` scope they give nothing, thus the span gets the prompt of the - scope. They never throw. -- They give `td.prompt.variables` only when input capture is on in the client. A span whose - effective captureInput is off drops `td.prompt.variables` from its attributes, also when the app - passes the result of these functions to its own span. + one message (Python: a `list` or `tuple`, not a subclass, so the read runs no code of the app). + They look at the first 1000 and the last 1000 items and give the name, version, label, and + variables of the tag with the highest serial number: the newest compile. Thus a message that a + placeholder brings from an older compile does not win over the compile that put it in the list. + They do not look inside a message. +- In a `withPrompt` / `with_prompt` scope they give nothing. A model-call span whose parent context + is in a scope gets the attributes of the scope and drops each `td.prompt.*` attribute that the + app passes, also when the app calls these functions outside the scope and passes an explicit + parent. They never throw: an error goes to `onError` / `on_error`. +- They give `td.prompt.variables` only when input capture is on in the client. Then they + serialize the variables, which calls the `mask` of the client. A span whose effective + captureInput is off drops `td.prompt.variables` from its attributes, also when the app passes + the result of these functions to its own span. +- A tagged message keeps the variables of its compile alive while the message lives, with the + lists of its placeholder variables. Thus a chat history that keeps a compiled message of each + turn keeps the history list of each turn. The SDKs keep no other reference to the variables, + except those of the last compile of each prompt object and of each running scope, for + `withPrompt` / `with_prompt`. - `compile()` of a text prompt returns a string, which cannot carry a tag. Outside a scope, the SDKs do not record a text prompt: use `withPrompt` / `with_prompt`. - The provider integrations call them with the request messages before the span starts, also diff --git a/packages/sdk/README.md b/packages/sdk/README.md index b73d74c..1fe9131 100644 --- a/packages/sdk/README.md +++ b/packages/sdk/README.md @@ -182,19 +182,29 @@ and TanStack AI integrations, and from other OpenTelemetry tracers, do not get t Outside `withPrompt`, those provider integrations record the prompt of the message objects that `compile()` returned. `compile()` of a chat prompt tags each message that it makes with its prompt -and variables, in a non-enumerable symbol property: the keys and the JSON of the message do not -change. A chat call that sends a tagged message gets `td.prompt.name`, `td.prompt.version`, and -`td.prompt.label` of the newest one (of the last 1000 messages), also when input capture is off, -and `td.prompt.variables` when it is on. A copy of a message (a spread, `structuredClone`, or -JSON) and a new message with the same text have no tag, so the SDK does not record them. Bedrock -Converse and Google GenAI take another message shape: change the compiled messages in place, or -use `withPrompt`. A text prompt compiles to a string, which cannot carry a tag: use `withPrompt`. -`withPrompt` wins over the tag. The integrations do it for OpenAI chat completions and responses, -Anthropic messages, Bedrock Converse, Google GenAI `generateContent`, and OpenRouter chat and -responses, but not for Bedrock InvokeModel (its body is bytes), embeddings, images, audio, video, -or OpenAI Realtime. For your own spans, `promptAttributesFor(messages)` gives the attributes to -pass as `attributes`; it gives nothing inside `withPrompt`, and a span with `captureInput: false` -drops `td.prompt.variables`. +and a copy of its variables, in a non-enumerable symbol property: the keys and the JSON of the +message do not change. A chat call that sends tagged messages gets `td.prompt.name`, +`td.prompt.version`, and `td.prompt.label` of the newest `compile()` among them, also when input +capture is off, and `td.prompt.variables` when it is on. The SDK looks at the first 1000 and the +last 1000 messages of the call. A message from a placeholder variable keeps the tag of its own, +older `compile()`, so it does not win over the `compile()` that put it in the list. A copy of a +message (a spread, `structuredClone`, or JSON) and a new message with the same text have no tag, +so the SDK does not record them. Bedrock Converse and Google GenAI take another message shape: +change the compiled messages in place, or use `withPrompt`. A text prompt compiles to a string, +which cannot carry a tag: use `withPrompt`. `withPrompt` wins over the tag. The integrations do it +for OpenAI chat completions and responses, Anthropic messages, Bedrock Converse, Google GenAI +`generateContent`, and OpenRouter chat and responses, but not for Bedrock InvokeModel (its body is +bytes), embeddings, images, audio, video, or OpenAI Realtime. + +A tagged message keeps the variables of its `compile()` alive while the message lives, with the +lists of its placeholder variables. If you keep compiled messages in a long chat history, put +copies (`{ ...message }`) in the history: else each turn keeps the history list of its own turn. + +For your own spans, `promptAttributesFor(messages)` gives the attributes to pass as `attributes`. +It gives nothing inside `withPrompt`, and a span with `captureInput: false` drops +`td.prompt.variables`. A model call whose parent is in `withPrompt` gets the prompt of the +`withPrompt` and none of the `td.prompt.*` attributes that you pass, also when its `parent` option +leads into a `withPrompt` that the call is not in. ## API diff --git a/packages/sdk/src/prompts.ts b/packages/sdk/src/prompts.ts index 48cfe27..8202159 100644 --- a/packages/sdk/src/prompts.ts +++ b/packages/sdk/src/prompts.ts @@ -97,8 +97,9 @@ export interface ChatPrompt extends PromptBase { /** * Fills in the variables and replaces each placeholder message with the messages in its * variable. `withPrompt` records the variables of the last call before it starts or inside it. - * It tags each message object that it makes with this prompt and these variables, so that a - * model call outside `withPrompt` that sends the message records them; see `promptAttributesFor`. + * It tags each message object that it makes with this prompt and a copy of these variables, so + * that a model call outside `withPrompt` that sends the message records them; see + * `promptAttributesFor`. The message keeps the variables alive while it lives. */ compile(variables?: PromptVariables): (ChatMessage | M)[]; } @@ -205,6 +206,8 @@ interface PromptScope { interface CompileTag { readonly prompt: Prompt; readonly variables: PromptVariables; + /** Counts the compiles of the process, so a higher one is newer. */ + readonly serial: number; } const FETCH_TIMEOUT_MS = 10_000; @@ -220,13 +223,14 @@ const MODEL_CALL_OPERATIONS = new Set(["chat", "text_completion", "gene const PROMPT_KEY = createContextKey("telemetry.dev prompt"); // Not enumerable, so JSON.stringify, spreads, and structuredClone leave the tag out of a copy. const COMPILED_BY = Symbol("telemetry.dev compiled by"); -/** How many messages promptAttributesFor looks at, from the end. */ -const MAX_TAG_LOOKUP = 1_000; +/** How many messages promptAttributesFor looks at, at each end of the list. */ +const TAG_LOOKUP = 1_000; const cache = new Map(); const compiledVariables = new WeakMap(); /** The Retry-After of the response that failed with this error, in milliseconds. */ const retryAfter = new WeakMap(); +let compiles = 0; /** * Fetches a prompt version from telemetry.dev and keeps it. When the copy is older than @@ -368,11 +372,11 @@ export function promptAttributes( /** * The td.prompt.* attributes for a model call outside `withPrompt` that sends `messages` (a list - * of messages, or one message): those of the newest message, of the last 1000, that `compile()` - * of a chat prompt returned. The tag is on the message object, so a copy of it gives nothing. Gives - * undefined inside `withPrompt`, whose prompt the span gets. `td.prompt.variables` needs input - * capture on in the client, and the span drops it when its own input capture is off. It never - * throws. + * of messages, or one message): those of the newest `compile()` of a chat prompt that made one of + * the messages. It looks at the first 1000 and the last 1000 messages. The tag is on the message + * object, so a copy of it gives nothing. Gives undefined inside `withPrompt`, whose prompt the span + * gets. `td.prompt.variables` needs input capture on in the client, and the span drops it when its + * own input capture is off. It never throws. */ export function promptAttributesFor(messages: unknown): Record | undefined { const config = currentClient().core?.config; @@ -408,17 +412,22 @@ function attributesOf( }); } +// The messages of a placeholder come from an older compile, so the newest compile, not the last +// tagged message, names the prompt of the request. Its own messages are at the start or the end of +// the list, unless placeholders put more than 1000 messages on each side of them. function newestTag(messages: unknown): CompileTag | undefined { if (!Array.isArray(messages)) return tagOf(messages); - const stop = Math.max(0, messages.length - MAX_TAG_LOOKUP); + const { length } = messages; + let newest: CompileTag | undefined; - for (let index = messages.length - 1; index >= stop; index -= 1) { + for (let index = 0; index < length; index += 1) { + if (index === TAG_LOOKUP && length > 2 * TAG_LOOKUP) index = length - TAG_LOOKUP; const tag = tagOf(messages[index]); - if (tag) return tag; + if (tag && (newest === undefined || tag.serial > newest.serial)) newest = tag; } - return undefined; + return newest; } function tagOf(message: unknown): CompileTag | undefined { @@ -613,14 +622,20 @@ function createPrompt(input: { responseFormat: content.responseFormat, isFallback: input.isFallback, compile(variables: PromptVariables = {}) { - compiledVariables.set(prompt, variables); + // A copy, so that a later change of the app's object does not change what a span records. + const values = { ...variables }; + compiledVariables.set(prompt, values); const scope = activeContext().getValue(PROMPT_KEY) as PromptScope | undefined; - if (scope?.prompt === prompt) scope.variables = variables; + if (scope?.prompt === prompt) scope.variables = values; - return type === "chat" - ? compileMessages(content.messages, Object.freeze({ prompt, variables })) - : fill(content.text ?? "", variables); + if (type === "text") return fill(content.text ?? "", values); + compiles += 1; + + return compileMessages( + content.messages, + Object.freeze({ prompt, variables: values, serial: compiles }), + ); }, }) as Prompt; diff --git a/packages/sdk/src/span.ts b/packages/sdk/src/span.ts index 4c4c38a..cf0c374 100644 --- a/packages/sdk/src/span.ts +++ b/packages/sdk/src/span.ts @@ -231,6 +231,15 @@ export function createSpanHandle( const parentCtx = withSessionParent(sessionContext, sessionId, core.config.apiKey); const attrs = fieldsToAttributes(options, meta.type, meta.capture); const operation = attrs["gen_ai.operation.name"] ?? SPAN_TYPE_OPERATIONS[meta.type]; + const prompt = promptAttributes(parentCtx, operation, meta.capture); + + // The prompt of withPrompt replaces all td.prompt.* attributes of the app, so that a span never + // mixes two prompts, also when its parent is in withPrompt and promptAttributesFor ran outside. + if (prompt) { + for (const key of Object.keys(attrs)) { + if (key.startsWith("td.prompt.")) delete attrs[key]; + } + } const span = core.tracer.startSpan( name, @@ -241,7 +250,7 @@ export function createSpanHandle( attributes: { "gen_ai.operation.name": SPAN_TYPE_OPERATIONS[meta.type], ...propagatedFromContext(parentCtx), - ...promptAttributes(parentCtx, operation, meta.capture), + ...prompt, ...attrs, }, }, diff --git a/packages/sdk/tests/prompts.test.ts b/packages/sdk/tests/prompts.test.ts index 8e68644..a10ebb6 100644 --- a/packages/sdk/tests/prompts.test.ts +++ b/packages/sdk/tests/prompts.test.ts @@ -1,5 +1,7 @@ import { createServer, type RequestListener } from "node:http"; import type { AddressInfo } from "node:net"; +import { setFlagsFromString } from "node:v8"; +import { runInNewContext } from "node:vm"; import { afterEach, beforeEach, expect, test, vi } from "vitest"; @@ -880,31 +882,157 @@ test("the newest compiled message wins, also after 170 messages of history", asy ); }); -test("promptAttributesFor looks at the last 1000 messages", async () => { +test("a message that a placeholder brings from an older compile does not win", async () => { + setup(); + const router = await promptOf({ + name: "router", + version: 1, + messages: [{ role: "user", content: "{{question}}" }], + }); + const answer = await promptOf({ + name: "answer", + version: 7, + messages: [ + { role: "system", content: "Answer for {{team}}." }, + { role: "placeholder", name: "history" }, + ], + }); + const chat = await promptOf({ + name: "chat", + messages: [{ role: "user", content: "{{question}}" }], + }); + const summary = await promptOf({ + name: "summary", + messages: [ + { role: "system", content: "Summarize the chat." }, + { role: "placeholder", name: "history" }, + ], + }); + + const routed = answer.compile({ + team: "billing", + history: router.compile({ question: "Where is #4?" }) as ChatMessage[], + }); + // The last 1000 messages are all from chat; the summary message is the first. + const conversation = Array.from( + { length: 1500 }, + (_, index) => chat.compile({ question: `question ${index}` }) as ChatMessage[], + ).flat(); + const summarized = summary.compile({ history: conversation }); + + expect(promptAttributesFor(routed)).toEqual({ + "td.prompt.name": "answer", + "td.prompt.version": 7, + "td.prompt.label": "production", + "td.prompt.variables": + '{"team":"billing","history":[{"role":"user","content":"Where is #4?"}]}', + }); + expect(promptAttributesFor(summarized)?.["td.prompt.name"]).toBe("summary"); +}); + +test("promptAttributesFor looks at the first and the last 1000 messages", async () => { setup(); const prompt = await promptOf(DESK_PROMPT); const [system] = prompt.compile({ team: "billing", order: "#4" }) as ChatMessage[]; const filler = (count: number) => Array.from({ length: count }, () => ({ role: "user" })); - expect(promptAttributesFor([system, ...filler(999)])).toBeDefined(); - expect(promptAttributesFor([system, ...filler(1000)])).toBeUndefined(); + expect(promptAttributesFor([...filler(1000), system, ...filler(999)])).toBeDefined(); + expect(promptAttributesFor([...filler(999), system, ...filler(1001)])).toBeDefined(); + expect(promptAttributesFor([...filler(1001), system, ...filler(999)])).toBeDefined(); + expect(promptAttributesFor([...filler(1000), system, ...filler(1000)])).toBeUndefined(); +}); + +test("compile records its variables, not later changes of the object", async () => { + const { spans } = setup(); + const prompt = await promptOf({ messages: [{ role: "user", content: "Order {{order}}" }] }); + const variables = { order: "#A" }; + + const first = prompt.compile(variables); + variables.order = "#B"; + const second = prompt.compile(variables); + variables.order = "#C"; + withPrompt(prompt, () => startSpan("scope", { type: "generation" }).end()); + await flush(); + + expect(promptAttributesFor(first)?.["td.prompt.variables"]).toBe('{"order":"#A"}'); + expect(promptAttributesFor(second)?.["td.prompt.variables"]).toBe('{"order":"#B"}'); + expect(spans.getFinishedSpans()[0]!.attributes["td.prompt.variables"]).toBe('{"order":"#B"}'); }); -test("withPrompt wins over a compiled message of another prompt", async () => { +test("the variables of a compile are freed with its messages", async () => { + setFlagsFromString("--expose-gc"); + const gc = runInNewContext("gc") as () => void; + // A WeakRef keeps its target until the current job ends, so collect in a new one. + const collect = async () => { + await new Promise((resolve) => setTimeout(resolve, 0)); + gc(); + }; + setup(); + const prompt = await promptOf({ + messages: [ + { role: "system", content: "Answer as the {{team}} desk." }, + { role: "placeholder", name: "history" }, + ], + }); + + let history: ChatMessage[] | undefined = [{ role: "user", content: "Hi" }]; + const freed = new WeakRef(history); + let messages = prompt.compile({ team: "billing", history }) as ChatMessage[] | undefined; + history = undefined; + // withPrompt keeps the variables of the last compile of a prompt object. + prompt.compile({ team: "returns" }); + await collect(); + expect(freed.deref()).toBeDefined(); + + expect(messages).toHaveLength(2); + messages = undefined; + await collect(); + expect(freed.deref()).toBeUndefined(); +}); + +test("withPrompt wins over a compiled message of another prompt, also through a parent", async () => { const { spans } = setup(); const scoped = await promptOf({ name: "scoped/prompt" }); - const other = await promptOf({ - name: "other/prompt", - messages: [{ role: "user", content: "Cancel order {{order}}." }], + // Fetched by version, so it has no label that the label of the scope could hide. + const { fetch } = fakeFetch( + reply({ + name: "other/prompt", + label: null, + version: 3, + messages: [{ role: "user", content: "Cancel order {{order}}." }], + }), + ); + const other = await getPrompt("other/prompt", { + apiKey: "td_live_a", + apiUrl: newApiUrl(), + version: 3, + fetch, }); const compiled = other.compile({ order: "#9" }); - withPrompt(scoped, () => { - startSpan("chat", { type: "generation", attributes: promptAttributesFor(compiled) }).end(); + const parent = withPrompt(scoped, () => { + startSpan("in scope", { type: "generation", attributes: promptAttributesFor(compiled) }).end(); + + return startSpan("parent"); }); + const attributes = promptAttributesFor(compiled); + startSpan("with parent", { type: "generation", parent, attributes }).end(); + parent.end(); await flush(); - expect(spans.getFinishedSpans()[0]!.attributes["td.prompt.name"]).toBe("scoped/prompt"); + expect(attributes?.["td.prompt.name"]).toBe("other/prompt"); + for (const name of ["in scope", "with parent"]) { + const span = spans.getFinishedSpans().find((finished) => finished.name === name)!; + expect( + Object.fromEntries( + Object.entries(span.attributes).filter(([key]) => key.startsWith("td.prompt.")), + ), + ).toEqual({ + "td.prompt.name": "scoped/prompt", + "td.prompt.version": 12, + "td.prompt.label": "production", + }); + } }); test("td.prompt.variables follows the input capture of the client and of the span", async () => { diff --git a/sdks/python-openai/tests/test_openai.py b/sdks/python-openai/tests/test_openai.py index 1ad3a36..55ab3c6 100644 --- a/sdks/python-openai/tests/test_openai.py +++ b/sdks/python-openai/tests/test_openai.py @@ -1222,10 +1222,6 @@ def text(self) -> str: ) -def prompt_attrs(span: ReadableSpan) -> dict[str, object]: - return {key: value for key, value in attrs(span).items() if key.startswith("td.prompt.")} - - @pytest.mark.parametrize( ("api", "messages", "capture_input", "older_core"), [ diff --git a/sdks/python/README.md b/sdks/python/README.md index e39a537..0345367 100644 --- a/sdks/python/README.md +++ b/sdks/python/README.md @@ -254,22 +254,36 @@ integrations: OpenAI, Anthropic, Amazon Bedrock, Google GenAI, LiteLLM, and Open other OpenTelemetry tracers do not get the attributes. Outside `with_prompt`, those provider integrations record the prompt of the message dicts that -`compile()` returned. `compile()` of a chat prompt returns instances of a `dict` subclass that -also name its prompt and variables: the keys and the JSON of the message do not change. A chat -call that sends such a message gets `td.prompt.name`, `td.prompt.version`, and `td.prompt.label` -of the newest one (of the last 1000 messages), also when input capture is off, and -`td.prompt.variables` when it is on. A copy of a message (`dict()`, `.copy()`, `copy.copy`, -`copy.deepcopy`, pickle, or JSON) and a new message with the same text do not name the prompt, so -the SDK does not record them. Bedrock Converse and Google GenAI take another message shape: change -the compiled dicts in place, or use `with_prompt`. A text prompt compiles to a string, which -cannot name its prompt: use `with_prompt`. `with_prompt` wins over the message. The integrations -do it for OpenAI chat completions and responses, Anthropic messages (for `messages.stream()`, at -`__enter__`), Bedrock Converse, Google GenAI `generate_content`, LiteLLM completion and responses, -and OpenRouter chat and responses, but not for Bedrock InvokeModel (its body is bytes), embeddings, -images, audio, video, or OpenAI Realtime. With an older `telemetry-dev` that has no -`prompt_attributes_for`, a provider integration records the prompt only in `with_prompt`. For your -own spans, `prompt_attributes_for(messages)` gives the attributes to pass as `attributes`; it gives -`{}` inside `with_prompt`, and a span with `capture_input=False` drops `td.prompt.variables`. +`compile()` returned. `compile()` of a chat prompt returns instances of a private `dict` subclass +that also hold its prompt and a copy of its variables. A chat call that sends such messages gets +`td.prompt.name`, `td.prompt.version`, and `td.prompt.label` of the newest `compile()` among +them, also when input capture is off, and `td.prompt.variables` when it is on. The SDK looks at +the first 1000 and the last 1000 messages of the call. A message from a placeholder variable keeps +the prompt of its own, older `compile()`, so it does not win over the `compile()` that put it in +the list. A copy of a message (`dict()`, `.copy()`, `copy.copy`, `copy.deepcopy`, pickle, JSON, +`type(message)(message)`, or `dataclasses.asdict`) and a new message with the same text do not +hold the prompt, so the SDK does not record them. Bedrock Converse and Google GenAI take another +message shape: change the compiled dicts in place, or use `with_prompt`. A text prompt compiles to +a string, which cannot hold its prompt: use `with_prompt`. `with_prompt` wins over the message. +The integrations do it for OpenAI chat completions and responses, Anthropic messages (for +`messages.stream()`, at `__enter__`), Bedrock Converse, Google GenAI `generate_content`, LiteLLM +completion and responses, and OpenRouter chat and responses, but not for Bedrock InvokeModel (its +body is bytes), embeddings, images, audio, video, or OpenAI Realtime. With an older +`telemetry-dev` that has no `prompt_attributes_for`, a provider integration records the prompt +only in `with_prompt`. + +A compiled message is a `dict` for `isinstance`, and its keys and its JSON (`json`, `orjson`, +pydantic) do not change. A check of the exact type (`type(message) is dict`), `yaml.safe_dump`, +and `marshal` reject it: give them `dict(message)`. A compiled message keeps the variables of its +`compile()` alive while the message lives, with the lists of its placeholder variables. If you +keep compiled messages in a long chat history, put copies (`dict(message)`) in the history: else +each turn keeps the history list of its own turn. + +For your own spans, `prompt_attributes_for(messages)` gives the attributes to pass as +`attributes`. It gives `{}` inside `with_prompt`, and a span with `capture_input=False` drops +`td.prompt.variables`. A model call whose parent is in `with_prompt` gets the prompt of the +`with_prompt` and none of the `td.prompt.*` attributes that you pass, also when its `parent` +context comes from a `with_prompt` that the call is not in. ## Auto-metrics diff --git a/sdks/python/src/telemetry_dev/_prompts.py b/sdks/python/src/telemetry_dev/_prompts.py index cb230fd..28ee83e 100644 --- a/sdks/python/src/telemetry_dev/_prompts.py +++ b/sdks/python/src/telemetry_dev/_prompts.py @@ -4,6 +4,7 @@ import contextlib import copy import hashlib +import itertools import json import random import re @@ -50,8 +51,8 @@ _VARIABLE = re.compile(r"\{\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*\}\}") _MODEL_CALL_OPERATIONS = frozenset({"chat", "text_completion", "generate_content"}) _PROMPT_KEY = otel_context.create_key("telemetry_dev.prompt") -# How many messages prompt_attributes_for looks at, from the end. -_MAX_TAG_LOOKUP = 1000 +# How many messages prompt_attributes_for looks at, at each end of the list. +_TAG_LOOKUP = 1000 class PromptModel(TypedDict): @@ -100,16 +101,21 @@ def compile( ) -> list[dict[str, Any]]: """Fill in the variables and replace each placeholder message with the list of messages in its variable. ``with_prompt`` records the variables of the last call before it starts - or inside it. It tags each message dict that it makes with this prompt and these - variables, so that a model call outside ``with_prompt`` that sends the message records - them; see ``prompt_attributes_for``.""" + or inside it. Each message that it makes is a ``dict`` subclass that also holds this + prompt and a copy of these variables, so that a model call outside ``with_prompt`` that + sends the message records them; see ``prompt_attributes_for``. The message keeps the + variables alive while it lives. A check of the exact type (``type(message) is dict``), + ``yaml.safe_dump``, and ``marshal`` reject it: give them ``dict(message)``.""" values = _record(self, variables, kwargs) - tag = _CompileTag(self, values) + tag = _CompileTag(self, values, next(_compiles)) compiled: list[dict[str, Any]] = [] for message in self.messages: if message.get("role") != "placeholder": - content = _fill(str(message["content"]), values) - compiled.append(_CompiledMessage(tag, role=message["role"], content=content)) + tagged = _CompiledMessage( + role=message["role"], content=_fill(str(message["content"]), values) + ) + tagged.compiled_by = tag + compiled.append(tagged) continue name = str(message["name"]) value = values.get(name) @@ -294,11 +300,16 @@ def prompt_attributes( def prompt_attributes_for(messages: object) -> dict[str, AttributeValue]: """The td.prompt.* attributes for a model call outside ``with_prompt`` that sends ``messages`` - (a list or tuple of messages, or one message): those of the newest message, of the last 1000, - that ``compile()`` of a chat prompt returned. The tag is on the dict that ``compile()`` made, so - a copy of it gives ``{}``. Gives ``{}`` inside ``with_prompt``, whose prompt the span gets. - ``td.prompt.variables`` needs input capture on in the client, and the span drops it when its - own input capture is off. It runs no code of the app and never raises.""" + (a list or tuple of messages, or one message): those of the newest ``compile()`` of a chat + prompt that made one of the messages. It looks at the first 1000 and the last 1000 messages. + The tag is on the dict that ``compile()`` made, so a copy of it gives ``{}``. Gives ``{}`` + inside ``with_prompt``, whose prompt the span gets. ``td.prompt.variables`` needs input + capture on in the client, and the span drops it when its own input capture is off. + + It reads only a ``list`` or ``tuple``, not a subclass, so the read runs no code of the app. + With input capture on, it serializes the variables, which calls the ``mask`` of the client and + can run code of a value, such as ``__repr__``. It never raises: it reports an error to + ``on_error`` and gives ``{}``.""" client = get_client() try: if isinstance(otel_context.get_value(_PROMPT_KEY), _PromptScope): @@ -443,32 +454,43 @@ class _CompileTag: prompt: ChatPrompt variables: dict[str, Any] + serial: int + """Counts the compiles of the process, so a higher one is newer.""" + + +_compiles = itertools.count() class _CompiledMessage(dict[str, Any]): - """A message that ``compile()`` made: a dict with the tag in a slot. A copy, pickle, or - ``copy.deepcopy`` of it is a plain dict without the tag.""" + """A message that ``compile()`` made: a dict with the tag in a slot. The constructor is the one + of ``dict``, so ``type(message)(message)`` and ``dataclasses.asdict`` rebuild it, without the + tag. A copy, pickle, or ``copy.deepcopy`` of it is a plain dict without the tag.""" __slots__ = ("compiled_by",) - - def __init__(self, compiled_by: _CompileTag, **fields: Any) -> None: - super().__init__(**fields) - self.compiled_by = compiled_by + compiled_by: _CompileTag def __reduce__(self) -> tuple[type[dict[str, Any]], tuple[dict[str, Any]]]: return dict, (dict(self),) def _newest_tag(messages: object) -> _CompileTag | None: - # Exact types: a subclass of list or tuple could run code of the app on each read. + """The messages of a placeholder come from an older compile, so the newest compile, not the + last tagged message, names the prompt of the request. Its own messages are at the start or the + end of the list, unless placeholders put more than 1000 messages on each side of them.""" if type(messages) is _CompiledMessage: - return messages.compiled_by + return getattr(messages, "compiled_by", None) + # Exact types: a subclass of list or tuple could run code of the app on each read. if type(messages) is not list and type(messages) is not tuple: return None - for message in reversed(cast("list[object]", messages)[-_MAX_TAG_LOOKUP:]): - if type(message) is _CompiledMessage: - return message.compiled_by - return None + items = cast("list[object]", messages) + if len(items) > 2 * _TAG_LOOKUP: + items = [*items[:_TAG_LOOKUP], *items[-_TAG_LOOKUP:]] + newest: _CompileTag | None = None + for item in items: + tag = getattr(item, "compiled_by", None) if type(item) is _CompiledMessage else None + if tag is not None and (newest is None or tag.serial > newest.serial): + newest = tag + return newest def _fill(text: str, values: Mapping[str, Any]) -> str: diff --git a/sdks/python/src/telemetry_dev/_spans.py b/sdks/python/src/telemetry_dev/_spans.py index d66f086..1d6ac64 100644 --- a/sdks/python/src/telemetry_dev/_spans.py +++ b/sdks/python/src/telemetry_dev/_spans.py @@ -748,11 +748,15 @@ def start_span( ctx, session_id if isinstance(session_id, str) else None, client.config.api_key ) initial.update(propagated) - initial.update( - prompt_attributes( - client, ctx, attrs.get(ATTR_OPERATION, operation), state.capture_input - ) + prompt = prompt_attributes( + client, ctx, attrs.get(ATTR_OPERATION, operation), state.capture_input ) + if prompt: + # The prompt of with_prompt replaces all td.prompt.* attributes of the app, so that a + # span never mixes two prompts, also when its parent is in with_prompt and + # prompt_attributes_for ran outside. + attrs = {key: value for key, value in attrs.items() if not key.startswith("td.prompt.")} + initial.update(prompt) initial.update(attrs) span = client.tracer.start_span( name, diff --git a/sdks/python/tests/test_prompts.py b/sdks/python/tests/test_prompts.py index 29297e3..423286f 100644 --- a/sdks/python/tests/test_prompts.py +++ b/sdks/python/tests/test_prompts.py @@ -3,6 +3,7 @@ import asyncio import contextvars import copy +import dataclasses import gc import itertools import json @@ -22,6 +23,7 @@ from urllib.parse import parse_qsl, urlsplit import pytest +from opentelemetry import context as otel_context from opentelemetry import trace import telemetry_dev @@ -966,26 +968,127 @@ def test_the_newest_compiled_message_wins_also_after_170_messages_of_history( ) -def test_prompt_attributes_for_looks_at_the_last_1000_messages(make: MakeClient) -> None: +def test_a_message_that_a_placeholder_brings_from_an_older_compile_does_not_win( + make: MakeClient, +) -> None: + make() + router = chat_prompt({"role": "user", "content": "{{question}}"}, name="router") + answer = chat_prompt( + {"role": "system", "content": "Answer for {{team}}."}, + {"role": "placeholder", "name": "history"}, + name="answer", + ) + chat = chat_prompt({"role": "user", "content": "{{question}}"}, name="chat") + summary = chat_prompt( + {"role": "system", "content": "Summarize the chat."}, + {"role": "placeholder", "name": "history"}, + name="summary", + ) + + routed = answer.compile(team="billing", history=router.compile(question="Where is #4?")) + # The last 1000 messages are all from chat; the summary message is the first. + conversation = [chat.compile(question=f"question {index}")[0] for index in range(1500)] + summarized = summary.compile(history=conversation) + + assert prompt_attributes_for(routed) == { + "td.prompt.name": "answer", + "td.prompt.version": 12, + "td.prompt.label": "production", + "td.prompt.variables": ( + '{"team":"billing","history":[{"role":"user","content":"Where is #4?"}]}' + ), + } + assert prompt_attributes_for(summarized)["td.prompt.name"] == "summary" + + +def test_prompt_attributes_for_looks_at_the_first_and_the_last_1000_messages( + make: MakeClient, +) -> None: make() system = chat_prompt(*DESK_MESSAGES).compile(team="billing", order="#4")[0] - assert prompt_attributes_for([system, *[{"role": "user"}] * 999]) != {} - assert prompt_attributes_for([system, *[{"role": "user"}] * 1000]) == {} + def around(before: int, after: int) -> list[dict[str, Any]]: + return [*[{"role": "user"}] * before, system, *[{"role": "user"}] * after] + + assert prompt_attributes_for(around(1000, 999)) != {} + assert prompt_attributes_for(around(999, 1001)) != {} + assert prompt_attributes_for(tuple(around(1001, 999))) != {} + assert prompt_attributes_for(around(1000, 1000)) == {} + + +def test_compile_records_its_variables_not_later_changes_of_the_mapping( + make: MakeClient, +) -> None: + memory = make() + prompt = chat_prompt({"role": "user", "content": "Order {{order}}"}) + variables = {"order": "#A"} + + first = prompt.compile(variables) + variables["order"] = "#B" + second = prompt.compile(variables) + variables["order"] = "#C" + with with_prompt(prompt): + start_span("scope", type="generation").end() + + assert prompt_attributes_for(first)["td.prompt.variables"] == '{"order":"#A"}' + assert prompt_attributes_for(second)["td.prompt.variables"] == '{"order":"#B"}' + [span] = memory.span_exporter.get_finished_spans() + assert (span.attributes or {})["td.prompt.variables"] == '{"order":"#B"}' + + +def test_a_compiled_message_rebuilds_like_a_dict_and_the_rebuilt_one_has_no_tag( + make: MakeClient, +) -> None: + make() + compiled = chat_prompt(*DESK_MESSAGES).compile(team="billing", order="#4") + @dataclasses.dataclass + class Conversation: + messages: list[dict[str, Any]] -def test_with_prompt_wins_over_a_compiled_message_of_another_prompt(make: MakeClient) -> None: + rebuilt = dataclasses.asdict(Conversation(messages=compiled))["messages"] + same_type = [type(message)(message) for message in compiled] + plain = [ + {"role": "system", "content": "Answer as the billing desk."}, + {"role": "user", "content": "Order #4: where is it?"}, + ] + + assert rebuilt == plain + assert same_type == plain + assert prompt_attributes_for(rebuilt) == {} + assert prompt_attributes_for(same_type) == {} + + +def test_with_prompt_wins_over_a_compiled_message_of_another_prompt_also_through_a_parent( + make: MakeClient, +) -> None: memory = make() scoped = chat_prompt({"role": "user", "content": "Hi"}, name="scoped/prompt") - compiled = chat_prompt( - {"role": "user", "content": "Cancel order {{order}}."}, name="other/prompt" - ).compile(order="#9") + # Fetched by version, so it has no label that the label of the scope could hide. + other = dataclasses.replace( + chat_prompt({"role": "user", "content": "Cancel order {{order}}."}, name="other/prompt"), + version=3, + label=None, + ) + compiled = other.compile(order="#9") with with_prompt(scoped): - start_span("chat", type="generation", attributes=prompt_attributes_for(compiled)).end() + start_span("in scope", type="generation", attributes=prompt_attributes_for(compiled)).end() + parent = otel_context.get_current() + attributes = prompt_attributes_for(compiled) + start_span("with parent", type="generation", parent=parent, attributes=attributes).end() - [span] = memory.span_exporter.get_finished_spans() - assert (span.attributes or {})["td.prompt.name"] == "scoped/prompt" + assert attributes["td.prompt.name"] == "other/prompt" + for span in memory.span_exporter.get_finished_spans(): + assert { + key: value + for key, value in dict(span.attributes or {}).items() + if key.startswith("td.prompt.") + } == { + "td.prompt.name": "scoped/prompt", + "td.prompt.version": 12, + "td.prompt.label": "production", + } def test_prompt_variables_follow_the_input_capture_of_the_client_and_of_the_span( From 3bc6f53a0afb0ad8b9fe19f8eaf7e5d812a93b0a Mon Sep 17 00:00:00 2001 From: ephraimduncan Date: Thu, 29 Oct 2026 20:15:11 +0000 Subject: [PATCH 4/9] fix(sdk): keep the variables of recent compiles only A compiled message kept the variables of its compile, with the lists in them. Thus a chat that keeps a compiled message of each turn in a new history list for each turn kept the history list of every turn. Memory increased with the square of the turn count: 49 MB in TypeScript and 38 MB in Python after 2000 turns, also with input capture off. compile() can get a message of an older compile in a variable, or in a list in a variable. If the older compile is 100 or more compiles older, it discards its variables. The newest compile identifies the prompt of a request, thus a call still records the variables of its own compile. A recent compile keeps its variables when a newer compile gets its messages. Thus a judge compile that gets the messages before their call does not remove them. After 2000 turns, the SDKs keep 3.7 MB (TypeScript) and 4.4 MB (Python) with input capture on, and 0.7 MB and 1.3 MB with it off. These numbers include the history of the app. If input capture is off in the client, compile() keeps no variables in the tag, because promptAttributesFor and prompt_attributes_for cannot record them. withPrompt and with_prompt keep the variables of the last compile of each prompt object, as before. compile() also copies each list in the variables. Thus a change to the history of the app after compile() does not change the variables that a call records. compile() does not copy the objects in a list, and the READMEs and docs/sdk-conformance.md tell this. A span that gets its prompt from withPrompt or with_prompt discards the td.prompt.* attributes of the app at start, and also in update, end, updateActiveSpan, and update_current_span. Refs #97 --- docs/sdk-conformance.md | 43 +++--- packages/sdk/README.md | 45 +++--- packages/sdk/src/prompts.ts | 68 +++++++-- packages/sdk/src/span.ts | 23 +++- packages/sdk/tests/prompts.test.ts | 160 ++++++++++++++++++---- sdks/python/README.md | 52 ++++--- sdks/python/src/telemetry_dev/_prompts.py | 57 ++++++-- sdks/python/src/telemetry_dev/_spans.py | 18 ++- sdks/python/tests/test_prompts.py | 140 ++++++++++++++++--- 9 files changed, 459 insertions(+), 147 deletions(-) diff --git a/docs/sdk-conformance.md b/docs/sdk-conformance.md index e3e6fec..b49779a 100644 --- a/docs/sdk-conformance.md +++ b/docs/sdk-conformance.md @@ -183,14 +183,15 @@ Outside a scope, the SDKs find the prompt from the message objects of the reques their text: - `compile()` of a chat prompt tags each message object that it makes with the prompt, a copy of - the top level of the variables of that call, and a serial number that grows with each chat - compile in the process. It does not tag the messages of a placeholder variable: the app passes - those, and they keep the tag of their own compile. TypeScript: a non-enumerable symbol - property; the keys and the JSON of a message do not change. Python: the messages are instances - of a private `dict` subclass that holds the tag in a slot. `isinstance(message, dict)` is true - and the keys and the JSON do not change, but a check of the exact type - (`type(message) is dict`), `yaml.safe_dump`, and `marshal` reject it. Its constructor is the - one of `dict`. + the variables of that call and of each list in them (not of the objects in a list), and a serial + number that grows with each chat compile in the process. With input capture off in the client + at that time, the tag holds no variables. It does not tag the messages of a placeholder + variable: the app passes those, and they keep the tag of their own compile. TypeScript: a + non-enumerable symbol property; the keys and the JSON of a message do not change. Python: the + messages are instances of a private `dict` subclass that holds the tag in a slot. + `isinstance(message, dict)` is true and the keys and the JSON do not change, but a check of the + exact type (`type(message) is dict`), `yaml.safe_dump`, and `marshal` reject it. Its + constructor is the one of `dict`. - A copy has no tag: a spread, `structuredClone`, or a JSON round trip (TypeScript); `dict()`, `.copy()`, `copy.copy`, `copy.deepcopy`, pickle, a JSON round trip, `type(message)(message)`, or `dataclasses.asdict` (Python). A new message with the same text has no tag either, thus a @@ -203,17 +204,21 @@ their text: They do not look inside a message. - In a `withPrompt` / `with_prompt` scope they give nothing. A model-call span whose parent context is in a scope gets the attributes of the scope and drops each `td.prompt.*` attribute that the - app passes, also when the app calls these functions outside the scope and passes an explicit - parent. They never throw: an error goes to `onError` / `on_error`. -- They give `td.prompt.variables` only when input capture is on in the client. Then they - serialize the variables, which calls the `mask` of the client. A span whose effective - captureInput is off drops `td.prompt.variables` from its attributes, also when the app passes - the result of these functions to its own span. -- A tagged message keeps the variables of its compile alive while the message lives, with the - lists of its placeholder variables. Thus a chat history that keeps a compiled message of each - turn keeps the history list of each turn. The SDKs keep no other reference to the variables, - except those of the last compile of each prompt object and of each running scope, for - `withPrompt` / `with_prompt`. + app passes, at start and in each update (`update`, `end`, `updateActiveSpan` / + `update_current_span`), also when the app calls these functions outside the scope and passes an + explicit parent. They never throw: an error goes to `onError` / `on_error`. +- They give `td.prompt.variables` only when input capture is on in the client, at the compile and + at the call. Then they serialize the variables, which calls the `mask` of the client. A span + whose effective captureInput is off drops `td.prompt.variables` from its attributes, also when + the app passes the result of these functions to its own span. +- A tag keeps the variables of its compile alive while a message of the compile lives. When a + chat compile gets a tagged message as a variable, or as an item of a list (Python: a list or + tuple) in a variable, and the tag is at least 100 serial numbers older, the older tag drops its + variables. Thus a chat history that keeps a compiled message of each turn keeps the variables of + about the last 100 compiles, not of each turn, and a request with only such messages records no + `td.prompt.variables`. The SDKs do not look deeper in a variable. They keep no other reference to + the variables, except those of the last compile of each prompt object and of each running scope, + for `withPrompt` / `with_prompt`. - `compile()` of a text prompt returns a string, which cannot carry a tag. Outside a scope, the SDKs do not record a text prompt: use `withPrompt` / `with_prompt`. - The provider integrations call them with the request messages before the span starts, also diff --git a/packages/sdk/README.md b/packages/sdk/README.md index 1fe9131..05c4dba 100644 --- a/packages/sdk/README.md +++ b/packages/sdk/README.md @@ -182,29 +182,36 @@ and TanStack AI integrations, and from other OpenTelemetry tracers, do not get t Outside `withPrompt`, those provider integrations record the prompt of the message objects that `compile()` returned. `compile()` of a chat prompt tags each message that it makes with its prompt -and a copy of its variables, in a non-enumerable symbol property: the keys and the JSON of the -message do not change. A chat call that sends tagged messages gets `td.prompt.name`, -`td.prompt.version`, and `td.prompt.label` of the newest `compile()` among them, also when input -capture is off, and `td.prompt.variables` when it is on. The SDK looks at the first 1000 and the -last 1000 messages of the call. A message from a placeholder variable keeps the tag of its own, -older `compile()`, so it does not win over the `compile()` that put it in the list. A copy of a -message (a spread, `structuredClone`, or JSON) and a new message with the same text have no tag, -so the SDK does not record them. Bedrock Converse and Google GenAI take another message shape: -change the compiled messages in place, or use `withPrompt`. A text prompt compiles to a string, -which cannot carry a tag: use `withPrompt`. `withPrompt` wins over the tag. The integrations do it -for OpenAI chat completions and responses, Anthropic messages, Bedrock Converse, Google GenAI -`generateContent`, and OpenRouter chat and responses, but not for Bedrock InvokeModel (its body is -bytes), embeddings, images, audio, video, or OpenAI Realtime. - -A tagged message keeps the variables of its `compile()` alive while the message lives, with the -lists of its placeholder variables. If you keep compiled messages in a long chat history, put -copies (`{ ...message }`) in the history: else each turn keeps the history list of its own turn. +and a copy of its variables and of each list in them, in a non-enumerable symbol property: the keys +and the JSON of the message do not change. The objects in a list are not copied, so a change to one +of them after `compile()` changes the variables that a call records. A chat call that sends tagged +messages gets `td.prompt.name`, `td.prompt.version`, and `td.prompt.label` of the newest `compile()` +among them, also when input capture is off, and `td.prompt.variables` when it is on. The SDK looks +at the first 1000 and the last 1000 messages of the call. A message from a placeholder variable +keeps the tag of its own, older `compile()`, so it does not win over the `compile()` that put it in +the list. A copy of a message (a spread, `structuredClone`, or JSON) and a new message with the same +text have no tag, so the SDK does not record them. Bedrock Converse and Google GenAI take another +message shape: change the compiled messages in place, or use `withPrompt`. A text prompt compiles to +a string, which cannot carry a tag: use `withPrompt`. `withPrompt` wins over the tag. The +integrations do it for OpenAI chat completions and responses, Anthropic messages, Bedrock Converse, +Google GenAI `generateContent`, and OpenRouter chat and responses, but not for Bedrock InvokeModel +(its body is bytes), embeddings, images, audio, video, or OpenAI Realtime. + +With input capture on in the client when `compile()` runs, a tagged message keeps the variables of +its `compile()` alive while the message lives. With input capture off, the tag keeps no variables. +When `compile()` gets a message in a variable, or in a list in a variable (for example a chat +history in a placeholder), and the message is from a `compile()` at least 100 compiles older, that +older `compile()` drops its variables. Thus a chat history of compiled messages keeps the variables +of about the last 100 compiles, not of each turn. A call that sends only messages of such an older +`compile()` records its prompt without `td.prompt.variables`. The SDK does not look deeper in a +variable for messages. For your own spans, `promptAttributesFor(messages)` gives the attributes to pass as `attributes`. It gives nothing inside `withPrompt`, and a span with `captureInput: false` drops `td.prompt.variables`. A model call whose parent is in `withPrompt` gets the prompt of the -`withPrompt` and none of the `td.prompt.*` attributes that you pass, also when its `parent` option -leads into a `withPrompt` that the call is not in. +`withPrompt` and none of the `td.prompt.*` attributes that you pass, at start or in `update`, +`end`, or `updateActiveSpan`, also when its `parent` option leads into a `withPrompt` that the call +is not in. ## API diff --git a/packages/sdk/src/prompts.ts b/packages/sdk/src/prompts.ts index 8202159..922222a 100644 --- a/packages/sdk/src/prompts.ts +++ b/packages/sdk/src/prompts.ts @@ -97,9 +97,11 @@ export interface ChatPrompt extends PromptBase { /** * Fills in the variables and replaces each placeholder message with the messages in its * variable. `withPrompt` records the variables of the last call before it starts or inside it. - * It tags each message object that it makes with this prompt and a copy of these variables, so - * that a model call outside `withPrompt` that sends the message records them; see - * `promptAttributesFor`. The message keeps the variables alive while it lives. + * It tags each message object that it makes with this prompt and a copy of these variables and + * of each list in them, so that a model call outside `withPrompt` that sends the message records + * them; see `promptAttributesFor`. The objects in a list are not copied. The tag keeps no + * variables when input capture is off in the client. It drops them when a compile 100 or more + * compiles newer takes one of its messages in a variable, as a chat history does. */ compile(variables?: PromptVariables): (ChatMessage | M)[]; } @@ -205,7 +207,8 @@ interface PromptScope { /** The `compile()` call that made a chat message. */ interface CompileTag { readonly prompt: Prompt; - readonly variables: PromptVariables; + /** Undefined when input capture was off, or when the messages became a history; see HISTORY_AGE. */ + variables: PromptVariables | undefined; /** Counts the compiles of the process, so a higher one is newer. */ readonly serial: number; } @@ -225,6 +228,13 @@ const PROMPT_KEY = createContextKey("telemetry.dev prompt"); const COMPILED_BY = Symbol("telemetry.dev compiled by"); /** How many messages promptAttributesFor looks at, at each end of the list. */ const TAG_LOOKUP = 1_000; +/** + * A compile drops its variables when a compile at least this many compiles newer takes one of its + * messages in a variable: its messages are then the history of a chat, and the newest compile names + * the request. Without this, a history that keeps a compiled message of each turn would keep the + * history list of every turn. + */ +const HISTORY_AGE = 100; const cache = new Map(); const compiledVariables = new WeakMap(); @@ -375,8 +385,9 @@ export function promptAttributes( * of messages, or one message): those of the newest `compile()` of a chat prompt that made one of * the messages. It looks at the first 1000 and the last 1000 messages. The tag is on the message * object, so a copy of it gives nothing. Gives undefined inside `withPrompt`, whose prompt the span - * gets. `td.prompt.variables` needs input capture on in the client, and the span drops it when its - * own input capture is off. It never throws. + * gets. `td.prompt.variables` needs input capture on in the client, at the compile and now, and + * the span drops it when its own input capture is off. A compile whose messages became the history + * of a newer compile gives no variables; see `compile()`. It never throws. */ export function promptAttributesFor(messages: unknown): Record | undefined { const config = currentClient().core?.config; @@ -622,8 +633,7 @@ function createPrompt(input: { responseFormat: content.responseFormat, isFallback: input.isFallback, compile(variables: PromptVariables = {}) { - // A copy, so that a later change of the app's object does not change what a span records. - const values = { ...variables }; + const values = copyVariables(variables); compiledVariables.set(prompt, values); const scope = activeContext().getValue(PROMPT_KEY) as PromptScope | undefined; @@ -631,20 +641,48 @@ function createPrompt(input: { if (type === "text") return fill(content.text ?? "", values); compiles += 1; - - return compileMessages( - content.messages, - Object.freeze({ prompt, variables: values, serial: compiles }), - ); + dropHistory(values, compiles); + // promptAttributesFor cannot record the variables when input capture is off in the client. + const captureInput = currentClient().core?.config.captureInput ?? true; + + return compileMessages(content.messages, values, { + prompt, + variables: captureInput ? values : undefined, + serial: compiles, + }); }, }) as Prompt; return prompt; } -function compileMessages(messages: readonly PromptMessage[], tag: CompileTag): unknown[] { - const { variables } = tag; +// A copy of each list too, so that a change of the app's history after compile() does not change +// what a span records. +function copyVariables(variables: PromptVariables): PromptVariables { + return Object.fromEntries( + Object.entries(variables).map(([name, value]) => [ + name, + Array.isArray(value) ? [...value] : value, + ]), + ); +} + +/** Drops the variables of each older compile whose message is in `variables`; see HISTORY_AGE. */ +function dropHistory(variables: PromptVariables, serial: number): void { + for (const value of Object.values(variables)) { + for (const item of Array.isArray(value) ? value : [value]) { + const tag = tagOf(item); + + if (tag && serial - tag.serial >= HISTORY_AGE) tag.variables = undefined; + } + } +} +function compileMessages( + messages: readonly PromptMessage[], + variables: PromptVariables, + tag: CompileTag, +): unknown[] { return messages.flatMap((message) => { if (message.role !== "placeholder") { const compiled = { role: message.role, content: fill(message.content, variables) }; diff --git a/packages/sdk/src/span.ts b/packages/sdk/src/span.ts index cf0c374..3ed9cd9 100644 --- a/packages/sdk/src/span.ts +++ b/packages/sdk/src/span.ts @@ -1,4 +1,5 @@ import { + type Attributes, type Context, defaultTextMapGetter, defaultTextMapSetter, @@ -98,6 +99,8 @@ interface SpanMeta { type: SpanType; capture: FieldCaptureConfig; onError?: (cause: Error) => void; + /** The span got its prompt from withPrompt. */ + scopedPrompt?: boolean; } // Remembers each span's type + capture flags so update()/end()/updateActiveSpan map @@ -190,11 +193,22 @@ function applyFields(span: Span, meta: SpanMeta, fields: SpanFields): void { if (fields.name) span.updateName(fields.name); const attrs = fieldsToAttributes(fields, meta.type, meta.capture); + if (meta.scopedPrompt) dropPromptAttributes(attrs); + if (Object.keys(attrs).length > 0) span.setAttributes(attrs); if (fields.error !== undefined) applyError(span, fields.error, meta.capture); } +// The prompt of withPrompt replaces each td.prompt.* attribute that the app passes, at start and in +// each update, so that a span does not mix two prompts when the app calls promptAttributesFor +// outside withPrompt. +function dropPromptAttributes(attrs: Attributes): void { + for (const key of Object.keys(attrs)) { + if (key.startsWith("td.prompt.")) delete attrs[key]; + } +} + function metaFor(core: ClientCore, options?: StartSpanOptions): SpanMeta { const cfg = core.config; @@ -232,14 +246,9 @@ export function createSpanHandle( const attrs = fieldsToAttributes(options, meta.type, meta.capture); const operation = attrs["gen_ai.operation.name"] ?? SPAN_TYPE_OPERATIONS[meta.type]; const prompt = promptAttributes(parentCtx, operation, meta.capture); + meta.scopedPrompt = prompt !== undefined; - // The prompt of withPrompt replaces all td.prompt.* attributes of the app, so that a span never - // mixes two prompts, also when its parent is in withPrompt and promptAttributesFor ran outside. - if (prompt) { - for (const key of Object.keys(attrs)) { - if (key.startsWith("td.prompt.")) delete attrs[key]; - } - } + if (meta.scopedPrompt) dropPromptAttributes(attrs); const span = core.tracer.startSpan( name, diff --git a/packages/sdk/tests/prompts.test.ts b/packages/sdk/tests/prompts.test.ts index a10ebb6..b464c46 100644 --- a/packages/sdk/tests/prompts.test.ts +++ b/packages/sdk/tests/prompts.test.ts @@ -16,6 +16,7 @@ import { shutdown, startActiveSpan, startSpan, + updateActiveSpan, withPrompt, } from "../src/index.ts"; import { setup } from "./helpers.ts"; @@ -909,10 +910,8 @@ test("a message that a placeholder brings from an older compile does not win", a ], }); - const routed = answer.compile({ - team: "billing", - history: router.compile({ question: "Where is #4?" }) as ChatMessage[], - }); + const question = router.compile({ question: "Where is #4?" }) as ChatMessage[]; + const routed = answer.compile({ team: "billing", history: question }); // The last 1000 messages are all from chat; the summary message is the first. const conversation = Array.from( { length: 1500 }, @@ -928,6 +927,13 @@ test("a message that a placeholder brings from an older compile does not win", a '{"team":"billing","history":[{"role":"user","content":"Where is #4?"}]}', }); expect(promptAttributesFor(summarized)?.["td.prompt.name"]).toBe("summary"); + // A newer compile that takes a message of a recent compile does not drop its variables. + expect(promptAttributesFor(question)).toEqual({ + "td.prompt.name": "router", + "td.prompt.version": 1, + "td.prompt.label": "production", + "td.prompt.variables": '{"question":"Where is #4?"}', + }); }); test("promptAttributesFor looks at the first and the last 1000 messages", async () => { @@ -942,45 +948,62 @@ test("promptAttributesFor looks at the first and the last 1000 messages", async expect(promptAttributesFor([...filler(1000), system, ...filler(1000)])).toBeUndefined(); }); -test("compile records its variables, not later changes of the object", async () => { +test("compile records its variables, not later changes of the object or of its lists", async () => { const { spans } = setup(); - const prompt = await promptOf({ messages: [{ role: "user", content: "Order {{order}}" }] }); - const variables = { order: "#A" }; + const prompt = await promptOf({ + messages: [ + { role: "placeholder", name: "history" }, + { role: "user", content: "Order {{order}}" }, + ], + }); + const history: ChatMessage[] = [{ role: "user", content: "Hi" }]; + const variables = { order: "#A", history }; const first = prompt.compile(variables); variables.order = "#B"; + history.push({ role: "assistant", content: "Hello" }); const second = prompt.compile(variables); variables.order = "#C"; + history.push({ role: "user", content: "Added after compile" }); withPrompt(prompt, () => startSpan("scope", { type: "generation" }).end()); await flush(); - expect(promptAttributesFor(first)?.["td.prompt.variables"]).toBe('{"order":"#A"}'); - expect(promptAttributesFor(second)?.["td.prompt.variables"]).toBe('{"order":"#B"}'); - expect(spans.getFinishedSpans()[0]!.attributes["td.prompt.variables"]).toBe('{"order":"#B"}'); + const secondVariables = + '{"order":"#B","history":[{"role":"user","content":"Hi"},{"role":"assistant","content":"Hello"}]}'; + expect(promptAttributesFor(first)?.["td.prompt.variables"]).toBe( + '{"order":"#A","history":[{"role":"user","content":"Hi"}]}', + ); + expect(promptAttributesFor(second)?.["td.prompt.variables"]).toBe(secondVariables); + expect(spans.getFinishedSpans()[0]!.attributes["td.prompt.variables"]).toBe(secondVariables); }); -test("the variables of a compile are freed with its messages", async () => { +// A WeakRef keeps its target until the current job ends, so collect in a new one. +async function collect(): Promise { setFlagsFromString("--expose-gc"); const gc = runInNewContext("gc") as () => void; - // A WeakRef keeps its target until the current job ends, so collect in a new one. - const collect = async () => { - await new Promise((resolve) => setTimeout(resolve, 0)); - gc(); - }; - setup(); - const prompt = await promptOf({ - messages: [ - { role: "system", content: "Answer as the {{team}} desk." }, - { role: "placeholder", name: "history" }, - ], - }); + await new Promise((resolve) => setTimeout(resolve, 0)); + gc(); +} - let history: ChatMessage[] | undefined = [{ role: "user", content: "Hi" }]; - const freed = new WeakRef(history); - let messages = prompt.compile({ team: "billing", history }) as ChatMessage[] | undefined; - history = undefined; +const RAG_PROMPT = { + messages: [ + { role: "system", content: "Answer from {{documents}}." }, + { role: "user", content: "{{question}}" }, + ], +}; + +test("the variables of a compile are freed with its messages", async () => { + setup(); + const prompt = await promptOf(RAG_PROMPT); + + let document: ChatMessage | undefined = { role: "user", content: "The refund policy" }; + const freed = new WeakRef(document); + let messages = prompt.compile({ documents: [document], question: "q" }) as + | ChatMessage[] + | undefined; + document = undefined; // withPrompt keeps the variables of the last compile of a prompt object. - prompt.compile({ team: "returns" }); + prompt.compile({ documents: [], question: "other" }); await collect(); expect(freed.deref()).toBeDefined(); @@ -990,7 +1013,79 @@ test("the variables of a compile are freed with its messages", async () => { expect(freed.deref()).toBeUndefined(); }); -test("withPrompt wins over a compiled message of another prompt, also through a parent", async () => { +test("with input capture off, compiled messages keep no variables", async () => { + setup({ captureInput: false }); + const prompt = await promptOf(RAG_PROMPT); + const documents: WeakRef[] = []; + const questions: ChatMessage[] = []; + + for (let turn = 0; turn < 200; turn += 1) { + const document: ChatMessage = { role: "user", content: `document ${turn}` }; + documents.push(new WeakRef(document)); + const compiled = prompt.compile({ documents: [document], question: `question ${turn}` }); + questions.push(compiled.at(-1) as ChatMessage); + } + await collect(); + + // withPrompt keeps the variables of the last compile of a prompt object. + expect(documents.filter((document) => document.deref()).length).toBe(1); + + await shutdown(); + setup(); + expect(promptAttributesFor(questions)).toEqual({ + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + }); +}); + +test("a chat history of compiled messages keeps the variables of a bounded number of turns", async () => { + setup(); + const prompt = await promptOf({ + messages: [ + { role: "system", content: "Answer as the {{team}} desk." }, + { role: "placeholder", name: "history" }, + { role: "user", content: "{{question}}" }, + ], + }); + const turns: WeakRef[] = []; + let history: ChatMessage[] = []; + let last: ChatMessage[] = []; + + // Each turn keeps its compiled question in the history. An unused variable marks the variables + // of the turn. + const chat = async (count: number) => { + for (let turn = 0; turn < count; turn += 1) { + const marker: ChatMessage = { role: "user", content: `turn ${turns.length}` }; + turns.push(new WeakRef(marker)); + last = prompt.compile({ + team: "billing", + question: `question ${turns.length}`, + history, + marker: [marker], + }) as ChatMessage[]; + history = [ + ...history, + last.at(-1)!, + { role: "assistant", content: `answer ${turns.length}` }, + ]; + } + await collect(); + + return turns.filter((turn) => turn.deref()).length; + }; + + const kept = await chat(150); + expect(kept).toBeLessThan(150); + expect(await chat(150)).toBe(kept); + + const variables = JSON.parse(promptAttributesFor(last)!["td.prompt.variables"] as string) as { + history: ChatMessage[]; + }; + expect(variables.history).toHaveLength(598); +}); + +test("withPrompt wins over a compiled message of another prompt, through a parent and in updates", async () => { const { spans } = setup(); const scoped = await promptOf({ name: "scoped/prompt" }); // Fetched by version, so it has no label that the label of the scope could hide. @@ -1018,10 +1113,15 @@ test("withPrompt wins over a compiled message of another prompt, also through a const attributes = promptAttributesFor(compiled); startSpan("with parent", { type: "generation", parent, attributes }).end(); parent.end(); + withPrompt(scoped, () => { + startSpan("updated", { type: "generation" }).update({ attributes }).end(); + startSpan("ended", { type: "generation" }).end({ attributes }); + startActiveSpan("active", { type: "generation" }, () => updateActiveSpan({ attributes })); + }); await flush(); expect(attributes?.["td.prompt.name"]).toBe("other/prompt"); - for (const name of ["in scope", "with parent"]) { + for (const name of ["in scope", "with parent", "updated", "ended", "active"]) { const span = spans.getFinishedSpans().find((finished) => finished.name === name)!; expect( Object.fromEntries( diff --git a/sdks/python/README.md b/sdks/python/README.md index 0345367..67207ed 100644 --- a/sdks/python/README.md +++ b/sdks/python/README.md @@ -255,35 +255,43 @@ other OpenTelemetry tracers do not get the attributes. Outside `with_prompt`, those provider integrations record the prompt of the message dicts that `compile()` returned. `compile()` of a chat prompt returns instances of a private `dict` subclass -that also hold its prompt and a copy of its variables. A chat call that sends such messages gets -`td.prompt.name`, `td.prompt.version`, and `td.prompt.label` of the newest `compile()` among -them, also when input capture is off, and `td.prompt.variables` when it is on. The SDK looks at -the first 1000 and the last 1000 messages of the call. A message from a placeholder variable keeps -the prompt of its own, older `compile()`, so it does not win over the `compile()` that put it in -the list. A copy of a message (`dict()`, `.copy()`, `copy.copy`, `copy.deepcopy`, pickle, JSON, -`type(message)(message)`, or `dataclasses.asdict`) and a new message with the same text do not -hold the prompt, so the SDK does not record them. Bedrock Converse and Google GenAI take another -message shape: change the compiled dicts in place, or use `with_prompt`. A text prompt compiles to -a string, which cannot hold its prompt: use `with_prompt`. `with_prompt` wins over the message. -The integrations do it for OpenAI chat completions and responses, Anthropic messages (for -`messages.stream()`, at `__enter__`), Bedrock Converse, Google GenAI `generate_content`, LiteLLM -completion and responses, and OpenRouter chat and responses, but not for Bedrock InvokeModel (its -body is bytes), embeddings, images, audio, video, or OpenAI Realtime. With an older -`telemetry-dev` that has no `prompt_attributes_for`, a provider integration records the prompt -only in `with_prompt`. +that also hold its prompt and a copy of its variables and of each list in them. The objects in a +list are not copied, so a change to one of them after `compile()` changes the variables that a call +records. A chat call that sends such messages gets `td.prompt.name`, `td.prompt.version`, and +`td.prompt.label` of the newest `compile()` among them, also when input capture is off, and +`td.prompt.variables` when it is on. The SDK looks at the first 1000 and the last 1000 messages of +the call. A message from a placeholder variable keeps the prompt of its own, older `compile()`, so +it does not win over the `compile()` that put it in the list. A copy of a message (`dict()`, +`.copy()`, `copy.copy`, `copy.deepcopy`, pickle, JSON, `type(message)(message)`, or +`dataclasses.asdict`) and a new message with the same text do not hold the prompt, so the SDK does +not record them. Bedrock Converse and Google GenAI take another message shape: change the compiled +dicts in place, or use `with_prompt`. A text prompt compiles to a string, which cannot hold its +prompt: use `with_prompt`. `with_prompt` wins over the message. The integrations do it for OpenAI +chat completions and responses, Anthropic messages (for `messages.stream()`, at `__enter__`), +Bedrock Converse, Google GenAI `generate_content`, LiteLLM completion and responses, and OpenRouter +chat and responses, but not for Bedrock InvokeModel (its body is bytes), embeddings, images, audio, +video, or OpenAI Realtime. With an older `telemetry-dev` that has no `prompt_attributes_for`, a +provider integration records the prompt only in `with_prompt`. A compiled message is a `dict` for `isinstance`, and its keys and its JSON (`json`, `orjson`, pydantic) do not change. A check of the exact type (`type(message) is dict`), `yaml.safe_dump`, -and `marshal` reject it: give them `dict(message)`. A compiled message keeps the variables of its -`compile()` alive while the message lives, with the lists of its placeholder variables. If you -keep compiled messages in a long chat history, put copies (`dict(message)`) in the history: else -each turn keeps the history list of its own turn. +and `marshal` reject it: give them `dict(message)`. + +With input capture on in the client when `compile()` runs, a compiled message keeps the variables +of its `compile()` alive while the message lives. With input capture off, it keeps no variables. +When `compile()` gets a message in a variable, or in a list or tuple in a variable (for example a +chat history in a placeholder), and the message is from a `compile()` at least 100 compiles older, +that older `compile()` drops its variables. Thus a chat history of compiled messages keeps the +variables of about the last 100 compiles, not of each turn. A call that sends only messages of such +an older `compile()` records its prompt without `td.prompt.variables`. The SDK does not look deeper +in a variable for messages. For your own spans, `prompt_attributes_for(messages)` gives the attributes to pass as `attributes`. It gives `{}` inside `with_prompt`, and a span with `capture_input=False` drops `td.prompt.variables`. A model call whose parent is in `with_prompt` gets the prompt of the -`with_prompt` and none of the `td.prompt.*` attributes that you pass, also when its `parent` -context comes from a `with_prompt` that the call is not in. +`with_prompt` and none of the `td.prompt.*` attributes that you pass, at start or in `update`, +`end`, or `update_current_span`, also when its `parent` context comes from a `with_prompt` that the +call is not in. ## Auto-metrics diff --git a/sdks/python/src/telemetry_dev/_prompts.py b/sdks/python/src/telemetry_dev/_prompts.py index 28ee83e..59d8a1b 100644 --- a/sdks/python/src/telemetry_dev/_prompts.py +++ b/sdks/python/src/telemetry_dev/_prompts.py @@ -14,7 +14,7 @@ import urllib.parse import urllib.request from collections import OrderedDict -from collections.abc import Callable, Generator, Mapping +from collections.abc import Callable, Generator, Mapping, Sequence from contextlib import contextmanager from contextvars import ContextVar from dataclasses import dataclass, field @@ -53,6 +53,11 @@ _PROMPT_KEY = otel_context.create_key("telemetry_dev.prompt") # How many messages prompt_attributes_for looks at, at each end of the list. _TAG_LOOKUP = 1000 +# A compile drops its variables when a compile at least this many compiles newer takes one of its +# messages in a variable: its messages are then the history of a chat, and the newest compile names +# the request. Without this, a history that keeps a compiled message of each turn would keep the +# history list of every turn. +_HISTORY_AGE = 100 class PromptModel(TypedDict): @@ -102,12 +107,20 @@ def compile( """Fill in the variables and replace each placeholder message with the list of messages in its variable. ``with_prompt`` records the variables of the last call before it starts or inside it. Each message that it makes is a ``dict`` subclass that also holds this - prompt and a copy of these variables, so that a model call outside ``with_prompt`` that - sends the message records them; see ``prompt_attributes_for``. The message keeps the - variables alive while it lives. A check of the exact type (``type(message) is dict``), - ``yaml.safe_dump``, and ``marshal`` reject it: give them ``dict(message)``.""" + prompt and a copy of these variables and of each list in them, so that a model call + outside ``with_prompt`` that sends the message records them; see + ``prompt_attributes_for``. The objects in a list are not copied. The message holds no + variables when input capture is off in the client. It drops them when a compile 100 or + more compiles newer takes one of its messages in a variable, as a chat history does. A + check of the exact type (``type(message) is dict``), ``yaml.safe_dump``, and ``marshal`` + reject it: give them ``dict(message)``.""" values = _record(self, variables, kwargs) - tag = _CompileTag(self, values, next(_compiles)) + serial = next(_compiles) + _drop_history(values, serial) + client = get_client() + # prompt_attributes_for cannot record the variables when input capture is off in the client. + kept = values if client is None or client.capture_input else None + tag = _CompileTag(self, kept, serial) compiled: list[dict[str, Any]] = [] for message in self.messages: if message.get("role") != "placeholder": @@ -304,7 +317,9 @@ def prompt_attributes_for(messages: object) -> dict[str, AttributeValue]: prompt that made one of the messages. It looks at the first 1000 and the last 1000 messages. The tag is on the dict that ``compile()`` made, so a copy of it gives ``{}``. Gives ``{}`` inside ``with_prompt``, whose prompt the span gets. ``td.prompt.variables`` needs input - capture on in the client, and the span drops it when its own input capture is off. + capture on in the client, at the compile and now, and the span drops it when its own input + capture is off. A compile whose messages became the history of a newer compile gives no + variables; see ``ChatPrompt.compile``. It reads only a ``list`` or ``tuple``, not a subclass, so the read runs no code of the app. With input capture on, it serializes the variables, which calls the ``mask`` of the client and @@ -439,7 +454,12 @@ class _PromptScope: def _record( prompt: Prompt, variables: Mapping[str, Any] | None, kwargs: Mapping[str, Any] ) -> dict[str, Any]: - values = {**(variables or {}), **kwargs} + # A copy of each list too, so that a change of the app's history after compile() does not + # change what a span records. + values: dict[str, Any] = { + name: list(cast("list[Any]", value)) if isinstance(value, list) else value + for name, value in {**(variables or {}), **kwargs}.items() + } _compiled_variables[prompt] = values scope = otel_context.get_value(_PROMPT_KEY) if isinstance(scope, _PromptScope) and scope.prompt is prompt: @@ -448,12 +468,13 @@ def _record( return values -@dataclass(frozen=True) +@dataclass(eq=False) class _CompileTag: """The ``compile()`` call that made a chat message.""" prompt: ChatPrompt - variables: dict[str, Any] + variables: dict[str, Any] | None + """None when input capture was off, or when the messages became a history; see _HISTORY_AGE.""" serial: int """Counts the compiles of the process, so a higher one is newer.""" @@ -473,6 +494,22 @@ def __reduce__(self) -> tuple[type[dict[str, Any]], tuple[dict[str, Any]]]: return dict, (dict(self),) +def _drop_history(values: Mapping[str, Any], serial: int) -> None: + """Drops the variables of each older compile whose message is in ``values``; see + _HISTORY_AGE.""" + for value in values.values(): + items: Sequence[object] = ( + cast("Sequence[object]", value) + if type(value) is list or type(value) is tuple + else (value,) + ) + for item in items: + if type(item) is _CompiledMessage: + tag = getattr(item, "compiled_by", None) + if tag is not None and serial - tag.serial >= _HISTORY_AGE: + tag.variables = None + + def _newest_tag(messages: object) -> _CompileTag | None: """The messages of a placeholder come from an older compile, so the newest compile, not the last tagged message, names the prompt of the request. Its own messages are at the start or the diff --git a/sdks/python/src/telemetry_dev/_spans.py b/sdks/python/src/telemetry_dev/_spans.py index 1d6ac64..6328fa3 100644 --- a/sdks/python/src/telemetry_dev/_spans.py +++ b/sdks/python/src/telemetry_dev/_spans.py @@ -103,6 +103,7 @@ class _SpanState: operation: str capture_input: bool capture_output: bool + prompt_from_scope: bool = False output_chunk_last_ms: float | None = None output_chunk_count: int = 0 output_chunk_sum_s: float = 0 @@ -300,6 +301,8 @@ def _apply_fields(client: Client, span: Span, state: _SpanState, fields: dict[st if name is not None: span.update_name(name) attrs = _fields_to_attributes(client, state, fields) + if state.prompt_from_scope: + attrs = _without_prompt(attrs) if attrs: span.set_attributes(attrs) @@ -308,6 +311,15 @@ def _apply_fields(client: Client, span: Span, state: _SpanState, fields: dict[st _record_error_on_span(client, span, state, error) +def _without_prompt( + attrs: dict[str, otel_types.AttributeValue], +) -> dict[str, otel_types.AttributeValue]: + """The prompt of with_prompt replaces each td.prompt.* attribute that the app passes, at start + and in each update, so that a span does not mix two prompts when the app calls + prompt_attributes_for outside with_prompt.""" + return {key: value for key, value in attrs.items() if not key.startswith("td.prompt.")} + + def _collect_fields( *, input: Any = NOT_GIVEN, @@ -752,10 +764,8 @@ def start_span( client, ctx, attrs.get(ATTR_OPERATION, operation), state.capture_input ) if prompt: - # The prompt of with_prompt replaces all td.prompt.* attributes of the app, so that a - # span never mixes two prompts, also when its parent is in with_prompt and - # prompt_attributes_for ran outside. - attrs = {key: value for key, value in attrs.items() if not key.startswith("td.prompt.")} + state.prompt_from_scope = True + attrs = _without_prompt(attrs) initial.update(prompt) initial.update(attrs) span = client.tracer.start_span( diff --git a/sdks/python/tests/test_prompts.py b/sdks/python/tests/test_prompts.py index 423286f..103c399 100644 --- a/sdks/python/tests/test_prompts.py +++ b/sdks/python/tests/test_prompts.py @@ -985,7 +985,8 @@ def test_a_message_that_a_placeholder_brings_from_an_older_compile_does_not_win( name="summary", ) - routed = answer.compile(team="billing", history=router.compile(question="Where is #4?")) + question = router.compile(question="Where is #4?") + routed = answer.compile(team="billing", history=question) # The last 1000 messages are all from chat; the summary message is the first. conversation = [chat.compile(question=f"question {index}")[0] for index in range(1500)] summarized = summary.compile(history=conversation) @@ -999,6 +1000,13 @@ def test_a_message_that_a_placeholder_brings_from_an_older_compile_does_not_win( ), } assert prompt_attributes_for(summarized)["td.prompt.name"] == "summary" + # A newer compile that takes a message of a recent compile does not drop its variables. + assert prompt_attributes_for(question) == { + "td.prompt.name": "router", + "td.prompt.version": 12, + "td.prompt.label": "production", + "td.prompt.variables": '{"question":"Where is #4?"}', + } def test_prompt_attributes_for_looks_at_the_first_and_the_last_1000_messages( @@ -1016,24 +1024,37 @@ def around(before: int, after: int) -> list[dict[str, Any]]: assert prompt_attributes_for(around(1000, 1000)) == {} -def test_compile_records_its_variables_not_later_changes_of_the_mapping( +def test_compile_records_its_variables_not_later_changes_of_the_mapping_or_of_its_lists( make: MakeClient, ) -> None: memory = make() - prompt = chat_prompt({"role": "user", "content": "Order {{order}}"}) - variables = {"order": "#A"} + prompt = chat_prompt( + {"role": "placeholder", "name": "history"}, + {"role": "user", "content": "Order {{order}}"}, + ) + history = [{"role": "user", "content": "Hi"}] + variables: dict[str, Any] = {"order": "#A", "history": history} first = prompt.compile(variables) variables["order"] = "#B" + history.append({"role": "assistant", "content": "Hello"}) second = prompt.compile(variables) variables["order"] = "#C" + history.append({"role": "user", "content": "Added after compile"}) with with_prompt(prompt): start_span("scope", type="generation").end() - assert prompt_attributes_for(first)["td.prompt.variables"] == '{"order":"#A"}' - assert prompt_attributes_for(second)["td.prompt.variables"] == '{"order":"#B"}' + second_variables = ( + '{"order":"#B","history":[{"role":"user","content":"Hi"},' + '{"role":"assistant","content":"Hello"}]}' + ) + assert ( + prompt_attributes_for(first)["td.prompt.variables"] + == '{"order":"#A","history":[{"role":"user","content":"Hi"}]}' + ) + assert prompt_attributes_for(second)["td.prompt.variables"] == second_variables [span] = memory.span_exporter.get_finished_spans() - assert (span.attributes or {})["td.prompt.variables"] == '{"order":"#B"}' + assert (span.attributes or {})["td.prompt.variables"] == second_variables def test_a_compiled_message_rebuilds_like_a_dict_and_the_rebuilt_one_has_no_tag( @@ -1059,7 +1080,7 @@ class Conversation: assert prompt_attributes_for(same_type) == {} -def test_with_prompt_wins_over_a_compiled_message_of_another_prompt_also_through_a_parent( +def test_with_prompt_wins_over_a_compiled_message_of_another_prompt_through_a_parent_and_in_updates( make: MakeClient, ) -> None: memory = make() @@ -1077,9 +1098,22 @@ def test_with_prompt_wins_over_a_compiled_message_of_another_prompt_also_through parent = otel_context.get_current() attributes = prompt_attributes_for(compiled) start_span("with parent", type="generation", parent=parent, attributes=attributes).end() + with with_prompt(scoped): + start_span("updated", type="generation").update(attributes=attributes).end() + start_span("ended", type="generation").end(attributes=attributes) + with start_span("current", type="generation"): + telemetry_dev.update_current_span(attributes=attributes) assert attributes["td.prompt.name"] == "other/prompt" - for span in memory.span_exporter.get_finished_spans(): + spans = memory.span_exporter.get_finished_spans() + assert sorted(span.name for span in spans) == [ + "current", + "ended", + "in scope", + "updated", + "with parent", + ] + for span in spans: assert { key: value for key, value in dict(span.attributes or {}).items() @@ -1147,30 +1181,94 @@ def __reversed__(self) -> Any: assert reads == [] +RAG_MESSAGES = [ + {"role": "system", "content": "Answer from {{documents}}."}, + {"role": "user", "content": "{{question}}"}, +] + + +class Document(dict[str, str]): + """A dict that a weak reference can point to.""" + + def test_the_variables_of_a_compile_are_freed_with_its_messages(make: MakeClient) -> None: make() - prompt = chat_prompt( - {"role": "system", "content": "Answer as the {{team}} desk."}, - {"role": "placeholder", "name": "history"}, - ) + prompt = chat_prompt(*RAG_MESSAGES) - class History(list[dict[str, str]]): - """A list that a weak reference can point to.""" - - history = History([{"role": "user", "content": "Hi"}]) - freed = weakref.ref(history) - messages = prompt.compile(team="billing", history=history) - del history + document = Document(content="The refund policy") + freed = weakref.ref(document) + messages = prompt.compile(documents=[document], question="q") + del document # with_prompt keeps the variables of the last compile of a prompt object. - prompt.compile(team="returns") + prompt.compile(documents=[], question="other") gc.collect() assert freed() is not None + assert len(messages) == 2 del messages gc.collect() assert freed() is None +def test_with_input_capture_off_compiled_messages_keep_no_variables(make: MakeClient) -> None: + make(capture_input=False) + prompt = chat_prompt(*RAG_MESSAGES) + documents: list[weakref.ref[Document]] = [] + questions: list[dict[str, Any]] = [] + + for turn in range(200): + document = Document(content=f"document {turn}") + documents.append(weakref.ref(document)) + questions.append(prompt.compile(documents=[document], question=f"question {turn}")[-1]) + gc.collect() + + # with_prompt keeps the variables of the last compile of a prompt object. + assert sum(1 for document in documents if document() is not None) == 1 + + telemetry_dev.shutdown() + make() + assert prompt_attributes_for(questions) == { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + } + + +def test_a_chat_history_of_compiled_messages_keeps_the_variables_of_a_bounded_number_of_turns( + make: MakeClient, +) -> None: + make() + prompt = chat_prompt( + {"role": "system", "content": "Answer as the {{team}} desk."}, + {"role": "placeholder", "name": "history"}, + {"role": "user", "content": "{{question}}"}, + ) + turns: list[weakref.ref[Document]] = [] + history: list[dict[str, Any]] = [] + last: list[dict[str, Any]] = [] + + # Each turn keeps its compiled question in the history. An unused variable marks the variables + # of the turn. + def chat(count: int) -> int: + nonlocal history, last + for _ in range(count): + marker = Document(content=f"turn {len(turns)}") + turns.append(weakref.ref(marker)) + last = prompt.compile( + team="billing", question=f"question {len(turns)}", history=history, marker=marker + ) + history = [*history, last[-1], {"role": "assistant", "content": f"answer {len(turns)}"}] + gc.collect() + return sum(1 for turn in turns if turn() is not None) + + kept = chat(150) + assert kept < 150 + assert chat(150) == kept + + variables = json.loads(str(prompt_attributes_for(last)["td.prompt.variables"])) + assert len(variables["history"]) == 598 + + def test_a_fallback_prompt_records_its_name_and_label_but_no_version( server: CaptureServer, make: MakeClient ) -> None: From abb8d10f680d1fa7ace6913a1d53b43a3e0db703 Mon Sep 17 00:00:00 2001 From: ephraimduncan Date: Thu, 29 Oct 2026 21:29:24 +0000 Subject: [PATCH 5/9] fix(sdk): give the base url to getPrompt in the prompt tests The tests of this branch gave apiUrl and api_url to getPrompt and get_prompt. The main branch removed these options in #101. Thus the tests give baseUrl, base_url, and newBaseUrl. Refs #97 --- packages/anthropic/tests/anthropic.test.ts | 2 +- packages/bedrock/tests/bedrock.test.ts | 2 +- packages/google-genai/tests/google-genai.test.ts | 2 +- packages/openai/tests/openai.test.ts | 2 +- packages/openrouter/tests/openrouter.test.ts | 2 +- packages/sdk/tests/prompts.test.ts | 6 +++--- sdks/python/tests/test_prompts.py | 4 ++-- 7 files changed, 10 insertions(+), 10 deletions(-) diff --git a/packages/anthropic/tests/anthropic.test.ts b/packages/anthropic/tests/anthropic.test.ts index 12eeae9..6516205 100644 --- a/packages/anthropic/tests/anthropic.test.ts +++ b/packages/anthropic/tests/anthropic.test.ts @@ -298,7 +298,7 @@ test.each([ const prompt = await getPrompt("support/reply", { apiKey: "td_live_a", - apiUrl: "https://prompts.anthropic.test", + baseUrl: "https://prompts.anthropic.test", fetch: async () => Response.json({ name: "support/reply", diff --git a/packages/bedrock/tests/bedrock.test.ts b/packages/bedrock/tests/bedrock.test.ts index 6098bea..0eb0e7f 100644 --- a/packages/bedrock/tests/bedrock.test.ts +++ b/packages/bedrock/tests/bedrock.test.ts @@ -443,7 +443,7 @@ test.each([ const prompt = await getPrompt("support/reply", { apiKey: "td_live_a", - apiUrl: "https://prompts.bedrock.test", + baseUrl: "https://prompts.bedrock.test", fetch: async () => Response.json({ name: "support/reply", diff --git a/packages/google-genai/tests/google-genai.test.ts b/packages/google-genai/tests/google-genai.test.ts index e7d5ec6..19d7fbf 100644 --- a/packages/google-genai/tests/google-genai.test.ts +++ b/packages/google-genai/tests/google-genai.test.ts @@ -253,7 +253,7 @@ test.each([ const prompt = await getPrompt("support/reply", { apiKey: "td_live_a", - apiUrl: "https://prompts.google-genai.test", + baseUrl: "https://prompts.google-genai.test", fetch: async () => Response.json({ name: "support/reply", diff --git a/packages/openai/tests/openai.test.ts b/packages/openai/tests/openai.test.ts index 4437d75..099ec37 100644 --- a/packages/openai/tests/openai.test.ts +++ b/packages/openai/tests/openai.test.ts @@ -2912,7 +2912,7 @@ test.each([ const prompt = await getPrompt("support/reply", { apiKey: "td_live_a", - apiUrl: "https://prompts.openai.test", + baseUrl: "https://prompts.openai.test", fetch: async () => Response.json({ name: "support/reply", diff --git a/packages/openrouter/tests/openrouter.test.ts b/packages/openrouter/tests/openrouter.test.ts index 3f5cb68..0d50a1b 100644 --- a/packages/openrouter/tests/openrouter.test.ts +++ b/packages/openrouter/tests/openrouter.test.ts @@ -329,7 +329,7 @@ test.each([ const prompt = await getPrompt("support/reply", { apiKey: "td_live_a", - apiUrl: "https://prompts.openrouter.test", + baseUrl: "https://prompts.openrouter.test", fetch: async () => Response.json({ name: "support/reply", diff --git a/packages/sdk/tests/prompts.test.ts b/packages/sdk/tests/prompts.test.ts index b464c46..f519e54 100644 --- a/packages/sdk/tests/prompts.test.ts +++ b/packages/sdk/tests/prompts.test.ts @@ -740,7 +740,7 @@ async function promptOf(overrides: Record): Promise { return getPrompt("support/reply", { apiKey: "td_live_a", - apiUrl: newApiUrl(), + baseUrl: newBaseUrl(), fetch, type: overrides.type as "chat", }); @@ -836,7 +836,7 @@ test("a rollout with two versions of the same messages records the version of ea }, }), ); - const options = { apiKey: "td_live_a", apiUrl: newApiUrl(), fetch }; + const options = { apiKey: "td_live_a", baseUrl: newBaseUrl(), fetch }; vi.spyOn(Math, "random").mockReturnValueOnce(0.159).mockReturnValueOnce(0.16); const rollout = await getPrompt("support/reply", options); @@ -1099,7 +1099,7 @@ test("withPrompt wins over a compiled message of another prompt, through a paren ); const other = await getPrompt("other/prompt", { apiKey: "td_live_a", - apiUrl: newApiUrl(), + baseUrl: newBaseUrl(), version: 3, fetch, }); diff --git a/sdks/python/tests/test_prompts.py b/sdks/python/tests/test_prompts.py index 103c399..324b01e 100644 --- a/sdks/python/tests/test_prompts.py +++ b/sdks/python/tests/test_prompts.py @@ -931,8 +931,8 @@ def draw(_stop: int) -> int: return next(draws) monkeypatch.setattr(prompts_module.random, "randrange", draw) - rollout = get_prompt("support/reply", api_key="td_live_a", api_url=server.url) - production = get_prompt("support/reply", api_key="td_live_a", api_url=server.url) + rollout = get_prompt("support/reply", api_key="td_live_a", base_url=server.url) + production = get_prompt("support/reply", api_key="td_live_a", base_url=server.url) from_rollout = rollout.compile(team="billing", order="#4") from_production = production.compile(team="billing", order="#4") From f6aa1c159742123600d6499653916c4dbf7b2ab7 Mon Sep 17 00:00:00 2001 From: ephraimduncan Date: Thu, 29 Oct 2026 23:19:43 +0000 Subject: [PATCH 6/9] fix(sdk): keep the variables of the last two chat turns A compile removed the variables of a compile 100 or more compiles before it, when it got a message of that compile. A chat that cuts its history to less than approximately 100 turns never gave such a message. Each tag kept a copy of the window, and the messages in the window kept the tags of previous turns. Thus every turn kept its variables: 2000 of 2000 turns in TypeScript and Python. The count also included the compiles of other requests. Thus 99 other compiles before a guard compile, or a guard compile at each step of an agent loop, removed the variables of a request before its model call. With this change, when a compile C gets a message of a compile B in a variable, each compile A whose message B got removes its variables. B keeps its own. The rule does not count compiles, thus the compiles of other requests do not change it. A chain of tags with variables is at most two tags long. The repros of the review keep the variables of 2 of 2000 turns in the two SDKs. This is true for one list that the app cuts in place to 20 messages, for a new list of the last 20 or 198 messages, and for a history with no window. With no window, the SDK keeps two copies of the history list, not approximately 100. The tests do the same check for 300 turns, also for a tuple (Python) and for one message that is not in a list. A guard compile after 150 other compiles, and a guard compile at each of 150 agent steps, keep the variables of the request. In Python, _drop_history reads the variables of a tag one time, because a compile in another thread can remove them. The READMEs and docs/sdk-conformance.md tell the new rule. They again recommend copies for compiled messages that the app keeps but does not give to compile(): a copy keeps no variables. Refs #97 --- docs/sdk-conformance.md | 18 +-- packages/sdk/README.md | 15 ++- packages/sdk/src/prompts.ts | 42 ++++--- packages/sdk/tests/prompts.test.ts | 129 +++++++++++++++------- sdks/python/README.md | 15 ++- sdks/python/src/telemetry_dev/_prompts.py | 48 +++++--- sdks/python/tests/test_prompts.py | 112 ++++++++++++++----- 7 files changed, 256 insertions(+), 123 deletions(-) diff --git a/docs/sdk-conformance.md b/docs/sdk-conformance.md index b49779a..49e27fe 100644 --- a/docs/sdk-conformance.md +++ b/docs/sdk-conformance.md @@ -211,14 +211,16 @@ their text: at the call. Then they serialize the variables, which calls the `mask` of the client. A span whose effective captureInput is off drops `td.prompt.variables` from its attributes, also when the app passes the result of these functions to its own span. -- A tag keeps the variables of its compile alive while a message of the compile lives. When a - chat compile gets a tagged message as a variable, or as an item of a list (Python: a list or - tuple) in a variable, and the tag is at least 100 serial numbers older, the older tag drops its - variables. Thus a chat history that keeps a compiled message of each turn keeps the variables of - about the last 100 compiles, not of each turn, and a request with only such messages records no - `td.prompt.variables`. The SDKs do not look deeper in a variable. They keep no other reference to - the variables, except those of the last compile of each prompt object and of each running scope, - for `withPrompt` / `with_prompt`. +- A tag keeps the variables of its compile alive while a message of the compile lives. A chat + compile C can get a tagged message of a compile B as a variable, or as an item of a list + (Python: a list or tuple) in a variable. Then each compile A whose message B got in the same way + removes its variables, and B keeps its own. Thus a chat history that keeps a compiled message of + each turn keeps the variables of the last two turns, for a history of any length or window, and + a request with only messages of previous turns records no `td.prompt.variables`. A guard prompt + that gets the messages of a request before its call does not remove their variables, also after + other compiles in the process. The SDKs do not look deeper in a variable. They keep no other + reference to the variables, except those of the last compile of each prompt object and of each + running scope, for `withPrompt` / `with_prompt`. - `compile()` of a text prompt returns a string, which cannot carry a tag. Outside a scope, the SDKs do not record a text prompt: use `withPrompt` / `with_prompt`. - The provider integrations call them with the request messages before the span starts, also diff --git a/packages/sdk/README.md b/packages/sdk/README.md index 05c4dba..9edf709 100644 --- a/packages/sdk/README.md +++ b/packages/sdk/README.md @@ -199,12 +199,15 @@ Google GenAI `generateContent`, and OpenRouter chat and responses, but not for B With input capture on in the client when `compile()` runs, a tagged message keeps the variables of its `compile()` alive while the message lives. With input capture off, the tag keeps no variables. -When `compile()` gets a message in a variable, or in a list in a variable (for example a chat -history in a placeholder), and the message is from a `compile()` at least 100 compiles older, that -older `compile()` drops its variables. Thus a chat history of compiled messages keeps the variables -of about the last 100 compiles, not of each turn. A call that sends only messages of such an older -`compile()` records its prompt without `td.prompt.variables`. The SDK does not look deeper in a -variable for messages. +When `compile()` gets a tagged message in a variable, or in a list in a variable, the `compile()` +that made the message keeps its variables. But each `compile()` whose message that `compile()` got +removes its variables. For a chat history in a placeholder, turn 3 thus removes the variables of +turn 1 and keeps those of turn 2. A chat history of compiled messages keeps the variables of its +last two turns, also when you cut it, and a call that sends only messages of previous turns records +its prompt without `td.prompt.variables`. A guard prompt that gets the messages before their call +does not remove their variables. The SDK does not look deeper in a variable for messages. If you +keep compiled messages and do not give them to `compile()`, each one keeps the variables of its +`compile()`: keep copies (`{ ...message }`) to free them. For your own spans, `promptAttributesFor(messages)` gives the attributes to pass as `attributes`. It gives nothing inside `withPrompt`, and a span with `captureInput: false` drops diff --git a/packages/sdk/src/prompts.ts b/packages/sdk/src/prompts.ts index 922222a..55169d5 100644 --- a/packages/sdk/src/prompts.ts +++ b/packages/sdk/src/prompts.ts @@ -100,8 +100,10 @@ export interface ChatPrompt extends PromptBase { * It tags each message object that it makes with this prompt and a copy of these variables and * of each list in them, so that a model call outside `withPrompt` that sends the message records * them; see `promptAttributesFor`. The objects in a list are not copied. The tag keeps no - * variables when input capture is off in the client. It drops them when a compile 100 or more - * compiles newer takes one of its messages in a variable, as a chat history does. + * variables when input capture is off in the client. When a newer compile gets one of these + * messages in a variable, the compiles whose messages this compile got remove their variables, + * and this compile keeps its own. Thus a chat history keeps the variables of its last two turns, + * and a guard prompt that gets the messages before their call does not remove them. */ compile(variables?: PromptVariables): (ChatMessage | M)[]; } @@ -207,7 +209,9 @@ interface PromptScope { /** The `compile()` call that made a chat message. */ interface CompileTag { readonly prompt: Prompt; - /** Undefined when input capture was off, or when the messages became a history; see HISTORY_AGE. */ + /** + * Undefined when input capture was off, or when the messages became a history; see dropHistory. + */ variables: PromptVariables | undefined; /** Counts the compiles of the process, so a higher one is newer. */ readonly serial: number; @@ -228,13 +232,6 @@ const PROMPT_KEY = createContextKey("telemetry.dev prompt"); const COMPILED_BY = Symbol("telemetry.dev compiled by"); /** How many messages promptAttributesFor looks at, at each end of the list. */ const TAG_LOOKUP = 1_000; -/** - * A compile drops its variables when a compile at least this many compiles newer takes one of its - * messages in a variable: its messages are then the history of a chat, and the newest compile names - * the request. Without this, a history that keeps a compiled message of each turn would keep the - * history list of every turn. - */ -const HISTORY_AGE = 100; const cache = new Map(); const compiledVariables = new WeakMap(); @@ -386,8 +383,8 @@ export function promptAttributes( * the messages. It looks at the first 1000 and the last 1000 messages. The tag is on the message * object, so a copy of it gives nothing. Gives undefined inside `withPrompt`, whose prompt the span * gets. `td.prompt.variables` needs input capture on in the client, at the compile and now, and - * the span drops it when its own input capture is off. A compile whose messages became the history - * of a newer compile gives no variables; see `compile()`. It never throws. + * the span drops it when its own input capture is off. The compile of a previous turn of a chat + * history gives no variables; see `compile()`. It never throws. */ export function promptAttributesFor(messages: unknown): Record | undefined { const config = currentClient().core?.config; @@ -641,7 +638,7 @@ function createPrompt(input: { if (type === "text") return fill(content.text ?? "", values); compiles += 1; - dropHistory(values, compiles); + dropHistory(values); // promptAttributesFor cannot record the variables when input capture is off in the client. const captureInput = currentClient().core?.config.captureInput ?? true; @@ -667,13 +664,26 @@ function copyVariables(variables: PromptVariables): PromptVariables, serial: number): void { +/** + * When a compile gets a message of a previous compile in `variables`, the compiles whose messages + * that previous compile got are the history of a chat and remove their variables. The previous + * compile keeps its own, because its model call can come after this compile, as with a guard + * prompt. Thus a chain of tags with variables is at most two tags long, for a history of any + * length. + */ +function dropHistory(variables: PromptVariables): void { + for (const tag of tagsIn(variables)) { + if (tag.variables) for (const older of tagsIn(tag.variables)) older.variables = undefined; + } +} + +/** The tags of the messages in `variables`: a value, or an item of a list in a value. */ +function* tagsIn(variables: PromptVariables): Generator { for (const value of Object.values(variables)) { for (const item of Array.isArray(value) ? value : [value]) { const tag = tagOf(item); - if (tag && serial - tag.serial >= HISTORY_AGE) tag.variables = undefined; + if (tag) yield tag; } } } diff --git a/packages/sdk/tests/prompts.test.ts b/packages/sdk/tests/prompts.test.ts index f519e54..381df5a 100644 --- a/packages/sdk/tests/prompts.test.ts +++ b/packages/sdk/tests/prompts.test.ts @@ -11,6 +11,7 @@ import { getPrompt, type Prompt, promptAttributesFor, + type PromptVariables, propagateAttributes, PromptError, shutdown, @@ -927,13 +928,6 @@ test("a message that a placeholder brings from an older compile does not win", a '{"team":"billing","history":[{"role":"user","content":"Where is #4?"}]}', }); expect(promptAttributesFor(summarized)?.["td.prompt.name"]).toBe("summary"); - // A newer compile that takes a message of a recent compile does not drop its variables. - expect(promptAttributesFor(question)).toEqual({ - "td.prompt.name": "router", - "td.prompt.version": 1, - "td.prompt.label": "production", - "td.prompt.variables": '{"question":"Where is #4?"}', - }); }); test("promptAttributesFor looks at the first and the last 1000 messages", async () => { @@ -1039,50 +1033,101 @@ test("with input capture off, compiled messages keep no variables", async () => }); }); -test("a chat history of compiled messages keeps the variables of a bounded number of turns", async () => { - setup(); - const prompt = await promptOf({ - messages: [ - { role: "system", content: "Answer as the {{team}} desk." }, - { role: "placeholder", name: "history" }, - { role: "user", content: "{{question}}" }, - ], - }); - const turns: WeakRef[] = []; - let history: ChatMessage[] = []; - let last: ChatMessage[] = []; - - // Each turn keeps its compiled question in the history. An unused variable marks the variables - // of the turn. - const chat = async (count: number) => { - for (let turn = 0; turn < count; turn += 1) { - const marker: ChatMessage = { role: "user", content: `turn ${turns.length}` }; +// Each row keeps the compiled question and the answer of a turn for the next turn: the variable +// that gets them, and the value of that variable after the turn. +test.each<[string, string, (kept: unknown, question: object, answer: object) => unknown]>([ + ["a new list each turn", "history", (kept, ...turn) => [...(kept as object[]), ...turn]], + [ + "one list trimmed in place to 20 messages", + "history", + (kept, ...turn) => { + const history = kept as object[]; + history.push(...turn); + history.splice(0, history.length - 20); + + return history; + }, + ], + [ + "a new list of the last 20 messages", + "history", + (kept, ...turn) => [...(kept as object[]), ...turn].slice(-20), + ], + ["the last question, not in a list", "previous", (_kept, question) => question], +])( + "a chat history in %s keeps the variables of its last two turns", + async (_shape, variable, next) => { + setup(); + const prompt = await promptOf({ + messages: [ + { role: "system", content: "Answer as the {{team}} desk." }, + { role: "placeholder", name: "history" }, + { role: "user", content: "{{question}}" }, + ], + }); + const turns: WeakRef[] = []; + let kept: unknown = variable === "history" ? [] : undefined; + let last: ChatMessage[] = []; + let older: ChatMessage | undefined; + + // An unused variable marks the variables of each turn. + for (let turn = 0; turn < 300; turn += 1) { + const marker: ChatMessage = { role: "user", content: `turn ${turn}` }; turns.push(new WeakRef(marker)); last = prompt.compile({ team: "billing", - question: `question ${turns.length}`, - history, + question: `question ${turn}`, marker: [marker], - }) as ChatMessage[]; - history = [ - ...history, - last.at(-1)!, - { role: "assistant", content: `answer ${turns.length}` }, - ]; + [variable]: kept, + } as PromptVariables) as ChatMessage[]; + const question = last.at(-1)!; + + if (turn === 297) older = question; + kept = next(kept, question, { role: "assistant", content: `answer ${turn}` }); } await collect(); - return turns.filter((turn) => turn.deref()).length; - }; + expect(turns.filter((turn) => turn.deref()).length).toBe(2); + expect(promptAttributesFor(last)?.["td.prompt.variables"]).toContain( + '"question":"question 299"', + ); + expect(promptAttributesFor(older)).toEqual({ + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + }); + }, +); - const kept = await chat(150); - expect(kept).toBeLessThan(150); - expect(await chat(150)).toBe(kept); +test("a guard prompt keeps the variables of its request, also after other compiles", async () => { + setup(); + const answer = await promptOf(RAG_PROMPT); + const guard = await promptOf({ + name: "guard", + messages: [ + { role: "system", content: "Is this safe?" }, + { role: "placeholder", name: "conversation" }, + ], + }); + const other = await promptOf({ name: "other", messages: [{ role: "user", content: "{{q}}" }] }); + const variables = '{"documents":"The refund policy","question":"Where is #4?"}'; - const variables = JSON.parse(promptAttributesFor(last)!["td.prompt.variables"] as string) as { - history: ChatMessage[]; - }; - expect(variables.history).toHaveLength(598); + const messages = answer.compile({ + documents: "The refund policy", + question: "Where is #4?", + }) as ChatMessage[]; + // Other requests of the server compile before this request runs its guard. + for (let index = 0; index < 150; index += 1) other.compile({ q: `${index}` }); + guard.compile({ conversation: messages }); + expect(promptAttributesFor(messages)?.["td.prompt.variables"]).toBe(variables); + + // An agent loop runs the guard before each step. + const conversation = [...messages]; + for (let step = 0; step < 150; step += 1) { + guard.compile({ conversation }); + conversation.push({ role: "assistant", content: `step ${step}` }); + } + expect(promptAttributesFor(conversation)?.["td.prompt.variables"]).toBe(variables); }); test("withPrompt wins over a compiled message of another prompt, through a parent and in updates", async () => { diff --git a/sdks/python/README.md b/sdks/python/README.md index 67207ed..d060322 100644 --- a/sdks/python/README.md +++ b/sdks/python/README.md @@ -279,12 +279,15 @@ and `marshal` reject it: give them `dict(message)`. With input capture on in the client when `compile()` runs, a compiled message keeps the variables of its `compile()` alive while the message lives. With input capture off, it keeps no variables. -When `compile()` gets a message in a variable, or in a list or tuple in a variable (for example a -chat history in a placeholder), and the message is from a `compile()` at least 100 compiles older, -that older `compile()` drops its variables. Thus a chat history of compiled messages keeps the -variables of about the last 100 compiles, not of each turn. A call that sends only messages of such -an older `compile()` records its prompt without `td.prompt.variables`. The SDK does not look deeper -in a variable for messages. +When `compile()` gets a compiled message in a variable, or in a list or tuple in a variable, the +`compile()` that made the message keeps its variables. But each `compile()` whose message that +`compile()` got removes its variables. For a chat history in a placeholder, turn 3 thus removes the +variables of turn 1 and keeps those of turn 2. A chat history of compiled messages keeps the +variables of its last two turns, also when you cut it, and a call that sends only messages of +previous turns records its prompt without `td.prompt.variables`. A guard prompt that gets the +messages before their call does not remove their variables. The SDK does not look deeper in a +variable for messages. If you keep compiled messages and do not give them to `compile()`, each one +keeps the variables of its `compile()`: keep copies (`dict(message)`) to free them. For your own spans, `prompt_attributes_for(messages)` gives the attributes to pass as `attributes`. It gives `{}` inside `with_prompt`, and a span with `capture_input=False` drops diff --git a/sdks/python/src/telemetry_dev/_prompts.py b/sdks/python/src/telemetry_dev/_prompts.py index 59d8a1b..4811f14 100644 --- a/sdks/python/src/telemetry_dev/_prompts.py +++ b/sdks/python/src/telemetry_dev/_prompts.py @@ -53,11 +53,6 @@ _PROMPT_KEY = otel_context.create_key("telemetry_dev.prompt") # How many messages prompt_attributes_for looks at, at each end of the list. _TAG_LOOKUP = 1000 -# A compile drops its variables when a compile at least this many compiles newer takes one of its -# messages in a variable: its messages are then the history of a chat, and the newest compile names -# the request. Without this, a history that keeps a compiled message of each turn would keep the -# history list of every turn. -_HISTORY_AGE = 100 class PromptModel(TypedDict): @@ -110,13 +105,15 @@ def compile( prompt and a copy of these variables and of each list in them, so that a model call outside ``with_prompt`` that sends the message records them; see ``prompt_attributes_for``. The objects in a list are not copied. The message holds no - variables when input capture is off in the client. It drops them when a compile 100 or - more compiles newer takes one of its messages in a variable, as a chat history does. A - check of the exact type (``type(message) is dict``), ``yaml.safe_dump``, and ``marshal`` - reject it: give them ``dict(message)``.""" + variables when input capture is off in the client. When a newer compile gets one of these + messages in a variable, the compiles whose messages this compile got remove their + variables, and this compile keeps its own. Thus a chat history keeps the variables of its + last two turns, and a guard prompt that gets the messages before their call does not + remove them. A check of the exact type (``type(message) is dict``), ``yaml.safe_dump``, and + ``marshal`` reject it: give them ``dict(message)``.""" values = _record(self, variables, kwargs) serial = next(_compiles) - _drop_history(values, serial) + _drop_history(values) client = get_client() # prompt_attributes_for cannot record the variables when input capture is off in the client. kept = values if client is None or client.capture_input else None @@ -318,8 +315,8 @@ def prompt_attributes_for(messages: object) -> dict[str, AttributeValue]: The tag is on the dict that ``compile()`` made, so a copy of it gives ``{}``. Gives ``{}`` inside ``with_prompt``, whose prompt the span gets. ``td.prompt.variables`` needs input capture on in the client, at the compile and now, and the span drops it when its own input - capture is off. A compile whose messages became the history of a newer compile gives no - variables; see ``ChatPrompt.compile``. + capture is off. The compile of a previous turn of a chat history gives no variables; see + ``ChatPrompt.compile``. It reads only a ``list`` or ``tuple``, not a subclass, so the read runs no code of the app. With input capture on, it serializes the variables, which calls the ``mask`` of the client and @@ -474,7 +471,8 @@ class _CompileTag: prompt: ChatPrompt variables: dict[str, Any] | None - """None when input capture was off, or when the messages became a history; see _HISTORY_AGE.""" + """None when input capture was off, or when the messages became a history; see + _drop_history.""" serial: int """Counts the compiles of the process, so a higher one is newer.""" @@ -494,9 +492,23 @@ def __reduce__(self) -> tuple[type[dict[str, Any]], tuple[dict[str, Any]]]: return dict, (dict(self),) -def _drop_history(values: Mapping[str, Any], serial: int) -> None: - """Drops the variables of each older compile whose message is in ``values``; see - _HISTORY_AGE.""" +def _drop_history(values: Mapping[str, Any]) -> None: + """When a compile gets a message of a previous compile in ``values``, the compiles whose + messages that previous compile got are the history of a chat and remove their variables. The + previous compile keeps its own, because its model call can come after this compile, as with a + guard prompt. Thus a chain of tags with variables is at most two tags long, for a history of + any length.""" + for tag in _tags_in(values): + # One read: a compile in another thread can remove the variables of the tag. + variables = tag.variables + if variables is not None: + for older in _tags_in(variables): + older.variables = None + + +def _tags_in(values: Mapping[str, Any]) -> Generator[_CompileTag]: + """The tags of the messages in ``values``: a value, or an item of a list or tuple in a value. + Exact types, so the read runs no code of the app.""" for value in values.values(): items: Sequence[object] = ( cast("Sequence[object]", value) @@ -506,8 +518,8 @@ def _drop_history(values: Mapping[str, Any], serial: int) -> None: for item in items: if type(item) is _CompiledMessage: tag = getattr(item, "compiled_by", None) - if tag is not None and serial - tag.serial >= _HISTORY_AGE: - tag.variables = None + if tag is not None: + yield tag def _newest_tag(messages: object) -> _CompileTag | None: diff --git a/sdks/python/tests/test_prompts.py b/sdks/python/tests/test_prompts.py index 324b01e..7b08c51 100644 --- a/sdks/python/tests/test_prompts.py +++ b/sdks/python/tests/test_prompts.py @@ -1000,13 +1000,6 @@ def test_a_message_that_a_placeholder_brings_from_an_older_compile_does_not_win( ), } assert prompt_attributes_for(summarized)["td.prompt.name"] == "summary" - # A newer compile that takes a message of a recent compile does not drop its variables. - assert prompt_attributes_for(question) == { - "td.prompt.name": "router", - "td.prompt.version": 12, - "td.prompt.label": "production", - "td.prompt.variables": '{"question":"Where is #4?"}', - } def test_prompt_attributes_for_looks_at_the_first_and_the_last_1000_messages( @@ -1234,8 +1227,45 @@ def test_with_input_capture_off_compiled_messages_keep_no_variables(make: MakeCl } -def test_a_chat_history_of_compiled_messages_keeps_the_variables_of_a_bounded_number_of_turns( +# How a chat keeps the compiled question and the answer of a turn for the next turn. +def _new_list(kept: Any, question: dict[str, Any], answer: dict[str, Any]) -> Any: + return [*kept, question, answer] + + +def _list_trimmed_in_place(kept: Any, question: dict[str, Any], answer: dict[str, Any]) -> Any: + kept.extend((question, answer)) + del kept[:-20] + return kept + + +def _new_list_of_20(kept: Any, question: dict[str, Any], answer: dict[str, Any]) -> Any: + return [*kept, question, answer][-20:] + + +def _tuple_of_20(kept: Any, question: dict[str, Any], answer: dict[str, Any]) -> Any: + return (*kept, question, answer)[-20:] + + +def _last_question(kept: Any, question: dict[str, Any], answer: dict[str, Any]) -> Any: + return question + + +@pytest.mark.parametrize( + ("variable", "first", "keep"), + [ + ("history", [], _new_list), + ("history", [], _list_trimmed_in_place), + ("history", [], _new_list_of_20), + ("history", (), _tuple_of_20), + ("previous", None, _last_question), + ], + ids=["new-list", "list-trimmed-in-place", "new-list-of-20", "tuple-of-20", "not-in-a-list"], +) +def test_a_chat_history_keeps_the_variables_of_its_last_two_turns( make: MakeClient, + variable: str, + first: Any, + keep: Callable[[Any, dict[str, Any], dict[str, Any]], Any], ) -> None: make() prompt = chat_prompt( @@ -1244,29 +1274,57 @@ def test_a_chat_history_of_compiled_messages_keeps_the_variables_of_a_bounded_nu {"role": "user", "content": "{{question}}"}, ) turns: list[weakref.ref[Document]] = [] - history: list[dict[str, Any]] = [] + kept = copy.copy(first) last: list[dict[str, Any]] = [] + older: dict[str, Any] = {} + + # An unused variable marks the variables of each turn. + for turn in range(300): + marker = Document(content=f"turn {turn}") + turns.append(weakref.ref(marker)) + last = prompt.compile( + {"team": "billing", "question": f"question {turn}", "marker": marker, variable: kept} + ) + if turn == 297: + older = last[-1] + kept = keep(kept, last[-1], {"role": "assistant", "content": f"answer {turn}"}) + gc.collect() - # Each turn keeps its compiled question in the history. An unused variable marks the variables - # of the turn. - def chat(count: int) -> int: - nonlocal history, last - for _ in range(count): - marker = Document(content=f"turn {len(turns)}") - turns.append(weakref.ref(marker)) - last = prompt.compile( - team="billing", question=f"question {len(turns)}", history=history, marker=marker - ) - history = [*history, last[-1], {"role": "assistant", "content": f"answer {len(turns)}"}] - gc.collect() - return sum(1 for turn in turns if turn() is not None) + assert sum(1 for turn in turns if turn() is not None) == 2 + assert '"question":"question 299"' in str(prompt_attributes_for(last)["td.prompt.variables"]) + assert prompt_attributes_for(older) == { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + } - kept = chat(150) - assert kept < 150 - assert chat(150) == kept - variables = json.loads(str(prompt_attributes_for(last)["td.prompt.variables"])) - assert len(variables["history"]) == 598 +def test_a_guard_prompt_keeps_the_variables_of_its_request_also_after_other_compiles( + make: MakeClient, +) -> None: + make() + answer = chat_prompt(*RAG_MESSAGES) + guard = chat_prompt( + {"role": "system", "content": "Is this safe?"}, + {"role": "placeholder", "name": "conversation"}, + name="guard", + ) + other = chat_prompt({"role": "user", "content": "{{q}}"}, name="other") + variables = '{"documents":"The refund policy","question":"Where is #4?"}' + + messages = answer.compile(documents="The refund policy", question="Where is #4?") + # Other requests of the server compile before this request runs its guard. + for index in range(150): + other.compile(q=str(index)) + guard.compile(conversation=messages) + assert prompt_attributes_for(messages)["td.prompt.variables"] == variables + + # An agent loop runs the guard before each step. + conversation = list(messages) + for step in range(150): + guard.compile(conversation=conversation) + conversation.append({"role": "assistant", "content": f"step {step}"}) + assert prompt_attributes_for(conversation)["td.prompt.variables"] == variables def test_a_fallback_prompt_records_its_name_and_label_but_no_version( From 1b099167bbd8acaf5008848ad9ba2a2ab8e35b0d Mon Sep 17 00:00:00 2001 From: ephraimduncan Date: Fri, 30 Oct 2026 00:25:02 +0000 Subject: [PATCH 7/9] fix(sdk): keep compiled variables for 5 minutes, not two turns With the last commit, a compile C removed the variables of a compile A when C got a message of a compile B, and B got a message of A. An inspection of this rule found four errors: - A chat that keeps the messages of a compile with no compiled messages in its variables kept the variables of each turn. Examples are a question prompt in a placeholder and a reply prompt for each answer. - Other code removed the variables of a shared message. A system message from startup had no variables after two turns of a chat that got it. - Two compiles before a call removed the variables of the request: a guard and then a judge of the guard, or turns of one session that add their question to the history before their calls. - The docs told the reader that a call with only messages of previous turns records no variables. But the call records the variables of the newest of them. A rule that looks at which compile got which message has one of these errors. A shared system message and a message of a chat history have the same shape. Thus a tag keeps its variables for 5 minutes after its compile. A call keeps them for 5 more minutes when the newest compile of the call is that compile. Other compiles do not change this time. After this time, a call records the prompt without td.prompt.variables. Each compile and each call removes the variables of the expired tags, also when the app keeps their messages. The SDKs find these tags by the minute of their time, through a weak reference. Thus the reference does not keep a tag in memory, and a clock change does not stop the minutes after it. In TypeScript, a runtime without WeakRef (Cloudflare Workers before 2025-05-05) keeps no variables in the tag. At one turn in 2 minutes, five chat shapes keep the variables of 3 of 300 turns in the two SDKs. A loop with no time between its compiles keeps all of their variables for 5 minutes. The shared system message, the judge of a guard, and the turns that share a history keep their variables. The tests do these checks, also with threads in Python. The READMEs, docs/sdk-conformance.md, and the docstrings tell the new rule. Refs #97 --- docs/sdk-conformance.md | 25 +- packages/sdk/README.md | 20 +- packages/sdk/src/prompts.ts | 110 +++--- packages/sdk/tests/prompts.test.ts | 410 +++++++++++++++++----- sdks/python/README.md | 20 +- sdks/python/src/telemetry_dev/_prompts.py | 99 +++--- sdks/python/tests/test_prompts.py | 339 ++++++++++++++---- 7 files changed, 759 insertions(+), 264 deletions(-) diff --git a/docs/sdk-conformance.md b/docs/sdk-conformance.md index 49e27fe..d0ed857 100644 --- a/docs/sdk-conformance.md +++ b/docs/sdk-conformance.md @@ -211,16 +211,21 @@ their text: at the call. Then they serialize the variables, which calls the `mask` of the client. A span whose effective captureInput is off drops `td.prompt.variables` from its attributes, also when the app passes the result of these functions to its own span. -- A tag keeps the variables of its compile alive while a message of the compile lives. A chat - compile C can get a tagged message of a compile B as a variable, or as an item of a list - (Python: a list or tuple) in a variable. Then each compile A whose message B got in the same way - removes its variables, and B keeps its own. Thus a chat history that keeps a compiled message of - each turn keeps the variables of the last two turns, for a history of any length or window, and - a request with only messages of previous turns records no `td.prompt.variables`. A guard prompt - that gets the messages of a request before its call does not remove their variables, also after - other compiles in the process. The SDKs do not look deeper in a variable. They keep no other - reference to the variables, except those of the last compile of each prompt object and of each - running scope, for `withPrompt` / `with_prompt`. +- A tag keeps the variables of its compile for 5 minutes. A call of these functions keeps them + for 5 more minutes when the newest compile of the call is that compile. After that time, + they give the attributes of the tag without `td.prompt.variables`, and the next compile or call + removes the variables from the tag, at most one minute late, also when the app keeps the + message. Other compiles do not change this time. Thus a chat history keeps the variables of the + compiles of the last 5 minutes, for a history of any length or window. A guard prompt, a queue, + or a compile in other code does not remove the variables of a request before a call in these 5 + minutes. A message that more than one request sends keeps its variables while a call names its + compile at least one time in 5 minutes. A request with only messages of earlier compiles gives + the variables of the newest of them, when they are less than 5 minutes old. +- The SDKs keep a weak reference to each tag with variables, by the minute of its time, thus the + reference does not keep a tag in memory. In a runtime without `WeakRef` (Cloudflare Workers with a + compatibility date before 2025-05-05, or with `disable_weak_ref`), the TypeScript tag keeps no + variables. The SDKs keep no other reference to the variables, except those of the last compile + of each prompt object and of each running scope, for `withPrompt` / `with_prompt`. - `compile()` of a text prompt returns a string, which cannot carry a tag. Outside a scope, the SDKs do not record a text prompt: use `withPrompt` / `with_prompt`. - The provider integrations call them with the request messages before the span starts, also diff --git a/packages/sdk/README.md b/packages/sdk/README.md index 9edf709..7c42fb5 100644 --- a/packages/sdk/README.md +++ b/packages/sdk/README.md @@ -198,16 +198,16 @@ Google GenAI `generateContent`, and OpenRouter chat and responses, but not for B (its body is bytes), embeddings, images, audio, video, or OpenAI Realtime. With input capture on in the client when `compile()` runs, a tagged message keeps the variables of -its `compile()` alive while the message lives. With input capture off, the tag keeps no variables. -When `compile()` gets a tagged message in a variable, or in a list in a variable, the `compile()` -that made the message keeps its variables. But each `compile()` whose message that `compile()` got -removes its variables. For a chat history in a placeholder, turn 3 thus removes the variables of -turn 1 and keeps those of turn 2. A chat history of compiled messages keeps the variables of its -last two turns, also when you cut it, and a call that sends only messages of previous turns records -its prompt without `td.prompt.variables`. A guard prompt that gets the messages before their call -does not remove their variables. The SDK does not look deeper in a variable for messages. If you -keep compiled messages and do not give them to `compile()`, each one keeps the variables of its -`compile()`: keep copies (`{ ...message }`) to free them. +its `compile()` for 5 minutes. A call keeps them for 5 more minutes when the newest `compile()` of +the call is that one. After that time, a call records the prompt without `td.prompt.variables`, +and the SDK removes the variables from the tag, also when you keep the message. Other compiles do +not change this time. Thus a chat history keeps the variables of the compiles of its last 5 +minutes, and a guard prompt or a queue before the call does not remove them. A message that you +compile one time and send in many calls keeps its variables while a call names its `compile()` at +least one time in 5 minutes. A call that sends only messages of earlier turns records the prompt of +the newest `compile()` among them, and its variables when they are less than 5 minutes old. With +input capture off, or in a runtime without `WeakRef` (Cloudflare Workers with a compatibility date +before 2025-05-05), the tag keeps no variables. For your own spans, `promptAttributesFor(messages)` gives the attributes to pass as `attributes`. It gives nothing inside `withPrompt`, and a span with `captureInput: false` drops diff --git a/packages/sdk/src/prompts.ts b/packages/sdk/src/prompts.ts index 55169d5..6f348ea 100644 --- a/packages/sdk/src/prompts.ts +++ b/packages/sdk/src/prompts.ts @@ -99,11 +99,10 @@ export interface ChatPrompt extends PromptBase { * variable. `withPrompt` records the variables of the last call before it starts or inside it. * It tags each message object that it makes with this prompt and a copy of these variables and * of each list in them, so that a model call outside `withPrompt` that sends the message records - * them; see `promptAttributesFor`. The objects in a list are not copied. The tag keeps no - * variables when input capture is off in the client. When a newer compile gets one of these - * messages in a variable, the compiles whose messages this compile got remove their variables, - * and this compile keeps its own. Thus a chat history keeps the variables of its last two turns, - * and a guard prompt that gets the messages before their call does not remove them. + * them; see `promptAttributesFor`. The objects in a list are not copied. The tag keeps the + * variables for 5 minutes after this compile and after each call that names it. Other compiles + * do not change this. The tag keeps no variables when input capture is off in the client, or + * when the runtime has no `WeakRef`. */ compile(variables?: PromptVariables): (ChatMessage | M)[]; } @@ -209,12 +208,12 @@ interface PromptScope { /** The `compile()` call that made a chat message. */ interface CompileTag { readonly prompt: Prompt; - /** - * Undefined when input capture was off, or when the messages became a history; see dropHistory. - */ + /** Undefined when input capture was off, or after `expires`; see expire. */ variables: PromptVariables | undefined; /** Counts the compiles of the process, so a higher one is newer. */ readonly serial: number; + /** When the tag removes its variables (Date.now). A call that names the tag moves it. */ + expires: number; } const FETCH_TIMEOUT_MS = 10_000; @@ -232,12 +231,21 @@ const PROMPT_KEY = createContextKey("telemetry.dev prompt"); const COMPILED_BY = Symbol("telemetry.dev compiled by"); /** How many messages promptAttributesFor looks at, at each end of the list. */ const TAG_LOOKUP = 1_000; +/** How long a tag keeps its variables after its compile, or after a call that names it. */ +const VARIABLES_TTL_MS = 300_000; +const MINUTE_MS = 60_000; const cache = new Map(); const compiledVariables = new WeakMap(); /** The Retry-After of the response that failed with this error, in milliseconds. */ const retryAfter = new WeakMap(); let compiles = 0; +/** + * The tags that keep variables, by the minute of their deadline. A weak reference, so that the + * variables of a message that the app dropped do not stay until the deadline. A minute that a + * clock change puts in the future does not stop the minutes after it. + */ +const deadlines = new Map[]>(); /** * Fetches a prompt version from telemetry.dev and keeps it. When the copy is older than @@ -383,8 +391,8 @@ export function promptAttributes( * the messages. It looks at the first 1000 and the last 1000 messages. The tag is on the message * object, so a copy of it gives nothing. Gives undefined inside `withPrompt`, whose prompt the span * gets. `td.prompt.variables` needs input capture on in the client, at the compile and now, and - * the span drops it when its own input capture is off. The compile of a previous turn of a chat - * history gives no variables; see `compile()`. It never throws. + * the span drops it when its own input capture is off. A compile gives no variables 5 minutes + * after it or after the last call that named it; see `compile()`. It never throws. */ export function promptAttributesFor(messages: unknown): Record | undefined { const config = currentClient().core?.config; @@ -392,10 +400,15 @@ export function promptAttributesFor(messages: unknown): Record now) tag.expires = now + VARIABLES_TTL_MS; + else tag.variables = undefined; + return attributesOf(tag.prompt, tag.variables, config) as Record; } catch (error) { reportError(config?.onError, error); @@ -638,21 +651,60 @@ function createPrompt(input: { if (type === "text") return fill(content.text ?? "", values); compiles += 1; - dropHistory(values); - // promptAttributesFor cannot record the variables when input capture is off in the client. - const captureInput = currentClient().core?.config.captureInput ?? true; - - return compileMessages(content.messages, values, { + const now = Date.now(); + expire(now); + const tag: CompileTag = { prompt, - variables: captureInput ? values : undefined, + variables: undefined, serial: compiles, - }); + expires: now + VARIABLES_TTL_MS, + }; + + // promptAttributesFor cannot record the variables when input capture is off in the client. + // Without WeakRef (Cloudflare Workers before 2025-05-05), expire could not remove them. + if ((currentClient().core?.config.captureInput ?? true) && typeof WeakRef === "function") { + tag.variables = values; + addDeadline(new WeakRef(tag), tag.expires); + } + + return compileMessages(content.messages, values, tag); }, }) as Prompt; return prompt; } +/** + * Removes the variables of each expired tag, also when the app keeps its messages. Thus a chat + * history keeps the variables of the last 5 minutes only, and other compiles do not change the time + * of a request. A tag that a call named gets its new deadline. + */ +function expire(now: number): void { + const minute = Math.floor(now / MINUTE_MS); + + for (const [due, refs] of deadlines) { + if (due >= minute) continue; + deadlines.delete(due); + + for (const ref of refs) { + const tag = ref.deref(); + + if (!tag) continue; + + if (tag.expires > now) addDeadline(ref, tag.expires); + else tag.variables = undefined; + } + } +} + +function addDeadline(ref: WeakRef, at: number): void { + const minute = Math.floor(at / MINUTE_MS); + const refs = deadlines.get(minute); + + if (refs) refs.push(ref); + else deadlines.set(minute, [ref]); +} + // A copy of each list too, so that a change of the app's history after compile() does not change // what a span records. function copyVariables(variables: PromptVariables): PromptVariables { @@ -664,30 +716,6 @@ function copyVariables(variables: PromptVariables): PromptVariables): void { - for (const tag of tagsIn(variables)) { - if (tag.variables) for (const older of tagsIn(tag.variables)) older.variables = undefined; - } -} - -/** The tags of the messages in `variables`: a value, or an item of a list in a value. */ -function* tagsIn(variables: PromptVariables): Generator { - for (const value of Object.values(variables)) { - for (const item of Array.isArray(value) ? value : [value]) { - const tag = tagOf(item); - - if (tag) yield tag; - } - } -} - function compileMessages( messages: readonly PromptMessage[], variables: PromptVariables, diff --git a/packages/sdk/tests/prompts.test.ts b/packages/sdk/tests/prompts.test.ts index 381df5a..81f712c 100644 --- a/packages/sdk/tests/prompts.test.ts +++ b/packages/sdk/tests/prompts.test.ts @@ -11,7 +11,6 @@ import { getPrompt, type Prompt, promptAttributesFor, - type PromptVariables, propagateAttributes, PromptError, shutdown, @@ -114,6 +113,7 @@ afterEach(async () => { await shutdown(); vi.useRealTimers(); vi.unstubAllEnvs(); + vi.unstubAllGlobals(); vi.restoreAllMocks(); }); @@ -1007,15 +1007,27 @@ test("the variables of a compile are freed with its messages", async () => { expect(freed.deref()).toBeUndefined(); }); -test("with input capture off, compiled messages keep no variables", async () => { - setup({ captureInput: false }); +// A row removes the global WeakRef, so the test keeps its own. +const TrackedRef = WeakRef; + +test.each<[string, () => void]>([ + ["input capture is off", () => setup({ captureInput: false })], + [ + "the runtime has no WeakRef (Cloudflare Workers before 2025-05-05)", + () => { + setup(); + vi.stubGlobal("WeakRef", undefined); + }, + ], +])("when %s, compiled messages keep no variables", async (_case, start) => { + start(); const prompt = await promptOf(RAG_PROMPT); const documents: WeakRef[] = []; const questions: ChatMessage[] = []; for (let turn = 0; turn < 200; turn += 1) { const document: ChatMessage = { role: "user", content: `document ${turn}` }; - documents.push(new WeakRef(document)); + documents.push(new TrackedRef(document)); const compiled = prompt.compile({ documents: [document], question: `question ${turn}` }); questions.push(compiled.at(-1) as ChatMessage); } @@ -1025,6 +1037,7 @@ test("with input capture off, compiled messages keep no variables", async () => expect(documents.filter((document) => document.deref()).length).toBe(1); await shutdown(); + vi.unstubAllGlobals(); setup(); expect(promptAttributesFor(questions)).toEqual({ "td.prompt.name": "support/reply", @@ -1033,101 +1046,328 @@ test("with input capture off, compiled messages keep no variables", async () => }); }); -// Each row keeps the compiled question and the answer of a turn for the next turn: the variable -// that gets them, and the value of that variable after the turn. -test.each<[string, string, (kept: unknown, question: object, answer: object) => unknown]>([ - ["a new list each turn", "history", (kept, ...turn) => [...(kept as object[]), ...turn]], +const CHAT_PROMPT = { + messages: [ + { role: "system", content: "Answer as the {{team}} desk." }, + { role: "placeholder", name: "history" }, + { role: "user", content: "{{question}}" }, + ], +}; + +interface ChatPrompts { + chat: Prompt; + question: Prompt; + rag: Prompt; + reply: Prompt; +} + +async function chatPrompts(): Promise { + return { + chat: await promptOf(CHAT_PROMPT), + question: await promptOf({ + messages: [{ role: "user", content: "Context: {{documents}}\n\n{{question}}" }], + }), + rag: await promptOf({ + messages: [ + { role: "system", content: "Answer as the {{team}} desk." }, + { role: "placeholder", name: "history" }, + { role: "placeholder", name: "current" }, + ], + }), + reply: await promptOf({ messages: [{ role: "assistant", content: "{{answer}}" }] }), + }; +} + +function answer(turn: number): ChatMessage { + return { role: "assistant", content: `answer ${turn}` }; +} + +// A row runs one turn of a chat. It puts `marker` in the variables of the compile that made the +// message that the chat keeps. It gives the request, the history for the next turn, and that message. +type ChatTurn = ( + prompts: ChatPrompts, + history: ChatMessage[], + turn: number, + marker: ChatMessage, +) => [request: ChatMessage[], history: ChatMessage[], kept: ChatMessage]; + +test.each<[string, ChatTurn]>([ [ - "one list trimmed in place to 20 messages", - "history", - (kept, ...turn) => { - const history = kept as object[]; - history.push(...turn); - history.splice(0, history.length - 20); - - return history; + "a new list each turn", + ({ chat }, history, turn, marker) => { + const request = chat.compile({ + team: "billing", + history, + question: `question ${turn}`, + marker: [marker], + }) as ChatMessage[]; + const kept = request.at(-1)!; + + return [request, [...history, kept, answer(turn)], kept]; }, ], [ "a new list of the last 20 messages", - "history", - (kept, ...turn) => [...(kept as object[]), ...turn].slice(-20), + ({ chat }, history, turn, marker) => { + const request = chat.compile({ + team: "billing", + history, + question: `question ${turn}`, + marker: [marker], + }) as ChatMessage[]; + const kept = request.at(-1)!; + + return [request, [...history, kept, answer(turn)].slice(-20), kept]; + }, ], - ["the last question, not in a list", "previous", (_kept, question) => question], -])( - "a chat history in %s keeps the variables of its last two turns", - async (_shape, variable, next) => { - setup(); - const prompt = await promptOf({ - messages: [ - { role: "system", content: "Answer as the {{team}} desk." }, - { role: "placeholder", name: "history" }, - { role: "user", content: "{{question}}" }, - ], - }); - const turns: WeakRef[] = []; - let kept: unknown = variable === "history" ? [] : undefined; - let last: ChatMessage[] = []; - let older: ChatMessage | undefined; - - // An unused variable marks the variables of each turn. - for (let turn = 0; turn < 300; turn += 1) { - const marker: ChatMessage = { role: "user", content: `turn ${turn}` }; - turns.push(new WeakRef(marker)); - last = prompt.compile({ + [ + "a full list that compile gets as copies", + ({ chat }, history, turn, marker) => { + const request = chat.compile({ team: "billing", + history: history.map((message) => ({ ...message })), question: `question ${turn}`, marker: [marker], - [variable]: kept, - } as PromptVariables) as ChatMessage[]; - const question = last.at(-1)!; - - if (turn === 297) older = question; - kept = next(kept, question, { role: "assistant", content: `answer ${turn}` }); - } - await collect(); - - expect(turns.filter((turn) => turn.deref()).length).toBe(2); - expect(promptAttributesFor(last)?.["td.prompt.variables"]).toContain( - '"question":"question 299"', - ); - expect(promptAttributesFor(older)).toEqual({ - "td.prompt.name": "support/reply", - "td.prompt.version": 12, - "td.prompt.label": "production", - }); - }, -); + }) as ChatMessage[]; + const kept = request.at(-1)!; + + return [request, [...history, kept, answer(turn)], kept]; + }, + ], + [ + "a question prompt for each turn, in a placeholder of the chat prompt", + ({ question, rag }, history, turn, marker) => { + const [kept] = question.compile({ + documents: `document ${turn}`, + question: `question ${turn}`, + marker: [marker], + }) as ChatMessage[]; + const request = rag.compile({ team: "billing", history, current: [kept!] }) as ChatMessage[]; + + return [request, [...history, kept!, answer(turn)], kept!]; + }, + ], + [ + "raw questions and a reply prompt for each answer", + ({ chat, reply }, history, turn, marker) => { + const request = chat.compile({ + team: "billing", + history, + question: `question ${turn}`, + }) as ChatMessage[]; + const [kept] = reply.compile({ answer: `answer ${turn}`, marker: [marker] }) as ChatMessage[]; + + return [request, [...history, { role: "user", content: `question ${turn}` }, kept!], kept!]; + }, + ], +])("a chat history in %s keeps the variables of the last 5 minutes", async (_shape, run) => { + vi.useFakeTimers({ toFake: ["Date"] }); + setup(); + const prompts = await chatPrompts(); + const markers: WeakRef[] = []; + let history: ChatMessage[] = []; + let request: ChatMessage[] = []; + let older: ChatMessage | undefined; + + // One turn in 2 minutes, for 300 turns. An unused variable marks the variables of each turn. + for (let turn = 0; turn < 300; turn += 1) { + if (turn > 0) vi.advanceTimersByTime(120_000); + const marker: ChatMessage = { role: "user", content: `turn ${turn}` }; + markers.push(new WeakRef(marker)); + let kept: ChatMessage; + [request, history, kept] = run(prompts, history, turn, marker); + // The model call of the turn. + promptAttributesFor(request); + + if (turn === 296) older = kept; + } + await collect(); + + // Turns 297, 298, and 299: 4, 2, and 0 minutes old. + expect(markers.filter((marker) => marker.deref()).length).toBe(3); + expect(promptAttributesFor(request)?.["td.prompt.variables"]).toContain("question 299"); + expect(promptAttributesFor(older)).toEqual({ + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + }); +}); + +function variablesOf(messages: unknown): unknown { + return promptAttributesFor(messages)?.["td.prompt.variables"]; +} + +const REFUND_VARIABLES = '{"documents":"The refund policy","question":"Where is #4?"}'; + +interface LivePrompts { + answer: Prompt; + guard: Prompt; + judge: Prompt; + other: Prompt; + chat: Prompt; + system: Prompt; + withSystem: Prompt; +} -test("a guard prompt keeps the variables of its request, also after other compiles", async () => { +// A row compiles one or more requests and runs other compiles before their calls. It gives the +// variables that each call records and the variables of the compile of its request. +type LiveRequests = (prompts: LivePrompts) => Promise<[recorded: unknown, compiled: string][]>; + +test.each<[string, LiveRequests]>([ + [ + "150 compiles of other requests, then a guard", + async ({ answer, guard, other }) => { + const request = answer.compile({ documents: "The refund policy", question: "Where is #4?" }); + for (let index = 0; index < 150; index += 1) other.compile({ q: `${index}` }); + guard.compile({ conversation: request }); + + return [[variablesOf(request), REFUND_VARIABLES]]; + }, + ], + [ + "a guard before each of 150 agent steps", + async ({ answer, guard }) => { + const conversation = answer.compile({ + documents: "The refund policy", + question: "Where is #4?", + }) as ChatMessage[]; + + for (let step = 0; step < 150; step += 1) { + guard.compile({ conversation }); + conversation.push({ role: "assistant", content: `step ${step}` }); + } + + return [[variablesOf(conversation), REFUND_VARIABLES]]; + }, + ], + [ + "a guard, then a judge of the messages of the guard", + async ({ answer, guard, judge }) => { + const request = answer.compile({ documents: "The refund policy", question: "Where is #4?" }); + judge.compile({ conversation: guard.compile({ conversation: request }) }); + + return [[variablesOf(request), REFUND_VARIABLES]]; + }, + ], + [ + "three turns of a session that add their question to the history before their calls", + async ({ chat }) => { + const history: ChatMessage[] = []; + let open!: () => void; + const queue = new Promise((resolve) => { + open = resolve; + }); + const turn = async (question: string) => { + const request = chat.compile({ team: "billing", history, question }) as ChatMessage[]; + history.push(request.at(-1)!); + await queue; + + return variablesOf(request); + }; + const turns = ["q1", "q2", "q3"].map(turn); + open(); + const [first, second, third] = await Promise.all(turns); + + return [ + [first, '{"team":"billing","history":[],"question":"q1"}'], + [second, '{"team":"billing","history":[{"role":"user","content":"q1"}],"question":"q2"}'], + [ + third, + '{"team":"billing","history":[{"role":"user","content":"q1"},{"role":"user","content":"q2"}],"question":"q3"}', + ], + ]; + }, + ], + [ + "a system message compiled at startup, after 3 turns of a chat that gets it", + async ({ system, withSystem }) => { + const startup = system.compile({ product: "Acme" }) as ChatMessage[]; + let history: ChatMessage[] = []; + + for (let turn = 0; turn < 3; turn += 1) { + const request = withSystem.compile({ + system: startup, + history, + question: `question ${turn}`, + }) as ChatMessage[]; + promptAttributesFor(request); + history = [...history, request.at(-1)!, answer(turn)]; + } + + return [ + [ + variablesOf([...startup, { role: "user", content: "Where is #4?" }]), + '{"product":"Acme"}', + ], + ]; + }, + ], +])("a request keeps its variables until its call: %s", async (_case, run) => { setup(); - const answer = await promptOf(RAG_PROMPT); - const guard = await promptOf({ - name: "guard", - messages: [ - { role: "system", content: "Is this safe?" }, - { role: "placeholder", name: "conversation" }, - ], + const calls = await run({ + answer: await promptOf(RAG_PROMPT), + guard: await promptOf({ + name: "guard", + messages: [ + { role: "system", content: "Is this safe?" }, + { role: "placeholder", name: "conversation" }, + ], + }), + judge: await promptOf({ + name: "judge", + messages: [ + { role: "system", content: "Check the answer of the guard." }, + { role: "placeholder", name: "conversation" }, + ], + }), + other: await promptOf({ name: "other", messages: [{ role: "user", content: "{{q}}" }] }), + chat: await promptOf(CHAT_PROMPT), + system: await promptOf({ + name: "system", + messages: [{ role: "system", content: "You support {{product}}." }], + }), + withSystem: await promptOf({ + messages: [ + { role: "placeholder", name: "system" }, + { role: "placeholder", name: "history" }, + { role: "user", content: "{{question}}" }, + ], + }), }); - const other = await promptOf({ name: "other", messages: [{ role: "user", content: "{{q}}" }] }); - const variables = '{"documents":"The refund policy","question":"Where is #4?"}'; - const messages = answer.compile({ - documents: "The refund policy", - question: "Where is #4?", - }) as ChatMessage[]; - // Other requests of the server compile before this request runs its guard. - for (let index = 0; index < 150; index += 1) other.compile({ q: `${index}` }); - guard.compile({ conversation: messages }); - expect(promptAttributesFor(messages)?.["td.prompt.variables"]).toBe(variables); - - // An agent loop runs the guard before each step. - const conversation = [...messages]; - for (let step = 0; step < 150; step += 1) { - guard.compile({ conversation }); - conversation.push({ role: "assistant", content: `step ${step}` }); + for (const [recorded, compiled] of calls) expect(recorded).toBe(compiled); +}); + +test("a compile keeps its variables for 5 minutes after it or after the last call that names it", async () => { + vi.useFakeTimers({ toFake: ["Date"] }); + setup(); + const chat = await promptOf(CHAT_PROMPT); + let history: ChatMessage[] = []; + + for (let turn = 0; turn < 5; turn += 1) { + const request = chat.compile({ team: "billing", history, question: `q${turn}` }); + promptAttributesFor(request); + history = [...history, request.at(-1) as ChatMessage, answer(turn)]; } - expect(promptAttributesFor(conversation)?.["td.prompt.variables"]).toBe(variables); + // These calls send only messages of the chat, so they name the newest compile among them. + const summary = [...history, { role: "user", content: "Summarize the chat." }]; + const earlier = history.slice(0, -2); + + expect(variablesOf(summary)).toContain('"question":"q4"'); + expect(variablesOf(earlier)).toContain('"question":"q3"'); + + vi.advanceTimersByTime(299_999); + expect(variablesOf(summary)).toContain('"question":"q4"'); + vi.advanceTimersByTime(1); + expect(variablesOf(earlier)).toBeUndefined(); + // The call 1 ms before moved the deadline of q4. + expect(variablesOf(summary)).toContain('"question":"q4"'); + vi.advanceTimersByTime(300_000); + expect(promptAttributesFor(summary)).toEqual({ + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + }); }); test("withPrompt wins over a compiled message of another prompt, through a parent and in updates", async () => { diff --git a/sdks/python/README.md b/sdks/python/README.md index d060322..05b5657 100644 --- a/sdks/python/README.md +++ b/sdks/python/README.md @@ -278,16 +278,16 @@ pydantic) do not change. A check of the exact type (`type(message) is dict`), `y and `marshal` reject it: give them `dict(message)`. With input capture on in the client when `compile()` runs, a compiled message keeps the variables -of its `compile()` alive while the message lives. With input capture off, it keeps no variables. -When `compile()` gets a compiled message in a variable, or in a list or tuple in a variable, the -`compile()` that made the message keeps its variables. But each `compile()` whose message that -`compile()` got removes its variables. For a chat history in a placeholder, turn 3 thus removes the -variables of turn 1 and keeps those of turn 2. A chat history of compiled messages keeps the -variables of its last two turns, also when you cut it, and a call that sends only messages of -previous turns records its prompt without `td.prompt.variables`. A guard prompt that gets the -messages before their call does not remove their variables. The SDK does not look deeper in a -variable for messages. If you keep compiled messages and do not give them to `compile()`, each one -keeps the variables of its `compile()`: keep copies (`dict(message)`) to free them. +of its `compile()` for 5 minutes. A call keeps them for 5 more minutes when the newest `compile()` +of the call is that one. After that time, a call records the prompt without +`td.prompt.variables`, and the SDK removes the variables from the message, also when you keep the +message. Other compiles do not change this time. Thus a chat history keeps the variables of the +compiles of its last 5 minutes, and a guard prompt, a queue, or a compile in another thread before +the call does not remove them. A message that you compile one time and send in many calls keeps its +variables while a call names its `compile()` at least one time in 5 minutes. A call that sends only +messages of earlier turns records the prompt of the newest `compile()` among them, and its +variables when they are less than 5 minutes old. With input capture off, the message keeps no +variables. For your own spans, `prompt_attributes_for(messages)` gives the attributes to pass as `attributes`. It gives `{}` inside `with_prompt`, and a span with `capture_input=False` drops diff --git a/sdks/python/src/telemetry_dev/_prompts.py b/sdks/python/src/telemetry_dev/_prompts.py index 4811f14..f2177cc 100644 --- a/sdks/python/src/telemetry_dev/_prompts.py +++ b/sdks/python/src/telemetry_dev/_prompts.py @@ -14,7 +14,7 @@ import urllib.parse import urllib.request from collections import OrderedDict -from collections.abc import Callable, Generator, Mapping, Sequence +from collections.abc import Callable, Generator, Mapping from contextlib import contextmanager from contextvars import ContextVar from dataclasses import dataclass, field @@ -23,7 +23,7 @@ from email.utils import parsedate_to_datetime from time import monotonic from typing import IO, Any, Literal, TypeAlias, TypedDict, cast, overload -from weakref import WeakKeyDictionary +from weakref import WeakKeyDictionary, ref from opentelemetry import context as otel_context from opentelemetry.context import Context @@ -53,6 +53,9 @@ _PROMPT_KEY = otel_context.create_key("telemetry_dev.prompt") # How many messages prompt_attributes_for looks at, at each end of the list. _TAG_LOOKUP = 1000 +# How long a tag keeps its variables after its compile, or after a call that names it. +_VARIABLES_TTL_S = 300.0 +_MINUTE_S = 60.0 class PromptModel(TypedDict): @@ -104,20 +107,20 @@ def compile( or inside it. Each message that it makes is a ``dict`` subclass that also holds this prompt and a copy of these variables and of each list in them, so that a model call outside ``with_prompt`` that sends the message records them; see - ``prompt_attributes_for``. The objects in a list are not copied. The message holds no - variables when input capture is off in the client. When a newer compile gets one of these - messages in a variable, the compiles whose messages this compile got remove their - variables, and this compile keeps its own. Thus a chat history keeps the variables of its - last two turns, and a guard prompt that gets the messages before their call does not - remove them. A check of the exact type (``type(message) is dict``), ``yaml.safe_dump``, and + ``prompt_attributes_for``. The objects in a list are not copied. The message keeps the + variables for 5 minutes after this compile and after each call that names it. Other + compiles do not change this. The message holds no variables when input capture is off in + the client. A check of the exact type (``type(message) is dict``), ``yaml.safe_dump``, and ``marshal`` reject it: give them ``dict(message)``.""" values = _record(self, variables, kwargs) - serial = next(_compiles) - _drop_history(values) + now = monotonic() + _expire(now) + tag = _CompileTag(self, None, next(_compiles), now + _VARIABLES_TTL_S) client = get_client() # prompt_attributes_for cannot record the variables when input capture is off in the client. - kept = values if client is None or client.capture_input else None - tag = _CompileTag(self, kept, serial) + if client is None or client.capture_input: + tag.variables = values + _add_deadline(ref(tag), tag.expires) compiled: list[dict[str, Any]] = [] for message in self.messages: if message.get("role") != "placeholder": @@ -315,8 +318,8 @@ def prompt_attributes_for(messages: object) -> dict[str, AttributeValue]: The tag is on the dict that ``compile()`` made, so a copy of it gives ``{}``. Gives ``{}`` inside ``with_prompt``, whose prompt the span gets. ``td.prompt.variables`` needs input capture on in the client, at the compile and now, and the span drops it when its own input - capture is off. The compile of a previous turn of a chat history gives no variables; see - ``ChatPrompt.compile``. + capture is off. A compile gives no variables 5 minutes after it or after the last call that + named it; see ``ChatPrompt.compile``. It reads only a ``list`` or ``tuple``, not a subclass, so the read runs no code of the app. With input capture on, it serializes the variables, which calls the ``mask`` of the client and @@ -326,11 +329,19 @@ def prompt_attributes_for(messages: object) -> dict[str, AttributeValue]: try: if isinstance(otel_context.get_value(_PROMPT_KEY), _PromptScope): return {} + now = monotonic() + _expire(now) tag = _newest_tag(messages) if tag is None: return {} + # One read: _expire in another thread can remove the variables of the tag. + variables = tag.variables + if tag.expires > now: + tag.expires = now + _VARIABLES_TTL_S + else: + variables = tag.variables = None capture = client if client is not None and client.capture_input else None - return _attributes_of(tag.prompt, tag.variables, capture) + return _attributes_of(tag.prompt, variables, capture) except Exception as exc: if client is not None: client.report("prompt_attributes_for failed", exc) @@ -471,13 +482,17 @@ class _CompileTag: prompt: ChatPrompt variables: dict[str, Any] | None - """None when input capture was off, or when the messages became a history; see - _drop_history.""" + """None when input capture was off, or after ``expires``; see _expire.""" serial: int """Counts the compiles of the process, so a higher one is newer.""" + expires: float + """When the tag removes its variables (monotonic). A call that names the tag moves it.""" _compiles = itertools.count() +# The tags that keep variables, by the minute of their deadline. A weak reference, so that the +# variables of a message that the app dropped do not stay until the deadline. +_deadlines: dict[int, list[ref[_CompileTag]]] = {} class _CompiledMessage(dict[str, Any]): @@ -492,34 +507,28 @@ def __reduce__(self) -> tuple[type[dict[str, Any]], tuple[dict[str, Any]]]: return dict, (dict(self),) -def _drop_history(values: Mapping[str, Any]) -> None: - """When a compile gets a message of a previous compile in ``values``, the compiles whose - messages that previous compile got are the history of a chat and remove their variables. The - previous compile keeps its own, because its model call can come after this compile, as with a - guard prompt. Thus a chain of tags with variables is at most two tags long, for a history of - any length.""" - for tag in _tags_in(values): - # One read: a compile in another thread can remove the variables of the tag. - variables = tag.variables - if variables is not None: - for older in _tags_in(variables): - older.variables = None - - -def _tags_in(values: Mapping[str, Any]) -> Generator[_CompileTag]: - """The tags of the messages in ``values``: a value, or an item of a list or tuple in a value. - Exact types, so the read runs no code of the app.""" - for value in values.values(): - items: Sequence[object] = ( - cast("Sequence[object]", value) - if type(value) is list or type(value) is tuple - else (value,) - ) - for item in items: - if type(item) is _CompiledMessage: - tag = getattr(item, "compiled_by", None) - if tag is not None: - yield tag +def _expire(now: float) -> None: + """Remove the variables of each expired tag, also when the app keeps its messages. Thus a chat + history keeps the variables of the last 5 minutes only, and other compiles do not change the + time of a request. A tag that a call named gets its new deadline. Threads add only to minutes + that are not due, and ``list()`` and ``pop()`` of a dict are atomic, so the threads share the + minutes without a lock.""" + minute = int(now // _MINUTE_S) + for due in list(_deadlines): + if due >= minute: + continue + for tag_ref in _deadlines.pop(due, ()): + tag = tag_ref() + if tag is None: + continue + if tag.expires > now: + _add_deadline(tag_ref, tag.expires) + else: + tag.variables = None + + +def _add_deadline(tag_ref: ref[_CompileTag], at: float) -> None: + _deadlines.setdefault(int(at // _MINUTE_S), []).append(tag_ref) def _newest_tag(messages: object) -> _CompileTag | None: diff --git a/sdks/python/tests/test_prompts.py b/sdks/python/tests/test_prompts.py index 7b08c51..0cd3e31 100644 --- a/sdks/python/tests/test_prompts.py +++ b/sdks/python/tests/test_prompts.py @@ -19,7 +19,7 @@ from email.utils import format_datetime from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from types import SimpleNamespace -from typing import Any, NoReturn +from typing import Any, NoReturn, TypeAlias from urllib.parse import parse_qsl, urlsplit import pytest @@ -1227,71 +1227,132 @@ def test_with_input_capture_off_compiled_messages_keep_no_variables(make: MakeCl } -# How a chat keeps the compiled question and the answer of a turn for the next turn. -def _new_list(kept: Any, question: dict[str, Any], answer: dict[str, Any]) -> Any: - return [*kept, question, answer] +CHAT_MESSAGES = ( + {"role": "system", "content": "Answer as the {{team}} desk."}, + {"role": "placeholder", "name": "history"}, + {"role": "user", "content": "{{question}}"}, +) + + +@dataclasses.dataclass +class ChatPrompts: + chat: ChatPrompt + question: ChatPrompt + rag: ChatPrompt + reply: ChatPrompt -def _list_trimmed_in_place(kept: Any, question: dict[str, Any], answer: dict[str, Any]) -> Any: - kept.extend((question, answer)) - del kept[:-20] - return kept +def _answer(turn: int) -> dict[str, Any]: + return {"role": "assistant", "content": f"answer {turn}"} -def _new_list_of_20(kept: Any, question: dict[str, Any], answer: dict[str, Any]) -> Any: - return [*kept, question, answer][-20:] +# What a row gives for one turn of a chat: the request, the history for the next turn, and the kept +# message. The row puts `marker` in the variables of the compile that made the kept message. +ChatTurn: TypeAlias = tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]] -def _tuple_of_20(kept: Any, question: dict[str, Any], answer: dict[str, Any]) -> Any: - return (*kept, question, answer)[-20:] +def _new_list( + p: ChatPrompts, history: list[dict[str, Any]], turn: int, marker: Document +) -> ChatTurn: + request = p.chat.compile( + team="billing", history=history, question=f"question {turn}", marker=marker + ) + return request, [*history, request[-1], _answer(turn)], request[-1] + +def _new_list_of_20( + p: ChatPrompts, history: list[dict[str, Any]], turn: int, marker: Document +) -> ChatTurn: + request = p.chat.compile( + team="billing", history=history, question=f"question {turn}", marker=marker + ) + return request, [*history, request[-1], _answer(turn)][-20:], request[-1] -def _last_question(kept: Any, question: dict[str, Any], answer: dict[str, Any]) -> Any: - return question + +def _full_list_as_copies( + p: ChatPrompts, history: list[dict[str, Any]], turn: int, marker: Document +) -> ChatTurn: + request = p.chat.compile( + team="billing", + history=[dict(message) for message in history], + question=f"question {turn}", + marker=marker, + ) + return request, [*history, request[-1], _answer(turn)], request[-1] + + +def _question_prompt_in_a_placeholder( + p: ChatPrompts, history: list[dict[str, Any]], turn: int, marker: Document +) -> ChatTurn: + [kept] = p.question.compile( + documents=f"document {turn}", question=f"question {turn}", marker=marker + ) + request = p.rag.compile(team="billing", history=history, current=[kept]) + return request, [*history, kept, _answer(turn)], kept + + +def _raw_questions_and_a_reply_prompt( + p: ChatPrompts, history: list[dict[str, Any]], turn: int, marker: Document +) -> ChatTurn: + request = p.chat.compile(team="billing", history=history, question=f"question {turn}") + [kept] = p.reply.compile(answer=f"answer {turn}", marker=marker) + return request, [*history, {"role": "user", "content": f"question {turn}"}, kept], kept @pytest.mark.parametrize( - ("variable", "first", "keep"), + "run", [ - ("history", [], _new_list), - ("history", [], _list_trimmed_in_place), - ("history", [], _new_list_of_20), - ("history", (), _tuple_of_20), - ("previous", None, _last_question), + _new_list, + _new_list_of_20, + _full_list_as_copies, + _question_prompt_in_a_placeholder, + _raw_questions_and_a_reply_prompt, + ], + ids=[ + "new-list", + "new-list-of-20", + "full-list-as-copies", + "question-prompt-in-a-placeholder", + "raw-questions-and-a-reply-prompt", ], - ids=["new-list", "list-trimmed-in-place", "new-list-of-20", "tuple-of-20", "not-in-a-list"], ) -def test_a_chat_history_keeps_the_variables_of_its_last_two_turns( +def test_a_chat_history_keeps_the_variables_of_the_last_5_minutes( make: MakeClient, - variable: str, - first: Any, - keep: Callable[[Any, dict[str, Any], dict[str, Any]], Any], + clock: list[float], + run: Callable[[ChatPrompts, list[dict[str, Any]], int, Document], ChatTurn], ) -> None: make() - prompt = chat_prompt( - {"role": "system", "content": "Answer as the {{team}} desk."}, - {"role": "placeholder", "name": "history"}, - {"role": "user", "content": "{{question}}"}, + prompts = ChatPrompts( + chat=chat_prompt(*CHAT_MESSAGES), + question=chat_prompt({"role": "user", "content": "Context: {{documents}}\n\n{{question}}"}), + rag=chat_prompt( + {"role": "system", "content": "Answer as the {{team}} desk."}, + {"role": "placeholder", "name": "history"}, + {"role": "placeholder", "name": "current"}, + ), + reply=chat_prompt({"role": "assistant", "content": "{{answer}}"}), ) - turns: list[weakref.ref[Document]] = [] - kept = copy.copy(first) - last: list[dict[str, Any]] = [] + markers: list[weakref.ref[Document]] = [] + history: list[dict[str, Any]] = [] + request: list[dict[str, Any]] = [] older: dict[str, Any] = {} - # An unused variable marks the variables of each turn. + # One turn in 2 minutes, for 300 turns. An unused variable marks the variables of each turn. for turn in range(300): + if turn > 0: + clock[0] += 120 marker = Document(content=f"turn {turn}") - turns.append(weakref.ref(marker)) - last = prompt.compile( - {"team": "billing", "question": f"question {turn}", "marker": marker, variable: kept} - ) - if turn == 297: - older = last[-1] - kept = keep(kept, last[-1], {"role": "assistant", "content": f"answer {turn}"}) + markers.append(weakref.ref(marker)) + request, history, kept = run(prompts, history, turn, marker) + # The model call of the turn. + prompt_attributes_for(request) + if turn == 296: + older = kept gc.collect() - assert sum(1 for turn in turns if turn() is not None) == 2 - assert '"question":"question 299"' in str(prompt_attributes_for(last)["td.prompt.variables"]) + # Turns 297, 298, and 299: 4, 2, and 0 minutes old. + assert sum(1 for marker in markers if marker() is not None) == 3 + assert "question 299" in str(prompt_attributes_for(request)["td.prompt.variables"]) assert prompt_attributes_for(older) == { "td.prompt.name": "support/reply", "td.prompt.version": 12, @@ -1299,32 +1360,184 @@ def test_a_chat_history_keeps_the_variables_of_its_last_two_turns( } -def test_a_guard_prompt_keeps_the_variables_of_its_request_also_after_other_compiles( - make: MakeClient, -) -> None: - make() - answer = chat_prompt(*RAG_MESSAGES) - guard = chat_prompt( - {"role": "system", "content": "Is this safe?"}, - {"role": "placeholder", "name": "conversation"}, - name="guard", - ) - other = chat_prompt({"role": "user", "content": "{{q}}"}, name="other") - variables = '{"documents":"The refund policy","question":"Where is #4?"}' +def _variables_of(messages: object) -> object: + return prompt_attributes_for(messages).get("td.prompt.variables") + + +REFUND_VARIABLES = '{"documents":"The refund policy","question":"Where is #4?"}' - messages = answer.compile(documents="The refund policy", question="Where is #4?") - # Other requests of the server compile before this request runs its guard. + +@dataclasses.dataclass +class LivePrompts: + answer: ChatPrompt + guard: ChatPrompt + judge: ChatPrompt + other: ChatPrompt + chat: ChatPrompt + system: ChatPrompt + with_system: ChatPrompt + + +# A row compiles one or more requests and runs other compiles before their calls. It gives the +# variables that each call records and the variables of the compile of its request. +LiveCalls: TypeAlias = list[tuple[object, str]] + + +def _other_compiles_then_a_guard(p: LivePrompts) -> LiveCalls: + request = p.answer.compile(documents="The refund policy", question="Where is #4?") for index in range(150): - other.compile(q=str(index)) - guard.compile(conversation=messages) - assert prompt_attributes_for(messages)["td.prompt.variables"] == variables + p.other.compile(q=str(index)) + p.guard.compile(conversation=request) + return [(_variables_of(request), REFUND_VARIABLES)] - # An agent loop runs the guard before each step. - conversation = list(messages) + +def _a_guard_before_each_agent_step(p: LivePrompts) -> LiveCalls: + conversation = p.answer.compile(documents="The refund policy", question="Where is #4?") for step in range(150): - guard.compile(conversation=conversation) + p.guard.compile(conversation=conversation) conversation.append({"role": "assistant", "content": f"step {step}"}) - assert prompt_attributes_for(conversation)["td.prompt.variables"] == variables + return [(_variables_of(conversation), REFUND_VARIABLES)] + + +def _a_guard_then_a_judge_of_the_guard(p: LivePrompts) -> LiveCalls: + request = p.answer.compile(documents="The refund policy", question="Where is #4?") + p.judge.compile(conversation=p.guard.compile(conversation=request)) + return [(_variables_of(request), REFUND_VARIABLES)] + + +def _turns_in_threads_that_share_a_history(p: LivePrompts) -> LiveCalls: + history: list[dict[str, Any]] = [] + turns = [threading.Event() for _ in range(4)] + compiled = threading.Barrier(3, timeout=5) + recorded: list[object] = [None] * 3 + + def turn(index: int) -> None: + turns[index].wait(5) + request = p.chat.compile(team="billing", history=history, question=f"q{index + 1}") + history.append(request[-1]) + turns[index + 1].set() + # A guard call or a queue: the call comes after the other turns compiled. + compiled.wait() + recorded[index] = _variables_of(request) + + threads = [threading.Thread(target=turn, args=(index,)) for index in range(3)] + for thread in threads: + thread.start() + turns[0].set() + for thread in threads: + thread.join(5) + return list( + zip( + recorded, + [ + '{"team":"billing","history":[],"question":"q1"}', + '{"team":"billing","history":[{"role":"user","content":"q1"}],"question":"q2"}', + '{"team":"billing","history":[{"role":"user","content":"q1"},' + '{"role":"user","content":"q2"}],"question":"q3"}', + ], + strict=True, + ) + ) + + +def _a_system_message_from_startup_after_a_chat_that_gets_it(p: LivePrompts) -> LiveCalls: + startup = p.system.compile(product="Acme") + history: list[dict[str, Any]] = [] + for turn in range(3): + request = p.with_system.compile( + system=startup, history=history, question=f"question {turn}" + ) + prompt_attributes_for(request) + history = [*history, request[-1], _answer(turn)] + return [ + ( + _variables_of([*startup, {"role": "user", "content": "Where is #4?"}]), + '{"product":"Acme"}', + ) + ] + + +@pytest.mark.parametrize( + "run", + [ + _other_compiles_then_a_guard, + _a_guard_before_each_agent_step, + _a_guard_then_a_judge_of_the_guard, + _turns_in_threads_that_share_a_history, + _a_system_message_from_startup_after_a_chat_that_gets_it, + ], + ids=[ + "150-other-compiles-then-a-guard", + "a-guard-before-each-of-150-agent-steps", + "a-guard-then-a-judge-of-the-guard", + "turns-in-threads-that-share-a-history", + "a-system-message-from-startup-after-a-chat-that-gets-it", + ], +) +def test_a_request_keeps_its_variables_until_its_call( + make: MakeClient, run: Callable[[LivePrompts], LiveCalls] +) -> None: + make() + calls = run( + LivePrompts( + answer=chat_prompt(*RAG_MESSAGES), + guard=chat_prompt( + {"role": "system", "content": "Is this safe?"}, + {"role": "placeholder", "name": "conversation"}, + name="guard", + ), + judge=chat_prompt( + {"role": "system", "content": "Check the answer of the guard."}, + {"role": "placeholder", "name": "conversation"}, + name="judge", + ), + other=chat_prompt({"role": "user", "content": "{{q}}"}, name="other"), + chat=chat_prompt(*CHAT_MESSAGES), + system=chat_prompt( + {"role": "system", "content": "You support {{product}}."}, name="system" + ), + with_system=chat_prompt( + {"role": "placeholder", "name": "system"}, + {"role": "placeholder", "name": "history"}, + {"role": "user", "content": "{{question}}"}, + ), + ) + ) + + for recorded, compiled in calls: + assert recorded == compiled + + +def test_a_compile_keeps_its_variables_for_5_minutes_after_it_or_after_the_last_call_that_names_it( + make: MakeClient, clock: list[float] +) -> None: + make() + chat = chat_prompt(*CHAT_MESSAGES) + history: list[dict[str, Any]] = [] + for turn in range(5): + request = chat.compile(team="billing", history=history, question=f"q{turn}") + prompt_attributes_for(request) + history = [*history, request[-1], _answer(turn)] + # These calls send only messages of the chat, so they name the newest compile among them. + summary = [*history, {"role": "user", "content": "Summarize the chat."}] + earlier = history[:-2] + start = clock[0] + + assert '"question":"q4"' in str(_variables_of(summary)) + assert '"question":"q3"' in str(_variables_of(earlier)) + + clock[0] = start + 299.5 + assert '"question":"q4"' in str(_variables_of(summary)) + clock[0] = start + 300 + assert _variables_of(earlier) is None + # The call 0.5 seconds before moved the deadline of q4. + assert '"question":"q4"' in str(_variables_of(summary)) + clock[0] = start + 600 + assert prompt_attributes_for(summary) == { + "td.prompt.name": "support/reply", + "td.prompt.version": 12, + "td.prompt.label": "production", + } def test_a_fallback_prompt_records_its_name_and_label_but_no_version( From b2a0e09e3a2fee5aa8c5d6951a0bac9dac019cfa Mon Sep 17 00:00:00 2001 From: ephraimduncan Date: Fri, 30 Oct 2026 01:14:04 +0000 Subject: [PATCH 8/9] fix(sdk): measure the variables time of compiled messages in elapsed time In #117, the prompt cache and the Retry-After wait moved to performance.now(), so that a change of the system clock does not move them. But a compile tag used Date.now() for the 5 minutes that it keeps its variables. Thus a clock that moved back an hour kept the variables of each compile for one more hour. A clock that moved forward removed the variables before 5 minutes. The Python tags use monotonic(). With this change, compile() and promptAttributesFor read performance.now(). The test of the 5 minutes moves the system clock back an hour, and the tag continues to remove the variables after 5 minutes. docs/sdk-conformance.md tells that the SDKs measure this time in elapsed time. Refs #97 --- docs/sdk-conformance.md | 3 ++- packages/sdk/src/prompts.ts | 13 ++++++++----- packages/sdk/tests/prompts.test.ts | 6 ++++-- 3 files changed, 14 insertions(+), 8 deletions(-) diff --git a/docs/sdk-conformance.md b/docs/sdk-conformance.md index d0ed857..54debda 100644 --- a/docs/sdk-conformance.md +++ b/docs/sdk-conformance.md @@ -212,7 +212,8 @@ their text: whose effective captureInput is off drops `td.prompt.variables` from its attributes, also when the app passes the result of these functions to its own span. - A tag keeps the variables of its compile for 5 minutes. A call of these functions keeps them - for 5 more minutes when the newest compile of the call is that compile. After that time, + for 5 more minutes when the newest compile of the call is that compile. The SDKs measure this + time in elapsed time, so a change of the system clock does not move it. After that time, they give the attributes of the tag without `td.prompt.variables`, and the next compile or call removes the variables from the tag, at most one minute late, also when the app keeps the message. Other compiles do not change this time. Thus a chat history keeps the variables of the diff --git a/packages/sdk/src/prompts.ts b/packages/sdk/src/prompts.ts index 6f348ea..7837933 100644 --- a/packages/sdk/src/prompts.ts +++ b/packages/sdk/src/prompts.ts @@ -212,7 +212,10 @@ interface CompileTag { variables: PromptVariables | undefined; /** Counts the compiles of the process, so a higher one is newer. */ readonly serial: number; - /** When the tag removes its variables (Date.now). A call that names the tag moves it. */ + /** + * When the tag removes its variables, in `performance.now()` time. A call that names the tag + * moves it. + */ expires: number; } @@ -242,8 +245,8 @@ const retryAfter = new WeakMap(); let compiles = 0; /** * The tags that keep variables, by the minute of their deadline. A weak reference, so that the - * variables of a message that the app dropped do not stay until the deadline. A minute that a - * clock change puts in the future does not stop the minutes after it. + * variables of a message that the app dropped do not stay until the deadline. The minutes are not + * sorted, since a tag that a call named goes back in at its new minute. */ const deadlines = new Map[]>(); @@ -400,7 +403,7 @@ export function promptAttributesFor(messages: unknown): Record([ }, ], ])("a chat history in %s keeps the variables of the last 5 minutes", async (_shape, run) => { - vi.useFakeTimers({ toFake: ["Date"] }); + vi.useFakeTimers({ toFake: ["Date", "performance"] }); setup(); const prompts = await chatPrompts(); const markers: WeakRef[] = []; @@ -1339,7 +1339,7 @@ test.each<[string, LiveRequests]>([ }); test("a compile keeps its variables for 5 minutes after it or after the last call that names it", async () => { - vi.useFakeTimers({ toFake: ["Date"] }); + vi.useFakeTimers({ toFake: ["Date", "performance"] }); setup(); const chat = await promptOf(CHAT_PROMPT); let history: ChatMessage[] = []; @@ -1362,6 +1362,8 @@ test("a compile keeps its variables for 5 minutes after it or after the last cal expect(variablesOf(earlier)).toBeUndefined(); // The call 1 ms before moved the deadline of q4. expect(variablesOf(summary)).toContain('"question":"q4"'); + // The system clock moves back an hour; 5 minutes still pass. + vi.setSystemTime(Date.now() - 3_600_000); vi.advanceTimersByTime(300_000); expect(promptAttributesFor(summary)).toEqual({ "td.prompt.name": "support/reply", From 0c2f2b6e17a534d403179dd277db9136ab17f815 Mon Sep 17 00:00:00 2001 From: ephraimduncan Date: Fri, 30 Oct 2026 01:14:06 +0000 Subject: [PATCH 9/9] docs(sdk): tell that a call with no compiled message records its input In #117, the READMEs tell that a model call outside withPrompt and with_prompt records the compiled messages as its input when input capture is on. This branch replaced that sentence. Its text told that the SDK does not record a copy of a compiled message. The READMEs now tell that such a call gets no td.prompt.* attributes. With input capture on, the call records the messages as its input. Refs #97 --- docs/sdk-conformance.md | 2 +- packages/sdk/README.md | 7 ++++--- sdks/python/README.md | 9 +++++---- 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/sdk-conformance.md b/docs/sdk-conformance.md index 54debda..d9790a8 100644 --- a/docs/sdk-conformance.md +++ b/docs/sdk-conformance.md @@ -195,7 +195,7 @@ their text: - A copy has no tag: a spread, `structuredClone`, or a JSON round trip (TypeScript); `dict()`, `.copy()`, `copy.copy`, `copy.deepcopy`, pickle, a JSON round trip, `type(message)(message)`, or `dataclasses.asdict` (Python). A new message with the same text has no tag either, thus a - message from another call, task, or prompt version is never recorded. + call never records the prompt of a message from another call, task, or prompt version. - `promptAttributesFor(messages)` / `prompt_attributes_for(messages)` take a list of messages or one message (Python: a `list` or `tuple`, not a subclass, so the read runs no code of the app). They look at the first 1000 and the last 1000 items and give the name, version, label, and diff --git a/packages/sdk/README.md b/packages/sdk/README.md index 7c42fb5..ab40381 100644 --- a/packages/sdk/README.md +++ b/packages/sdk/README.md @@ -190,9 +190,10 @@ among them, also when input capture is off, and `td.prompt.variables` when it is at the first 1000 and the last 1000 messages of the call. A message from a placeholder variable keeps the tag of its own, older `compile()`, so it does not win over the `compile()` that put it in the list. A copy of a message (a spread, `structuredClone`, or JSON) and a new message with the same -text have no tag, so the SDK does not record them. Bedrock Converse and Google GenAI take another -message shape: change the compiled messages in place, or use `withPrompt`. A text prompt compiles to -a string, which cannot carry a tag: use `withPrompt`. `withPrompt` wins over the tag. The +text have no tag, so a call that sends only such messages gets no `td.prompt.*` attributes; with +input capture on, the call still records them as its input. Bedrock Converse and Google GenAI take +another message shape: change the compiled messages in place, or use `withPrompt`. A text prompt +compiles to a string, which cannot carry a tag: use `withPrompt`. `withPrompt` wins over the tag. The integrations do it for OpenAI chat completions and responses, Anthropic messages, Bedrock Converse, Google GenAI `generateContent`, and OpenRouter chat and responses, but not for Bedrock InvokeModel (its body is bytes), embeddings, images, audio, video, or OpenAI Realtime. diff --git a/sdks/python/README.md b/sdks/python/README.md index 05b5657..a3010b1 100644 --- a/sdks/python/README.md +++ b/sdks/python/README.md @@ -263,10 +263,11 @@ records. A chat call that sends such messages gets `td.prompt.name`, `td.prompt. the call. A message from a placeholder variable keeps the prompt of its own, older `compile()`, so it does not win over the `compile()` that put it in the list. A copy of a message (`dict()`, `.copy()`, `copy.copy`, `copy.deepcopy`, pickle, JSON, `type(message)(message)`, or -`dataclasses.asdict`) and a new message with the same text do not hold the prompt, so the SDK does -not record them. Bedrock Converse and Google GenAI take another message shape: change the compiled -dicts in place, or use `with_prompt`. A text prompt compiles to a string, which cannot hold its -prompt: use `with_prompt`. `with_prompt` wins over the message. The integrations do it for OpenAI +`dataclasses.asdict`) and a new message with the same text do not hold the prompt, so a call that +sends only such messages gets no `td.prompt.*` attributes; with input capture on, the call still +records them as its input. Bedrock Converse and Google GenAI take another message shape: change the +compiled dicts in place, or use `with_prompt`. A text prompt compiles to a string, which cannot hold +its prompt: use `with_prompt`. `with_prompt` wins over the message. The integrations do it for OpenAI chat completions and responses, Anthropic messages (for `messages.stream()`, at `__enter__`), Bedrock Converse, Google GenAI `generate_content`, LiteLLM completion and responses, and OpenRouter chat and responses, but not for Bedrock InvokeModel (its body is bytes), embeddings, images, audio,