The agent returned a wrong answer, or a right one after forty seconds, and your logs say INFO step 3 complete. You can't tell which model call was slow, which tool timed out and got retried, or whether the prompt cache did anything. A trace answers all of that in one view, if the spans are shaped well.
This post builds that shape in Python with OpenTelemetry: one invoke_agent span per run, a chat span per model call and an execute_tool span per tool call, using the GenAI attribute names from the OpenTelemetry spec. The script uses a stub model and a virtual clock, so it needs no API key, and I ran it twice with byte-identical output. It also includes a small checker for the conventions you can put in CI.
If you want the backend side (Grafana, Datadog, alert rules, a wrapper around the Anthropic client), read LLM observability with OpenTelemetry and agent observability with LangSmith and Braintrust. This post covers the part that post predates: the agent-level span layout and a status check on the conventions themselves, which have moved and changed.
What is the status of the OpenTelemetry GenAI conventions?
On 2026-10-09 I opened the old page at opentelemetry.io/docs/specs/semconv/gen-ai/. It now says the GenAI semantic conventions have moved to the semantic-conventions-genai repository and are no longer maintained on the main site. I cloned that repository (its latest commit was dated 2026-10-07) and read the spans, agent spans and client inference documents directly.
What I found:
- Every spans document I read is marked Status: Development. The README's schema URL section says
TODO. - The repo's changelog fragments include entries labelled breaking, deprecation, enhancement and clarification that were not yet in a release (the
CHANGELOG.mdshows only an empty "Unreleased" section). - One breaking entry removed the cache-token attributes from the internal
invoke_agentspan, saying cache breakdowns there aggregate across models and are misleading, and that consumers should aggregate them from inference spans. - I searched the repository's docs, model and changelog directories for
gen_ai.systemand found no match. The provider is nowgen_ai.provider.name. The older post linked above still usesgen_ai.system, which is a sign of how fast this moved. - The Python package
opentelemetry-semantic-conventions0.65b0, which I installed, exportsGEN_AI_USAGE_CACHE_CREATION_INPUT_TOKENSwith the valuegen_ai.usage.cache_creation.input_tokens, while the spec repo documentsgen_ai.usage.cache_write.input_tokens. Packages lag the spec.
The practical rules: pin versions, write attribute names as string literals in one module you control, and re-read the spec before upgrading. Don't expect a dashboard built on one version's names to survive the next.
What should the span tree for an agent look like?
The spec defines three operations that matter for an agent loop, each with a span-name pattern and a kind:
| Operation | Span name | Kind | Required attributes (as read) |
|---|---|---|---|
invoke_agent (in process) | invoke_agent {agent name} | INTERNAL | gen_ai.operation.name |
chat (model call) | chat {request model} | CLIENT, or INTERNAL for an in-process model | gen_ai.operation.name, gen_ai.provider.name |
execute_tool | execute_tool {tool name} | INTERNAL | gen_ai.operation.name, gen_ai.tool.name |
Nest them the way the work nests. The agent span is the root, each model call and each tool call is its child, and a failed tool call sits next to the model call that asked for it.
Three details from the spec that change how you write the code:
- Sampling attributes go in at creation. The spec says
gen_ai.operation.name,gen_ai.provider.name,gen_ai.request.modeland the server address and port SHOULD be provided when the span is created, because samplers decide at that moment. In the script they are passed tostart_span, not set afterwards. - Don't invent a conversation id. If your framework or app has a real session or thread id, set
gen_ai.conversation.id. The spec says not to fall back to a new UUID, a trace id or a hash of the content. The demo uses an app-managed ticket id. - Manual instrumentation of tools is expected. The spec says tools are often executed by application code and encourages developers to follow the convention for their own tool calls. Auto-instrumentation will not know about your functions.
How do you count tokens when caching is on?
This one bites cost dashboards. The spec's description of gen_ai.usage.input_tokens says the value SHOULD include all types of input tokens, including cached tokens, and gives an example: 100 text tokens of which 40 are cached plus 200 image tokens gives 300 total input tokens, with 40 as cache-read tokens. Cache reads and writes are reported separately as gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_write.input_tokens.
Anthropic's usage block is shaped differently. Its prompt caching docs say input_tokens is only the tokens after the last cache breakpoint, and that total input is cache_read_input_tokens + cache_creation_input_tokens + input_tokens. So the mapping function adds the three before it sets the GenAI attribute. If you forward usage.input_tokens as-is, a healthy cached run looks like it uses almost no input, and any cost per token you compute from it will be wrong. The prompt caching loop post shows how the three numbers move when the prefix breaks.
The script: a traced agent loop with a stub model
Install the SDK (I used opentelemetry-api and opentelemetry-sdk 1.44.0 on Python 3.13) and save the file below as agent_trace.py. Run it with python3 -I agent_trace.py.
How it works, briefly. timed() starts a span on a virtual clock, makes it current so children nest under it, and records errors by hand (use_span is told not to do it, otherwise you would get the exception recorded twice). A scripted list stands in for the model: two tool calls, then a final answer. The second tool call times out. The agent catches that, as a real agent would hand the error back to the model, and carries on. The token numbers in the script are invented for the demo. IDs come from a seeded generator, so the trace and span ids are repeatable.
"""Trace a stubbed agent loop with OpenTelemetry using GenAI semantic-convention attribute names.
No network and no API key: the "model" is a scripted stub, the clock is virtual, and ids are seeded,
so the printed output is identical on every run.
"""
import json
import random
import re
from contextlib import contextmanager
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.trace.id_generator import IdGenerator
from opentelemetry.trace import SpanKind, Status, StatusCode
MODEL = "stub-model-1"
START_NS = 1_791_536_400 * 1_000_000_000 # 2026-10-09T09:00:00Z
class SeededIds(IdGenerator):
def __init__(self, seed):
self.rng = random.Random(seed)
def generate_span_id(self):
return self.rng.getrandbits(64) or 1
def generate_trace_id(self):
return self.rng.getrandbits(128) or 1
class Clock:
ns = START_NS
@contextmanager
def timed(tracer, name, kind, ms, attributes):
"""Start a span on the virtual clock, make it current, and record errors by hand."""
span = tracer.start_span(name, kind=kind, start_time=Clock.ns, attributes=attributes)
with trace.use_span(span, end_on_exit=False, record_exception=False, set_status_on_exception=False):
try:
yield span
except Exception as exc:
span.set_attribute("error.type", type(exc).__qualname__)
span.record_exception(exc, timestamp=Clock.ns + ms * 1_000_000)
span.set_status(Status(StatusCode.ERROR, str(exc)))
raise
finally:
Clock.ns += ms * 1_000_000
span.end(end_time=Clock.ns)
def usage_to_attrs(u):
"""Anthropic-style usage -> GenAI attributes. The spec says input_tokens includes cached tokens."""
read = u.get("cache_read_input_tokens", 0)
write = u.get("cache_creation_input_tokens", 0)
return {
"gen_ai.usage.input_tokens": u["input_tokens"] + read + write,
"gen_ai.usage.output_tokens": u["output_tokens"],
"gen_ai.usage.cache_read.input_tokens": read,
"gen_ai.usage.cache_write.input_tokens": write,
}
# Scripted stub "model" replies. The token numbers are invented for this demo.
SCRIPT = [
{"id": "resp_001", "stop": "tool_use", "tool": ("search_orders", "call_a1"), "ms": 420,
"usage": {"input_tokens": 120, "cache_creation_input_tokens": 1700, "cache_read_input_tokens": 0, "output_tokens": 64}},
{"id": "resp_002", "stop": "tool_use", "tool": ("decide_refund", "call_a2"), "ms": 380,
"usage": {"input_tokens": 310, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 1700, "output_tokens": 48}},
{"id": "resp_003", "stop": "end_turn", "tool": None, "ms": 510,
"usage": {"input_tokens": 150, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 1700, "output_tokens": 120}},
]
def run_tool(name):
if name == "decide_refund":
raise TimeoutError("refund API timed out after 5s")
return "ok"
def run_agent(tracer):
with timed(tracer, "invoke_agent support_agent", SpanKind.INTERNAL, 0, {
"gen_ai.operation.name": "invoke_agent",
"gen_ai.agent.name": "support_agent",
"gen_ai.conversation.id": "ticket-1001", # app-managed id, not invented by the tracer
}) as agent:
total_in = total_out = 0
for step, reply in enumerate(SCRIPT, start=1):
with timed(tracer, f"chat {MODEL}", SpanKind.CLIENT, reply["ms"], {
"gen_ai.operation.name": "chat",
"gen_ai.provider.name": "anthropic",
"gen_ai.request.model": MODEL,
}) as chat:
attrs = usage_to_attrs(reply["usage"])
chat.set_attributes({**attrs, "gen_ai.response.id": reply["id"],
"gen_ai.response.finish_reasons": [reply["stop"]]})
total_in += attrs["gen_ai.usage.input_tokens"]
total_out += attrs["gen_ai.usage.output_tokens"]
if reply["tool"]:
tool, call_id = reply["tool"]
try:
with timed(tracer, f"execute_tool {tool}", SpanKind.INTERNAL, 90 if tool == "search_orders" else 5000, {
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": tool,
"gen_ai.tool.call.id": call_id,
"gen_ai.tool.type": "function",
}):
run_tool(tool)
except TimeoutError:
pass # the agent recovers: the error goes back to the model as a tool result
agent.set_attributes({"gen_ai.usage.input_tokens": total_in, "gen_ai.usage.output_tokens": total_out,
"app.steps": len(SCRIPT)})
def render(spans):
children, by_id = {}, {}
for s in spans:
by_id[s.context.span_id] = s
children.setdefault(s.parent.span_id if s.parent else None, []).append(s)
shown = ("gen_ai.operation.name", "gen_ai.tool.name", "gen_ai.usage.input_tokens",
"gen_ai.usage.output_tokens", "gen_ai.usage.cache_read.input_tokens",
"gen_ai.usage.cache_write.input_tokens", "gen_ai.response.finish_reasons", "error.type")
def walk(span, depth):
ms = (span.end_time - span.start_time) // 1_000_000
status = span.status.status_code.name
print(f"{' ' * depth}{span.name} [{span.kind.name}] {ms} ms status={status}")
for k in shown:
if k in span.attributes and not (k == "gen_ai.operation.name"):
print(f"{' ' * depth} {k} = {span.attributes[k]}")
for c in sorted(children.get(span.context.span_id, []), key=lambda x: x.start_time):
walk(c, depth + 1)
for root in children[None]:
walk(root, 0)
RULES = {
"chat": ("gen_ai.operation.name", "gen_ai.provider.name"),
"execute_tool": ("gen_ai.operation.name", "gen_ai.tool.name"),
"invoke_agent": ("gen_ai.operation.name",),
}
NAME_PATTERNS = {"chat": r"^chat \S+$", "execute_tool": r"^execute_tool \S+$", "invoke_agent": r"^invoke_agent( \S+)?$"}
KINDS = {"chat": "CLIENT", "execute_tool": "INTERNAL", "invoke_agent": "INTERNAL"}
def as_dicts(spans):
return [{"name": s.name, "kind": s.kind.name, "error": s.status.status_code == StatusCode.ERROR,
"attrs": dict(s.attributes)} for s in spans]
def check_conventions(items):
"""Small subset of the spec: required attributes, span-name shape, span kind."""
problems = []
for s in items:
attrs = s["attrs"]
op = attrs.get("gen_ai.operation.name")
if op not in RULES:
problems.append(f"{s['name']}: unknown or missing gen_ai.operation.name {op!r}")
continue
for req in RULES[op]:
if req not in attrs:
problems.append(f"{s['name']}: missing required attribute {req}")
if not re.match(NAME_PATTERNS[op], s["name"]):
problems.append(f"{s['name']}: span name does not match the {op} pattern")
if s["kind"] != KINDS[op]:
problems.append(f"{s['name']}: kind is {s['kind']}, convention says {KINDS[op]}")
if s["error"] and "error.type" not in attrs:
problems.append(f"{s['name']}: ERROR status without error.type")
return problems
def rollup(spans):
chats = [s for s in spans if s.attributes.get("gen_ai.operation.name") == "chat"]
total_in = sum(s.attributes["gen_ai.usage.input_tokens"] for s in chats)
read = sum(s.attributes["gen_ai.usage.cache_read.input_tokens"] for s in chats)
errors = [s.name for s in spans if s.status.status_code == StatusCode.ERROR]
print(f"chat calls: {len(chats)}, input tokens: {total_in}, of which cache reads: {read} ({100 * read / total_in:.0f}%)")
print(f"error spans: {errors}")
if __name__ == "__main__":
exporter = InMemorySpanExporter()
provider = TracerProvider(resource=Resource.create({"service.name": "support-agent-demo", "service.instance.id": "demo-1"}), id_generator=SeededIds(7))
provider.add_span_processor(SimpleSpanProcessor(exporter))
run_agent(provider.get_tracer("agent-demo"))
spans = exporter.get_finished_spans()
print("== trace tree ==")
render(spans)
print("\n== rollup ==")
rollup(spans)
print("\n== convention check (good run) ==")
print(check_conventions(as_dicts(spans)) or "no problems")
print("\n== convention check (deliberately broken copies) ==")
broken = as_dicts(spans)
del broken[0]["attrs"]["gen_ai.provider.name"] # first finished span is a chat span
broken[1]["name"] = "execute_tool" # the name lost its tool suffix
broken[1]["kind"] = "CLIENT"
del broken[3]["attrs"]["error.type"] # the failed tool call lost its error.type
for p in check_conventions(broken):
print(p)
print("\n== raw span: the failed tool call ==")
failed = [s for s in spans if s.status.status_code == StatusCode.ERROR][0]
raw = json.loads(failed.to_json())
raw["events"][0]["attributes"]["exception.stacktrace"] = "<traceback omitted for the article>"
print(json.dumps(raw, indent=2))
What the trace shows
Output of the first section, == trace tree ==, on 2026-10-09. Durations come from the virtual clock, so they are the numbers I put in the script and say nothing about real latency.
invoke_agent support_agent [INTERNAL] 6400 ms status=UNSET
gen_ai.usage.input_tokens = 5680
gen_ai.usage.output_tokens = 232
chat stub-model-1 [CLIENT] 420 ms status=UNSET
gen_ai.usage.input_tokens = 1820
gen_ai.usage.output_tokens = 64
gen_ai.usage.cache_read.input_tokens = 0
gen_ai.usage.cache_write.input_tokens = 1700
gen_ai.response.finish_reasons = ('tool_use',)
execute_tool search_orders [INTERNAL] 90 ms status=UNSET
gen_ai.tool.name = search_orders
chat stub-model-1 [CLIENT] 380 ms status=UNSET
gen_ai.usage.input_tokens = 2010
gen_ai.usage.output_tokens = 48
gen_ai.usage.cache_read.input_tokens = 1700
gen_ai.usage.cache_write.input_tokens = 0
gen_ai.response.finish_reasons = ('tool_use',)
execute_tool decide_refund [INTERNAL] 5000 ms status=ERROR
gen_ai.tool.name = decide_refund
error.type = TimeoutError
chat stub-model-1 [CLIENT] 510 ms status=UNSET
gen_ai.usage.input_tokens = 1850
gen_ai.usage.output_tokens = 120
gen_ai.usage.cache_read.input_tokens = 1700
gen_ai.usage.cache_write.input_tokens = 0
gen_ai.response.finish_reasons = ('end_turn',)
Read it the way you would read a real trace. The run took 6.4 seconds, and 5 of those were one tool call that failed. The model calls took 0.4 to 0.5 seconds each. Cache reads rose from 0 to 1,700 after the first call, so caching worked from call two. The agent span carries total input and output tokens, but, following the changelog entry above, no cache breakdown. The rollup function does that aggregation from the chat spans:
chat calls: 3, input tokens: 5680, of which cache reads: 3400 (60%)
error spans: ['execute_tool decide_refund']
Sixty percent of input tokens came from the cache. That figure is only as good as the three-way token sum in the mapping function.
The raw span for the failed tool call
This is the failed execute_tool span as the SDK serialises it. I removed the resource block, replaced the stack trace text and kept everything else.
{
"name": "execute_tool decide_refund",
"context": {
"trace_id": "0x6513270e269e0d37f2a74de452e6b438",
"span_id": "0xe8e25d940ed90475",
"trace_state": "[]"
},
"kind": "SpanKind.INTERNAL",
"parent_id": "0x0c5c7fd0a6a3a450",
"start_time": "2026-10-09T09:00:00.890000Z",
"end_time": "2026-10-09T09:00:05.890000Z",
"status": {
"status_code": "ERROR",
"description": "refund API timed out after 5s"
},
"attributes": {
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": "decide_refund",
"gen_ai.tool.call.id": "call_a2",
"gen_ai.tool.type": "function",
"error.type": "TimeoutError"
},
"events": [
{
"name": "exception",
"timestamp": "2026-10-09T09:00:05.890000Z",
"attributes": {
"exception.type": "TimeoutError",
"exception.message": "refund API timed out after 5s",
"exception.stacktrace": "<traceback omitted for the article>",
"exception.escaped": "False"
}
}
],
"links": []
}
Note the error.type attribute next to the ERROR status. The spec marks error.type as conditionally required when the operation ended in an error, and its description says to use a low-cardinality identifier such as the exception's canonical name, not the message. The message lives in the status description and the exception event, where it won't blow up your metric cardinality.
A convention check you can run in CI
Tracing code drifts. Someone renames a span, forgets the provider attribute or switches the kind. The check_conventions function in the script checks the subset of the spec shown in the table above: required attributes, span-name shape, span kind, and error spans without error.type. It runs on plain dicts so you can feed it spans from a test exporter.
On the real spans it finds nothing:
no problems
Then I broke four things on purpose (removed the provider, wrote an execute_tool name without a tool, set the wrong kind, removed error.type from the failed span):
chat stub-model-1: missing required attribute gen_ai.provider.name
execute_tool: span name does not match the execute_tool pattern
execute_tool: kind is CLIENT, convention says INTERNAL
execute_tool decide_refund: ERROR status without error.type
It's strict in one place where the spec isn't. The spec says chat spans should be CLIENT but may be INTERNAL for an in-process model, so loosen KINDS if you run models locally. It is also a subset of the spec: it doesn't check value types or the opt-in attributes. Treat it as a smoke test, not validation.
Privacy: what to record and what to leave out
Everything above records identifiers, counts and names, not content. That's deliberate. The spec treats these as Opt-In attributes: gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, gen_ai.tool.definitions, gen_ai.tool.call.arguments and gen_ai.tool.call.result. Its capture section says instructions, inputs and outputs are considered sensitive, that instrumentations SHOULD NOT capture them by default, and that the default usage pattern is to record none of them. (The spec names OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT as an example of an opt-in switch, in its notes on the memory attributes, so expect instrumentation libraries to use a switch like that.) In your own code, take the same approach:
- Default to ids, sizes and counts:
app.prompt_chars,app.tool.args_bytes, tool name, response id. - Gate content capture behind a setting that is off in production and expires.
- Scrub before you record. Tool results are the usual leak: a CRM lookup returns an email address and it lands in your tracing backend.
- Remember the backend's retention. A trace store is another database of user data.
For Indian teams working under the DPDP Act, treat this as an engineering default and check obligations with a lawyer; this post doesn't give legal advice.
Sending it somewhere real
The script keeps spans in memory. To ship them, replace SimpleSpanProcessor(exporter) with a BatchSpanProcessor around an OTLP exporter from opentelemetry-exporter-otlp-proto-http, with the endpoint set by the standard environment variables. I did not run that part (no collector here), so it is untested. Keep the demo's habit of passing sampling-relevant attributes at span creation, and sample at the collector or with a parent-based sampler rather than dropping spans inside your loop.
Two follow-ups worth doing once traces flow. Put cache_read over input_tokens on a dashboard so a prefix break shows up the same day. And join tool-span error rates to your tool descriptions, because a tool that fails often is often a tool the model is calling wrongly. For loop-level guards, see token budgeting for agents, and for the framework-specific route, OpenAI Agents SDK tracing. The agent observability lesson covers what to look for once you have a trace.
Checklist
- One
invoke_agentspan per run,chatper model call,execute_toolper tool call, nested. - Sampling-relevant attributes passed when the span starts.
- Span names and kinds follow the patterns above.
- Token attributes: input includes cached tokens, cache read and write reported separately.
- Tool errors: ERROR status,
error.type, exception event. - No prompt, tool argument or tool result content unless explicitly switched on and scrubbed.
- Attribute names in one module; SDK and instrumentation versions pinned.
-
check_conventions(or similar) runs in CI against a test exporter.
What I did not verify
- No real model or tool was traced. The stub model, token numbers and durations are invented, and the output shows structure, not performance.
- The spec was read from a shallow clone of the
semantic-conventions-genairepository at its 2026-10-07 commit, plus the moved-notice page on opentelemetry.io. The Recording Errors document it links to was not read. - The OTLP export path and any backend (Grafana, Datadog, Jaeger) were not tested.
- The GenAI conventions for providers other than Anthropic, and the MCP conventions that also live in the repository, were not covered.
