# How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools

Source: https://greptime.com/blogs/2026-05-09-opentelemetry-genai-semantic-conventions

Observability for LLM applications doesn't work like traditional microservices. An LLM call produces far more telemetry than an HTTP request, but the harder problem is the shape of that data. Prompts and completions are large text blobs. Tool call parameters take a different structure every time. An agent's multi-step reasoning can't be captured in a fixed schema. And beyond the usual "which model was called and how long did it take," you also need to know how many tokens were consumed, how much the call cost, and whether the answer was any good.

Traditional OTel semantic conventions don't cover any of this. `http.request.method` and `db.system.name` are meaningless for LLM calls. The community needed a dedicated spec.

In April 2024, OpenTelemetry formed the GenAI Special Interest Group (GenAI SIG) under the Semantic Conventions SIG[^1]. The original scope was LLM client call tracing. Since then it has expanded to cover agent orchestration, MCP tool calling, content capture, and quality evaluation: six layers in all. As of this writing, the OTel docs site shows Semantic Conventions v1.41.0[^2], with the latest GitHub release at v1.41.1 (a k8s codegen fix only, no GenAI changes)[^15].

Each layer below covers what the spec defines, why it's designed that way, and where it stands in maturity. If you read our earlier piece on [Agent Observability](https://greptime.com/blogs/2025-12-11-agent-observability), this is the spec companion: how the community is standardizing around the challenges that article described.

![OTel GenAI Semantic Conventions — Six-Layer Structure](/blogs/2026-05-09-opentelemetry-genai-semantic-conventions/01-six-layers.png)

---

## How the Spec Evolved: From Early Days to v1.41

When GenAI SIG kicked off in April 2024, the goal was straightforward: define a standard set of attributes for LLM client calls (model name, token usage, latency) so that different instrumentation libraries produce consistent telemetry. Early versions introduced `gen_ai.system`, `gen_ai.request.model`, and a pair of histogram metrics.

v1.37 was the turning point. It overhauled how chat history is recorded[^16]. The old per-message events (one event per message) were replaced by three aggregated attributes (`gen_ai.system_instructions`, `gen_ai.input.messages`, `gen_ai.output.messages`) stored either on the span or in a new `gen_ai.client.inference.operation.details` event. The reason was practical: per-message events flooded multi-turn conversations with fine-grained events that were painful to query and correlate.

Every subsequent release has included GenAI changes:

| Version | GenAI Changes |
|---------|---------------|
| v1.37[^16] | Chat history revamp; `gen_ai.system` replaced by `gen_ai.provider.name` |
| v1.38[^17] | Evaluation event; tool definitions and call details; `invoke_agent` kind guidance; embeddings dimension; multimodal JSON schema |
| v1.39[^18] | MCP semantic conventions |
| v1.40[^19] | Retrieval span; cache token attributes; Anthropic input token calculation guide |
| v1.41[^20] | `execute_tool` span naming requires tool name; reasoning tokens; `invoke_workflow`; streaming metrics; split `invoke_agent` into CLIENT/INTERNAL |

![OTel GenAI Semantic Conventions — Evolution Timeline](/blogs/2026-05-09-opentelemetry-genai-semantic-conventions/04-timeline.png)

As of May 2026, the GenAI and MCP semantic conventions remain in **Development** status[^2]. The docs say it plainly: "This transition plan will be updated to include stable version before the GenAI conventions are marked as stable." No public timeline for stabilization. The 2026 Semantic Conventions Roadmap[^3] is collecting proposals from each sub-SIG, but nothing's committed.

Attribute names and structures may still change. The core concepts have settled though, and the spec provides `OTEL_SEMCONV_STABILITY_OPT_IN` to manage version transitions. Building on the spec today is a reasonable bet.

---

## Layer 1: Client Spans — Standardizing Model Calls

This is the base layer of the spec. It defines the spans produced when application code calls a GenAI model[^4].

### Inference

Each LLM call generates a span with `gen_ai.operation.name` set to `chat`, `text_completion`, or `generate_content` (for multimodal). Span kind is `CLIENT`, since models typically run on remote services.

Core attributes:

| Attribute | Meaning | Example |
|-----------|---------|---------|
| `gen_ai.provider.name` | Provider identifier | `openai`, `anthropic`, `aws.bedrock` |
| `gen_ai.request.model` | Requested model | `gpt-4o-mini` |
| `gen_ai.response.model` | Actual responding model | `gpt-4o-mini-2024-07-18` |
| `gen_ai.usage.input_tokens` | Input token count | `142` |
| `gen_ai.usage.output_tokens` | Output token count | `87` |
| `gen_ai.response.finish_reasons` | Completion reason | `["stop"]`, `["tool_calls"]` |

Here's what this looks like in a trace viewer (Jaeger, Grafana Tempo, etc.):

```json
{
  "operationName": "chat gpt-4o-mini",
  "spanKind": "CLIENT",
  "duration": "1.23s",
  "attributes": {
    "gen_ai.operation.name":          "chat",
    "gen_ai.provider.name":           "openai",
    "gen_ai.request.model":           "gpt-4o-mini",
    "gen_ai.response.model":          "gpt-4o-mini-2024-07-18",
    "gen_ai.usage.input_tokens":      142,
    "gen_ai.usage.output_tokens":     87,
    "gen_ai.response.finish_reasons": ["stop"],
    "server.address":                 "api.openai.com",
    "server.port":                    443
  }
}
```

> Note: This is a logical view for readability. The actual OTLP wire format uses `KeyValue` arrays with typed values (`stringValue`, `intValue`, etc.) and nanosecond Unix timestamps.

`server.address` and `server.port` are general networking attributes. They're not part of the GenAI spec, but instrumentation typically emits them anyway.

`provider.name` and `request.model` are intentionally separate. The same model name can be accessed through different providers (Azure OpenAI and OpenAI direct both serve GPT-4o), and `provider.name` determines which provider-specific attributes apply.

`request.model` vs `response.model` matters in practice too: you request `gpt-4o`, but the response might come from `gpt-4o-2024-08-06`. For fine-tuned models, `response.model` should be more specific than the base name.

### Embeddings and Retrievals

Embeddings (`gen_ai.operation.name=embeddings`) cover vector embedding operations, with `gen_ai.embeddings.dimension.count` recording the vector dimension. Retrievals cover the retrieval step in RAG pipelines.

### One-Line Integration

With Python + OpenAI SDK, instrumentation takes one line[^5]:

```python
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

OpenAIInstrumentor().instrument()

# OpenAI SDK calls will emit semconv-compliant spans and metrics.
# Prompt/completion events require content capture to be enabled explicitly.
client.chat.completions.create(model="gpt-4o-mini", messages=[...])
```

`opentelemetry-instrumentation-openai-v2` is the most mature GenAI instrumentation. Anthropic, AWS Bedrock, and others are covered by community libraries.

---

## Layer 2: Agent & Workflow Spans — Beyond Microservices

This is where GenAI semantic conventions diverge most from traditional OTel. Distributed tracing has HTTP spans, RPC spans, and DB spans, but nothing for "agent invocation." The GenAI spec introduces a new set of operation types[^6].

### create_agent

Describes agent creation, typically for remote agent services (OpenAI Assistants API, AWS Bedrock Agents). Span kind is `CLIENT`.

```
Span: create_agent support-router
Kind: CLIENT
Attributes:
  gen_ai.operation.name = create_agent
  gen_ai.agent.name = support-router
  gen_ai.provider.name = openai
```

### invoke_agent

Invokes an agent to perform a task. v1.41 explicitly splits two scenarios[^20]: `CLIENT` for remote calls, `INTERNAL` for local framework execution (e.g. LangGraph running agent logic in-process).

### invoke_workflow

Describes execution of a predefined workflow (added in v1.41[^20]). Agents reason autonomously; workflows follow predetermined paths.

### execute_tool

Tool execution span, kind `INTERNAL`. Starting with v1.41, the tool name must appear in the span name (`execute_tool {gen_ai.tool.name}`)[^20]. `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result` are recorded only when privacy policies permit.

### Why These Span Types Matter

Our [Agent Observability](https://greptime.com/blogs/2025-12-11-agent-observability) article discussed a core challenge: the agent's reasoning process shows up as a black box in traces. Agent span conventions crack that open:

```
invoke_agent research-assistant (INTERNAL)
├── chat gpt-4o (CLIENT)                ← Model decides to search
├── execute_tool web_search (INTERNAL)  ← Search executed
├── chat gpt-4o (CLIENT)               ← Continues reasoning with results
├── execute_tool summarize (INTERNAL)   ← Summarization
└── chat gpt-4o (CLIENT)               ← Generates final answer
```

Every step carries standardized attributes, and any compatible backend (Jaeger, Tempo, Datadog) can render the structure correctly.

---

## Layer 3: MCP Semantic Conventions — Fixing Broken Traces

Model Context Protocol (MCP) spread fast in 2025, but brought an observability problem: traces from the agent side and the MCP server side are disconnected.

Glama's analysis laid it out clearly[^7]: the agent produces Trace A, the MCP server produces Trace B, no context propagation between them. The OTel MCP semantic conventions, introduced in v1.39[^18], fix this[^8].

### Core Design

MCP runs on JSON-RPC, but the spec recommends MCP conventions over generic RPC semantic conventions. MCP spans carry context that generic RPC conventions miss, such as session and tool call details.

Client span example (stdio transport):

```
Span: tools/call get-weather
Kind: CLIENT
Attributes:
  gen_ai.operation.name = execute_tool
  mcp.method.name = tools/call
  mcp.session.id = session-xyz
  mcp.protocol.version = 2025-03-26
  gen_ai.tool.name = get-weather
  jsonrpc.request.id = 42
  network.transport = pipe            # stdio maps to pipe
```

For HTTP transport, use `network.transport = tcp` (or `quic`) with `network.protocol.name = http`[^8].

When both sides propagate W3C Trace Context, the server span nests under the client span, and trace continuity is preserved across the protocol boundary.

### Compatibility with execute_tool

The spec handles deduplication: if MCP instrumentation detects that an outer GenAI instrumentation already tracks tool execution, it enriches the existing span with MCP-specific attributes (`mcp.method.name`, `mcp.session.id`, etc.) instead of creating a duplicate.

A complete agent + MCP call chain[^8]:

```
invoke_agent weather-forecast-agent (INTERNAL)
├── chat {model} (CLIENT)                          ← GenAI model
├── tools/call get-weather (CLIENT)                ← MCP client
│   └── tools/call get-weather (SERVER)            ← MCP server
└── chat {model} (CLIENT)                          ← GenAI model
```

### MCP-Specific Metrics

Four MCP metrics: `mcp.client.operation.duration` / `mcp.server.operation.duration` (operation latency) and `mcp.client.session.duration` / `mcp.server.session.duration` (session lifetime).

---

## Layer 4: Events and Content Capture — Balancing Privacy and Observability

Traditional OTel HTTP spans rarely need to worry about "should we record the request body?" LLM applications are different: prompt and completion content is both the most valuable debugging data and the most sensitive.

### Two Core Events

`gen_ai.client.inference.operation.details` (added in v1.37)[^9]: records full input and output for a GenAI call. It's opt-in, and backends can process it as events/logs, decoupled from trace lifecycle and storage policies.

`gen_ai.evaluation.result`: records quality evaluation results via `gen_ai.evaluation.score.value` and `gen_ai.evaluation.score.label`[^9]. A relevancy evaluator might return score.value=0.85, score.label="relevant".

Here's the `operation.details` event with content capture enabled:

```json
{
  "eventName": "gen_ai.client.inference.operation.details",
  "attributes": {
    "gen_ai.system_instructions": [
      {"type": "text", "content": "You are a helpful customer support agent."}
    ],
    "gen_ai.input.messages": [
      {
        "role": "user",
        "parts": [{"type": "text", "content": "Where is my order #12345?"}]
      }
    ],
    "gen_ai.output.messages": [
      {
        "role": "assistant",
        "parts": [{"type": "text", "content": "Your order #12345 shipped this morning and should arrive tomorrow."}],
        "finish_reason": "stop"
      }
    ]
  }
}
```

When recorded on the `operation.details` event, messages follow the spec's JSON schema in structured form[^9]. On span attributes, where backends lack structured attribute support, serialized JSON strings work too.

Content is absent by default. Many instrumentations gate it behind `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true`. In production, most teams end up on mode three: external storage with a reference URL on the span.

### Three Content Recording Modes

The spec defines three approaches[^4]:

**Not recorded (default).** Content capture is off.

**On span attributes.** `gen_ai.input.messages` and `gen_ai.output.messages` as span attributes. Convenient, but size-limited and visible to anyone with trace access.

**External storage + span reference.** Full content in external storage (S3, GreptimeDB, etc.), span holds only a reference URL. Independent IAM and retention policies. The spec recommends this for production with significant telemetry volume or sensitive data[^4].

![Three Content Capture Modes Compared](/blogs/2026-05-09-opentelemetry-genai-semantic-conventions/03-capture-modes.png)

---

## Layer 5: Metrics — The Two Essential Client Histograms

The GenAI spec defines multiple metrics on both client and server sides. The two most used client histograms[^10]:

### gen_ai.client.operation.duration

End-to-end latency per GenAI operation, in seconds. Dimensions: `gen_ai.operation.name`, `gen_ai.request.model`, `gen_ai.provider.name`.

### gen_ai.client.token.usage

Token consumption per operation, unit `{token}`. Recommended bucket boundaries follow exponential growth: `[1, 4, 16, 64, 256, 1024, 4096, 16384, 65536, 262144, 1048576, 4194304, 16777216, 67108864]`, covering 1 token to 67M tokens.

Two counting rules worth knowing. When a provider reports both used and billable tokens, the instrumentation should report the billable count. When token counts can't be obtained efficiently, the instrumentation should omit them rather than guess.

These two metrics answer most operational questions: which model is most expensive, where latency is worst, how token consumption is trending. In our [GenAI demo](https://greptime.com/blogs/2026-03-10-llm-observability-otel-genai-greptimedb), they're written to GreptimeDB via OTLP and queried with PromQL:

```promql
# p95 token consumption
histogram_quantile(0.95,
  sum(rate(gen_ai_client_token_usage_bucket[5m])) by (le, gen_ai_token_type)
)
```

---

## Layer 6: Provider-Specific Conventions — From Generic to Specialized

Generic GenAI attributes cover common ground. Each provider has unique capabilities, handled through provider-specific conventions.

### OpenAI

OpenAI has the most detailed provider convention so far[^11]. With `gen_ai.provider.name` set to `openai`, the spec adds:

- `gen_ai.usage.cache_read.input_tokens`: tokens read from the provider's cache
- `gen_ai.usage.cache_creation.input_tokens`: tokens written to the provider's cache
- `gen_ai.usage.reasoning.output_tokens`: tokens consumed during reasoning (o1/o3 series, added in v1.41[^20])

Cached input is typically cheaper than regular input (exact discounts vary by model). Reasoning tokens are an additional cost specific to reasoning models.

### Anthropic, AWS Bedrock, Azure AI Inference

Anthropic (`gen_ai.provider.name=anthropic`) includes a calculation guide for `gen_ai.usage.input_tokens`[^19], because its billing model differs from OpenAI's. AWS Bedrock (`aws.bedrock`) and Azure AI Inference (`azure.ai.inference`) extend platform-specific attributes.

Design principle: `gen_ai.provider.name` is the discriminator. It determines which provider-specific attributes should appear. An OpenAI span shouldn't carry `aws.bedrock.*` attributes, and vice versa.

---

## A Complete Trace

Combining all six layers, here's what a full trace looks like when an agent calls an external tool via MCP:

![Agent + MCP Complete Trace Waterfall](/blogs/2026-05-09-opentelemetry-genai-semantic-conventions/02-trace-waterfall.png)

```
invoke_agent support-router (INTERNAL, trace=t1)
│
├── chat gpt-4o (CLIENT)
│     gen_ai.provider.name = openai
│     gen_ai.request.model = gpt-4o
│     gen_ai.usage.input_tokens = 1523
│     gen_ai.usage.output_tokens = 42
│     gen_ai.response.finish_reasons = ["tool_calls"]
│
├── tools/call query-orders (CLIENT)                  ← MCP client
│     mcp.method.name = tools/call
│     mcp.session.id = sess-abc
│     gen_ai.tool.name = query-orders
│   │
│   └── tools/call query-orders (SERVER)              ← MCP server
│
└── chat gpt-4o (CLIENT)
      gen_ai.usage.input_tokens = 2841
      gen_ai.usage.output_tokens = 256
      gen_ai.usage.cache_read.input_tokens = 1523     ← OpenAI-specific
      gen_ai.response.finish_reasons = ["stop"]

Metrics (same time window):
  gen_ai.client.operation.duration{model=gpt-4o}
  gen_ai.client.token.usage{model=gpt-4o, token_type=input}
  mcp.client.operation.duration{method=tools/call}

Events (opt-in):
  gen_ai.client.inference.operation.details → full prompt/completion
  gen_ai.evaluation.result → score.value=0.92, score.label="relevant"
```

Note: metrics use Prometheus-style shorthand (`{model=...}`). Actual attribute names are `gen_ai.request.model`, `gen_ai.token.type`, etc., exported to Prometheus as `gen_ai_request_model`, `gen_ai_token_type`. Prometheus 3.0 has since added native support for OpenTelemetry naming conventions, so cross-system alignment no longer requires manual conversion.

A single `trace_id` links the entire chain, from the agent's initial decision through the MCP server's execution to the final response.

---

## Current Status and Adoption

The spec is still in Development status. v1.36 is the transition baseline: existing instrumentations default to the old attribute format, while `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` switches to the latest version[^2].

The OpenAI Python SDK instrumentation is the most mature[^5]. Community libraries like OpenLLMetry[^12] cover Anthropic, Cohere, and AWS Bedrock. Framework instrumentations for LangGraph and CrewAI are in progress.

Datadog was one of the first commercial platforms to natively support v1.37+ GenAI semantic conventions[^21]. Elastic's 2026 observability report finds that 85% of organizations use some form of GenAI for observability, and 89% of OTel production users rate vendor compliance as "critical" or "very important"[^14].

We built a full LLM observability stack on this spec in our [GenAI demo](https://greptime.com/blogs/2026-03-10-llm-observability-otel-genai-greptimedb), with OTLP going straight to GreptimeDB and all three signal types in unified storage:

```bash
docker compose --profile load up -d
# Grafana: http://localhost:3000
```

---

## Wrap-Up

The six layers covered above (Client Spans, Agent Spans, MCP conventions, Events, Metrics, and Provider conventions) together span the full observability chain, from the initial model call to agent orchestration to tool execution.

The two pieces worth paying closest attention to today are Agent Spans and MCP conventions. Both are new to OTel, both target concrete problems (black-box agent reasoning and broken MCP traces), and both are likely to set the pattern for how the ecosystem instruments agents going forward. The three-mode design for content capture is the spec's pragmatic answer to a real conflict between privacy and debuggability.

The spec itself is still moving fast. Every release from v1.37 to v1.41 has touched GenAI. For teams building LLM or agent systems, the easiest place to start is the OpenAI Python SDK instrumentation. From there, the [spec docs](https://opentelemetry.io/docs/specs/semconv/gen-ai/) and our [demo](https://github.com/GreptimeTeam/demo-scene/tree/main/genai-observability) show how the layers fit together in practice.

## References

[^1]: [OpenTelemetry GenAI SIG](https://github.com/open-telemetry/community/blob/main/projects/gen-ai.md)
[^2]: [Semantic Conventions for Generative AI Systems](https://opentelemetry.io/docs/specs/semconv/gen-ai/) (docs v1.41.0, Development)
[^3]: [Semantic Conventions 2026 Roadmap](https://github.com/open-telemetry/semantic-conventions/issues/3330)
[^4]: [GenAI Client Spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/)
[^5]: [opentelemetry-instrumentation-openai-v2](https://pypi.org/project/opentelemetry-instrumentation-openai-v2/)
[^6]: [GenAI Agent and Framework Spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/)
[^7]: [Glama: OpenTelemetry for MCP Analytics and Agent Observability](https://glama.ai/blog/2025-11-29-open-telemetry-for-model-context-protocol-mcp-analytics-and-agent-observability)
[^8]: [MCP Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/mcp/)
[^9]: [GenAI Events](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-events/)
[^10]: [GenAI Metrics](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/)
[^11]: [OpenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/openai/)
[^12]: [OpenLLMetry project](https://horovits.medium.com/opentelemetry-for-genai-and-the-openllmetry-project-81b9cea6a771)
[^14]: [Elastic: Observability trends for 2026](https://www.elastic.co/blog/2026-observability-trends-generative-ai-opentelemetry)
[^15]: [Semantic Conventions v1.41.1](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.41.1)
[^16]: [v1.37.0 Release](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.37.0)
[^17]: [v1.38.0 Release](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.38.0)
[^18]: [v1.39.0 Release](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.39.0)
[^19]: [v1.40.0 Release](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.40.0)
[^20]: [v1.41.0 Release](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.41.0)
[^21]: [Datadog: LLM OTel Semantic Convention](https://www.datadoghq.com/blog/llm-otel-semantic-convention/)
