Observability & Tracing

Enable OpenTelemetry tracing to monitor and debug your agent’s execution with tools like Laminar, MLflow, Honeycomb, or any OTLP-compatible backend.

A full setup example is available here!

Overview

The OpenHands SDK provides built-in OpenTelemetry (OTEL) tracing support, allowing you to monitor and debug your agent’s execution in real-time. You can send traces to any OTLP-compatible observability platform including:

• Laminar - AI-focused observability with browser session replay support
• MLflow - Open-source AI platform with tracing, evaluation, and LLM governance
• Honeycomb - High-performance distributed tracing
• Any OTLP-compatible backend - Including Jaeger, Datadog, New Relic, and more

The SDK automatically traces:

• Agent execution steps
• Tool calls and executions
• LLM API calls (via LiteLLM integration)
• Browser automation sessions (when using browser-use)
• Conversation lifecycle events

Quick Start

Tracing is automatically enabled when you set the appropriate environment variables. The SDK detects the configuration on startup and initializes tracing without requiring code changes.

Using Laminar

Laminar provides specialized AI observability features including browser session replays when using browser-use tools:

# Set your Laminar project API key
export LMNR_PROJECT_API_KEY="your-laminar-api-key"

That’s it! Run your agent code normally and traces will be sent to Laminar automatically.

For self-hosted Laminar deployments, you can also configure custom ports:

export LMNR_PROJECT_API_KEY="your-laminar-api-key"
export LMNR_HTTP_PORT=8000
export LMNR_GRPC_PORT=8001

Using OpenTelemetry (OTLP) Backends

For OpenTelemetry (OTLP) compatible backends, set the following environment variables:

# Required: Set the OTLP endpoint
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://your-otlp-backend/v1/traces"

# Required: Set additional headers required by your backend (format: comma-separated key=value pairs, URL-encoded)
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="key=value,key2=value2"

# Recommended: Explicitly set the protocol (most OTLP backends require HTTP)
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"  # use "grpc" only if your backend supports it

View the platform-specific configuration sections below for which values to use.

• MLflow - Open-source AI platform with tracing, evaluation, and governance
• Honeycomb - High-performance distributed tracing
• Jaeger - Open-source distributed tracing
• Generic OTLP Collector - For other backends

Alternative Configuration Methods

You can also use these alternative environment variable formats:

# Short form for endpoint
export OTEL_ENDPOINT="http://localhost:4317"

# Alternative header format
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20<KEY>"

# Alternative protocol specification
export OTEL_EXPORTER="otlp_http"  # or "otlp_grpc"

How It Works

The OpenHands SDK uses the Laminar SDK as its OpenTelemetry instrumentation layer. When you set the environment variables, the SDK:

1. Detects Configuration: Checks for OTEL environment variables on startup
2. Initializes Tracing: Configures OpenTelemetry with the appropriate exporter
3. Instruments Code: Automatically wraps key functions with tracing decorators
4. Captures Context: Associates traces with conversation IDs for session grouping
5. Exports Spans: Sends trace data to your configured backend

What Gets Traced

The SDK automatically instruments these components:

• agent.step - Each iteration of the agent’s execution loop
• Tool Executions - Individual tool calls with input/output capture
• LLM Calls - API requests to language models via LiteLLM
• Conversation Lifecycle - Message sending, conversation runs, and title generation
• Browser Sessions - When using browser-use, captures session replays (Laminar only)

Trace Hierarchy

Traces are organized hierarchically:

conversation

conversation.run

agent.step

llm.completion

tool.execute

agent.step

llm.completion

Each conversation gets its own session ID (the conversation UUID), allowing you to group all traces from a single conversation together in your observability platform.

Note that in tool.execute the tool calls are traced, e.g., bash, file_editor.

Configuration Reference

Environment Variables

The SDK checks for these environment variables (in order of precedence):

Variable	Description	Example
LMNR_PROJECT_API_KEY	Laminar project API key	your-laminar-api-key
LMNR_HTTP_PORT	HTTP port for self-hosted Laminar	8000
LMNR_GRPC_PORT	gRPC port for self-hosted Laminar	8001
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT	Full OTLP traces endpoint URL	https://api.honeycomb.io:443/v1/traces
OTEL_EXPORTER_OTLP_ENDPOINT	Base OTLP endpoint (traces path appended)	http://localhost:4317
OTEL_ENDPOINT	Short form endpoint	http://localhost:4317
OTEL_EXPORTER_OTLP_TRACES_HEADERS	Authentication headers for traces	x-honeycomb-team=YOUR_API_KEY
OTEL_EXPORTER_OTLP_HEADERS	General authentication headers	Authorization=Bearer%20TOKEN
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL	Protocol for traces endpoint	http/protobuf, grpc
OTEL_EXPORTER	Short form protocol	otlp_http, otlp_grpc

Header Format

Headers should be comma-separated key=value pairs with URL encoding for special characters:

# Single header
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="x-honeycomb-team=abc123"

# Multiple headers
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer%20abc123,X-Custom-Header=value"

Protocol Options

The SDK supports both HTTP and gRPC protocols:

• http/protobuf or otlp_http - HTTP with protobuf encoding (recommended for most backends)
• grpc or otlp_grpc - gRPC with protobuf encoding (use only if your backend supports gRPC)

Platform-Specific Configuration

Laminar Setup

1. Sign up at laminar.sh
2. Create a project and copy your API key
3. Set the environment variable:

export LMNR_PROJECT_API_KEY="your-laminar-api-key"

Self-Hosted Laminar: If you are running a self-hosted Laminar instance, you can configure the HTTP and gRPC ports via environment variables:

export LMNR_PROJECT_API_KEY="your-laminar-api-key"
export LMNR_HTTP_PORT=8000
export LMNR_GRPC_PORT=8001

Browser Session Replay: When using Laminar with browser-use tools, session replays are automatically captured, allowing you to see exactly what the browser automation did.

MLflow Setup

MLflow is an open-source AI platform that accepts OpenTelemetry traces out of the box, alongside evaluation and LLM governance capabilities.

1. Start your MLflow tracking server:

uvx mlflow server

For other deployment options (pip, Docker Compose, etc.), see Set Up MLflow Server.

2. Configure the environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:5000"
export OTEL_EXPORTER_OTLP_HEADERS="x-mlflow-experiment-id=123"  # Replace "123" with your MLflow experiment ID
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"

Navigate to the MLflow UI (e.g., http://localhost:5000), select the experiment, and open the Traces tab to view the recorded traces.

Honeycomb Setup

1. Sign up at honeycomb.io
2. Get your API key from the account settings
3. Configure the environment:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://api.honeycomb.io:443/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="x-honeycomb-team=YOUR_API_KEY"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"

Jaeger Setup

For local development with Jaeger:

# Start Jaeger all-in-one container
docker run -d --name jaeger \
  -p 4317:4317 \
  -p 16686:16686 \
  jaegertracing/all-in-one:latest

# Configure SDK
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="grpc"

Access the Jaeger UI at http://localhost:16686

Generic OTLP Collector

For other backends, use their OTLP endpoint:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://your-otlp-collector:4317/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer%20YOUR_TOKEN"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"

Advanced Usage

Disabling Observability

To disable tracing, simply unset all OTEL environment variables:

unset LMNR_PROJECT_API_KEY
unset OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
unset OTEL_EXPORTER_OTLP_ENDPOINT
unset OTEL_ENDPOINT

The SDK will automatically skip all tracing instrumentation with minimal overhead.

Custom Span Attributes

The SDK automatically adds these attributes to spans:

• conversation_id - UUID of the conversation
• tool_name - Name of the tool being executed
• action.kind - Type of action being performed
• session_id - Groups all traces from one conversation

Debugging Tracing Issues

If traces aren’t appearing in your observability platform:

1. Verify Environment Variables:

   import os
   
   otel_endpoint = os.getenv('OTEL_EXPORTER_OTLP_TRACES_ENDPOINT')
   otel_headers = os.getenv('OTEL_EXPORTER_OTLP_TRACES_HEADERS')
   
   print(f"OTEL Endpoint: {otel_endpoint}")
   print(f"OTEL Headers: {otel_headers}")
   

2. Check SDK Logs: The SDK logs observability initialization at debug level:

   import logging
   
   logging.basicConfig(level=logging.DEBUG)
   

3. Test Connectivity: Ensure your application can reach the OTLP endpoint:

   curl -v https://api.honeycomb.io:443/v1/traces
   

4. Validate Headers: Check that authentication headers are properly URL-encoded

Troubleshooting

Traces Not Appearing

Problem: No traces showing up in observability platform

Solutions:

• Verify environment variables are set correctly
• Check network connectivity to OTLP endpoint
• Ensure authentication headers are valid
• Look for SDK initialization logs at debug level

High Trace Volume

Problem: Too many spans being generated

Solutions:

• Configure sampling at the collector level
• For Laminar with non-browser tools, browser instrumentation is automatically disabled
• Use backend-specific filtering rules

Performance Impact

Problem: Concerned about tracing overhead

Solutions:

• Tracing has minimal overhead when properly configured
• Disable tracing in development by unsetting environment variables
• Use asynchronous exporters (default in most OTLP configurations)

Example: Full Setup

This example is available on GitHub: examples/01_standalone_sdk/27_observability_laminar.py

examples/01_standalone_sdk/27_observability_laminar.py

"""
Observability & Laminar example

This example demonstrates enabling OpenTelemetry tracing with Laminar in the
OpenHands SDK. Set LMNR_PROJECT_API_KEY and run the script to see traces.
"""

import os

from pydantic import SecretStr

from openhands.sdk import LLM, Agent, Conversation, Tool
from openhands.tools.terminal import TerminalTool


# Tip: Set LMNR_PROJECT_API_KEY in your environment before running, e.g.:
#   export LMNR_PROJECT_API_KEY="your-laminar-api-key"
# For non-Laminar OTLP backends, set OTEL_* variables instead.

# Configure LLM and Agent
api_key = os.getenv("LLM_API_KEY")
model = os.getenv("LLM_MODEL", "openhands/claude-sonnet-4-5-20250929")
base_url = os.getenv("LLM_BASE_URL")
llm = LLM(
    model=model,
    api_key=SecretStr(api_key) if api_key else None,
    base_url=base_url,
    usage_id="agent",
)

agent = Agent(
    llm=llm,
    tools=[Tool(name=TerminalTool.name)],
)

# Create conversation and run a simple task
conversation = Conversation(agent=agent, workspace=".")
conversation.send_message("List the files in the current directory and print them.")
conversation.run()
print(
    "All done! Check your Laminar dashboard for traces "
    "(session is the conversation UUID)."
)

Running the Example

export LMNR_PROJECT_API_KEY="your-laminar-api-key"
cd software-agent-sdk
uv run python examples/01_standalone_sdk/27_observability_laminar.py

Next Steps

• Metrics Tracking - Monitor token usage and costs alongside traces
• LLM Registry - Track multiple LLMs used in your application
• Security - Add security validation to your traced agent executions