Inkdown
Start writing

Study

70 files·12 subfolders

Shared Workspace

Study
AI eng

10_TRACING

Shared from "Study" on Inkdown

Tracing - Comprehensive Deep Dive

Overview

Tracing in the OpenAI Agents SDK provides comprehensive observability into agent runs. Think of tracing as a "flight recorder" or "audit log" that captures everything that happens during an agent execution - from the initial input to the final output, including all model calls, tool executions, handoffs, and more. This is essential for debugging, monitoring, and understanding agent behavior.

Core Concepts

What is Tracing?

Tracing is the systematic recording of events that occur during an agent run. It captures:

  • Timing - When each operation occurred
  • Duration - How long each operation took
  • Inputs/Outputs - What data was passed (unless sensitive)
  • Relationships - How operations relate to each other
  • Metadata - Additional context about the run
Why Tracing Matters
basic-ques
core
Revision w/ Whiteboard
CN Basics - 1
CN Basics - 2
DNS
Event loop
programming-language-concepts.md
zero-language-explanation.md
DB
Quick
databases-deep-dive.md
01-introduction.md
02-relational-databases.md
03-database-design.md
04-indexing.md
05-transactions-acid.md
06-nosql-databases.md
07-query-optimization.md
08-replication-ha.md
09-sharding-partitioning.md
10-caching-strategies.md
11-cap-theorem.md
12-connection-pooling.md
13-backup-recovery.md
14-monitoring.md
15-database-selection.md
README.md
JS
core topics
Event loop
Merlin Backend
01-Orchestration.md
02-DeepResearch.md
03-Search.md
04-Scraping.md
05-Streaming.md
06-MultiProviderLLM.md
07-MemoryAndContext.md
08-ErrorHandling.md
09-RateLimiting.md
10-TaskQueue.md
11-SecurityAndAuth.md
Orchestration-2nd-draft
Mobile
Build Alternative
Bundling
metro-bundler-deep-dive.md
OpenAI Agents Python
00_OVERVIEW.md
01_AGENT_SYSTEM.md
02_RUNNER_SYSTEM.md
03_TOOL_SYSTEM.md
04_ITEMS_SYSTEM.md
05_GUARDRAILS.md
06_HANDOFFS.md
07_MEMORY_SESSIONS.md
08_MODEL_PROVIDERS.md
09_SANDBOX_SYSTEM.md
10_TRACING.md
11_RUN_STATE.md
12_CONTEXT.md
13_LIFECYCLE_HOOKS.md
14_CONFIGURATION.md
15_ERROR_HANDLING.md
16_STREAMING.md
17_EXTENSIONS.md
18_MCP_INTEGRATION.md
19_BEST_PRACTICES.md
20_ARCHITECTURE_PATTERNS.md
opencode-study
context-handling
core
Python
Alembic
Basics
sqlalchemy - fastapi
SQLAlchemy overview
tweets
system_design_for_agentic_apps.md
Agent Loop
  • Debugging - Understand what went wrong in a failed run
  • Performance - Identify bottlenecks and slow operations
  • Observability - Monitor agent behavior in production
  • Analysis - Analyze patterns and optimize workflows
  • Compliance - Maintain audit trails for regulatory requirements
  • Testing - Verify agent behavior matches expectations

    • Trace Structure

    Trace Hierarchy

    Traces are organized hierarchically:

    Plain text

    Analogy: Think of it like a tree where the Trace is the trunk and Spans are branches.

    Trace Class
    Python
    Span Class
    Python

    Span Types

    Agent Span

    Represents an agent's execution:

    Python

    AgentSpanData includes:

    • Agent name
    • Agent instructions
    • Input items
    • Output items
    • Tool calls
    • Handoffs
    Generation Span

    Represents a model generation:

    Python

    GenerationSpanData includes:

    • Model name
    • System instructions
    • Input items
    • Output items
    • Token usage
    • Tool calls
    Function Span

    Represents a function execution:

    Python

    FunctionSpanData includes:

    • Function name
    • Arguments
    • Return value
    • Duration
    Guardrail Span

    Represents a guardrail execution:

    Python

    GuardrailSpanData includes:

    • Guardrail name
    • Guardrail type (input/output/tool)
    • Input data
    • Output data
    • Tripwire triggered
    Handoff Span

    Represents a handoff between agents:

    Python

    HandoffSpanData includes:

    • From agent name
    • To agent name
    • Handoff arguments
    • Handoff result
    Custom Span

    Represents a custom operation:

    Python
    Tool Span

    Represents a tool execution:

    Python
    MCP Tools Span

    Represents MCP tool operations:

    Python

    Tracing Configuration

    RunConfig Tracing Settings

    Configure tracing for a run:

    Python
    Global Tracing Settings

    Set global tracing defaults:

    Python
    Environment Variables

    Configure tracing via environment:

    Bash

    Trace Processors

    TracingProcessor Interface

    Custom trace processors allow you to handle traces:

    Python
    Adding Trace Processors

    Add processors to handle traces:

    Python
    Multiple Processors

    You can have multiple processors:

    Python

    Built-in Processors

    Console Processor

    Print traces to console:

    Python
    File Processor

    Save traces to files:

    Python
    OpenAI Processor

    Send traces to OpenAI:

    Python

    Manual Tracing

    Creating Traces Manually

    Create traces for custom operations:

    Python
    Getting Current Trace

    Access the current trace context:

    Python
    Getting Current Span

    Access the current span context:

    Python

    Span Data Types

    AgentSpanData
    Python
    GenerationSpanData
    Python
    FunctionSpanData
    Python
    GuardrailSpanData
    Python

    Trace Errors

    SpanError

    Errors in spans are captured:

    Python
    Error Handling in Spans

    Errors are automatically captured:

    Python
    Manual Error Recording

    Manually record errors:

    Python

    Trace Export

    OpenAI Trace Export

    Export traces to OpenAI:

    Python
    Custom Export

    Implement custom export:

    Python

    Trace Analysis

    Analyzing Trace Duration

    Calculate total duration:

    Python
    Analyzing Token Usage

    Aggregate token usage across spans:

    Python
    Analyzing Tool Usage

    Count tool executions:

    Python
    Analyzing Handoff Patterns

    Track handoff patterns:

    Python

    Tracing Best Practices

    1. Use Descriptive Span Names

    Use clear, descriptive span names:

    Python
    2. Include Relevant Metadata

    Add helpful metadata:

    Python
    3. Handle Sensitive Data

    Be careful with sensitive data:

    Python
    4. Use Appropriate Granularity

    Choose the right level of detail:

    Python
    5. Clean Up Old Traces

    Implement trace cleanup:

    Python

    Common Tracing Patterns

    1. Distributed Tracing

    Link traces across services:

    Python
    2. Performance Monitoring

    Track performance metrics:

    Python
    3. Error Tracking

    Track errors in traces:

    Python
    4. Compliance Logging

    Maintain compliance logs:

    Python
    5. Cost Tracking

    Estimate costs from traces:

    Python

    Tracing and Streaming

    Streaming Traces

    Traces are built incrementally during streaming:

    Python
    Real-time Span Updates

    Spans are updated as operations complete:

    Python

    Tracing and Sessions

    Session-Aware Tracing

    Traces can include session information:

    Python
    Cross-Session Tracing

    Link traces across sessions:

    Python

    Tracing Security

    Sensitive Data Protection

    Protect sensitive data in traces:

    Python
    Access Control

    Control who can access traces:

    Python

    Tracing Performance

    Performance Impact

    Tracing has minimal performance impact:

    Python
    Optimizing Tracing

    Optimize for performance:

    Python

    Tracing Debugging

    Debugging with Traces

    Use traces to debug issues:

    Python
    Visualizing Traces

    Create visual representations:

    Python

    Summary

    Tracing provides comprehensive observability. Key takeaways:

    1. Traces capture entire agent runs
    2. Spans represent individual operations
    3. Span hierarchy shows operation relationships
    4. Agent spans track agent execution
    5. Generation spans track model calls
    6. Function spans track function execution
    7. Guardrail spans track guardrail execution
    8. Handoff spans track agent delegation
    9. Custom spans track custom operations
    10. Trace processors handle completed traces
    11. Configuration controls tracing behavior
    12. Sensitive data can be excluded
    13. Manual tracing for custom operations
    14. Current trace/span context access
    15. Span data provides operation details
    16. Span errors capture failures
    17. Trace export to external systems
    18. Trace analysis enables insights
    19. Performance monitoring via traces
    20. Compliance via audit trails

    Tracing is essential for building observable, debuggable, and monitorable agent systems.