Skip to content

Agent Monitoring

Agent Monitoring correlates Agent/LLM requests with the entire application request chain, tracing the complete flow of every conversation and precisely measuring the number of Tokens consumed by each generation task.

When using Agent Monitoring, you can:

  • View the complete request flow of a single request: clearly see the entire process from receiving a user question, processing it (such as database queries), to invoking the LLM model and returning the answer
  • Analyze performance bottlenecks: precisely measure the duration of each step (such as model invocation and data retrieval) to identify latency promptly
  • Correlate upstream and downstream services: associate Agent/LLM requests with related application and infrastructure metrics for comprehensive root cause analysis

Core Capabilities

The most essential part of Agent Monitoring is establishing a quantifiable correlation between inputs (Prompts), outputs (Completions), and system behavior. Its core capabilities are reflected in three dimensions:

1. End-to-End Tracing

In Agent/LLM invocation frameworks, use Traces and Spans to precisely trace the entire request path and locate latency bottlenecks.

2. Output Quality Evaluation

Use Prompt Eval and Schema Eval to evaluate model outputs online or in batch, and view scores and rule-matching results. Prompt Eval supports describing evaluation objectives to generate configuration and selecting the context to pass in via custom evaluation input. Outputs can be reviewed through manual annotation, and samples can be consolidated into a dataset for subsequent batch evaluation.

3. Cost Metering

Automatically collect and correlate the Token consumption (broken down by input/output), model type, and invocation parameters of each request, providing cost allocation capabilities based on multiple business dimensions.

Getting Started

Application List

Go to Application List to create and manage monitoring applications. Agent Monitoring supports two integration types: Agent and LLM. The Agent type is further divided into Agent Applications and Agent Frameworks. The integration methods for each type are as follows:

  • Agent Applications: Supports Codex, Claude Code, OpenClaw, Hermes, Qoder, WorkBuddy, OpenCode, CodeBuddy, Cursor, DeepSeek Harness, Opm, mimo, and pi. For the integration method of each type, see Create an Agent Monitoring Application;
  • Agent Frameworks: Supports AgentScope, Spring AI Alibaba, and Deep Agents, integrated manually via OpenTelemetry;
  • LLM Monitoring Applications: Support integration via Langfuse.

After completing the parameter configuration in the guided setup, you can start collecting data.

The Summary page allows you to customize the Agent types to display via Display Items and adjust the display order of each type.

Explorer

Once data is ingested, you can search and filter data by Session or Trace in the Explorer:

  • User list: Aggregates User usage by user_id; view Agent type, instance, Session, request volume, Token or credit consumption, and last active time;
  • Session list: View Session ID, input/output Tokens, risk event count, and alert level; drill down to the details page to view the Trace waterfall chart, model/Skill/Tool call proportions, call details, and risk events;
  • Trace list: View Trace ID, associated Session, duration, Tokens, status, risky operation count, and alert level; drill down to view Span details, risky operations, tool call records, and Skill call records.

Analysis Dashboard

Use the Analysis Dashboard to get an overview of application health in charts. You can filter by application type, and the Analysis Dashboard displays different content for different application types:

  • LLM Monitoring Applications: Displays overview metrics such as request count, Span count, request error rate, total Token consumption, and average response time, as well as request trend charts, Token consumption trend charts, request share by model, and Token usage ranking by model.

  • Agent Monitoring Applications: Displays content across four modules: Runtime Overview, Models & Tokens, Tool Module, and Skill Module.

    • Runtime Overview: Total operation count, total Tool, LLM, and Skill calls, total Token count, and the distribution of model, Tool, and Skill calls
    • Models & Tokens: LLM call trends, input/output Token trends, Token trends by model, LLM call duration percentiles, and average time to first Token (TTFT)
    • Tool Module: Number of active Tools, maximum duration, call trends, average duration, and Top 10 anomalies
    • Skill Module: Number of active Skills, maximum duration, call trends, average duration, and Top 10 anomalies