Skip to content

Agent Monitoring

Agent Monitoring associates Agent / LLM requests with the entire application tracing, tracking the complete flow of each conversation and accurately measuring the Token consumption for each generation task.

When using Agent Monitoring services, you can:

  • View the complete trace of a single request: clearly see the entire process from user question reception, processing (e.g., database queries), to calling the LLM model and returning the answer.
  • Analyze performance bottlenecks: accurately measure the time spent on each step (e.g., model invocation, data retrieval) and identify delays promptly.
  • Correlate upstream and downstream services: associate Agent / LLM requests with related application and infrastructure metrics for comprehensive root cause analysis.

Core Capabilities

The core of Agent Monitoring is establishing quantifiable associations between input (Prompt), output (Completion), and system behavior. Its core capabilities are reflected in three dimensions:

1. Full-Trace Tracing

Within the Agent / LLM invocation framework, use Traces and Spans to precisely trace the full request trace and locate latency bottlenecks.

2. Quality Output Evaluation

Internally, a rule engine and AI evaluation automatically optimize output content.

3. Cost Measurement

Automatically collect and associate Token consumption (input/output breakdown), model type, and invocation parameters for each request, providing cost allocation capabilities across multiple business dimensions.

Getting Started

Application List

Enter the Application List to create and manage monitoring applications. You can create a new Agent Monitoring Application or LLM Monitoring Application. Agent Monitoring Applications support Codex, Claude Code, OpenClaw, Hermes, Qoder, WorkBuddy, and AgentScope. Among these, AgentScope is manually integrated via OpenTelemetry, while other types use the obs-agent-connector installer for integration. LLM Monitoring Applications support integration via Langfuse. After completing the parameter configuration in the page guide, you can start collecting data.

The summary page supports customizing the Agent types to display via "Display Items" and adjusting the display order of each type.

Explorer

After data ingestion, you can search and filter data by Session or Trace dimension in the Explorer:

  • Session List: View Session ID, Input/Output Tokens, number of risk events, and alert levels. Drill down to the detail page to view the Trace waterfall chart, Model/Skill/Tool invocation ratio, invocation details, and risk events.
  • Trace List: View Trace ID, associated Session, duration, Tokens, status, number of risky operations, and alert levels. Drill down to view Span details, risky operations, tool invocation records, and Skill invocation records.

Analysis Dashboard

Use the Analysis Dashboard to get an overview of application running status in chart form. It supports filtering by application type. The content displayed in the Analysis Dashboard varies by application type:

  • LLM Monitoring Application: Displays overview metrics such as request count, Span count, request error rate, total Token consumption, average response time, as well as request trend chart, Token consumption trend chart, request count ratio by model, and Token usage ranking by model.

  • Agent Monitoring Application: Displays in four modules: Running Overview, Models and Tokens, Tool Module, and Skill Module.

    • Running Overview: Total operations, total Tool, LLM, and Skill invocations, total Tokens, and distribution of model, Tool, and Skill invocations.
    • Models and Tokens: LLM invocation trend, Input/Output Token trend, Token trend by model, and LLM invocation duration percentiles.
    • Tool Module: Active Tool count, maximum duration, invocation trend, average duration, and Top 10 anomalies.
    • Skill Module: Active Skill count, maximum duration, invocation trend, average duration, and Top 10 anomalies.