Trace Details¶

In the Trace Explorer, click any trace to view its details, including relative time, duration, HTTP method, URL, status code, Trace ID, flame graph, span list, service call relationships, and associated logs, hosts, metrics, and network information. For frontend applications (e.g., browsers), you can also view the request time distribution, including queue, first byte, and download time percentages.

When you open trace details, the system first displays the waterfall chart or flame graph based on the Default Trace Detail View setting in Preferences.

Note

The RUM SDK must be 2.2.10 or later to display this data. If cross-origin scenarios exist, adjust the header configuration.

For more details, refer to Web Application Integration.

The flame graph, span list, and waterfall chart display up to 10,000 spans. Use the Offset setting to view additional spans.

Flame Graph¶

The flame graph clearly shows the flow and execution time of each span. On the right, you can view the service list and response time. Click a span to view its JSON content in the trace details. Use the mouse wheel to zoom in and out for detailed information.

For more details, refer to Using Flame Graphs to Analyze Trace Performance.

Flame Graph Trace ExplanationExecution Time Percentage CalculationAsynchronous Calls

The flame graph above shows that this trace involves two services: CloudCare and MySQL. The trace starts with a POST request to the CloudCare service, then executes ActionResource.executeAction, and finally executes MySQL statements. During the execution of ActionResource.executeAction, MySQL statements are executed multiple times. The execution time calculations for CloudCare and MySQL are as follows:

CloudCare service execution time = D1 + D2 + D3 + D4 + D5 + D6 + D7 + D8 + D9 + D10 + D11
MySQL service execution time = span2 + span3 + ... + span11

For specific executed statements and execution times, refer to the span list.

The service execution time percentage in the flame graph indicates the proportion of each service's time in the total trace time. For example, the trace below involves two services: CloudCare and MySQL, with execution time percentages of 42.37% and 57.63% respectively. CloudCare has 2 spans, and MySQL has 10 spans.

MySQL service execution time percentage calculation: sum of all span execution times / total trace duration.

Calculation explanation: In the figure below, the MySQL service has 10 spans. Click each span to get its execution time. For example, the span shown has an execution time of 5.08 ms. Repeat this for the remaining 9 spans and sum them.

CloudCare service execution time percentage calculation: (total trace duration - MySQL service execution time) / total trace duration.

Calculation explanation: In the figure below, the CloudCare service spans the entire trace. The remaining time after subtracting the MySQL service execution time is the CloudCare service execution time (see the time marked in red). You can also view each span's execution time and percentage directly in the span list.

In the flame graph, regardless of whether services are synchronous or asynchronous, you can clearly trace the performance details of each call. For example, the flame graph shows which requests are asynchronous, their start time, end time, and total elapsed time.

Span List¶

Displays the resource name, count, duration, execution time, and percentage of all spans in this trace.
You can search by resource name or span ID.
Click any span to view its JSON content in the trace details, with synchronized display in the flame graph.
If there are errors, error indicators are shown.
Click Error Spans to directly display the filtered results.

Waterfall Chart¶

View the parent-child relationships between resources.

The waterfall chart displays span data in chronological order. The left panel shows span data with the execution time percentage of each resource. The right panel shows the waterfall chart by time.

Enter a resource name or span ID to search.
Collapse or expand resources with one click.
Click to switch the execution time format.
Click Error Spans to directly display the filtered results.

Service Call Relationships¶

Provides an overall view of call relationships between different services, including call counts, execution time, average latency, and call count for individual services.

Click any service node to view its details:

Service Operations and Resources: Lists the specific operations (e.g., API endpoints, methods) and related resources within the service.
Error Status Tracking:
- If the service has errors, the error details are displayed directly.
- Click to view the parent span of the service, allowing you to trace the upstream call source.

Note

The system determines the color of services in this section based on the error result of the corresponding service in the flame graph in the trace details.

Uninstrumented HTTP External Services¶

When a trace passes through an uninstrumented HTTP external service, the system identifies the service based on upstream and downstream span information and generates a virtual Gateway node in the service call relationships.

The call relationship may display as:

When a single caller can be identified: Caller Service → Gateway → Downstream Service.
When a single caller cannot be identified: Root → Gateway → Downstream Service.

The Gateway node is generated only when all of the following conditions are met:

The downstream is an HTTP Server span.
The parent_id of the Server span is valid and not 0.
No corresponding parent span is found after supplementary queries.
The span contains a recognizable external service address.

A Gateway node is not generated when only an HTTP Client span exists.

The virtual Gateway has no independent span, so call count, latency, and error count are not calculated. Related metrics show -, and the node uses a neutral status display.

Warning

The virtual Gateway is displayed only in the Service Call Relationships section of the trace details, not in the waterfall chart or flame graph.

The Root span is separately queried using trace_id and parent_id=0. It may take up to approximately 6 hours for the data to be fully identified. For traces with more than 10,000 spans, the parent span may not be included in the query results, potentially causing misidentification.

Quick Actions¶

Action	Description
Fullscreen View / Restore Default Size	Click the fullscreen icon in the upper right corner of the trace details to expand the flame graph horizontally. Click the restore icon to return to the detail page.
Locate Current Span	Click to zoom in and locate the current span.
Expand/Collapse Minimap	Click the expand/collapse minimap icon on the left side of the trace details to quickly navigate the flame graph by selecting ranges, dragging, or scrolling on the minimap.
View Global Trace	Click the view global trace icon on the left side of the trace details to view the entire trace in the flame graph.
Double-click Span	Enlarges the span in the center of the flame graph for quick positioning and viewing of its context spans.
Click Service Name	Highlights the corresponding spans. Click again to restore the default selection of all spans. You can quickly filter and view spans belonging to a specific service.

Extended Attributes¶

In the search bar, enter a field name or value to quickly locate data.
After selecting a field alias, it is displayed after the field name. You can choose as needed.
On the trace details page, you can view the related field attributes of the current trace in Extended Attributes:

Field	Attribute
Filter Field Value	Adds the field to the explorer to view all data related to this field. You can filter the trace list in the Trace Explorer. See Figure 1
Inverse Filter Field Value	Adds the field to the explorer to view all data except this field.
Add to Display Columns	Adds the field to the explorer list for display.
Copy	Copies the field to the clipboard.

Some fields do not support filter and aggregation logic

TrueWatch uses full-text indexing for some fields. These fields do not support filter and aggregation logic. The full-text index fields are as follows:

Category	Field
Objects, Resource Catalog	`message`
Logs, Backup Logs	`message`
Security	`message` / `title`
Network	`message`
Trace	`error_message` / `error_stack`
Events	`message` / `title` / `df_message` / `df_title`
RUM Errors	`error_message` / `error_stack`
RUM Long Tasks	`long_task_message` / `long_task_stack`

Figure 1

Error Details¶

On the trace details page, if the current trace contains errors, you can select the error span filter option in the sidebar. The system automatically highlights all error spans. Click any highlighted span to view its associated error stack, logs, and other details.

For more details, refer to Error Tracking.

Service Context¶

By retrieving the object classification from the infrastructure resource catalog and selecting the latest object based on create_time, you can quickly view the current service's runtime information, service dependencies, and integration information.

Correlated Analysis¶

Correlated LogsCode HotspotsCorrelated HostsCorrelated ContainersCorrelated PodsCorrelated Network

You can view log data associated with all services in the entire trace via Logs (correlation field: trace_id). You can customize the display columns. To view more detailed log content, click the log content to jump to the log details page, or click the jump button to open the log page.

The system matches the log index mapping based on the current trace's service, env, version, etc., and automatically selects the corresponding log index. The original index "Match Service" configuration continues to map based on the service field.

If you have administrator or higher permissions, you can customize the correlation fields. Click the settings button next to the correlation field, select the fields you want to correlate in the dialog box, and manually enter, drag to reorder, etc. Confirm to complete the configuration.

Note: Custom fields for correlated logs and custom fields for service list correlated analysis affect each other. If you configure custom fields in the service list, they will be displayed synchronously here.

When the application uses the ddtrace collector and has both APM distributed tracing and Profile performance tracing data collection enabled, the system provides span-level correlated code hotspots, including execution time, methods, and execution time percentage.

Click View Profile Details to jump to the Profile details page to view more correlated code.

On the trace details page, you can view the metric view and attribute view of the related host via Hosts (correlation field: host).

Metric View: View the performance metric status of the related host from 30 minutes before the trace ends to 30 minutes after the trace ends, including CPU, memory, and other performance metrics.

Attribute View: Helps you trace back the actual state of the host object when the trace was generated. Supports viewing the latest object data generated within the corresponding time period for the related host, including basic host information and integration runtime status. If cloud host collection is enabled, you can also view cloud provider information.

Note: TrueWatch retains host object historical data for the last 48 hours by default. If no host historical data corresponding to the current trace time is found, you will not be able to view the attribute view of the related host.

On the trace details page, you can view the metric view and attribute view of the related container via Containers (correlation field: container_name).

Metric View: Supports viewing the performance metric status of the related container from 30 minutes before the trace ends to 30 minutes after the trace ends, including container CPU, memory, and other performance metrics.
Attribute View: Helps you trace back the actual state of the container object when the trace was generated. Supports viewing the latest object data generated within the corresponding time period for the related container, including basic container information and attribute details.

On the trace details page, you can view the attribute view and metric view of the related Pod via Pod (correlation field: pod_name).

Metric View: Supports viewing the performance metric status of the related container Pod from 30 minutes before the trace ends to 30 minutes after the trace ends, including container CPU, memory, and other performance metrics.
Attribute View: Helps you trace back the actual state of the container Pod object when the trace was generated. Supports viewing the latest object data generated within the corresponding time period for the related container Pod, including basic container information and attribute details.

TrueWatch supports viewing network topology and summary data across multiple dimensions including Host, Pod, Deployment, and Service via Network.

Match Fields:

To view the related network in the details page, you need to configure the corresponding correlation fields. The corresponding field tags must be configured during data collection; otherwise, the correlated network view cannot be matched and displayed on the details page.

Host: Match field host.
Pod:

Match Field Priority
namespace, pod_name
namespace, pod
pod_name
pod

Deployment:

Match Field Priority
namespace, deployment_name
namespace, deployment
deployment_name
deployment

Service:

Match Field Priority
namespace, service_name
namespace, service

Note:

If the correlation fields for Host, Pod, Deployment, and Service are all found simultaneously, the network data is displayed in this order when entering the details page.
If a correlation field is not found, it is displayed in gray at the end. Clicking it prompts No matching network view found.