Trace Details¶

In the Trace Explorer, clicking on any trace will display the trace details, including relative time, duration, HTTP method, URL, status code, TraceId, flame graph, Span list, service call relationships, and associated logs, hosts, metrics, and network information. For frontend applications (e.g., browsers), you can also view the request time distribution, including queue, first packet, and download time percentages.

Note

The RUM SDK must be version 2.2.10 or higher to see this data display. If cross-domain issues exist, header configuration adjustments are required.

For more details, refer to Web Application Integration.

The flame graph/Span list/waterfall chart can display up to 10,000 Spans. You can view unshown Spans by setting the offset:

Flame Graph¶

The flame graph clearly shows the flow and execution time of each Span. The right side displays the service list and response time. Clicking on a Span allows you to view its JSON content in the trace details. Use the mouse wheel to zoom in for specific information.

For more details, refer to Using Flame Graphs to Analyze Trace Performance.

Flame Graph Trace ExplanationExecution Time Percentage Calculation ExplanationAsynchronous Calls

As shown in the flame graph above, this trace involves two services: CloudCare and Mysql. The trace starts with a POST request to the CloudCare service, followed by executing ActionResource.executeAction, and finally executing Mysql statements. During the execution of ActionResource.executeAction, multiple Mysql statements are executed. The execution times for CloudCare and Mysql are calculated as follows:

CloudCare service execution time = D1+D2+D3+D4+D5+D6+D7+D8+D9+D10+D11
Mysql service execution time = span2+span3+......+span11

For specific statements and execution time details, refer to the Span list.

The service execution time percentage in the flame graph refers to the ratio of each service's execution time to the total trace duration. In the example below, this trace involves two services: CloudCare and Mysql, with execution time percentages of 42.37% and 57.63%, respectively. The CloudCare service has 2 Spans, and the Mysql service has 10 Spans.

Mysql service execution time percentage calculation: sum of all Span execution times / total trace duration.

Calculation explanation: The Mysql service in the image below has 10 Spans. Clicking on each Span shows its execution time. For example, the Span execution time is 5.08ms. Similarly, obtain the execution times for the remaining 9 Spans and sum them.

CloudCare service execution time percentage calculation: (total trace duration - Mysql service execution time) / total trace duration

Calculation explanation: The CloudCare service in the image below spans the entire trace. The execution time for CloudCare is the remaining time after subtracting the Mysql service execution time (see the red line in the image). The execution time percentage can also be viewed directly in the Span list, which shows each Span's execution time and percentage.

In the flame graph, whether the service calls are synchronous or asynchronous, you can clearly trace the performance data details of each trace. For example, the flame graph can clearly show which requests are asynchronous, their start time, end time, and total duration.

Span List¶

Displays all Spans in the trace, including resource name, count, duration, execution time, and percentage.
You can search by resource name or Span ID.
Clicking on any Span allows you to view its JSON content in the trace details and synchronize it with the flame graph.
If there are errors, error prompts will be displayed.
Click Error Spans to directly display the filtered results.

Waterfall Chart¶

You can view the parent-child relationships between resources.

The waterfall chart displays Span data in chronological order. The left list shows Span data, displaying each resource's execution time percentage. The right side shows the waterfall chart in chronological order.

Search by resource name or Span ID;
Collapse or expand resources on the left with one click;
Click to switch the execution time format;
Click Error Spans to directly display the filtered results.

Service Call Relationships¶

Displays the call relationships and call counts between services. You can search and filter by service, resource, or Span ID.

The system displays service colors based on the error results from the trace details > Flame Graph. Red indicates the presence of errors.

If you have configured service binding relationships in the user view, such as service:mysql, clicking on the service card allows you to quickly view the associated view. Click to jump to the details page.

Quick Actions¶

Action	Description
Full Screen/Restore Default Size	You can click the full-screen icon in the top-right corner of the trace details to horizontally expand the flame graph; click the restore default size icon to return to the details page.
Locate Current Span	Click to zoom in and locate the current Span
Expand/Collapse Mini Map	You can click the expand/collapse mini map icon on the left side of the trace details to quickly view the flame graph by selecting intervals, dragging, or scrolling on the mini map.
View Global Trace	You can click the view global trace icon on the left side of the trace details to view the global trace in the flame graph.
Double-Click Span	The Span is magnified in the middle of the flame graph, allowing you to quickly locate and view its context-related Spans.
Click Service Name	The corresponding Span is highlighted. Clicking the service name again restores the default selection of all Spans. You can quickly filter and view the Spans corresponding to the service by clicking the service name.

Extended Attributes¶

In the search bar, you can quickly search and locate by entering field names or values;

After checking the field alias, you can view it after the field name; you can choose as needed.

In the trace details page, you can view the relevant field attributes of the current trace in Extended Attributes:

Field	Attribute
Filter Field Value	Add this field to the explorer to view all data related to this field. You can filter and view the trace list related to this field in the trace explorer. See Figure 1.
Reverse Filter Field Value	Add this field to the explorer to view all data except this field.
Add to Display Column	Add this field to the explorer list for viewing.
Copy	Copy this field to the clipboard.

Some fields do not support filtering, aggregation, and related logic

TrueWatch has some fields that use full-text indexing. These fields do not support filtering, aggregation, and related logic. The full-text indexed fields are as follows:

Category	Field
Objects, Resource Catalog	`message`
Logs, Backup Logs	`message`
Security	`message` / `title`
Network	`message`
Traces	`error_message` / `error_stack`
Events	`message` / `title` / `df_message` / `df_title`
RUM Errors	`error_message` / `error_stack`
RUM Long Tasks	`long_task_message` / `long_task_stack`

Figure 1

Error Details¶

In the trace details page, if there are error traces, you can view the related error details.

For more error trace analysis, refer to Error Tracking.

Service Context¶

By obtaining the object classification under the infrastructure resource catalog and selecting the latest object by create_time, you can quickly view the current service's operational information, service dependencies, and integration information.

Correlation Analysis¶

Correlated LogsCode HotspotsCorrelated HostsCorrelated ContainersCorrelated PodsCorrelated Network

You can view logs correlated with the current trace (correlated field: trace_id) via Logs. You can customize the display columns. To view more detailed log content, click on the log content to jump to the log details page, or click the jump button to open it in the logs page.

If you have administrator or higher permissions, you can customize the correlated fields. Click the settings button next to the correlated field, select the fields to correlate in the pop-up dialog, and confirm to complete the configuration. Manual input and drag-and-drop ordering are supported.

Note: Custom fields for correlated logs and service list correlation analysis configurations affect each other. If custom fields are configured in the service list, they will be displayed here as well.

When the application uses the ddtrace collector with both APM trace and Profile performance trace data collection enabled, the system provides Span-level correlation to view code hotspots, including execution time, methods, and execution time percentage.

Click View Profile Details to jump to the Profile details page and view more correlated code.

In the trace details page, you can view related host metric views and attribute views (correlated field: host) via Hosts.

Metric View: View the performance metrics status of related hosts from 30 minutes before the trace ends to 30 minutes after, including CPU, memory, and other performance metric views.

Attribute View: Helps you trace the real state of the host object when the trace was generated. It supports viewing the latest object data generated by the related host within the corresponding time, including basic host information and integration status. If cloud host collection is enabled, you can also view cloud vendor information.

Note: TrueWatch saves host object history data for the last 48 hours by default. If no host history data corresponding to the trace time is found, you will not be able to view the attribute view of the correlated host.

In the trace details page, you can view related container metric views and attribute views (correlated field: container_name) via Containers.

Metric View: Supports viewing the performance metrics status of related containers from 30 minutes before the trace ends to 30 minutes after, including CPU, memory, and other performance metric views.
Attribute View: Helps you trace the real state of the container object when the trace was generated. It supports viewing the latest object data generated by the related container within the corresponding time, including basic container information and attribute information.

In the trace details page, you can view related Pod attribute views and metric views (correlated field: pod_name) via Pods.

Metric View: Supports viewing the performance metrics status of related container Pods from 30 minutes before the trace ends to 30 minutes after, including CPU, memory, and other performance metric views.
Attribute View: Helps you trace the real state of the container Pod object when the trace was generated. It supports viewing the latest object data generated by the related container Pod within the corresponding time, including basic container information and attribute information.

TrueWatch supports viewing network topology and overview data from multiple dimensions including Host, Pod, Deployment, and Service in Network.

Matching Fields:

To view related networks in the details page, you need to match the corresponding correlated fields, i.e., configure the corresponding field tags during data collection. Otherwise, you will not be able to match and view the related network views in the details page.

Host: Match field host.
Pod:

Matching Field Priority
namespace, pod_name
namespace, pod
pod_name
pod

Deployment:

Matching Field Priority
namespace, deployment_name
namespace, deployment
deployment_name
deployment

Service:

Matching Field Priority
namespace, service_name
namespace, service

Note:

If matching fields for Host, Pod, Deployment, and Service are found simultaneously, the network data is displayed in this order when entering the details page;
If no matching fields are found, it is displayed in gray at the end, with a prompt No network view matched.