Trace Details¶
In the Trace Explorer, clicking any trace displays its details, including relative time, duration, HTTP method, URL, status code, Trace ID, flame graph, Span list, service call relationships, as well as associated logs, hosts, metrics, and network information. For frontend applications (such as browsers), you can also view the request time distribution, including queue, first packet, and download time proportions.
Note
- The RUM SDK must be version 2.2.10 or above to see this data. If there are cross-domain situations, you need to adjust the header configuration.
For more details, refer to Web Application Integration.
- The flame graph/Span list/waterfall graph displays a maximum of 10,000 Spans. You can use the Offset setting to view Spans not displayed:
Flame Graph¶
The flame graph clearly shows the flow and execution time of each Span. On the right side, you can view the service list and response time. Clicking a Span allows you to view its JSON content in the trace details, and you can zoom in/out with the mouse wheel to view specific information.
For more details, refer to Using Flame Graphs to Analyze Trace Performance.
From the flame graph above, it can be seen that this trace includes two services: CloudCare and Mysql. The trace starts with a POST request to the CloudCare service, then executes ActionResource.executeAction, and finally executes Mysql statements. During the entire execution of ActionResource.executeAction, Mysql statements are executed multiple times. The execution time calculation methods for CloudCare and Mysql are:
- CloudCare service execution time = D1+D2+D3+D4+D5+D6+D7+D8+D9+D10+D11
- Mysql service execution time = span2+span3+......+span11
For details on specific executed statements and execution times, refer to the Span list.
The service execution time proportion in the flame graph refers to the ratio of each service's time to the total time of the trace. Taking the following figure as an example, this trace includes two services: CloudCare and Mysql, with execution time proportions of 42.37% and 57.63% respectively. The CloudCare service has 2 Spans, and the Mysql service has 10 Spans.
- Mysql service execution time proportion calculation method: Sum of execution times of all Spans / Total duration of the current trace.
Calculation method description: In the figure below, the Mysql service has a total of 10 Spans. You can click each Span to get its execution time. From the figure, the execution time of this Span is 5.08ms. Then, obtain the execution times of the remaining 9 Spans in the same way and sum them up.
- CloudCare service execution time proportion calculation method: (Total duration of the current trace - Mysql service execution time) / Total duration of the current trace
Calculation method description: In the figure below, the CloudCare service runs through the entire current trace. The remaining time after subtracting the Mysql service execution time is the CloudCare service execution time (see the red line part). The execution time proportion can also be directly viewed through the Span list for each Span's execution time and proportion.
In the flame graph, whether services are synchronous or asynchronous, every detail of each trace's performance data can be clearly tracked. For example, the flame graph allows you to clearly see which requests are asynchronous, their start times, end times, and total time spent.
Span List¶
- Displays the resource name, count, duration, execution time, and proportion of all Spans in the trace.
- You can enter a resource name or Span ID to search and match.
- Clicking any Span allows you to view its JSON content in the trace details, and the flame graph will synchronously display.
- If there are errors, error prompts will be displayed.
- Click Error Spans to directly display the filtered results.
Waterfall Graph¶
You can view the parent-child relationships between various resources.
The waterfall graph displays Span data in order of start time. The left list shows Span data, displaying the execution time proportion of each resource. The right side shows the waterfall graph in chronological order.
- Enter the resource name or Span ID corresponding to the Span to search and match.
- Collapse or expand the left resources with one click.
- Click to switch the execution time format.
- Click Error Spans to directly display the filtered results.
Service Call Relationships¶
Displays the overall call relationships and call counts between different services, as well as the execution time, average duration, and call count of individual services.
Click any service node to view its detailed information:
-
Service Operations and Resources: Lists the specific operations (such as API endpoints, methods, etc.) and related resources included in the service.
-
Error Status Tracking:
-
If the service has errors, error details will be directly displayed.
-
Click to view the parent Span of the service to trace the upstream call source.
-
Note
The system determines the color of the service display here based on the error result corresponding to the service counted in the trace details > Flame Graph.
Shortcut Operations¶
Operation |
Description |
|---|---|
| Fullscreen View / Restore Default Size | You can click the fullscreen view icon in the upper right corner of the trace details to expand the flame graph horizontally; click the restore default size icon to restore the details page. |
| Locate Current Span | Click to zoom in and locate the current Span. |
| Expand/Collapse Mini Map | You can click the expand/collapse mini map icon on the left side of the trace details to quickly view the flame graph by selecting intervals, dragging, or scrolling on the mini map. |
| View Global Trace | You can click the view global trace icon on the left side of the trace details to view the global trace in the flame graph. |
| Double-click Span | Zooms in to display the Span in the middle of the flame graph, allowing you to quickly locate and view its context-related Spans. |
| Click Service Name | Highlights the corresponding Span. Clicking the service name again restores the default selection of all Spans. You can quickly filter and view the corresponding Spans for a service by clicking the service name. |
Extended Attributes¶
- In the search bar, you can enter a field name or value to quickly search and locate.
- After checking the field alias, you can view it after the field name. You can choose as needed.
- On the trace details page, you can view the relevant field attributes of the current trace in Extended Attributes:
Field |
Attribute |
|---|---|
| Filter Field Value | Adds the field to the explorer to view all data related to the field. You can filter and view the trace list of related fields in the Trace Explorer. Refer to Figure 1 |
| Reverse Filter Field Value | Adds the field to the explorer to view data other than this field. |
| Add to Display Column | Adds the field to the explorer list for viewing. |
| Copy | Copies the field to the clipboard. |
Some Fields Do Not Support Filtering and Aggregation Logic
TrueWatch has some fields indexed with full-text search, which do not support filtering, aggregation, and related logic. The full-text index fields are as follows:
| Category | Fields |
|---|---|
| Objects, Resource Catalog | message |
| Logs, Backup Logs | message |
| Security | message / title |
| Network | message |
| Traces | error_message / error_stack |
| Events | message / title / df_message / df_title |
| RUM Errors | error_message / error_stack |
| RUM Long Tasks | long_task_message / long_task_stack |
Figure 1
Error Details¶
On the trace details page, if there is an error in the current trace, you can check the error span filter in the sidebar. The system will automatically highlight all error Spans. Clicking any highlighted Span allows you to view its associated error stack, logs, and other details.
For more details, refer to Error Tracking.
Service Context¶
By obtaining the object classification under the infrastructure resource catalog and selecting the most recent object based on create_time, you can quickly view the running information, service dependencies, and integration information of the current service.
Related Analysis¶
You can use Logs to view the log data associated with all services in the entire trace (related field: trace_id). You can customize the display columns. If you need to view more detailed log content, click the log content to jump to the log details page, or click the jump button to open the log page.
The system will match the log index mapping based on the service, env, version and other fields of the current trace, and automatically select the corresponding log index. The original index "Match Service" configuration will continue to take effect based on the service field mapping.
If you have administrator permissions or higher, you can customize the related fields. Click the settings button to the right of the related field, select the fields to relate in the dialog box, support manual input, drag and drop to reorder, etc. Confirm to complete the configuration.
Note: The custom fields for related logs and the configuration of custom fields for related analysis in the Service List affect each other. If you configure custom fields in the Service List, they will be displayed here synchronously.
When the application uses the ddtrace collector and enables both APM trace collection and Profile performance tracing data collection, the system provides Span-level association viewing of code hotspots, including execution time, methods, and execution time proportion.
Click View Profile Details to jump to the Profile details page to view more related code.
On the trace details page, you can use Hosts to view the metric views and attribute views of related hosts (related field: host).
- Metric View: View the performance metric status of the related host within 30 minutes before the trace ends to 30 minutes after the trace ends, including CPU, memory, and other performance metric views.
- Attribute View: Helps you backtrack the actual situation of the host object when the trace was generated. Supports viewing the latest object data generated within the corresponding time for the related host, including basic host information and integration running status. If cloud host collection is enabled, cloud provider information can also be viewed.
Note: TrueWatch saves the host object history data for the last 48 hours by default. If no host historical data corresponding to the current trace time is found, you will not be able to view the attribute view of the related host.
On the trace details page, you can use Containers to view the metric views and attribute views of related containers (related field: container_name).
-
Metric View: Supports viewing the performance metric status of the related container within 30 minutes before the trace ends to 30 minutes after the trace ends, including container CPU, memory, and other performance metric views.
-
Attribute View: Helps you backtrack the actual situation of the container object when the trace was generated. Supports viewing the latest object data generated within the corresponding time for the related container, including basic container information and attribute information.
On the trace details page, you can use Pods to view the attribute views and metric views of related Pods (related field: pod_name).
-
Metric View: Supports viewing the performance metric status of the related Pod within 30 minutes before the trace ends to 30 minutes after the trace ends, including container CPU, memory, and other performance metric views.
-
Attribute View: Helps you backtrack the actual situation of the Pod object when the trace was generated. Supports viewing the latest object data generated within the corresponding time for the related Pod, including basic container information and attribute information.
TrueWatch supports you in viewing network topology maps and overview data in Networks, including Host, Pod, Deployment, and Service dimensions.
Matching Fields:
To view related networks on the details page, you need to match the corresponding related fields. That is, you need to configure the corresponding field labels during data collection. Otherwise, you cannot view the related network views on the details page.
-
Host: Match field
host. -
Pod:
| Matching Field Priority |
|---|
| namespace, pod_name |
| namespace, pod |
| pod_name |
| pod |
- Deployment:
| Matching Field Priority |
|---|
| namespace, deployment_name |
| namespace, deployment |
| deployment_name |
| deployment |
- Service:
| Matching Field Priority |
|---|
| namespace, service_name |
| namespace, service |
Note:
- If related fields for Host, Pod, Deployment, and Service are queried simultaneously, the network data will be displayed in this order when entering the details page.
- If no related fields are found, they will be displayed in gray at the end, and clicking will prompt No matching network view found.