Log Details¶

In the log list, click the expand icon on the left side of a single log to slide out the details page for that log. You can view detailed information about the log, including the time it was generated, host, source, service, content, extended fields, context, and more.

View Complete Log¶

When logs are reported to TrueWatch, if a single log data exceeds 1M, the system will split it into multiple logs according to the 1M standard. For example, a 2.5M log will be split into 3 logs, 1M, 1M, and 0.5M respectively. The completeness of the split logs can be viewed through the following fields:

On the log details page, if the current log has been split into multiple pieces, a View Complete Log button will appear in the upper right corner. Clicking this button will open a new page listing all related logs in their split order. The page will also highlight the log selected before the jump with color for quick location of upstream and downstream logs.

TOBY AI AI Error Analysis¶

TrueWatch provides the capability to analyze error logs with one click. It leverages large language models to automatically extract key information from logs, combines online search engines and operational knowledge bases to quickly analyze potential fault causes and provide preliminary solutions.

1. Filter all logs with status error.

2. Click on a single data entry to expand its details page.

3. Click TOBY AI AI Error Analysis in the upper right corner.

4. You can then start the anomaly analysis.

Error Details¶

If the current log contains error_stack or error_message field information, the system will provide you with error details related to that log.

To view more log error information, you can go to Log Error Tracing.

Attribute Fields¶

Click on attribute fields for quick filtering and viewing. You can view host, process, trace, and container data related to the log.

Log Content¶

• The log content automatically switches between JSON and text viewing modes based on the message type. If the message field does not exist in the log, the log content section is not displayed. The log content supports expand and collapse, defaulting to expanded state. When collapsed, only one line height is displayed.

• For logs with source:bpf_net_l4_log, JSON and packet viewing modes are automatically provided. The packet mode displays client, server, time, and other information, and supports switching between absolute and relative time display, defaulting to absolute time. The switched configuration is saved in the local browser.

JSON Search¶

In JSON-formatted logs, JSON search can be performed on both key and value. After clicking, the explorer search bar will add the format @key:value for searching.

For multi-level JSON data, use . to represent the hierarchical relationship. For example, @key1.key2:value means searching for the value corresponding to key2 under key1.

For more details, refer to JSON Search.

Extended Fields¶

• In the search bar, you can enter field names or values for quick search and location.

• After checking the field alias, you can view it after the field name.

• Hover over an extended field, click the dropdown icon, and you can choose to perform the following operations on that field:

  • Filter Field Value

  • Exclude Field Value

  • Add to Display Columns

  • Perform Dimensional Analysis: Click to jump to Analysis Mode > Time Series Chart.

  • Copy

Context Logs¶

The context query function of the log service helps you trace related records before and after an abnormal log occurred through a timeline, quickly locating the root cause of the problem.

• On the log details page, you can directly view the context logs of this data entry.

• The left dropdown allows you to select an index to filter corresponding data.

• Data sorting.

• Directly jump to a new page of the log explorer based on the current details page.

• Display Item Configuration.

• Settings.

L::RE(`.*`):(`message`) { `index` = 'default' and `host` = "ip-172-31-204-89.cn-northwest-1" AND `source` = "kodo-log" AND `service` = "kodo-inner" AND `filename` = "0.log" and `log_read_lines` >= 1354140 and `log_read_lines` <= 1354200}  sorder by log_read_lines

Log Context Page¶

Click to jump to the log context page. You can manage all current data through the following operations:

• Enter text in the search box to search and locate data.

• Click the button on the side to switch the system's default auto-wrap to content overflow mode. In this mode, each log is displayed as a single line, and you can scroll left and right as needed.

• Directly locate to the current log.

• Go to top/bottom.

• Load 100 logs up/down.

Correlation Analysis¶

The system supports correlation analysis of log data. In addition to error details, extended fields, and context logs, you can also get a one-stop understanding of the host, container, network, etc., corresponding to the log.

Built-in Pages¶

For built-in pages like Host, Container, Pod, etc., you can perform the following operations:

(Taking the "Host" built-in page as an example)

• Edit the fields displayed on the current page. The system will automatically match corresponding data based on the fields.

• Choose to jump to the metric view or host details page.

• Filter the time range.

Built-in Views¶

In addition to the views displayed by default by the system here, you can also bind user views.

1. Enter the built-in view binding page.

2. View the default associated fields. You can choose to keep or delete fields, and you can also add new key:value fields.

3. Select a view.

4. After binding is complete, you can view the bound built-in view in the host object details. You can click the jump button to go to the corresponding built-in view page.