Skip to content

Log Collection

TrueWatch has comprehensive log collection capabilities, primarily divided into host log collection and K8S container log collection. The installation methods for DataKit differ between the two, and the log collection methods also vary. The collected log data is uniformly aggregated to TrueWatch for unified storage, search, and analysis, helping us quickly locate and resolve issues.

This article mainly introduces how to collect logs in a host environment. For log collection in a K8S environment, please refer to the best practice Several Methods for Log Collection in Kubernetes Clusters.

Prerequisites

Install DataKit.

Alternatively, you can log in to TrueWatch, go to Integration > DataKit, and select Linux, Windows, or MacOS based on your host system to obtain the DataKit installation command and steps.

Log Collector Configuration

After DataKit is installed, you can enable standard log collection or custom log collection to collect log data from various sources such as system logs, application logs like Nginx, Redis, Docker, ES, and more.

Navigate to the conf.d/samples directory under the DataKit installation directory, copy logging.conf.sample and rename it to logging.conf for configuration. After configuration, restart DataKit for the changes to take effect.

For details, please refer to Host Log Collection.

By enabling the standard log collectors supported by TrueWatch, such as Nginx, Redis, ES, etc., you can enable log collection with one click.

Note

When configuring a log collector, you need to enable the Pipeline function for logs to extract the time and status fields:

  • time: The generation time of the log. If the time field is not extracted or parsing of this field fails, the system's current time is used by default.
  • status: The level of the log. If the status field is not extracted, stauts is set to unknown by default.

For more details, please refer to the documentation Pipeline Configuration and Usage.

Log Data Storage

After the log collector is configured, restart DataKit, and the log data will be uniformly reported to the TrueWatch workspace.

  • For users with a large volume of log data, we can configure Log Index or Log Blacklist to save on data storage costs.
  • For users who need long-term storage of logs, we can use Log Backup to preserve log data.
Writing Data with Large Time Deviation

When writing data with timestamps that have a significant deviation from the current time, it can harm the efficiency of the min-max index of data blocks in the storage engine. This leads to the need to scan a large number of data blocks even when querying a very small time range, severely degrading query performance.

To mitigate this issue, the system defaults to filtering out data points whose timestamps deviate from the current time by more than 12 hours during write operations (only the timed-out data points are discarded, not the entire data packet). This mechanism helps maintain index effectiveness and improves query efficiency.

Kodo-X adds configuration:

kodo-x.yaml: Add 3 parameters. enable_discard_expired_data: Whether to enable discarding data with large time deviation. Enabled by default. discard_expired_seconds: The criterion for judging large time deviation. Default value is 12 hours.

global:
   enable_discard_expired_data: true
   discard_expired_seconds: 12 * 3600

discard_data_type: The type of data to discard. Default value:

DiscardDataType: map[string]bool{
                // "metering": true,
                // "TAE":      true,
                // "AE":       true,
                "B":  true,
                "CO": true,
                "D":  true,
                "E":  true,
                "EL": true,
                "L":  true,
                // "NM":       true,
                "N":  true,
                "OH": true,
                "O":  true,
                "P":  true,
                // "RM":       true,
                "R": true,
                "S": true,
                // "TM":       true,
                "T": true,
            },