Log Pipeline User Guide¶

Pipeline supports text parsing for log data in different formats. By writing Pipeline scripts, you can customize the extraction of structured logs that meet your requirements and use the extracted fields as attributes. With attribute fields, you can quickly filter relevant logs and perform data correlation analysis, helping you rapidly locate and resolve issues.

TrueWatch provides an official Pipeline script library with built-in log parsing Pipelines for various scenarios. It also supports creating custom Pipeline scripts. The following sections describe how to use the custom Pipeline feature.

Prerequisites¶

You need to create a TrueWatch account and install DataKit on your host.
Enable the log collector and turn on the Pipeline feature in the configuration file.

Custom Pipeline Script File¶

Step 1: Enable the Log Collector (Using DataKit Logs as an Example)¶

After installing DataKit on the host, navigate to the /usr/local/datakit/conf.d/log directory, copy logging.conf.sample and rename it to logging.conf. Edit logging.conf to configure the path and source of the DataKit logs. For example:

logfiles = ["/var/log/datakit/log"]
source = "datakit"

Note: In the log collector, the Pipeline feature is enabled by default. The Pipeline = "" field can be left empty—the system automatically matches a script file with the same name as the source. If the log source and Pipeline file name do not match, you must configure Pipeline = "xxxxxx.p" in the log collector.

[[inputs.logging]]
  ## required
  logfiles = [
    "/var/log/datakit/log",
  ]
  # only two protocols are supported:TCP and UDP
  # sockets = [
  #      "tcp://0.0.0.0:9530",
  #      "udp://0.0.0.0:9531",
  # ]
  ## glob filteer
  ignore = [""]

  ## your logging source, if it's empty, use 'default'
  source = "datakit"

  ## add service tag, if it's empty, use $source.
  service = ""

  ## grok pipeline script name
  ## If no specific pipeline is specified, the script with the same name as the source is automatically used.
  pipeline = ""

After configuration, restart DataKit using the command datakit --restart to apply the changes.

Step 2: Determine the Fields to Extract Based on the Collected Logs¶

After enabling the log collector, you can view the collected DataKit logs in the TrueWatch workspace. Observe and analyze the DataKit logs to determine the fields to extract, such as log timestamp, log level, log module, module content, and the log message itself.

Step 3: Create a Custom Pipeline¶

In the TrueWatch workspace, go to Logs > Pipelines and click Create Pipeline to create a new Pipeline file.

Filter Logs¶

Select "datakit" as the log source. A Pipeline file with the same name as the source is automatically generated.

Define Parsing Rules¶

Define the parsing rules for your logs. Multiple script functions are supported. You can view the syntax of each function directly from the script function list provided by TrueWatch.

In this example, based on the log observation, you can write the Pipeline script. Use the add_pattern() function to define custom patterns first, then reference those custom patterns in the Grok processor to parse the logs. The example below uses rename and default_time to optimize the extracted fields.

add_pattern('_dklog_date', '%{YEAR}-%{MONTHNUM}-%{MONTHDAY}T%{HOUR}:%{MINUTE}:%{SECOND}%{INT}')
add_pattern('_dklog_level', '(DEBUG|INFO|WARN|ERROR|FATAL)')
add_pattern('_dklog_mod', '%{WORD}')
add_pattern('_dklog_source_file', '(/?[\\w_%!$@:.,-]?/?)(\\S+)?')
add_pattern('_dklog_msg', '%{GREEDYDATA}')

grok(_, '%{_dklog_date:log_time}%{SPACE}%{_dklog_level:level}%{SPACE}%{_dklog_mod:module}%{SPACE}%{_dklog_source_file:code}%{SPACE}%{_dklog_msg:msg}')
rename("time", log_time) # Rename log_time to time
default_time(time)       # Use the time field as the timestamp of the output data

For more Pipeline parsing rules, see the document Text Data Processing (Pipeline).

Test with a Log Sample¶

After writing the script rules, you can input sample log data to test whether the parsing rules are correct.

Note:

Testing with a log sample is optional.
When the custom Pipeline is saved, the sample log test data is saved together.

Step 4: Save the Pipeline File¶

After configuring all required fields, save the Pipeline file. The custom Pipeline file will then appear in the log Pipelines list.

Pipeline files created in the TrueWatch workspace are saved uniformly in the /usr/local/datakit/Pipeline_remote directory.

Note: DataKit has two Pipeline directories. DataKit automatically matches Pipeline files in these directories.

Pipeline: Directory for the official Pipeline library.
Pipeline_remote: Directory for custom Pipeline files from the TrueWatch workspace.
If the same Pipeline file name exists in both directories, DataKit gives priority to the file in Pipeline_remote.

Step 5: View the Extracted Fields in TrueWatch¶

In the TrueWatch workspace, select the log source datakit in the log explorer. On the log details page, you can see the fields and their values under the "Attributes" section. These are the fields displayed after log parsing. For example:

code: container/input.go:167
level: ERROR
module: container

Clone the Official Pipeline Library Script¶

Step 1: Enable the Nginx Collector¶

After installing DataKit on the host, navigate to /usr/local/datakit/conf.d/nginx, copy nginx.conf.sample and rename it to nginx.conf. Edit nginx.conf to configure the path to the Nginx logs and the Pipeline. For example:

files = ["/var/log/nginx/access.log","/var/log/nginx/error.log"]
Pipeline = "nginx.p" (The "" can be left empty—DataKit automatically matches a script file with the same name as the source.)

Note: When the Pipeline is enabled, DataKit automatically matches the Pipeline script file based on the log source. If the log source and Pipeline file name do not match, you must configure the collector accordingly. For example, if the log source is nginx and the Pipeline file name is nginx1.p, you must configure Pipeline = "nginx1.p".

[[inputs.nginx]]
        url = "http://localhost/server_status"
        # ##(optional) collection interval, default is 30s
        # interval = "30s"
        use_vts = false
        ## Optional TLS Config
        # tls_ca = "/xxx/ca.pem"
        # tls_cert = "/xxx/cert.cer"
        # tls_key = "/xxx/key.key"
        ## Use TLS but skip chain & host verification
        insecure_skip_verify = false
        # HTTP response timeout (default: 5s)
        response_timeout = "20s"

        [inputs.nginx.log]
                files = ["/var/log/nginx/access.log","/var/log/nginx/error.log"]
        #       # grok pipeline script path
                pipeline = "nginx.p"
        [inputs.nginx.tags]
        # some_tag = "some_value"
        # more_tag = "some_other_value"
        # ...

After configuration, restart DataKit using the command datakit --restart to apply the changes.

For more Nginx collector configuration, refer to Nginx.

Step 2: Determine the Fields to Extract Based on the Collected Logs¶

After enabling the Nginx collector, configuring the log file path, and enabling the Pipeline, you can view the collected Nginx logs in the TrueWatch workspace. In the log details, you can see the fields parsed by the Pipeline file. Observe and analyze the logs to determine whether the extracted fields need optimization.

Step 3: Clone and Customize the Official Pipeline¶

In the TrueWatch workspace, go to Logs > Pipelines, click Official Pipeline Library, select and clone the nginx.p Pipeline file.

In Filter Logs, select "nginx".
In Define Parsing Rules, optimize the parsing rules.
In Test with Log Sample, input Nginx log data and test according to the configured parsing rules.