Full-line Index¶

What Is a Full-line Index¶

A full-line index is a full-text search method for logs. When enabled, the system includes all business fields in the log into the full-text index. You can search for logs containing a value by entering that value without specifying the field name in advance.

For example, a log contains the following fields:

{
  "service": "order-api",
  "trace_id": "8d4f2a",
  "order_id": "O-1001",
  "amount": 128.5
}

After the full-line index is enabled, searching directly for 8d4f2a, O-1001, or 128.5 will retrieve this log.

A full-line index is suitable for the following scenarios:

• Logs have already been extracted into multiple structured fields and no longer rely on a single message block.
• You need to quickly search across different business fields such as trace_id, order ID, or user ID.
• The same log index contains multiple log structures, and it is impossible to predefine a uniform search field for all logs.
• You want to retain field-based querying capability while also supporting full-text search across business fields.

Differences from message-only Index¶

A full-line index only changes the scope of full-text search and does not replace field filtering. For already extracted fields, you can still use field conditions such as service:order-api for precise filtering, aggregation, or analysis.

How It Works¶

The system uses variant to uniformly carry the business fields in the full-line index and builds a full-text index only for variant.

No message:
Business fields → variant → Full-text index

Message exists:
message + Other business fields → variant → Full-text index

The specific rules are as follows:

• A full-line index includes business fields of the log, not system fields.
• When the log does not have a message, other business fields can still participate in full-text search.
• When the log has a message, the system does not delete the field; instead, it writes the message as a regular business field into variant.
• The system does not build two full-text indexes for both message and variant.
• The same log index can store logs with or without a message simultaneously.

Whether message exists depends on the original log content and the actual processing result of DataKit and Pipeline. Enabling a full-line index does not require the deletion of message.

Enabling a Full-line Index¶

1. Go to Logs > Index.
2. Create a new log index, or edit an existing one.
3. Expand Advanced Options.
4. Under Full-text Index Field, select Full-line Index.
5. Save the configuration.

Searching with a Full-line Index¶

After configuration and log ingestion, go to Logs > Explorer and select the corresponding log index.

The explorer supports text search, field filtering, combined search, JSON search, and DQL query. For complete search syntax and usage, see Explorer Search.

Full-text Search¶

Enter a business field value in the search box to search across all business fields without specifying the field name.

Using the above order log as an example:

• Enter O-1001 to match order_id.
• Enter 8d4f2a to match trace_id.
• Enter order-api to match service.
• If the log retains message, you can also search for content within message.

Text search tokenizes the input. To match an exact, continuous string, wrap the search content in English double quotes. For more details, see Text Search.

Field Filtering¶

If you already know the field name, you can still use field conditions to narrow the search scope. For example:

service:order-api

Full-text search is suitable when you are unsure which field contains the content. Field filtering is suitable when you know the field name and need precise filtering or aggregation analysis. Both can be used together. For the complete syntax of field filtering, see Filtering.

Search Notes¶

• Full-line full-text search only matches business fields, not system fields.
• When the log does not have a message, the explorer will combine the current log's business fields to display the log content.
• System fields are not included as part of the full-line index log content.
• If business fields have not yet been extracted from the original log, the full-line index can only search fields that actually exist. Therefore, for JSON or plain-text logs, extract the required fields first using the methods described below.

Extracting Log Fields¶

A full-line index indexes business fields that already exist in the log, but it does not automatically parse and split the content of message. If the original log is still a single block of JSON or plain text, you can extract the content into structured fields using DataKit or Pipeline.

After field extraction, whether to keep the original message depends on your actual needs:

• Keep message if you need to view the full original text or maintain compatibility with existing usage habits.
• Delete message if the structured fields are sufficient for display and querying.
• Regardless of whether message is retained, extracted business fields can participate in the full-line full-text index.

Extracting JSON Fields with DataKit¶

This is suitable for logs where each line is a standard JSON object and you want to extract all top-level fields directly. DataKit 2.9.0 and later can enable json_as_fields.

When enabled, DataKit converts the top-level properties of the JSON root object into log fields after character decoding, ANSI sanitization, and multi-line merging, and then executes the Pipeline.

Host Log Collection¶

Edit logging.conf:

[[inputs.logging]]
  logfiles = ["/var/log/order/*.json"]
  source = "order"
  service = "order-api"
  json_as_fields = true

After saving the configuration, restart DataKit.

Kubernetes Container Log Collection¶

You can enable JSON field mode for a specific container using Pod Annotation:

metadata:
  annotations:
    datakit/order-api.logs: >-
      [{"source":"order","service":"order-api","json_as_fields":true}]

Here, order-api is the container name. You can also use the same JSON configuration in the container environment variable DATAKIT_LOGS_CONFIG.

Log Streaming¶

Enable it in logstreaming.conf:

[inputs.logstreaming]
  json_as_fields = true

json_as_fields is a collector configuration, not an HTTP URL parameter; this configuration does not apply to influxdb, firelens, or firehose types.

Extraction Results¶

Original log:

{"timestamp":"2026-08-18T10:00:00+08:00","level":"INFO","service":"order-api","trace_id":"8d4f2a","order_id":"O-1001","amount":128.5,"labels":{"channel":"web"}}

After successful conversion:

• timestamp, level, service, trace_id, order_id, and amount become independent fields.
• The labels object is saved as a compact JSON string.
• DataKit no longer generates an additional message that holds the entire original JSON.
• If the original JSON itself contains a message, that field is retained normally.

The main field conversion rules are as follows:

• Top-level strings, booleans, integers, and decimals keep their types; objects and arrays are saved as compact JSON strings; null is ignored.
• If the JSON is invalid, the root is not an object, or there are no valid fields, the system falls back to the original message.
• . in field names is converted to _, newline characters are converted to spaces, and field names are up to 256 bytes.
• Each log retains at most 1024 fields, excluding tags.
• JSON fields override collector-level tags or fields with the same name.
• time, source, date, and storage_index in JSON are renamed to json_time, json_source, json_date, and json_storage_index respectively.

For the complete rules, see JSON Field Mode.

Extracting Fields with Pipeline¶

This is suitable for the following scenarios:

• You only need to extract a subset of fields.
• You need to rename or transform fields.
• The log is not standard JSON.
• You already have a Pipeline cleaning process and want to continue using it.

Extracting All Top-Level Scalar Fields¶

DataKit 2.2.0 and later can use json_all():

# Original data:
# {"service":"order-api","status":"info","trace_id":"8d4f2a","order_id":"O-1001","amount":128.5}

json_all(_, key_patterns=["*"])

After processing, you get independent fields such as service, status, trace_id, order_id, and amount.

json_all() only extracts top-level strings, numbers, and booleans; it does not recursively expand objects or arrays, nor does it save null. When neither include_keys nor key_patterns is configured, no fields are extracted. To extract all top-level scalar fields, you must explicitly set key_patterns=["*"].

Extracting Only Specified Fields¶

json(_, service)
json(_, level, status)
json(_, trace_id)
json(_, order_id)
json(_, amount)

This way, you can control the range of fields that enter the full-line index and also unify fields from different logs to the same name.

Objects or arrays are not extracted by json_all(). To retain such content, you can use json() to specify field extraction; the extracted result is saved as a JSON string.

Deleting message After Field Extraction¶

When extracting fields through Pipeline, the original log is still saved in message by default. If the structured fields are already sufficient for display and querying, you can call drop_origin_data() after field extraction to stop saving the original content.

json_all(_, key_patterns=["*"])
drop_origin_data()

You can also delete after extracting specified fields:

json(_, service)
json(_, level, status)
json(_, trace_id)
json(_, order_id)
json(_, amount)
drop_origin_data()

If the JSON itself contains a message and that field has also been extracted, you can keep or delete it as needed. To explicitly remove it when not needed, use:

json_all(_, key_patterns=["*"])
drop_origin_data()
drop_key(message)

The three functions have different effects:

• drop_origin_data(): stops outputting the original text saved during initialization; the entire log is still uploaded.
• drop_key(message): deletes the already extracted message field.
• drop(): discards the entire log; the log is not uploaded. It cannot be used to delete message.

For more syntax, see json(), json_all(), drop_origin_data(), and drop_key().

Verifying Pipeline Locally¶

After saving the Pipeline script, you can run it on the DataKit host:

datakit pipeline -P full_line_json.p \
  -T '{"service":"order-api","status":"info","trace_id":"8d4f2a","order_id":"O-1001","amount":128.5}'

Check that the output meets the following expectations:

• All required fields have been extracted.
• If drop_origin_data() is configured, the original message has been deleted.
• The log is not marked as drop: true.
• Fields such as status and log time conform to business expectations.

FAQ¶

Does a full-line index require deleting message?¶

No. When message exists, the system writes it as a regular business field into variant and does not create a separate full-text index for message. Whether to delete message depends on the need to retain and display the original content.

After enabling a full-line index, why can't I still search JSON fields in message?¶

If the entire JSON is still just a string content of message, its properties are not independent business fields. Although you can search for text within message, you cannot directly use these properties for field filtering or aggregation. We recommend extracting the required fields first using DataKit or Pipeline.

When there is no message, how does the log explorer display content?¶

When the log does not have a message, the log explorer combines the business fields of the current log to display the log content. System fields are not included as part of the log content.

Can the same index contain logs with and without message at the same time?¶

Yes. Both types of logs write business fields into variant under the same rules, and variant participates in the full-text index.

Can a full-line index search system fields?¶

No. A full-line index only includes business fields, not system fields. System fields can still be queried through field filtering and other methods.