DQL Function Reference¶

DQL provides a rich set of functions for data aggregation, transformation, and matching. This document details the semantics, parameters, and usage of each function.

Aggregation Functions¶

Aggregation functions are used to aggregate multiple rows of data into a single value, typically used with time windows (time-expr) and grouping (BY clause).

Basic Aggregation¶

sum¶

Calculates the sum of field values.

Syntax:

sum(field)

Parameters:

• field: Numeric field

Examples:

// Calculate total request count
M::http_requests:(sum(request_count)) [1h]

// Calculate total request count grouped by service
M::http_requests:(sum(request_count)) [1h] BY service

avg¶

Calculates the average of field values.

Syntax:

avg(field)

Parameters:

• field: Numeric field

Examples:

// Calculate average response time
M::response_time:(avg(duration)) [1h] BY endpoint

// Calculate average CPU usage
M::cpu:(avg(usage)) [1h] BY host

count¶

Counts the number of data rows.

Syntax:

count(field)
count(*)

Parameters:

• field: Any field, counts the number of non-null values
• *: Counts all rows

Examples:

// Count log entries
L::nginx:(count(*)) [1h]

// Count requests with response time
M::response_time:(count(duration)) [1h] BY service

min / max¶

Calculates the minimum or maximum value of a field.

Syntax:

min(field)
max(field)

Parameters:

• field: Numeric field

Examples:

// Find the maximum response time
M::response_time:(max(duration)) [1h] BY endpoint

// Find the range of CPU usage
M::cpu:(min(usage), max(usage)) [1h] BY host

first / last¶

Gets the first or last value (in chronological order).

Syntax:

first(field)
last(field)
last_row(field)

Parameters:

• field: Any field

Description:

• first: Returns the earliest value in time
• last: Returns the latest value in time; if the field is an array type, it will be expanded
• last_row: Returns the latest value in time; array types are not expanded

Examples:

// Get the latest status value
M::system:(last(status)) [1h] BY host

// Get initial and final values
M::counter:(first(value), last(value)) [1h] BY metric

any¶

Returns any non-null value. Suitable for obtaining sample data or scenarios where a specific aggregation order is not required.

Syntax:

any(field)

Parameters:

• field: Any field

Examples:

// Get a sample message
L::logs:(any(message)) [1h] BY service

// Get a sample error stack trace
L::error_logs:(any(stack_trace)) [1h] BY error_type

spread¶

Calculates the range (difference between maximum and minimum).

Syntax:

spread(field)

Parameters:

• field: Numeric field

Examples:

// Calculate the fluctuation range of response time
M::response_time:(spread(duration)) [1h] BY endpoint

stddev¶

Calculates the standard deviation.

Syntax:

stddev(field)

Parameters:

• field: Numeric field

Examples:

// Calculate the standard deviation of response time
M::response_time:(stddev(duration)) [1h] BY endpoint

mode¶

Calculates the mode (most frequent value).

Syntax:

mode(field)

Parameters:

• field: Any field

Examples:

// Find the most common response status code
M::http:(mode(status)) [1h] BY endpoint

count_series¶

Counts the number of time series (groups). Returns how many independent time series exist within the current query scope.

Syntax:

count_series(field)

Parameters:

• field: Any field (typically * or any existing field)

Examples:

// Count how many hosts are reporting CPU metrics
M::cpu:(count_series(*)) [1h]

// Count how many instances each service has
M::http_requests:(count_series(*)) [1h] BY service

Statistical Aggregation (Estimation Functions)¶

The following functions use probabilistic data structures for estimation, suitable for large-scale data scenarios, balancing accuracy and performance.

count_distinct¶

Calculates the number of distinct values in a field (estimated).

Syntax:

count_distinct(field)

Parameters:

• field: Any field

Algorithm Description:

Uses the HyperLogLog algorithm for cardinality estimation: - Number of registers: 2¹⁶ = 65536 - Uses LogLog-Beta estimation method - Standard error: approximately 0.4%

Use Cases:

• Counting unique users (UV)
• Calculating the number of distinct IP addresses
• Analyzing the number of unique request IDs

Examples:

// Count unique users
L::access_logs:(count_distinct(user_id)) [1d] BY service

// Count distinct accessing IPs
L::nginx:(count_distinct(client_ip)) [1h] BY endpoint

percentile¶

Calculates the percentile of a field (estimated).

Syntax:

percentile(field, n)
pXX(field)  // Shorthand form

Parameters:

• field: Numeric field
• n: Percentile, range 0-100

Shorthand Forms:

• p50(field) is equivalent to percentile(field, 50)
• p95(field) is equivalent to percentile(field, 95)
• p99(field) is equivalent to percentile(field, 99)

Algorithm Description:

Uses logarithmic linear interpolation histogram for estimation: - Bucket range: 10⁻⁹ to 10¹⁸, covering most numerical scenarios - Each order of magnitude is divided into 128 buckets - Uses linear interpolation in logarithmic space to improve accuracy

Use Cases:

• Calculating P99, P95 of response times
• Analyzing tail latency of performance metrics
• Evaluating Service Level Agreement (SLA) compliance

Examples:

// Calculate P99 of response time
M::response_time:(percentile(duration, 99)) [1h] BY service

// Using shorthand form
M::response_time:(p99(duration)) [1h] BY service

// Calculate multiple percentiles simultaneously
M::response_time:(p50(duration), p95(duration), p99(duration)) [1h] BY service

median¶

Calculates the median, equivalent to percentile(field, 50).

Syntax:

median(field)

Examples:

// Calculate the median of response time
M::response_time:(median(duration)) [1h] BY service

Histogram Functions¶

DQL provides three histogram-related functions, suitable for different data sources and scenarios:

histogram_auto (Recommended)¶

Automatically generates distribution histograms, specifically designed for numerical distribution statistics of detailed data like logs and Trace.

Features:

• No need to specify bucket boundaries; automatically adapts to data distribution
• Uses logarithmic linear interpolation histogram algorithm, covering numerical range from 10⁻⁹ to 10¹⁸
• Returns both quantile statistics and bucket distribution information

Syntax:

histogram_auto(field)

Parameters:

• field: Numeric field

Return Values: | Column Name | Description | | --------------- | ------------------------------- | | lower_bounds | Array of lower bounds for each bucket | | upper_bounds | Array of upper bounds for each bucket | | counts | Array of counts for each bucket | | min | Minimum value | | p50 | Median | | p75 | 75th percentile | | p90 | 90th percentile | | p95 | 95th percentile | | p99 | 99th percentile | | max | Maximum value |

Algorithm Description: Uses estimated histogram (logarithmic linear interpolation), dividing each order of magnitude into 128 buckets, suitable for distribution statistics of large-scale data.

Use Cases:

• Analyzing response time distribution in logs
• Statistics of latency distribution in Trace
• Exploratory data analysis without preset bucket boundaries

Examples:

// Analyze response time distribution in Nginx access logs
L::nginx:(histogram_auto(response_time)) [1h]

// Statistics of request latency distribution by service
L::app_logs:(histogram_auto(duration)) [1h] BY service

// Statistics of latency distribution for Trace calls
T::http_client:(histogram_auto(elapsed)) [1h] BY operation

Result Example:

Note: lower_bounds, upper_bounds, counts are array types, representing the boundaries and counts of each bucket.

histogram (Deprecated)¶

Generates a histogram with specified bucket boundaries. This function is Deprecated, recommended to use histogram_auto instead.

Description: histogram requires manually specifying bucket boundary parameters, making it less flexible. histogram_auto can automatically adapt to data distribution, cover a wider numerical range, and return richer statistical information.

Syntax:

histogram(field, left_bound, right_bound, bucket_size [, threshold])

Parameters:

• field: Numeric field
• left_bound: Left boundary
• right_bound: Right boundary
• bucket_size: Bucket size
• threshold (optional): Minimum count per bucket; buckets below this value are not returned

Return Values: Returns two columns: bucket_le (right boundary of bucket) and count (count)

Examples:

// Generate a histogram in the range 0-1000ms, with one bucket per 100ms
M::response_time:(histogram(duration, 0, 1000, 100)) [1h]

// Recommended to use histogram_auto instead
M::response_time:(histogram_auto(duration)) [1h]

Result Example:

histogram_quantile¶

Calculates quantiles from Prometheus histogram metrics.

Features:

• Specifically designed to handle histogram-type metrics reported by Prometheus
• Relies on the le label (or VictoriaMetrics' vmrange label) to identify bucket boundaries
• Input data should be cumulative counts

Syntax:

histogram_quantile(field, q)

Parameters:

• field: Histogram count field (e.g., http_request_duration_bucket)
• q: Quantile, range 0-1 (e.g., 0.99 represents P99)

Use Case Comparison:

le Label Handling Mechanism:

histogram_quantile relies on the le label (less than or equal) to identify histogram bucket boundaries:

1. Prometheus format (default): Uses the le label directly to represent the bucket upper boundary
2. le value is numeric (e.g., "0.1", "1", "10") or "+Inf" (infinity)

3. Data should be cumulative counts

4. VictoriaMetrics format: Uses the vmrange label to represent ranges

5. Format is "lower...upper" (e.g., "0.1...0.2")
6. Data is range counts (non-cumulative)
7. The function automatically converts range counts to cumulative counts

Calculation Process: 1. Sort all buckets by le value 2. If in vmrange format, accumulate counts to convert to cumulative distribution 3. Ensure bucket counts are monotonically increasing (fix any abnormal data) 4. Use linear interpolation to calculate the target quantile

Differences from PromQL:

Equivalent Examples:

Assume a histogram metric http_request_duration_bucket, containing le labels (e.g., 0.1, 0.5, 1, 5, +Inf) and service label.

Scenario 1: Calculate P99 latency

DQL:

M::http_request_duration:(histogram_quantile(duration_bucket, 0.99)) [1h] BY service

PromQL equivalent:

histogram_quantile(0.99, sum(rate(http_request_duration_bucket[1h])) by (le, service))

Scenario 2: Calculate P95 latency for each service (multiple groups)

DQL:

M::http_request_duration:(
    histogram_quantile(duration_bucket, 0.95)
) [1h] BY service, endpoint

PromQL equivalent:

histogram_quantile(0.95, sum(rate(http_request_duration_bucket[1h])) by (le, service, endpoint))

Scenario 3: Calculate P50 (median) and P99

DQL:

M::http_request_duration:(
    histogram_quantile(duration_bucket, 0.50) as p50,
    histogram_quantile(duration_bucket, 0.99) as p99
) [1h] BY service

PromQL equivalent:

label_join(
  histogram_quantile(0.50, sum(rate(http_request_duration_bucket[1h])) by (le, service)), "quantile", "", "0.50"
)
or
label_join(
  histogram_quantile(0.99, sum(rate(http_request_duration_bucket[1h])) by (le, service)), "quantile", "", "0.99"
)

Note: PromQL requires label_join or label_replace to distinguish results of different quantiles.

Notes:

• Input data must contain le or vmrange label, otherwise calculation is impossible
• If there is no +Inf bucket, the upper boundary of the last bucket will be used as the maximum value
• Buckets with count 0 or NaN are skipped
• Returns -Inf when quantile < 0, returns +Inf when quantile > 1

TopN Functions¶

top¶

Gets the top N largest values.

Syntax:

top(field, n)

Parameters:

• field: Numeric field
• n: Number of values to return

Examples:

// Get the 5 requests with the longest response time
M::response_time:(top(duration, 5)) [1h] BY service

// Get the 10 hosts with the highest traffic
M::network:(top(bytes, 10)) [1h]

Result Example:

Note: Returns multiple rows, each containing a TopN value.

bottom¶

Gets the top N smallest values.

Syntax:

bottom(field, n)

Parameters:

• field: Numeric field
• n: Number of values to return

Examples:

// Get the 5 requests with the shortest response time
M::response_time:(bottom(duration, 5)) [1h] BY service

Result Example:

Note: Returns multiple rows, each containing a BottomN value.

Value Collection Functions¶

distinct¶

Returns all distinct values of a field.

Syntax:

distinct(field)

Examples:

// Get all distinct status codes
M::http:(distinct(status)) [1h] BY endpoint

Result Example:

Note: Returns multiple rows, each containing a distinct value.

distinct_by_collapse¶

Gets distinct values of a field by collapse strategy, and retains the last value of other fields during deduplication.

Syntax:

distinct_by_collapse(field, [last_field1, last_field2, ...])

Parameters:

• field: Field for deduplication basis
• last_fields (optional): List of fields whose last value needs to be retained

Description: Unlike distinct, distinct_by_collapse retains information from other related fields (taking the last value) during deduplication, suitable for scenarios requiring contextual information.

Examples:

// Get distinct user IDs and retain the last access time for each user
L::access_logs:(distinct_by_collapse(user_id, [timestamp])) [1h]

// Get distinct hosts and retain the last status and message
O::HOST:(distinct_by_collapse(host, [status, message])) [1h]

Result Example:

Note: Returns the deduplicated main field values, and the last value of other fields specified by last_fields.

collect¶

Collects all values (including duplicates).

Syntax:

collect(field [, limit])

Parameters:

• field: Any field
• limit (optional): Maximum number to collect

Examples:

// Collect all response times
M::response_time:(collect(duration)) [1h] BY service

// Collect up to 100 values
M::response_time:(collect(duration, 100)) [1h] BY service

Result Example:

Note: Returns array type, containing all collected values (may include duplicates).

collect_distinct¶

Collects all distinct values.

Syntax:

collect_distinct(field [, limit])

Parameters:

-## 请继续翻译 field: Any field - limit (optional): Maximum number to collect

Examples:

// Collect all distinct error types
L::error_logs:(collect_distinct(error_type)) [1h] BY service

Result Example:

Note: Returns array type, containing all deduplicated values.

field_values¶

Gets all values of a field, returns array type.

Syntax:

field_values(field)

Examples:

// Get all tag values
M::metrics:(field_values(tags)) [1h] BY metric_name

Result Example:

Note: Returns array type, containing all values of the field.

Filter Aggregation¶

count_filter¶

Counts the number of field values within a specified list.

Syntax:

count_filter(field, [value1, value2, ...])

Parameters:

• field: Any field
• values: List of values

Examples:

// Count requests with specific status codes
M::http:(count_filter(status, [200, 201, 204])) [1h] BY endpoint

// Count error-level logs
L::logs:(count_filter(level, ["error", "critical"])) [1h] BY service

Helper Functions¶

default¶

Sets a default value for a field, returns the default value when the field is empty.

Syntax:

default(field, default_value)

Parameters:

• field: Any field
• default_value: Default value (can be numeric, string, boolean, or null)

Examples:

// Set default response time for null values
M::response_time:(default(duration, 0)) [1h] BY service

Time Series Functions¶

Time series functions are used to process data that changes over time, especially Counter-type metrics.

Rollup Functions¶

Rollup functions are used for preprocessing raw time series data within time windows. For detailed explanation, please refer to Rollup Functions in this article.

Writing Instructions:

• Rollup is written in the time clause, e.g., [rate], [1h::5m:rate].
• Writing in the outer query (e.g., rate(DQL)) belongs to outer functions, not Rollup.

Supported Rollup Functions:

Examples:

// Calculate request QPS
M::http_requests:(sum(request_count)) [1h::5m:rate] BY service

// Shorthand form
M::cpu:(max(usage)) [rate]

Growth Rate Calculation¶

rate¶

Calculates the growth rate of a metric (per second).

Syntax:

rate(field)

Description: rate calculates the average growth rate of a Counter metric within a time window. For monotonically increasing Counter-type metrics, directly aggregating raw values is meaningless; growth rate needs to be calculated first.

Use Cases:

• Calculating request QPS
• Calculating data write rate
• Analyzing traffic growth trends

Examples:

// Calculate request QPS
M::http_requests:(sum(request_count)) [rate] BY service

// Calculate data write rate
M::data_ingestion:(sum(bytes)) [rate] BY source

irate¶

Calculates the instantaneous growth rate of a metric.

Syntax:

irate(field)

Description: Unlike rate, irate uses only the last two data points to calculate the growth rate, reflecting instantaneous change rate, more suitable for alerting scenarios.

Examples:

// Calculate instantaneous request QPS
M::http_requests:(sum(request_count)) [irate] BY service

increase¶

Calculates the increase amount of a metric.

Syntax:

increase(field)

Description: increase returns the total increase amount within the time window, not the growth rate.

Examples:

// Calculate total request increase
M::http_requests:(sum(request_count)) [increase] BY service

rate_over_sum¶

Calculates the average per second (sum / time window seconds).

Syntax:

rate_over_sum(field)

Description: Equivalent to sum(field) / time window (seconds), used to calculate the average per second. Often used in the Rollup stage to convert cumulative values to per-second rates.

Difference from rate:

• rate: Calculates growth rate of Counter (handles resets)
• rate_over_sum: Simply divides sum by time window seconds

Examples:

// Calculate average requests per second
M::http_requests:(rate_over_sum(request_count)) [1h] BY service

rate_over_count¶

Calculates count per second (count / time window seconds).

Syntax:

rate_over_count(field)

Description: Equivalent to count(field) / time window (seconds), used to calculate occurrences per second.

Examples:

// Calculate error log entries per second
L::error_logs:(rate_over_count(*)) [1h] BY error_type

Difference Calculation¶

This section describes function semantics. The same functions can be used both as Rollup (e.g., [rate], [increase]) and written in query expressions (e.g., rate(field), increase(field)); they execute at different stages, prioritize selection based on business needs.

rate / deriv¶

Calculates rate of change (derivative). rate is used for Counter-type metrics (ignores negative values), deriv is used for Gauge-type metrics (preserves negative values).

Aliases: rate alias is non_negative_derivative; deriv alias is derivative (PromQL style)

Syntax:

// Counter metrics: Calculate non-negative rate of change (ignores negative values due to resets)
rate(field)

// Gauge metrics: Calculate full rate of change (including negative values)
deriv(field)

Function Selection:

Examples:

// Counter metrics: Calculate request QPS
M::requests:(rate(count)) [1h::5m] BY service

// Gauge metrics: Calculate memory usage change rate
M::memory:(deriv(used)) [1h::5m] BY host

increase / difference¶

Calculates difference between adjacent values. increase is used for Counter-type metrics (ignores negative values), difference is used for Gauge-type metrics (preserves negative values).

Description: increase and difference are two independent functions with different behaviors, not aliases.

Syntax:

// Counter metrics: Calculate non-negative difference (ignores negative values due to resets)
increase(field)

// Gauge metrics: Calculate full difference (including negative values)
difference(field)

Function Selection:

Examples:

// Counter metrics: Calculate request increase amount
M::requests:(increase(count)) [1h::5m] BY service

// Gauge metrics: Calculate request count change (may increase or decrease)
M::requests:(difference(count)) [1h::5m] BY service

Moving Calculation¶

moving_average¶

Calculates moving average.

Syntax:

moving_average(field, n)

Parameters:

• field: Numeric field
• n: Window size (number of data points)

Examples:

// Calculate 5-point moving average
M::cpu:(moving_average(usage, 5)) [1h::1m] BY host

cumsum¶

Calculates cumulative sum.

Syntax:

cumsum(DQL_expression)

Examples:

// Calculate cumulative request count
cumsum(M::requests:(sum(count)) [1h::5m] BY service)

Transformation Functions¶

Transformation functions are used for mathematical operations, type conversions, or string processing on field values.

Mathematical Functions¶

abs¶

Calculates absolute value.

Syntax:

abs(field)

Examples:

// Calculate absolute value of temperature deviation
M::temperature:(abs(deviation)) [1h] BY sensor

round / ceil / floor¶

Rounding functions.

Syntax:

round(field)  // Round to nearest integer
ceil(field)   // Round up
floor(field)  // Round down

Examples:

// Round up response time
M::response_time:(ceil(duration)) [1h]

// Round percentage
M::cpu:(round(usage)) [1h] BY host

log / log2 / log10¶

Logarithmic functions.

Syntax:

log(field)    // Natural logarithm
log2(field)   // Base-2 logarithm
log10(field)  // Base-10 logarithm

Examples:

// Calculate logarithmically transformed value
M::metrics:(log(value)) [1h] BY metric_name

Type Conversion¶

int / uint / float / string / bool¶

Type conversion functions.

Syntax:

int(field)    // Convert to signed integer
uint(field)   // Convert to unsigned integer
float(field)  // Convert to floating-point
string(field) // Convert to string
bool(field)   // Convert to boolean

Examples:

// Convert string to numeric
L::logs:(int(response_time)) [1h] BY service

// Convert numeric to string for concatenation
M::metrics:(string(value)) [1h] BY metric_name

String Functions¶

lower¶

Converts string to lowercase.

Syntax:

lower(field)

Parameters:

• field: String field

Return Value: Returns the converted lowercase string.

upper¶

Converts string to uppercase.

Syntax:

upper(field)

Parameters:

• field: String field

Return Value: Returns the converted uppercase string.

trim¶

Removes whitespace characters from both ends of a string.

Syntax:

trim(field)

Parameters:

• field: String field

Return Value: Returns the string with whitespace removed from both ends.

ltrim¶

Removes whitespace characters from the left side of a string.

Syntax:

ltrim(field)

Parameters:

-## 请继续翻译 field: String field

Return Value: Returns the string with whitespace removed from the left side.

rtrim¶

Removes whitespace characters from the right side of a string.

Syntax:

rtrim(field)

Parameters:

• field: String field

Return Value: Returns the string with whitespace removed from the right side.

length¶

Returns the length of a string (counted by characters).

Syntax:

length(field)

Parameters:

• field: String field

Return Value: Returns the string length.

substr¶

Extracts a substring.

Syntax:

substr(field, start)
substr(field, start, length)

Parameters:

• field: String field
• start: Starting position (0-based, negative numbers indicate starting from the end)
• length (optional): Length of substring

Return Value: Returns the extracted substring.

Examples:

// Extract first 100 characters of message
L::logs:(substr(message, 0, 100)) [1h]

// Extract last 10 characters
L::logs:(substr(message, -10)) [1h]

Result Example:

regexp_extract¶

Regular expression extraction.

Syntax:

regexp_extract(field, pattern)
regexp_extract(field, pattern, n)

Parameters:

• field: String field
• pattern: Regular expression
• n (optional): Extract the nth capture group, default is 0 (entire match)

Return Value: Returns a single string, extracting the content of the nth capture group. Returns null if no match.

Examples:

// Extract error code
L::logs:(regexp_extract(message, 'error_code: (\d+)', 1)) [1h]

// Extract IP address
L::nginx:(regexp_extract(message, '(\d+\.\d+\.\d+\.\d+)', 1)) [1h]

Result Example:

regexp_extract_all¶

Extracts all matching results.

Syntax:

regexp_extract_all(field, pattern)
regexp_extract_all(field, pattern, n)

Return Value: Returns a string array containing all matched substrings.

Examples:

// Extract all numbers
L::logs:(regexp_extract_all(message, '\d+', 0)) [1h]

// Extract all IP addresses
L::logs:(regexp_extract_all(message, '\d+\.\d+\.\d+\.\d+', 0)) [1h]

Result Example:

md5¶

Calculates MD5 hash value.

Syntax:

md5(field)

Return Value: Returns a 32-character hexadecimal string (lowercase).

Examples:

// Calculate MD5 of message
L::logs:(md5(message)) [1h]

Result Example:

concat¶

String concatenation.

Syntax:

concat(field, ...)

Return Value: Returns a single concatenated string.

Examples:

// Concatenate multiple fields
L::logs:(concat(service, ":", level)) [1h]

// Results: "api:error", "web:info", etc.

Result Example:

set¶

Deduplicates and sorts an array field.

Syntax:

set(DQL_expression)

Return Value: Returns a deduplicated and sorted array.

Examples:

// Get all distinct tags
set(M::metrics:(tags) [1h] BY metric_name)

// Deduplicate collect results
set(M::http:(collect(status)) [1h] BY endpoint)

Result Example:

Log Clustering¶

drain¶

Uses the Drain algorithm to cluster logs and generate log templates.

Syntax:

drain(field, similarity_threshold, max_clusters)

Parameters:

• field: String field (usually message)
• similarity_threshold: Similarity threshold, range (0, 1], higher values mean stricter clustering
• max_clusters: Maximum number of clusters, range [1, 10000]

Algorithm Description:

Drain is a log clustering algorithm based on a parse tree, capable of automatically identifying constant and variable parts in logs to generate log templates.

Use Cases:

• Log pattern recognition
• Anomaly log clustering analysis
• Log noise reduction

Examples:

// Cluster logs, similarity 0.7, maximum 1000 clusters
L::logs:(drain(message, 0.7, 1000)) [1h]

// Stricter clustering
L::logs:(drain(message, 0.9, 500)) [1h]

Result Example:

Note: Returns string-type log templates, where <...> indicates variable parts.

Matching Functions¶

Matching functions are used for text matching in WHERE clauses and can also be used as expressions returning boolean values.

Substring Matching¶

match¶

Checks if a field contains a specified substring.

Syntax:

match(pattern)
match(field, pattern)

Parameters:

• pattern: Substring to match
• field: Field name (optional, can be omitted in WHERE clause)

Examples:

// Use in WHERE
L::logs:(message) {match(message, "error")} [1h]

// Shorthand form
L::logs:(message) {match("error")} [1h]

// As an expression
L::logs:(match(message, "timeout")) [1h] BY match_result

Phrase Matching¶

search¶

Tokenized phrase matching, supports mixed Chinese and English.

Syntax:

search(query)
search(field, query)

Parameters:

• query: Query phrase
• field: Field name (optional)

Matching Rules:

• Chinese: Tokenized by character for matching
• English: Tokenized by word boundaries (space, punctuation separated)
• Case-insensitive

Examples:

// Match logs containing "connection timeout"
L::logs:(message) {search("connection timeout")} [1h]

// Chinese matching
L::logs:(message) {search("连接超时")} [1h]

// Mixed Chinese and English
L::logs:(message) {search("error 错误")} [1h]

Regular Expression Matching¶

re / regex / regexp¶

Regular expression matching.

Syntax:

re(pattern)
re(field, pattern)
regex(pattern)
regex(field, pattern)
regexp(pattern)
regexp(field, pattern)

Parameters:

• pattern: Regular expression (supports PromRegex syntax)
• field: Field name (optional)

Examples:

// Match logs starting with error
L::logs:(message) {re("error.*")} [1h]

// Match error codes of specific format
L::logs:(message) {regexp(message, "ERR-\d{4}")} [1h]

// Use regex in data source
M::re('cpu.*'):(usage) [1h]

Wildcard Matching¶

wildcard¶

Wildcard pattern matching.

Syntax:

wildcard(pattern)
wildcard(field, pattern)

Parameters:

• pattern: Wildcard pattern
• *: Matches any number of characters
• ?: Matches a single character
• field: Field name (optional)

Examples:

// Match messages starting with error
L::logs:(message) {wildcard("error*")} [1h]

// Match specific format
L::logs:(message) {wildcard(message, "ERR-????")} [1h]

CIDR Matching¶

cidr¶

IP address network segment matching.

Syntax:

cidr(cidr)
cidr(field, cidr)

Parameters:

• cidr: Network segment in CIDR notation, e.g., 192.168.1.0/24
• field: IP address field (optional)

Examples:

// Match internal IPs
L::nginx:(*) {cidr(client_ip, "10.0.0.0/8")} [1h]

// Match specific network segment
L::nginx:(*) {cidr(client_ip, "192.168.1.0/24")} [1h]

Field Existence Check¶

exists¶

Checks if a field exists. exists() is a special placeholder, typically used on the right side of a comparison expression.

Syntax:

exists()

Recommended Usage:

field = exists()   // Field exists (non-null)
field != exists()  // Field does not exist or is null

Examples:

// Find logs with error_type field
L::logs:(message) {error_type = exists()} [1h]

// Find logs without error_type field
L::logs:(message) {error_type != exists()} [1h]

Query String Syntax¶

query_string¶

Uses query string syntax for complex matching.

Syntax:

query_string(query)
query_string(field, query)

Parameters:

• query: Query string
• field: Field name (optional, defaults to full-text matching)

Query String Syntax:

1. Term Matching¶

foo              # Match content containing foo
"foo bar"        # Exact phrase match, must appear consecutively
foo\ bar         # Escape space, match "foo bar" as a whole

2. Wildcards¶

foo*             # Match content starting with foo
foo?bar          # ? matches a single character
"foo*bar"        # Wildcards inside quotes are not parsed, matched literally

3. Regular Expressions¶

/foo.*bar/       # Regular expression enclosed in slashes
/joh?n(ath[oa]n)/  # Complex regex

4. Boolean Operators¶

foo AND bar      # Logical AND, both must be contained
foo OR bar       # Logical OR, at least one must be contained
NOT foo          # Logical NOT, does not contain foo

# Shorthand forms
foo && bar       # Equivalent to foo AND bar
foo || bar       # Equivalent to foo OR bar
!foo             # Equivalent to NOT foo

5. Grouping¶

(foo OR bar) AND baz       # Use parentheses to change precedence
!(status 429 reading)      # Negate the entire expression

6. Default Operator¶

When multiple terms are separated by spaces, OR is used by default (configurable to AND):

foo bar          # Equivalent to foo OR bar

7. Tokenization Rules¶

The specific behavior of the queryString function depends on the underlying storage engine used by the current workspace:

• ScopeDB environment: queryString performs case-insensitive contains search, i.e., matches whether any substring of the query string is contained in the field value, without involving tokenization;

• Doris environment: queryString performs matching based on full-text index tokenization, and its semantics are directly related to the tokenization results. The tokenizer used by Doris follows the default word boundary specification of Unicode Standard Annex #29.

Examples¶

// Simple term matching
L::logs:(message) {query_string("error timeout")} [1h]

// Boolean combination
L::logs:(message) {query_string("error AND NOT timeout")} [1h]

// Regular expression
L::logs:(message) {query_string("/ERR-\d{4}/")} [1h]

// Complex query
L::logs:(message) {query_string("(error OR warn) AND service")} [1h]

// Specify field
L::logs:(*) {query_string(message, "error AND timeout")} [1h]

// Chinese query
L::logs:(message) {query_string("错误 AND 超时")} [1h]

Outer Functions¶

Usage Recommendation: Outer functions are legacy design. For scenarios that can be solved using Rollup + aggregation functions (e.g., [rate], [last], [increase], etc.), prioritize using the Rollup approach. Outer functions should only be used in scenarios where Rollup is not applicable (e.g., requiring secondary calculation on aggregated results).

Outer functions operate on the entire DQL query result, used for secondary calculation on the time series data output by the query. Outer functions wrap the entire DQL expression, rather than being written inside the Select clause.

In-Query Functions vs Outer Functions¶

• In-Query Functions: Used inside DQL expressions, such as sum, avg, max, etc.
• Outer Functions: Wrap the entire DQL query result, performing post-processing on the output time series.

Writing Comparison:

// Rollup (time clause): First calculate growth rate for each time series, then aggregate
M::http_requests:(sum(request_count)) [rate] BY service

// Outer function: First get query result, then perform secondary calculation
rate(M::http_requests:(sum(request_count)) [1h::1m] BY service)

Syntax:

outer_function(DQL_expression)

Examples:

// In-query function: Average raw data
M::cpu:(avg(usage)) [1h::5m] BY host

// Outer function: Calculate moving average on query result
moving_average(M::cpu:(avg(usage)) [1h::5m] BY host, 5)

Cumulative Calculation¶

cumsum¶

Calculates cumulative sum, computing the accumulated value of all previous points for each point in the time series.

Syntax:

cumsum(DQL_expression)

Examples:

// Calculate cumulative request count
cumsum(M::requests:(sum(count)) [1h::5m] BY service)

Difference and Derivative¶

Recommended Usage: These functions should preferably be used as Rollup functions (e.g., [rate], [deriv]). The outer function form is only used for secondary calculation on aggregated results.

The following functions can be used as outer functions:

Examples:

// Outer function: Calculate derivative on query result
derivative(M::cpu:(avg(usage)) [1h::5m] BY host)

// Recommended: Use Rollup approach
M::cpu:(deriv(usage)) [1h::5m:last] BY host

Moving Calculation¶

moving_average¶

Calculates moving average on query result.

Recommended Usage: Prioritize using the Rollup approach moving_average(field, n). The outer function form is only used for secondary smoothing on aggregated results.

Syntax:

moving_average(DQL_expression, n)

Parameters:

• DQL_expression: DQL query expression
• n: Window size (number of data points)

Examples:

// Outer function: Calculate moving average on query result
moving_average(M::cpu:(avg(usage)) [1h::1m] BY host, 5)

// Recommended: Use Rollup approach
M::cpu:(moving_average(usage, 5)) [1h::1m] BY host

TopN¶

top / bottom¶

Recommended Usage: Prioritize using the Rollup approach top(field, n) or bottom(field, n). The outer function form is only used for secondary filtering on aggregated results.

Gets TopN or BottomN from query result.

Syntax:

top(DQL_expression, n)
bottom(DQL_expression, n)

Examples:

// Outer function: Get TopN from query result
top(M::response_time:(max(duration)) [1h::5m] BY service, 5)

// Recommended: Use Rollup approach
M::response_time:(top(duration, 5)) [1h::5m] BY service

Null Value Filling¶

fill¶

Fills null values in query result.

For detailed explanation, please refer to fill function.

Examples:

// Fill null values with 0
fill(M::cpu:(avg(usage)) [1h::5m] BY host, 0)

// Fill with linear interpolation
fill(M::cpu:(avg(usage)) [1h::5m] BY host, LINEAR)

Other Outer Functions¶

The following functions can also be used as outer functions:

Combined Usage¶

Outer functions can be combined:

// Calculate moving average then round
round(moving_average(M::cpu:(avg(usage)) [1h::1m] BY host, 5))

// Calculate moving average of growth rate
moving_average(rate(M::requests:(sum(count)) [1h::5m] BY service), 3)

eval Expression Calculation¶

eval is a special function that allows expression calculation outside the query, referencing results from multiple subqueries for combined operations.

Syntax¶

eval(expression, name1=(query1), name2=(query2), ..., alias="result_name")

Parameters:

• expression: Mathematical expression, using name.field to reference subquery results
• name=(query): Named subquery
• alias: Result alias (optional)

Note: eval also supports the historical writing name="query", but name=(query) is recommended for better type expression and readability.

How It Works¶

1. Execute all named subqueries
2. Align results of each subquery by time
3. Calculate the expression for each time point
4. Return calculation results

Use Cases¶

• Calculating ratios of multiple metrics (e.g., error rate, utilization rate)
• Comparing metrics across different time periods
• Combining calculation results from multiple data sources

Examples¶

Calculate Error Rate¶

// Calculate error rate = error count / total requests * 100
eval(a / b * 100,
     a=(M::http:(sum(error_count)) [1h] BY service),
     b=(M::http:(sum(request_count)) [1h] BY service),
     alias="error_rate")

Calculate CPU Usage Rate¶

// Usage rate = used / total * 100
eval(used / total * 100,
     used=(M::memory:(sum(used_bytes)) [1h] BY host),
     total=(M::memory:(sum(total_bytes)) [1h] BY host),
     alias="memory_usage_percent")

Calculate Baseline Comparison Growth Rate¶

// Calculate growth rate of current value relative to baseline value
eval(current / baseline - 1,
     current=(M::sales_current:(sum(amount)) [7d]),
     baseline=(M::sales_baseline:(sum(amount)) [7d]),
     alias="growth_rate")

Reference Fields from Subqueries¶

// Reference specific fields from subqueries
eval(a.usage / b.total * 100,
     a=(M::cpu:(avg(usage) as usage) [1h] BY host),
     b=(M::cpu:(avg(total) as total) [1h] BY host),
     alias="cpu_percent")

Notes¶

• Time windows of all subqueries must be compatible
• Grouping dimensions of subqueries should be consistent
• Field names referenced in expressions use name.field format
• If there is only one subquery, field names can be used directly

Other Functions¶

fill¶

Fills specified values for null values in query result.

Recommended Usage: fill is recommended to be used as an outer function, acting on the entire query result:

fill(M::cpu:(avg(usage)) [1h::5m] BY host, 0)

Although writing fill(avg(usage), 0) in the Select clause is also supported, fill actually fills results after aggregation is complete, so the outer function form better matches its working mechanism.

Syntax (outer function):

fill(DQL_expression, value)
fill(DQL_expression, LINEAR)
fill(DQL_expression, PREVIOUS)

Parameters:

• DQL_expression: DQL query expression
• value: Fill value, supports multiple modes:
• Specific value: numeric, string, null
• LINEAR: Linear interpolation
• PREVIOUS: Fill with previous non-null value

Examples:

// Recommended: Use as outer function
fill(M::cpu:(avg(usage)) [1h::5m] BY host, 0)

// Fill with linear interpolation
fill(M::cpu:(avg(usage)) [1h::5m] BY host, LINEAR)

// Fill with previous value
fill(M::cpu:(avg(usage)) [1h::5m] BY host, PREVIOUS)

now¶

Returns current timestamp (milliseconds).

Syntax:

now()

Examples:

// Query recently updated data
O::HOST:(*) {last_update_time > now() - 600000}  // Updated within 10 minutes

unwrap¶

Unwraps aggregation result packaging.

Syntax:

unwrap(field)

Examples:

// Unwrap aggregated field
M::cpu:(unwrap(usage)) [1h] BY host

Show Functions¶

Show functions are used to view metadata (e.g., measurement, tag, field, cardinality and series count), often used for modeling troubleshooting and pre-query exploration.

General Syntax¶

show_xxx(arg1=..., arg2=...){ where_conditions } [time_window] LIMIT n OFFSET m

• where, time_window, LIMIT, OFFSET are all optional.
• LIMIT/OFFSET cannot be negative.

M Namespace Built-in Show Functions¶

Cardinality-related functions use HyperLogLog merging at the underlying layer, return values are estimates.

Non-M Namespace Show Functions (Suffix Pattern)¶

For non-M namespaces, the following suffix patterns are supported:

• show_<namespace>_source
• show_<namespace>_class
• show_<namespace>_type
• show_<namespace>_field
• show_<namespace>_label

Where <namespace> is automatically mapped from the middle segment of the function name, for example:

• show_logging_source -> L
• show_tracing_field -> T
• show_object_source -> O

Common examples:

show_logging_source()
show_tracing_field('mysql')
show_logging_field('*')
show_logging_label(name='env')
show_logging_label(names=['env', 'team'])

Parameter and Behavior Description¶

• from: Measurement list, supports string or string array.
• keyin: Tag key list, supports string or string array.
• field: Field list, supports string or string array (for metric show field filtering).
• For show_*_field:
• Unnamed parameters (e.g., 'mysql') are usually used as source filtering.
• '*' is equivalent to not specifying source.
• Named parameters are converted to where filtering conditions.
• For show_*_label:
• Requires named parameters; names is treated as an alias for name.

Constraints and Notes¶

• show_tag_value and show_tag_value_cardinality must provide keyin.
• show_series_count_by_tag_value must provide keyin.
• show_series_count_by_* must provide from, or provide equivalent source constraints in where (e.g., @__source__ condition).
• show_<namespace>_source, show_<namespace>_class, show_<namespace>_type currently share the same execution path, returning deduplicated source list.
• Currently parser does not support show_<namespace>_index syntax (even though the execution layer has corresponding branches).
• When Query API does not provide show time range, some log show queries fall back to the most recent 30-minute window for execution.

Return Examples¶

The following examples only show typical column structure and sample rows; actual results are affected by tenant data, filtering conditions, time range, and LIMIT/OFFSET.

Function Classification Quick Reference¶

Basic Aggregation¶

Statistical Aggregation¶

Filter Aggregation¶

Histogram Functions¶

Set Functions¶

Helper Functions¶

Time Series Functions¶

Rollup Functions¶

Transformation Functions¶

Matching Functions¶

Outer Functions (Prioritize Rollup)¶

Expression Calculation¶