Skip to content

DQL Function Reference


DQL provides a rich set of functions for data aggregation, transformation, and matching. This document describes the semantics, parameters, and usage of each function in detail.

Aggregation Functions

Aggregation functions aggregate multiple rows of data into a single value, typically used with a time window (time-expr) and grouping (BY clause).

Basic Aggregation

sum

Calculates the sum of field values.

Syntax:

sum(field)

Parameters:

  • field: Numeric field

Examples:

// Calculate total request count
M::http_requests:(sum(request_count)) [1h]

// Calculate total request count per service
M::http_requests:(sum(request_count)) [1h] BY service

avg

Calculates the average of field values.

Syntax:

avg(field)

Parameters:

  • field: Numeric field

Examples:

// Calculate average response time
M::response_time:(avg(duration)) [1h] BY endpoint

// Calculate average CPU usage
M::cpu:(avg(usage)) [1h] BY host

count

Counts the number of data rows.

Syntax:

count(field)
count(*)

Parameters:

  • field: Any field, counts non-null values
  • *: Counts all rows

Examples:

// Count log entries
L::nginx:(count(*)) [1h]

// Count requests with response time
M::response_time:(count(duration)) [1h] BY service

min / max

Calculates the minimum or maximum value of a field.

Syntax:

min(field)
max(field)

Parameters:

  • field: Numeric field

Examples:

// Find the maximum response time
M::response_time:(max(duration)) [1h] BY endpoint

// Find the range of CPU usage
M::cpu:(min(usage), max(usage)) [1h] BY host

first / last

Gets the first or last value (in time order).

Syntax:

first(field)
last(field)
last_row(field)

Parameters:

  • field: Any field

Notes:

  • first: Returns the earliest value
  • last: Returns the latest value; if the field is an array type, it will be expanded
  • last_row: Returns the latest value; array types are not expanded

Examples:

// Get the latest status value
M::system:(last(status)) [1h] BY host

// Get the initial and final values
M::counter:(first(value), last(value)) [1h] BY metric

any

Returns any non-null value. Useful for retrieving sample data when no specific aggregation order is needed.

Syntax:

any(field)

Parameters:

  • field: Any field

Examples:

// Get any message sample
L::logs:(any(message)) [1h] BY service

// Get any error stack trace
L::error_logs:(any(stack_trace)) [1h] BY error_type

SHIFT

Expression-level SHIFT references aggregated values from a historical window in the projection, aligning them with the current aggregation result:

Syntax:

<aggregate-or-subquery-measure> SHIFT <duration>

Current values, historical values, and derived calculations are all explicitly expressed in the projection expression. When referencing the same historical value multiple times, write the same SHIFT expression repeatedly:

Example:

L::logs:(
  service,
  count(*) AS requests,
  count(*) SHIFT 7d AS requests_last_week,
  CASE
    WHEN (count(*) SHIFT 7d) = nil OR (count(*) SHIFT 7d) = 0 THEN nil
    ELSE count(*) / (count(*) SHIFT 7d)
  END AS requests_ratio
)[1d:1h] BY service

duration must be a positive fixed duration, e.g., 1h, 7d.

Usage Restrictions:

  • Can only appear in a projection and only apply to an aggregation expression whose row identity remains stable across windows within the current projection, or to a subquery measure derived from such an aggregation;
  • Cannot be applied to a projection alias, dimension column, WHERE, BY, HAVING, ORDER BY, SORDER BY just declared in the same SELECT, nor can SHIFT be nested;
  • Aggregations whose output rows are determined by each window's data, such as distinct, distinct_by_collapse, field_values, histogram, cannot be used as operands; expressions like uint(field) that only transform raw fields are not aggregation measures, while scalar transformations wrapping stable aggregations like abs(sum(field)) can be used;
  • When filtering or sorting based on historical values, do it in an outer SELECT;
  • All expression-level SHIFT in the same SELECT can contain at most 16 different offsets in total.

Query-level SHIFT (see Time Shift in DQL Main Document) can be combined with expression-level SHIFT: the query-level offset first determines the base window of the entire query, then the expression-level offset reads earlier aggregated values relative to that base window.


spread

Calculates the range (difference between maximum and minimum).

Syntax:

spread(field)

Parameters:

  • field: Numeric field

Examples:

// Calculate the fluctuation range of response time
M::response_time:(spread(duration)) [1h] BY endpoint

stddev

Calculates the standard deviation.

Syntax:

stddev(field)

Parameters:

  • field: Numeric field

Examples:

// Calculate the standard deviation of response time
M::response_time:(stddev(duration)) [1h] BY endpoint

mode

Calculates the mode (the value that appears most frequently).

Syntax:

mode(field)

Parameters:

  • field: Any field

Examples:

// Find the most common response status code
M::http:(mode(status)) [1h] BY endpoint

count_series

Counts the number of time series (groups). Returns how many distinct time series exist within the current query range.

Syntax:

count_series(field)

Parameters:

  • field: Any field (typically * or any existing field)

Examples:

// Count how many hosts are reporting CPU metrics
M::cpu:(count_series(*)) [1h]

// Count how many instances each service has
M::http_requests:(count_series(*)) [1h] BY service

Statistical Aggregation (Estimation Functions)

The following functions use probabilistic data structures for estimation, suitable for large data volumes, balancing accuracy and performance.

count_distinct

Estimates the number of distinct values of a field.

Syntax:

count_distinct(field)

Parameters:

  • field: Any field

Algorithm Notes:

Uses the HyperLogLog algorithm for cardinality estimation: - Number of registers: 2¹⁶ = 65536 - Uses LogLog-Beta estimation method - Standard error: approximately 0.4%

Use Cases:

  • Count unique users (UV)
  • Count distinct IP addresses
  • Analyze unique request IDs

Examples:

// Count unique users
L::access_logs:(count_distinct(user_id)) [1d] BY service

// Count distinct IP addresses
L::nginx:(count_distinct(client_ip)) [1h] BY endpoint

percentile

Estimates the percentile of a field.

Syntax:

percentile(field, n)
pXX(field)  // Shorthand

Parameters:

  • field: Numeric field
  • n: Percentile, range 0-100

Shorthand Forms:

  • p50(field) is equivalent to percentile(field, 50)
  • p95(field) is equivalent to percentile(field, 95)
  • p99(field) is equivalent to percentile(field, 99)

Algorithm Notes:

Uses a log-linear interpolation histogram for estimation: - Bucket range: 10⁻⁹ to 10¹⁸, covering most numeric scenarios - 128 buckets per order of magnitude - Linear interpolation in log space improves accuracy

Use Cases:

  • Calculate P99, P95 of response time
  • Analyze tail latency of performance metrics
  • Evaluate Service Level Agreement (SLA) compliance

Examples:

// Calculate P99 of response time
M::response_time:(percentile(duration, 99)) [1h] BY service

// Using shorthand
M::response_time:(p99(duration)) [1h] BY service

// As a Rollup function call: first compute P95 on each time series, then aggregate by service
M::response_time:(avg(duration)) [1h::5m:percentile(95)] BY service

// Calculate multiple percentiles simultaneously
M::response_time:(p50(duration), p95(duration), p99(duration)) [1h] BY service

median

Calculates the median, equivalent to percentile(field, 50).

Syntax:

median(field)

Examples:

// Calculate the median response time
M::response_time:(median(duration)) [1h] BY service

Histogram Functions

DQL provides three histogram-related functions for different data sources and scenarios:

Function Use Case Data Source Type Recommendation
histogram_auto Value distribution statistics for detail data (logs, Traces) Detail model (Logs/Trace) ⭐⭐⭐ Recommended
histogram Histogram with fixed bucket boundaries Detail model (Logs/Trace) ⭐⭐ Deprecated
histogram_quantile Calculate quantiles from Prometheus histogram metrics Prometheus metrics ⭐⭐⭐ Recommended

Automatically generates a distribution histogram, designed for value distribution statistics of detail data such as logs and Traces.

Features:

  • No need to specify bucket boundaries; automatically adapts to data distribution
  • Uses a log-linear interpolation histogram algorithm, covering the numeric range from 10⁻⁹ to 10¹⁸
  • Returns both quantile statistics and bucket distribution information

Syntax:

histogram_auto(field)

Parameters:

  • field: Numeric field

Return Values: | Column | Description | | ---------------- | ---------------------------- | | lower_bounds | Array of lower bucket bounds | | upper_bounds | Array of upper bucket bounds | | counts | Array of bucket counts | | min | Minimum value | | p50 | 50th percentile | | p75 | 75th percentile | | p90 | 90th percentile | | p95 | 95th percentile | | p99 | 99th percentile | | max | Maximum value |

Algorithm Notes: Uses an estimated histogram (log-linear interpolation) with 128 buckets per order of magnitude, suitable for distribution statistics on large datasets.

Use Cases:

  • Analyze response time distribution in logs
  • Analyze duration distribution in Traces
  • Exploratory data analysis without preset bucket boundaries

Examples:

// Analyze response time distribution in Nginx access logs
L::nginx:(histogram_auto(response_time)) [1h]

// Analyze request duration distribution per service
L::app_logs:(histogram_auto(duration)) [1h] BY service

// Analyze duration distribution of Trace calls
T::http_client:(histogram_auto(elapsed)) [1h] BY operation

Result Example:

lower_bounds upper_bounds counts min p50 p75 p90 p95 p99 max
[0, 10, 100] [10, 100, 1000] [1000, 500, 100] 0.5 45 120 280 450 850 1200

Note: lower_bounds, upper_bounds, counts are array types representing the boundaries and counts of each bucket.


histogram (Deprecated)

Generates a histogram with specified bucket boundaries. This function is Deprecated; use histogram_auto instead.

Notes: histogram requires manually specifying bucket boundary parameters, which is less flexible. histogram_auto automatically adapts to data distribution, covers a wider range of values, and returns richer statistical information.

Syntax:

histogram(field, left_bound, right_bound, bucket_size [, threshold])

Parameters:

  • field: Numeric field
  • left_bound: Left boundary
  • right_bound: Right boundary
  • bucket_size: Bucket size
  • threshold (optional): Minimum count per bucket; buckets below this value are not returned

Return Values: Returns two columns: bucket_le (upper boundary of the bucket) and count (count)

Examples:

// Generate a histogram with range 0-1000ms and 100ms per bucket
M::response_time:(histogram(duration, 0, 1000, 100)) [1h]

// Recommended replacement using histogram_auto
M::response_time:(histogram_auto(duration)) [1h]

Result Example:

bucket_le count
100 1500
200 2800
300 3500
... ...
1000 5000

histogram_quantile

Calculates quantiles from Prometheus histogram metrics.

Features:

  • Specifically designed for processing histogram-type metrics reported by Prometheus
  • Relies on the le label (or vmrange label for VictoriaMetrics) to identify bucket boundaries
  • Input data should be cumulative counts

Syntax:

histogram_quantile(field, q)

Parameters:

  • field: Histogram count field (e.g., http_request_duration_bucket)
  • q: Quantile, range 0-1 (e.g., 0.99 for P99)

Use Case Comparison:

Scenario Recommended Function Notes
Analyze response time distribution in logs histogram_auto Logs are detail data, no pre-aggregated histogram
Calculate P99 of Prometheus histogram metrics histogram_quantile Metrics are pre-aggregated with le label
Analyze duration distribution of Trace calls histogram_auto Traces are detail data

le Label Handling Mechanism:

histogram_quantile relies on the le label (less than or equal) to identify histogram bucket boundaries:

  1. Prometheus format (default): Uses the le label directly to represent the bucket upper boundary
  2. le value is numeric (e.g., "0.1", "1", "10") or "+Inf" (infinity)
  3. Data should be cumulative counts

  4. VictoriaMetrics format: Uses the vmrange label to represent a range

  5. Format: "lower_bound...upper_bound" (e.g., "0.1...0.2")
  6. Data is range counts (non-cumulative)
  7. The function automatically converts range counts to cumulative counts

Calculation Process: 1. Sort all buckets by le value 2. If vmrange format, accumulate counts to convert to cumulative distribution 3. Ensure bucket counts are monotonically increasing (fix potential anomalous data) 4. Use linear interpolation to calculate the target quantile

Differences from PromQL:

Feature DQL PromQL
Function Type Aggregation function Transformation function
Input Data Directly reads metrics with le label Requires sum(rate(...)) by (le)
Usage histogram_quantile(field, 0.99) histogram_quantile(0.99, sum(rate(...)) by (le))
Data Format Supports both le and vmrange labels Only supports le label
Grouping Via DQL's BY clause Explicit grouping via by (le)

Equivalent Examples:

Assume a histogram metric http_request_duration_bucket with le labels (e.g., 0.1, 0.5, 1, 5, +Inf) and service label.

Scenario 1: Calculate P99 latency

DQL:

M::http_request_duration:(histogram_quantile(duration_bucket, 0.99)) [1h] BY service

PromQL equivalent:

histogram_quantile(0.99, sum(rate(http_request_duration_bucket[1h])) by (le, service))

Scenario 2: Calculate P95 latency per service (multiple groups)

DQL:

M::http_request_duration:(
    histogram_quantile(duration_bucket, 0.95)
) [1h] BY service, endpoint

PromQL equivalent:

histogram_quantile(0.95, sum(rate(http_request_duration_bucket[1h])) by (le, service, endpoint))

Scenario 3: Calculate P50 (median) and P99

DQL:

M::http_request_duration:(
    histogram_quantile(duration_bucket, 0.50) as p50,
    histogram_quantile(duration_bucket, 0.99) as p99
) [1h] BY service

PromQL equivalent:

label_join(
  histogram_quantile(0.50, sum(rate(http_request_duration_bucket[1h])) by (le, service)), "quantile", "", "0.50"
)
or
label_join(
  histogram_quantile(0.99, sum(rate(http_request_duration_bucket[1h])) by (le, service)), "quantile", "", "0.99"
)

Note: PromQL requires label_join or label_replace to distinguish results for different quantiles.

Notes:

  • Input data must contain the le or vmrange label; otherwise, calculation is impossible
  • If there is no +Inf bucket, the upper boundary of the last bucket is used as the maximum
  • Buckets with a count of 0 or NaN are skipped
  • When the quantile is < 0, returns -Inf; when > 1, returns +Inf

TopN Functions

top

Gets the top N largest values.

Syntax:

top(field, n)

Parameters:

  • field: Numeric field
  • n: Number of values to return

Examples:

// Get the 5 requests with the longest response time
M::response_time:(top(duration, 5)) [1h] BY service

// Get the 10 hosts with the highest traffic
M::network:(top(bytes, 10)) [1h]

Result Example:

service top(duration, 5)
api 1250
api 1180
api 1050
api 980
api 920

Note: Returns multiple rows, each containing one TopN value.


bottom

Gets the bottom N smallest values.

Syntax:

bottom(field, n)

Parameters:

  • field: Numeric field
  • n: Number of values to return

Examples:

// Get the 5 requests with the shortest response time
M::response_time:(bottom(duration, 5)) [1h] BY service

Result Example:

service bottom(duration, 5)
api 12
api 18
api 25
api 32
api 45

Note: Returns multiple rows, each containing one BottomN value.


Value Collection Functions

distinct

Returns all distinct values of a field.

Syntax:

distinct(field)

Examples:

// Get all distinct status codes
M::http:(distinct(status)) [1h] BY endpoint

Result Example:

endpoint distinct(status)
/api/v1 200
/api/v1 404
/api/v1 500
/health 200

Note: Returns multiple rows, each containing one distinct value.


distinct_by_collapse

Gets distinct values of a field using a collapse strategy, retaining the last value of other fields during deduplication.

Syntax:

distinct_by_collapse(field, [last_field1, last_field2, ...])

Parameters:

  • field: Field to deduplicate by
  • last_fields (optional): List of fields whose last values should be retained

Notes: Unlike distinct, distinct_by_collapse retains information from other related fields (taking the last value) during deduplication, suitable for scenarios where context information needs to be preserved.

Examples:

// Get distinct user IDs, retaining the last access time for each user
L::access_logs:(distinct_by_collapse(user_id, [timestamp])) [1h]

// Get distinct hosts, retaining the last status and message
O::HOST:(distinct_by_collapse(host, [status, message])) [1h]

Result Example:

user_id last(timestamp) last(path)
user001 1704067200000 /checkout
user002 1704067100000 /product
user003 1704067000000 /home

Note: Returns the deduplicated primary field value, along with the last values of the fields specified in last_fields.


collect

Collects all values (including duplicates).

Syntax:

collect(field [, limit])

Parameters:

  • field: Any field
  • limit (optional): Maximum number of values to collect

Examples:

// Collect all response times
M::response_time:(collect(duration)) [1h] BY service

// Collect at most 100 values
M::response_time:(collect(duration, 100)) [1h] BY service

Result Example:

service collect(duration)
api [120, 135, 98, 142, ...]
web [45, 52, 48, 61, ...]

Note: Returns an array type containing all collected values (may include duplicates).


collect_distinct

Collects all distinct values.

Syntax:

collect_distinct(field [, limit])

Parameters:

  • field: Any field
  • limit (optional): Maximum number of values to collect

Examples:

// Collect all distinct error types
L::error_logs:(collect_distinct(error_type)) [1h] BY service

Result Example:

service collect_distinct(error_type)
api ["timeout", "connection refused", "404"]
web ["200", "301", "404"]

Note: Returns an array type containing all deduplicated values.


field_values

Gets all values of a field, returning an array type.

Syntax:

field_values(field)

Examples:

// Get all tag values
M::metrics:(field_values(tags)) [1h] BY metric_name

Result Example:

metric_name field_values(tags)
cpu_usage ["host:A", "env:prod", "team:backend"]
memory_used ["host:B", "env:staging", "team:frontend"]

Note: Returns an array type containing all values of the field.


Filtered Aggregation

count_filter

Counts the number of field values that are in a specified list.

Syntax:

count_filter(field, [value1, value2, ...])

Parameters:

  • field: Any field
  • values: List of values

Examples:

// Count requests with specific status codes
M::http:(count_filter(status, [200, 201, 204])) [1h] BY endpoint

// Count error-level logs
L::logs:(count_filter(level, ["error", "critical"])) [1h] BY service

Helper Functions

default

Sets a default value for a field, returning the default when the field is null.

Syntax:

default(field, default_value)

Parameters:

  • field: Any field
  • default_value: Default value (can be numeric, string, boolean, or null)

Examples:

// Set a default response time for null values
M::response_time:(default(duration, 0)) [1h] BY service

Time Series Functions

Time series functions are used to process data that changes over time, especially Counter-type metrics.

Rollup Functions

Rollup functions are used to preprocess raw time series data within a time window. For detailed information, please refer to Rollup Functions in this document.

Syntax Notes:

  • Rollup is written in the time clause, e.g., [rate], [1h::5m:rate].
  • Written outside the query (e.g., rate(DQL)) is an outer function, not a Rollup.
  • For differences in execution phase and parameters between Rollup shorthand, Rollup function calls, and explicit aggregation calls, please refer to the DQL Main Document's Rollup Functions.

The time clause supports Rollup shorthand and also allows passing additional algorithm parameters to Rollup functions, e.g., [1h::1m:ewma(0.3)]. Parameters in the time clause only represent algorithm parameters; the input field is still determined by the Select field. Functions that require multiple input fields should use explicit aggregation calls in the Select clause.

Functions Supporting Rollup Shorthand:

Function Description
rate Calculates rate of change (per second)
irate Calculates instant rate of change
increase Calculates increase
deriv Calculates derivative (rate of change)
difference Calculates difference
non_negative_derivative Calculates non-negative derivative
non_negative_difference Calculates non-negative difference
rate_over_sum Calculates per-second average
rate_over_count Calculates per-second count
sum Sum
avg Average
min Minimum
max Maximum
count Count
first First value
last Last value
stddev Standard deviation
mode Mode
spread Range
any Any value
slope Linear trend slope
zscore Latest point Z-Score
mad_score Latest point MAD anomaly score
change_score Series change score

Functions Supporting Rollup Function Calls:

Function Description
ewma(alpha) Exponentially Weighted Moving Average
moving_average(n) Moving average
percentile(p) Percentile

Examples:

// Calculate request QPS
M::http_requests:(sum(request_count)) [1h::5m:rate] BY service

// Shorthand
M::cpu:(max(usage)) [rate]

// Rollup function call with algorithm parameter
M::cpu:(avg(usage)) [1h::1m:ewma(0.3)] BY host

Rate Calculation

rate

Calculates the rate of change of a metric (per second).

Syntax:

rate(field)

Notes: rate calculates the average rate of change of a Counter metric within a time window. For monotonically increasing Counter-type metrics, aggregating raw values directly is meaningless; the rate of change must be calculated first.

Use Cases:

  • Calculate request QPS
  • Calculate data ingestion rate
  • Analyze traffic growth trends

Examples:

// Calculate request QPS
M::http_requests:(sum(request_count)) [rate] BY service

// Calculate data ingestion rate
M::data_ingestion:(sum(bytes)) [rate] BY source

irate

Calculates the instant rate of change of a metric.

Syntax:

irate(field)

Notes: Unlike rate, irate only uses the last two data points to calculate the rate of change, reflecting the instantaneous rate of change, making it more suitable for alerting scenarios.

Examples:

// Calculate instant request QPS
M::http_requests:(sum(request_count)) [irate] BY service

increase

Calculates the increase of a metric.

Syntax:

increase(field)

Notes: increase returns the total increase within the time window, not the rate of change.

Examples:

// Calculate total request count increase
M::http_requests:(sum(request_count)) [increase] BY service

rate_over_sum

Calculates the per-second average value (sum / time window seconds).

Syntax:

rate_over_sum(field)

Notes: Equivalent to sum(field) / time_window_seconds, used to calculate the per-second average. Often used in the Rollup phase to convert cumulative values to a per-second rate.

Difference from rate:

  • rate: Calculates the rate of change of a Counter (handles resets)
  • rate_over_sum: Simply divides the sum by the time window seconds

Examples:

// Calculate average requests per second
M::http_requests:(rate_over_sum(request_count)) [1h] BY service

rate_over_count

Calculates the per-second count (count / time window seconds).

Syntax:

rate_over_count(field)

Notes: Equivalent to count(field) / time_window_seconds, used to calculate the number of occurrences per second.

Examples:

// Calculate error log entries per second
L::error_logs:(rate_over_count(*)) [1h] BY error_type

Difference Calculation

This section describes function semantics. The same function can be used both as a Rollup (e.g., [rate], [increase]) and in a query expression (e.g., rate(field), increase(field)); the execution phase differs, so choose the position based on business requirements.

rate / deriv

Calculates the rate of change (derivative). rate is used for Counter-type metrics (ignores negative values), while deriv is used for Gauge-type metrics (preserves negative values).

Aliases: rate is aliased as non_negative_derivative; deriv is aliased as derivative (PromQL style)

Syntax:

// Counter metric: calculate non-negative rate of change (ignores negative values due to resets)
rate(field)

// Gauge metric: calculate full rate of change (includes negative values)
deriv(field)

Function Selection:

Function Description Use Case
rate Calculates only non-negative rate of change Counter-type metrics (monotonically increasing)
deriv Calculates full rate of change (including negative values) Gauge-type metrics (can increase or decrease)

Examples:

// Counter metric: calculate request QPS
M::requests:(rate(count)) [1h::5m] BY service

// Gauge metric: calculate memory usage rate of change
M::memory:(deriv(used)) [1h::5m] BY host

increase / difference

Calculates the difference between consecutive values. increase is used for Counter-type metrics (ignores negative values), while difference is used for Gauge-type metrics (preserves negative values).

Note: increase and difference are two independent functions with different behaviors; they are not aliases.

Syntax:

// Counter metric: calculate non-negative difference (ignores negative values due to resets)
increase(field)

// Gauge metric: calculate full difference (includes negative values)
difference(field)

Function Selection:

Function Description Use Case
increase Calculates only non-negative difference Counter-type metrics (monotonically increasing)
difference Calculates full difference (including negative values) Gauge-type metrics (can increase or decrease)

Examples:

// Counter metric: calculate request increase
M::requests:(increase(count)) [1h::5m] BY service

// Gauge metric: calculate request count change (may increase or decrease)
M::requests:(difference(count)) [1h::5m] BY service

Moving Calculations

moving_average

Calculates the moving average.

Syntax:

moving_average(field, n)

Parameters:

  • field: Numeric field
  • n: Window size (number of data points)

Examples:

// Calculate 5-point moving average
M::cpu:(moving_average(usage, 5)) [1h::1m] BY host

// As a Rollup function call: first compute 5-point moving average on each time series, then aggregate by host
M::cpu:(avg(usage)) [1h::1m:moving_average(5)] BY host

Time Series Analysis Aggregation

The following functions are used as aggregation functions within a query, computing a single numeric result from the value sequence in each time window and group.

ewma

Calculates the Exponentially Weighted Moving Average (EWMA). alpha is the smoothing coefficient and must be explicitly provided.

Syntax:

ewma(field, alpha)

Parameters:

  • field: Numeric field
  • alpha: Smoothing coefficient, range (0, 1]; larger values give higher weight to recent data points

Notes:

  • ewma has no implicit default alpha; ewma(field) will result in an error.
  • When used as a Rollup in the time clause, write it as [...:ewma(alpha)]; parameters in the time clause only pass alpha, not the field name.
  • ewma requires alpha, so it does not support the Rollup shorthand [...:ewma] without parameters.

Examples:

// Explicit aggregation call: compute EWMA in the Select aggregation phase
M::cpu:(ewma(usage, 0.3)) [1h::1m] BY host

// Rollup function call: first compute EWMA on each time series, then aggregate by host
M::cpu:(avg(usage)) [1h::1m:ewma(0.3)] BY host

slope

Calculates the linear trend slope of the sequence over time, with time in seconds.

Syntax:

slope(field)

Notes:

  • At least 2 valid points are required.
  • Returns null when there is no time change or insufficient valid points.

Examples:

// Calculate the growth trend of memory usage
M::memory:(slope(used)) [1h::5m] BY host

zscore

Calculates the Z-Score of the latest point relative to the mean and standard deviation within the window.

Syntax:

zscore(field)

Notes:

  • Result: (latest - mean) / stddev.
  • At least 2 valid points are required; returns null when the standard deviation is 0.

Examples:

// Evaluate the deviation of the latest response time relative to the historical window
M::response_time:(zscore(duration)) [1h::5m] BY service

mad_score

Calculates the MAD (Median Absolute Deviation) anomaly score of the latest point.

Syntax:

mad_score(field)

Notes:

  • Uses the median and MAD to measure the deviation of the latest point, which is more robust to outliers than mean/standard deviation.
  • At least 2 valid points are required; returns null when the MAD is 0.

Examples:

// Detect an anomaly in the latest latency point using a robust anomaly score
M::latency:(mad_score(p95)) [1h::5m] BY service

change_score

Calculates the change score of a sequence within a window, used to detect time series where the mean has significantly shifted.

Syntax:

change_score(field)

Notes:

  • Enumerates possible split points, compares the mean difference between the left and right segments, and normalizes by the pooled standard deviation.
  • At least 4 valid points are required; returns null when insufficient.

Examples:

// Detect whether the error rate has changed abruptly within the window
M::error_rate:(change_score(value)) [1h::5m] BY service

corr

Calculates the Pearson correlation coefficient between two numeric fields.

Syntax:

corr(left_field, right_field)

Notes:

  • Return value range is typically [-1, 1].
  • At least 2 pairs of valid points are required; returns null when either field has no variation.
  • corr requires two input fields and does not support the time clause Rollup syntax.

Examples:

// Calculate the correlation between CPU usage and request count
M::service_metric:(corr(cpu_usage, request_count)) [1h::5m] BY service

cumsum

Calculates the cumulative sum.

Syntax:

cumsum(DQL_expression)

Examples:

// Calculate cumulative request count
cumsum(M::requests:(sum(count)) [1h::5m] BY service)

Transformation Functions

Transformation functions are used to perform mathematical operations, type conversions, or string processing on field values.

Mathematical Functions

abs

Calculates the absolute value.

Syntax:

abs(field)

Examples:

// Calculate the absolute temperature deviation
M::temperature:(abs(deviation)) [1h] BY sensor

round / ceil / floor

Rounding functions.

Syntax:

round(field)          // Round to nearest integer
round(field, digits)  // Round to specified decimal places
ceil(field)           // Round up
floor(field)          // Round down

Examples:

// Round up response time
M::response_time:(ceil(duration)) [1h]

// Round percentage
M::cpu:(round(usage)) [1h] BY host

// Round average response time to 2 decimal places
L::log:(round(avg(duration), 2)) BY api

log / log2 / log10

Logarithmic functions.

Syntax:

log(field)    // Natural logarithm
log2(field)   // Base 2 logarithm
log10(field)  // Base 10 logarithm

Examples:

// Calculate the log-transformed value
M::metrics:(log(value)) [1h] BY metric_name

Type Conversion

int / uint / float / string / bool

Type conversion functions.

Syntax:

int(field)    // Convert to signed integer
uint(field)   // Convert to unsigned integer
float(field)  // Convert to floating point
string(field) // Convert to string
bool(field)   // Convert to boolean

Examples:

// Convert string to numeric
L::logs:(int(response_time)) [1h] BY service

// Convert numeric to string for concatenation
M::metrics:(string(value)) [1h] BY metric_name

String Functions

lower

Converts a string to lowercase.

Syntax:

lower(field)

Parameters:

  • field: String field

Return Value: Returns the converted lowercase string.


upper

Converts a string to uppercase.

Syntax:

upper(field)

Parameters:

  • field: String field

Return Value: Returns the converted uppercase string.


trim

Removes leading and trailing whitespace from a string.

Syntax:

trim(field)

Parameters:

  • field: String field

Return Value: Returns the string with leading and trailing whitespace removed.


ltrim

Removes leading whitespace from a string.

Syntax:

ltrim(field)

Parameters:

  • field: String field

Return Value: Returns the string with leading whitespace removed.


rtrim

Removes trailing whitespace from a string.

Syntax:

rtrim(field)

Parameters:

  • field: String field

Return Value: Returns the string with trailing whitespace removed.


length

Returns the length of a string (in characters).

Syntax:

length(field)

Parameters:

  • field: String field

Return Value: Returns the string length.


substr

Extracts a substring.

Syntax:

substr(field, start)
substr(field, start, length)

Parameters:

  • field: String field
  • start: Starting position (0-based, negative values indicate position from the end)
  • length (optional): Substring length

Return Value: Returns the extracted substring.

Examples:

// Extract the first 100 characters of a message
L::logs:(substr(message, 0, 100)) [1h]

// Extract the last 10 characters
L::logs:(substr(message, -10)) [1h]

Result Example:

message substr(message, 0, 10) substr(message, -5)
"Error: connection timeout" "Error: con" "eout"

regexp_extract

Regular expression extraction.

Syntax:

regexp_extract(field, pattern)
regexp_extract(field, pattern, n)

Parameters:

  • field: String field
  • pattern: Regular expression
  • n (optional): Extract the nth capturing group, default is 0 (the entire match)

Return Value: Returns a single string containing the content of the nth capturing group. Returns null if no match.

Examples:

// Extract error code
L::logs:(regexp_extract(message, 'error_code: (\d+)', 1)) [1h]

// Extract IP address
L::nginx:(regexp_extract(message, '(\d+\.\d+\.\d+\.\d+)', 1)) [1h]

Result Example:

service regexp_extract(message, 'error_code: (\d+)', 1)
api "404"
api "500"
web null

regexp_extract_all

Extracts all matching results.

Syntax:

regexp_extract_all(field, pattern)
regexp_extract_all(field, pattern, n)

Return Value: Returns a string array containing all matching substrings.

Examples:

// Extract all numbers
L::logs:(regexp_extract_all(message, '\d+', 0)) [1h]

// Extract all IP addresses
L::logs:(regexp_extract_all(message, '\d+\.\d+\.\d+\.\d+', 0)) [1h]

Result Example:

message regexp_extract_all(message, '\d+.\d+.\d+.\d+', 0)
Request from 192.168.1.1 to 10.0.0.1 ["192.168.1.1", "10.0.0.1"]

regexp_replace

Replaces matched text using a regular expression.

Syntax:

regexp_replace(field, pattern, replacement)

Parameters:

  • field: String field
  • pattern: Regular expression
  • replacement: Replacement string; use $1, ${1}, $2, etc. to reference capturing groups; use ${1} when the capturing group is immediately followed by a letter, digit, or underscore for disambiguation (e.g., ${1}_suffix); use $$ to output a literal $

Return Value: Returns the replaced string. If no match, returns the original string.

Examples:

// Remove numbers from a message, then aggregate by prefix
L::logs:(substr(regexp_replace(message, '\\d+', ''), 0, 20) AS prefix, count(*)) [1h] BY prefix

// Normalize user IDs using capturing groups
L::logs:(regexp_replace(message, 'user=([0-9]+)', 'uid=$1') AS normalized) [1h]

md5

Calculates the MD5 hash.

Syntax:

md5(field)

Return Value: Returns a 32-character hexadecimal string (lowercase).

Examples:

// Calculate MD5 of a message
L::logs:(md5(message)) [1h]

Result Example:

service md5(message)
api 5d41402abc4b2a76b9719d911017c592
web 098f6bcd4621d373cade4e832627b4f6

concat

String concatenation.

Syntax:

concat(field, ...)

Return Value: Returns a single concatenated string.

Examples:

// Concatenate multiple fields
L::logs:(concat(service, ":", level)) [1h]

// Result: "api:error", "web:info", etc.

Result Example:

service level concat(service, ":", level)
api error "api:error"
web info "web:info"

set

Deduplicates and sorts an array field.

Syntax:

set(DQL_expression)

Return Value: Returns a deduplicated and sorted array.

Examples:

// Get all distinct tags
set(M::metrics:(tags) [1h] BY metric_name)

// Deduplicate the result of collect
set(M::http:(collect(status)) [1h] BY endpoint)

Result Example:

metric_name set(tags)
cpu_usage ["env:prod", "host:A", "team:backend"]
memory_used ["env:staging", "host:B", "team:frontend"]

Log Clustering

drain

Uses the Drain algorithm to group similar logs into the same cluster, returning a representative log sample for that cluster.

Syntax:

drain(field, similarity_threshold)
drain(field, similarity_threshold, max_clusters)

Parameters:

  • field: String field or string expression, typically message
  • similarity_threshold: Similarity threshold, range (0, 1]; larger values mean only more similar logs are grouped together
  • max_clusters: Optional, maximum number of clusters, range [1, 10000]; default is 1000 when omitted

Return Value:

Returns a string representative log sample. The sample is the original log at the time of cluster creation, not the generalized template maintained internally by Drain.

Algorithm Notes:

Drain is a log clustering algorithm based on a parse tree. The function continuously trains the clusterer during execution: when a new log matches an existing cluster, it returns that cluster's representative sample; when no match is found, it creates a new cluster and uses the current log as its representative sample.

Use Cases:

  • Group statistics by similar logs
  • Anomaly log clustering analysis
  • Log noise reduction

Examples:

// Group and count by similar logs, similarity 0.7, max 1000 clusters
L::logs:(count(*)) [1h] BY drain(message, 0.7, 1000) AS sample

// Omit max_clusters, default max 1000 clusters
L::logs:(count(*)) [1h] BY drain(message, 0.9) AS sample

// Concatenate multiple fields before clustering
L::logs:(count(*)) [1h] BY drain(concat(service, " ", message), 0.7) AS sample

Result Example:

sample count(*)
"Request from 192.168.1.1 to /api/users took 35 ms" 128
"Query SELECT * FROM orders executed in 18 ms" 42

Matching Functions

Matching functions are used for text matching in WHERE clauses and can also be used as expressions that return Boolean values.

Substring Matching

match

Checks if a field contains a specified substring.

Syntax:

match(pattern)
match(field, pattern)

Parameters:

  • pattern: Substring to match
  • field: Field name (optional; can be omitted in WHERE clause)

Examples:

// Use in WHERE
L::logs:(message) {match(message, "error")} [1h]

// Shorthand
L::logs:(message) {match("error")} [1h]

// As an expression
L::logs:(match(message, "timeout")) [1h] BY match_result

Phrase Matching

Tokenized phrase matching, supporting mixed Chinese and English.

Syntax:

search(query)
search(field, query)

Parameters:

  • query: Query phrase
  • field: Field name (optional)

Matching Rules:

  • Chinese: Tokens are segmented by character
  • English: Matches by word boundaries (separated by spaces, punctuation)
  • Case-insensitive
  • When a non-empty query consists of exactly one pair of double quotes enclosing the entire value, it is treated as a phrase match: all tokens must be adjacent and in the same order; the double quotes are not part of the match content
  • In the above special double-quote form, if the content is only whitespace or punctuation, it still matches as a continuous literal value
  • An empty query "" is not considered a special form; it is treated as a normal tokenized search parameter
  • Other forms containing double quotes are valid regular tokenized search parameters and are not interpreted specially, e.g., only one double quote, extra double quotes inside the value, or double quotes not at both ends

Examples:

// Match logs containing "connection timeout"
L::logs:(message) {search("connection timeout")} [1h]

// Chinese matching
L::logs:(message) {search("连接超时")} [1h]

// Mixed Chinese and English
L::logs:(message) {search("error 错误")} [1h]

// Adjacent and ordered phrase matching; will not match "build succeeded, parse error"
L::logs:(message) {search("\"build error\"")} [1h]

// Only punctuation, matches as continuous literal substring
L::logs:(message) {search("\"---\"")} [1h]

Regular Expression Matching

re / regex / regexp

Regular expression matching, case-sensitive.

Syntax:

re(pattern)
re(field, pattern)
regex(pattern)
regex(field, pattern)
regexp(pattern)
regexp(field, pattern)

Parameters:

  • pattern: Regular expression (supports PromRegex syntax)
  • field: Field name (optional)

Examples:

// Match logs starting with "error", will not match logs starting with "ERROR"
L::logs:(message) {re("error.*")} [1h]

// Match a specific error code format
L::logs:(message) {regexp(message, "ERR-\d{4}")} [1h]

// Use regex in data source
M::re('cpu.*'):(usage) [1h]

Wildcard Matching

wildcard

Wildcard pattern matching, case-sensitive.

Syntax:

wildcard(pattern)
wildcard(field, pattern)

Parameters:

  • pattern: Wildcard pattern
  • *: Matches any number of characters
  • ?: Matches a single character
  • field: Field name (optional)

Examples:

// Match messages starting with "error", will not match messages starting with "ERROR"
L::logs:(message) {wildcard("error*")} [1h]

// Match a specific format
L::logs:(message) {wildcard(message, "ERR-????")} [1h]

CIDR Matching

cidr

IP address CIDR matching.

Syntax:

cidr(cidr)
cidr(field, cidr)

Parameters:

  • cidr: CIDR notation network, e.g., 192.168.1.0/24
  • field: IP address field (optional)

Examples:

// Match internal IP addresses
L::nginx:(*) {cidr(client_ip, "10.0.0.0/8")} [1h]

// Match a specific network segment
L::nginx:(*) {cidr(client_ip, "192.168.1.0/24")} [1h]

Field Existence Check

exists

Checks if a field exists. exists() is a special placeholder typically used on the right side of a comparison expression.

Syntax:

exists()

Recommended Usage:

field = exists()   // Field exists (non-null)
field != exists()  // Field does not exist or is null

Examples:

// Find logs with an error_type field
L::logs:(message) {error_type = exists()} [1h]

// Find logs without an error_type field
L::logs:(message) {error_type != exists()} [1h]

Query String Syntax

query_string

Uses query string syntax for complex matching.

Syntax:

query_string(query)
query_string(field, query)

Parameters:

  • query: Query string
  • field: Field name (optional; defaults to full text search)

Query String Syntax:

1. Term Matching
foo              # Match content containing foo
"foo bar"        # Exact phrase match, must appear consecutively
foo\ bar         # Escape space, match "foo bar" as a whole
2. Wildcards
foo*             # Match content starting with foo
foo?bar          # ? matches a single character
"foo*bar"        # Wildcards inside quotes are not parsed, matched as literals
3. Regular Expressions
/foo.*bar/       # Regular expression enclosed in slashes
/joh?n(ath[oa]n)/  # Complex regex
4. Boolean Operators
foo AND bar      # Logical AND, both must be present
foo OR bar       # Logical OR, at least one must be present
NOT foo          # Logical NOT, must not contain foo

# Shorthand
foo && bar       # Equivalent to foo AND bar
foo || bar       # Equivalent to foo OR bar
!foo             # Equivalent to NOT foo
5. Grouping
(foo OR bar) AND baz       # Use parentheses to change precedence
!(status 429 reading)      # Negate the entire expression
6. Default Operator

When multiple terms are separated by spaces, the default operator is OR (configurable to AND):

foo bar          # Equivalent to foo OR bar
7. Tokenization Rules

The behavior of the queryString function depends on the underlying storage engine used by the current workspace:

  • ScopeDB environment: queryString performs a case-insensitive contains search, i.e., it checks whether the field value contains any substring of the query string, without tokenization;

  • Doris environment: queryString matches based on full-text index tokenization, and its semantics are directly related to the tokenization result. The tokenizer used by Doris follows the default word boundary specification of Unicode Standard Annex #29.

Examples
// Simple term matching
L::logs:(message) {query_string("error timeout")} [1h]

// Boolean combination
L::logs:(message) {query_string("error AND NOT timeout")} [1h]

// Regular expression
L::logs:(message) {query_string("/ERR-\d{4}/")} [1h]

// Complex query
L::logs:(message) {query_string("(error OR warn) AND service")} [1h]

// Specify field
L::logs:(*) {query_string(message, "error AND timeout")} [1h]

// Chinese query
L::logs:(message) {query_string("错误 AND 超时")} [1h]

Outer Functions

Usage Recommendation: Outer functions are a legacy design. For scenarios that can be solved with Rollup + aggregation functions (e.g., [rate], [last], [increase], etc.), prefer the Rollup approach. Outer functions should only be used in scenarios where Rollup is not applicable (e.g., when secondary calculations on aggregation results are required). Functions like dbscan, forecast that require center detection/prediction over the entire query result have no equivalent Rollup syntax and should be used as outer functions.

Outer functions operate on the entire DQL query result, performing secondary calculations on the output time series data. Outer functions wrap the entire DQL expression, rather than being written inside the Select clause.

Query-Internal Functions vs. Outer Functions

  • Query-Internal Functions: Used inside a DQL expression, e.g., sum, avg, max, etc.
  • Outer Functions: Wrap the entire DQL query result, performing post-processing on the output time series.

Syntax Comparison:

// Rollup (time clause): first compute rate of change on each time series, then aggregate
M::http_requests:(sum(request_count)) [rate] BY service

// Outer function: first get the query result, then perform secondary calculation
rate(M::http_requests:(sum(request_count)) [1h::1m] BY service)

Syntax:

outer_function(DQL_expression)

Examples:

// Query-internal function: average the raw data
M::cpu:(avg(usage)) [1h::5m] BY host

// Outer function: calculate moving average on the query result
moving_average(M::cpu:(avg(usage)) [1h::5m] BY host, 5)

Cumulative Calculations

cumsum

Calculates the cumulative sum, summing all previous points for each point in the time series.

Syntax:

cumsum(DQL_expression)

Examples:

// Calculate cumulative request count
cumsum(M::requests:(sum(count)) [1h::5m] BY service)

Difference and Derivative

Recommended Use: These functions should preferably be used as Rollup functions (e.g., [rate], [deriv]); the outer function form is only for secondary calculations on aggregation results.

The following functions can be used as outer functions:

Function Description
derivative(DQL) Calculates derivative (rate of change)
difference(DQL) Calculates difference from previous value
non_negative_derivative(DQL) Calculates non-negative derivative
non_negative_difference(DQL) Calculates non-negative difference
rate(DQL) Calculates rate of change (per second)
irate(DQL) Calculates instant rate of change

Examples:

// Outer function: calculate derivative on the query result
derivative(M::cpu:(avg(usage)) [1h::5m] BY host)

// Recommended: use Rollup approach
M::cpu:(deriv(usage)) [1h::5m:last] BY host

Moving Calculations

moving_average

Calculates the moving average on the query result.

Recommended Use: Prefer the Rollup approach moving_average(field, n); the outer function form is only for secondary smoothing of aggregation results.

Syntax:

moving_average(DQL_expression, n)

Parameters:

  • DQL_expression: DQL query expression
  • n: Window size (number of data points)

Examples:

// Outer function: calculate moving average on the query result
moving_average(M::cpu:(avg(usage)) [1h::1m] BY host, 5)

// Recommended: use Rollup approach
M::cpu:(moving_average(usage, 5)) [1h::1m] BY host

Time Series Analysis and Anomaly Detection

dbscan

Performs DBSCAN outlier detection on the numeric time series in the query result, outputting numeric anomaly flags.

Syntax:

dbscan(DQL_expression)
dbscan(DQL_expression, eps)

Parameters:

  • DQL_expression: DQL query expression. Typically used for multi-series query results with time windows and BY grouping.
  • eps: Neighborhood distance threshold, optional, default 0.5, range (0, 3.0].

Return Value:

  • Outputs a corresponding dbscan(column) numeric column for each numeric column.
  • 1 indicates an outlier point, 0 indicates a non-outlier point.
  • Returns null when there are fewer than 5 valid numeric points, indicating insufficient samples for detection.

Notes:

  • At least 5 valid numeric points are required; returns null when insufficient.
  • The current implementation performs one-dimensional DBSCAN on each numeric column in the input table separately, preserving the time column.

Examples:

// Detect CPU usage outliers with default eps=0.5
dbscan(M::cpu:(avg(usage)) [1h::5m] BY host)

// Explicitly specify eps
dbscan(M::cpu:(avg(usage)) [1h::5m] BY host, 0.8)

forecast

Performs linear trend forecasting on the numeric time series in the query result, outputting predicted values for future time points.

Syntax:

forecast(DQL_expression)
forecast(DQL_expression, steps)

Parameters:

  • DQL_expression: DQL query expression
  • steps: Number of forecast steps, optional, default 5, must be a positive integer

Return Value:

  • Outputs future steps time points.
  • Each numeric column outputs a corresponding forecast(column) numeric column.
  • Non-numeric columns do not participate in forecasting.

Notes:

  • The current implementation uses linear trend forecasting.
  • Returns null when there are fewer than 2 valid numeric points.

Examples:

// Forecast the next 5 time points
forecast(M::cpu:(avg(usage)) [1h::5m] BY host)

// Forecast the next 3 time points
forecast(M::cpu:(avg(usage)) [1h::5m] BY host, 3)

TopN

top / bottom

Recommended Use: Prefer the Rollup approach top(field, n) or bottom(field, n); the outer function form is only for secondary filtering of aggregation results.

Gets the TopN or BottomN of the query result.

Syntax:

top(DQL_expression, n)
bottom(DQL_expression, n)

Examples:

// Outer function: get TopN on the query result
top(M::response_time:(max(duration)) [1h::5m] BY service, 5)

// Recommended: use Rollup approach
M::response_time:(top(duration, 5)) [1h::5m] BY service

Null Value Filling

fill

Fills null values in the query result.

For detailed information, please refer to fill function.

Examples:

// Fill null values with 0
fill(M::cpu:(avg(usage)) [1h::5m] BY host, 0)

// Linear interpolation fill
fill(M::cpu:(avg(usage)) [1h::5m] BY host, LINEAR)

Other Outer Functions

The following functions can also be used as outer functions:

Function Description
abs(DQL) Absolute value
round(DQL[, digits]) Round
ceil(DQL) Round up
floor(DQL) Round down
log(DQL) / log2(DQL) / log10(DQL) Logarithmic transformation
set(DQL) Deduplicate and sort
concat(DQL, ...) String concatenation

Combined Usage

Outer functions can be combined:

// Calculate moving average and round
round(moving_average(M::cpu:(avg(usage)) [1h::1m] BY host, 5))

// Calculate moving average of the rate
moving_average(rate(M::requests:(sum(count)) [1h::5m] BY service), 3)

eval Expression Calculation

eval is a special function that allows expression calculation outside the query, referencing results from multiple sub-queries for combined operations.

Syntax

eval(expression, name1=(query1), name2=(query2), ..., alias="result_name")

Parameters:

  • expression: Mathematical expression, using name.field to reference sub-query results
  • name=(query): Named sub-query
  • alias: Result alias (optional)

Note: eval also supports the legacy syntax name="query", but the name=(query) syntax is recommended for better type expression and readability.

How It Works

  1. Executes all named sub-queries
  2. Aligns the results of each sub-query by time
  3. Evaluates the expression for each time point
  4. Returns the calculation result

Use Cases

  • Calculate ratios of multiple metrics (e.g., error rate, utilization)
  • Compare metrics across different time periods
  • Combine calculation results from multiple data sources

Examples

Calculate Error Rate

// Calculate error rate = error_count / total_requests * 100
eval(a / b * 100,
     a=(M::http:(sum(error_count)) [1h] BY service),
     b=(M::http:(sum(request_count)) [1h] BY service),
     alias="error_rate")

Calculate CPU Usage

// Usage = used / total * 100
eval(used / total * 100,
     used=(M::memory:(sum(used_bytes)) [1h] BY host),
     total=(M::memory:(sum(total_bytes)) [1h] BY host),
     alias="memory_usage_percent")

Calculate Baseline Comparison Growth Rate

// Calculate growth rate of current value relative to baseline
eval(current / baseline - 1,
     current=(M::sales_current:(sum(amount)) [7d]),
     baseline=(M::sales_baseline:(sum(amount)) [7d]),
     alias="growth_rate")

Reference Sub-query Fields

// Reference specific fields from sub-queries
eval(a.usage / b.total * 100,
     a=(M::cpu:(avg(usage) as usage) [1h] BY host),
     b=(M::cpu:(avg(total) as total) [1h] BY host),
     alias="cpu_percent")

Notes

  • All sub-query time windows must be compatible
  • Grouping dimensions of sub-queries should be consistent
  • Fields referenced in expressions use the name.field format
  • If there is only one sub-query, you can directly use the field name

Other Functions

fill

Fills null values in the query result with a specified value.

Recommended Usage: fill is recommended to be used as an outer function, operating on the entire query result:

fill(M::cpu:(avg(usage)) [1h::5m] BY host, 0)

Although the syntax fill(avg(usage), 0) is also supported in the Select clause, fill actually fills the result after aggregation is complete, so the outer function form better matches its working mechanism.

Syntax (Outer Function):

fill(DQL_expression, value)
fill(DQL_expression, LINEAR)
fill(DQL_expression, PREVIOUS)

Parameters:

  • DQL_expression: DQL query expression
  • value: Fill value, supports multiple modes:
  • Specific value: numeric, string, null
  • LINEAR: Linear interpolation
  • PREVIOUS: Fill with the previous non-null value

Examples:

// Recommended: use as an outer function
fill(M::cpu:(avg(usage)) [1h::5m] BY host, 0)

// Linear interpolation fill
fill(M::cpu:(avg(usage)) [1h::5m] BY host, LINEAR)

// Fill with previous value
fill(M::cpu:(avg(usage)) [1h::5m] BY host, PREVIOUS)

now

Returns the current timestamp (in milliseconds).

Syntax:

now()

Examples:

// Query recently updated data
O::HOST:(*) {last_update_time > now() - 600000}  // Updated within the last 10 minutes

unwrap

Unwraps the wrapper of an aggregation result.

Syntax:

unwrap(field)

Examples:

// Unwrap aggregated field
M::cpu:(unwrap(usage)) [1h] BY host

Show Functions

Show functions are used to view metadata (such as measurements, tags, fields, cardinality, and series count), commonly used for modeling troubleshooting and pre-query exploration.

General Syntax

show_xxx(arg1=..., arg2=...){ where_conditions } [time_window] LIMIT n OFFSET m
  • where, time_window, LIMIT, OFFSET are all optional.
  • LIMIT/OFFSET cannot be negative.

Built-in Show Functions in M Namespace

Function Parameters Return Columns Description
show_measurement Optional re('pattern') name List measurements
show_tag_key Optional from=['measurement'] tagKey List tag keys
show_field_key Optional from=['measurement'] fieldKey, fieldType List field keys (current fieldType is float)
show_tag_value keyin=['tagKey'] (required), optional from key, value List tag values
show_measurement_cardinality No mandatory parameters count Number of measurements
show_series_cardinality No mandatory parameters count Series cardinality (estimated)
show_tag_key_cardinality No mandatory parameters count Tag key cardinality (estimated)
show_tag_value_cardinality keyin=['tagKey'] (required) count Value cardinality of a specified tag key (estimated)
show_field_key_cardinality No mandatory parameters count Field key cardinality (estimated)
show_series_count_by_field_key from=['measurement'] (recommended) name, count Series count by field key
show_series_count_by_tag_key from=['measurement'] (recommended) name, count, value_count Series count and value count by tag key
show_series_count_by_tag_value keyin=['tagKey'] (required), from=['measurement'] (recommended) name, count Series count by value of a specified tag key

Cardinality-related functions use HyperLogLog merge under the hood, returning estimated values.

Show Functions for Non-M Namespaces (Suffix Pattern)

For non-M namespaces, the following suffix patterns are supported:

  • show_<namespace>_source
  • show_<namespace>_class
  • show_<namespace>_type
  • show_<namespace>_field
  • show_<namespace>_label

The <namespace> is automatically mapped from the middle segment of the function name, for example:

  • show_logging_source -> L
  • show_tracing_field -> T
  • show_object_source -> O

Common examples:

show_logging_source()
show_tracing_field('mysql')
show_logging_field('*')
show_logging_label(name='env')
show_logging_label(names=['env', 'team'])

Parameters and Behavior Notes

  • from: List of measurements, supports a string or an array of strings.
  • keyin: List of tag keys, supports a string or an array of strings.
  • field: List of fields, supports a string or an array of strings (used for metric show field filtering).
  • For show_*_field:
  • An unnamed parameter (e.g., 'mysql') is typically used as a source filter.
  • '*' is equivalent to not specifying a source.
  • Named parameters are converted to where filter conditions.
  • For show_*_label:
  • Requires named parameters; names is treated as an alias for name.

Constraints and Notes

  • show_tag_value and show_tag_value_cardinality must provide keyin.
  • show_series_count_by_tag_value must provide keyin.
  • show_series_count_by_* must provide from, or an equivalent source constraint in the where clause (e.g., @__source__ condition).
  • show_<namespace>_source, show_<namespace>_class, show_<namespace>_type currently share the same execution path, returning a deduplicated list of sources.
  • The parser currently does not support the show_<namespace>_index syntax (even if the execution layer has a corresponding branch).
  • When the Query API does not provide a show time range, some log show queries will fall back to the last 30-minute window.

Return Examples

The following examples only show typical column structures and sample rows; actual results will be affected by tenant data, filter conditions, time range, and LIMIT/OFFSET.

Query Typical Columns Sample Rows (Illustrative)
show_measurement() name cpu, disk, memory
show_tag_value(from=['cpu'], keyin=['host']) key, value host, web-01; host, web-02
show_series_count_by_tag_key(from=['cpu']) name, count, value_count host, 3200, 120; service, 2800, 35
show_tag_value_cardinality(keyin=['host']) count 120
show_logging_field('*') fieldKey, fieldType, fieldIndices service, keyword, ["idx_service"]
show_logging_source() source nginx, mysql, redis

Function Category Quick Reference Table

Basic Aggregation

Function Description Exact/Estimate
sum Sum Exact
avg Average Exact
count Count Exact
count_distinct Distinct count Estimate (HyperLogLog, error ≈ 0.4%)
min / max Minimum / Maximum Exact
first / last First / Last value Exact
any Any value Exact

Statistical Aggregation

Function Description Exact/Estimate
percentile / pXX Percentile Estimate (log histogram)
median Median Estimate
stddev Standard deviation Exact
mode Mode Exact
spread Range Exact
count_series Time series count Exact

Time Series Analysis Aggregation

Function Description Notes
ewma Exponentially Weighted Moving Average Supports [...:ewma(alpha)]
slope Linear trend slope Supports Rollup shorthand
zscore Latest point Z-Score Supports Rollup shorthand
mad_score Latest point MAD anomaly score Supports Rollup shorthand
change_score Series change score Supports Rollup shorthand
corr Pearson correlation coefficient between two fields Requires two input fields

Filtered Aggregation

Function Description Exact/Estimate
top / bottom TopN / BottomN Exact
count_filter Conditional count Exact

Histogram Functions

Function Description Applicable Data Source Exact/Estimate
histogram_auto Automatic histogram (Recommended) Logs, Trace detail data Estimate
histogram Fixed bucket boundary histogram (Deprecated) Logs, Trace detail data Exact
histogram_quantile Calculate quantiles from Prometheus histogram Prometheus metrics Estimate

Collection Functions

Function Description Exact/Estimate
distinct List of distinct values Exact
distinct_by_collapse Collapse deduplication (retain other fields) Exact
collect Collect all values Exact
collect_distinct Collect distinct values Exact

Helper Functions

Function Description Exact/Estimate
default Set default value Exact

Time Series Functions

Function Description
rate Rate of change (per second)
irate Instant rate of change
increase Increase
derivative Derivative
difference Difference
non_negative_derivative Non-negative derivative
non_negative_difference Non-negative difference
moving_average Moving average
cumsum Cumulative sum
ewma Exponentially Weighted Moving Average
slope Trend slope
zscore Latest point Z-Score
mad_score MAD anomaly score
change_score Change score
corr Correlation coefficient

Rollup Functions

Function Description
rate Calculates rate of change (per second)
irate Calculates instant rate of change
increase Calculates increase
rate_over_sum Calculates per-second average
rate_over_count Calculates per-second count
deriv Calculates derivative
difference Calculates difference
sum Sum
avg Average
min Minimum
max Maximum
count Count
first First value
last Last value
stddev Standard deviation
mode Mode
spread Range
any Any value
ewma(alpha) Exponentially Weighted Moving Average
moving_average(n) Moving average
percentile(p) Percentile
slope Linear trend slope
zscore Latest point Z-Score
mad_score Latest point MAD anomaly score
change_score Series change score

Transformation Functions

Function Description
abs Absolute value
round / ceil / floor Rounding
log / log2 / log10 Logarithm
int / uint / float / string / bool Type conversion
substr Substring
regexp_extract Regex extraction
regexp_extract_all Extract all regex matches
regexp_replace Regex replacement
md5 MD5 hash
concat String concatenation
set Array deduplication and sort
drain Log clustering

Matching Functions

Function Description
match Substring matching
phrase / search Tokenized phrase matching
re / regex / regexp Regex matching
wildcard Wildcard matching
cidr CIDR network matching
query_string Query string syntax
exists Field existence check

Outer Functions

Function Description
cumsum Cumulative sum
rate / irate Rate of change (non-negative)
deriv Derivative (allows negative values)
increase Increase (non-negative)
difference Difference (allows negative values)
moving_average Moving average
top / bottom TopN
dbscan DBSCAN outlier detection
forecast Linear trend forecasting
fill Null value filling (recommended as outer)
abs / round / ceil / floor Mathematical operations
set Deduplicate and sort
concat String concatenation

Expression Calculation

Function Description
eval Multi-query expression calculation