Mutation Detection¶
Current Document Positioning
This document is the second step in the configuration process of detection rules. After completion, please return to the main document to continue with the third step: Event Notification.
By comparing the absolute change amount or relative percentage change of the same metric in two different time periods, it is determined whether an anomaly has occurred. This method is commonly used to track metric peaks or fluctuations. When an anomaly is detected, it can more accurately generate event records for subsequent analysis and processing.
Suitable for monitoring relative changes or rates of change between short-term and long-term data. For example, set the MySQL connection count metric to a percentage difference greater than 500% between the average of the last 15 minutes and the average of the past 1 day. This means that if the average connection count in the last 15 minutes exceeds 5 times the average connection count of the past day, the system will trigger an alert.
It is recommended to use statistical functions such as average (avg), maximum (max), minimum (min), etc., to calculate these metrics, rather than using the last value (last) function, to reduce the impact of abnormal data and improve monitoring accuracy.
Detection Metrics¶
Define the detection data source and aggregation method based on DQL (❗️Please avoid selecting high-cardinality fields as detection dimensions. If configured improperly with overly loose trigger conditions, it may cause frequent alerts. The current query returns a maximum of 100,000 records).
Detect mutation anomalies by comparing metric data from two time periods:
Result = [Time Period A] vs [Time Period B] detection metric [Difference / Difference Percentage]
Configuration Elements¶
| Configuration Item | Description |
|---|---|
| Workspace | Defaults to the current workspace, can be switched to other authorized workspaces After authorization, you can use the detection metrics of other workspaces under the current account to create monitors. Once the rule is successfully created, cross-workspace alert configuration can be achieved. Note that when you select another workspace, the detection metric dropdown list will only display the data types authorized for the current workspace. |
| Data Source Type | Metrics, Logs, APM, RUM, etc. |
| Query Method | Simple Query, Expression Query |
| Filter Conditions | Filter the detection metric data based on metric tags to limit the data range for detection; supports adding one or more tag filters; supports fuzzy matching and fuzzy non-matching filter conditions |
| Aggregation Algorithm | Avg by, Min by, Max by, Sum by, Last, First by, Count by, Count_distinct by, p50, p75, p90, p99 |
| Detection Dimensions | Any string type (keyword) field in the data can be selected as a detection dimension. Currently, up to three fields can be selected as detection dimensions. By combining multiple detection dimension fields, a specific detection object can be determined. The system will determine whether the statistical metric corresponding to a detection object meets the trigger condition threshold. If satisfied, an event will be generated.* (For example, selecting detection dimensions host and host_ip, the detection object can be {host: host1, host_ip: 127.0.0.1}.) * |
| Alias | Custom detection metric name |
Time Period Configuration¶
| Configuration Item | Description |
|---|---|
| Time Period A | Recent data time period as the baseline |
| Time Period B | Historical data time period for comparison |
| Comparison Method | Difference: Absolute difference between the two time periods (A - B) Difference Percentage: Relative change percentage ((A - B) / B x 100%) |
- Available Time Periods:
| Time Period Type | Options |
|---|---|
| Historical Same Period | Last Month, Last Week, Yesterday, 1 Hour Ago, Previous Period, Last 15 Minutes, Last 30 Minutes, Last 1 Hour, Last 4 Hours, Last 12 Hours, Last 1 Day |
| Recent Time | Last 1 Minute, Last 5 Minutes, Last 15 Minutes, Last 30 Minutes, Last 1 Hour, Last 4 Hours, Last 12 Hours, Last 1 Day |
Note
The detection intervals "Yesterday" and "1 Hour Ago" compare the difference or difference percentage of the detection metric within the same time range; other detection intervals compare the difference or difference percentage of the detection metric between two different time periods.
Click to view Query Method Details.
Detection Frequency¶
The execution frequency of the detection rule, automatically matching the detection interval with the larger time range among the two selected detection intervals.
- Default selection: 5 minutes
Trigger Conditions¶
Configure trigger conditions and severity levels. When the query result contains multiple values, if any value meets the trigger condition, an event will be generated.
Supports configuring Critical, Error, Warning, Info four-level thresholds, as well as OK recovery conditions.
| Level | Configuration | Description |
|---|---|---|
| Critical | When Result >= [value] |
Highest level alert, requires immediate handling |
| Error | When Result >= [value] |
High level alert, requires priority handling |
| Warning | When Result >= [value] |
Medium level alert, requires attention |
| Info | When Result >= [value] |
Low level alert, requires awareness |
| OK | [N] consecutive detections with no event generated |
After the detection rule takes effect, if within the configured custom number of detections, the data detection result returns from abnormal (Critical, Error, Warning, Info) to normal, a recovery alert event will be triggered. ❗️ Recovery alert events are not restricted by Alert Silence. If the recovery alert event detection count is not set, the alert event will not recover and will remain in the Events > Unrecovered Events List |
For more details, refer to Event Level Description.
Trigger Precondition¶
Enabled by default. As the entry threshold for mutation detection, the mutation detection rule will only proceed with judgment when the detection value meets the precondition threshold.
- Configuration format: Only execute the following judgment when the
[time period]detection value[operator][threshold] - Supported operators:
>,>=,<,<=(default selected>) - After disabling this configuration, the system will directly proceed with the mutation detection rule judgment without setting an entry threshold.
Mutation Direction¶
An event is triggered when the mutation direction is [direction]:
- Up: Detection data increases (rises) mutation
- Down: Detection data decreases (drops) mutation
- Up or Down: Detection data fluctuates in both directions (rises or drops) mutation
Mass Alert Protection¶
Enabled by default.
When the number of alerts generated by a single detection exceeds the preset threshold, the system will automatically switch to a summary strategy by status: Instead of processing alert objects one by one, a small number of summary alerts are generated based on the event status and pushed.
This ensures timely notification while significantly reducing alert noise and avoiding timeout risks due to processing too many alerts.
When this switch is enabled, the event details generated by subsequent monitor detections of anomalies will not display historical records and related events.
Data Gap¶
The processing strategy when the detection metric query result is empty within the detection interval:
| Option | Description |
|---|---|
| Do Not Trigger Event (Default) | Links the time range of the detection interval, and based on the query results of the detection metric in the last few minutes, determines whether to generate an event. Suitable for scenarios where data loss is allowed |
| Treat Query Result as 0 | Links the time range of the detection interval, treats the query result of the detection metric in the last few minutes as 0, and re-compares it with the threshold configured in the Trigger Conditions above to determine whether to trigger an abnormal event |
| Custom Fill and Trigger Event | Supports custom filling of the detection interval value and separately triggering the following event types: Data Gap Event, Critical Event, Error Event, Warning Event, Info Event, and OK Event. ❗️When selecting this strategy, it is recommended to configure the custom data gap time ≥ the detection interval time interval; if the configured time ≤ the detection interval time interval, data gap and anomaly may occur simultaneously, in which case the data gap processing result will be applied first |
When configuring trigger conditions, data gap, and information generation simultaneously, the priority for judgment is: Data Gap > Trigger Conditions > Information Event Generation.
That is: first determine whether there is a data gap, then determine whether the threshold is triggered, and finally determine whether to generate an information event.
Information Generation¶
After enabling this option, you need to configure the Information Generation Condition. When the detection result does not trigger any of the "Critical", "Error", "Warning", or "Info" thresholds, and the information generation condition is met, the system will write an "Info" event.
Suitable for scenarios where normal state changes or low-priority information need to be recorded.
Subsequent Configuration¶
After completing the above detection configuration, please continue to configure:
-
Event Notification: Define event title, content, notification members, data gap handling, and associated faults;
-
Alert Configuration: Select alert strategy, set notification targets and silence periods;
-
Association: Associate with dashboards for quick navigation to view data;
-
Permissions: Set operation permissions to control who can edit/delete this monitor.