Synthetic Testing Anomaly Detection¶
Current Document Positioning
This document is the second step in the monitor rule configuration process. After completing the configuration, please return to the main document to continue with the third step: Event Notifications.
Used to monitor synthetic test (dialing test) data within the workspace. You can set threshold ranges for the specified data volume generated by dialing test tasks within a certain time period. Once the data volume reaches these thresholds, the system will trigger an alert. Supports monitoring based on performance metrics and quantity statistics for protocol types such as HTTP, TCP, ICMP, WEBSOCKET, and multistep tests.
Suitable for monitoring scenarios such as URL availability deployed in production environments, service port connectivity, and network latency. For example, monitor whether the average response time or availability rate of key business interfaces meets the standards.
Detection Configuration¶
Detection Frequency¶
Set the time period for executing the detection.
-
Preset options: 1 minute, 5 minutes (default), 10 minutes, 15 minutes, 30 minutes, 1 hour, 6 hours, 12 hours, 24 hours
-
Crontab mode: Click "Switch to Crontab mode" to configure a custom period, supporting scheduled task execution based on seconds, minutes, hours, days, months, weeks, etc.
Detection Interval¶
Set the time range of data queried for each detection (❗️The detection interval should be greater than or equal to the detection frequency, and must match the actual data reporting cycle to avoid missed detections or false positives).
| Detection Frequency | Detection Interval (Dropdown options) |
|---|---|
| 30s | 1m/5m/15m/30m/1h/3h |
| 1m | 1m/5m/15m/30m/1h/3h |
| 5m | 5m/15m/30m/1h/3h |
| 15m | 15m/30m/1h/3h/6h |
| 30m | 30m/1h/3h/6h |
| 1h | 1h/3h/6h/12h/24h |
| 6h | 6h/12h/24h |
| 12h | 12h/24h |
| 24h | 24h |
- Custom format: Custom input detection interval, e.g., 20m (last 20 minutes), 2h (last 2 hours), 1d (last 1 day).
Detection Metrics¶
Set the metrics for detection data. Supports setting the specified data generated by all or a single dialing test task in the current workspace as detection metrics (❗️Avoid selecting high-cardinality fields as detection dimensions. Improper configuration with overly loose trigger conditions may cause frequent alerts. The maximum number of returned records for a single query is 100,000).
Supports two query modes:
-
Dialing test metrics (based on specific performance metrics)
-
Quantity statistics (based on data source query)
Dialing Test Metrics¶
Detect based on specific performance metrics of synthetic tests.
| Configuration Item | Description |
|---|---|
| Dialing Test Type | Includes HTTP Dialing Test, TCP Dialing Test, ICMP Dialing Test, WEBSOCKET Dialing Test, Multistep Test, and other protocol types |
| Dialing Test Address | Supports monitoring all or a single dialing test task of synthetic tests in the current workspace; obtains the corresponding dialing test task list based on the selected dialing test type |
| Metrics | Supports detection based on metric dimensions, including: Average Response Time, P50 Response Time, P75 Response Time, P90 Response Time, P99 Response Time, Availability Rate, Number of Error Requests, Number of Requests, etc. |
| Detection Dimensions | Fields of string type (keyword) in the configuration data can be selected as detection dimensions. Currently, a maximum of three dimension fields can be selected. Through the combination of multiple detection dimension fields, a specific detection object can be determined. The system will judge whether the statistical metric corresponding to a detection object meets the trigger condition threshold. If satisfied, an event will be generated.* (For example, if the detection dimensions host and host_ip are selected, the detection object can be {host: host1, host_ip: 127.0.0.1}.) * |
| Filter Conditions | Filter the detection metric data based on metric labels to limit the data range for detection; supports adding one or more label filters; supports fuzzy match and fuzzy not match filter conditions |
Quantity Statistics¶
Perform statistical detection based on synthetic test data sources.
You can query and count dialing test tasks based on different dialing test types by selecting the corresponding data source (http_dial_testing, tcp_dial_testing, icmp_dial_testing, websocket_dial_testing, multi_dial_testing, etc.).
Supports limiting the detection range through keyword search or label filtering.
In addition to simple queries, expression query methods are also supported.
Additional Information¶
Additional fields are only used for extra queries and will not be used for trigger condition judgment. You can configure them in event notifications. If multiple matching values are detected, one record will be returned randomly.
Trigger Conditions¶
Configure trigger conditions and severity levels. When the query result has multiple values, if any value meets the trigger condition, an event will be generated.
Supports configuring four levels of thresholds: Critical, Error, Warning, Info, and OK recovery conditions.
| Level | Configuration | Description |
|---|---|---|
| Critical | When Result >= [value] |
Highest severity alert, requires immediate handling |
| Error | When Result >= [value] |
High severity alert, requires priority handling |
| Warning | When Result >= [value] |
Medium severity alert, requires attention |
| Info | When Result >= [value] |
Low severity alert, needs to be noted |
| OK | No event generated for [N] consecutive detections |
After the monitor rule takes effect, if the data detection result changes from abnormal (Critical, Error, Warning, Info) back to normal within the configured custom detection count, a recovery alert event will be triggered. ❗️ Recovery alert events are not limited by alert muting. If the recovery alert event detection count is not set, the alert event will not recover and will always appear in the Events > Unrecovered Events List |
For more details, refer to Event Level Description.
Advanced Options¶
Consecutive Trigger Judgment¶
When enabled, an event is generated only when the trigger condition is continuously met, avoiding false positives from transient fluctuations (❗️The maximum configuration limit is 10 times).
Bulk Alert Protection¶
Enabled by default.
When the number of alerts generated by a single detection exceeds the preset threshold, the system automatically switches to a summary-by-state strategy: Instead of processing alert objects one by one, it generates a small number of summary alerts based on the event state and pushes them.
This ensures timely notification while significantly reducing alert noise, avoiding the risk of timeout due to processing too many alerts.
When this switch is enabled, subsequent events detected by the monitor will not display historical records and related events in the event details.
Data Interruption¶
The handling strategy when the query result of the detection metric is empty within the detection interval:
| Option | Description |
|---|---|
| Do not trigger events (default) | Linked to the detection interval time range, determines whether to generate an event based on the query result of the detection metric in the last few minutes. Suitable for scenarios where data loss is allowed. |
| Treat query result as 0 | Linked to the detection interval time range, treats the query result of the detection metric in the last few minutes as 0, and re-compares it with the thresholds configured in the Trigger Conditions above to determine whether to trigger an abnormal event. |
| Custom fill and trigger event | Supports custom filling of the detection interval value and triggers the following event types separately: Data Interruption Event, Critical Event, Error Event, Warning Event, Info Event, and Recovery Event. ❗️When selecting this strategy, it is recommended that the custom data interruption time configuration is ≥ the detection interval time interval; if the configured time ≤ the detection interval time interval, data interruption and anomaly may be satisfied simultaneously, in which case the data interruption handling result will be applied first. |
When trigger conditions, data interruption, and information generation are configured simultaneously, the priority for triggering is: Data Interruption > Trigger Conditions > Information Event Generation.
That is: first determine whether there is a data interruption, then determine whether the threshold is triggered, and finally determine whether to generate an information event.
Information Generation¶
When this option is enabled, you need to configure the Information Generation Condition. The system will only write an "Information" event when the detection result does not trigger any of the thresholds "Critical", "Error", "Warning", or "Info", and the information generation condition is met.
Suitable for scenarios where normal state changes or low-priority information need to be recorded.
Subsequent Configuration¶
After completing the above detection configuration, please continue to configure:
-
Event Notifications: Define the event title, content, notification members, data interruption handling, and related incidents;
-
Alert Configuration: Select an alert strategy, set notification targets and mute periods;
-
Association: Associate with dashboards for quick navigation to view data;
-
Permissions: Set operation permissions to control who can edit/delete this monitor.