Frequently Asked Questions¶

Event Viewing and Time Range¶

"Unrecovered Events" only shows the last 48 hours by default. How do I view earlier unrecovered events?

There is a Time Widget in the top-right corner of the page that allows you to freely adjust the time range.

What is the difference between the "All Events" and "Unrecovered Events" tabs? Why can both filter out unrecovered events?

Key difference: "Unrecovered Events" is a shortcut that automatically filters for you; "All Events" requires manually adding filter conditions but offers greater flexibility.

After adjusting the Time Widget, will my previously set filter conditions be reset?

No. Time range adjustment is independent. Your filter conditions (event severity, alert policy, monitor name, etc.) remain unchanged, and the system applies those filters within the new time range.

Event Content and Variables¶

Variables in event content (such as {{df_dimension_tags}}, {{Result}}) appear empty or improperly formatted. How do I debug this?

Variable source: Variables in event content are defined in the monitor's Event Notification configuration section, and the system substitutes them based on actual monitoring data.

Common causes of empty values:

1. The DQL query does not return the corresponding field
2. Variable name typos (case-sensitive)
3. Detection dimension tags are empty

Debugging methods:

1. Check the Detection Metrics section in the event details to confirm the raw query results
2. Check whether the monitor's DQL query statement is correct

Common variable reference:

• {{Result}} - Detection value
• {{df_dimension_tags}} - JSON of detection dimension tags
• {{df_status}} - Event status
• {{df_monitor_name}} - Monitor name
• {{date}} - Event timestamp

What is the difference between the event title and the event content? Which one is sent in notifications?

• Event title: Displayed in the event list and also serves as the alert notification title (email subject, DingTalk message title, etc.)
• Event content: Shown on the event details page and serves as the default alert notification body

Custom notification content:

The monitor supports enabling the "Custom Notification Content" toggle. When enabled:

• Event content is still saved in the event details
• But alert notifications are sent using the custom template
• Suitable for scenarios where content needs to be customized for different notification channels

What template functions are supported in event content? Can numbers be formatted as percentages?

Currently supported template functions:

Event Status and Recovery¶

Is event recovery automatic or manual? Why do some events remain unrecovered?

Automatic recovery: When the monitor detects that metrics have returned to normal, it automatically generates a recovery event and df_status changes to ok

Manual recovery: Events can be manually recovered from the event list or details page

Common causes of unrecovered events:

1. Monitor configuration issue: Recovery conditions are not correctly configured
2. Data gap events: No recovery policy configured or data has not been re-reported
3. Detection logic issue: Threshold settings prevent automatic recovery determination

Will events generated while a monitor is muted appear in the event list? What is their status?

Yes, they will appear in the event list, but:

• No alert notifications will be sent
• No incidents will be created in sync (if sync creation is configured)
• Event status is recorded normally (critical/warning, etc.)

Muting only pauses notification sending; it does not affect event generation or recording.

Data Gap Events¶

What is a data gap event? In which scenarios should it be enabled?

A data gap occurs when the monitor does not find the expected data within the detection period.

Configuration location: The "Data Gap Event" configuration area of the monitor (supported by some monitor types)

Three processing strategies:

1. Do not trigger events - Mute handling
2. Trigger recovery events - Data gap is treated as abnormal recovery
3. Trigger data gap events - Generate a dedicated gap alert (severity configurable)

Why does a data gap event sometimes not trigger when data has clearly not been reported?

TrueWatch uses an "edge-triggered" mechanism to determine data gaps:

In the previous round, X was detected; in the current round, X is not detected, so X has a data gap.

Key limitations:

• No alert is generated when no data is found during the first detection (the system does not know what "should" be present)
• A "data present" state must have been reached first; only then, if data is not detected again, is it determined to be a gap

Troubleshooting suggestions:

1. Confirm the monitor has been running normally for at least one detection cycle and has queried data
2. Check the "detection range drift" mechanism (the actual detection time range drifts by 1 minute)
3. Check monitor execution logs to confirm whether the DQL query was successful

What is the generation logic for data gap events and data recovery events?

Alternating generation mechanism:

• Data gap events and data recovery events always alternate
• Consecutive data gap events are not generated
• Consecutive data recovery events are also not generated

Determination flow:

No data on first detection → No alert
↓
Data detected → Record "data present" state
↓
No data detected again → Trigger data gap event
↓
Data re-reported → Trigger data recovery event
↓
No data detected again → Trigger data gap event (new)

Event Correlation and Troubleshooting¶

How are "related events" on the event details page correlated? Why is it sometimes empty?

Correlation logic: - Based on the same detection dimension tags (such as host, service, etc.) - Based on time windows (related events within the same time period)

Common causes of empty correlation:

1. The dimension tag does not exist in other events
2. No other related events within the time window
3. The current event type does not support correlation (some monitor types)

What does "Related SLO" showing 0 mean? When will there be data?

• Showing 0: Indicates the event is not associated with any SLO
• With data: Only when the event is triggered by an SLO will the associated SLO information be displayed
• Events triggered by monitors are not associated with SLOs by default

What does the "detection interval" dashed line in the historical trend chart mean? Can the time range be adjusted?

• Detection interval dashed line: Marks the specific detection time window that triggered the alert
• Chart display: Shows the trend of the detection metric over a longer time range

Click the "Get Chart Query" button in the top-right corner of the chart to jump to the metrics or log explorer, where you can flexibly adjust the time range for longer-term trend analysis.

Event Sources and Fields¶

What are the field differences for events from different sources (monitor, AI monitor, audit, OpenAPI)?

Different df_source values correspond to different additional fields:

What is the functional difference between custom events written via OpenAPI and system-generated events?

Functional differences: - Custom events can set fields such as df_status, df_title, df_message - df_dimension_tags can be specified for correlation - They are not automatically associated with dashboards like monitor events - Recovery logic must be handled manually (by calling the recovery API)

Use cases: External system integration, custom business alerts, bulk importing historical events, etc.

Audit Events¶

What operations do audit events specifically record? Where can I view them?

Viewing location: Management > Basic Settings > Security > Operation Audit

General scope of records:

• Adding/deleting data authorization (cross-workspace authorization)
• Add, delete, and modify operations on configurations such as monitors, SLOs, and alert policies
• Workspace member permission changes

Field characteristics: df_source = audit, includes audit-specific fields such as operator, operation time, and operation type.

Common Issue Troubleshooting¶

Why does the time shown in the event details not match the actual incident occurrence time?

This is normal, for two reasons:

1. Scheduled trigger time vs actual execution time:

   • The time displayed for an event is the monitor's scheduled trigger time (based on the aligned Crontab time)
   • Not the actual time the event was generated in the system

2. Detection range drift:

   • The actual data range detected is: scheduled trigger time - detection range - 1 minute to scheduled trigger time - 1 minute
   • Therefore, the timestamp of the incident data may be earlier than the event display time

Troubleshooting suggestion: Check the "Detection Metrics" section in the event details to confirm the actual detected data time range.

I can query the incident data directly on the platform, but the monitor did not generate an event. Why?

Common causes:

1. Data write delay: The incident data was not yet queryable at detection execution time (the monitor automatically drifts by 1 minute to avoid this, but it fails if the delay exceeds 1 minute)
2. DQL query failure: The detection process was interrupted due to a query failure
3. Monitor muted: Notifications are not sent during a mute period, but events are still generated (check the event list)
4. Threshold configuration: The actual detection value did not reach the trigger threshold

How long can event data be retained? How do I export or archive it?

Long-term retention options:

• Use Dataway Sink to stream event data to external storage
• Use OpenAPI to periodically pull event data to local storage
• Use Data Forwarding to forward to external systems such as Kafka, S3, etc.