Create Data Forwarding Rules¶

Prerequisites¶

Only available for Commercial Plan.

Create¶

Go to Data Forwarding > Forwarding Rules > Create.

After a data forwarding rule is created, the system will perform rule validation every 5 minutes.

Step 1: Enter Rule Name¶

The name of the current data forwarding rule.

Step 2: Define Forwarding Rules¶

1. Data Source

Includes LOG, APM, RUM, LLM, Event, Audit Event.

2. Filter Conditions

Supports custom logic operations between conditions. Multiple conditions can be added.

All conditions: Only log data that meets all filter conditions will be saved to data forwarding.
Any condition: Log data that meets any one of the filter conditions will be saved to data forwarding.

Condition operators are shown in the table below:

Condition Operator	Match Type
in, not in	Exact match, supports multiple values (separated by commas)

If no filter conditions are added here, it means saving all data.

Pipeline Processing for Forwarded Data

Central Pipeline scripts affect forwarding condition filtering and final content.

When creating a script, you can check "Enable Pipeline processing for forwarded data": If checked, data is processed by the script first and then filtered and stored; if not checked, the original data is forwarded directly.

3. Advanced Settings

When LOG is selected as the data source, you can further configure:

Include Extended Fields: By default, only the message field content of logs meeting the conditions is forwarded. If "Include Extended Fields" is checked, the entire log data meeting the conditions will be forwarded.
- When creating multiple data forwarding rules, rules with "Include Extended Fields" checked are matched first. If different rules hit the same piece of data, the logic of including extended fields takes precedence to display the entire log data.
Limit Index: Drop-down to select a native direct-write index. After enabling, only log data from the selected index will be forwarded. (❗️ After enabling Limit Index, the performance consumption of this data forwarding rule will be significantly reduced.)

Step 3: Select Archive Type¶

To provide more comprehensive data forwarding storage methods, the system supports the following storage paths.

TrueWatch: Matched log data will be saved to TrueWatch-side OSS, S3, OBS object storage.

Note

All the above archive types are available for the entire site.
When selecting TrueWatch as the data forwarding storage object, the minimum log data storage period defaults to 180 days, and the rule cannot be canceled once created. Fees will be charged daily during the storage period. You can go to Manage > Workspace Settings > Change Data Storage Policy to modify it.

Storage Format¶

Select the data storage format as needed:

JSON: Text format (❗️ When forwarding data to TrueWatch, only JSON storage format is supported).
Parquet: Columnar storage format.

Feature	JSON	Parquet
Functional Positioning	Standard format for immediate consumption and integration	Optimized format for low-cost archiving and offline analysis
Data State	Hot data that can be reviewed. Forwarded data will be normally stored and displayed within the TrueWatch platform (e.g., in the Log Explorer), maintaining consistency and visibility on both ends.	Cold data for external processing. After data is forwarded, the original logs will still be normally stored and displayed within the TrueWatch platform. However, the copy forwarded in Parquet format, due to format limitations, cannot be reloaded or reviewed within the TrueWatch interface.
Core Scenarios	While maintaining data observability within TrueWatch, provides a copy for external systems (e.g., SIEM, self-built log libraries) to consume in real-time, which is directly parseable.	Generates a dedicated copy for external big data systems to perform efficient batch analysis, with more optimal storage costs, for data meeting the conditions.
Business Use	Downstream business systems can obtain readable logs, completely consistent with those within the TrueWatch interface, quasi-real-time via API, object storage files, or message queues, for: Integrating with security audit (SIEM) platforms. Flowing into self-built log analysis tools for secondary processing.	Downstream big data systems can periodically (e.g., hourly/daily) read Parquet files in object storage in batches, for: Compliance archiving of long-term historical logs. Offline aggregation analysis and report generation across months/years.
Key Impact	Data dual-write, visible on both ends. Incurs additional storage costs but ensures data consistency and immediate availability both inside and outside the TrueWatch ecosystem.	Format-specific, with review limitations. The generated Parquet copy is optimized for external analysis and is not suitable for backflow viewing. Therefore, this copy itself cannot be reviewed within TrueWatch. However, the observability of the original logs within TrueWatch is not affected in any way.

Encrypted Storage¶

After enabling encrypted storage, the system will perform symmetric encryption on the forwarded data. If you need to query or view this data later, the system can restore the encrypted data to its original content for display.

What is symmetric encryption?

Symmetric encryption is an encryption method that uses the same key to encrypt and decrypt data, just like a key that can both lock and unlock the same lock.

Step 4: Define Data Viewing Permissions¶

Set viewing permissions for forwarded data to enhance data security.

No restrictions: All members of the workspace can view the forwarded data.
Custom: Specify member roles that can view the forwarded data.