Forward Data to TiDB Cloud Lake¶

After selecting TiDB Cloud Lake as the forwarding destination, TrueWatch generates independent Parquet batches according to the forwarding rules, loads them into the specified Database.TargetTable via the system-managed intermediate storage, and enables long-term SQL and business correlation analysis.

Prerequisites¶

Before starting configuration, ensure:

Lake DSN: Already obtained and contains a valid Database and Warehouse. If not yet available, first obtain the Lake DSN.
Forwarding Rule: Already completed rule name and filter condition settings.
SQL User Permissions: Write permission is required if the target table already exists; if the target table does not exist, the user also needs permission to create a table in the target Database.

Permission Recommendations

It is recommended to use a dedicated SQL user for data forwarding, granting only the necessary write and table creation permissions on the target Database and target table. Do not use a personal account with global administrative privileges.

Configure¶

Step 1: Select the Forwarding Destination¶

In Select Forwarding Destination, select TiDB Cloud Lake from the Analytics Platform group.

After selection, the following configurations are fixed by the system:

Configuration Item	Fixed Value	Description
Forwarding Destination Type	Analytics Platform (AP)	Loads observability data into TiDB Cloud Lake for long-term SQL and business correlation analysis
Storage Format	Parquet	Default storage format
Access Type	SQL Connection Authentication	Uses Lake DSN to verify identity and perform the load

Step 2: Fill in Connection Information¶

Configuration Item	Required	Description
Lake DSN	Yes	Paste the complete DSN copied from the TiDB Cloud Lake Connect page
Database	Automatic	Automatically parsed from the Lake DSN path; read-only display, cannot be modified individually
Warehouse	Automatic	Automatically parsed from the Lake DSN parameters; read-only display, cannot be modified individually
Target Table	Yes	Enter only the table name; the actual load target is `Database.TargetTable`
Data Wait Time	Yes	Options: 15 minutes, 30 minutes, or 1 hour; default is 1 hour

After filling in the Lake DSN, first confirm that Database and Warehouse have been automatically parsed, then fill in the Target Table.

System-Managed Storage

The system generates an independent Parquet batch according to the current rule and loads it into Database.TargetTable. If the target table does not exist, it will be automatically created according to the system-managed structure of the current forwarding data type when the rule creation is completed.

Step 3: Test Connection¶

Click Test Connection. The system validates:

The format and authentication information of the Lake DSN.
Whether Database and Warehouse are valid.
Whether the network is reachable.
Whether the target table exists.
If the target table exists, whether the SQL user has write permission and whether the table structure is compatible.
If the target table does not exist, whether the SQL user has permission to create a table in the target Database.

After the test, the page displays the target table status, permission check results, and issues that need to be addressed based on the check results.

Test Connection Does Not Create the Target Table

The test connection checks the existence of the target table, related permissions, and structure compatibility, but does not create the target table. If the target table does not exist, the system only prompts the check result; the actual table creation is performed when the rule creation is completed.

Step 4: Set Data Access Permissions¶

Inherits the permission settings from the data forwarding rule:

Unrestricted: All workspace members can view the forwarded data.
Only specific members can view: Only designated roles or members can view.

Step 5: Complete Rule Creation¶

When the rule creation is completed, the system rechecks the connection information and target table status, and handles them as follows:

Target table existsTarget table does not exist

Rechecks the SQL user's write permission and table structure compatibility. If the check fails, the rule is still created and enabled, but a warning indicator [！] is displayed in the rule list. Please fix the write permission or table structure issue according to the prompt.

Creates the target table according to the system-managed structure of the current forwarding data type. If table creation permission is lacking or creation fails, the rule is still created and enabled, but a warning indicator [！] is displayed in the rule list. Please fix the permission or table creation issue according to the prompt.

After the rule is created and enabled, data matching the rule is sealed into Parquet batches according to the data wait time, and then loaded into the target table.

Data Wait Time¶

The data wait time affects the file sealing and batch loading frequency:

Option	Use Case
15 minutes	Preferred when you want to query new data in the Lake faster and can accept more frequent load tasks
30 minutes	Balances timeliness and load frequency
1 hour	Default value, suitable for long-term archiving and batch analysis

Data forwarding is not real-time, row-by-row writing. After the rule is enabled, you need to wait for the first batch to be sealed and loaded.

Verify in TiDB Cloud Lake¶

Wait for at least one data wait cycle.
Open TiDB Cloud Lake.
Execute a read-only query using the Database, Warehouse, and Target Table from the rule.
Check whether data matching the forwarding conditions has appeared.

You can start with a query that limits the number of rows:

SELECT *
FROM database_name.target_table
LIMIT 10;

Replace database_name and target_table in the example with the actual values.

Configuration Results¶

After successful configuration, the following should be satisfied:

Rule status is enabled.
Database and Warehouse are consistent with the Lake DSN.
The target table exists and is writable.
At least one Parquet batch has been loaded successfully.
Data matching the rule can be queried in TiDB Cloud Lake.

Notes¶

The dataset is not a required field for TiDB Cloud Lake connection and is not filled in separately in the forwarding destination configuration; the scope of forwarded data is determined by the rule's data type and filter conditions.
Changing the Target Table affects only subsequent loads; it does not migrate data already written to the old table.
After modifying the TiDB Cloud Lake SQL user password, you need to obtain a new Lake DSN and edit the rule.
If a rule load fails, the system retains the failure reason and retries; after fixing permission, connection, or table structure issues, the load can be re-executed.

If the test connection, rule creation, or load fails, refer to TiDB Cloud Lake FAQ.