Forward Data to TiDB Cloud Lake¶

After selecting TiDB Cloud Lake as the forwarding destination, TrueWatch generates independent Parquet batches according to the forwarding rules, loads them via system-managed intermediate storage to the specified Database.TargetTable, and supports long-term SQL and business correlation analysis.

Prerequisites¶

Before you start configuring, confirm:

• Lake DSN: Already obtained and contains a valid Database and Warehouse. If you don't have one, first Obtain a Lake DSN;
• Forwarding rule: Already completed Rule name and filter condition setup;
• SQL user permissions: Write permission is required if the target table already exists; if the target table does not exist, permission to create tables in the target Database is also required.

Configuration Steps¶

Step 1: Select the Forwarding Destination¶

In Select Forwarding Destination, choose TiDB Cloud Lake from the Analysis Platform group.

After selection, the following configurations are fixed by the system:

Step 2: Fill in Connection Information¶

After filling in the Lake DSN, first confirm that Database and Warehouse have been automatically parsed, then fill in the Target Table.

Step 3: Test the Connection¶

Click Test Connection. The system verifies:

• The format and authentication information of the Lake DSN;
• Whether Database and Warehouse are valid;
• Whether the network is reachable;
• Whether the target table exists;
• If the target table exists, whether the SQL user has write permission and whether the table schema is compatible;
• If the target table does not exist, whether the SQL user has permission to create tables in the target Database.

After the test, the page indicates the target table status, permission check results, and any issues that need resolution based on the check results.

Step 4: Set Data Viewing Permissions¶

Use the permission settings of the data forwarding rule:

• No restriction: All workspace members can view the forwarded data;
• Only specified members can view: Only specified roles or members can view.

Step 5: Complete Rule Creation¶

When completing rule creation, the system rechecks the connection information and target table status, and handles them as follows:

After the rule is created and enabled, data matching the rule is buffered into Parquet batches according to the data waiting time, and then loaded into the target table.

Data Waiting Time¶

The data waiting time affects the file buffering and batch loading frequency:

Data forwarding does not write records in real time one by one. After the rule is enabled, wait for the first batch to be buffered and loaded.

Verify in TiDB Cloud Lake¶

1. Wait for at least one data waiting period;
2. Open TiDB Cloud Lake;
3. Execute a read-only query using the Database, Warehouse, and Target Table in the rule;
4. Check whether data matching the forwarding conditions has appeared.

You can first verify with a row-limited query:

SELECT *
FROM database_name.target_table
LIMIT 10;

Replace the Database and Target Table in the example with the actual values.

Configuration Result¶

After successful configuration, the following should be satisfied:

• The rule status is Enabled;
• Database and Warehouse match the Lake DSN;
• The target table exists and is writable;
• At least one Parquet batch has been loaded successfully;
• Data matching the rule can be queried in TiDB Cloud Lake.

Notes¶

• Dataset is not a required field for TiDB Cloud Lake connection and is not separately filled in the forwarding destination configuration; the scope of forwarded data is determined by the data type and filter conditions of the rule.
• Changing the Target Table only affects subsequent loads; it does not migrate data already written to the old table.
• After changing the TiDB Cloud Lake SQL user password, re-obtain the Lake DSN and edit the rule.
• If rule loading fails, the system retains the failure reason and retries; after fixing permissions, connection, or table schema, you can re-execute.

If the test connection, rule creation, or loading fails, see TiDB Cloud Lake FAQ.