Browser Dialtesting

Version-2.1.0

Browser dialtesting is a BROWSER task type under the inputs.dialtesting collector. It simulates page access through the Lightpanda browser engine, including opening pages, clicking elements, entering text, waiting for selectors, and asserting titles or text. It reports page performance, step details, failure reasons, and optional failure screenshots.

For basic dialtesting node configuration, see Network Dialtesting. This page only describes browser-specific configuration, deployment, and troubleshooting.

Enable Browser Dialtesting¶

Browser dialtesting is enabled by default. To configure it explicitly, set it in dialtesting.conf:

[[inputs.dialtesting]]
  # Other dialtesting node settings are omitted.

  [inputs.dialtesting.browser]
    enabled = true

Browser dialtesting currently supports Linux dialtesting nodes only. On non-Linux platforms, DataKit service mode does not run BROWSER tasks even when enabled = true is configured, except when using local debug verification mode.

DataKit runs BROWSER tasks with the embedded browser runner and the Lightpanda browser engine. Task-level advance_options.engine is overwritten by the node configuration. [inputs.dialtesting.browser].engine defaults to and currently only accepts lightpanda.

DataKit resolves Lightpanda in the following order:

1. [inputs.dialtesting.browser].engine_path
2. LIGHTPANDA_EXECUTABLE_PATH
3. lightpanda from PATH

Use max_concurrency to limit browser tasks running at the same time. 0 means no limit. On resource-limited nodes, 1 is recommended.

Kubernetes Deployment¶

For Kubernetes, use the DataKit image directly:

pubrepo.truewatch.com/truewatch/datakit:<version>

The DataKit image includes Lightpanda and can run BROWSER tasks directly. To use a custom Lightpanda binary, mount the executable into the container and configure engine_path:

[[inputs.dialtesting]]
  [inputs.dialtesting.browser]
    enabled = true
    engine = "lightpanda"
    engine_path = "/opt/datakit-browser/bin/lightpanda"
    max_concurrency = 10

Host Deployment¶

For host deployment, install Lightpanda on the dialtesting node first, then confirm browser dialtesting remains enabled. The following examples use a Linux host.

Install Lightpanda¶

The DataKit image uses the GuanceCloud Lightpanda 0.4.0-g1 release, built on Ubuntu 22.04. To install the same build on x86_64 Linux:

curl -fL -o lightpanda \
  https://github.com/GuanceCloud/browser/releases/download/0.4.0-g1/lightpanda-x86_64-linux
echo "3da11a5e0ce793480648074b6dfcc3f91c5386fd394e904aece28270f410b756  lightpanda" | sha256sum -c -
sudo install -m 0755 lightpanda /usr/local/bin/lightpanda
rm lightpanda

For arm64/aarch64 Linux, use:

curl -fL -o lightpanda \
  https://github.com/GuanceCloud/browser/releases/download/0.4.0-g1/lightpanda-aarch64-linux
echo "ee84aec580d936b843b07c968e64d6a92360f5d3cbdde4780031588f4e4abdfd  lightpanda" | sha256sum -c -
sudo install -m 0755 lightpanda /usr/local/bin/lightpanda
rm lightpanda

Verify the installation:

lightpanda version
lightpanda serve --help

Configure DataKit¶

Copy the dialtesting input configuration:

cd /usr/local/datakit/conf.d/samples
sudo cp dialtesting.conf.sample ../dialtesting.conf

Edit /usr/local/datakit/conf.d/dialtesting.conf, confirm browser dialtesting remains enabled, and set the browser engine and path explicitly:

[[inputs.dialtesting]]
  server = "https://dflux-dial.truewatch.com"
  region_id = "<your-private-node-id>"
  ak = "<your-ak>"
  sk = "<your-sk>"
  pull_interval = "1m"
  time_out = "30s"

  [inputs.dialtesting.browser]
    enabled = true
    engine = "lightpanda"
    engine_path = "/usr/local/bin/lightpanda"
    max_concurrency = 10

  [inputs.dialtesting.tags]
    region = "<your-region>"

If you do not want to set engine_path in the configuration file, set the browser path with an environment variable:

export LIGHTPANDA_EXECUTABLE_PATH=/usr/local/bin/lightpanda

If DataKit runs as a systemd service, exporting the variable in the current shell usually does not pass it to the DataKit service process. For host deployment, setting engine_path in dialtesting.conf is recommended. If you prefer environment variables, write LIGHTPANDA_EXECUTABLE_PATH into the DataKit service environment configuration and restart the service.

Restart DataKit after updating the configuration:

sudo datakit service restart

Custom CA Certificates¶

For sites whose certificates are issued by an enterprise or private CA, import the trusted CA used by Lightpanda on the dialtesting node:

[inputs.dialtesting.browser]
  ca_cert_file = "/etc/datakit/certs/internal-ca.pem"
  # ca_cert_dir = "/etc/datakit/certs"

Use PEM certificates. ca_cert_dir loads certificate files from a directory, and both settings can be used together. Paths must be absolute paths on the dialtesting node, and certificate files must not contain private keys.

The equivalent environment variables are ENV_INPUT_DIALTESTING_BROWSER_CA_CERT_FILE and ENV_INPUT_DIALTESTING_BROWSER_CA_CERT_DIR. In Kubernetes, mount the CA certificate into the DataKit container through a ConfigMap or Secret and configure its in-container path.

The imported CA does not disable TLS verification: the engine still verifies the certificate chain, hostname, and validity period. This is a node-level trust setting and cannot be overridden by an individual BROWSER task. Lightpanda itself replaces the active trust store when custom CA arguments are present, so DataKit passes a detected system CA directory and the configured custom CA file or directory as separate arguments. The system and custom certificate files remain in their original directories while both are loaded into the same in-memory trust store.

Private Network and Proxy¶

DataKit blocks internal network targets by default. On a private dialtesting node that must access loopback, RFC1918, or link-local addresses, disable this restriction in the dialtesting input:

[[inputs.dialtesting]]
  disable_internal_network_task = false

DataKit also passes this setting to Lightpanda. With the default disable_internal_network_task = true and no custom CIDR list, Lightpanda starts with --block-private-networks. When disabled_internal_network_cidr_list is configured, DataKit passes those exact ranges through --block-cidrs instead of blocking every private range. When disable_internal_network_task is false, Lightpanda allows private-network requests. No engine-specific environment variable is required.

Lightpanda 0.4.0-g1 supports the following default HTTP proxy setting:

[inputs.dialtesting.browser]
  proxy_url = "http://proxy.example.com:8080"
  # proxy_url = "http://user:password@proxy.example.com:8080"

The equivalent environment variable is ENV_INPUT_DIALTESTING_BROWSER_PROXY_URL. Proxy precedence is: task advance_options.proxy_url > proxy_url in browser_config > node-level browser.proxy_url. If the proxy intercepts HTTPS traffic, import its CA through ca_cert_file or ca_cert_dir as well.

Verify with a Local Task¶

If no BROWSER task is available from the console yet, use a local JSON task to verify the browser execution path first. browser_config is a YAML string. It is easier to write the browser script as YAML first, then put it into the JSON task.

Browser script example:

name: browser-homepage
target: https://example.com
timeout_ms: 60000
viewport:
  width: 1280
  height: 720
steps:
  - name: open page
    action: goto
    url: https://example.com
  - name: assert title
    action: assert_title
    contains: Example

Create /tmp/dialtesting-browser-task.json. When writing JSON, put the YAML above into browser_config as a string and represent line breaks with \n:

{
  "BROWSER": [
    {
      "name": "browser-homepage",
      "url": "https://example.com",
      "status": "OK",
      "frequency": "1m",
      "post_url": "https://openway.truewatch.com?token=<your-token>",
      "browser_config": "name: browser-homepage\ntarget: https://example.com\ntimeout_ms: 60000\nviewport:\n  width: 1280\n  height: 720\nsteps:\n  - name: open page\n    action: goto\n    url: https://example.com\n  - name: assert title\n    action: assert_title\n    contains: Example\n"
    }
  ]
}

Temporarily set server in dialtesting.conf to the local file URL and keep browser dialtesting enabled:

[[inputs.dialtesting]]
  server = "file:///tmp/dialtesting-browser-task.json"
  pull_interval = "10s"
  time_out = "30s"

  [inputs.dialtesting.browser]
    enabled = true
    engine = "lightpanda"
    engine_path = "/usr/local/bin/lightpanda"
    max_concurrency = 10

After verification, restore server, region_id, ak, sk, and other settings to the real dialtesting node configuration.

Then verify with debug mode or by restarting DataKit:

datakit debug --input-conf /usr/local/datakit/conf.d/dialtesting.conf

If DataKit runs as a service:

sudo datakit service restart

After 1~2 pull intervals, check metrics:

curl -s http://127.0.0.1:9529/metrics | grep datakit_dialtesting

Normally, datakit_dialtesting_task_number{protocol="BROWSER"} is greater than 0, and datakit_dialtesting_worker_send_points_number{protocol="BROWSER",status="ok"} keeps increasing.

BROWSER Task Example¶

In custom dialtesting tasks, a BROWSER task uses browser_config to define the browser script. browser_config is a YAML string that describes page navigation, interactions, and assertions.

Common browser_config fields:

steps can use actions and assertions such as goto, click, fill, wait_for_selector, wait_for_url, assert_title, assert_url, and assert_text.

wait_for_url accepts contains, equals, or text and polls until the URL matches or the step/script timeout expires. assert_title, assert_url, and assert_text always poll as well. A configured step timeout_ms takes precedence; otherwise the script timeout applies. DataKit uses a 60-second default step timeout. For example:

- name: wait for dashboard redirect
  action: wait_for_url
  contains: https://console.example.com/dashboard
  timeout_ms: 15000
- name: assert dashboard title
  action: assert_title
  contains: Dashboard
  timeout_ms: 5000

In the full task JSON, browser_config is inside the BROWSER task object:

{
  "BROWSER": [
    {
      "name": "browser-homepage",
      "url": "https://example.com",
      "status": "OK",
      "frequency": "1m",
      "post_url": "https://openway.truewatch.com?token=<your-token>",
      "browser_config": "<browser_config YAML string>",
      "advance_options": {
        "screenshot_on_failure": true
      }
    }
  ]
}

Screenshot Support¶

Set advance_options.screenshot_on_failure to true to capture a PNG after a browser step fails. DataKit uploads the image and records its metadata in steps[].screenshot; upload failures are reported in screenshot_upload_error.

Lightpanda screenshots are text-oriented semantic renderings rather than pixel-perfect browser images. They are useful for checking the page content and DOM state at the point of failure, but do not reproduce all CSS, images, canvas content, or other Chrome rendering details. A screenshot is unavailable when Lightpanda cannot start, the browser process exits, or the browser session has already reached its overall timeout.

Troubleshooting¶

Use DataKit metrics on the dialtesting node to check task and reporting status:

curl -s http://127.0.0.1:9529/metrics | grep datakit_dialtesting

Check these metrics first:

datakit_dialtesting_task_number
datakit_dialtesting_worker_send_points_number
datakit_dialtesting_dataway_send_failed_number
datakit_dialtesting_worker_cached_points_number
datakit_dialtesting_worker_dropped_points_number

Check browser engine availability with:

echo $LIGHTPANDA_EXECUTABLE_PATH
$LIGHTPANDA_EXECUTABLE_PATH version
command -v lightpanda

Troubleshoot common issues as follows:

• No tasks are pulled: check server, region_id, ak, and sk, and confirm that datakit_dialtesting_task_number{protocol="BROWSER"} is greater than 0.
• BROWSER tasks are available in the console but not executed on the node: confirm that [inputs.dialtesting.browser].enabled = false is not explicitly configured, and check whether the DataKit log contains browser.enabled is false or unsupported.
• Results are not reported: check that task post_url is reachable, and that datakit_dialtesting_dataway_send_failed_number, datakit_dialtesting_worker_cached_points_number, and datakit_dialtesting_worker_dropped_points_number do not keep increasing.
• Browser fails to start: check that engine_path, LIGHTPANDA_EXECUTABLE_PATH, or lightpanda from PATH is accessible to the DataKit process.
• Browser dependencies are missing: in Kubernetes, use the datakit:<version> image directly; on hosts, confirm that Lightpanda is installed correctly.
• Screenshot is not uploaded: confirm that advance_options.screenshot_on_failure is true, then check screenshot_upload_error. Browser startup failures, process exits, and overall task timeouts may not leave a live session to capture.

Normally, the node can pull BROWSER tasks, datakit_dialtesting_worker_send_points_number{status="ok"} keeps increasing, and datakit_dialtesting_dataway_send_failed_number, datakit_dialtesting_worker_cached_points_number, and datakit_dialtesting_worker_dropped_points_number do not keep increasing.