rsdatakit

rsdatakit is a lightweight Rust collector for Linux hosts and Kubernetes nodes. It sends data to Dataway in the DataKit/Guance Point v2 format.

Features¶

Linux amd64 and arm64 are supported. Windows, macOS, OTLP Metrics, OTLP Logs, eBPF, dial testing, remote configuration, and the DataKit plugin system are not supported.

Requirements¶

• Linux hosts must expose /proc and /sys; journald collection also requires the systemd journal runtime library.
• Managing the rsdatakit service with the installer requires systemd and systemctl.
• Kubernetes collection must run as a DaemonSet on each collected node and be allowed to read Pod metadata, Kubelet Summary, and host container-log directories.
• A reachable Dataway URL and token are required. Restrict access to configuration and state directories.

Installation¶

RSDATAKIT_DATAWAY_URL="https://openway.truewatch.com" \
RSDATAKIT_DATAWAY_TOKEN="<DATAWAY_TOKEN>" \
  sh -c "$(curl -fsSL https://static.truewatch.com/rsdatakit/install.sh)"

RSDATAKIT_DATAWAY_URL="https://openway.truewatch.com" \
RSDATAKIT_DATAWAY_TOKEN="<DATAWAY_TOKEN>" \
  sh -c "$(curl -fsSL https://static.truewatch.com/rsdatakit/install-<VERSION>.sh)"

RSDATAKIT_UPGRADE=1 \
  sh -c "$(curl -fsSL https://static.truewatch.com/rsdatakit/install.sh)"

systemctl is-active rsdatakit
systemctl status rsdatakit --no-pager
journalctl -u rsdatakit -n 30 --no-pager

wget https://static.truewatch.com/rsdatakit/rsdatakit.yaml

wget -O rsdatakit.yaml https://static.truewatch.com/rsdatakit/rsdatakit-<VERSION>.yaml

kubectl create namespace rsdatakit-system --dry-run=client -o yaml | kubectl apply -f -
kubectl -n rsdatakit-system create secret generic rsdatakit-secrets \
  --from-literal=dataway-url='https://openway.truewatch.com' \
  --from-literal=dataway-token='<DATAWAY_TOKEN>' \
  --from-literal=otlp-token='<OTLP_TOKEN>' \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl apply -f rsdatakit.yaml

kubectl -n rsdatakit-system rollout status daemonset/rsdatakit
kubectl -n rsdatakit-system get pods -l app.kubernetes.io/name=rsdatakit

kubectl -n rsdatakit-system logs daemonset/rsdatakit --tail=100

Configuration¶

The default configuration path is /etc/rsdatakit/rsdatakit.toml. Use --config <path> or RSDATAKIT_CONFIG to select another file. Configuration is read only at startup; restart the service after changing it.

The following is the smallest useful configuration:

[agent]
# Local state directory for the WAL and file-log read positions. Must be absolute.
state_dir = "/var/lib/rsdatakit"
# Optional standalone health-listener address. Leave empty to disable it.
health_addr = ""

[dataway]
# Workspace Dataway endpoint.
url = "https://openway.example.com"
# Workspace token.
token = "<DATAWAY_TOKEN>"
# Per-request Dataway timeout in seconds. Valid range: 1–60.
timeout_secs = 15
# Tags added to all reported data.
global_tags = { env = "prod" }
# Target index. Uses the workspace default index when empty.
storage_index = ""

[metrics]
# Host metrics collection interval in seconds. Valid range: 5–300.
interval_secs = 15
# Mount points whose capacity is collected. Maximum: 32. Each must be absolute.
mounts = ["/"]
# Also collect virtual interfaces such as veth, docker0, and cni.
enable_virtual_interfaces = false

[hostobject]
# HOST object reporting interval in seconds. Valid range: 60–3600.
interval_secs = 300

[wal]
# Local write-ahead log directory. Must be absolute.
dir = "/var/lib/rsdatakit/wal"
# Total WAL capacity in bytes. Valid range: 4 MiB–256 MiB; the oldest unsent data is evicted when full.
max_bytes = 268435456

agent.state_dir, logs.state_dir, and wal.dir must be absolute paths. The default WAL capacity is 256 MiB. Set wal.max_bytes according to the acceptable Dataway outage duration and the volume of logs and traces. At process startup, RSDATAKIT_DATAWAY_URL and RSDATAKIT_DATAWAY_TOKEN can override the endpoint and token separately; RSDATAKIT_HEALTH_ADDR overrides the standalone health address.

Host collection¶

Host metrics and the HOST object start by default; they do not have a separate enabled switch.

The host metric interval is limited to 5–300 seconds, and the HOST interval to 60–3600 seconds. CPU, network, and disk I/O rates require two samples, so rate fields may be absent on the first run.

Metrics¶

The tables below describe only the fields currently emitted by rsdatakit. Measurement and field names are compatible with their DataKit counterparts, but rsdatakit does not implement every extended field exposed by the corresponding DataKit collectors.

cpu¶

mem¶

disk¶

One Point is emitted for each mount listed in [metrics].mounts.

diskio¶

One Point is emitted for every Linux block device with non-zero I/O counters.

net¶

One Point is emitted for every interface that passes the configured filter and has a previous sample.

system¶

Object¶

HOST¶

Log collection¶

Logs are disabled by default. An empty paths array collects no files. Configure only the paths you need instead of a broad global glob.

[logs]
# Enables file-log collection.
enabled = true
# Absolute path globs for log files to collect.
paths = ["/var/log/my-service/*.log"]
# Log data source name.
source = "my-service"
# service tag added to logs.
service = "my-service"
# Reports first-level members of a top-level JSON object as log fields.
json_as_fields = false
# Maximum number of active files read simultaneously.
max_files = 32
# Interval for rediscovering matching files in seconds.
discovery_secs = 5
# copytruncate log-rotation detection strategy.
copytruncate_mode = "auto"
# Local directory that stores file read positions. Must be absolute.
state_dir = "/var/lib/rsdatakit"

# Effective only when built with the journald feature and libsystemd is available.
journald_enabled = false
# Collects only this systemd unit. Leave empty for all accessible journal entries.
journald_unit = "my-service.service"

[logs.multiline]
# Continuation pattern. Leave empty to report one log point per line.
continuation_regex = "^\\s+"
# Maximum wait time for continuation lines in milliseconds.
flush_timeout_ms = 1000
# Maximum bytes in a merged log entry.
max_bytes = 262144

Log positions, the journald cursor, and WAL data are kept in the state directory. File logs resume from a confirmed position after restart. A position advances only after the data reaches WAL or is accepted by Dataway. Matched continuation lines are sent as a single multiline log.

json_as_fields is disabled by default. When enabled, a log whose top level is a JSON object reports its first-level members as fields. Strings, booleans, int64 values, and finite float64 values retain their types; objects and arrays are encoded as compact JSON strings; null and numbers outside the supported range are omitted. Invalid JSON, a non-object top level, or an object with no usable fields retains the original message. JSON fields override collector fields with the same name. Field names are normalized using Point rules and limited to 256 bytes, with at most 1024 fields per Point. The reserved keys time, source, date, and storage_index are renamed to their json_* forms and do not change the Point timestamp, log source, or storage index. The feature can also be enabled with RSDATAKIT_LOG_JSON_AS_FIELDS=true; the former json_extract setting and RSDATAKIT_LOG_JSON_EXTRACT environment variable remain compatibility aliases.

journald_enabled = true adds systemd journal collection; it does not change whether file logs are collected. Release packages include journald support. For a manual source build, use:

cargo build --release --locked --features journald

Log data¶

The log measurement is [logs].source. Kubernetes container logs commonly start with source kubernetes; when metadata is matched, the measurement is changed to the container name.

Kubernetes container logs also receive the following tags when metadata is available:

OpenTelemetry Trace¶

OTLP is disabled by default. When enabled, rsdatakit receives OTLP Trace and forwards it to the tracing data category:

[otlp]
# Starts OTLP HTTP and gRPC receivers.
enabled = true
# OTLP/HTTP listen address.
http_addr = "127.0.0.1:4318"
# OTLP/gRPC listen address. Must differ from http_addr.
grpc_addr = "127.0.0.1:4317"
# Bearer token required for non-local listen addresses. It may be empty for local listening.
bearer_token = ""
# Maximum bytes per OTLP request.
max_request_bytes = 4194304
# Maximum number of concurrent OTLP requests.
max_concurrent_requests = 8
# Total memory budget in bytes for admitted OTLP requests.
max_inflight_bytes = 16777216

When an HTTP or gRPC address is not loopback-only, bearer_token is mandatory. Clients use Authorization: Bearer <token>. Only Trace is accepted today; OTLP Metrics and OTLP Logs are not supported. /health can be served independently through agent.health_addr; disabling OTLP does not bind ports 4317 or 4318. The connection idle timeout is 30 seconds. HTTP headers and bodies, gRPC connection startup, and gRPC request-body reception and decoding each have a 65-second absolute deadline. The gRPC body deadline stops after the complete message and trailers are decoded; subsequent mapping, queue admission, and Dataway/WAL receipt waiting have no whole-handler timeout, and an active request is not interrupted by the connection idle timeout. A gRPC body that exceeds its receive deadline returns DeadlineExceeded and releases the global admission budget.

Trace data¶

OTLP Spans are written to the tracing category with measurement opentelemetry. The start and duration fields use microseconds; the Point timestamp is the Span start time in nanoseconds.

Common OTel Resource and Span attributes are flattened, for example db.system → db_system, db.operation.name → db_operation, http.request.method → http_method, http.response.status_code → http_status_code, rpc.system → rpc_system, and messaging.system → messaging_system. String, boolean, integer, and floating-point values retain their types; arrays and objects are encoded as JSON. At most 256 attributes are processed from each attribute source. Attributes outside the semantic mapping allowlist do not become top-level fields, but remain available in the message JSON.

Semantic attribute fields¶

The table below lists the current allowlisted OTel attribute mappings. Except for http_status_code, a field preserves the input primitive type: string, bool, int, or float. Arrays and key/value objects are JSON-encoded strings.

In Kubernetes mode, Trace data can receive the same cluster, Node, Pod, Container, workload, and allowlisted label tags as container logs. Identity supplied by OTel Resource data wins; missing values are filled from the local-node Pod cache.

Kubernetes collection¶

Enable Kubernetes mode with [kubernetes].enabled = true. The DaemonSet reads local Pod metadata, Kubelet /stats/summary, and /var/log/pods on the node:

[kubernetes]
# Enables Kubernetes collection.
enabled = true
# Reports Pod and container metrics.
metrics_enabled = true
# Reports Pod and container objects.
objects_enabled = true
# Kubernetes cluster name.
cluster_name = "default"
# Local Pod metadata refresh interval in seconds.
refresh_secs = 60
# Pod and container object reporting interval in seconds.
object_interval_secs = 300
# Allowlist of Pod labels added to reported data.
label_tags = ["app.kubernetes.io/name"]

metrics_enabled and objects_enabled are independent. Metrics are collected every 60 seconds by default. Objects are sent at startup and then every 5 minutes. label_tags is an allowlist of Pod labels to add as tags, preventing uncontrolled cardinality.

rsdatakit accesses Kubelet over HTTPS with the automatically mounted ServiceAccount bearer token and relies on the nodes/stats RBAC permission for authorization. To support node-local self-signed certificates and clusters whose Kubelet certificate lacks a Node IP SAN, the Kubelet connection does not validate the server certificate. The Kubernetes API uses a separate client and continues to validate its server certificate with the ServiceAccount CA.

The supplied DaemonSet also mounts /var/log/pods, /var/lib/docker/containers, and /var/lib/rsdatakit. Container logs come from local node files; rsdatakit does not call the Kubernetes API for each log line. Pod metadata is added from a shared local-node cache.

Metrics¶

Optional fields absent from Kubelet Summary are not emitted. CPU, memory, file-system, and network counters come from the local Kubelet /stats/summary; requests, limits, readiness, restarts, and workload tags come from Pod API metadata.

kube_pod¶

One Point is emitted for every Pod on the local node.

docker_containers (Kubernetes metric)¶

One Point is emitted for every container on the local node. Container network traffic is reported at Pod level by kube_pod, so this measurement has no network fields.

Objects¶

Objects use DataKit-compatible classes. The object message is a JSON snapshot of current fields. When the Kubelet Summary is available in the cycle, the resource fields for the matching Pod or container are also carried into the object and are listed explicitly below.

kubelet_pod object¶

docker_containers object¶

Prometheus collection¶

Prometheus scraping is part of Kubernetes mode. It is disabled in the standalone binary configuration and enabled in the supplied DaemonSet example:

[kubernetes.prometheus]
# Enables local Prometheus target discovery and scraping.
enabled = true
# Scrape interval per target in seconds.
scrape_interval_secs = 30
# Timeout per scrape in seconds.
scrape_timeout_secs = 10
# Keeps the full exporter metric name as the field name.
keep_exist_metric_name = true
# Discovers Pods annotated with prometheus.io/scrape.
discover_pod_annotations = true
# Discovers Services annotated with prometheus.io/scrape.
discover_service_annotations = true
# Discovers Prometheus Operator PodMonitors.
discover_pod_monitors = true
# Discovers Prometheus Operator ServiceMonitors.
discover_service_monitors = true

Trusted explicit targets can be configured in addition to discovery:

[[kubernetes.prometheus.instances]]
# Target source type.
role = "pod" # node | pod | service | endpoints
# Namespaces containing the targets.
namespaces = ["default"]
# Kubernetes label selector.
selector = "app=nginx"
# Scrape scheme.
scheme = "http"
# Numeric or named port.
port = "metrics"
# HTTP metrics path.
path = "/metrics"
# Measurement name. Derived automatically when empty.
measurement = ""
# Tags added to all metrics from this target.
tags = { team = "platform" }

Targets can be discovered through:

• Pod/Service annotations such as prometheus.io/scrape, port, scheme, and path;
• Prometheus Operator PodMonitor and ServiceMonitor resources;
• trusted static node, pod, service, and endpoints instances.

Only targets confirmed to belong to the local node are scraped, avoiding duplicate cluster-wide scraping by DaemonSet replicas. Measurements are derived from exporter metric names or an explicit measurement setting. collector.up reports the scrape result of each target.

Disable unneeded discover_* options. When Service annotation and ServiceMonitor discovery are off and no explicit service or endpoints instance is configured, rsdatakit does not subscribe to cluster-wide Service or EndpointSlice resources.

Metric conversion rules¶

Exporter measurements and fields are generated dynamically from metric names and cannot be listed like fixed host metrics:

Every scrape also emits collector status:

collector¶

Run and verify¶

sudo RUST_LOG=rsdatakit=info rsdatakit --config /etc/rsdatakit/rsdatakit.toml

sudo systemctl restart rsdatakit
sudo systemctl status rsdatakit
sudo journalctl -u rsdatakit -f

kubectl -n rsdatakit-system rollout status daemonset/rsdatakit
kubectl -n rsdatakit-system get pods -l app=rsdatakit
kubectl -n rsdatakit-system logs daemonset/rsdatakit --tail=100

After rsdatakit started appears in the startup log, verify these data sets in the workspace:

After configuring the standalone health address or enabling OTLP HTTP, inspect health with, for example:

curl http://127.0.0.1:4319/health

degraded indicates a current Dataway, WAL, or collector fault. HTTP 503 indicates unhealthy queue/WAL pressure or a stopped background task.

Notes¶

• Logs and traces enter WAL and retry after network errors, HTTP 5xx, or HTTP 429. Other 4xx responses are not retried. Metrics and objects are periodic snapshots and are not stored in WAL.
• Size wal.max_bytes for the log/Trace volume and acceptable Dataway outage window.
• Inject Dataway and OTLP tokens through a Secret or restricted configuration file; never commit them.
• Allowlist only required Pod labels in kubernetes.label_tags to avoid high-cardinality tags.
• RUST_LOG=rsdatakit=debug produces substantial logs and should be used only for short diagnostics.