OpenLIT OTel GPU Collector

The OpenLIT OTel GPU Collector reports GPU hardware, GPU process, host, and Linux/NVIDIA CUDA eBPF metrics to DataKit through OpenTelemetry OTLP.

Configuration¶

Prerequisites¶

• DataKit is installed, and the OTLP network path from the OpenLIT Collector to DataKit is reachable.
• For NVIDIA GPU containers, install the NVIDIA driver, Docker, and NVIDIA Container Toolkit on the host. Map /dev/kfd and /dev/dri for AMD, or /dev/dri for Intel.
• Process-level GPU attribution requires the container to use the host PID namespace. Linux/NVIDIA CUDA eBPF metrics also require BPF/PERFMON capabilities and lockable memory. Without these permissions, device-level hw.gpu.* metrics still work, but the process and eBPF views are incomplete.

Enable the DataKit OpenTelemetry Receiver¶

cd /usr/local/datakit/conf.d
sudo cp samples/opentelemetry.conf.sample opentelemetry.conf
sudo vim opentelemetry.conf

[[inputs.opentelemetry]]
  customer_tags_all = true

  [inputs.opentelemetry.grpc]
    addr = "127.0.0.1:4317"
    max_payload = 16777216

When the Collector and DataKit run on different hosts or do not use the host network, change addr to 0.0.0.0:4317, set the Collector endpoint to port 4317 of DataKit's actual private IP address or DNS name, and restrict sources through a firewall or security group. Do not expose port 4317 to the public network without protection.

Restart DataKit:

sudo datakit service -R

Start the OpenLIT OTel GPU Collector¶

The following example uses an NVIDIA GPU with the Collector and DataKit on the same host:

docker pull ghcr.io/openlit/otel-gpu-collector:latest

docker run -d \
  --name otel-gpu-collector \
  --restart unless-stopped \
  --network host \
  --gpus all \
  --pid=host \
  --cap-add BPF \
  --cap-add PERFMON \
  --ulimit memlock=-1:-1 \
  -e OTEL_SERVICE_NAME=openlit-otel-gpu-collector \
  -e OTEL_RESOURCE_ATTRIBUTES='deployment.environment=production,team=ml,host.name=gpu-host-01' \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
  -e OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
  -e OTEL_METRIC_EXPORT_INTERVAL=15000 \
  -e OTEL_GPU_EBPF_ENABLED=true \
  ghcr.io/openlit/otel-gpu-collector:latest

For AMD containers, replace --gpus all with --device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri. For Intel, use --device /dev/dri:/dev/dri. AMD and Intel do not use the CUDA eBPF path, so empty CUDA-specific views are an expected capability limitation.

Kubernetes Deployment¶

OpenLIT officially supports running one Collector on each GPU node as a DaemonSet. This integration does not reproduce the complete YAML, which may change between versions. Refer directly to the official OpenLIT Kubernetes DaemonSet configuration, and point OTEL_EXPORTER_OTLP_ENDPOINT to a DataKit or OpenTelemetry Collector service reachable from the cluster.

For Kubernetes deployments, retain hostPID: true, GPU device access, the kubelet PodResources mount, and the BPF permissions required by eBPF. To derive host.type, cloud provider, and region from Node labels or providerID, grant the ServiceAccount the nodes/get permission as shown in the official example. Refer to the official documentation for identity configuration differences among EKS, GKE, AKS, and self-managed clusters.

Verification¶

1. Query otel_service under Metrics in TrueWatch, and confirm that hw.gpu.up, hw.gpu.utilization, hw.gpu.memory.usage, and other metrics are continuously reported.
2. If the process or CUDA eBPF charts have no data, check the host PID setting, GPU device mappings, BPF/PERFMON capabilities, memlock, whether the workload has loaded libcudart, and whether host security policies block eBPF uprobes.

Metrics¶

OpenLIT metrics are written to otel_service through OTLP. The following table covers GPU hardware, host system, Collector process, GPU workload process, and CUDA eBPF metrics from the official OpenLIT OTel GPU Collector Metrics Reference.