DataKit Operator¶

DataKit Operator uses a Kubernetes Admission Webhook to inject tracing, log collection, and profiling components into newly created Pods. It also provides in-cluster Pod query and DataKit central election coordination APIs.

Overview¶

DataKit Operator provides the following features:

Injection occurs during the Pod CREATE phase and does not modify running Pods. After changing the Operator configuration, an injection image, or workload Annotations, recreate the Pod for the change to take effect.

Admission is fail-open: if injection fails, the Operator logs the error but does not block creation of the application Pod. The webhook in the deployment manifest also uses failurePolicy: Ignore.

Prerequisites¶

• Kubernetes must support admissionregistration.k8s.io/v1; Kubernetes v1.24 or later is recommended.
• The MutatingAdmissionWebhook Admission Controller must be enabled in the cluster.
• Cluster nodes must be able to pull the configured images. In offline environments, synchronize the images to a private registry in advance.

Installation¶

kubectl create namespace datakit
wget https://static.truewatch.com/datakit-operator/datakit-operator.yaml
kubectl apply -f datakit-operator.yaml
kubectl get pod -n datakit

NAME                                READY   STATUS    RESTARTS   AGE
datakit-operator-f948897fb-5w5nm    1/1     Running   0          15s

helm install datakit-operator datakit-operator \
    --repo https://pubrepo.truewatch.com/chartrepo/truewatch \
    -n datakit --create-namespace

helm -n datakit list

helm -n datakit get values datakit-operator -a -o yaml > values.yaml
helm upgrade datakit-operator datakit-operator \
    --repo https://pubrepo.truewatch.com/chartrepo/truewatch \
    -n datakit \
    -f values.yaml

helm uninstall datakit-operator -n datakit

Configuration¶

The Operator uses JSON configuration. Deployment manifests generally store the configuration in a ConfigMap and load it through the ENV_JSON_CONFIG environment variable.

admission_inject_v2 is used starting with v1.8.0:

{
    "server_listen": "0.0.0.0:9543",
    "log_level": "info",
    "admission_inject_v2": {
        "ddtraces": [],
        "otels": [],
        "logfwds": [],
        "flameshots": [],
        "profilers": []
    },
    "admission_mutate": {
        "loggings": []
    }
}

The preceding example shows only the configuration structure. The distributed deployment templates retain one Java DDTrace rule that matches Pods in the default Namespace; otels is empty, so OpenTelemetry injection is not enabled automatically. This default is a runtime policy that preserves compatibility with existing deployments and does not mean that the Operator supports only Java.

The Operator does not detect the language in application containers automatically. DDTrace also supports Python, PHP, and Node.js, while OpenTelemetry supports Java, Python, and Node.js. To enable these capabilities, follow the DDTrace automatic injection and OpenTelemetry automatic injection documentation and add rules that use mutually exclusive language labels.

The legacy admission_inject configuration remains supported. A valid ddtrace, logfwd, or profiler entry in the legacy configuration overrides the corresponding v2 rules. Do not maintain two valid configurations for the same feature during an upgrade.

Cluster API¶

DataKit Operator v1.8.1 and later provide the Cluster API. This API uses the Operator's Pod informer cache to proxy Pod data queries, reducing direct access to the API Server from each DataKit instance.

The Cluster API is enabled by default. At startup, the Operator checks whether its ServiceAccount has permission to read Pods. If permission is insufficient, the related routes are not registered. Before the Pod cache is synchronized, or if synchronization fails, the API returns 503, while other features continue to run. The minimum required permissions are:

- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

The API uses the Operator Service. Its default address is https://datakit-operator.datakit.svc:443:

Starting with v1.8.9, use view=ebpf-v1 to obtain a reduced Pod representation for eBPF:

curl -k "https://datakit-operator.datakit.svc:443/v1/cluster/api/v1/pods?view=ebpf-v1"

DataKit Central Election¶

Starting with DataKit Operator v1.9.1, DataKit Operator can replace DataWay/Kodo as the central election service for DataKit. This replaces only the election service; collected data is still uploaded through DataWay.

This feature is available only in Kubernetes. DataKit Operator and DataKit must run in the same Kubernetes cluster.

Before enabling the feature, note:

1. Compatible versions: DataKit Operator v1.9.1 or later and DataKit 2.12.0 or later are required. Configure the environment variables manually.
2. Lease and RBAC permissions: DataKit Operator stores election state in Kubernetes Leases and requires the corresponding read/write permissions. Lease is a native Kubernetes resource. No CRD installation or manual Lease creation is needed; DataKit Operator creates Leases on demand.

Enable Central Election¶

First upgrade DataKit Operator and grant the required permissions. Then configure the following environment variables consistently on every DataKit in the same election group and restart DataKit:

- name: ENV_ENABLE_ELECTION
  value: "true"
- name: ENV_ELECTION_OPERATOR_URL
  value: "https://datakit-operator.datakit.svc:443"

The example uses the default Service and namespace. Adjust the address for a custom deployment.

DataKit decides whether to use DataKit Operator only at startup. If the address is unset, the version is unsupported, permissions are missing, or the service is temporarily unavailable, DataKit continues to use DataWay/Kodo election. It does not switch services at runtime. Restart DataKit to reassess after changing the configuration or restoring DataKit Operator.

Before switching election services, confirm that DataKit Operator's election feature is available. Stop all DataKits in the same election group, apply the same configuration, and then start them again. Check their startup logs to confirm that they all use the same election service, avoiding duplicate collection caused by mixing services during a normal rolling update.

Add Permissions to an Existing Deployment¶

The new default YAML and Helm Chart include the Lease Role/RoleBinding. Existing deployments do not need to replace the entire YAML: after upgrading DataKit Operator, apply the following RBAC separately. Save it as datakit-operator-election-rbac.yaml. For a custom namespace or ServiceAccount, adjust metadata.namespace, subjects.namespace, and subjects.name. Leases are created in the Kubernetes namespace where DataKit Operator runs.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: datakit-operator-election
  namespace: datakit
rules:
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: datakit-operator-election
  namespace: datakit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: datakit-operator-election
subjects:
- kind: ServiceAccount
  name: datakit-operator
  namespace: datakit

kubectl apply -f datakit-operator-election-rbac.yaml
kubectl -n datakit rollout restart deployment/datakit-operator
kubectl -n datakit rollout status deployment/datakit-operator

After granting the permissions, configure and restart DataKit as described above. A missing Lease object is not an error. Missing Lease permissions make central election unavailable but do not affect other DataKit Operator services.

Injection Rules¶

Every injection configuration must first match a Pod through selectors. An Annotation can reject injection or, when check_annotation is enabled, further restrict injection. It cannot trigger injection independently of selectors.

Selector Configuration¶

namespace_selectors and label_selectors are both arrays:

• Multiple selectors in the same array are matched with OR.
• When both arrays are configured, both the Namespace and Label dimensions must match.
• When an injection rule configures only one dimension, only that dimension is used. If neither dimension is configured, the current rule is disabled and evaluation continues with subsequent rules.
• Logging mutation rules require both dimensions. If either dimension is missing, the current rule is disabled.

namespace_selectors uses Go regular expressions. The string "*" is shorthand for matching all Namespaces; use ^ and $ for exact matches. label_selectors uses Kubernetes Label Selector syntax and extends =, ==, and != with glob matching.

Empty strings, whitespace-only values, invalid Namespace regular expressions, and invalid or empty-constraint Label selectors are invalid entries. At startup, the Operator logs a warning and ignores each invalid entry. If a configured dimension has no valid entries left, the current rule is disabled and evaluation continues with subsequent rules. Use "*" to explicitly match all Namespaces.

The following rule matches only Pods in the production Namespace that have the admission.datakit/ddtrace-language=java label:

{
    "namespace_selectors": ["^production$"],
    "label_selectors": ["admission.datakit/ddtrace-language=java"]
}

The same Pod may match multiple DDTrace or OTel rules. The Operator selects the first rule in configuration order that satisfies the Annotation conditions. Once selected, it does not fall back to subsequent rules because of an invalid language, image, or other configuration error. Use mutually exclusive labels for rules targeting different languages.

When DDTrace and OTel both match, DDTrace takes precedence. If DDTrace injection fails, the Operator does not fall back to OTel.

Annotation Configuration¶

Add Annotations to the Pod or to .spec.template.metadata.annotations in a controller such as a Deployment.

These switch Annotations use permissive semantics: only a value that can be parsed as false disables the corresponding feature. A missing or unparseable Annotation is treated as true. Even when explicitly set to true, the Pod must still match the corresponding configuration rule.

metadata:
  annotations:
    admission.datakit/ddtrace.enabled: "false"
    admission.datakit/otel.enabled: "true"

check_annotation¶

DDTrace, OTel, logfwd, and legacy Profiler rules support check_annotation:

The mappings are:

When check_annotation: true and the Pod provides the corresponding version Annotation, DDTrace, OTel, and Profiler replace the tag in the rule's image without changing the image registry or name. Feature switch Annotations always apply, regardless of check_annotation.

Image Pull Policy¶

DDTrace, OTel, logfwd, Flameshot, and Profiler rules under admission_inject_v2 support image_pull_policy alongside image. Valid values are Always, IfNotPresent, and Never, with exact casing. Missing, empty, or invalid values (including incorrect JSON value types) fall back to Always; invalid values produce a warning.

For example, configure the corresponding rule in admission_inject_v2.ddtraces as follows:

{
    "name": "ddtrace-java",
    "language": "java",
    "namespace_selectors": ["default"],
    "image": "pubrepo.truewatch.com/datakit-operator/dd-lib-java-init:v1.60.4-ext",
    "image_pull_policy": "IfNotPresent"
}

IfNotPresent reuses the local image when it is already present on the node; Never requires the image to be available on the node in advance. Use fixed versions to avoid reusing an old cached image with a mutable tag. This setting controls newly injected containers only. Helm's image.pullPolicy still controls only the Operator's own image.

After changing the configuration, restart the Operator and recreate the application Pods. Existing containers keep their pull policies. The deprecated admission_inject retains the default Always; to use the new setting, remove any active legacy configuration that overrides the corresponding v2 rules.

Supported Injection Features¶

Environment Variable Value References¶

The envs field in an injection rule supports literal values and placeholders that are converted to native Kubernetes fieldRef, resourceFieldRef, and secretKeyRef references.

Environment variables preserve configuration order. Later values can reference earlier variables using Kubernetes $(VAR) syntax:

{
    "envs": {
        "POD_NAME": "{fieldRef:metadata.name}",
        "POD_NAMESPACE": "{fieldRef:metadata.namespace}",
        "RESOURCE_TAGS": "pod_name=$(POD_NAME),pod_namespace=$(POD_NAMESPACE)"
    }
}

Unrecognized placeholders are injected as ordinary strings. resourceFieldRef references only the first application container. If that container does not declare the corresponding request or limit, the environment variable is not injected.

{secretKeyRef:*}¶

The Secret reference format is:

{secretKeyRef:<secret-name>.<key>}

For example:

{
    "envs": {
        "ACCESS_KEY_ID": "{secretKeyRef:flameshot-oss.access_key_id}",
        "ACCESS_KEY_SECRET": "{secretKeyRef:flameshot-oss.access_key_secret}"
    }
}

The Operator does not read the Secret; Kubernetes resolves the reference when the container starts. The Secret must be in the same Namespace as the application Pod. If the Secret or key does not exist, the Pod enters CreateContainerConfigError. Because . separates the Secret name from the key, the Secret name in this format cannot contain ..

FAQ¶

Disable Pod Changes¶

Add the following to the Pod template:

admission.datakit/enabled: "false"

Why Did Injection Not Take Effect?¶

Check the following in order:

1. Whether the Pod was recreated after the configuration update.
2. Whether its Namespace and Label match the same rule.
3. Whether admission.datakit/enabled or a feature switch Annotation is false.
4. When check_annotation: true, whether the correct version or configuration Annotation is present.
5. Whether the Operator logs contain warnings about selectors, images, environment variables, or security context conflicts.

What Should I Consider on AWS EKS?¶

The EKS control plane must be able to access port 9543 on the Operator webhook. If injection does not take effect, check whether the cluster security groups and network policies allow traffic in that direction.