eBPF

eBPF collector, collecting host network TCP, UDP connection information, Bash execution log, etc. This collector mainly includes ebpf-net, ebpf-conntrack and ebpf-bash three plugins:

• ebpf-net:

  • Data category: Network
  • It is composed of netflow, httpflow and dnsflow, which are used to collect host TCP/UDP connection statistics and host DNS resolution information respectively;

• ebpf-bash:

  • Data category: Logging
  • Collect Bash execution log, including Bash process number, user name, executed command and time, etc.;

• ebpf-conntrack: Version-1.8.0 · Experimental

  • Add two tags dst_nat_ip and dst_nat_port to the network flow data.

• ebpf-trace:

  • Application call relationship tracking.

• bpf-netlog:

  • Data categories: Logging, Network
  • This plugin implements the collection of network logs bpf_net_l4_log/bpf_net_l7_log, and can also replace ebpf-net's netflow/httpflow data collection when the kernel does not support eBPF;

Configuration¶

Preconditions¶

For DataKit before v1.5.6, you need to execute the installation command to install:

• v1.2.13 ~ v1.2.18
  • Install time specify environment variable：DK_INSTALL_EXTERNALS="datakit-ebpf"
  • After the DataKit is installed, manually install the eBPF collector: datakit install --datakit-ebpf
• v1.2.19+
  • specify environment variable：DK_INSTALL_EXTERNALS="ebpf" when installing
  • After the DataKit is installed, manually install the eBPF collector: datakit install --ebpf
• v1.5.6+
  • No manual installation required

When deploying in Kubernetes environment, you must mount the host's' /sys/kernel/debug directory into pod, refer to the latest datakit.yaml;

Linux Kernel Version Requirement¶

Native Linux i386/i686 kernels use the linux/386 DataKit and datakit-ebpf binaries. The 32-bit collector checks the kernel architecture at startup. Use linux/amd64 on an x86_64 kernel. The kernel must enable eBPF, kprobes and perf events, and the collector requires root privileges.

i386 network statistics cover IPv4/IPv6 TCP and UDP, RTT, retransmits, connection failures, processes, network namespaces and DNS association. UDP is disabled by default. Set enable_udp = true in [[inputs.ebpf]] or use ENV_INPUT_EBPF_ENABLE_UDP=true. DNAT association also requires ebpf-conntrack and the corresponding kernel modules.

At startup, complete BTF fields from the running kernel at /sys/kernel/btf/vmlinux directly provide structure offsets without requiring private-socket probe events. External BTF, cached candidates and incomplete layouts still require validation against actual connections. The offset cache is bound to the complete kernel release and architecture and is resolved again after a kernel update. Failure to confirm required base offsets reports a startup error in the collector log as well as stderr. NAT and HTTP offsets reuse the same BTF parse; complete fields from the current kernel do not trigger additional offset guessing. Without BTF, L7 sequence offsets use the existing bounded private-socket validation instead of assuming their positions relative to RTT fields. Confirmed diagnostic fields are reused without repeating the same challenges. An unavailable optional L7 sequence offset disables that application capability while verified netflow collection continues. Private test sockets, packet rings and temporary probes are released after startup validation.

In addition to CentOS 7.6+ and Ubuntu 16.04, other distributions recommend that the Linux kernel version is higher than 4.9, otherwise the eBPF collector may not start.

If you want to enable the eBPF-conntrack plugin, usually requires a higher kernel version, such as v5.4.0 etc., please confirm whether the symbols in the kernel contain nf_ct_delete and __nf_conntrack_hash_insert, you can execute the following command to view:

cat /proc/kallsyms | awk '{print $3}' | grep "^nf_ct_delete$\|^__nf_conntrack_hash_insert$"

SELinux-enabled System¶

For SELinux-enabled systems, you need to shut them down (pending subsequent optimization), and execute the following command to shut them down:

setenforce 0

Collector Configuration¶

[[inputs.ebpf]]
  daemon = true
  name = 'ebpf'
  cmd = "/usr/local/datakit/externals/datakit-ebpf"
  args = [
    "--datakit-apiserver", "0.0.0.0:9529",
  ]
  envs = []

  ## Resource limits.
  ## The collector automatically exits when the limit is exceeded.
  ## Can configure the number of cpu cores, memory size and network bandwidth.
  ##
  # cpu_limit = "2.0"
  # mem_limit = "4GiB"
  # net_limit = "100MiB/s"

  ## automatically takes effect when running DataKit in 
  ## Kubernetes daemonset mode
  ##
  # kubernetes_url = "https://kubernetes.default:443"
  # bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
  ##
  ## or 
  # bearer_token_string = "<your-token-string>"

  ## k8s workload labels
  ##
  # workload_labels = ["app"]  
  # workload_label_prefix = ""

  ## operator url
  ##
  # operator_url = "https://datakit-operator.datakit.svc:443"

  ## all supported plugins:
  ## - "ebpf-net"  :
  ##     contains L4-network(netflow), L7-network(httpflow, dnsflow) collection
  ## - "ebpf-bash" :
  ##     log bash
  ## - "ebpf-conntrack":
  ##     add two tags "dst_nat_ip" and "dst_nat_port" to the network flow data
  ## - "ebpf-trace":
  ##     param trace_server must be set simultaneously.
  ## - "bpf-netlog":
  ##     contains L4-network log (bpf_net_l4_log), L7-network log (bpf_net_l7_log), 
  ##              L4-network(netflow), L7-network(httpflow, dnsflow) collection
  enabled_plugins = [
    "ebpf-net",
  ]

  ## Collect UDP flows in addition to TCP.
  enable_udp = false

  ## Network diagnostic metrics and bounded structured problem logs.
  ## These use their own budgets, independent of netflow sampling.
  network_diagnostics = true
  ## Default: 40 node metric timelines with stable global tags.
  ## Full metrics add collector fields and per-netns/interface timelines.
  network_diagnostics_full_metrics = false
  # network_diagnostics_max_groups = 4096
  ## Both limits apply; allow four minutes of burst tokens.
  # network_diagnostics_log_bytes_per_minute = 524288
  # network_diagnostics_log_points_per_minute = 240

  ## If you enable the ebpf-net plugin, you can configure:
  ##  - "httpflow" (* enabled by default)
  ##  - "httpflow-tls"
  ##
  l7net_enabled = [
    "httpflow",
    # "httpflow-tls"
  ]


  ## datakit-ebpf pprof service
  pprof_host = "127.0.0.1"
  pprof_port = "6061"

  ## netlog blacklist
  ##
  # netlog_blacklist = "ip_saddr=='127.0.0.1' || ip_daddr=='127.0.0.1'"

  ## bpf-netlog plugin collection metric and log
  ##
  netlog_metric = true
  netlog_log = false

  ## bpf-netlog L7 log protocols.
  ## "http" enables both HTTP/1.x and HTTP/2. Use "http1" only when HTTP/1.x is needed alone.
  ## Available values: "http", "http1", "http2", "grpc".
  ##
  # netlog_l7log_protocols = ["http"]

  ## bpf-netlog L7 log HTTP headers.
  ## Empty uses the built-in recommended allowlist, ["none"] disables header capture.
  ##
  # netlog_l7log_headers = [
  #   "host",
  #   "x-request-id",
  #   "traceparent",
  # ]

  ## bpf-netlog AF_PACKET capture ring limits.
  ## In Kubernetes, datakit-ebpf uses smaller defaults to reduce resident memory.
  ##
  # netlog_fallback_sockets = 0
  # netlog_fallback_blocks = 0
  # netlog_shared_blocks = 0

  ## Advanced datakit-ebpf capacity tuning can be added to envs above.
  ## Prefer changing these only when self-monitoring shows drops caused by limits.
  ##
  ##   "DK_EBPF_NETFLOW_MAP_MAX_ENTRIES=131072",
  ##   "DK_EBPF_L4LOG_CONN_CACHE_LIMIT=131072",
  ##   "DK_EBPF_L7FLOW_CONN_MAP_LIMIT=131072",

  ## eBPF trace generation server center address.
  trace_server = ""

  ## trace all processes containing any specified environment variable
  trace_env_list = [
    # "DK_BPFTRACE_SERVICE",
    # "DD_SERVICE",
    # "OTEL_SERVICE_NAME",
  ]

  ## deny tracking any process containing any specified environment variable
  trace_env_blacklist = []

  ## trace all processes containing any specified process names,
  ## can be used with trace_namedenyset
  ##
  trace_name_list = []

  ## deny tracking any process containing any specified process names
  ##
  trace_name_blacklist = [

    ## The following two processes are hard-coded to never be traced,
    ## and do not need to be set:
    ##
    # "datakit",
    # "datakit-ebpf",
  ]

  ## conv other trace id to datadog trace id (base 10, 64-bit) 
  conv_to_ddtrace = false

  ## If the system does not enable ipv6, it needs to be changed to true
  ##
  ipv6_disabled = false

  ## ephemeral port strart from <ephemeral_port>
  ##
  # ephemeral_port = 10001

  ## Dynamic network path discovery.
  ## datakit-ebpf only reports candidates; DataKit netpath runs traceroute
  ## and uploads netpath points.
  ##
  # network_path_enabled = false
  # network_path_api = "http://127.0.0.1:9529/v1/netpath/candidates"
  # network_path_token = ""
  # network_path_flush_interval = "10s"
  # network_path_batch_size = 100
  # network_path_http_timeout = "3s"
  # network_path_queue_size = 4096

  # interval = "60s"

  # sampling_rate = "0.50"
  # sampling_rate_pts_per_min = "1500"

  [inputs.ebpf.tags]
    # some_tag = "some_value"
    # more_tag = "some_other_value"

#############################
## Parameter description (if marked * is required)
#############################
##  --hostname               : Host name, this parameter can change the value of the host tag when the collector uploads data, the priority is: specify this parameter >
##                             ENV_HOSTNAME value in datakit.conf (if it is not empty, this parameter will be added automatically at startup) >
##                             collector Get it yourself (the default).
##  --datakit-apiserver      : DataKit API Server address, default value 0.0.0.0:9529 .
##  --log                    : Log output path, default <DataKitInstallDir>/externals/datakit-ebpf.log.
##  --log-level              : Log level, the default value is 'info'.
##  --service                : The default value is 'ebpf'.

- name: ENV_ENABLE_INPUTS
       value: cpu,disk,diskio,mem,swap,system,hostobject,net,host_processes,container,ebpf

Environment variables and configuration items¶

The following environment variables can be used to adjust the eBPF collection configuration in Kubernetes:

Configuration items:

• enabled_plugins:

  • Description: Used to configure the built-in plugins for the collector
  • Environment variable: ENV_INPUT_EBPF_ENABLED_PLUGINS
  • Example: ebpf-net,ebpf-trace

• l7net_enabled

  • Description: Enable http protocol data collection
  • Environment variable: ENV_INPUT_EBPF_L7NET_ENABLED
  • Example: httpflow

• interval

  • Description: Set the sampling time interval
  • Environment variable: ENV_INPUT_EBPF_INTERVAL
  • Example: 1m30s

The interval setting controls aggregation for ebpf-net netflow/httpflow/dnsflow and bpf-netlog netflow/httpflow; its default and minimum are 60s. DNS request matching timeouts are independent of this interval. Counts and total latency accumulate directly, with the average weighted by the number of requests rather than by averaging group averages.

DNSFlow uses one packet socket in the collector network namespace to observe visible plaintext DNS traffic. It does not open a socket in each Pod namespace or filter by Kubernetes namespace. Traffic from multiple business namespaces may be visible on the host; traffic confined to interfaces outside that namespace is not guaranteed to be covered. Aggregation keys remain in client/server order, and the local observation side is resolved at export so address discovery during a window does not split its statistics.

DNS enrichment matches the client IP and answer address within the observation scope. Answers captured on the host can also match a Pod socket by the original client IP; the capture netns is not the socket netns. There is no arbitrary-namespace or global IP fallback; cross-scope matching excludes loopback, unspecified, and link-local client addresses. TCP DNS candidates are matched as of connection creation or initial adoption and retained with that socket generation beyond TTL expiry. TTL starts at packet capture time. Initially unresolved bindings retry only on existing report periods, accepting earlier answers processed late while rejecting answers observed after the connection. The cache keeps expired metadata for only one additional cleanup interval (one minute) for these historical lookups; current-time lookups still enforce TTL. It does not retain complete DNS history. Existing stronger application-level enrichment is unchanged. Incoming server_domain uses the local server endpoint; dst_domain continues to describe the raw remote endpoint and cannot stand in for the incoming server domain. Retiring the connection releases the candidate, and new connections consult the current cache. UDP retains observation-time enrichment. Conflicting DNS candidates in one aggregation bucket leave its DNS domain unset without adding dimensions or points. This is DNS inference, not proof of the application hostname; earlier, invisible, or encrypted DNS cannot be reconstructed.

DNSFlow separates routine summaries from diagnostic detail while retaining the dnsflow measurement and existing field types:

A <nil> client namespace or Pod means that Kubernetes metadata was not matched; it does not identify node traffic by itself. A node client IP can also represent hostNetwork traffic or a DNS forwarder after SNAT. Use client_ip, server_ip and available Pod tags to distinguish the observed legs; client, forwarding and upstream observations of one resolution must not be summed as unique application queries.

The three scopes have disjoint counts: abnormal requests are not counted again in routine summaries. Fast NXDOMAIN remains a routine result with its own rcode=3 summary; slow NXDOMAIN retains diagnostic detail. Abnormal observations also continue through the existing ebpf_network_event channel when enabled, under its existing log budgets. Disabling that channel does not remove query scope DNSFlow data. This change adds no capture socket, proc scan or namespace switch. Service attribution uses existing Kubernetes metadata in memory, cached within the aggregation window under the DNS aggregate entry limit. Reverse domain enrichment continues to consume original DNS answers.

• ipv6_disabled

  • Description: Whether the system does not support IPv6
  • Environment variable: ENV_INPUT_EBPF_IPV6_DISABLED
  • Example: false

• ephemeral_port

  • Description: Ephemeral port threshold used only for UDP direction inference. TCP roles require connection or protocol evidence, rather than port magnitude. Netflow/httpflow aggregates with a known role roll up the client port to * and preserve the server port. Unknown roles roll up both ports to *, retain address pairs and traffic totals, and omit derived client/server fields. Connection details and local diagnostics retain the original ports.
  • Environment variable: ENV_INPUT_EBPF_EPHEMERAL_PORT
  • Example: 32768

• pprof_host

  • Description: pprof host
  • Environment variable: ENV_INPUT_EBPF_PPROF_HOST
  • Example: 127.0.0.1

• pprof_port

  • Description: pprof port
  • Environment variable: ENV_INPUT_EBPF_PPROF_PORT
  • Example: 6061

• trace_server

  • Description: The address of DataKit ELinker/DataKit to enable the ebpftrace collector
  • Environment variable: ENV_INPUT_EBPF_TRACE_SERVER
  • Example: <ip>:<port>

• trace_name_blacklist

  • Description: The process with the specified process name will be disabled from collecting trace data
  • Environment variable: ENV_INPUT_EBPF_TRACE_NAME_BLACKLIST
  • Example:

• trace_env_blacklist

  • Description: Any process containing any of the specified environment variable names will be disabled from collecting trace data
  • Environment variable: ENV_INPUT_EBPF_TRACE_ENV_BLACKLIST
  • Example: DKE_DISABLE_ETRACE

• trace_env_list

  • Description: Link data for processes with any specified environment variables will be traced and reported
  • Environment variable: ENV_INPUT_EBPF_TRACE_ENV_LIST
  • Example: DK_BPFTRACE_SERVICE,DD_SERVICE,OTEL_SERVICE_NAME

• trace_name_list

  • Description: Processes whose names are in the specified set will be traced and reported
  • Environment variable: ENV_INPUT_EBPF_TRACE_NAME_LIST
  • Example: chrome,firefox

• conv_to_ddtrace

  • Description: Convert all application side link IDs to decimal strings for compatibility purposes, not used unless necessary
  • Environment variable: ENV_INPUT_EBPF_CONV_TO_DDTRACE
  • Example: false

• netlog_blacklist

  • Description: Used to filter packets after packet capture
  • Environment variable: ENV_INPUT_EBPF_NETLOG_BLACKLIST
  • Example: ip_saddr=='127.0.0.1' \|\| ip_daddr=='127.0.0.1'

• netlog_metric

  • Description: Collect network metrics from network packet analysis
  • Environment variable: ENV_INPUT_EBPF_NETLOG_METRIC
  • Example: true

• netlog_log

  • Description: Collect network logs from network packet analysis
  • Environment variable: ENV_INPUT_EBPF_NETLOG_LOG
  • Example: false

• netlog_l7log_protocols

  • Description: Set the L7 log protocols parsed by bpf-netlog. http enables both HTTP/1.x and HTTP/2; use http1 only when HTTP/1.x is needed alone. Available values: http, http1, http2, grpc
  • Environment variable: ENV_INPUT_EBPF_NETLOG_L7LOG_PROTOCOLS
  • Example: http,http2,grpc

• netlog_l7log_headers

  • Description: Set the HTTP headers recorded in bpf-netlog L7 logs. Empty uses the built-in recommended allowlist; none disables header capture. Sensitive headers are always ignored
  • Environment variable: ENV_INPUT_EBPF_NETLOG_L7LOG_HEADERS
  • Example: host,x-request-id,traceparent

• netlog_fallback_sockets

  • Description: Set the maximum number of bpf-netlog fallback network namespace capture sockets. Fallback capture is disabled by default in Kubernetes and can be raised when needed
  • Environment variable: ENV_INPUT_EBPF_NETLOG_FALLBACK_SOCKETS
  • Example: 8

• netlog_fallback_blocks

  • Description: Set the number of AF_PACKET ring blocks for bpf-netlog fallback capture
  • Environment variable: ENV_INPUT_EBPF_NETLOG_FALLBACK_BLOCKS
  • Example: 4

• netlog_shared_blocks

  • Description: Set the number of AF_PACKET ring blocks for bpf-netlog shared host-peer capture
  • Environment variable: ENV_INPUT_EBPF_NETLOG_SHARED_BLOCKS
  • Example: 64

• cpu_limit

  • Description: The maximum number of CPU cores used per unit time. When the upper limit is reached, the collector exits.
  • Environment variable: ENV_INPUT_EBPF_CPU_LIMIT
  • Example: "2.0"`

• mem_limit

  • Description: Memory size usage limit
  • Environment variable: ENV_INPUT_EBPF_MEM_LIMIT
  • Example: "4GiB"

• net_limit

  • Description: Network bandwidth (any network card) limit
  • Environment variable: ENV_INPUT_EBPF_NET_LIMIT
  • Example: "100MiB/s"

• sampling_rate

  • Description: The sampling rate when the eBPF collector reports data, ranging from 0.01 to 1.00; Mutually exclusive with the samping_rate_pts_per_min setting
  • Environment variable: ENV_INPUT_EBPF_SAMPLING_RATE
  • Example: 0.50

• sampling_rate_pts_per_min

  • Description: Set the data volume threshold per minute when the eBPF collector reports data, and dynamically adjust the sampling rate
  • Environment variable: ENV_INPUT_EBPF_SAMPLING_RATE_PTSPERMIN
  • Example: 1500

• workload_labels

  • Description: Set all specified labels of the K8s workload to be added to the data
  • Environment variable: ENV_INPUT_EBPF_WORKLOAD_LABELS
  • Example: app,project_id

• workload_label_prefix

  • Description: Add a prefix to the k8s workload label
  • Environment variable: ENV_INPUT_EBPF_WORKLOAD_LABEL_PREFIX
  • Example: k8s_workload_label_

• operator_url

  • Description: Sets the operator address used to collect workload tags from the Kubernetes cluster.
  • Environment variable: ENV_INPUT_EBPF_OPERATOR_URL
  • Example: https://datakit-operator.datakit.svc:443

Advanced Collection Limits¶

The following are advanced capacity parameters for the datakit-ebpf process. They are not recommended for routine tuning. Increase them through inputs.ebpf.envs or DaemonSet environment variables only when self-monitoring metrics or the summary log show pressure signals such as limit, queue_full, pending_query_limit, or bpf_map_fill_ratio staying close to 1.

DataKit forwards DK_EBPF_* environment variables to datakit-ebpf; explicit entries in inputs.ebpf.envs take precedence for the same key. Connection map capacity is applied when the collector starts and still defaults to 65536. After configuring a larger capacity, verify the effective capacity / max_entries for the relevant maps in self-monitoring; support for tuning or an edited configuration does not resize a running map.

For bpf-netlog, the HTTP/HTTP2 limits above constrain unfinished transactions. A batch of completed transactions wakes the consumer without waiting for the 8-second maintenance interval. All connections in one capture instance share a 16 MiB allowance for completed data awaiting log encoding or aggregation, released after consumption. This accounts for retained structures, strings and request/response headers; it is not a process RSS limit. If prolonged output blocking exhausts the allowance, completed-record eviction is included in collection-quality counters with reason completed_budget.

HTTP/2 supports negotiated HPACK dynamic-table size changes. In-band table-size updates drive decoding and dictionary eviction, so cross-direction syscall completion order does not impose a second negotiation state machine; see RFC 9113 and RFC 7541. Dynamic-table decoding is bounded to 64 KiB per direction. A larger advertisement alone does not allocate an equally large table; an encoded update exceeding the bound stops decoding that connection and increments collection-quality counters. httpflow exports a complete transaction after both request and response sides end. Cancellation and GOAWAY rejection release unfinished state without generating a successful transaction.

For RST_STREAM observed before the corresponding request event, HTTP/2 retains a terminal marker in the existing connection index until that request is observed. Markers do not consume the allowance for unfinished transactions; their count is bounded by the same stream setting. Exhausting that marker allowance stops decoding the connection and records terminal_limit in collection-quality counters. Server-push transactions are not exported, but their headers are still decompressed to maintain the HPACK dictionary without occupying normal request capacity.

DNS-over-TCP request/response matching includes a local connection generation to distinguish reconnects reusing ports and DNS IDs. This identity adds no exported field, tag or aggregation dimension. SYN-ACK validation includes observed SYN data for Fast Open. A reset must match the current direction's contiguous sequence position; handshake refusals instead validate the SYN acknowledgement interval, preventing old packets from retiring a new connection.

FIN validation checks the reassembly sequence range and the observed peer acknowledgement range before changing close state; unmatched controls leave the current connection intact. Idle cleanup expires incomplete payloads while retaining clean sequence identities within the existing 4096 directional-entry limit, allowing split messages after idle reuse. Admission pressure reclaims closed or mutually idle identities first. Expired incomplete payloads invalidate pending queries as a collection gap rather than reporting a network DNS timeout.

eBPF Tracing function¶

ebpf-trace collects and analyzes the network data read and written by the process on the host, and tracks the kernel-level threads/user-level threads (such as golang goroutine) of the process to generate link eBPF Span. This data needs to be collected by ebpftrace for further processing.

When using, you need to deploy the eBPF collector with link data collection enabled on multiple nodes, then you need to send all eBPF Span data to the same DataKit ELinker/DataKit with the ebpftrace collector plug-in enabled. For more configuration details, see the eBPF link document

The blacklist function of the bpf-netlog plug-in¶

Filter rule example:

Single rule:

The following rules filter network data with ip 1.1.1.1 and port 80. (Line breaks allowed after operator)

(ip_saddr == "1.1.1.1" || ip_saddr == "1.1.1.1") &&
      (src_port == 80 || dst_port == 80)

Multiple rules:

Use ; or \n to separate the rules. If any rule is met, the data will be filtered.

udp
ip_saddr == "1.1.1.1" && (src_port == 80 || dst_port == 80);
ip_saddr == "10.10.0.1" && (src_port == 80 || dst_port == 80)

ipnet_contains("127.0.0.0/8", ip_saddr); ipv6

Data available for filtering:

This filter is used to filter network data. Comparable data is as follows:

Operator:

Operators from highest to lowest:

function:

1. ipnet_contains

   Function signature: fn ipnet_contains(ipnet: str, ipaddr: str) bool

   Description: Determine whether the address is within the specified network segment

   Example:

   ipnet_contains("127.0.0.0/8", ip_saddr)
   

   If the ip_saddr value is "127.0.0.1", then this rule returns true and the TCP connection packet/UDP packet will be filtered.

2. has_prefix

   Function signature: fn has_prefix(s: str, prefix: str) bool

   Description: Specifies whether the field contains a certain prefix

   Example:

   has_prefix(k8s_src_pod, "datakit-") || has_prefix(k8s_dst_pod, "datakit-")
   

   This rule returns true if the pod name is datakit-kfez321.

Network Diagnostics¶

Network diagnostics are enabled by default with ebpf-net. Results are uploaded to DataKit and can be investigated without visiting the host's /debug/netflow endpoint.

TCP first retains a packet-proven NAT destination in the flow snapshot identified by the socket pointer and birth time. The evidence stays with its counters until user space consumes them, so conntrack deletion or source-port reuse does not erase an unconsumed association. Conflicting destinations for one TCP generation remain N/A.

With IPVS conntrack enabled and the required kernel hooks available, the MASQ/NAT path associates the actual skb with conntrack reply publication. Preexisting connections use the completed mapping after the IPVS update function returns. Without BTF, two distinct conntrack owners must uniquely confirm the same skb pointer field before it is used. Unproved preexisting mappings remain N/A during calibration or after conflicting evidence. The steady path does not traverse the IPVS connection table or retain packet history.

UDP and connections without this socket proof use a fallback lookup based on address family, TCP/UDP protocol, source network namespace and the complete endpoint tuple. Multiple owners suspend only that tuple’s fallback association. It can recover without a collector restart once the source version and membership are stable and ownership is unique. UDP does not pin one backend to its entire socket lifetime. Cross-namespace DNAT requires observed packet ingress and a veth relationship verified in both directions. Unverified associations remain N/A; not all CNI and traffic-offload paths are guaranteed to be covered. Map admission failures are reported as collection gaps.

Enabling httpflow-tls alone starts its application runtime. Failure to initialize optional application offsets or probes preserves independently verified base network collection; startup logs and enabled network diagnostics report the unavailable application capability. DNS reassembly capacity drops invalidate affected pending requests instead of turning known capture gaps into DNS no-response incidents.

# Inside [[inputs.ebpf]]
network_diagnostics = true
network_diagnostics_full_metrics = false
network_diagnostics_max_groups = 4096
network_diagnostics_log_bytes_per_minute = 524288
network_diagnostics_log_points_per_minute = 240

Environment variables: ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS, ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS_FULL_METRICS, ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS_MAX_GROUPS, ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS_LOG_BYTES_PER_MINUTE, and ENV_INPUT_EBPF_NETWORK_DIAGNOSTICS_LOG_POINTS_PER_MINUTE. Effective budgets are 4096–1048576 bytes/minute and 1–4096 records/minute; at most 4096 groups are admitted. Budgets apply per node. UDP collection remains controlled by enable_udp. Plaintext DNS capture and namespace counters have independent availability fields.

The netflow tcp_role_observed string field describes TCP roles first observed in this metric window: client, server, or unknown; aggregates containing multiple roles use mixed. The field is omitted without new role observations. It is not a connection count. direction and conn_side use available connection or protocol evidence; unproven TCP roles remain unknown, rather than inferring a server from port magnitude.

For TCP connections established before attachment, ebpf-net and bpf-netlog share listener snapshots refreshed per network namespace. Connect/accept or packet handshake evidence takes precedence over verified protocol roles, which take precedence over local listener inference. A missing listener does not establish a client role. Matching checks address family, address and port. Failed refreshes preserve complete snapshots; listener inference expires after 30 seconds without a successful refresh. Inference is not latched as handshake evidence and can be corrected by later direct evidence. ebpf-net moves contributions already counted in the current window. bpf-netlog aggregates live connections with their latest roles at the reporting boundary and settles connections removed earlier when they leave, without an additional per-connection migration ledger. Both retain traffic totals and leave previously exported windows unchanged. Discovery does not manufacture historical establishments, handshake latency or traffic before attachment. In network diagnostics, role_source=listener identifies listener inference; tcp_role_observed retains its kernel-observed meaning and can differ from an inferred direction. Refreshes reuse the namespace inventory and INET_DIAG queries without walking process FDs or requiring BTF.

When TCP handshake capture is missed, bpf-netlog can join translated packet endpoints to the original socket during gathering and use its observed role or separately identified listener evidence to identify the service port. This requires a unique NAT mapping, a verified source namespace and a matching connection lifetime; conntrack original/reply orientation alone does not establish application roles. Conflicting or retired mappings and sockets without role evidence remain unknown, including closing counters that may be aggregated separately. Traffic totals are retained, and detail logs keep the endpoints observed by packet capture.

bpf-netlog counts captured unique TCP sequence ranges without waiting for protocol reassembly to fill gaps. A changed initial sequence distinguishes tuple reuse from SYN retransmission. The old generation is settled before replacement, keeping connection states separate without inventing an unobserved close. Deduplication retains bounded sequence ranges, without extra packet payloads. Incomplete range history is exposed through the local collection-quality counter dkebpf_exporter_bpf_events_dropped_total{component="l4log",event="tcp_accounting",reason="range_history"}.

Known L7 admission-loss markers belong to exact socket generations. Live generations do not lose their markers merely through expiry, and birth times do not ban unrelated older connections. Marker capacity is bounded; inability to retain one records event="admission_marker",reason="limit" and uses the existing loss-epoch protection against silent mismatches. Four fixed listener workers schedule due namespaces independently and prioritize renewal of confirmed listeners. Namespace inventory discovery runs separately from listener reads.

To reduce payload size, additional netflow counters are emitted only for nonzero increments. Connection-duration and SRTT fields are emitted only with a nonzero sample count; valid zero-valued measurements remain present. Existing fields are unaffected. Queries must tolerate absent additional fields and check collection capability and evidence completeness; an absent field alone does not prove the absence of network faults.

Default Metrics and Timeline Cardinality¶

With the default network_diagnostics_full_metrics = false, each node emits one group of 41 metric fields under its stable global tags, without netns or interface labels. Source counters, deltas, resets and expiry remain active for fault detection; logs retain namespace/interface attribution and samples. Netflow fields and semantics are unchanged.

connection_evidence_complete describes collection completeness for connection evidence on the node, not successful platform ingestion. If DataKit rejects a feed, it returns HTTP 503 and the collector retains queued data for retry. HTTP 200 acknowledges acceptance by DataKit; Pipeline, filtering and output still affect the final result. netflow_queue_full_points_total counts points discarded before queue admission. These points are not retried, so assess completeness using queue and output-loss counter increments as well.

With one collector instance per node and stable host/global tags, 100, 1000 and 10000 nodes produce approximately 4100, 41000 and 410000 timelines respectively, independently of connection, process, Pod and namespace counts. Changing global tags still creates new timelines. Longer reporting intervals reduce point counts, not the cardinality of distinct label combinations.

For continuous per-source metrics, enable network_diagnostics_full_metrics = true, or use standalone --network-diagnostics-full-metrics / DKE_NETWORK_DIAGNOSTICS_FULL_METRICS=true. Full mode adds collector fields and source cumulative counters, deltas, timestamps and interface counters keyed by host + netns; cardinality grows with namespace count and churn. The complete metric definitions below include optional fields not emitted by default.

Full-mode network stack and interface fields use lowercase snake case with explicit protocols and receive/send directions. *_total is the kernel source cumulative count; *_delta is the observed increase since the previous metric window. An initial or reset sample establishes a new baseline and contributes no increment for that sample; increments already observed earlier in the same metric window are retained. IPv4 uses ipv4_*; IPv6 uses ipv6_*, icmpv6_*, and udpv6_*. icmp_* and udp_* mean ICMPv4 and IPv4 UDP respectively. Counters can overlap and must not be added to derive unique faults or a wire loss rate.

Node-level udp_no_port_total counts observed udp_no_port diagnostic increments during this collector session. In full mode, the same field under scope=namespace is the Linux Udp.NoPorts cumulative count and can include history before attachment. These scopes are different views of evidence and must not be summed.

TCP retransmits, tcp_retransmit_total, and tcp_retransmit log counts use increments of the socket kernel total_retrans counter, including TLP, rather than the requested segment budget at function entry. They are not unique lost packets, failed requests, or an end-to-end loss rate; local transmit failures can still occur after kernel retransmit accounting. Existing connections first adopted through I/O use their first observed value as a baseline, excluding pre-attachment history.

tcp_retransmit_counter_available=false means a verified counter offset is unavailable; zero does not establish an absence of retransmissions. Without BTF, bounded private-socket self-tests validate the field during startup offset resolution only. Disabling network diagnostics preserves the original netflow retransmit counter; its BTF-less fallback validates only this field without a packet ring or other diagnostic challenges. Log capabilities.tcp_retransmit_events indicates whether the successful-retransmit tracepoint is available. When available, it refreshes queue evidence on retransmission; otherwise cumulative counters catch up on a later retransmit attempt, I/O, or final connection sample. Compare at_ns with snapshot_at_ns: an older snapshot cannot establish recovery.

Each connection sample also includes retransmits_since_observed, tcp_retransmit_counter_available, socket_errno, and socket_error_observed_at_ns. The retransmit count is the cumulative increment since this collector adopted the socket, rather than a count for the log aggregation window. An unavailable counter does not establish an absence of retransmissions. Socket errno and its observation time remain independent of application I/O errors. Closed connections no longer expose current queue values, but retain observed retransmits, RTO fires, and socket errors. Ordinary clue logs group observations by service and problem type, retaining at most three connection samples per group. Severe connection evidence is retained as described below.

ack_progress_observed_at_ns is the observed reference for the no-progress check. It updates on ACK advancement, an empty unacknowledged queue, or the first observation of newly outstanding data. It is not the exact time of the first packet loss; a preceding idle period does not count as a stalled send.

ACK-stall detection requires locally outstanding send data. When the peer has acknowledged a request but the response is lost on its return path, the client can show zero retransmits, zero unacknowledged data and no RST or socket error. These zero values do not rule out a network failure; connection_evidence_complete does not mean every failure type is covered. HTTP transactions from httpflow do not provide MySQL request-wait observations. Investigate such database connections using client request/read-timeout evidence, server logs or peer-side TCP retransmissions. TCP_USER_TIMEOUT bounds transmission waits for unacknowledged/unsent data and does not replace an application response-read timeout.

An established TCP connection with a valid ACK-stall snapshot (outstanding sequence space and at least 10 seconds without progress) or an observed hard socket/I/O error also produces problem_type=tcp_connection_issue. It uses the existing ebpf_network_event/v1 schema and a connection_id field, never a connection ID metric label. Each group belongs to one connection and retains at most three snapshots: the first abnormal observation, the latest significant change, and closure, identified by phase=started/updated/closed and reason. Initial and final evidence enter the existing output loop promptly; intermediate changes are reported at most once every five minutes. The same ACK stall no longer emits a duplicate service-level sample log, while its aggregate counter remains. For this log type, occurrences counts evidence updates; read retransmits_since_observed for retransmit counts.

At the first severe connection observation, the existing PID-to-Pod cache supplies k8s_pod_name, k8s_namespace and k8s_workload_name when available. They remain log fields and message.entity context through closure. An unmapped PID leaves them absent; a later Pod reusing the IP is not used to backfill an old connection. This adds no file or namespace scans, metric tags, or aggregation dimensions to normal netflow.

event_status=closed means TCP CLOSE or final socket retirement was observed; an application close call need not have occurred yet. first_seen_ns is the first abnormal observation, not the exact first packet-loss time, and connection age is not issue duration. A missing start or collection coverage gap sets evidence_partial=true. A timeout errno alone cannot distinguish retransmission exhaustion, TCP_USER_TIMEOUT, or TCP keep-alive timeout. Existing group, memory, queue, and log budgets apply. Groups are released when their final log is queued, and inactive groups retain the existing 30-minute expiry. Budget pressure, export rejection, or capture loss can prevent complete retention; inspect the existing drop and completeness metrics. Healthy connections and isolated retransmissions do not start this record.

Rules and Attribution¶

Ordinary connection clues pass an additional log qualification step. Raw cumulative counts and netflow data remain available:

The first qualified ordinary connection clue normally enters the output queue within 5 seconds. Subsequent qualified updates for the same group are summarized at most once every 5 minutes. Confirmed connect/I/O errors, network stack drops, DNS and collection-quality events retain their one-minute update interval. Qualification does not automatically turn info into an alert or change metric fields, netflow fields or raw observation counts. occurrences includes observations since the preceding report, including earlier observations below the log threshold; it is not a unique connection count. Qualified evidence remains pending when output budgets are exhausted. Shutdown does not force an unqualified clue into a log.

These are conservative initial investigation rules; informational clues do not automatically become fault alerts. See connect(2) for connection results. UDP asynchronous errors may refer to an earlier datagram, so UDP I/O error logs retain socket/process scope and omit an unverified peer. See Linux SNMP counters for overlap/offload semantics, TCP RFC 9293 for half-close, zero windows and sequence space, and RFC 1035 for DNS response codes.

Volume, Retention and Delivery¶

Problems are grouped by type and process/service/namespace/service endpoint, with at most three representative samples per group. Business payloads are not collected; plaintext DNS diagnostics may include query names. Logs have both encoded-byte and record token budgets: 512 KiB/minute and 240 records/minute sustained by default; either budget can defer emission. Bursts are capped at 2 MiB and 960 records. Each encoded record is capped at 16 KiB; oversized records increment output_rejected_points_total. Unchanged observations in the same group are not repeatedly encoded and rejected; new observations may retry, while factual counts remain retained. For one continuously running collector, sustained new-log admission is bounded by approximately 720 MiB and 345600 records/day per node, plus the initial burst, retries and transport overhead. Both maxima need not be reached together. Existing netflow, additional netflow fields and diagnostic metrics are outside this budget.

Group accounting has a 16 MiB budget; the output queue has an 8 MiB / 2048-batch budget; the source counter table has at most 16384 keys. Groups and queues grow with observations and pending output, without preallocating their full budgets at startup. TCP fault values share the existing connection lifecycle and add 144 bytes plus map key/management overhead per admitted connection. These limits are not an overall RSS limit. Metrics do not use PID, connection ID or remote address as labels. Namespace/interface points are disabled by default; full mode also omits interfaces with no error increments.

Listener endpoints release local slots after 30 minutes without an observed SYN, counted by listener_idle_evictions_total. Listener snapshots are sampled clues; a SYN racing idle retirement may be omitted. Inactive problem groups and stale source counters expire after 30 minutes. Restart does not replay memory, and no persistent directory is required. Platform retention is determined by the platform's storage policy. Silence is not recovery: only confirmed hard connect errors can emit recovered, after two complete quiet windows with successful connects in the same group. Other clues report observations and updates.

Admission failures, omitted samples, rate limiting, full queues, permanent rejection and undelivered shutdown points are counted explicitly. Byte and record tokens are spent only when a new log enters the queue; queue rejection and metrics do not spend log tokens. Transient failures retry the same encoded batch and event_id without spending tokens again. The platform can deduplicate by collector_session + event_id; end-to-end exactly-once delivery is not guaranteed. Export blockage delays the first log.

Investigation and Capability Limits¶

Busy snapshots increment flow_snapshot_deferred_total and make that window incomplete; cumulative traffic waits for the next stable scan, without claiming actual loss.

1. Check connection_evidence_complete, *_available, namespace_counter_sample_at_ns, coverage and collector loss counters in ebpf_network; full mode additionally exposes per-source sample_at_ns / sample_age_seconds. Missing capability or stale data does not mean no fault.
2. Use netflow traffic, outcomes and observed roles to narrow the process, service and peer. Connections predating attachment may lack role evidence: direction and conn_side then remain unknown, aggregates retain address-pair traffic totals, and raw ports remain available in connection details or local diagnostics.
3. Search ebpf_network_event by time, problem_type and scope. Its message contains ebpf_network_event/v1, entity, samples, evidence_partial and capabilities. Process/endpoints are fields; listener/namespace events do not invent a PID.
4. Correlate both endpoints and stack evidence to identify connection stage, service queues, local consumption, DNS or collection problems. A single endpoint usually cannot identify the exact network hop dropping packets.

Socket errno, TCP queues, send windows and listener queues prefer validated kernel BTF fields. Missing BTF or fields trigger startup validation on local TCP sockets. Socket waits and sampling share an eight-second deadline; temporary probes and packet sockets are then released, without increasing runtime sampling frequency. Failed or ambiguous validation, missing kernel functions or missing loopback capture permissions explicitly disable the affected optional capability. Traffic, connection outcomes, I/O errors and stack counters remain available. Terminal queue snapshots are marked unavailable, while socket errors and reset evidence are retained.

Namespace sources cover candidates from observed connections and the collector itself; every 10 seconds at most 128 namespaces are rotated through, with at most 32 interfaces each. Coverage counts are reported. Encrypted DNS, inaccessible namespaces and transient queue pressure between scans are not guaranteed to be observed.

Network Diagnostic Metrics¶

ebpf_network¶

Network aggregation data¶

For all of the following data collections, a global tag named host is appended by default (the tag value is the host name of the DataKit), or other tags can be specified in the configuration by [inputs.ebpf.tags]:

 [inputs.ebpf.tags]
  # some_tag = "some_value"
  # more_tag = "some_other_value"
  # ...

netflow¶

dnsflow¶

httpflow¶

Logging¶

ebpf_network_event¶

bash¶

bpf_net_l4_log¶

bpf_net_l7_log¶

Tracing¶

dketrace¶

Probe and System Overhead¶

Go association is enabled only when tracing targets are configured. It prefers probes with lower overhead and automatically falls back when they are unsupported. Related probes are detached after the last target and its approximately 30-second reuse window expire, and reinstalled when a new target appears.

TLS probes are shared across targets and retained for up to approximately 30 seconds after the last target disappears, reducing repeated attachment during brief restarts. Idle probes are reclaimed early under resource pressure and cleaned up when the collector stops. These optimizations do not change exported fields or data formats.

File read budgets are not disk I/O rate limits or read timeouts. Assess actual resource usage using node metrics.

Prometheus Self Monitoring¶

After pprof_port is configured, datakit-ebpf exposes both endpoints on the same address:

• /debug/pprof/
• /metrics

It is recommended to expose them only on localhost or an internal address, for example:

pprof_host = "127.0.0.1"
pprof_port = "6061"

OOM-focused Metrics¶

When neither the collector memory limit nor GOMEMLIMIT is explicitly set, the collector reads the effective cgroup v1/v2 limit at startup, including visible ancestors and the v1 hierarchical limit. A DataKit child uses 35% of that limit as its Go runtime soft budget; a standalone collector uses 70%. An unreadable parent identity selects 35%. For a shared 2000 MiB cgroup, the child's Go budget is approximately 700 MiB. Detection runs once at startup; restart the collector after changing its container limit.

This soft limit encourages Go GC. It excludes AF_PACKET/perf mmap, kernel BPF maps and the DataKit parent, and does not guarantee that total container RSS stays below the cgroup limit. An existing GOMEMLIMIT takes precedence. Explicit mem_limit retains its resource-limit/exit behavior and sets the Go budget using DK_EBPF_GO_MEMORY_LIMIT_RATIO (default 0.8). Hosts without a finite cgroup limit receive no automatic Go limit.

The idle L7 event pool has seven size classes, each bounded to approximately 512 KiB (at most 3.5 MiB total accounted capacity). Active events remain charged to async queues and decoder budgets. TCP gap retries reuse syscall-event ordering state. Network fields and connection/byte accounting semantics are unchanged.

When an L7 worker queue or its byte budget is full, admission rejects the current event immediately and marks the affected socket generation. The perf callback does not wait for queue space or acquire that connection's decoder lock. Its worker releases decoder state and pending data that is no longer in sync, and suppresses a suspect result produced concurrently with admission loss. A bounded lifetime tombstone prevents later records from starting mid-stream when no user-space pipe existed yet. Queue and decoder memory budgets are unchanged; sustained overload can still lose samples. Inspect async-event, perf-lost and decoder-budget counters for coverage.

On kernels that return perf overflow errors, L7 events use the existing bounded recovery queue and resume perf delivery after ordered recovery. dkebpf_exporter_perf_lost_samples_total counts perf transport notifications; an increase does not establish the same number of lost L7 transactions. Also inspect dkebpf_exporter_l7_event_recovery_total{result="delivered"} and actual sequence or decoder gaps. Some older kernels, including Linux 4.15, return success even after perf overflow, so this return value cannot recover those events. Sequence gaps that cannot be recovered still invalidate collection evidence and must not be interpreted as complete transactions. Sustained recovery-queue overload can also leave unrecoverable gaps.

A stream confirmed as unable to recover synchronization releases pending payloads and decoder state promptly, rather than continuing to consume healthy connections' budgets. Already queued close notifications still retire the socket generation. Known HTTP body gaps that can be safely recovered retain their existing handling.

httpflow captures application I/O through eBPF syscall/socket probes, with supported uprobes for TLS plaintext. It no longer starts supplemental L7 AF_PACKET capture, namespace listeners or TCP payload caches. Independent bpf-netlog and DNS packet capture retain their existing behavior.

sendfile / splice may expose completed byte ranges without their payload. Observed HTTP headers and known body lengths allow byte accounting to advance. Unobserved headers or chunk framing produce explicit capture gaps rather than complete transactions; later L7 transactions on that connection are not guaranteed to recover. TCP connection and byte accounting remain independent.

Application I/O capture is bounded. When the HTTP body length is known, the parser advances by completed I/O bytes and can skip uncaptured body data. Missing headers, chunk framing or other protocol boundaries invalidate decoding and stop unreliable transaction pairing. Inspect deliberate truncation, user-memory read failures and perf/queue event loss separately; none directly represents a transaction loss rate.

The following metrics are the primary signals for OOM investigation:

• dkebpf_runtime_memory_bytes{type="rss"}
• Process resident memory, the closest signal to actual OOM pressure
• dkebpf_runtime_memory_bytes{type="hwm"}
• Historical peak RSS
• dkebpf_runtime_memory_bytes{type="go_limit"}
• Current finite Go runtime soft budget; not a total RSS cap
• dkebpf_runtime_memory_bytes{type="go_total"}
• Major Go runtime memory retained from the OS
• dkebpf_runtime_memory_bytes{type="heap_alloc"}
• Actively allocated Go heap memory
• dkebpf_runtime_memory_bytes{type="non_go_estimate"}
• Approximate value computed as rss - go_total
• Useful when RSS grows faster than Go heap, indicating cgo, native buffers, mmap, or other non-Go memory
• dkebpf_runtime_memory_ratio{type="non_go_vs_rss"}
• Estimated non-Go memory as a ratio of RSS

Additional detailed Go memory breakdowns are also exposed, including:

• heap_inuse
• heap_idle
• heap_released
• heap_unused
• stack_inuse
• stack_sys
• mspan_inuse
• mspan_sys
• mcache_inuse
• mcache_sys
• gc_sys
• other_sys
• buck_hash_sys

Supporting metrics:

• dkebpf_runtime_memory_objects
• dkebpf_runtime_memory_ops_total
• dkebpf_runtime_gc
• dkebpf_runtime_goroutines

Aggregation and Cache Pressure Metrics¶

To avoid high-cardinality self metrics, the eBPF collector only exposes low-cardinality aggregation pressure metrics:

• dkebpf_exporter_agg_entries{component}
• Current number of keys in an aggregation pool
• dkebpf_exporter_agg_flush_points_total{component,result}
• Total number of points emitted by flushes
• dkebpf_exporter_agg_flush_duration_seconds{component,result}
• Flush duration
• dkebpf_exporter_cache_entries{component,cache}
• Current cache or queue length

The current coverage includes:

• netflow
• dnsflow
• l4log_netflow
• l4log_httpflow
• l7flow_http
• l4log
• l4log_host_peer_shared

It also covers several intermediate states, for example:

• dnsflow pending query / packet queue / flush queue
• l4log connection pool / two-msl pool / blacklist cache
• l7flow conn map / closed conn map / span buffer

perf and TPacket Metrics¶

The collector also exposes cumulative metrics for perf event streams and afpacket.TPacket capture rings:

• dkebpf_exporter_perf_lost_samples_total{component,stream}
• Number of lost perf samples observed in user space
• dkebpf_exporter_perf_read_errors_total{component,stream}
• Number of perf reader errors
• dkebpf_exporter_tpacket_stats_total{component,type}
• Cumulative TPacket statistics
• type includes:
  • packets
  • drops
  • queue_freezes

These metrics are intentionally exposed as cumulative counters instead of separate rate gauges. Use Prometheus rate() or irate() when you need rates.

Exporter Send-path Metrics¶

If memory growth is caused by write-path slowdown or send backlog, focus on:

The sender keeps at most 16 MiB of queued plus in-flight encoded data. Batches contain at most 64 points and normally at most 4 MiB; an indivisible larger point retains all fields and is rejected as batch_too_large if it cannot fit its admission budget. Raw packet logs and detailed tracing cannot consume the last 2 MiB reserved for other inputs. Detached packet logs and detailed tracing may wait up to 5 seconds for admission; ordinary input waits, including DNS consumption, remain bounded to 1 second. DK_EBPF_EXPORTER_ENQUEUE_TIMEOUT overrides both defaults. Persistent downstream overload still causes explicitly counted drops; headroom does not guarantee delivery or reserve HTTP workers.

Diagnostic map admission failures are additionally split by the fixed map labels flow_faults, connect_failures, udp_errors, listener_stats and reason labels capacity, memory, exists, missing, other in the local self metric dkebpf_exporter_diagnostic_map_update_failures_total{map,reason}. These counters and health.diagnostic_map_failures in /debug/netflow supplement the existing aggregate failure count; they are not additional platform measurements. Map occupancy includes the error sample and listener maps.

Connection-failure and UDP hard-error samples use completion time plus PID/TID as their internal map key, so simultaneous tasks do not collide on one timestamp. Sample start/end times and existing exported fields keep their original meaning.

Error-sample consumption starts before data probes attach and runs independently of full flow scans and namespace file reads, with a bounded pass every 100 ms. Supported kernels atomically retrieve/delete batches; older kernels reuse native buffers for individual operations. Neither path introduces an unbounded queue or increases map capacity.

• dkebpf_exporter_sender_queue_length
• Number of pending send tasks
• dkebpf_exporter_sender_batch_points
• Distribution of points per send batch
• dkebpf_exporter_sender_batch_bytes
• Distribution of encoded bytes per send batch
• dkebpf_exporter_sender_requests_total{result}
• Total number of send requests
• dkebpf_exporter_sender_request_duration_seconds{result}
• Send request duration

result uses low-cardinality enum values such as:

• ok
• request_error
• http_4xx
• http_5xx

Periodic Summary Log¶

In addition to Prometheus metrics, datakit-ebpf also sends one low-frequency summary log to DataKit on each collection interval:

• category: Logging
• measurement: ebpf_monitor
• source: ebpf_monitor
• input: ebpf-monitor

This log is intended as a supplement to metrics. A single log contains:

• runtime memory summary
• aggregation and cache sizes
• NIC group counts and route counts
• exporter queue size and send-result deltas
• perf lost / read-error deltas
• TPacket packets / drops / queue-freezes deltas
• BPF map entries, capacities, fill ratios, and update-failure deltas
• Actual output-queue discarded-point deltas, distinct from temporary queuing or retries
• perf buffer / socket / filter configuration and state summaries
• for example, perf reader sizing, shared host-peer socket mode, filter attach outcome, and socket-side drops/freezes context

New map update failures, output-queue drops, or perf read errors set the summary status to warn. Unchanged historical failure totals are not counted as new failures. Existing cache.entries details remain available; cache_entries_total and cache.total_entries exclude failure counters, cumulative counters, and loss epochs.

The log contains a larger message field encoded as a fixed-schema JSON summary, while keeping a small set of important numeric fields for filtering and search.

For perf/ringbuf configuration, socket-buffer-side status, and shared socket/filter mode, prefer the periodic summary log as a supplement instead of introducing more standalone Prometheus metrics. This keeps time-series growth under control and avoids low-value alert noise.

Troubleshooting Suggestions¶

Use the following order for diagnosis:

1. rss keeps growing, while heap_alloc and go_total do not This usually indicates non-Go memory pressure. Check non_go_estimate and non_go_vs_rss
2. agg_entries and cache_entries keep growing without falling back This usually indicates aggregation or intermediate-state buildup
3. sender_queue_length keeps growing and sender_request_duration_seconds becomes larger This usually indicates write-path slowdown and queued flush results
4. bpf_map_fill_ratio stays close to 1 This usually indicates kernel-side BPF map pressure and may correlate with event drops

DDTrace is a tracing project initiated and open-sourced by Datadog. Copyright belongs to Datadog and other contributors. TrueWatch supports DDTrace data ingestion. Use of DDTrace is subject to the applicable project's open-source license.