DDTrace

The DataKit ddtrace collector is a DataDog Trace protocol receiver. A DDTrace SDK or Java Agent in your application sends trace payloads to DataKit over HTTP; DataKit then parses, processes, and uploads them. The collector neither installs an SDK in the application nor instruments application code for you.

The data path is:

application code → DDTrace SDK / Java Agent → DataKit HTTP receiver → TrueWatch

Traces, profiling data, and runtime/JMX metrics use different receivers:

• This collector receives traces through the DataKit HTTP port, normally 9529.
• Profiling requires the separate Profiling collector.
• JMX and runtime metrics sent through DogStatsD require the separate StatsD collector, normally on port 8125.

Before You Begin¶

1. Install DataKit, enable this collector, and instrument the application with the SDK or Agent for its language.
2. Ensure that the application can reach DataKit. DataKit listens on localhost:9529 by default. If the application is on another host or Pod, adjust the HTTP listener and restrict access with firewalls, a Kubernetes Service, or NetworkPolicy.
3. Explicitly configure the trace destination in the application, for example DD_AGENT_HOST=<datakit-host> and DD_TRACE_AGENT_PORT=9529. The common upstream Datadog Agent default is 8126; do not rely on a default and do not confuse the trace port with the StatsD port 8125.
4. Set stable DD_SERVICE, DD_ENV, and DD_VERSION values. They determine the service, environment, and version shown in the tracing UI.

Language-Specific Setup¶

Configuration¶

This section configures the DataKit receiver. Configure the SDK's destination, service identity, and sampling separately in the application's startup options or environment variables; see the language-specific guide.

[[inputs.ddtrace]]
  ## DDTrace Agent endpoints register by version respectively.
  ## Endpoints can be skipped listen by remove them from the list.
  ## NOTE: DO NOT EDIT.
  endpoints = ["/v0.3/traces", "/v0.4/traces", "/v0.5/traces"]

  ## customer_tags will work as a whitelist to prevent tags send to data center.
  ## All . will replace to _ ,like this :
  ## "project.name" to send to center is "project_name"
  # customer_tags = ["sink_project", "custom_dd_tag", "reg:key_*"]

  ## Keep rare tracing resources list switch.
  ## If some resources are rare enough(not presend in 1 hour), those resource will always send
  ## to data center and do not consider samplers and filters.
  # keep_rare_resource = false

  ## By default every error presents in span will be send to data center and omit any filters or
  ## sampler. If you want to get rid of some error status, you can set the error status list here.
  # omit_err_status = ["404"]

  ## compatible otel: It is possible to compatible OTEL Trace with DDTrace trace.
  ## make span_id and parent_id to hex encoding.
  # compatible_otel=true

  ##  It is possible to compatible B3/B3Multi TraceID with DDTrace.
  # trace_id_64_bit_hex=true

  ## api:/telemetry/proxy/api/v2/apmtelemetry is collect jvm metadata.
  ## data is: app-dependencies-loaded,app-client-configuration-change,app-integrations-change ...
  ## default is true.
  # apmtelemetry_route_enable = true

  ## When true, the tracer generates 128 bit Trace IDs, 
  ## and encodes Trace IDs as 32 lowercase hexadecimal characters with zero padding.
  ## default is true.
  # trace_128_bit_id = true

  ## delete trace message
  # del_message = true

  ## max spans limit on each trace. default 100000 or set to -1 to remove this limit.
  # trace_max_spans = 100000

  ## max trace body(Content-Length) limit. default 32MiB or set to -1 to remove this limit.
  # max_trace_body_mb = 32

  ## tracing_metric_enable: trace_hits trace_hits_by_http_status trace_latency trace_errors trace_errors_by_http_status trace_apdex.
  ## Extract the above metrics from the collection traces.
  # tracing_metric_enable = true

  ## Blacklist of metric tags: There are many labels in the metric: "tracing_metrics".
  ## If you want to remove certain tag, you can use the blacklist to remove them.
  ## By default, it includes: source,span_name,env,service,status,version,resource,http_status_code,http_status_class
  ## and "customer_tags", k8s related tags, and others service.
  # tracing_metric_tag_blacklist = ["resource","operation","tag_x"]

  ## Whitelist of metric tags: There are many labels in the metric: "tracing_metrics".
  # tracing_metric_tag_whitelist = []

  ## Ignore tracing resources map like service:[resources...].
  ## The service name is the full service name in current application.
  ## The resource list is regular expressions uses to block resource names.
  ## If you want to block some resources universally under all services, you can set the
  ## service name as "*". Note: double quotes "" cannot be omitted.
  # [inputs.ddtrace.close_resource]
  #   service1 = ["resource1", "resource2", ...]
  #   service2 = ["resource1", "resource2", ...]
  #   "*" = ["close_resource_under_all_services"]
  #   ...

  ## Sampler config uses to set global sampling strategy.
  ## sampling_rate used to set global sampling rate.
  # [inputs.ddtrace.sampler]
  #   sampling_rate = 1.0

  # [inputs.ddtrace.tags]
  #   key1 = "value1"
  #   key2 = "value2"
  #   ...

  ## Threads config controls how many goroutines an agent cloud start to handle HTTP request.
  ## buffer is the size of jobs' buffering of worker channel.
  ## threads is the total number fo goroutines at running time.
  # [inputs.ddtrace.threads]
  #   buffer = 100
  #   threads = 8

  ## Storage config a local storage space in hard dirver to cache trace data.
  ## path is the local file path used to cache data.
  ## capacity is total space size(MB) used to store data.
  # [inputs.ddtrace.storage]
  #   path = "./ddtrace_storage"
  #   capacity = 5120

customer_tags promotes selected meta fields to top-level tags. A . in a literal field name becomes _; for example, http.route becomes http_route. A regular expression must begin with reg: and uses Go regular-expression syntax; for example, reg:^key_.*$ matches fields beginning with key_. Validate expressions in a test environment first: an invalid expression prevents the collector from initializing correctly.

Notes on Multi-Tool Trace Propagation¶

A traditional DDTrace Trace ID is a 64-bit integer, whereas W3C tracecontext uses a 128-bit, 32-character hexadecimal Trace ID. DDTrace places the upper 64 bits in _dd.p.tid; DataKit uses that field to reconstruct the full 128-bit ID.

Use a consistent propagation protocol across the call path and configure DataKit for the protocol actually emitted:

• When linking tracecontext traffic with OpenTelemetry, enable compatible_otel=true so span and parent IDs are formatted in hexadecimal. trace_128_bit_id is enabled by default and combines _dd.p.tid with the lower 64-bit Trace ID.
• When using b3multi and the upstream sends a 64-bit hexadecimal Trace ID, enable trace_id_64_bit_hex=true.
• After changing propagation, validate one cross-service request. A protocol mismatch normally produces disconnected service topology rather than an ingestion failure.

For more combinations, see Multi-Tracing Propagation.

Inject Pod and Node Information¶

When an application runs in Kubernetes, use the Downward API to put Pod, Namespace, and Node values in DD_TAGS. They travel with application spans. Add the fields to DataKit customer_tags as well if you need them as top-level tags in the trace list.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  replicas: 3
  template:
    metadata:
      labels:
        app: my-app
        service: my-service
    spec:
      containers:
        - name: my-app
          image: my-app:v0.0.1
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: DD_TAGS
              value: pod_name:$(POD_NAME),pod_namespace:$(POD_NAMESPACE),host:$(NODE_NAME)
            - name: DD_SERVICE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['service']

Kubernetes environment substitution can reference only environment variables defined earlier in the same container. Define POD_NAME, POD_NAMESPACE, and NODE_NAME before DD_TAGS.

After the application starts, enter the corresponding Pod and verify if the ENV is in effect:

$ env | grep DD_
...

Once injected successfully, you can see the Pod and Node names where the Span is located in the final Span data.

endpoints = ["/v0.3/traces", "/v0.4/traces", "/v0.5/traces"]

# [inputs.ddtrace.sampler]
# sampling_rate = 1.0

HTTP Settings¶

DataKit listens on localhost:9529 by default. If traces arrive from a remote host or another Pod, set a reachable listener in datakit.conf, for example:

[http_api]
  listen = "0.0.0.0:9529"

Use this only on a controlled network. Do not expose trace endpoints directly to the Internet; apply firewall rules, security groups, Kubernetes Services/NetworkPolicies, or TLS as appropriate. DataKit receives /v0.3/traces, /v0.4/traces, and /v0.5/traces by default. Do not change endpoints unless you understand the compatibility impact.

If DDTrace data is sent to DataKit, you can view it on the DataKit monitor:

Enable Disk Cache¶

Disk cache defers processing of HTTP request bodies during traffic spikes, reducing memory and processing peaks. It is not long-term archival and does not replace SDK-side sampling. The directory must be writable, have enough capacity, and be on persistent storage when recovery after a container restart matters. capacity is in MiB.

[inputs.ddtrace.storage]
  path = "/path/to/ddtrace-disk-storage"
  capacity = 5120

DDTrace SDK Configuration¶

After configuring the collector, configure the SDK. Supported variables and precedence vary by language and SDK version. The variables below describe common concepts. When an SDK supports DD_TRACE_AGENT_URL, that URL normally overrides separate host and port values, so do not set conflicting destinations.

Environment Variable Settings¶

In addition to setting the project name, environment name, and version number during application initialization, you can also set them in the following two ways:

• Inject environment variables via the command line

DD_TAGS="project:your_project_name,env=test,version=v1" ddtrace-run python app.py

• Configure receiver-side tags directly in ddtrace.conf. They affect every DDTrace payload entering this DataKit and are suitable for shared deployment tags, not application-specific service identity:

# tags are key-value pairs configured for ddtrace
[inputs.ddtrace.tags]
  some_tag = "some_value"
  more_tag = "some_other_value"

APMTelemetry¶

Version-1.35.0 · Experimental

The Java Agent can report startup configuration, heartbeat, dependencies, and loaded integrations through /telemetry/proxy/api/v2/apmtelemetry. DataKit accepts this route by default; disable apmtelemetry_route_enable if it is not needed. View the data in the TrueWatch Infrastructure Resource Directory to troubleshoot startup options, dependency versions, and loaded instrumentation.

Data may vary significantly across different languages and versions; please refer to the actual received data.

Fixed Tag Extraction¶

Starting with DataKit 1.21.0, DataKit no longer promotes every Span.Meta field to a top-level tag. It extracts only the commonly used fields below to control tag cardinality and indexing cost.

The following is a list of tags that may be extracted:

Fields outside the list remain in the span meta and are available in trace details. Whether they can be used as top-level filters depends on the UI and indexing configuration.

Starting with DataKit 1.22.0, add fields to the customer_tags allowlist when they must become top-level tags. A . becomes _ during extraction. Add only stable, low-cardinality fields; never promote values such as user IDs or request IDs.

Common Troubleshooting Paths¶

Collected Data Field Description¶

Tracing¶

ddtrace¶

Following is tags/fields of tracing data

Metrics¶

tracing_metrics¶

Based on DDTrace's span data, we count span count, span cost metrics

Custom Objects¶

The extended Java Agent can report its configuration, integration list, dependencies, and service metadata after startup. These resource objects currently apply only to the Java Agent. Common events include:

• app_client_configuration_change: Contains the agent's configuration information
• app_dependencies_loaded: Dependency list (including package names and version information)
• app_integrations_change: Integration list (including package names and whether the agent is enabled)
• Other host information, service information, etc.

tracing_service¶

Collect service, host, process APM telemetry message.

More Readings¶

• DataKit Tracing Field Definition
• DataKit General Tracing Data Collection Description
• Proper Use of Regular Expressions for Configuration
• Multi-Tracing Propagation
• Java Integration and Exception Description
• DDTrace Sampling Strategy and Notes on Multi-Tool Tracing Propagation