OpenTelemetry Go (otelc)¶

OpenTelemetry Go Compile-Time Instrumentation uses otelc to automatically inject OpenTelemetry SDK initialization and component instrumentation logic during the Go compilation phase. The application does not need to modify business code; simply replace the original go build with go tool otelc go build to send telemetry data via OTLP to DataKit, which then forwards it to TrueWatch.

This document uses otelc v1.1.0, a host-installed DataKit, and OTLP/gRPC as an example to complete Trace integration for a Go HTTP service:

Go source -- go tool otelc go build --> Instrumented Go binary -- OTLP --> DataKit --> TrueWatch

Note: The "zero-code" in otelc means that business code does not need to manually import and initialize the OpenTelemetry SDK; it does not mean that rebuilding is unnecessary. A normal Go binary that has already been built cannot be directly instrumented after the fact; you must recompile using otelc and deploy the new artifact.

Prerequisites¶

• Go 1.25 or higher;
• The project uses Go Modules and can be compiled normally with a plain go build;
• DataKit is installed and connected to the target TrueWatch workspace;
• The Go application has network reachability to DataKit: OTLP/gRPC uses port 4317 by default, OTLP/HTTP uses DataKit HTTP port 9529;
• The framework or components used by the application are supported by otelc v1.1.0;
• The application does not initialize the OpenTelemetry SDK more than once. Projects that need to manage the SDK lifecycle themselves should use the OpenTelemetry Go SDK integration approach.

This document uses a Linux host and a net/http service for explanation; it does not cover Kubernetes deployment.

1. Enable the OpenTelemetry Collector¶

Navigate to the DataKit OpenTelemetry collector directory. If the configuration file does not yet exist, copy the sample configuration:

cd /usr/local/datakit/conf.d/opentelemetry
sudo cp opentelemetry.conf.sample opentelemetry.conf

Ensure that opentelemetry.conf contains at least the following configuration:

[[inputs.opentelemetry]]
  # Whitelist of custom attributes to retain as TrueWatch tags.
  customer_tags = ["team", "project"]

  [inputs.opentelemetry.http]
    http_status_ok = 200
    trace_api = "/otel/v1/traces"
    metric_api = "/otel/v1/metrics"

  [inputs.opentelemetry.grpc]
    addr = "127.0.0.1:4317"

The above configuration corresponds to the following receive endpoints:

When the application and DataKit are not on the same host, adjust the gRPC addr to a listen address reachable by the application, for example 0.0.0.0:4317, and configure the firewall or other network access controls accordingly. Do not expose the OTLP receive port directly to the public network.

Restart DataKit for the configuration to take effect:

sudo datakit service restart

Check DataKit and the gRPC port:

curl http://127.0.0.1:9529/v1/ping
ss -lnt | grep 4317

2. Integrate the Application with OpenTelemetry¶

Pre-instrumentation Check¶

Navigate to the Go Module root directory of the application and first confirm that the original project can be built normally:

go version
go build ./...

If the current directory does not yet have a go.mod, initialize the Module:

go mod init example.com/my-service

Check whether the project already directly depends on the OpenTelemetry SDK or Contrib instrumentation libraries:

go list -m all | grep 'go.opentelemetry.io'

otelc injects the SDK initialization logic. If the project already has another set of SDK initialization or duplicate HTTP instrumentation, it may cause duplicate spans, provider overwrites, or dependency version conflicts. This document recommends that business code does not directly integrate the SDK.

Install otelc¶

Use the Go tool directive to install and pin otelc v1.1.0:

go get -tool go.opentelemetry.io/otelc/tool/cmd/otelc@v1.1.0
go mod tidy

Verify the tool version:

go tool otelc version

Expected output:

otelc version v1.1.0

In production builds, pin the version in go.mod and go.sum; do not use an unpinned @latest.

Compile with otelc¶

Keep the original build parameters unchanged, only prepend go tool otelc before go build:

go tool otelc go build -o my-service .

Example with common build flags:

mkdir -p ./bin
go tool otelc go build \
  -trimpath \
  -ldflags="-s -w" \
  -o ./bin/my-service \
  ./cmd/my-service

otelc go currently supports go build, go install, and go test. The first instrumentation build needs to download and compile OpenTelemetry dependencies, which is usually noticeably slower than a normal go build; when the command prompt reappears in the terminal, the build is complete.

After the build finishes, the matched rules are recorded in .otelc-build/matched.json. Check for the HTTP server hook:

jq -e '[.. | objects | .name?] | index("server_hook") != null' \
  .otelc-build/matched.json >/dev/null

Note: The release process must deploy the binary produced by go tool otelc go build. If the artifact is later overwritten by a plain go build, the runtime will not include automatic instrumentation.

Configure OTLP/gRPC and Start¶

The following configuration enables only net/http Trace and reports to the local DataKit via OTLP/gRPC:

export OTEL_SERVICE_NAME="my-service"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=prod,service.version=1.0.0,team=backend"

export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="none"
export OTEL_LOGS_EXPORTER="none"

export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4317"
export OTEL_EXPORTER_OTLP_INSECURE="true"

export OTEL_GO_ENABLED_INSTRUMENTATIONS="nethttp"

./my-service

The runtime parameters must be set in the actual runtime environment of the instrumented binary, not just on the build host. After the application starts, request an endpoint handled by a supported component to generate a verifiable span.

In production, use TLS endpoints and manage certificates and authentication headers via Secrets. http://127.0.0.1:4317 is only suitable for localhost access.

Use OTLP/HTTP¶

If you want to switch to OTLP/HTTP + Protobuf, replace the protocol and endpoint:

export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:9529/otel"
export OTEL_EXPORTER_OTLP_INSECURE="true"

The exporter appends /v1/traces or /v1/metrics based on the data type. When setting only the Trace endpoint, you can also use:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://127.0.0.1:9529/otel/v1/traces"

3. Data Reporting Parameters¶

Resource and Exporter Parameters¶

Instrumentation, Sampling, and Debug Parameters¶

OTEL_GO_ENABLED_INSTRUMENTATIONS and OTEL_GO_DISABLED_INSTRUMENTATIONS only control the instrumentation that was already compiled into the binary. When both variables are present, the whitelist is applied first, then the blacklist is excluded.

In production, set the sampling rate based on traffic and data budget. When application-side sampling and DataKit-side sampling are both enabled, the final retention rate is compounded and reduced; plan the sampling location uniformly.

Supported Components¶

otelc v1.1.0 includes built-in rules covering the following common components:

Actual support scope may be affected by component versions and build methods. After upgrading application dependencies or otelc, re-check .otelc-build/matched.json and perform trace regression tests.

Field Mapping¶

The DataKit OpenTelemetry collector maps common OpenTelemetry Span Attributes to TrueWatch trace fields:

To retain other attributes as TrueWatch tags, configure customer_tags in the DataKit opentelemetry.conf. Do not promote high-cardinality values (such as user IDs, order numbers) to tags in bulk, and do not report sensitive information such as passwords, tokens, or full database connection strings.

Verify Integration¶

1. Confirm that the tool version and instrumentation build are both successful:

go tool otelc version
test -x ./my-service

1. Start the application and confirm that the following log entries appear, and there are no OTLP export errors:

trace provider initialized with auto-export
OpenTelemetry initialized
HTTP server instrumentation initialized

1. Request an application endpoint to generate a trace:

curl http://127.0.0.1:18080/ping

1. Check that the build rules include server_hook:

jq -e '[.. | objects | .name?] | index("server_hook") != null' \
  .otelc-build/matched.json

1. When using OTLP/gRPC, you can observe the receive count increase on the DataKit host:

curl -fsS http://127.0.0.1:9529/metrics \
  | grep 'opentelemetry.proto.collector.trace.v1.TraceService/Export'

1. Go to TrueWatch APM > Traces, search by service:my-service, and verify that you can see the trace generated by the request.

Frequently Asked Questions¶

go.mod file not found¶

go get -tool must be executed within a Go Module. Go to the project root directory, or first run:

go mod init example.com/my-service

Build hangs at WORK=/tmp/go-build...¶

The first instrumentation build compiles many dependencies. As long as the go tool otelc go build process is still running, wait for it to finish; the build is complete only when the command prompt reappears in the terminal. Do not execute the application binary mid-build.

Connection failed to the request port¶

First confirm that the build has finished and the instrumented binary has been started, then check the listening port:

ss -lntp | grep 18080

Application runs normally but no Trace in TrueWatch¶

Check, in order:

• Whether the deployed binary was produced by go tool otelc go build;
• Whether OTEL_SERVICE_NAME, the exporter, protocol, and endpoint are configured in the actual running process;
• Whether .otelc-build/matched.json contains the expected rules;
• Whether OTEL_GO_ENABLED_INSTRUMENTATIONS includes the target component;
• Whether the DataKit OpenTelemetry collector is enabled and restarted for the changes to take effect;
• Whether the network from the application to DataKit and ports 4317 or 9529 are reachable.

For build issues, you can temporarily enable detailed logging:

OTELC_DEBUG=1 go tool otelc go build -o my-service .

Detailed logs are located at .otelc-build/debug.log. Disable debug after troubleshooting to avoid log growth.

Application does not exit after the first Ctrl+C¶

The runtime injected by otelc v1.1.0 listens for SIGINT and SIGTERM; the first signal is used to flush telemetry data, but the application is still responsible for its own exit flow. Simple applications that do not implement graceful shutdown may need a second signal. Production services should use the Go standard library to implement graceful HTTP server shutdown and reserve time for telemetry flushing.

References¶

• OpenTelemetry Go Compile-Time Instrumentation
• otelc v1.1.0 Release
• DataKit OpenTelemetry Collector
• OpenTelemetry SDK Environment Variable Specification