OpenTelemetry Go (otelc)¶

OpenTelemetry Go Compile-Time Instrumentation uses otelc to inject OpenTelemetry SDK initialization and component instrumentation during compilation. You do not need to modify business code. Replace the original go build with go tool otelc go build, then report telemetry to DataKit through OTLP and forward it to TrueWatch.

This document uses otelc v1.1.0, a host-installed DataKit, and OTLP/gRPC as examples to show Trace access for a Go HTTP service:

Go source code -- go tool otelc go build --> instrumented Go binary -- OTLP --> DataKit --> TrueWatch

Note: “Zero-code” in otelc means that you do not need to manually add and initialize the OpenTelemetry SDK in business code. It does not mean you can skip rebuilding. Existing Go binaries cannot be instrumented in place. You must rebuild and redeploy with otelc.

Prerequisites¶

• Go 1.25 or later;
• The project uses Go Modules and can be built successfully with a normal go build;
• DataKit is installed and already connected to the target TrueWatch workspace;
• The Go application can reach DataKit: OTLP/gRPC uses 4317 by default, and OTLP/HTTP uses the DataKit HTTP port 9529;
• The frameworks or libraries used by the application are supported by otelc v1.1.0;
• The application does not initialize another OpenTelemetry SDK pipeline repeatedly. If you need to manage the SDK lifecycle yourself, use the OpenTelemetry Go SDK method instead.

This document uses a Linux host and a net/http service as examples and does not cover Kubernetes deployment.

1. Enable the OpenTelemetry collector¶

Go to the conf.d/opentelemetry directory under the DataKit installation path. If the configuration file does not exist yet, copy it from the sample:

cd /usr/local/datakit/conf.d/opentelemetry
sudo cp opentelemetry.conf.sample opentelemetry.conf

Confirm that opentelemetry.conf contains at least the following settings:

[[inputs.opentelemetry]]
  # Add custom attributes here if you want to keep them as tags in TrueWatch.
  customer_tags = ["team", "project"]

  [inputs.opentelemetry.http]
    http_status_ok = 200
    trace_api = "/otel/v1/traces"
    metric_api = "/otel/v1/metrics"

  [inputs.opentelemetry.grpc]
    addr = "127.0.0.1:4317"

The configuration above enables the following receive endpoints:

If the application and DataKit are not on the same host, change addr to a listening address reachable by the application, such as 0.0.0.0:4317, and adjust the firewall or other network access controls at the same time. Do not expose the OTLP receive port directly to the public internet.

Restart DataKit to apply the configuration:

sudo datakit service restart

Check DataKit and the gRPC port:

curl http://127.0.0.1:9529/v1/ping
ss -lnt | grep 4317

2. Instrument the application¶

Pre-build checks¶

Go to the Go Module root of the application and first confirm that the original project can be built normally:

go version
go build ./...

If the current directory does not contain go.mod, initialize the module first:

go mod init example.com/my-service

Check whether the project already depends on OpenTelemetry SDK or Contrib instrumentation packages:

go list -m all | grep 'go.opentelemetry.io'

otelc injects SDK initialization logic during compilation. If the project already initializes another SDK pipeline or instruments the same component in another way, you may see duplicate spans, provider overrides, or dependency conflicts.

Install otelc¶

Install and pin otelc v1.1.0 with the Go tool directive:

go get -tool go.opentelemetry.io/otelc/tool/cmd/otelc@v1.1.0
go mod tidy

Confirm the tool version:

go tool otelc version

Expected output:

otelc version v1.1.0

Pin the version in go.mod and go.sum for production builds. Do not rely on an unpinned @latest.

Build with otelc¶

Keep the existing build parameters and only add go tool otelc before go build:

go tool otelc go build -o my-service .

Example with common build options:

mkdir -p ./bin
go tool otelc go build \
  -trimpath \
  -ldflags="-s -w" \
  -o ./bin/my-service \
  ./cmd/my-service

otelc go currently supports go build, go install, and go test. The first instrumented build downloads and compiles OpenTelemetry dependencies, so it is usually noticeably slower than a normal go build. The build is complete only after the shell prompt returns.

After the build finishes, .otelc-build/matched.json records the matched rules. For example, to verify the HTTP server hook:

jq -e '[.. | objects | .name?] | index("server_hook") != null' \
  .otelc-build/matched.json >/dev/null

Important: your release pipeline must deploy the binary generated by go tool otelc go build. If it is later overwritten by a normal go build, the runtime binary will no longer contain automatic instrumentation.

Configure OTLP/gRPC and run¶

The following example enables only net/http tracing and reports through OTLP/gRPC to a local DataKit:

export OTEL_SERVICE_NAME="my-service"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=prod,service.version=1.0.0,team=backend"

export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="none"
export OTEL_LOGS_EXPORTER="none"

export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4317"
export OTEL_EXPORTER_OTLP_INSECURE="true"

export OTEL_GO_ENABLED_INSTRUMENTATIONS="nethttp"

./my-service

Apply these runtime parameters where the instrumented binary actually runs, not only on the build host. After the application starts, send requests to an endpoint handled by a supported component so that spans can be generated.

For production, use a TLS endpoint and manage certificates or authentication headers through a secret system. http://127.0.0.1:4317 is only suitable for local access.

Use OTLP/HTTP¶

To switch to OTLP/HTTP + Protobuf, replace the protocol and endpoint:

export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:9529/otel"
export OTEL_EXPORTER_OTLP_INSECURE="true"

The exporter appends /v1/traces or /v1/metrics based on the signal type. If you only want to set the Trace endpoint, you can also use:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://127.0.0.1:9529/otel/v1/traces"

3. Reporting parameters¶

Resource and Exporter parameters¶

Instrumentation, sampling, and debugging parameters¶

OTEL_GO_ENABLED_INSTRUMENTATIONS and OTEL_GO_DISABLED_INSTRUMENTATIONS only control instrumentation that has already been compiled into the binary. If both are set, the allowlist is applied first and the denylist is applied afterwards.

Plan sampling in one place. If application-side sampling and DataKit-side sampling are both enabled, the final retention rate is reduced multiplicatively.

Supported components¶

otelc v1.1.0 includes rules for the following common components:

The actual support scope still depends on the component version and build method. After upgrading application dependencies or otelc, check .otelc-build/matched.json again and run trace regression tests.

Field mapping¶

The DataKit OpenTelemetry collector converts common OpenTelemetry span attributes into TrueWatch trace fields:

If you want to keep other attributes as tags in TrueWatch, configure customer_tags in DataKit opentelemetry.conf. Do not promote high-cardinality fields such as user IDs or order IDs to tags in bulk, and do not report sensitive data such as passwords, tokens, or full database connection strings.

Verify the integration¶

1. Confirm that the tool version and instrumented build are both successful:

go tool otelc version
test -x ./my-service

1. Start the application and confirm that the logs show the runtime initialization and no OTLP export error;

2. Send a request to generate a trace:

curl http://127.0.0.1:18080/ping

1. Confirm that the build rules contain server_hook:

jq -e '[.. | objects | .name?] | index("server_hook") != null' \
  .otelc-build/matched.json

1. When OTLP/gRPC is used, check whether the receive count is increasing on the DataKit host:

curl -fsS http://127.0.0.1:9529/metrics \
  | grep 'opentelemetry.proto.collector.trace.v1.TraceService/Export'

1. In TrueWatch Application Performance Monitoring > Traces, query service:my-service and confirm that the trace generated by the request is visible.

FAQ¶

go.mod file not found¶

go get -tool must be executed inside a Go Module. Go to the project root, or initialize the module first:

go mod init example.com/my-service

The build seems to stop at WORK=/tmp/go-build...¶

The first instrumented build compiles many dependencies. As long as the go tool otelc go build process is still running, keep waiting. The build is finished only after the shell prompt returns. Do not start the application binary before the build is complete.

The request port cannot be reached¶

First confirm that the build has finished and the instrumented binary has been started, then check the listening port:

ss -lntp | grep 18080

The service runs but no Trace appears in TrueWatch¶

Check the following items in order:

• whether the deployed binary was generated by go tool otelc go build;
• whether OTEL_SERVICE_NAME, exporter settings, protocol, and endpoint are configured in the actual runtime process;
• whether .otelc-build/matched.json contains the expected rule;
• whether OTEL_GO_ENABLED_INSTRUMENTATIONS includes the target component;
• whether the DataKit OpenTelemetry collector is enabled and has been restarted;
• whether the network and ports 4317 or 9529 from the application to DataKit are reachable.

For build troubleshooting, temporarily enable detailed logs:

OTELC_DEBUG=1 go tool otelc go build -o my-service .

Detailed logs are written to .otelc-build/debug.log. Disable debug logging after troubleshooting to avoid continuous log growth.

The first Ctrl+C does not exit the application¶

The runtime injected by otelc v1.1.0 listens for SIGINT and SIGTERM. The first signal is used to flush telemetry data, but the application is still responsible for its own shutdown flow. Simple applications without graceful shutdown may require a second signal. Production services should implement graceful HTTP server shutdown and leave enough time for telemetry flushing.

References¶

• OpenTelemetry Go Compile-Time Instrumentation
• otelc v1.1.0 Release
• DataKit OpenTelemetry collector
• OpenTelemetry SDK environment variable specification