Trace without pain: Tracing transactions with OpenTelemetry auto-instrumentation

Integration with Kiali — When properly configured, you can view distributed tracing platform data from the Kiali console.

High scalability — The distributed tracing platform back end is designed to have no single points of failure and to scale with the business needs.

Distributed Context Propagation — Enables you to connect data from different components together to create a complete end-to-end trace.

Backwards compatibility with Zipkin — Red Hat OpenShift distributed tracing platform has APIs that enable it to be used as a drop-in replacement for Zipkin, but Red Hat is not supporting Zipkin compatibility in this release.

Gateway — The Gateway handles authentication, authorization, and forwarding requests to the Distributor or Query Frontend service.

Distributor — The Distributor accepts spans in multiple formats including Jaeger, OpenTelemetry, and Zipkin. It routes spans to Ingesters by hashing the traceID and using a distributed consistent hash ring.

Ingester — The Ingester batches a trace into blocks, creates bloom filters and indexes, and then flushes it all to the back end.

Query Frontend — The Query Frontend is responsible for sharding the search space for an incoming query. The search query is then sent to the Queriers. The Query Frontend deployment exposes the Jaeger UI through the Tempo Query sidecar.

Querier - The Querier is responsible for finding the requested trace ID in either the Ingesters or the back-end storage. Depending on parameters, it can query the Ingesters and pull Bloom indexes from the back end to search blocks in object storage.

Compactor — The Compactors stream blocks to and from the back-end storage to reduce the total number of blocks.

OpenTelemetry Collector - The OpenTelemetry Collector is a vendor-agnostic way to receive, process, and export telemetry data. The OpenTelemetry Collector supports open-source observability data formats, for example, Jaeger and Prometheus, sending to one or more open-source or commercial back-ends. The Collector is the default location instrumentation libraries export their telemetry data.

Review deployment configuration for the frontend application here: https://github.com/voraviz/openshift-otel/blob/main/config/frontend.yaml

Open the Web Terminal in OpenShift console, then run these commands to deploy the frontend application.

oc project userX-otel
PROJECT=userX-otel
cd openshift-otel
oc create -f config/frontend.yaml -n $PROJECT

deployment.apps/frontend created
service/frontend created
route.route.openshift.io/frontend created

Run this command to update the frontend Deployment resource by adding an annotation to the Deployment for auto-instrumentation.

oc patch deployment/frontend \
  -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-nodejs":"true"}}}}}' \
  -n $PROJECT

deployment.apps/frontend patched

Run this command to verify that auto-instrumentation is working as init-container.

oc get po $(oc get po -l app=frontend -o custom-columns='Name:.metadata.name' -n $PROJECT --no-headers) -n $PROJECT -o jsonpath='{.status.initContainerStatuses}'|jq

[
  {
    "containerID": "cri-o://8461a346ece60ba52018f6395b1133298c26d532a0d411a30e5d147bee089fc0",
    "image": "ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.53.0",
    "imageID": "ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-
     nodejs@sha256:70ba757df71d0596aaccac91f439e8be7f81136b868205e79178e8fd3c36a763",
    "lastState": {},
    "name": "opentelemetry-auto-instrumentation-nodejs",
    "ready": true,
    "restartCount": 0,
    "started": false,
    "state": {
      "terminated": {
        "containerID": "cri-o://8461a346ece60ba52018f6395b1133298c26d532a0d411a30e5d147bee089fc0",
        "exitCode": 0,
        "finishedAt": "2025-05-05T07:22:25Z",
        "reason": "Completed",
        "startedAt": "2025-05-05T07:22:23Z"
      }
    },
    "volumeMounts": [
      {
        "mountPath": "/otel-auto-instrumentation-nodejs",
        "name": "opentelemetry-auto-instrumentation-nodejs"
      },
      {
        "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
        "name": "kube-api-access-6zqld",
        "readOnly": true,
        "recursiveReadOnly": "Disabled"
      }
    ]
  }
]

Run these commands to set environment variables for the frontend Deployment. These environment variables will be used by OTel auto-instrumentation.

oc set env deploy frontend OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector-headless:4318 -n $PROJECT
oc set env deploy frontend OTEL_SERVICE_NAME=frontend -n $PROJECT
oc set env deploy frontend OTEL_PROPAGATORS=tracecontext,b3 -n $PROJECT

Review deployment configuration for the simple-go application here: https://github.com/voraviz/openshift-otel/blob/main/config/simple-go.yaml

For the frontend application, we added annotation and set environment variables with oc patch and oc set env commands to the Deployment after we’ve created it .

template:
  metadata:
    annotations:
      instrumentation.opentelemetry.io/inject-go: "true"
      instrumentation.opentelemetry.io/otel-go-auto-target-exe: /app/api
      openshift.io/required-scc: otel-go-instrumentation-scc

      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://otel-collector-headless:4318
      - name: OTEL_SERVICE_NAME
        value: simple-go
      - name: OTEL_GO_AUTO_TARGET_EXE
        value: /app/api
      - name: OTEL_PROPAGATORS
        value: tracecontext,b3

Run this command to deploy simple-go application.

oc create -f config/simple-go.yaml -n $PROJECT

deployment.apps/simple-go created
service/simple-go created

Run this command to verify that simple-go application Pod status is Running.

oc get po -l app=simple-go -n $PROJECT

NAME                        READY   STATUS    RESTARTS   AGE
simple-go-56f8c644b-w4sxw   2/2     Running   0          7m42s

Run this command to verify that auto-instrumentation is working with as sidecar container.

oc get po $(oc get po -l app=simple-go -o custom-columns='Name:.metadata.name' -n $PROJECT --no-headers) -n $PROJECT -o jsonpath='{.spec.containers[1].name}'

opentelemetry-auto-instrumentation

Review deployment configuration for the backend application here: https://github.com/voraviz/openshift-otel/blob/main/config/backend.yaml

The backend Deployment configuration already has the annotation and environment variables requried for OTel auto-instrumentation.

  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"

      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://otel-collector-headless:4318
      - name: OTEL_SERVICE_NAME
        value: backend
      - name: OTEL_PROPAGATORS
        value: tracecontext,b3

Run this command to deploy backend application,

oc create -f config/backend.yaml -n $PROJECT

Verify that auto-instrumentation is working as init-container.

oc get po $(oc get po -l app=backend -o custom-columns='Name:.metadata.name' -n $PROJECT --no-headers) -n $PROJECT -o jsonpath='{.status.initContainerStatuses}'|jq|head -n 8

[
  {
    "containerID": "cri-o://f88eea516f6b51b9bf772b17f88f73ccd920728a8ff4237a855d89a08d2499a9",
    "image": "ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.33.6",
    "imageID": "ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java@sha256:502d3170177a0676db8b806eba047a520af9bb83400e734fc64f24a593b2ca64",
    "lastState": {},
    "name": "opentelemetry-auto-instrumentation-java",
    "ready": true,
...

Run this command to check application container’s logs. Look for JAVA_TOOL_OPTIONS in the logs, you should see the Otel auto-instrumentation Java agent is configured.

oc logs $(oc get po -l app=backend -o custom-columns='Name:.metadata.name' -n $PROJECT --no-headers) -n $PROJECT | head -n 4

Defaulted container "backend" out of: backend, opentelemetry-auto-instrumentation-java (init)
INFO exec -a "java" java -Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -cp "." -jar /deployments/quarkus-run.jar
INFO running in /deployments
Picked up JAVA_TOOL_OPTIONS:  -javaagent:/otel-auto-instrumentation-java/javaagent.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

Switch to Developer perspective, then select Topology menu on the left. All applications should be up and running.