Custom OTel Collector — Crossplane Implementation

Users create a CustomOTelCollector resource in their namespace. Crossplane composes it into an OpenTelemetryCollector and an Instrumentation resource in that same namespace.

CustomOTelCollector (namespaced XR)
  ├─► function-kcl  (build-otel-collector step)
  │     └─► Object (provider-kubernetes)
  │           └─► OpenTelemetryCollector
  ├─► function-kcl  (build-otel-instrumentation step)
  │     └─► Object (provider-kubernetes)
  │           └─► Instrumentation  (named "grpc")
  └─► function-auto-ready

Kyverno ClusterPolicy (generate-otel-provider-config)
  └─► watches CustomOTelCollector creation
        └─► generates ProviderConfig (kubernetes.m.crossplane.io) in the same namespace

---

Prerequisites

1. Crossplane

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane crossplane-stable/crossplane \
  --namespace crossplane-system \
  --create-namespace
kubectl wait deployment/crossplane -n crossplane-system \
  --for=condition=Available --timeout=120s

2. cert-manager

Required by the OpenTelemetry Operator.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
kubectl wait --for=condition=Available deployment/cert-manager-webhook \
  -n cert-manager --timeout=120s

3. OpenTelemetry Operator

Provides the OpenTelemetryCollector and Instrumentation CRDs that Crossplane will manage.

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
kubectl wait --for=condition=Available deployment/opentelemetry-operator-controller-manager \
  -n opentelemetry-operator-system --timeout=120s

---

Installation

Step 1 — Install Crossplane functions and provider-kubernetes

kubectl apply -f install/functions.yaml
kubectl apply -f install/provider-kubernetes.yaml

functions.yaml installs:

• function-kcl — runs the KCL config-merging logic
• function-auto-ready — marks the XR Ready when its composed resources are Ready

provider-kubernetes.yaml installs:

• DeploymentRuntimeConfig — gives the provider a stable ServiceAccount (provider-kubernetes)
• Provider — allows Crossplane to manage arbitrary Kubernetes resources
• ProviderConfig named in-cluster using InjectedIdentity (no credentials Secret needed)

Step 2 — Wait for provider to become Healthy

kubectl wait provider/provider-kubernetes --for=condition=Healthy --timeout=120s

Step 3 — Install the Helm chart (XRD, Composition, RBAC, Kyverno policy)

helm upgrade --install custom-otel-collector chart/ -n crossplane-system

Wait for the XRD to become established:

kubectl wait xrd/customotelcollectors.k8s.flow.rws.nl \
  --for=condition=Established --timeout=60s

---

Usage

Create a CustomOTelCollector resource in any namespace:

kubectl apply -f sample/custom-otel-collector.yaml

Or write your own:

apiVersion: k8s.flow.rws.nl/v1alpha1
kind: CustomOTelCollector
metadata:
  name: my-collector
  namespace: my-app
spec:
  extraProcessors:
    filter/audit_logs:
      error_mode: ignore
      logs:
        log_record:
          - 'not IsMatch(body, ".*AUDIT.*")'
  pipelines:
    splunk:
      extraProcessors:
        - filter/audit_logs
    # flow not specified → logs/flow pipeline omitted from collector

Spec fields

Field	Type	Description
extraProcessors	object	Additional OTel processors merged into the baseline processors: section. Keys are processor names (e.g. filter/audit_logs), values are processor config objects.
pipelines.splunk	object	Optional. Omit to exclude the logs/splunk pipeline from the collector.
pipelines.splunk.extraProcessors	[]string	Processor names injected between memory_limiter and batch in the logs/splunk pipeline.
pipelines.flow	object	Optional. Omit to exclude the logs/flow pipeline from the collector.
pipelines.flow.extraProcessors	[]string	Processor names injected between memory_limiter and batch in the logs/flow pipeline.
instrumentation.grpc.dotnet.image	string	Per-resource override for the .NET auto-instrumentation image. Falls back to instrumentation.dotnet.image in chart values.
instrumentation.grpc.go.image	string	Per-resource override for the Go auto-instrumentation image. Falls back to instrumentation.go.image in chart values.
instrumentation.grpc.java.image	string	Per-resource override for the Java auto-instrumentation image. Falls back to instrumentation.java.image in chart values.

Check status

kubectl get customotelcollector -n my-app
kubectl get opentelemetrycollector -n my-app
kubectl get instrumentation -n my-app
kubectl get object

Deletion

Deleting the CustomOTelCollector removes the composed OpenTelemetryCollector and Instrumentation:

kubectl delete customotelcollector my-collector -n my-app

---

What gets created

OpenTelemetryCollector

A deployment-mode collector named <xr-name> with:

• A baseline pipeline (traces, optional logs/splunk, optional logs/flow)
• Exporters configurable via chart values (exporters.flow, exporters.splunk, exporters.traces)
• memory_limiter + batch processors by default; extended via spec.extraProcessors

The OTel Operator creates a Service named <xr-name>-collector that the Instrumentation automatically points to.

Instrumentation

---

grpc

An opentelemetry.io/v1alpha1 Instrumentation resource named grpc in the same namespace as the XR.

Configured for gRPC export to the collector created by the same CustomOTelCollector:

http://<xr-name>-collector.<namespace>.svc.cluster.local:4317

All signals (traces, metrics, logs) are routed to this single endpoint. The following env vars are set on all instrumented workloads:

Env var	Value
OTEL_EXPORTER_OTLP_PROTOCOL	grpc
OTEL_LOGS_EXPORTER	otlp
OTEL_METRICS_EXPORTER	otlp
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT	collector endpoint
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT	collector endpoint
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT	collector endpoint

Language-specific settings:

Language	Extra env vars	Image
.NET	OTEL_TRACES_EXPORTER=otlp, OTEL_DOTNET_AUTO_LOGGER=console	instrumentation.grpc.dotnet.image on XR, else instrumentation.dotnet.image in values
Go	OTEL_TRACES_EXPORTER=otlp	instrumentation.grpc.go.image on XR, else instrumentation.go.image in values
Java	OTEL_TRACES_EXPORTER=otlp	instrumentation.grpc.java.image on XR, else instrumentation.java.image in values

To pin a specific auto-instrumentation image per collector:

spec:
  instrumentation:
    grpc:
      java:
        image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.5.0
      dotnet:
        image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:1.2.0

To annotate a workload so it gets instrumented:

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"     # Java
    instrumentation.opentelemetry.io/inject-dotnet: "true"   # .NET
    instrumentation.opentelemetry.io/inject-go: "true"       # Go

---

Kyverno ProviderConfig policy

provider-kubernetes (kubernetes.m.crossplane.io) is a namespaced provider variant. Unlike the standard cluster-scoped provider-kubernetes, it creates resources using a ProviderConfig that must exist in the same namespace as the composed resource. This means every namespace where a CustomOTelCollector is deployed needs its own ProviderConfig.

Rather than requiring users to manually create a ProviderConfig in each namespace, the chart installs a Kyverno ClusterPolicy (generate-otel-provider-config) that automatically generates one:

User creates CustomOTelCollector in namespace "my-app"
  → Kyverno detects the new resource
  → Kyverno generates ProviderConfig "in-cluster" in namespace "my-app"
  → Crossplane can now compose Objects in that namespace

The generated ProviderConfig uses InjectedIdentity credentials — no secrets or extra configuration needed. It is kept in sync (synchronize: true) and also applied retroactively to pre-existing CustomOTelCollector resources (generateExisting: true).

To disable this policy (e.g. if you manage ProviderConfig resources yourself):

kyverno:
  providerConfigPolicy:
    install: false

---

Chart values reference

Value	Default	Description
providerKubernetes.providerConfig.name	in-cluster	Name of the ProviderConfig used by provider-kubernetes
exporters.flow.name	otlphttp/loki	Exporter name for the logs/flow pipeline
exporters.flow.endpoint	http://flow-loki-write:3100/otlp	Endpoint for the flow exporter
exporters.flow.tlsInsecure	true	Disable TLS verification for the flow exporter
exporters.splunk.name	otlp/splunk	Exporter name for the logs/splunk pipeline
exporters.splunk.endpoint	http://splunk-collector.observability:4317	Endpoint for the Splunk exporter
exporters.splunk.tlsInsecure	true	Disable TLS verification for the Splunk exporter
exporters.traces.name	otlp/tempo	Exporter name for the traces pipeline
exporters.traces.endpoint	http://flow-tempo:4317	Endpoint for the traces exporter
exporters.traces.tlsInsecure	true	Disable TLS verification for the traces exporter
processors.memoryLimiter.checkInterval	1s	How often the memory limiter checks usage
processors.memoryLimiter.limitMib	400	Memory limit in MiB
processors.batch.timeout	1s	Maximum time before a batch is sent
instrumentation.collectorPort	4317	gRPC port on the collector service
instrumentation.dotnet.image	""	Default .NET auto-instrumentation image (cluster-wide)
instrumentation.go.image	""	Default Go auto-instrumentation image (cluster-wide)
instrumentation.java.image	""	Default Java auto-instrumentation image (cluster-wide)
kyverno.providerConfigPolicy.install	true	Install the Kyverno ClusterPolicy that auto-generates a ProviderConfig in each namespace where a CustomOTelCollector is deployed
rbac.install	true	Install ClusterRole/ClusterRoleBinding for provider-kubernetes and Crossplane
rbac.serviceAccountName	provider-kubernetes	ServiceAccount used by the provider-kubernetes pod
rbac.serviceAccountNamespace	crossplane-system	Namespace of the provider-kubernetes ServiceAccount