Collectors and agents

What a collector does #

A collector runs next to your workload: it receives telemetry from your instrumented services, batches it, and forwards it to Sentinel. Batching is what makes ingest affordable — a few hundred spans become one request instead of hundreds.

The OpenTelemetry Collector is the standard choice, and the configurations below assume it.

The OpenTelemetry Collector #

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 8192
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20

exporters:
  otlp/sentinel:
    endpoint: https://sentinel.example.com/otlp
    headers:
      x-sentinel-api-key: ${SENTINEL_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/sentinel]

The memory_limiter processor is worth keeping on any shared host: it sheds load before the collector is killed by the operating system, which turns a memory spike into backpressure instead of a restart.

Tuning batch size #

Batch size trades latency against throughput. A short timeout keeps dashboards responsive; a large batch is more efficient.

• Interactive debugging — timeout: 1s, send_batch_size: 512.
• Normal production — timeout: 5s, send_batch_size: 8192.
• High-volume services — timeout: 10s, send_batch_size: 16384.

Kubernetes #

Run the collector as a DaemonSet so each node forwards its own workloads. The pod name and node name become resource attributes, which is what populates the Kubernetes views.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
spec:
  selector:
    matchLabels: { app: otel-collector }
  template:
    metadata:
      labels: { app: otel-collector }
    spec:
      hostNetwork: true
      containers:
        - name: collector
          image: otel/opentelemetry-collector:latest
          args: ["--config=/etc/otel/config.yaml"]
          env:
            - name: SENTINEL_API_KEY
              valueFrom:
                secretKeyRef: { name: sentinel, key: api-key }
          volumeMounts:
            - { name: config, mountPath: /etc/otel }
      volumes:
        - name: config
          configMap: { name: otel-collector }

The Kubernetes infrastructure agent #

A second, read-only agent enriches telemetry with cluster metadata and exposes a health endpoint for probes. It runs with read-only access to the Kubernetes API and has no cluster-level write permissions.

• k8sattributes — pod, namespace and workload metadata on every signal.
• kubeletstats — CPU, memory, disk and network per container.
• Health endpoint — 0.0.0.0:13133, reporting collector readiness.

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

receivers:
  k8sattributes:
  kubeletstats:
    collection_interval: 20s

What the agent does not do #

The agent reports on the workloads it observes. It does not manage them: it creates nothing, changes nothing and deploys nothing in your cluster.

Verifying a collector #

Send a span and look for it. A successful send returns 200; the span appears under APM → Traces within a few seconds. If it does not appear, the usual causes are an incorrect endpoint path, a missing API key header, or a firewall between the collector and Sentinel.