Dynamic instrumentation

How it works #

A probe is a small piece of code attached to a live process. It watches for a condition — a function call, an error, a specific argument — and emits telemetry when that condition is met.

The agent pulls its probe configuration from Sentinel on startup and refreshes it on a short interval, so enabling or disabling a probe takes effect without a restart or a redeploy.

GET /otlp/v1/probes

Supported languages #

Probes are available for Java, Python, .NET, PHP, Node.js, Go and Rust. The instrumentation is language-specific, and a probe is always attached to one process type.

Probe limits #

Probes run inside your process, so their size and complexity are bounded. These limits protect the host.

• A pattern may be up to 256 characters.
• A probe program may contain up to 256 steps.
• A probe may repeat up to 1,000 times.
• A probe expression tree may be 64 levels deep and hold 2,000 nodes.
• A probe document is limited to 64 KiB.

Capabilities #

Some probes use a matcher that runs in linear time. An agent advertises the linear-matches capability when its own matcher provides that guarantee, and Sentinel withholds probes that depend on it from agents that do not.

This keeps a probe from stalling the thread it is attached to: a probe is only sent to an agent that can evaluate its conditions within a bounded time.

Choosing where to probe #

• Always on — attached at startup and active until removed.
• On demand — attached when a specific investigation needs it.
• One-shot — fires once, then detaches.

Start narrow. A probe on a hot path runs on every request; scope it to a specific service, version or environment before widening it.

Writing a condition #

A probe declares what to match and what to emit. The emitted attributes become searchable fields, so name them the way you will query them.

{
  "name": "payment-retries",
  "service": "checkout",
  "env": "production",
  "type": "always_on",
  "language": "java",
  "target": {
    "class": "com.acme.checkout.PaymentClient",
    "method": "charge",
    "matchType": "substring",
    "match": "card_declined"
  },
  "emit": {
    "message": "payment declined",
    "status": "warn",
    "attributes": {
      "payment.provider": "stripe",
      "order.source": "checkout"
    }
  }
}

Operational guidance #

• Attach probes to non-production environments first whenever possible.
• Remove a probe once the investigation that needed it is finished.
• Watch the agent's memory after adding probes to a hot path.
• A probe can be scoped to a version, which is the safest way to compare two deployments.