Log pipelines

What a pipeline does #

A pipeline is an ordered list of rules applied to each log as it is received. Rules can drop records, redact fields, add attributes, or change how a record is stored.

Pipelines run on the ingest path, before storage. They apply to every record your organisation sends, which makes them the right place for anything you do not want to store.

Writing a pipeline #

• Open Logs → Pipelines and create a pipeline.
• Add rules in the order you want them evaluated.
• Save. New logs pick the change up immediately; stored logs are unaffected.

Rule types #

• Filter — keep only records matching a condition, or drop them.
• Redact — replace or remove an attribute's value.
• Rename — move a value from one attribute to another.
• Add attribute — attach a constant attribute to matching records.
• Parse — extract structured fields from the message.

Order and conflicts #

Rules are evaluated top to bottom and the first match wins. Once a rule claims a record, the rules below it are not evaluated.

Redaction #

Redaction runs before the record is stored, so the original value is never written. This is the place to remove credentials, tokens and personal data.

{
  "name": "Remove credentials",
  "type": "filter",
  "condition": {
    "kind": "or",
    "terms": [
      {"kind": "word", "value": "authorization"},
      {"kind": "keyed", "key": "@api_key", "value": "*"},
      {"kind": "keyed", "key": "@password", "value": "*"}
    ]
  }
}

Remapping #

Remapping rules reshape attributes after parsing. They are how you turn a field a library emits into a name your organisation already uses, without changing the emitting code.

• Rename http.status to http.status_code so error searches find it.
• Strip a prefix such as myapp. from every attribute name.
• Move a value out of the message and into a structured attribute.

Verifying a change #

After saving, search the logs for records the rule should have caught. Pipelines apply to newly received records, so allow a few minutes before judging the result.