Indexes, exclusions and what is kept
How a log is assigned to an index, and what each index controls.
What an index is #
An index is a rule that claims matching logs and decides how they are stored. Everything about storage — retention, exclusions and a daily cap — travels with the index rather than with the log.
- Filter — which logs this index claims.
- Retention — how long claimed logs are kept.
- Exclusions — attributes whose values are never stored.
- Daily cap — the maximum volume the index accepts per day.
Which index claims a log #
Indexes are evaluated in order and the first matching index claims the log. A log that matches no index uses your organisation's default index.
Order decides ownership
Put specific indexes above general ones. An index matching everything on env:production will claim a log that a more specific index below it also matches.
Exclusions #
An excluded attribute is removed before the log is written. Use it for values you never need to query and would rather not hold.
- Excluded values are removed at write time, not at read time.
- Search cannot find a log by an excluded value, because the value is not stored.
- Excluding an attribute does not drop the log itself.
Excluded logs and archiving #
Archiving runs as its own consumer, independent of index routing. A log that an index excludes is still archived when it matches an archive filter, and it still counts toward log-based metrics.
Exclusions and archives are independent. Excluding an attribute controls what live search holds; archiving controls what is kept off-platform. Choose each for its own purpose.
Retention #
Retention is set per index, so different classes of log can age differently. A high-volume index carrying short-lived debug output can be given a short retention while a low-volume index of audit records is kept for months.
Daily caps #
An index can carry a daily volume cap. When the cap is reached, further logs matching that index are dropped for the rest of the day and the drop is counted, so a runaway service cannot consume an index's whole budget.
Caps are counted and visible
Every index reports how many logs it accepted and how many it dropped for the current day. A cap that is silently reached looks like missing data; the counter tells you which index is at its limit.
Designing an index set #
- Errors — narrow filter, long retention, no exclusions.
- Debug output — broad filter, short retention, a daily cap.
- Audit records — very narrow filter, longest retention, exclusions off.
Start with three. You can always add a fourth once you can see where the volume is actually going.