Sampling Without Losing the Story

Teams often treat sampling as a finance lever first and a measurement decision second. That order creates blind spots exactly where rare failures concentrate: unusual tenants, uncommon locales, or edge payloads.

Prefer head-based sampling for routine traffic and keep a guaranteed path for errors and high-latency traces. When budgets tighten, reduce cardinality in labels before you starve the trace volume that explains a slow dependency.

Document the sampling policy next to the signal catalog. On-call engineers should know which stories the data can still tell at 2 a.m., and which questions require a temporary sampling override.

In our Hong Kong engagements, the teams that recover fastest are not the ones with the densest storage—they are the ones who can restate what was sampled and why.