Quantiles at Scale: Choosing the Right Estimation Algorithms for Observability - Mike Shi

Don't miss out! Join us at our next KubeCon + CloudNativeCon events in Mumbai, India (18-19 June, 2026), Yokohama, Japan (29-30 July, 2026), and Shanghai, China (8-9 September, 2026). Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io Quantiles at Scale: Choosing the Right Estimation Algorithms for Observability - Mike Shi, ClickHouse Quantiles like p90 and p99 sit at the heart of observability. They define dashboards, drive SLOs, and shape how teams reason about system performance. They are also some of the most expensive metrics to compute, and the cost grows fast as data volumes increase. To keep up, observability systems rely heavily on approximate quantile algorithms such as sketches and probabilistic data structures, including t-digest. These approaches work well at small and medium scale, but at tens or hundreds of petabytes, things start to creak and limitations become apparent. We share hard won lessons from operating ClickHouse at extreme scale, where quantile estimation must remain accurate and affordable over hundreds of petabytes of data. We break down the most common quantile algorithms used in observability today, explain their real trade offs, and show when each approach makes sense. We also explore a critical design decision: when quantiles should be computed on the fly at query time versus pre aggregated during ingestion. The goal is to give you a practical framework for choosing quantile algorithms that scale, rather than blindly relying on defaults that stop working as your data grows.