Taming Observability at Scale in a Multi-Cluster Kubernetes Platform at Bloom... Joe Nathan Abellard

Don't miss out! Join us at our next KubeCon + CloudNativeCon events in Mumbai, India (18-19 June, 2026), Yokohama, Japan (29-30 July, 2026), and Shanghai, China (8-9 September, 2026). Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io Taming Observability at Scale in a Multi-Cluster Kubernetes Platform at Bloomberg - Joe Nathan Abellard, Bloomberg Bloomberg runs a managed, multi-cluster Kubernetes platform built atop Karmada to support AI and streaming analytics workloads. This comes with challenges around observability at scale. To meet disaster recovery requirements, we use a multi-region architecture where each Karmada control plane is hosted on management clusters spanning multiple regions. This helps ensure high availability, but also adds complexity related to observability. For example, how do we aggregate and visualize metrics across multiple Prometheus servers when each management cluster has a dedicated Prometheus setup? This talk covers our multi-region architecture to meet DR requirements and our Prometheus stack with Thanos for global metrics aggregation. We’ll explore how we choose the right signals and define meaningful alerts in a complex multi-cluster environment to curb alert fatigue, while ensuring timely issue detection. We’ll also discuss the challenges of defining SLIs and SLOs in a multi-tenant platform.