Apache Spark Upgrade Agent on EMR | Let's Talk About Data

"Upgrading Apache Spark applications takes 3-6 months per major version. Troubleshooting production failures consumes 20-40% of data engineering time. These two workflows are the biggest time sinks teams face today. In this session, we'll live-demo two AI-powered agents that compress these workflows from weeks to minutes. The Spark Troubleshooting Agent diagnoses real production failures — executor OOM, Iceberg schema drift, YARN container kills — by automatically correlating signals across driver logs, container diagnostics, and Spark event history. The Spark Upgrade Agent takes a PySpark application from Spark 3.5 to 4.0 on EMR Serverless — iteratively fixing breaking changes through runtime validation until the job succeeds, with data quality comparison at the end. Everything happens through natural language. No manual log searching. No reading migration guides. Just describe the problem and let the agent work."