Automating MCP Server Testing: Engineering Reliability for Agentic Systems - Neethu Elizabeth Simon

Join us at the premier vendor-neutral open source conference, where developers and technologists come together to collaborate, share knowledge, and explore the latest innovations and advancements in open source technology. Learn more at https://events.linuxfoundation.org/ Automating MCP Server Testing: Engineering Reliability for Agentic Systems - Neethu Elizabeth Simon, Arm AI agents don’t fail like traditional software. They don’t just throw exceptions, they drift. They misinterpret tools, invoke the wrong functions or behave differently across environments. When deploying Arm’s Open Source custom MCP server to power AI assistants for architecture development, migration, and optimization, we faced a critical question: how do we test a system built for nondeterministic interaction? In this talk, I’ll share how we moved from manual validation to a repeatable, CI-enforced testing strategy using Pytest and Testcontainers. We spin up real MCP server in Docker during tests, validating tool discovery, invocation, and protocol compliance end-to-end. This isn’t about mocking LLM output. It’s about testing the contract between agents and tools. The key insight: treat your MCP server like production infrastructure, not experimental glue code. Because “it worked on my machine” is not a deployment strategy. Session takeaways: • A demo of Arm’s Open Source MCP server(github.com/arm/mcp) • Why unit tests are insufficient for agent-facing systems • How we run MCP server inside containerized test environments • How GitHub Actions automate CI integration testing