Context Rot: Its Implications and Potential Solutions with Jeff Huber

Watch the entire AI Superstream: https://learning.oreilly.com/videos/ai-superstream-context/0642572273927/0642572273927-video403838/?utm_medium=social&utm_source=youtube&utm_campaign=free+trial&utm_content=ai+superstream "Long context was not quite the panacea that the labs were making it out to be," says Jeff Huber, CEO of Chroma, so his team spent 2024 running experiments to figure out exactly when and why it breaks, then codified it in a research report called _Context Rot._ In this talk from AI Superstream Jeff walks through the _Context Rot_ research: why needle-in-a-haystack benchmarks are nearly meaningless, how distractors and semantic ambiguity tank model performance far sooner than you'd expect, and why precision in context matters just as much as recall. Most serious builders already cap their trust in models at 40–100K tokens—regardless of what the labs advertise. Follow O'Reilly on: LinkedIn: https://www.linkedin.com/company/oreilly/ Facebook: http://facebook.com/OReilly Instagram: https://www.instagram.com/oreillymedia BlueSky: https://bsky.app/profile/oreilly.bsky.social