3Blue1Brown

Reinventing Entropy | Compression & Intelligence Part 1

What is the fundamental compressibility of language? Check out our virtual career fair: https://3b1b.co/talent See new projects before they go live: https://3b1b.co/support Animation credit: Manim scenes by Aaron Gostein and Grant Sanderson Shannon’s story, as well as those for various pi creatures, by Mitchell Zemil. Lunar robot and prediction/compression coin by Paul Dancstep NanoGPT animations by Clayton Rabideau The way of visualizing entropy shown here is something I first came across in this excellent post by Chris Olah: https://colah.github.io/posts/2015-09-Visual-Information/ Shannon’s “A Mathematical Theory of Communication” https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf Shannon’s “Prediction and Entropy of Printed English” https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf Scientific American article that mentions the story with Von Neumann suggesting the name Entropy: https://www.esalq.usp.br/lepse/imgs/conteudo_thumb/Energy-and-Information.pdf Timestamps: 0:00 - On “Compression is intelligence.” 3:28 - The warmup example 10:46 - What perfect compression looks like 14:47 - Defining information 17:40 - Information of language 24:29 - Defining Entropy 31:14 - 3b1b Talent ------------------ These animations are largely made using a custom Python library, manim. See the FAQ comments here: https://3b1b.co/faq#manim Music by Vincent Rubinetti. https://vincerubinetti.bandcamp.com/album/the-music-of-3blue1brown https://open.spotify.com/album/1dVyjwS8FBqXhRunaG5W5u ------------------ 3blue1brown is a channel about animating math, in all senses of the word animate. If you're reading the bottom of a video description, I'm guessing you're more interested than the average viewer in lessons here. It would mean a lot to me if you chose to stay up to date on new ones, either by subscribing here on YouTube or otherwise following on whichever platform below you check most regularly. Mailing list: https://3blue1brown.substack.com Twitter: https://twitter.com/3blue1brown Bluesky: https://bsky.app/profile/3blue1brown.com Instagram: https://www.instagram.com/3blue1brown Reddit: https://www.reddit.com/r/3blue1brown Facebook: https://www.facebook.com/3blue1brown Patreon: https://patreon.com/3blue1brown Website: https://www.3blue1brown.com