I Built an LLM From Scratch

In this video CJ explains chat bots, neural networks, tokenization, embeddings, transformers and more. He shows real working code for everything and gives historical context along the way. View the code here: https://github.com/w3cj/how-llms-work 00:00 - intro 01:54 - history of chatbots 04:21 - chat bot code 06:49 - black box thinking 07:41 - what are neural networks? 08:33 - history of neural networks 10:27 - XOR neural network code 15:35 - what is tokenization? 16:32 - history of tokenization 17:55 - tokenizer code 21:37 - what are embeddings? 22:04 - history of embeddings 23:33 - what is a vector? 24:39 - embedding code 30:48 - history of transformers 32:33 - what is a transformer? 33:11 - what is self attention? 34:25 - what is a feed forward network? 34:37 - what are stacked transformer blocks? 41:37 - what are softmax, temperature and top-p? 42:54 - the autoregressive loop 43:28 - the context window 44:45 - the full picture 45:26 - what is fine-tuning 46:39 - RLHF 47:24 - tool calling 48:36 - the AI summit 49:51 - final thoughts Claude Shannon | https://en.wikipedia.org/wiki/Claude_Shannon Betty Shannon | https://en.wikipedia.org/wiki/Betty_Shannon Prediction and Entropy of Printed English (1951) | https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf Alan Turing | https://en.wikipedia.org/wiki/Alan_Turing Turing Test | https://en.wikipedia.org/wiki/Computing_Machinery_and_Intelligence Joseph Weizenbaum | https://en.wikipedia.org/wiki/Joseph_Weizenbaum ELIZA | https://en.wikipedia.org/wiki/ELIZA ELIZA - Weizenbaum (1966) | https://dl.acm.org/doi/10.1145/365153.365168 ELIZA Archaeology Project | https://sites.google.com/view/elizaarchaeology/blog/3-weizenbaums-secretary Computer Power and Human Reason | https://archive.org/details/computerpowerhum0000weiz_v0i3 Kenneth Colby | https://en.wikipedia.org/wiki/Kenneth_Colby PARRY | https://en.wikipedia.org/wiki/PARRY Artificial Paranoia (1971) (PDF) | https://courses.cs.umbc.edu/graduate/671/fall20/resources/colby_71.pdf ELIZA connected to PARRY - RFC 439 | https://datatracker.ietf.org/doc/html/rfc439 ALICE | https://en.wikipedia.org/wiki/Artificial_Linguistic_Internet_Computer_Entity SmarterChild | https://en.wikipedia.org/wiki/SmarterChild McCulloch & Pitts (1943) | https://en.wikipedia.org/wiki/A_Logical_Calculus_of_the_Ideas_Immanent_in_Nervous_Activity Frank Rosenblatt | https://en.wikipedia.org/wiki/Frank_Rosenblatt Perceptron | https://en.wikipedia.org/wiki/Perceptron Rosenblatt's Perceptron - Cornell | https://news.cornell.edu/stories/2019/09/professors-perceptron-paved-way-ai-60-years-too-soon New Navy Device Learns by Doing - NYT (1958) | https://timesmachine.nytimes.com/timesmachine/1958/07/08/issue.html Perceptrons | https://en.wikipedia.org/wiki/Perceptrons_(book) Backpropagation - Rumelhart, Hinton & Williams (1986) | https://www.nature.com/articles/323533a0 BPE - Philip Gage (1994) (PDF) | https://www.derczynski.com/papers/archive/BPE_Gage.pdf Byte Pair Encoding | https://en.wikipedia.org/wiki/Byte-pair_encoding BPE for NMT - Sennrich, Haddow & Birch (2015) | https://arxiv.org/abs/1508.07909 Gottlob Frege | https://en.wikipedia.org/wiki/Gottlob_Frege Foundations of Arithmetic - Frege (1884) | https://archive.org/details/foundationsofari00fregrich Context Principle | https://en.wikipedia.org/wiki/Context_principle J.R. Firth | https://en.wikipedia.org/wiki/John_Rupert_Firth Studies in Linguistic Analysis - Firth (1957) | https://archive.org/details/studiesinlinguis0000vari Word2Vec - Mikolov et al. (2013) | https://arxiv.org/abs/1301.3781 Word2Vec Negative Sampling - Mikolov et al. (2013) | https://arxiv.org/abs/1310.4546 Attention Mechanism - Bahdanau, Cho & Bengio (2014) | https://arxiv.org/abs/1409.0473 GNMT - Wu et al. (2016) | https://arxiv.org/abs/1609.08144 Xavier Initialization - Glorot & Bengio (2010) | https://proceedings.mlr.press/v9/glorot10a.html Attention Is All You Need - Vaswani et al. (2017) | https://arxiv.org/abs/1706.03762 Transformer | https://en.wikipedia.org/wiki/Transformer_(deep_learning) Common Crawl | https://en.wikipedia.org/wiki/Common_Crawl RLHF | https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback Dartmouth Workshop (1956) | https://en.wikipedia.org/wiki/Dartmouth_workshop Dartmouth AI Proposal (1955) | https://www-formal.stanford.edu/jmc/history/dartmouth/dartmouth.html Mamba - Gu & Dao (2023) | https://arxiv.org/abs/2312.00752 xLSTM - Beck et al. (2024) | https://arxiv.org/abs/2405.04517 Jamba - Lieber et al. (2024) | https://arxiv.org/abs/2403.19887 JEPA - Yann LeCun (2022) | https://openreview.net/pdf?id=BZ5a1r-kVsf ---------------------------------------------------------------------------- http://www.syntax.fm Brought to you by https://sentry.io/syntax #LLM #explained #machinelearning #webdevelopment #webdeveloper #javascript #typescript #syntax #syntaxfm