8 ms·
The GPT Architecture, on a Napkin
- macleginn 4y agoA small, very clearly written and well commented implementation: https://github.com/karpathy/minGPT https://github.com/karpathy/minGPT
- bogomipz 4y agoThis was a great read. Does anyone know what software the author may have used for the hand-written or napkin-sketch looking graphics?
- pnt12 4y agoLooks similar to excalidraw but may be something else.
- chronolitus 4y agoHi! I'm the author (Daniel). I used OneNote on some old surface tablet I had lying around, but these days I'm not sure I would use it again (for example because it doesn't support exporting parts of a page to .svg)
- bogomipz 4y agoThanks, yes that does sound like an unfortunate limitation but I wanted to say your graphics really look fantastic! I really enjoyed reading this.
- deleted 4y ago[deleted]
- mysterydip 4y agoIf one were to make a markov chain with the same amount of input data, would the result be the same? Markov chain chatbots have been a thing for years, just on a much more limited set of data.
- igorkraw 4y agoIt is a Markov chain. Your input context is your state.
- n2d4 4y agoNo. The state size is 50257^2048. The vast majority of states have never been seen and will never be seen in all of humanity. For an example, if your training set consists of the words rain and thunder used interchangeably a lot, but the word "today" is only used once in the sentence "there is no rain today", then a Markov chain based on the data would never output "there is no thunder today", but a transformer might. In other words, information compression (eg. equating rain with thunder) isn't just for practicability, it's a necessary requirement for (the current generation of) good language models.
- mysterydip 4y agoAh, that's what I was missing. Thanks!
- wklm 4y agoVery cool write-up, would love it even more if author included the references.
- chronolitus 4y agoGood point, will do!
- mudrockbestgirl 4y agoThat's a great summary, but it's important to understand that much more goes into training these models. The architecture is not any kind of secret sauce, or special in any way. It's just a typical Transformer. I call this "architecture porn" - people love looking at neural net architectures and think that's the key to success. If only you know the algorithm! It's so simple! But reality is usually much messier. The real training code will be littered with hundreds of ugly little tricks to make it work. A large part of it will be input preprocessing and data engineering, tricks to deal with exploding/vanishing gradients, monitoring, learning rate schedules and optimizer cycling, complexity for distributed training, regularization tricks, changing parts of the architecture for performance reasons (like attention), and so on.
- WanderPanda 4y agoI‘ve recently come to the conclusion that the magic of fully connected neural networks is that there are almost no tricks to reach close to sota. Dense layers + relu + adam = it just works
- blackbear_ 4y agoSorry but this is just wrong, using only fully connected layers would result in pretty bad performance on images, text, audio, etc., or at the very least require much more data to perform well. At least use the right type of architecture for each data modality, then I agree that the basic version won't perform much worse than sota in the real world.
- jerrygenser 4y agoI think part of parent is wrong but part is correct. There are many rules of thumb that took the last 5+ years to discover but are now quite standard. You are nit picking on fully connected, but if we add dropout, weight initialization, and adaptive learning rate to what they said, then we are fairly close to being able at least get a deep architecture to overfit a toy dataset and be off to the races for then applying it to a larger dataset.
- devxpy 4y agoWould also love this to include InsturctGPT architecture with its RL reward model!
- cuuupid 4y agoNote on the sinusoidal encoding, the reason it’s used is generally speaking twofold: 1 - To encode position somehow (which author details) 2 - Because sine is easy “noise” for the network to learn. There’s also a bunch of cool tricks here even down to the PyTorch implementation to optimize this encoding by exploiting the nature of sine/cosine which is an added reason for its popularity in Transformer architectures. If you like math I recommend diving into it as it’s quick but fun! (Side note it’s also falling out of fashion for other encoding methods. e.g. rotary positional encoding is vastly popular in the reformer branch of transformers)
- WanderPanda 4y agoIs treating the position embeddings as trainable weights already out of fashion again?
- alexmolas 4y agoaccording to the Attention is all you need paper there wasn't huge differences between using sin/cos and trainable weights. But the paper is a little bit old, so I don't know what is the current sota regarding positional embeddings
- deleted 4y ago[deleted]
- yunwal 4y agoI’m new to this stuff, but as I understand it, the “Attention is all you need” paper stated that training the positional encoding weights didn’t improve results for language models specifically, but other papers found that vision transformers performed better with trainable weights.
- khazhoux 4y agoFor a newbie in all this, what's the reason that position isn't just encoded as an integer?
- ec109685 4y agoThis was helpful. What are the basics that allow ChatGPT to output numbers and solve math problems?
- sva_ 4y agoDid you read the article? Under "Encoding"/"Decoding".
- ec109685 4y agoI guess each digit is a word? For example, how does the explanation in the article produce a module that can solve this: “ To calculate the hypotenuse of a triangle with one side that is 12 inches long and another i S side that is 36 centimeters long, a 6th grader might say something like this: "First, we need to convert the 36 centimeters into inches so that both sides of the triangle are in the same units. We can do this by dividing 36 by 2.54, which is the number of centimeters in one inch. This gives us 14.173228 inches. Then, we can use the Pythagorean theorem to find the length of the hypotenuse. The Pythagorean theorem says that in a right triangle, the square of the length of the hypotenuse is equal to the sum of the squares of the lengths of the other two sides. So we can use this formula to find the length of the hypotenuse: a^2 + b^2 = c^2. In our triangle, the length of one side is 12 inches, and the length of the other side is 14.173228 inches. So we can plug those numbers into the formula like this: 12^2 + 14.173228^2 = c^2. Then we just need to do the math to find the value of c. 12^2 is 144, and 14.173228^2 is 201.837296. So if we add those two numbers together, we get 346.837296. And if we take the square root of that number, we get the length of the hypotenuse, which is 18.816199 inches.”
- heisenzombie 4y agoWell, it's wrong for one: It correctly gets the division (I assume it has that fact memorized) but 14.173228^2 is 200.88 not 201.83. it then also does the addition wrong, and the square root is also wrong. You gotta be REAL careful with ChatGPT output that sounds convincing and technical. It's very good at convincingly making stuff up, even math-y science-y sounding stuff.
- seydor 4y agoIt s not intutive why the QK^T matrix is importance, similarity makes more sense
- yunwal 4y agoI have a more intuitive explanation of Q and K. Q (Query) is like a search query. K (Key) is like a set of tags or attributes of each word that the query can look for. Imagine the self-attention scores for the sentence ”The chicken crossed the road because it wanted to get to the other side.” Let’s look specifically at the word “it”. We can imagine the matrix Q_it as a representation of attributes of the word we want “it” to gain context from. For example, “it” is a pronoun, and therefore gains context from nouns. Therefore, the matrix Q_it might have some representation of “noun” in it, because we want it to take context from nouns. In other words, one of the goals during the training process of a transformer network is to train a weight W_q to map a word to a matrix representation of attributes of words it gains context from. So W_q should map any word that’s a pronoun to a noun query. Similarly, K can be thought of as a representation of attributes of each word. So K_chicken should also have a “noun” tag. So one of the goals during training is to train W_k, that maps the latent-space representation of a word (chicken) to a matrix representation (K_chicken) of it’s important attributes (noun). When we take the dot product of Q and K, what we’re finding is the similarity between those 2 matrices. In other words Q_it * K_chicken should have a high value because “it” is looking for nouns in it’s query, and “chicken” is a noun. Obviously this is a very human-centric explanation and how the weights W_q and W_k are trained in practice may not align perfectly with human interpretable concepts, but hopefully helps with understanding.
- sAbakumoff 4y agoPersonalized chatGPT + Boston Dynamics collaboration in creating best friends who can talk about anything
- deleted 4y ago[deleted]
- oars 4y agoGreat article about ChatGPT architecture, thank you.
- _8j50 4y agoIf I was let's say China, what stops me from replicating GPT!
- randyrand 4y agovery little! just competent engineers and lots of gpus
- frozencell 4y agoWhat does it mean?
- visarga 4y agoJust $10M. They have their own GPT-3 class models.
- dragonwriter 4y ago> If I was let’s say China, what stops me from replicating GPT! Well, since its described in a paper and there are open source GPT implementations (e.g., GPT-NeoX), probably not much.
- sideshowb 4y agoI wonder what is the reason different blocks are summed even though they semantically represent very different things - e.g. Positional encoding plus embedding. Intuitively wouldn't you expect the architecture to work better if you kept those things as separate inputs to the next layer? Won't the network have some blind spots where a given word will effectively be invisible if it appears in the wrong position?
- boulos 4y agoAm I the only person that was disappointed that there wasn't actually an architecture diagram on a napkin? The final diagram [1] is also much less "readable" than the original figures from the paper. [1] https://dugas.ch/artificial_curiosity/img/GPT_architecture/fullarch.png https://dugas.ch/artificial_curiosity/img/GPT_architecture/f...