12 ms·
A Baby GPT
- marcodiego 4y ago> This is a baby GPT with two tokens 0/1 and context length of 3, viewing it as a finite state markov chain. What is actually the difference between this and a "pure" Markov chain? Is GPT simply a Markov chain? EDIT: Better explained here: https://colab.research.google.com/drive/1SiF0KZJp75rUeetKOWqpsA8clmHP6jMg?usp=sharing https://colab.research.google.com/drive/1SiF0KZJp75rUeetKOWq...
- karpathy 4y agoYes but in the same way as saying that computers are just a Markov chain.
- nurettin 4y agoAfter all the epic content and research, I think this was your worst explanation ever.
- karpathy 4y ago:(
- HopenHeyHi 4y ago[flagged]
- ShamelessC 4y ago[flagged]
- yinser 4y agoPay no mind to him I think it’s brilliant. There, canceled out :) Many of us look to you as the GOAT of teaching
- nurettin 4y agoSorry, I just didn't like the comparison between saying GPT is a markov chain and computers are a markov chain. Really learned something from the actual colab. Thank you, appreciated. Nobody likes a sad karpathy.
- whiplash451 4y agoUnfair and irrelevant. Keep it up, Andrej!
- ImprobableTruth 4y agoComputers are (with a couple caveats) deterministic, so I think the better analogy would be to say that it's the same as saying that "computers are just state machines". But this is a bit nitpicky. The core idea is of course conveyed either way, that yes, this very abstract model does describe it while also completely missing that the magic is in how it's done.
- gary_0 4y agoIf I can continue the nitpickery, plenty of computer algorithms are deterministic but also pseudorandom--including Markov chains. Only when the randomness is outside your control are things truly non-deterministic. A simulation with many possible outcomes is still deterministic if you get to choose the "seed". In other words, statistical permutations are deterministic if you control all the input variables (in terms of computer science; physics is another story).
- dr_kiszonka 4y agoComputers have memory! Oh, wait... so does ChatGPT. Or am I forgetting something...
- transcoderx 4y agoSince every state has a set of probabilities for the next state, I guess you could call it that. It would be very misleading, though.
- deleted 4y ago[deleted]
- Imnimo 4y agoEven in the case of a full size GPT, one could equivalently write out a lookup table for every possible permutation of tokens in the context window and the resulting output distribution. The table would of course be impossibly large.
- oezi 4y agoDue to combinatorial explosion this would be infeasible even for a single input vector.
- KRAKRISMOTT 4y agoThe mathematical modeling of most NLP tasks is Markovian. (More specifically, they tend to be HMMs, Hidden Markov Models). One of the popular "solutions" for Markovian processes is the Metropolis algorithm which you might find relevant. Note in statistical learning the mathematical formulation generally operates as part of the loss function, the actual "model" is usually a neural network (in the context of deep or variational learning).
- lamp987 4y ago>Is GPT simply a Markov chain? Yes, but much more efficiently stored and looked up than just a massive table. The way its very large space is encoded in a small amount of memory is what makes GPT so interesting.
- jonny_eh 4y agoA neural net instead of a table?
- nuclearnice3 4y agoA markov chain maps states to states. You move from word "I" to word "will" or "can" or "want" The transition probabilities could be stored in a huge table. You have a word and the probability of each other word. Draw a number and hop. [1,2] Alternatively you could ask a neural net for the next state. Pass it the current token or the last 4000 token. [1] https://towardsdatascience.com/text-generation-with-markov-chains-an-introduction-to-using-markovify-742e6680dc33 https://towardsdatascience.com/text-generation-with-markov-c... [2] https://crowintelligence.org/2021/05/27/a-simple-markov-chain-text-generator-in-python/ https://crowintelligence.org/2021/05/27/a-simple-markov-chai...
- oezi 4y agoGPT is not architecturally implemented as a Markov chain. But given enough time one could model it as one.
- ttul 4y agoSort of, but only if you consider that the states are massive. It’s not just “what’s the next word after this one.” It’s, “what’s the next word after this one, considering the other several hundred other words that came first.” So, the input sequence is an extremely high-dimensional state value in a gigantic Markov chain.
- toxik 4y agoTo me, that sounds like not a Markov chain. Yes if you consider the Markov chain over entire sequences, but that’s not what people mean with Markovian.
- nothrowaways 4y agoNo, gpt is generative.
- kgwgk 4y agoHow is a Markov chain “non-generative”?
- drexlspivey 4y agoShould have named it picoGPT
- deleted 4y ago[deleted]
- Jknitgirl 4y ago[flagged]
- adamnemecek 4y agoI have written a paper on this recently viewing transformers as Hopf algebras https://arxiv.org/abs/2302.01834 https://arxiv.org/abs/2302.01834. This paper is very relevant https://arxiv.org/abs/1206.3620 https://arxiv.org/abs/1206.3620.
- Blahah 4y agoCan you explain the implications of this for people who aren't expert in the field?
- adamnemecek 4y agoI argue that Hopf algebras, which are spicy tensors, provide a better autodiff and are a better model for SSMs, convnets, transformers, and diffusion models.
- ilaksh 4y agoHow do we use them in place of transformers? Would they allow something like the power of ChatGPT but with interpretability or modularity? Would they make it easier to create systems that behave like hybrid architectures or automatically combine different types of models (algebras?) It sounds so promising. But I have to assume that if it really is then this can be translated to code that can run on Colab or something, at least by some genius. Until it is I have to be a little skeptical and I'm not going to be able to access it anyway.
- adamnemecek 4y agoBecause it would show that all the models are just slightly different Hopf algebras. And yes to interpretability and modularity. It would provide guidance for building new architectures.
- dontwearitout 4y agoI've seen you pop up and recommend Hopf algebras and your paper in several deep learning threads here on HN. I am interested in your work, but find the topic very opaque and unintuitive (despite a background in physics and CS). Do you have any recommended educational resources for a gentle introduction to this?
- alexahn 4y agoI have a hunch that a lot of the deep relationships between words in languages can be inferred by using something like binary regular expressions. Defining states by a context length have a close analogue to this technique to speed up applying regular expressions over large data: https://swtch.com/~rsc/regexp/regexp4.html https://swtch.com/~rsc/regexp/regexp4.html. Now imagine if context lengths could be grouped together to form higher types, that could then be used in a regular expression. For example, currently 3 bits are grouped to form the context window, and in the realm of regular expressions, one could say this forms the "3 bit alphabet". Now a higher level alphabet could be built upon the "3 bit alphabet", maybe a "9 bit alphabet" from a grouping of 3 symbols from the "3 bit alphabet". In other words, imagine if you had a transition diagram, but instead of each state being a context window that relates to direct data, you had something like a parsing graph for a programming language. Now imagine you had a statistical method to gradually build these nested parsing structures. In a way convolutions and regular expressions are similar, they are a way of partitioning data to eventually signify different values to different parts, except convolutions have better statistical properties.
- roflyear 4y agoHmm I imagine there's a way to "freeze" neural nets into these types of systems to optimize them?
- alexahn 4y agoMy hunch is that all probabilistic methods eventually degrade and they cannot capture "rules". Otherwise you would end up with a classical computer. I think what we are really looking for is a way to learn "rules" such that they remain stable over a certain number of iterations. Hopefully enough iterations to do useful work. Once you have a way to do this, you can extend the number of iterations by further imprinting on the system. The simplest example of this is if you have an oscillating signal whose amplitude is slowly decaying. A certain number of cycles will have an amplitude above a certain threshold that allows for useful work.
- knome 4y agoThe problem with using a regex as you describe is that GPT isn't working at the level of words, its working at a level of meaning and intent. You can have it translate rather meaningfully between metaphors that exist in completely different contexts. Somewhere in the 90+ levels of interconnected neurons, the literal meanings are abstracted and re-encoded. A regex isn't going to accomplish that. edit: you might be able to match this little tiny GPT, but not one of the billions-of-parameters varieties.
- egl2021 4y agoThank you, Andrej. You're a pedagogical gem!
- voz_ 4y agoI made a version using PyTorch 2.0, if anyone cares: https://colab.research.google.com/drive/1iar9NEmOjWf-RPCxVklwL6EIa4FPZRee?usp=sharing https://colab.research.google.com/drive/1iar9NEmOjWf-RPCxVkl... The Good: replacing gpt(x) w/ opt_gpt(x) speeds it up ~1.5-2x at first blush (really crude measurement) The Bad: The very first run took 43s on the initial compile! Gross! We had a mode for progress bar somewhere, but it would be cool if it was on by default. It would also be neat if there was a better set of utils around preheating. The Ugly: Nothing ugly?
- Buttons840 4y agoI've been reading this paper with pseudocode for various transformers and finding it helfpul: https://arxiv.org/abs/2207.09238 https://arxiv.org/abs/2207.09238 "This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (not results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models."
- Jiocus 4y agoFinally, a concise explanation why this generation of chatbots –while absolutely superior to previous art (where applicable)— still throws me back to discovering markov chains*, 'cat'-ing countless amount of text into a 'corpora.txt' to build models of various personalities. The idea that a larger corpora == better was quickly established, but I hit a ceiling quite soon on early RPi's. It was possible to seed a start word for text generation, a Baby prompt we might call it? Nice feature was being able to make a mini chain (model) of the user input sentence, then merge that statistics into the bots chain. The verbatim record of what had been said was obscured but some words and their relations —the experience— was kept. So I imagined, at least. * I recommend giving the 'markovify' README a view: https://github.com/jsvine/markovify#basic-usage https://github.com/jsvine/markovify#basic-usage edit: LLMs still can't deliver an as convincing display of an Aleister Crowley-persona, but I can barely handle a prompt so we're even.
- bulbosaur123 4y agoAutoGPT seems to be utterly superior to BabyAGI. Reasoning: you can actually control and steer AutoGPT unlike BabyAGI which runs on full auto-pilot. This is a crucial feature. The only benefit I see with BabyAGI is that it allows to use local models. Ideally AutoGPT should add a feature to use local models as well so we don't have to waste money on OpenAI API use. What am I missing here?
- kolinko 4y agoThe link is not about BabyAGI.
- fancyfredbot 4y agoWould an LLM predicting the next bit rather than the next token have any advantages over GPT? In some ways it seems more general - for example it can predict an image or even executable machine code. I'm guessing that it's more difficult to train however?
- deleted 4y ago[deleted]
- thethimble 4y agoGPT’s vocab size is north of tens of thousands. Included in this vocab are the 256 unique byte values (2^8). This means GPT can already generate/interpret all byte sequences. The vocab size is larger than 256 (or in your example 2) because some sequences of bytes occur much more frequently than others. This allows the model to generalize more efficiently. By analogy consider that we think of “words” as atomic concepts much more than we think of the individual lines that make up the letters in a word - even though the former can be thought of being composed of the latter.
- informal007 4y agoMay be all neural network are build base on statistic routain, But we can't enumerate all word like 010 in computer, so we compress the data. neural network is like the combine statistic routain and compress base large amount of data.
- informal007 4y agoThis is a very good idea to research ChatGPT if open ai open the method that how to make ChatGPT.