14 ms·
A GPT in 60 Lines of NumPy
- durdn 4y agoVery impressive. Recently I watched this really amazing lecture on building GPT from scratch from Karpathy, I was blown away: https://www.youtube.com/watch?v=kCc8FmEb1nY&t=642s https://www.youtube.com/watch?v=kCc8FmEb1nY&t=642s
- est 4y ago> GPT-3 was trained on 300 billion tokens of text from the internet and books: > GPT-3 is 175 billion parameters Total newbie here. What does these two numbers mean? If running huge number of texts through BPE, we get a array with length of 300B ? What's the number if we de-dup these tokens? (size of vocab?) 175B parameters means there are somewhat useful 175B floats in the pre-trained neural network?
- code_runner 4y agoI’ll do my best. Number of params is the number of weights. Basically the number of learnable variables. Number of tokens is how many tokens it saw during training. Vocab size is the number of distinct tokens. The relationship between params/tokens/compute power is something people have studied a good deal and how it affects model performance. https://arxiv.org/pdf/2203.15556.pdf https://arxiv.org/pdf/2203.15556.pdf
- lvwarren 4y agomake this change and it will run under Python 3.8 in google colab #name = name.removeprefix("model/") name = name[len('model/'):] in function: load_gpt2_params_from_tf_ckpt in the utils.py module
- deleted 4y ago[deleted]
- lvwarren 4y agoMake this change in utils.py: def load_gpt2_params_from_tf_ckpt(tf_ckpt_path, hparams): [...] #name = name.removeprefix("model/") name = name[len('model/'):] and you're cool example will run in Google Colab under Python 3.8 otherwise the 3.9 Jupyter patching is a headache.
- eric_hui 4y agofantastic article about GPT. Thank you for sharing
- sva_ 4y agoImpressive, but only forward pass.
- thwayunion 4y agoIt's an excellent learning tool :) Doing the backward pass in the same style would be a great tool for teaching.
- anigbrowl 4y agoI think the completeness and self-contained-ness more than offsets the limited scope. One of the problems in the ML field is rapidly multiplying logistical complexity, and I appreciate an example that is (somewhat) functional but simple enough to fit on a postcard and using very basic components.
- time_to_smile 4y agojust replace the numpy code with jax.numpy as you should have a fully differentiable model ready for training!
- pumanoir 4y agoFor someone not familiar with jax, if I do the suggested replacement. What'd be the little extra code to make it do the backward pass? Or is it all automatic and we literally would not need extra lines of code?
- time_to_smile 4y agoBackprop is just an implementation detail when doing automatic differentiation, basically setting up how you would apply the chain rule to your problem. JAX is able to differentiate arbitrary python code (so long as it uses JAX for the numeric stuff) automatically so the backprop is abstracted away. If you have the forward model written, to train it all you have to do with wrap it in whatever loss function you want, and the use JAX's `grad` with respect to the model parameters and you can use that to find the optimum using your favorite gradient optimization algorithm. This is why JAX is so awesome. Differentiable programming means you only have to think about problems in terms of the forward pass and then you can trivially get the derivative of that function without having to worry about the implementation details.
- adamnemecek 4y agoIt turns out that transformers have a learning mechanism similar to autodiff but better since it happens mostly within the single layers as opposed to over the whole graph. I wrote a paper on this recently https://arxiv.org/abs/2302.01834v1 https://arxiv.org/abs/2302.01834v1. The math is crazy.
- naasking 4y agoCan this all be done on the GPU so the CPU doesn't need to be involved to adjust the weights?
- LukeB42 4y agoCan you explain like I'm 5 why this matters distinctly from how transformers are normally trained with autodiff and what its possible applications are?
- adamnemecek 4y agoI’m talking about attention only transformers. Those don’t have an autodiff but still learn. The math is actually really cool.
- lostmsu 4y ago> attention only transformers Can you share any good link on the subject?
- adamnemecek 4y agohttps://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2021/framework/index.html
- lostmsu 4y agoMaybe I am missing something, but I don't see any learning without autodiff.
- master_yoda_1 4y ago[flagged]
- enlyth 4y agoThat's a bit overly harsh and missing the point of the article
- LukeB42 4y agoTerrible joke and COMPLETELY missing the point.
- kurisufag 4y agothe LoC count has nothing to do with the amount of processing required for training -- if anything they're probably inversely related. the title (correctly) describes a small LoC count, which makes no assertions about the other things you mentioned.
- master_yoda_1 4y agothe article DOES NOT describe GPT3 and this article title is misleading, and the author is lying.
- joelfried 4y agoAnger… fear… aggression. The dark side are they. Easily they flow, quick to join you in a fight. If once you start down the dark path, forever will it dominate your destiny, consume you it will, as it did Obi-Wan’s apprentice. -- Master Yoda, Return of the Jedi
- kurisufag 4y agogpt3 =/= gpt https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf https://cdn.openai.com/research-covers/language-unsupervised...
- simonw 4y agoWhat's the lie? The article title is "GPT in 60 lines of NumPy". I'd agree with you if it said "GPT3 in 60 lines of NumPy" - but it doesn't say that.
- eddsh1994 4y agoWhy do people in ML put imports inside function definitions?
- reallymental 4y agoScope-dependent imports. What if a package is just required for that particular function, and once that function is done, the imported package is no longer required?
- w0m 4y agoI do sometimes - just depends on the context and how often the function(xor library) is going to get called. Here - they put `import fire` only in the `if __name__ == "__main__":` - that seems reasonable to me as anyone pulling in the library from elsewhere doesn't need the pollution.
- theptip 4y agoDoes that import have side effects? Are we really worried about adding an entry to the imports dict if not? Or put differently, what cases do we actually get a negative effect from just importing at the top?
- apetresc 4y agoOh yeah, imports in Python are not just, like, extending a namespace like in many other languages. They, at runtime, go and run the module's __init__ and can have arbitrary side effects - an entire program can run (although usually shouldn't) just in the import. Imports of large modules often take entire seconds. It is absolutely worthwhile to avoid unnecessary imports if possible.
- theptip 4y agoI know they _can_ have side-effects, I’ve just never seen a case where it actually mattered, and I have used Python professionally for 10 years. So I’m curious if this is more common in ML libraries or something.
- terran57 4y agoFrom the article: "Of course, you need a sufficiently large model to be able to learn from all this data, which is why GPT-3 is 175 billion parameters and probably cost between $1m-10m in compute cost to train.[2]" So, perhaps better title would be "GPT in 60 Lines of Numpy (and $1m-$10m)"
- MuffinFlavored 4y agoWill one business model be for OpenAI to "license" out access to their trained model? How large is the model on disk(s) once it is trained?
- shagie 4y agoFrom various sources, the model itself is about 800 GB on disk.
- theptip 4y agoPerhaps I’m missing your point, but isn’t that what they do with their API right now? You pay for text completions, and can fine-tune their model with your data.
- veqq 4y agoBut you can't run the code on your own machine.
- est 4y ago> But you can't run the code on your own machine. iirc GPT-3 itself alone is some 500TB in size. You need a really, really big machine to run LLMs, the first L means Large.
- mattnewton 4y agoOf course, if they leaked the model weight’s and a local inference binary for it they would lose the ability to charge for it. Clones with the weights would crop up all over the place.
- master_yoda_1 4y ago[flagged]
- moyix 4y agoThe title doesn't say GPT3, it says GPT (unless it's been edited since you posted this?).
- barbazoo 4y agoSo much criticism in the comments. I appreciated the write-up and the code samples. For some people not in ML like myself it's hard to understand the concept behind GPT and this made it a little bit clearer.
- tysam_and 4y agoI think this is a factor of putting one's self out there. I've had this happen on ML projects I've put out too, though being hyper-engaged in trying to thoughtfully respond to all (or as many as possible of) the comments section for me has seemed to lower negativity a bit just because it brings up the 'person-in-the-room' effect up to an online audience...at least, so I think! :D I thought it was a great post and manky kudos to the author for putting themselves out like that! I really appreciated this and any work that does this kind of effort in onboarding people and giving people tools to understand something well really I think has some of the most long-term impact to the field. Lowering barriers to entry, making resources accessible to all, and decreasing experimentation cycle time I think are some of the most critical components to making any progress at all in the field beyond a basic pittance. Imagine if everyone had easy access to, knowledge about, and rapid experimentation results in things like quantum mechanics, large-algorithm testing, painting arts, musical arts, etc. It would drive things so much further forward at an individual and field-based level so quickly. <3 :)))) :D :D ;D :D :D :))))))))
- victor9000 4y agoPeople respond with negativity for a variety of reasons, and it often has more to do with the commenter than the content.
- ghotli 4y agoWow, what a nice succinct way to put that. Thanks
- freecodyx 4y agoSince most models require little code compared to big software projects, why not use c++ or any other compiled language directly. Python with it’s magic functions, shortcuts is just hiding too much complexity which can result in bug performance issues. Plus code is more hard to maintain
- stavros 4y agoWhich magic functions and shortcuts in the posted code do you feel might introduce bugs?
- freecodyx 4y agoIn general, the article is fine.
- aiisjustanif 4y agoI’m curious as well, what specific lines would cause an issue or abstracts too much.
- CaptainNegative 4y ago> Python with it’s magic functions, shortcuts is just hiding too much complexity One counterpoint would be that verbosity, especially in the heavy syntax style of languages such as C++, distracts the reader and helps bugs hide in plain sight. For a silly example, imagine trying to read and verify the correctness of an academic paper from its uncompiled LaTeX source.
- mhh__ 4y agoA lot of AI (not a huge amount but more than you'd think) people can't code in any sense that would get them a job at a normal software company, Python is easy and fast enough to last until the model is obsolete.
- lspears 4y agoFor those interested I would also check out Andrej Karpathy's YouTube video on building GPT from scratch: https://youtu.be/kCc8FmEb1nY https://youtu.be/kCc8FmEb1nY
- azath92 4y agoKarpathy has a bunch of great resources on this front! His minGPT writeup is excellent https://github.com/karpathy/minGPT https://github.com/karpathy/minGPT His more recent project nanoGPT which references this video is a much more capable, but still learning friendly, implementation.
- simonw 4y agoThis article is an absolutely fantastic introduction to GPT models - I think the clearest I've seen anywhere, at least for the first section that talks about generating text and sampling. Then it got to the training section, which starts "We train a GPT like any other neural network, using gradient descent with respect to some loss function". It's still good from that point on, but it's not as valuable as a beginner's introduction.
- ly3xqhl8g9 4y agoFor those curious about writing a "gradient descent with respect to some loss function" starting from an empty .py file (and a numpy import, sure), can't recommend enough Harrison "sentdex" Kinsley's videos/book Neural Networks from Scratch in Python [1]. [1] https://youtu.be/Wo5dMEP_BbI?list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3 https://youtu.be/Wo5dMEP_BbI?list=PLQVvvaa0QuDcjD5BAw2DxE6OF... https://nnfs.io https://nnfs.io
- artdigital 4y agoThe beginning of Andrew Ng’s machine learning course on coursera does that too, it touches on the math a bit and explains how to imagine gradient descent in 3d space Didn’t do the full course, but after the first few chapters I was able to write a very basic implementation in raw python (emphasizing here on “very basic”)
- aulin 4y agothere is so much material on deep learning basics these days that I think we can finally skip reintroducing gradient descent in every tutorial, can't we?
- Joker_vD 4y agoThe idea of "find in which direction function decreases most quickly and go that direction" is really deep, and its implementation via this cutting-edge mathematical concept of "gradient" also deserves a whole section as well.
- thomasfromcdnjs 4y agoThis reads really well, thank you very much.
- deleted 4y ago[deleted]
- voz_ 4y agoWonderfully written, I love the amount of detail put into the diagrams. Would love breakdowns like this for more stuff :)
- insane_dreamer 4y agonice and clear. a worthy contribution to the subject.
- jaykmody 4y agoHey ya'll author here! Thank you for all the nice and constructive comments! For clarity, this is ONLY the forward pass of the model. There's no training code, batching, kv cache for efficiency, GPU support, etc ... The goal here was to provide a simple yet complete technical introduction to the GPT as an educational tool. Tried to make the first two sections something any programmer can understand, but yeah, beyond that you're gonna need to know some deep learning. Btw, I tried to make the implementation as hackable as possible. For example, if you change the import from `import numpy as np` to `import jax.numpy as np`, the code becomes end-to-end differentiable: def lm_loss(params, inputs, n_head) -> float: x, y = inputs[:-1], inputs[1:] output = gpt(x, **params, n_head=n_head) loss = np.mean(-np.log(output[y])) return loss grads = jax.grad(lm_loss)(params, inputs, n_head) You can even support batching with `jax.vmap` (https://jax.readthedocs.io/en/latest/_autosummary/jax.vmap.html https://jax.readthedocs.io/en/latest/_autosummary/jax.vmap.h...): gpt2_batched = jax.vmap(gpt2, in_axes=0) gpt2_batched(batched_inputs) # [batch, seq_len] -> [batch, seq_len, vocab] Of course, with JAX comes in-built GPU and even TPU support! As far as training code and KV Cache for inference efficiency, I leave that as an exercise for the reader lol
- Teamteam16 4y agoTteam5049@gmail.com
- tysam_and 4y ago"hackable" and "simple yet complete technical introduction" Music to my ears, well done and don't worry too much about the negative comments! They'll come out for anything you do I think. I saw a tweet from someone the other day talking about how they massively increased their training speed by changing part of their architecture to have dimensions that were a factor of 64 rather than a prime-like kind of number. One of the comments below it? ~"Seems very architecture specific." lol. So don't sweat it! <3 Great work and thanks for putting yourself out there, super job! :D :D :D :D :)))))) <3 :D :D :fireworks:
- pavelstoev 4y agoWe do GPU-specific training and inference speedups, at CentML.
- deleted 4y ago[deleted]
- ultrasounder 4y agoI also learnt a ton from NLPDemystified-https://www.nlpdemystified.org https://www.nlpdemystified.org. In fact I used this resource first before attempting Andrej Karpathy's https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html. I find Nitin's voice soothing and am able to focus more. I also found the pacing good and the course introduces a lots of concepts a beginner level and also points to appropriate resources along the way(spacy for instance). Overall an exciting time to be a total beginner looking to grok NLP concepts.
- eslaught 4y agoI know this probably isn't intended for performance, but it would be fun to run this in cuNumeric [1] and see how it scales. [1]: https://github.com/nv-legate/cunumeric https://github.com/nv-legate/cunumeric
- quantupus 4y ago[flagged]
- teaearlgraycold 4y agoReminds me the scene from Westworld where they explain their failed prototypes of the human mind with millions of lines of code. The version that finally worked was only a few dozen.
- est 4y agoIf I maintain an open source project, could I build a doc page using a small GPT allowing users to query FAQ and common methods using natural language?
- hummus_bae 4y agoYou can build anything using numpy. You can build a supercomputer out of duct tape if you want to. Spinning up a db, serving an API, doing natural language processing, ... whatever you want. That said, there are niche solutions that do these things well and that can save you a lot of work. There are also frameworks (such as Pyntango/Jupyter/Nbspectrum) that lets you spin up a simple API quickly.
- qwerty456127 4y agoHow powerful/heavy it is? Some time ago here was a post about implementing a GPT on a very constrained computer (under a gigabyte of RAM, some old CPU, no GPU (?)) as opposed to an ordinary kind of GPT requiring terabytes of RAM. I immediately thought it would be nice to do something in the middle: taking full advantage of a reasonably modern multicore CPU with AVX support, a humble yet again reasonably modern OpenCL-capable GPU and some 32 Gigabytes of RAM.