4 ms·
I think many people don't appreciate just how simple state-of-the-art neural net techniques are. It's great to have an illustration of just how little code you
by modeless 6y ago
I think many people don't appreciate just how simple state-of-the-art neural net techniques are. It's great to have an illustration of just how little code you need to get the results that have amazed people.
You could say that it relies on PyTorch which is a lot of code, but most of the complexity comes from the need to do GPU/TPU/SIMD acceleration. A truly from-scratch CPU-only implementation in C would still not be a large amount of code.
- typon 6y agoIt's kind of like saying Redis is just an dict. The optimizations in the entire software/hardware stack is what makes these innovations possible. A C implementation would not work in reality for large models.
- modeless 6y agoHere is a C implementation of GPT-2. https://bellard.org/nncp/gpt2tc.html https://bellard.org/nncp/gpt2tc.html I don't disagree that hardware acceleration is key in enabling these models, but I still find it interesting how simple the core techniques are.
- read_if_gay_ 6y agoIs there anything Bellard hasn’t done?
- huac 6y agoThis is very cool, thanks for sharing! From the readme (https://bellard.org/nncp/readme-gpt2tc.txt https://bellard.org/nncp/readme-gpt2tc.txt), the program benchmarks very comparably to CMIX, which is the top algorithm on the Large Text Compression Benchmark (http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/text.html). I'm guessing that any GPT implementation would be ineligible for the benchmark because of its file size but impressive nonetheless.
- svnpenn 6y agoHm, I notice the source code is missing? I did I overlook it
- xkapastel 6y agoThat's not really a C implementation of GPT-2 since it cannot be used to do the thing everyone cares about: self-supervised learning from text. In fact, it doesn't even use the weights in the same way GPT-2 does, so it's not clear how close it is to GPT-2's inference mode. The source isn't even on the page.
- dheera 6y ago> A C implementation You mean CPU implementation. C language isn't the culprit here, and low-level parts of ML frameworks are normally written in C or C++ as well and compiled with nvcc or equivalent.
- 2bitencryption 6y agoAs a developer with interested in neural-net based ML, my eyes start to glaze over a bit when I see so much crazy-looking numpy operations even just to get a trivial representation of data. I guess it's just something I have to get used to, but I wish there was an interface to do the same nd-array based logic that was designed more for developers like me rather than data scientists who perform surgery on 50d arrays all day long.
- modeless 6y agoI agree; check out NamedTensor for an attempt to fix this: http://nlp.seas.harvard.edu/NamedTensor http://nlp.seas.harvard.edu/NamedTensor
- dheera 6y agoHave you checked out numba? You can use regular Python for loops and numpy calls and it will compile it. Instead of numpy vectorization mess.
- Der_Einzige 6y agoGood luck with anything of non trivial types. Numba is awesome but also quite finicky.
- newen 6y agoSorry if this sounds condescending but just learn linear algebra and numpy. Linear algebra really is a minimum requirement if you want to get anywhere near ML. College freshmen can learn linear algebra and numpy and so can you. You'll only get even more lost and frustrated if you try to go into it thinking in terms of for loops.
- vmladenov 6y agoDon’t look at code, because it bakes in so many derived values and simplifications that it’s hard to recover the original ideas. If you can find the original math, and walk through the derivatives (seeing zeros crop up and get simplified away), it starts to make a lot more sense.
- p1esk 6y agoIt's simple in the same way Conway's Game of Life is simple.
- antupis 6y agoKarpathy is also very good at making complex architectures to very simple.
- fho 6y agoIt really depends on what level allow yourself to start. If you hand derive all equations in your program you are going to have a hard time ... but automatic differentiation (ad) and existing linear algebra libraries (for gradient descent/ADAM) make the the job pretty easy. I recently dabbled in automatic (song-)mixing ... reimplemented the code of some open audio plugins in a way that they were auto-derivable and "trained" the whole thing on some multitracks with reference mixes. Sadly that didn't resulted in a good mix ... but in theory it should work. What's your "simple deep learning" story?