3 ms·
It's kind of like saying Redis is just an dict. The optimizations in the entire software/hardware stack is what makes these innovations possible. A C implementa
by typon 6y ago
It's kind of like saying Redis is just an dict. The optimizations in the entire software/hardware stack is what makes these innovations possible. A C implementation would not work in reality for large models.
- modeless 6y agoHere is a C implementation of GPT-2. https://bellard.org/nncp/gpt2tc.html https://bellard.org/nncp/gpt2tc.html I don't disagree that hardware acceleration is key in enabling these models, but I still find it interesting how simple the core techniques are.
- read_if_gay_ 6y agoIs there anything Bellard hasn’t done?
- huac 6y agoThis is very cool, thanks for sharing! From the readme (https://bellard.org/nncp/readme-gpt2tc.txt https://bellard.org/nncp/readme-gpt2tc.txt), the program benchmarks very comparably to CMIX, which is the top algorithm on the Large Text Compression Benchmark (http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/text.html). I'm guessing that any GPT implementation would be ineligible for the benchmark because of its file size but impressive nonetheless.
- svnpenn 6y agoHm, I notice the source code is missing? I did I overlook it
- xkapastel 6y agoThat's not really a C implementation of GPT-2 since it cannot be used to do the thing everyone cares about: self-supervised learning from text. In fact, it doesn't even use the weights in the same way GPT-2 does, so it's not clear how close it is to GPT-2's inference mode. The source isn't even on the page.
- dheera 6y ago> A C implementation You mean CPU implementation. C language isn't the culprit here, and low-level parts of ML frameworks are normally written in C or C++ as well and compiled with nvcc or equivalent.