5 ms·
Huggingface is currently replicating it. Replications of small models indicate that they don't lie any significant amount. The architecture is cheap to train.
by nabla9 2y ago
Huggingface is currently replicating it.
Replications of small models indicate that they don't lie any significant amount. The architecture is cheap to train.
Berkeley Researchers Replicate DeepSeek R1's Core Tech for Just $30: A Small Model RL Revolution https://xyzlabs.substack.com/p/berkeley-researchers-replicate-deepseek https://xyzlabs.substack.com/p/berkeley-researchers-replicat...
- benterix 2y agoWhoah, that's incredible! I remember a year ago I was hoping that in a decade from now it would be great to run GPT4-class models on my own hardware. The reality seems to be far more exciting.
- moritzwarhier 2y agoI first sneered at the idea of LLM generated LLM training sets, but is this what might be driving the big efficiency leap? Asking as someone who honestly only superficially followed the developments since the end of 2023 or so
- bufferoverflow 2y agoYou call R1 a small model? It's a 671-billion parameter model.
- elorant 2y agoThere are multiple variations of the model starting from 1.5B parameters.
- bufferoverflow 2y agoThose are distillations of the model.
- rsanek 2y agohave you used those? in my experience even the 70B distillation is far worse than what you can expect from o1 / the R1 available on the web
- elorant 2y agoNo, I haven't. I've used Perplexity's R1 but I don't know how many parameters it has. It's quite good, although too slow.