5 ms·
Looking at the R1 paper, if the benchmark are correct, even the 1.5b and 7b models are outperforming Claude 3.5 Sonnet, and you can run these models on a 8-16GB
by huydotnet 2y ago
Looking at the R1 paper, if the benchmark are correct, even the 1.5b and 7b models are outperforming Claude 3.5 Sonnet, and you can run these models on a 8-16GB macbook, that's insane...
- csomar 2y agoI think because they are trained on Claude/O1, they tend to have comparable performance. The small models quickly fails on complex reasoning. The larger the models, the better the reasoning is. I wonder, however, if you can hit a sweet spot with 100gb of ram. That's enough for most professional to be able to run it on an M4 laptop and will be a death sentence for OpenAI and Anthropic.
- elashri 2y agoAt the price of $5,000 before taxes. There would be better and most cost effective options to run models that will require that much memory.
- csomar 2y agoIt is a laptop. The memory is also shared which means if you are looking for a non-gaming workload, you can use it. If you have laptop equivalents in the same memory range, feel free to share.
- rfoo 2y agoI have laptop equivalents in the same memory range and is at least $2,500 cheaper. Unfortunately, it does not have "unified memory", a somewhat "powerful GPU", and of course no local LLM hype behind it. Instead, I've decided to purchase a laptop with 128GB RAM with $2,500 and then another $2,160 for 10 years Claude subscription, so I can actually use my 128GB RAM at the same time as using a LLM.
- kridsdale1 2y agoOk, but that means you’re not getting full privacy. It’s a trade off.
- csomar 2y agoThat's not the same thing. Also, can you share this 128GB $2500 laptop?
- kergonath 2y agoI see this comment all the time. But realistically if you want more than 1 token/s you’re going to need geforces, and that would cost quite a lot as well, for 100 GB.
- nenaoki 2y agohttps://nvidianews.nvidia.com/news/nvidia-puts-grace-blackwell-on-every-desk-and-at-every-ai-developers-fingertips https://nvidianews.nvidia.com/news/nvidia-puts-grace-blackwe... GB10, or DIGITS, is $3,000 for 1 PFLOP (@4-bit) and 128GB unified memory. Storage configurable up to 4TB. Can be paired to run 405B (4-bit), probably not very fast though (memory bandwidth is slower than a typical GPU's, and is the main bottleneck for LLM inference).
- justincormack 2y agoNot shipping until May or so.
- kergonath 2y agoThat’s not something I can get, so it’s not really relevant. There is always a better device around the corner.
- kamikazeturtles 2y ago> I think because they are trained on Claude/O1, they tend to have comparable performance. Why does having comparable performance indicate having been trained on a preexisting model's output? I read a similar claim in relation to another model in the past, so I'm just curious how this works technically.
- wordpad25 2y agobecause the valley is burning money and GPUs training these and somebody else comes out with another model for a tiny fraction of cost it's an easy assumption to make it was trained on synthetic data
- byefruit 2y agoDo you have any evidence for this accusation? O1's reasoning traces aren't even shown, are you suggesting they've somehow exfiltrated them?