5 ms·

An interesting idea, but there's no discussion about benchmarks in the article, cpu or other hardware requirements beyond 230 GB RAM for the cluster. It seems

An interesting idea, but there's no discussion about benchmarks in the article, cpu or other hardware requirements beyond 230 GB RAM for the cluster. It seems impractical that a home would have 4 machines with 64 gb ram that would be dedicated to a distributed system. Max, core count, 16 cores from Consumer AMD CPUs? From a cost perspective build a system with 256gb ram and AMD Epic CPU? The only thing I can think of that pushes a need for a distributed system is multi-GPU across multiple systems.

RealStickman_ 2y ago
If you just want to play with this a bit, around 250$/month should give you enough metal when renting from cheap VDS/dedicated server providers.
- malux85 2y ago
  Surely not with GPUs? Could you direct me to them please?
  - RealStickman_ 2y ago
    This project only supports CPU inference at the moment, so no, no GPUs. I just estimated what some 256GB or RAM might cost monthly on the low end.
- wkat4242 2y ago
  250$ is a lot of money to me though. I spent €290 on my 16GB GPU for my AI server but that was once off and I really had to think about it. I'd love to see a llama model that fits now economically inside 16GB. The 8b is a bit too small when quantised even to 8 bits. A 16-20b model would be perfect. But I think for 400b models to be viable, the hardware pricing really needs to catch up.
  - RealStickman_ 2y ago
    > 250$ is a lot of money to me though. I spent €290 on my 16GB GPU for my AI server but that was once off and I really had to think about it. Agreed, it's a lot of money and definitely more than I'd be willing to spend. However, compared to running such 400b models on a GPU cluster it's extremely cheap (and much slower)
adrian_b 2y ago
Unless you buy some obsolete Epyc CPU with a small number of cores, which would be slower than even one up-to-date Ryzen 9 9950X (which can beat even 32-core older Epycs in compute-limited tasks that use vector instructions), 4 Ryzen boxes with 16 cores & 64 GB ECC memory each should be significantly cheaper (probably not exceeding $6000) and also faster than one box with an Epyc CPU and 256 GB, where the CPU alone could be more expensive than the complete Ryzen systems, if bought at retail prices. I actually happen to have in my home a cluster of five eATX cases, which stay on two adjacent IKEA metallic tables and which are connected in a ring of direct 10 Gb/s Ethernet links (i.e. they all have dual-port NICs). Therefore it is not really impractical, even if such a configuration may be infrequent. In the distant past, I was using dual-socket Xeon or Opteron motherboards in those cases, because such MBs and CPUs were much cheaper than today. Then, by the time of Zen 1, when the Epyc CPUs were still very cheap, the upgrades have been to single-socket Epyc MBs. More recently, Epyc became much more expensive and Ryzen CPUs have much better performance per dollar, so the latest upgrades have been to Ryzen MBs. I look forward to the launch of Ryzen 9 9950X, which should double the throughput of the vector operations, 5 years after Ryzen 9 3950X (Zen 2) has done the same in 2019. As my PC, I use an Intel NUC. Whenever I need to execute something for which that would be too weak, I launch it on one or more of the servers, by using Wake-on-LAN and shutting them down after the task is completed. In this way, my average electrical power consumption is much less than if I used a beefy desktop, while the peak performance is much higher.
- imtringued 2y ago
  > 4 Ryzen boxes with 16 cores & 64 GB ECC memory each should be significantly cheaper (probably not exceeding $6000) Yeah, but then the question arises: why? Assuming 4x $700 for a mini PC setup, you still only get like 360GB/s memory bandwidth. It is a lot of money with nothing to show for it, really. Your model will run at 2 tokens per second. A GPU setup within the same budget could run a 70B model at high speeds. This is why I am kinda bullish on the XDNA NPU from AMD. It represents the low end. People who run small models below 16GB in size. Since the hardware will be available on almost every AMD laptop, it will be much easier to actually utilize in say a video game. As of today, it is kind of annoying when an LLM based game asks you for an API key. The out of the box experience isn't that good.
duffyjp 2y ago
There's a bit of info on the project page. It's CPU only, which is why GPU and VRAM is never mentioned: https://github.com/b4rtaz/distributed-llama https://github.com/b4rtaz/distributed-llama I'm using an old x99 board for my desktop currently. If I swap out the i7 with a Xeon it can take up to 512GB of ram. That would be pricey, but I could do 256 and the Xeon for under $300 total. Still a lot for a toy, and I'm sure it would be super slow...

RealStickman_ 2y ago

If you just want to play with this a bit, around 250$/month should give you enough metal when renting from cheap VDS/dedicated server providers.

malux85 2y ago
Surely not with GPUs? Could you direct me to them please?
- RealStickman_ 2y ago
  This project only supports CPU inference at the moment, so no, no GPUs. I just estimated what some 256GB or RAM might cost monthly on the low end.
wkat4242 2y ago
250$ is a lot of money to me though. I spent €290 on my 16GB GPU for my AI server but that was once off and I really had to think about it. I'd love to see a llama model that fits now economically inside 16GB. The 8b is a bit too small when quantised even to 8 bits. A 16-20b model would be perfect. But I think for 400b models to be viable, the hardware pricing really needs to catch up.
- RealStickman_ 2y ago
  > 250$ is a lot of money to me though. I spent €290 on my 16GB GPU for my AI server but that was once off and I really had to think about it. Agreed, it's a lot of money and definitely more than I'd be willing to spend. However, compared to running such 400b models on a GPU cluster it's extremely cheap (and much slower)

adrian_b 2y ago

Unless you buy some obsolete Epyc CPU with a small number of cores, which would be slower than even one up-to-date Ryzen 9 9950X (which can beat even 32-core older Epycs in compute-limited tasks that use vector instructions), 4 Ryzen boxes with 16 cores & 64 GB ECC memory each should be significantly cheaper (probably not exceeding $6000) and also faster than one box with an Epyc CPU and 256 GB, where the CPU alone could be more expensive than the complete Ryzen systems, if bought at retail prices. I actually happen to have in my home a cluster of five eATX cases, which stay on two adjacent IKEA metallic tables and which are connected in a ring of direct 10 Gb/s Ethernet links (i.e. they all have dual-port NICs). Therefore it is not really impractical, even if such a configuration may be infrequent. In the distant past, I was using dual-socket Xeon or Opteron motherboards in those cases, because such MBs and CPUs were much cheaper than today. Then, by the time of Zen 1, when the Epyc CPUs were still very cheap, the upgrades have been to single-socket Epyc MBs. More recently, Epyc became much more expensive and Ryzen CPUs have much better performance per dollar, so the latest upgrades have been to Ryzen MBs. I look forward to the launch of Ryzen 9 9950X, which should double the throughput of the vector operations, 5 years after Ryzen 9 3950X (Zen 2) has done the same in 2019. As my PC, I use an Intel NUC. Whenever I need to execute something for which that would be too weak, I launch it on one or more of the servers, by using Wake-on-LAN and shutting them down after the task is completed. In this way, my average electrical power consumption is much less than if I used a beefy desktop, while the peak performance is much higher.

imtringued 2y ago
> 4 Ryzen boxes with 16 cores & 64 GB ECC memory each should be significantly cheaper (probably not exceeding $6000) Yeah, but then the question arises: why? Assuming 4x $700 for a mini PC setup, you still only get like 360GB/s memory bandwidth. It is a lot of money with nothing to show for it, really. Your model will run at 2 tokens per second. A GPU setup within the same budget could run a 70B model at high speeds. This is why I am kinda bullish on the XDNA NPU from AMD. It represents the low end. People who run small models below 16GB in size. Since the hardware will be available on almost every AMD laptop, it will be much easier to actually utilize in say a video game. As of today, it is kind of annoying when an LLM based game asks you for an API key. The out of the box experience isn't that good.

duffyjp 2y ago

There's a bit of info on the project page. It's CPU only, which is why GPU and VRAM is never mentioned: https://github.com/b4rtaz/distributed-llama https://github.com/b4rtaz/distributed-llama I'm using an old x99 board for my desktop currently. If I swap out the i7 with a Xeon it can take up to 512GB of ram. That would be pricey, but I could do 256 and the Xeon for under $300 total. Still a lot for a toy, and I'm sure it would be super slow...