5 ms·

I have a used workstation I got for $2k (with 768GB of RAM) - using the Q4 model, I can get about 1.5 tokens/sec and use very large contexts. It's pretty awesom

I have a used workstation I got for $2k (with 768GB of RAM) - using the Q4 model, I can get about 1.5 tokens/sec and use very large contexts. It's pretty awesome to be able to run it at home.

MysticFear 2y ago
Would love to know more info & specs of your workstation.
- mechagodzilla 2y ago
  It's an HP Z8 G4 (dual-socket 18-core, 3 GHz Xeons, 24x32GB of DDR4-2666, and then a crappy GPU, 8TB HDD, 1TB SSD). It can accommodate 3 dual-slot GPUs, but I was mostly interested in playing with frontier models where holding all the weights in VRAM requires a ~$500k machine. It can run the full Deepseek R1, Llama3-405B, etc, usually around 1-2 tokens/sec.
nomel 2y ago
For me, where electricity is $0.45/kWh, assuming 1kW consumption, it would be around $80 USD/million!
- CyberDildonics 2y ago
  I think you might have to show your math on that one.
  - nomel 2y ago
    They said 1.5 tokens/second. 1 mil tokens is 667k seconds is 185 hours per million tokens. 1kW * 185hr * $0.45/kWh = $80 per million tokens. Again, assuming 1kW, which may be high (or low). The cost of the physical computation is electricity cost.
    - CyberDildonics 2y ago
      They said it has a crappy GPU, the whole computer probably only uses 200 - 250 watts.
      - nomel 2y ago
        No way. 768GB of ram will have significant power draw. DDR4 (which this probably is) is something like 3W/8GB. That's > 250W alone. So, say 500W. That's, for me in my expensive electricity city, $40/million tokens, with the pretty severe rate limit of 5600 tokens/hours. If you're in Texas, that would be closer to $10/million tokens! Now you're at the same price as GPT-4o.
        menaerus 2y ago
        But you can run and experiment with any model of your liking. And your data does not leave your desktop environment. You can build services. I don't think anybody doing this is doing it to save $20 a month.
        nomel 2y ago
        Yes. I was only making a monetary comparison. Related, you can get a whole lot of cloud computing for $2k, for those same experiments, on much faster hardware. But yes, the data stays local. And, it's fun. This comment chain is pretty funny.

I have a used workstation I got for $2k (with 768GB of RAM) - using the Q4 model, I can get about 1.5 tokens/sec and use very large contexts. It's pretty awesome to be able to run it at home.

MysticFear 2y ago

Would love to know more info & specs of your workstation.

mechagodzilla 2y ago
It's an HP Z8 G4 (dual-socket 18-core, 3 GHz Xeons, 24x32GB of DDR4-2666, and then a crappy GPU, 8TB HDD, 1TB SSD). It can accommodate 3 dual-slot GPUs, but I was mostly interested in playing with frontier models where holding all the weights in VRAM requires a ~$500k machine. It can run the full Deepseek R1, Llama3-405B, etc, usually around 1-2 tokens/sec.

nomel 2y ago

For me, where electricity is $0.45/kWh, assuming 1kW consumption, it would be around $80 USD/million!

CyberDildonics 2y ago
I think you might have to show your math on that one.
- nomel 2y ago
  They said 1.5 tokens/second. 1 mil tokens is 667k seconds is 185 hours per million tokens. 1kW * 185hr * $0.45/kWh = $80 per million tokens. Again, assuming 1kW, which may be high (or low). The cost of the physical computation is electricity cost.
  - CyberDildonics 2y ago
    They said it has a crappy GPU, the whole computer probably only uses 200 - 250 watts.
    - nomel 2y ago
      No way. 768GB of ram will have significant power draw. DDR4 (which this probably is) is something like 3W/8GB. That's > 250W alone. So, say 500W. That's, for me in my expensive electricity city, $40/million tokens, with the pretty severe rate limit of 5600 tokens/hours. If you're in Texas, that would be closer to $10/million tokens! Now you're at the same price as GPT-4o.
      - menaerus 2y ago
        But you can run and experiment with any model of your liking. And your data does not leave your desktop environment. You can build services. I don't think anybody doing this is doing it to save $20 a month.
        nomel 2y ago
        Yes. I was only making a monetary comparison. Related, you can get a whole lot of cloud computing for $2k, for those same experiments, on much faster hardware. But yes, the data stays local. And, it's fun. This comment chain is pretty funny.