5 ms·

In case you're wondering what it takes to run it, the answer is 8x H200 141GB [1] which costs $250k [2]. 1. https://github.com/MiniMax-AI/MiniMax-M1/issues/2#i

In case you're wondering what it takes to run it, the answer is 8x H200 141GB [1] which costs $250k [2]. 1. https://github.com/MiniMax-AI/MiniMax-M1/issues/2#issuecomment-2982368797 https://github.com/MiniMax-AI/MiniMax-M1/issues/2#issuecomme... 2. https://www.ebay.com/itm/335830302628 https://www.ebay.com/itm/335830302628

incomingpain 1y ago
That's full quantization. If you run Q4 or Q8 you can run this on <$10,000 equipment.
- cma 1y ago
  And if you add in heavy sparsification it should fit and run on a raspberry pi.
  - rvz 1y ago
    So in around 6 months, we will see that the person who bought this H200 in the listing just got scammed for $250k and will realize that you just needed specific quantizations to the model and a few optimizations to run locally. Unless they want to train their own model, buying this for inference for $250k is unnecessary and still isn't enough for a full production deployment.
  - vFunct 1y ago
    It's already sparsified from the 150T parameter model..
    - yorwba 1y ago
      It took me several hours to realize that 150 trillion parameters is a reference to the number of synapses in a human brain.
- deadbabe 1y ago
  No point in running anything but full quantization.
  - deleted 1y ago
    [deleted]
  - esafak 1y ago
    Quantization doesn't work? Really?
- tgtweak 1y ago
  My experience with heavily quantized models is they do better than a similar sized unquantized model but don't really perform anywhere near the pre-quantized model.
  - incomingpain 1y ago
    >My experience with heavily quantized models is they do better than a similar sized unquantized model but don't really perform anywhere near the pre-quantized model. People have tested it. Q8 has essentially no drop in quality, Q4 is measurable but still not realistically a problem. If this impacts you, just pay for the commercial saas option.
    - haolez 1y ago
      This assumes that the benchmarks are representative of real usage scenarios. I'm not saying that there is bad faith, but that benchmarking is really hard in the context of LLMs.
      - incomingpain 1y ago
        >This assumes that the benchmarks are representative of real usage scenarios. I'm not saying that there is bad faith, but that benchmarking is really hard in the context of LLMs. It's a fair point, but the conclusion is 'i dont know' I could assume that it gets better because it'll keep to simpler code.
GTP 1y ago
How many parameters does this model have?
- 7moritz7 1y ago
  456 bn, about 46 bn active at a time (it's moe)
kridsdale1 1y ago
Can’t you run it on a Mac Studio with 512GB? That’s about $8,500.
- hu3 1y ago
  It's also 1/20 of the speed. So not very useable.

incomingpain 1y ago

That's full quantization. If you run Q4 or Q8 you can run this on <$10,000 equipment.

cma 1y ago
And if you add in heavy sparsification it should fit and run on a raspberry pi.
- rvz 1y ago
  So in around 6 months, we will see that the person who bought this H200 in the listing just got scammed for $250k and will realize that you just needed specific quantizations to the model and a few optimizations to run locally. Unless they want to train their own model, buying this for inference for $250k is unnecessary and still isn't enough for a full production deployment.
- vFunct 1y ago
  It's already sparsified from the 150T parameter model..
  - yorwba 1y ago
    It took me several hours to realize that 150 trillion parameters is a reference to the number of synapses in a human brain.
deadbabe 1y ago
No point in running anything but full quantization.
- deleted 1y ago
  [deleted]
- esafak 1y ago
  Quantization doesn't work? Really?
tgtweak 1y ago
My experience with heavily quantized models is they do better than a similar sized unquantized model but don't really perform anywhere near the pre-quantized model.
- incomingpain 1y ago
  >My experience with heavily quantized models is they do better than a similar sized unquantized model but don't really perform anywhere near the pre-quantized model. People have tested it. Q8 has essentially no drop in quality, Q4 is measurable but still not realistically a problem. If this impacts you, just pay for the commercial saas option.
  - haolez 1y ago
    This assumes that the benchmarks are representative of real usage scenarios. I'm not saying that there is bad faith, but that benchmarking is really hard in the context of LLMs.
    - incomingpain 1y ago
      >This assumes that the benchmarks are representative of real usage scenarios. I'm not saying that there is bad faith, but that benchmarking is really hard in the context of LLMs. It's a fair point, but the conclusion is 'i dont know' I could assume that it gets better because it'll keep to simpler code.

GTP 1y ago

How many parameters does this model have?

7moritz7 1y ago
456 bn, about 46 bn active at a time (it's moe)

kridsdale1 1y ago

Can’t you run it on a Mac Studio with 512GB? That’s about $8,500.

hu3 1y ago
It's also 1/20 of the speed. So not very useable.