4 ms·

Yeah, they're the good guys. I suspect the open source work is mostly advertisements for them to sell consulting and services to enterprises. Otherwise, the wor

Yeah, they're the good guys. I suspect the open source work is mostly advertisements for them to sell consulting and services to enterprises. Otherwise, the work they do doesn't make sense to offer for free.

arcanemachiner 8mo ago
I hope that is exactly what is happening. It benefits them, and it benefits us.
danielhanchen 8mo ago
Haha for now our primary goal is to expand the market for local AI and educate people on how to do RL, fine-tuning and running quants :)
- WanderPanda 8mo ago
  Amazing work and people should really appreciate that the opportunity costs of your work are immense (given the hype). On another note: I'm a bit paranoid about quantization. I know people are not good at discerning model quality at these levels of "intelligence" anymore, I don't think a vibe check really catches the nuances. How hard would it be to systematically evaluate the different quantizations? E.g. on the Aider benchmark that you used in the past? I was recently trying Qwen 3 Coder Next and there are benchmark numbers in your article but they seem to be for the official checkpoint, not the quantized ones. But it is not even really clear (and chatbots confuse them for benchmarks of the quantized versions btw.) I think systematic/automated benchmarks would really bring the whole effort to the next level. Basically something like the bar chart from the Dynamic Quantization 2.0 article but always updated with all kinds of recent models.
  - Zetaphor 8mo ago
    This would be amazing
    - danielhanchen 8mo ago
      Working on it! :)
  - jychang 8mo ago
    > How hard would it be to systematically evaluate the different quantizations? E.g. on the Aider benchmark that you used in the past? Very hard. $$$ The benchmarks are not cheap to run. It'll cost a lot to run them for each quant of each model.
    - danielhanchen 8mo ago
      Yes sadly very expensive :( Maybe a select few quants could happen - we're still figuring out what is the most economical and most efficient way to benchmark!
      - illusive4080 8mo ago
        Roughly how much does it cost to run one of the popular benchmarks? Are we talking $1,000, $10,000, or $100k?
        danielhanchen 8mo ago
        Oh it's more time that's the issue - each benchmark takes 1-3 hours ish to run on 8 GPUs, so running on all quants per model release can be quite painful. Assume AWS spot say $20/hr B200 for 8 GPUs, then $20 ish per quant, so assuming benchmark is on BF16, 8bit, 6, 5, 4, 3, 2 bits then 7 ish tests so $140 per model ish to $420 ish/hr. Time wise 7 hours to 1 day ish. We could run them after a model release which might work as well. This is also on 1 benchmark.
  - danielhanchen 8mo ago
    Thanks! Yes we actually did think about that - it can get quite expensive sadly - perplexity benchmarks over short context lengths with small datasets are doable, but it's not an accurate measure sadly. We're actually investigating currently what would be the best efficient course of action on evaluating quants - will keep you posted!

arcanemachiner 8mo ago

I hope that is exactly what is happening. It benefits them, and it benefits us.

danielhanchen 8mo ago

Haha for now our primary goal is to expand the market for local AI and educate people on how to do RL, fine-tuning and running quants :)

WanderPanda 8mo ago
Amazing work and people should really appreciate that the opportunity costs of your work are immense (given the hype). On another note: I'm a bit paranoid about quantization. I know people are not good at discerning model quality at these levels of "intelligence" anymore, I don't think a vibe check really catches the nuances. How hard would it be to systematically evaluate the different quantizations? E.g. on the Aider benchmark that you used in the past? I was recently trying Qwen 3 Coder Next and there are benchmark numbers in your article but they seem to be for the official checkpoint, not the quantized ones. But it is not even really clear (and chatbots confuse them for benchmarks of the quantized versions btw.) I think systematic/automated benchmarks would really bring the whole effort to the next level. Basically something like the bar chart from the Dynamic Quantization 2.0 article but always updated with all kinds of recent models.
- Zetaphor 8mo ago
  This would be amazing
  - danielhanchen 8mo ago
    Working on it! :)
- jychang 8mo ago
  > How hard would it be to systematically evaluate the different quantizations? E.g. on the Aider benchmark that you used in the past? Very hard. $$$ The benchmarks are not cheap to run. It'll cost a lot to run them for each quant of each model.
  - danielhanchen 8mo ago
    Yes sadly very expensive :( Maybe a select few quants could happen - we're still figuring out what is the most economical and most efficient way to benchmark!
    - illusive4080 8mo ago
      Roughly how much does it cost to run one of the popular benchmarks? Are we talking $1,000, $10,000, or $100k?
      - danielhanchen 8mo ago
        Oh it's more time that's the issue - each benchmark takes 1-3 hours ish to run on 8 GPUs, so running on all quants per model release can be quite painful. Assume AWS spot say $20/hr B200 for 8 GPUs, then $20 ish per quant, so assuming benchmark is on BF16, 8bit, 6, 5, 4, 3, 2 bits then 7 ish tests so $140 per model ish to $420 ish/hr. Time wise 7 hours to 1 day ish. We could run them after a model release which might work as well. This is also on 1 benchmark.
- danielhanchen 8mo ago
  Thanks! Yes we actually did think about that - it can get quite expensive sadly - perplexity benchmarks over short context lengths with small datasets are doable, but it's not an accurate measure sadly. We're actually investigating currently what would be the best efficient course of action on evaluating quants - will keep you posted!