3 ms·

I recently set up the 26B A4B model up on vLLM on an RTX3090 (4-bit) after a hiatus from local models. Just completely blown away by the speed and quality you c

I recently set up the 26B A4B model up on vLLM on an RTX3090 (4-bit) after a hiatus from local models. Just completely blown away by the speed and quality you can get now for sub-$1k investment. I tried first with Qwen but it was unstable and had ridiculously long thinning traces!

jszymborski 6mo ago
The A4B model is blazing fast and the model is super good at general inquiries. Notably worse than Qwen 3.6 for coding tasks but that says more about the Qwen model.
- maille 6mo ago
  Bad at coding, but would it be good at code review?
  - avadodin 6mo ago
    Good compared to what? Nothing? Probably better.
2ndorderthought 6mo ago
Some of the early quants for qwen3.6 were broken. It's still finicky but with a little hand holding it's crazy. Local models are the future it's awesome
aimxhaisse 6mo ago
It even fits on a 3060 with turboquant / Q4 at decent speed (40T/s) for ~200$ (:
moffkalast 6mo ago
The 31B is surprisingly fast too, for a dense model. Runs tg at least twice as fast as it ought to on my machine when compared to other 30B, probably due to the hybrid attention I guess. Ingestion is somewhat slower though.