3 ms·
Kimi K2.6 does not run well on 256GB.
by Tepix 5mo ago
Kimi K2.6 does not run well on 256GB.
- girvo 5mo agoTrue, I might be thinking of some of the communities four-Spark clusters for it; it’s already int4 right?
- Tepix 5mo agoYeah, the default quants are 595GB. Even four Sparks would require a quant lower than 4bit
- zozbot234 5mo agoHave you tried it? It would be slow for sure, but the main limitation AIUI would actually be storing the context in RAM - models like Kimi and GLM have high demands there which limit your ability to get meaningful aggregate throughput via large batches.
- Tepix 5mo agoNo need to try really. 1100b weights with 256GB RAM that‘s less than 1.8 bits per weight if you want a little bit of context. How is that supposed to give good results?