3 ms·
Have you tried it? It would be slow for sure, but the main limitation AIUI would actually be storing the context in RAM - models like Kimi and GLM have high dem
by zozbot234 5mo ago
Have you tried it? It would be slow for sure, but the main limitation AIUI would actually be storing the context in RAM - models like Kimi and GLM have high demands there which limit your ability to get meaningful aggregate throughput via large batches.
- Tepix 5mo agoNo need to try really. 1100b weights with 256GB RAM that‘s less than 1.8 bits per weight if you want a little bit of context. How is that supposed to give good results?