5 ms·
It's worse at SQL generation than llama3 according to their own post. https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake
by mritchie712 2y ago
It's worse at SQL generation than llama3 according to their own post.
https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/#:~:text=on%20enterprise%20intelligence.-,Table%203,-.%20Full%20Metrics%20Table https://www.snowflake.com/blog/arctic-open-efficient-foundat...
- CharlesW 2y agoTo be fair, that's comparing their 17B model with the 70B Llama 3 model.
- ru552 2y agoTo stay fair, their "17B" model sits at 964GB on your disk and the 70B Llama 3 model sits at 141GB. unquantized GB numbers for both
- CharlesW 2y agoSorry, it sounds like you know a lot more than I do about this, and I'd appreciate it if you'd connect the dots. Is your comment a dig at either Snowflake or Llama? Where are you finding the unquantized size of Llama 3 70B? Isn't it extremely rare to do inference with large unquantized models?
- fsiefken 2y agoto stay fairer, the required extra disk space for snowflake-arctic is cheaper then the required extra ram memory for llama3
- Manabu-eo 2y agoFor decent performance, you need to keep all the parameters on memory for both. Well, with a raid-0 of two PCIe 5 SSDs (or 4 PCIe 4) you might get 1 t/s loading experts from disk on snowflake-artic... but that is slooow.