18 ms·

llama3 narrowly beats arctic at SQL generation (80.2 vs 79.0) and Mixtral 8x22B scored 79.2. You'd think SQL would be the one thing they'd be sure to smoke oth

llama3 narrowly beats arctic at SQL generation (80.2 vs 79.0) and Mixtral 8x22B scored 79.2. You'd think SQL would be the one thing they'd be sure to smoke other models on. 0 - https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/#:~:text=Table%203.%20Full%20Metrics%20Table.%20Comparing%20Snowflake%20Arctic%20with%20DBRX%2C%20LLAMA%2D3%208B%2C%20LLAMA%2D3%2070B%2C%20Mixtral%208x7B%2C%20Mixtral%208x22B%20(instruction%2Dtuned%20or%20chat%20variants%20if%20available).1 https://www.snowflake.com/blog/arctic-open-efficient-foundat...

sp332 2y ago
Yeah but that's a 70B model. You can see on the Inference Efficiency chart that it takes more than 3x as much compute to run it compared to this one.
- msp26 2y ago
  Most people are vram constrained not compute constrained.
  - kaibee 2y ago
    Cloud providers aren’t though.
  - Manabu-eo 2y ago
    But those people usually have more system RAM than VRAM. At those scales, most people become bandwidth and compute constrained using CPU inference instead of multiple GPUs. In those cases, an MOE with a low number of active parameters is the fastest.
- karmasimida 2y ago
  But you do need to hold all 128 experts in memory? Or not? Or they simply consider inference efficiency as latency
  - giantrobot 2y ago
    I believe the main draw of the MoE model is they don't all need to be in memory at once. They can be swapped based on context. In aggregate you get the performance of a much larger model (384b tokens) while using much less memory than such a model would require. If you had enough memory it could all be loaded but it doesn't need to be.
    - sp332 2y ago
      Technically you could, but it would take much longer to do all that swapping.
    - Manabu-eo 2y ago
      Wrong. MoE models like this one usually chose a different and unpredictable mix of experts for each token, and as such you need all parameters at memory at once. It lessens the number of parameters that need to be moved from memory to compute chip for each token, not from disk to memory.
    - qeternity 2y ago
      "Expert" in MoE has no bearing on what you might think of as a human expert. It's not like there is one expert that is proficient at science, and one that is proficient in history. For a given inference request, you're likely to activate all the experts at various points. But for each individual forward pass (e.g. each token), you are only activating a few.
  - rajhans 2y ago
    Arctic dev here. Yes keeping all experts in memory is the recommendation here and understandably that is a barrier to some. But once you have 1 H100 node or two (gpu middle-class I guess...?), then a few things to note: 1. FP6/FP8 inference is pretty good. How to on a single node: https://github.com/Snowflake-Labs/snowflake-arctic/tree/main/inference https://github.com/Snowflake-Labs/snowflake-arctic/tree/main... (vllm support coming soon!) 2. Small number of activated parameters shine in batch inference case for cloud providers.
    - 2y ago
adrien-treuille 2y ago
Actually, Snowflake doesn’t use Arctic for SQL codegen internally. They use a different model chained with mistral-large… and they do smoke the competition. https://medium.com/snowflake/1-1-3-how-snowflake-and-mistral-push-the-boundary-in-text2sql-169bfa391331 https://medium.com/snowflake/1-1-3-how-snowflake-and-mistral...
- mritchie712 2y ago
  smoke? it's the same as gpt4 https://medium.com/snowflake/1-1-3-how-snowflake-and-mistral-push-the-boundary-in-text2sql-169bfa391331#:~:text=When%20we%20combine%20the%20Snowflake%20fine%2Dtuned%20model%20with%20Mistral%20Large%20in%20a%20chained%20setup%2C%20we%20are%20able%20to%20improve%20execution%20accuracy%20by%206%20percentage%20points%20from%20~40%25%20to%20%3E46%25%20in%20our%20internal%20evaluation%20dataset%20%E2%80%94%20outperforming%20GPT%2D4%3A https://medium.com/snowflake/1-1-3-how-snowflake-and-mistral...

sp332 2y ago

Yeah but that's a 70B model. You can see on the Inference Efficiency chart that it takes more than 3x as much compute to run it compared to this one.

msp26 2y ago
Most people are vram constrained not compute constrained.
- kaibee 2y ago
  Cloud providers aren’t though.
- Manabu-eo 2y ago
  But those people usually have more system RAM than VRAM. At those scales, most people become bandwidth and compute constrained using CPU inference instead of multiple GPUs. In those cases, an MOE with a low number of active parameters is the fastest.
karmasimida 2y ago
But you do need to hold all 128 experts in memory? Or not? Or they simply consider inference efficiency as latency
- giantrobot 2y ago
  I believe the main draw of the MoE model is they don't all need to be in memory at once. They can be swapped based on context. In aggregate you get the performance of a much larger model (384b tokens) while using much less memory than such a model would require. If you had enough memory it could all be loaded but it doesn't need to be.
  - sp332 2y ago
    Technically you could, but it would take much longer to do all that swapping.
  - Manabu-eo 2y ago
    Wrong. MoE models like this one usually chose a different and unpredictable mix of experts for each token, and as such you need all parameters at memory at once. It lessens the number of parameters that need to be moved from memory to compute chip for each token, not from disk to memory.
  - qeternity 2y ago
    "Expert" in MoE has no bearing on what you might think of as a human expert. It's not like there is one expert that is proficient at science, and one that is proficient in history. For a given inference request, you're likely to activate all the experts at various points. But for each individual forward pass (e.g. each token), you are only activating a few.
- rajhans 2y ago
  Arctic dev here. Yes keeping all experts in memory is the recommendation here and understandably that is a barrier to some. But once you have 1 H100 node or two (gpu middle-class I guess...?), then a few things to note: 1. FP6/FP8 inference is pretty good. How to on a single node: https://github.com/Snowflake-Labs/snowflake-arctic/tree/main/inference https://github.com/Snowflake-Labs/snowflake-arctic/tree/main... (vllm support coming soon!) 2. Small number of activated parameters shine in batch inference case for cloud providers.
  - 2y ago

adrien-treuille 2y ago

Actually, Snowflake doesn’t use Arctic for SQL codegen internally. They use a different model chained with mistral-large… and they do smoke the competition. https://medium.com/snowflake/1-1-3-how-snowflake-and-mistral-push-the-boundary-in-text2sql-169bfa391331 https://medium.com/snowflake/1-1-3-how-snowflake-and-mistral...

mritchie712 2y ago
smoke? it's the same as gpt4 https://medium.com/snowflake/1-1-3-how-snowflake-and-mistral-push-the-boundary-in-text2sql-169bfa391331#:~:text=When%20we%20combine%20the%20Snowflake%20fine%2Dtuned%20model%20with%20Mistral%20Large%20in%20a%20chained%20setup%2C%20we%20are%20able%20to%20improve%20execution%20accuracy%20by%206%20percentage%20points%20from%20~40%25%20to%20%3E46%25%20in%20our%20internal%20evaluation%20dataset%20%E2%80%94%20outperforming%20GPT%2D4%3A https://medium.com/snowflake/1-1-3-how-snowflake-and-mistral...