3 ms·
The problem with these Open weights LLMs hosted by these provider is that we don't what's the precision of the LLM, that makes a huge difference in the speed an
by amrrs 2y ago
The problem with these Open weights LLMs hosted by these provider is that we don't what's the precision of the LLM, that makes a huge difference in the speed and cost (compute).
I think Together recently introduced a different price tier based on precision but otherwise it is usually dark.
- Whiteshadow12 2y agoSomething Open Router get's close to that https://openrouter.ai/models/meta-llama/llama-3.1-405b-instruct https://openrouter.ai/models/meta-llama/llama-3.1-405b-instr...
- amrrs 2y agoAmazing, just noticed the mention of precision
- alextttty 2y agoExactly, always best to rely on your hardware, we need to collect/add more data from self hosted models on different gpus/clouds to compare
- edgoode 2y agoWe have 15+ clouds you can try on our platform if you're looking for a place to compare inference engines Email me at ed at shadeform dot ai if we can help
- s-macke 2y agoI have realized that in my benchmarks [1]. Llama 3 was significantly better than Llama 3.1, which was puzzling. Then I realized, that I changed the provider. And the new one quantized Llama 3.1 with fp8. Then I tried Hyperbolic [2], because they offer the model in different quantizations. As result, Llama 3.1 was better than Llama 3 or at least on par. [1] https://github.com/s-macke/AdventureAI https://github.com/s-macke/AdventureAI [2] https://app.hyperbolic.xyz/models https://app.hyperbolic.xyz/models