4 ms·

IMHO Qwen are shipping the best OSS models you can run locally on consumer GPUs right now, getting great results from both qwen2.5-coder:32b and qwq:32b running

by mythz 2y ago

IMHO Qwen are shipping the best OSS models you can run locally on consumer GPUs right now, getting great results from both qwen2.5-coder:32b and qwq:32b running at ~18 tok/s and running on older NVidia A4000 GPUs. Definitely my first choices for local workloads. It's also great to see qwq's open chain of thought baked in a OSS LLM so you can see it reason with itself in real-time, it's the kind of secret sauce that proprietary LLMs like o1 would prefer to keep hidden to try build a moat around. We've got a lot to thank Meta and Qwen for in continually releasing improving high quality OSS models which also encourages others to follow. High quality OSS models are the best thing keeping the cost of LLMs down, you can get unbelievable value on OpenRouter with qwen2.5-coder:32b at $0.08/$0.18 M/tok qwq:32 available at $0.15/$0.60 M/tok which is more than 18x cheaper than Anthropic's latest budget Haiku 3.5 model at $0.80/$4 M/tok (4x price hike over Haiku 3.0).

simonw 2y ago
Yeah, Haiku 3.5 doesn't really count as a budget model - Anthropic have kept the original Claude 3 Haiku around for their budget entry. I collected notes on the lowest cost hosted LLMs from the major vendors when I wrote up Amazon Nova last week: https://simonwillison.net/2024/Dec/4/amazon-nova/ https://simonwillison.net/2024/Dec/4/amazon-nova/ Nova Micro is $0.035/$0.14 and Google's Gemini 1.5 Flash 8B is $0.0375/$0.15 - just beating those OpenRouter prices, but it may well be that the Qwen models provide better results.
- mythz 2y ago
  Yeah I'm currently using Gemini 2 Flash (exp) free quota for a premium hosted model, it's a surprisingly great model, IMO Google has caught up with the leaders with their latest experimental models. I've also tested Nova's models, which are pretty high quality and exception value (lite/micro) for their performance. Also worth shouting out you can get Meta's latest llama-3.3:70b (comparable to llama3.1:405b but must faster and cheaper) within GroqCloud's free quotas running at an impressive 276 tok/s.
  - sharih 2y ago
    Groq limits context window to 8192 is that your experience too
- antupis 2y ago
  Have you tested tool calling capabilities of these cheaper models?
  - simonw 2y ago
    Not yet. A lot of them make bold claims and there are benchmarks like https://gorilla.cs.berkeley.edu/leaderboard.html https://gorilla.cs.berkeley.edu/leaderboard.html but I don't have my own function calling test harness setup yet.
  - leobg 2y ago
    You know that you can make any model call and use tools simply by giving it few shot examples and writing your own parsing logic. I’ve done it many times for clients, both at the prompt and at the fine-tune level.
foo42 2y ago
Previously I've always been very skeptical of rosy pictures of a possible future where "everyone has an ai that's there to accomplish tasks for them" - given that I imagined such ai (if it ever came to exist) being run by the usual big tech who have their own incentives not so cleanly aligned with our own. Right now, with the availability of open weights for cutting-edge models, it feels like this wave of technological advance is pleasantly decentralised however. I can download and run a model and tinker with things which at least feel like the seeds of such a future, where I _might_ be able to build things with my own interests at heart. But what happens if these models stop being shared, and how likely is that? Reading about the vast quantities of compute deployed to train them, replicating the successes of the main players with a community of volunteers just seems an order of magnitude less achievable than traditional OSS efforts like Linux. This wave feels so tied to massive scale for its success, what do we do if big-tech stop handing out models?
- mythz 2y ago
  I think we're all fortunate in that the companies behind the best OSS models, i.e. Meta/llama and Alibaba/Qwen are funding their compute and R&D from secondary business models instead of VC capital or an AI company who's primary business model is direct revenue from their models and will be seeking ROI. That's why I don't expect we can rely on Mistral AI to OSS their best models in the long run since that's their primary business model. This is reflected in their hosting costs which charge a healthy premium that's always more expensive than OpenRouter providers hosting their OSS models. But I don't see why Meta and Alibaba would stop releasing their best models as OSS, since they benefit from the tooling, optimizations and software ecosystems being developed around their OSS models and don't benefit from a future where the best AI models are centralized behind the big tech Corps. As long as their core business remain profitable I don't expect them to stop improving and sharing their OSS models.
- d3VwsX 2y ago
  I read an article about humanoid robots yesterday and it scared me that it seemed like the expectation is still that the robot will be 24/7 online and "thinking" using some cloud brain. The current models described more in detail all used Open AI as a brain. Having a personal robot would be great, but they have to invent a fully offline real positronic brain before I will consider allowing one in my house. Fully open source might be too much to hope for, but that would obviously be the ideal. If it is closed source it definitely should be offline. I can have another, carefully sandboxed, AI in my computer that can help out with tasks that require online access. No need for the two types to be built into the same device.

by mythz 2y ago

simonw 2y ago

Yeah, Haiku 3.5 doesn't really count as a budget model - Anthropic have kept the original Claude 3 Haiku around for their budget entry. I collected notes on the lowest cost hosted LLMs from the major vendors when I wrote up Amazon Nova last week: https://simonwillison.net/2024/Dec/4/amazon-nova/ https://simonwillison.net/2024/Dec/4/amazon-nova/ Nova Micro is $0.035/$0.14 and Google's Gemini 1.5 Flash 8B is $0.0375/$0.15 - just beating those OpenRouter prices, but it may well be that the Qwen models provide better results.

mythz 2y ago
Yeah I'm currently using Gemini 2 Flash (exp) free quota for a premium hosted model, it's a surprisingly great model, IMO Google has caught up with the leaders with their latest experimental models. I've also tested Nova's models, which are pretty high quality and exception value (lite/micro) for their performance. Also worth shouting out you can get Meta's latest llama-3.3:70b (comparable to llama3.1:405b but must faster and cheaper) within GroqCloud's free quotas running at an impressive 276 tok/s.
- sharih 2y ago
  Groq limits context window to 8192 is that your experience too
antupis 2y ago
Have you tested tool calling capabilities of these cheaper models?
- simonw 2y ago
  Not yet. A lot of them make bold claims and there are benchmarks like https://gorilla.cs.berkeley.edu/leaderboard.html https://gorilla.cs.berkeley.edu/leaderboard.html but I don't have my own function calling test harness setup yet.
- leobg 2y ago
  You know that you can make any model call and use tools simply by giving it few shot examples and writing your own parsing logic. I’ve done it many times for clients, both at the prompt and at the fine-tune level.

foo42 2y ago

Previously I've always been very skeptical of rosy pictures of a possible future where "everyone has an ai that's there to accomplish tasks for them" - given that I imagined such ai (if it ever came to exist) being run by the usual big tech who have their own incentives not so cleanly aligned with our own. Right now, with the availability of open weights for cutting-edge models, it feels like this wave of technological advance is pleasantly decentralised however. I can download and run a model and tinker with things which at least feel like the seeds of such a future, where I _might_ be able to build things with my own interests at heart. But what happens if these models stop being shared, and how likely is that? Reading about the vast quantities of compute deployed to train them, replicating the successes of the main players with a community of volunteers just seems an order of magnitude less achievable than traditional OSS efforts like Linux. This wave feels so tied to massive scale for its success, what do we do if big-tech stop handing out models?

mythz 2y ago
I think we're all fortunate in that the companies behind the best OSS models, i.e. Meta/llama and Alibaba/Qwen are funding their compute and R&D from secondary business models instead of VC capital or an AI company who's primary business model is direct revenue from their models and will be seeking ROI. That's why I don't expect we can rely on Mistral AI to OSS their best models in the long run since that's their primary business model. This is reflected in their hosting costs which charge a healthy premium that's always more expensive than OpenRouter providers hosting their OSS models. But I don't see why Meta and Alibaba would stop releasing their best models as OSS, since they benefit from the tooling, optimizations and software ecosystems being developed around their OSS models and don't benefit from a future where the best AI models are centralized behind the big tech Corps. As long as their core business remain profitable I don't expect them to stop improving and sharing their OSS models.
d3VwsX 2y ago
I read an article about humanoid robots yesterday and it scared me that it seemed like the expectation is still that the robot will be 24/7 online and "thinking" using some cloud brain. The current models described more in detail all used Open AI as a brain. Having a personal robot would be great, but they have to invent a fully offline real positronic brain before I will consider allowing one in my house. Fully open source might be too much to hope for, but that would obviously be the ideal. If it is closed source it definitely should be offline. I can have another, carefully sandboxed, AI in my computer that can help out with tasks that require online access. No need for the two types to be built into the same device.