4 ms·
As is now traditional for new LLM releases, I used Qwen 3 (32B, run via Ollama on a Mac) to summarize this Hacker News conversation about itself - run at the po
by simonw 2y ago
As is now traditional for new LLM releases, I used Qwen 3 (32B, run via Ollama on a Mac) to summarize this Hacker News conversation about itself - run at the point when it hit 112 comments.
The results were kind of fascinating, because it appeared to confuse my system prompt telling it to summarize the conversation with the various questions asked in the post itself, which it tried to answer.
I don't think it did a great job of the task, but it's still interesting to see its "thinking" process here: https://gist.github.com/simonw/313cec720dc4690b1520e5be3c944976 https://gist.github.com/simonw/313cec720dc4690b1520e5be3c944...
- littlestymaar 2y agoAren't all Qwen models known to perform poorly with system prompt though?
- notfromhere 2y agoQwen does decently, DeepSeek doesn't like system prompts. For Qwen you really have to play with parameters
- simonw 2y agoI hadn't heard that, but it would certainly explain why the model made a mess of this task. Tried it again like this, using a regular prompt rather than a system prompt (with the https://github.com/simonw/llm-hacker-news https://github.com/simonw/llm-hacker-news plugin for the hn: prefix): llm -f hn:43825900 \ 'Summarize the themes of the opinions expressed here. For each theme, output a markdown header. Include direct "quotations" (with author attribution) where appropriate. You MUST quote directly from users when crediting them, with double quotes. Fix HTML entities. Output markdown. Go long. Include a section of quotes that illustrate opinions uncommon in the rest of the piece' \ -m qwen3:32b This worked much better! https://gist.github.com/simonw/3b7dbb2432814ebc8615304756395558 https://gist.github.com/simonw/3b7dbb2432814ebc8615304756395...
- littlestymaar 2y agoWow, it hallucinates quotes a lot!
- croemer 2y agoSeems to truncate the input to only 2048 input tokens
- simonw 2y agoOops! That's an Ollama default setting. You can fix that by increasing the num_ctx setting - I'll try running this again. The num_predict setting controls output size.
- hbbio 2y agoI also have a benchmark that I'm using for my nanoagent[1] controllers. Qwen3 is impressive in some aspects but it thinks too much! Qwen3-0.6b is showing even better performance than Llama 3.2 3b... but it is 6x slower. The results are similar to Gemma3 4b, but the latter is 5x faster on Apple M3 hardware. So maybe, the utility is to run better models in cases where memory is the limiting factor, such as Nvidia GPUs? [1] github.com/hbbio/nanoagent
- phh 2y agoWhat's cool with those models is that you can tweak the thinking process, all the way down to "no thinking". It's maybe not available in your inference engine though
- hbbio 2y agoFeel free to add a PR :) What is the parameter?
- ammo1662 2y agoJust add "/no_think" in your prompt. https://qwenlm.github.io/blog/qwen3/#advanced-usages https://qwenlm.github.io/blog/qwen3/#advanced-usages
- hbbio 2y agoThanks! Turns out just is not the word here. My benchmark is made using conversations, where there is a SystemMessage and some structured content in a UserMessage. But Qwen3 seems to ignore /no_think when appended to the SystemMessage. I can try to add it to the structured content but that will be a bit weird. Would have been better to have a "think" parameter like temperature.
- simonw 2y agoHah, and now we can't summarize this thread any more because your comment will turn thinking off!
- 2y ago
- rcdwealth 2y ago[dead]
- manmal 2y agoOne person on Reddit claimed the first unsloth release was buggy - if you used that, maybe you can retry with the fixed version?
- hobofan 2y agoI think this was only regarding the chat template that was provided in the metadata (this was also broken in the official release). However, I doubt that this would impact this test, as most inference frameworks will just error if provided with a broken template.
- daemonologist 2y agoIt was - Unsloth put up a message on their HF for a while to only use the Q6 and larger. I'm not sure to what extent this affected prediction accuracy though.
- anentropic 2y agoThis sounds like a task where you wouldn't want to use the 'thinking' mode
- claiir 2y agoo1-preview had this same issue too! You’d give it a long conversation to summarize, and if the conversation ended with a question, o1-preview would answer that, completely ignoring your instructions. Generally unimpressed with Qwen3 from my own personal set of problems.