5 ms·
I've been running this on my laptop with the Unsloth 20.9GB GGUF in LM Studio: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/blob/main/Qwen3.6-35B-A3B-UD-
by simonw 7mo ago
I've been running this on my laptop with the Unsloth 20.9GB GGUF in LM Studio: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/blob/main/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/blob/mai...
It drew a better pelican riding a bicycle than Opus 4.7 did! https://simonwillison.net/2026/Apr/16/qwen-beats-opus/ https://simonwillison.net/2026/Apr/16/qwen-beats-opus/
- jamwise 7mo agoI've had some really gnarly SVGs from Claude. Here's what I got after many iterations trying to draw a hand: https://imgur.com/a/X4Jqius https://imgur.com/a/X4Jqius
- giantg2 7mo agoProbably because all the training material of humans drawing hands are garbage haha.
- bertili 7mo agoIt's fascinating that a $999 Mac Mini (M4 32GB) with almost similar wattage as a human brain gets us this far.
- johanvts 7mo agoInteresting thought, I looked it up out of curiosity and fund 155w max (but realistically more like 80w sustained) for the mac under load, and just around 20watts for the brain, surprisingly almost constant whether “under load” or not.
- petu 7mo ago> 155w max (but realistically more like 80w sustained) 155W PSU seems to be unified with M4 Pro model, plus there's reserve for peripherals (~55W for 5 USB/Thunderbolt ports). Apple lists 65W for base M4 Mac itself: https://support.apple.com/en-am/103253 https://support.apple.com/en-am/103253 Notebookcheck found same number: https://www.notebookcheck.net/Apple-Mac-Mini-M4-review-Smaller-faster-and-louder.918832.0.html#c12271268:~:text=We%20measured%20around%2040%20watts%20when%20gaming%20and%20a%20maximum%20of%2062%2E5%20watts https://www.notebookcheck.net/Apple-Mac-Mini-M4-review-Small...
- fragmede 7mo agoI clocked my M4 at 108 Watts while running inference using Qwen3.6-35b-a3b via Al dente.
- slekker 7mo agoHow does it do with the "car wash" benchmark? :D
- danielhanchen 7mo agoOh that is pretty good! And the SVG one!
- cyclopeanutopia 7mo agoBut that you also gave a win to Qwen on flamingo is pretty outrageous! :) Tthe right one looks much better, plus adding sunglasses without prompting is not that great. Hopefully it won't add some backdoor to the generated code without asking. ;)
- simonw 7mo agoI love how the Chinese models often have an unprompted predilection to add flair. GLM-5.1 added a sparkling earring to a north Virginia opossum the other day and I was delighted: https://simonwillison.net/2026/Apr/7/glm-51/ https://simonwillison.net/2026/Apr/7/glm-51/
- deleted 7mo ago[deleted]
- jubilanti 7mo agoI wonder when pelican riding a bicycle will be useless as an evaluation task. The point was that it was something weird nobody had ever really thought about before, not in the benchmarks or even something a team would run internally. But now I'd bet internally this is one of the new Shirley Cards.
- abustamam 7mo agoSimon has an article on this https://simonwillison.net/2025/Nov/13/training-for-pelicans-riding-bicycles/ https://simonwillison.net/2025/Nov/13/training-for-pelicans-...
- rafaelmn 7mo agoI mean look at the result where he asked about a unicycle - the model couldn't even keep the spokes inside the wheels - would be rudimentary if it "learned" what it means to draw a bicycle wheel and could transfer that to unicycle.
- duzer65657 7mo agoit's the frame that's surprisingly - and consistentnly - wrong. You'd think two triangles would be pretty easy to repro; once you get that the rest is easy. It's not like he's asking "draw a pelican on a four-bar linkage suspension mountainbike..."
- Reddit_MLP2 7mo agoThis is older, but even humans don't have a great concept of how a bicycle works... https://twistedsifter.com/2016/04/artist-asks-people-to-draw-bicycle-from-memory-and-renders-results/ https://twistedsifter.com/2016/04/artist-asks-people-to-draw...
- yndoendo 7mo agoWouldn't this be more about being capable of mentally remembering how a bicycle looks versus how it works? This reminds me of Pictionary. [0] Some people are good and some are really bad. I am really bad a remembering how items look in my head and fail at drawing in Pictionary. My drawing skills are tied to being able to copy what I see. [0] https://en.wikipedia.org/wiki/Pictionary https://en.wikipedia.org/wiki/Pictionary
- prirun 7mo agoThe flamingo on Qwen's unicycle is sitting on the tire, not the seat. That wins because of sunglasses?
- evilduck 7mo agoCan a benchmark meant as a joke not use a fun interpretation of results? The Qwen result has far better style points. Fun sunglasses, a shadow, a better ground, a better sky, clouds, flowers, etc. If we want to get nitty gritty about the details of a joke, a flamingo probably couldn't physically sit on a unicycle's seat and also reach the pedals anyways.
- akavel 7mo agoWell, maybe the flamingo is a really good unicyclist... https://youtu.be/Rrpgd5oIKwI https://youtu.be/Rrpgd5oIKwI
- yabutlivnWoods 7mo agoTransparency of the wheel Stylized gradients on the flamingo Flowers Ground/grass has a stylized look and feel ...despite a miss along the Y-axis where it's below the seat, couple oddly organized tail feathers, spokes, the composition overall is much closer to a production quality entity Opus 4.7 looks like 20 seconds in MS paint. Qwen3.6 looks incomplete due to the sitting position, but like a WIP I could see on a designer coworkers screen if I walk up and interrupt them. Click and drag it up, adjust tail feathers and spokes, you're there or much closer, to a usable output
- culi 7mo agothe more I look at these images the more convinced I become that world models are the major missing piece and that these really are ultimately just stochastic sentence machines. Maybe Chomsky was right
- bmitc 7mo ago> that these really are ultimately just stochastic sentence machines I thought that's exactly what they are?
- mastermage 7mo agoI am so perplexed what exactly where people thinking they were. Its nothing else than highly sofisticated statistics.
- tmountain 7mo agoFrom that perspective, which is totally correct, it makes you wonder what other domains of knowledge look like when pushed to the boundaries of our capabilities as a species.
- mastermage 7mo agoThat is a genuinely thought provoking idea.
- culi 7mo agoDo you know of any other statistical model that can "hallucinate". They clearly have emergent capabilities that come from scale that are absent in any other statistical model we've ever dreamt up. We know that LLMs build complex internal representations of language, logic, and concepts rather than just shallow word-counting. If you deny that then you probably have an elementary understanding of how they work. Not even Chomsky denies that. The real argument imo is whether those internal representations constitute an actual "understanding" of the world or just flatten out to something much less interesting.
- MeteorMarc 7mo agoInteresting, qwen has the pelican driving on the left lane. Coincidence or has it something to do with the workers providing the RL data?
- rubiquity 7mo agoCould be on a bike path where bikes are on the left and pedestrians to the right.
- rdslw 7mo agointeresting, I just tried this very model, unsloth, Q8, so in theory more capable than Simon's Q4, and get those three "pelicans". definitely NOT opus quality. lmstudio, via Simon's llm, but not apple/mlx. Of course the same short prompt. Simon, any ideas? https://ibb.co/gFvwzf7M https://ibb.co/gFvwzf7M https://ibb.co/dYHRC3y https://ibb.co/dYHRC3y https://ibb.co/FLc6kggm https://ibb.co/FLc6kggm (tried here temperature 0.7 instead of pure defaults)
- strobe 7mo agotry Unsloth recommended settings Thinking mode for general tasks: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0 Thinking mode for precise coding tasks (e.g. WebDev): temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0 Instruct (or non-thinking) mode for general tasks: temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0 Instruct (or non-thinking) mode for reasoning tasks: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0 (Please note that the support for sampling parameters varies according to inference frameworks.)
- kelnos 7mo agoI'm not sure how you can give the flamingo win to Qwen: * It's sitting on the tire, not the seat. * Is that weird white and black thing supposed to be a beak? If so, it's sticking out of the side of its face rather than the center. * The wheel spokes are bizarre. * One of the flamingo's legs doesn't extend to the pedal. * If you look closely at the sunglasses, they're semi-transparent, and the flamingo only has one eye! Or the other eye is just on a different part of its face, which means the sunglasses aren't positioned correctly. Or the other eye isn't. * (subjective) The sunglasses and bowtie are cute, but you didn't ask for them, so I'd actually dock points for that. * (subjective) I guess flamingos have multiple tail feathers, but it looks kinda odd as drawn. In contrast, Opus's flamingo isn't as detailed or fancy, but more or less all of it looks correct.
- deleted 7mo ago[deleted]
- withinboredom 7mo agoHe literally said it came down to the comment in the SVG. Points for taste, not correctness. Basically.
- monksy 7mo agoHey I really enjoy your blog. On some things I end up finding a blog post of yours thats a year+ old and at other times, you and I are investigating similar things. I just pulled Qwen3.6 - 35b -A3B (Can't believe thats a A3B coming from 35b). I'm impressed about the reach of your blog, and I'm hoping to get into blogging similar things. I currently have a lot on my backlog to blog about. In short, keep up the good work with an interesting blog!
- bwv848 7mo agoI've been trying the Q4_K_M version, and sometimes it gets stuck in a loop. Gemma 4 doesn’t have this issue.
- yencabulator 7mo agoThis has happened before with quantizations and other backends (ones not used by the research lab). Give it a week, download latest versions of everything, and try again.
- Readerium 7mo agoperhaps increasing repitition_penalty might be helpful
- mobiuscog 7mo agoI'm having the same issues, the more I use it. The repetition penalty doesn't seem to help. I get some really amusing 'reflective' responses, but I think it needs a bit more cooking. Maybe I'll try another variant.
- GistNoesis 7mo agoThanks for pointing to the GGUF. I just tried this GGUF with llama.cpp in its UD Q4_K_XL version on my custom agentic oritened task consisiting of wiki exploration and automatic database building ( https://github.com/GistNoesis/Shoggoth.db/ https://github.com/GistNoesis/Shoggoth.db/ ) I noted a nice improvement over QWen3.5 in its ability to discover new creatures in the open ended searching task, but I've not quantified it yet with numbers. It also seems faster, at around 140 token/s compared to 100 token/s , but that's maybe due to some different configuration options. Some little difference with QWen3.5 : to avoid crashes due to lack of memory in multimodal I had to pass --no-mmproj-offload to disable the gpu offload to convert the images to tokens otherwise it would crash for high resolutions images. I also used quantized kv store by passing -ctk q8_0 -ctv q8_0 and with a ctx-size 150000 it only need 23099 MiB of device memory which means no partial RAM offloading when I use a RTX 4090.
- jaspanglia 7mo agoThe real question is what the next truly weird, un-optimized prompt will be. Something involving a sloth debugging a quantum computer in MS Paint?"
- realityfactchex 7mo agoHere's a reproduction attempt (LM Studio, same Qwen3.6-35B-A3B-GGUF model as linked in parent, M1 Max 64GB, <90 seconds): https://files.catbox.moe/r3oru2.png https://files.catbox.moe/r3oru2.png - My Qwen 3.6 result had sun and cloud in sky, similar to the second Opus 4.7 result in Simon's post. - My Qwen 3.6 result had no grass (except as a green line), but all three results in Simon's post had grass (thick). - My Qwen 3.6 result had visible "tailing air motion" like Simon's Qwen 3.6 result. - My Qwen 3.6 result had a "sun with halo" effect that none of Simon's results had. But, I know, it's more about the pelican and the bicycle.
- _ache_ 7mo agoThe bicycle frame is ok. Simon's was better but at least it's not broken like Opus 4.7. I can't comment that flamingo.
- quietsegfault 7mo agoThe qwen flamingo looks like it’s smoking’ a doobie.
- Scrounger 7mo agoI've been running qwen3.6:35b-a3b-q4_K_M (22.3GB) via Ollama. Is the 20.9GB GGUF version better or negligible in comparison?
- logicallee 7mo agowhat kind of specs does your laptop have? do you know how many tokens/second you get on it?