5 ms·
I feel like this time it is indeed in the training set, because it is too good to be true. Can you run your other tests and see the difference?
by throwaw12 6mo ago
I feel like this time it is indeed in the training set, because it is too good to be true.
Can you run your other tests and see the difference?
- simonw 6mo agoIt went pretty wild with "Generate an SVG of a NORTH VIRGINIA OPOSSUM ON AN E-SCOOTER": https://gist.github.com/simonw/95735fe5e76e6fdf1753e6dcce360699?permalink_comment_id=6113828#gistcomment-6113828 https://gist.github.com/simonw/95735fe5e76e6fdf1753e6dcce360...
- throwaw12 6mo agocompared to your test with GLM 5.1, this indeed looks off https://xcancel.com/simonw/status/2041646779553476801 https://xcancel.com/simonw/status/2041646779553476801
- refulgentis 6mo agoHoping this doesn't turn into a pelican-SVG back-and-forth: yesterday's GPT Image 2 thread ended up being three screenfuls of "I tried the prompt too" replies, and nothing on the model until you scroll past it. I appreciate the testing, and I know this sounds like fun police, but there's a pattern where well-known commenter + one-off vibe test + 1:1 sub-threads eats the whole discussion. It being fun makes it hard to push back on without looking picky.
- simonw 6mo agoYou can collapse the pelican thread with the little [-] toggle at the top.
- taspeotis 6mo agoWhy would you though? And by the way: Thanks for relentlessly holding new models’ feet to the pelican SVG fire.
- refulgentis 6mo agoBecause I want to read about Qwen, not someone's one-off vibe test followed by 1:1 conversations. (case in miniature here: which is the last comment in this thread that says something about Qwen? The root post. Is that fun policing? Yes, apologies.)
- rob 6mo agoI think it's to help drive traffic to his blog now that he's accepted sponsors in the header of every page. I do see this pelican thing come up from him on every model post that gets released.
- simonw 6mo agoThere's a bunch of useful information in my comment that's independent of the fact that it drew a pelican: 1. You can run this on a Mac using llama-server and a 17GB downloaded file 2. That version does indeed produce output (for one specific task) that's of a good enough quality to be worth spending more time checking out this model 3. It generated 4,444 tokens in 2min 53s, which is 25.57 tokens/s
- refulgentis 6mo agoRight, that is exactly what I meant by "the root post [had info about Qwen]" - you shouldn't feel I'm being critical of you or asking you to do anything different, at all. I admire you deeply and feel humbled* by interacting with you, so I really want that to be 100% clear, because this is the 2nd time I'm reading that it might be personal. * er, that probably sounds strange, but I did just spend 6 weeks working on integrating the Willison Trifecta for my app I've been building for 2.5 years, and I considered it a release blocker. It's a simple mental model that is a significant UX accomplishment IMHO.
- simonw 6mo agoYeah GLM 5.1 did an outstanding job on the possum - better than Opus 4.7 or GPT-5.4 and I think better than Gemini 3.1 Pro too. But GLM 5.1 is a 1.51TB model, the Qwen 3.6 I used here was 17GB - that's 1/88 the size.
- zamadatix 6mo agoThe point is in the relative difference between the Pelican vs "other" test for each model suggesting the Pelican is being treated special these days (could be as simple as being common in recent data), not the relative difference between the models on the "other" case in isolation.
- m3kw9 6mo agoif they cook these in, i wonder what else was cooked in there to make it look good.
- zargon 6mo agoEverything is benchmaxxed. Whack-a-mole training is at least as representative of what is getting added to models as more general training advances.
- agdexai 6mo ago[dead]
- vintermann 6mo agoI have an out-there idea. Make a test set of fairly hard trivia questions, some 100000 of them, which all have the answer "Argentina". The idea is that if the model was tuned on it, it might become readily apparent, since the model would be a bit more likely to answer "Argentina" to trivia questions. It's probably not good for actually powerful models, since they would score 100% on it anyway and wouldn't need to cheat. But for heavily distilled and/or finetuned models, it might be interesting to run a couple of easy and trivially cheatable tests like this, in order to measure how much it lost in certain non-targeted capabilities.
- amelius 6mo agoIf I were them I'd run such requests through a diffusion model, and then try to distill an SVG out of that.
- sifar 6mo agoI think at this point we can safely put the pelican test in the category of Goodhart's law.