6 ms·
It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in o
by OsrsNeedsf2P 3mo ago
It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website
- eminence32 3mo agoPelicans and bikes can be in the training set without them training for this specific benchmark.
- j_maffe 3mo agoYes and that would improve its ability to draw SVGs of pelicans on bikes, no?
- asasidh 3mo agoand that is bad because ?
- program_whiz 3mo agothe nature of the test was to see if the models can effectively compose an image of a novel concept outside the training set. If they are trained on it, it ceases to be an interesting test to some extent.
- cyanydeez 3mo agoit's still interesting because there's no pelican-on-bike model, and if you're training a model well enough, then it should be obvious when a model has reached "AGI" or whatever.
- wasabi991011 3mo agoI would urge you to re-read the blog post you are commenting on. It pretty clearly explains how it is an interesting test independently of "see[ing] if the models can effectively compose an image of a novel concept outside the training set".
- barrenko 3mo agoWould it? Tongue in cheek.
- deleted 3mo ago[deleted]
- freedomben 3mo ago> Yes and that would improve its ability to draw SVGs of pelicans on bikes, no? I would think the opposite because unless people have been hand drawing these with high quality, the training would be on much crappier versions that old AIs have done.
- semilin 3mo agoThey can be in the training set but not deliberately trained for. There may be a lot of people posting pelican svgs, but not typically because they're high quality and worth replicating.
- podgietaru 3mo agoMore to it, the actual bloody companies are using them as a reference. Maybe it’s a 3d version, not an svg - but it clearly shows they’re on the radar of these companies.
- cebert 3mo agoSimon has stated a few times that he knows it’s possible that pelicans could be in the training sets. He also has other tests he doesn’t share publicly. He’s just a fan of pelicans.
- hungryhobbit 3mo agoFrom the article it doesn't even sound like he cares about pelicans at all, and doesn't think they are a good way to compare models anymore ... but people are used to seeing the test now, and it does serve as a common "hello world" unit of work.
- oceanplexian 3mo agoImagine if we applied this train of logic to humans. "That artist saw a pelican at the beach once!" [cue the outrage] "He's not a real artist, he's a cheater and produces nothing original!"
- computably 3mo agoExcept, of course, LLMs are not humans, and they do not learn or "reason" in a way which even remotely resembles humans. Plus obviously humans can still overfit to a specific style of test.
- program_whiz 3mo agoThis is a sight-reading test. If a musician practices a piece for thousands of hours, it would no longer be an effective sight reading / creativity test. The purpose of the test was to see how models would compose something novel requiring the ability to compose orthogonal, normally unrelated, components into a coherent image.
- alexjplant 3mo agoWe do. People who, for example, memorize question banks to pass certification tests without knowing the underlying material are equally frowned upon for not having the problem solving skills that they purport to. I'll leave the contrasts between LLMs and people to the well-written sibling comments.
- xigoi 3mo agoMore like “This artist won the drawing competition because someone told her the theme in advance and the specifically practiced drawing pelicans for hundreds of hours.”
- andy_xor_andrew 3mo agoDid you read the post? It's not even that long. He explicitly mentions this...
- Barbing 3mo agoAre they responding to: “I’m still not convinced that labs are training for the benchmark—if they were, I’d expect much better results.”
- Certhas 3mo agoIn my reading, "training for the benchmark" is very, very different from "this benchmark is in the training data".
- drcongo 3mo agoClearly not. There's a subset of HN users who rush to post this same thing every single time.
- Topfi 3mo agoMaybe it gets posted every time because besides a personal believe by the person popularising this "benchmark", there is no reason to assume that certain labs aren't intentionally training to game this and every other lab at least unintentionally gets improvements for this specific combination of animal and action because the internet is full of both good and bad examples, often ranked, which does inevitably become training data. I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts. Just operating on a feeling that labs don't optimise for this (as mentioned, even if they don't training data is filled with these) is not solid enough that criticism shouldn't be leveraged when it comes up.
- simonw 3mo ago> I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts Please share those again! One of the things I'm most looking forward to is a lab producing a model that creates a really great pelican riding a bicycle and then a terrible sloth riding a skateboard (or whatever). I've not seen that myself yet.
- HardCodedBias 3mo agoA person from Google famously put on her linkedin that her job was to optimize SVG for Gemini 3.0.
- Chu4eeno 3mo agoSVG output is useful, though. I often ask whatever LLM I have open to generate placeholder icons whenever I need them.
- simonw 3mo agoThe pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!
- OtherShrezzing 3mo agoRespectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model. In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts. Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and producing a watercolour of a pelican. They’re all rendered in an approximately uniform style, even though the svg format has a basically unlimited possibility space.
- simonw 3mo agoYou know what, that's actually something I hadn't considered before. There's definitely a bias towards a pelican cycling from left to right on a red bicycle against a blue sky and green grass. Blue sky and green grass aren't that surprising, but the color and direction are interesting. When I finally build the proper gallery I'll throw in a few other creature-vehicle combinations, and track some characteristics like which direction, color of bicycle, general pelican geometry etc. It will be interesting to see if other creatures end up with coincidentally similar design choices or if that's unique to the pelican-bicycle combination.
- port3000 3mo agoYeah I asked Nano Banana to make a render of our company office and was scarily accurate
- OtomotO 3mo agoIt's incredible people still discuss the pelicans... But then again, the ad just works.
- segmondy 3mo agoIt's incredible you can't reason to see if pelican on a bike is a thing. It's not! This has been discussed to death. You can ask any model to generate anything. Generate an SVG of earthworm and a robin boxing. Guess what? The smarter the model the better the image, doesn't matter if it's a vision model or not. I rolled my eyes at this eval when I first saw it, then I tried various ridiculous things and noticed a very strong correlation. Things that are absolutely not in the training set.
- jhalloran 3mo agoThis reminded me about the news cycle last year that we were running out of training data (and how silly that was)
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- Jimmc414 3mo ago[dead]