3 ms·
The pelican has looked very same-y across all frontier models, same color bike, same camera angle, etc. I suspect this challenge is already too embedded in the
by sempron64 4mo ago
The pelican has looked very same-y across all frontier models, same color bike, same camera angle, etc. I suspect this challenge is already too embedded in the training data to be a good signal when it succeeds, and maybe even when it fails in pathological ways mirroring existing AI pelicans on the internet.
- tripleee 4mo ago[flagged]
- jurgenaut23 4mo agoTrue that
- scrollaway 4mo agoDo you seriously have a dedicated “bad takes on AI” hn account?
- tripleee 4mo agoyeah, although I do combine it with "replies to snarky questions" for efficiency
- yreg 4mo agoI really don't understand what's interesting about this test and why is it always on top.
- simonw 4mo agoIt's funny.
- depr 4mo agoSame reason you would always see the same top comments on reddit during a certain era.
- yreg 4mo agoThat’s what I think too, but we should actively go against such culture here because hn is not reddit.
- gunsle 4mo agoIt basically is at this point, if you haven’t noticed. Complete with the same America bad, Elon bad, democrats good midwit progressive politics.
- replwoacause 4mo agoElon does suck. Objectively.
- anhner 4mo agoDon't forget EU bad! Because they won't let Apple screw over consumers.
- ankit_mishra 4mo agoIs this Straw Man and Ad Hominem ?
- clydethefrog 4mo agoAlmost all Musk related negative news gets [flagged] and never hits the the front page, so there is still a silent base on the other "team" apparently.
- WithinReason 4mo agoIt's a meme, and HN loves upvoting memes. Just like Reddit!
- luqtas 4mo agobecause you can't still ask LLMs to port DOOM to hardware X or Y
- inglor_cz 4mo agoIt has become a funny meme, much like "My hovercraft is full of eels!"
- port11 4mo agoThe ultimate measure of an LLM is whether it can produce a capable image of a pelican riding a bicycle. All other use cases are but a distraction!
- quantumwoke 4mo agoVariations of this comment have been posted for over a year. The pelican has now morphed into part of HN culture rather than a legitimate benchmark, but it's still valuable as a meme.
- brazukadev 4mo agoit is more an example of gaming (the HN system) than meme.
- h4ny 4mo agoWas it ever a good test? How do you even objectively assess what a good pelican on a bike is anyway?
- fwipsy 4mo agoSVG generation is a good test because it's extremely easy to subjectively assess with visual reasoning where humans are strong. However, pelican on a bike specifically may be overused at this point.
- kayge 4mo agoDo you think the models are ready for the next level? I believe that would be: Pelican feeding Spaghetti to Will Smith.
- stratos123 4mo agoI'd be very surprised if this is in the training data given that most models mess it up to this day. E.g. look at the ones from Opus.
- Fuzzwah 4mo agoThe "big beak!" comment in the svg source makes me think it's definitely a gamed "benchmark" at this point.