3 ms·
I literally cannot believe that people are wasting their time doing this either as a benchmark or for fun. After every single language model release, no less.
by throwuxiytayq 6mo ago
I literally cannot believe that people are wasting their time doing this either as a benchmark or for fun. After every single language model release, no less.
- sharkjacobs 6mo agoIt feels like the results stopped being interesting a little while ago but the practice has become part of simonw's brand, and it gives him something to post even when there is nothing interesting to say about another incremental improvement to a model, and so I don't imagine he'll stop.
- stephbook 6mo agoI, for one, expected progress. Uneven, sometimes delayed, but ever increasing progress. But that Opus pelican?
- segmondy 6mo agoI can't believe you're such a party pooper. It's exciting times, the silly things do matter!
- recursive 6mo agoFun is so un-productive. Everyone doing things for "fun" is going to be sorry when they look back and realizes they were wasting time having a "good time" rather than optimizing their KPIs.
- throwuxiytayq 6mo agoSarcasm aside, asking LLMs do draw pelicans is your idea of fun? I'm worried for you.
- recursive 6mo agoNo. I've never done it. However the stuff I do is even weirder. Thanks for your concern.
- cedws 6mo agoIt’s not a waste of time. As the boundaries of AI are pushed we increasingly struggle to define what intelligence actually is. It becomes more useful to test what models cannot do instead of what they can. Random tasks like the pelican test can show how general the intelligence really is, putting aside the obvious flaw that the labs can optimise for such a simple public benchmark.
- throwuxiytayq 6mo agoThe whole point of this benchmark is that it asks the model to work in a modality it is not trained in and does not understand well. The result is largely meaningless. This is just like the people who are endlessly surprised by the fact that a raw LLM does not work with numbers well, or miscounts letters. In short, this test benchmarks the intelligence of the person running it, not of the model.
- cedws 6mo agoThe rasterised SVG is just a different representation of the same data. A sufficiently advanced LLM may not need to 'see' the rasterised image to be able to draw a good picture. A human could draw a very basic image through raw SVG just by mentally plotting points.
- Marciplan 6mo agoI also can't understand how this goes so viral every time on Hackernews lol
- bschwindHN 6mo agoI do wonder how much energy collectively has been burned on this useless "benchmark".
- casey2 6mo agoThat's what happens in a monopoly enviornment. Literally everyone and every company becomes dancing monkeys for teracaps