3 ms·
> and there’s zero chance any AI lab would train a model for such a ridiculous task. I'm not sure that's true anymore considering how popular Simon's blog is
by shepherdjerred 5mo ago
> and there’s zero chance any AI lab would train a model for such a ridiculous task.
I'm not sure that's true anymore considering how popular Simon's blog is
- nickvec 5mo agoSimon mentions further along in his article that given Jeff Dean’s post referencing the pelican-riding-a-bike task (and how good current models are at doing it), that it’s no longer a great benchmark to use. Enter the opossum riding an e-scooter!
- aaronbrethorst 5mo agoBanana man on the Segway
- _puk 5mo ago> So maybe the AI labs have been paying attention after all! > I think this mainly demonstrates that the pelican on the bicycle has firmly exceeded its limits as a useful benchmark. As acknowledged in the article.
- kzrdude 5mo agoGemini 3.1 basically takes it home on that benchmark, anyway, it's done.
- sunaookami 5mo agoGemini is heavily benchmaxxed and sucks in agentic coding so no surprise.
- simonw 5mo agoThat bit probably works better in the talk, it was a setup for a joke later on.
- muzani 5mo agoIt's practically a benchmark now. Some friends have been specifically training models to count the R's in "strawberry"