3 ms·

Yeah, GLM have been beating Anthropic on the pelicans for a while now. (I suspect that's more of an indication that Anthropic have chosen not to waste resource

• bel8 4mo ago
  That's one possibility albeit quite charitable. I'd be inclined to think the same personally if GLM 5.2 wasn't also rocking in other areas too.
• kamranjon 4mo ago
  This is interesting, I haven’t actually heard you suggest that the labs are focusing on this benchmark before. Have you come around to this position as a result of the quality of pelicans you’ve been getting? The reason I thought this was an interesting benchmark is because it’s a non-image generating model creating an image using SVG code, so it kinda spans capabilities. If an AI lab trained a model specifically for animals riding bicycles it seem trivial to modify the prompt and determine if it was trained specifically for that or if it’s generalized a skill and can also generate a proper orangutan walking on stilts or an armadillo on a skateboard, this sort of thing?
  • simonw 4mo ago
    That's what I've been doing - trying different animals in different vehicles. I'd love to find a lab who does a good pelican on a bicycle but sucks at other combinations, but sadly that's not happened yet. Google Gemini have openly boasted about their animals on vehicles results! https://x.com/JeffDean/status/2024525132266688757 https://x.com/JeffDean/status/2024525132266688757