4 ms·

pelican on a bicycle benchmark probably getting saturated... especially as it's become a popular way to demonstrate model ability quickly

• AlecSchueler 1y ago
  But where is the training set of good pelicans on bikes coming from? You think they have people jigging them up internally?
  • eli 1y ago
    Assuming they updated the crawled training data, just having a bunch of examples of specifically pelicans on bicycles from other models is likely to make a difference.
    • AlecSchueler 1y ago
      But then how does the quality increase? Normally we hear that when models are trained on the output of other models the style becomes very muted and various other issues start to appear. But this probably the best pelicans on a bicycle I've ever seen, by quite some margin.
      • Kuinox 1y ago
        Just compare it with a human on a bicycle, you would see that LLMs are weirdly good at drawing pelicans in SVG but not humans.
        • AlecSchueler 1y ago
          I thought a human would be a considerable step up in complexity but I asked it first for a pelican[0] and then for a rat [1] to get out of the bird world and it did a great job on both. But just fot thrills I also asked for a "punk rocker"[2] and the result--while not perfect--is leaps and bounds above anything from the last generation. 0 -- ok, here's the first hurdle! It's giving me "something went wrong" when I try to get a share link on any of my artifacts. So for now it'll have to be a "trust me bro" and I'll try to edit this comment soon.