5 ms·

I've seen a bunch of tweets like this recently, as far as I can tell they're all from people using https://aistudio.google.com/ https://aistudio.google.com/ who

by simonw 1y ago

I've seen a bunch of tweets like this recently, as far as I can tell they're all from people using https://aistudio.google.com/ https://aistudio.google.com/ who got served an A/B test. A few more in this genre: https://x.com/cannn064/status/1973818263168852146 https://x.com/cannn064/status/1973818263168852146 - "Make a SVG of a PlayStation 4 controller" https://x.com/cannn064/status/1973415142302830878 https://x.com/cannn064/status/1973415142302830878 "Create a single, self-contained HTML5 file that mimics a macOS Sonoma-style desktop: translucent menu bar with live clock, magnifying dock, draggable/resizable windows, and a dynamic wallpaper. No external assets; use inline SVG for icons." https://x.com/synthwavedd/status/1973405539708056022 https://x.com/synthwavedd/status/1973405539708056022 "Write full HTML, CSS and Javascript for a very realistic page on Apple's website for the new iPhone 18" I've not seen it myself so I'm not sure how confident they are that it's Gemini 3.0.

ceejayoz 1y ago
> a very realistic page on Apple's website… Is this supposed to be a good example? It looks like something I'd put together, and you don't want me doing design work.
ajcp 1y ago
At this point until I see one run through the Pelican Benchmark I can't really take a new model seriously.
- diggan 1y ago
  Unfortunately, as every public benchmark, once it ends up in the training sets and/or the developers aware of it, it stops being effective, and I think we've started to reach that point. The only thing I've found to give me some sort of quantitative idea of how good a new model is, is my own private benchmarks. It doesn't cover everything I want to use LLMs for, and only has 20-30 tests per "category", but at least I'm 99% sure it isn't in the training datasets.
  - simonw 1y ago
    I have a few "SVG of an X riding a Y" tests that I don't publish online which I run occasionally to see if a model is suspiciously better at drawing a pelican riding a bicycle than some other creature on some other form of transport. I would be so entertained if I found out an AI lab had wasted their time cheating on my dumb benchmark!
    - ajcp 1y ago
      -> I would be so entertained if I found out an AI lab had wasted their time cheating on my dumb benchmark! Que intro: "The gang wastes their time cheating on a dumb benchmark"
      - mcny 1y ago
        A shower thought I just had: there must be some AI training company somewhere that has injested all It is always sunny in Philadelphia, not just the text but all the video from all episodes somehow...
    - Imustaskforhelp 1y ago
      Please do let us know through your blog post if you ever find AI labs to cheat on your benchmark. But now I am worried that since you have shared that you do SVG of an X riding a Y thing, maybe these models will try to cheat on the whole SVG of X riding Y thing instead of hyper focusing the pelican. So now I suppose you might need to come up with an entirely new thing though :)
      - throwup238 1y ago
        There are so many X and Y combinations that I find it hard to believe they could realistically train for a even a small fraction of them. Someone has to generate the graphics output for the training. A duck billed platypus riding a unicycle? A man o' war riding a pyrosome? A chicken riding a Quetzalcoatlus? A tardigrade riding a surf board?
epolanski 1y ago
Jm2c I couldn't care less about this vibecode style benchmarks and I'm sick of them. The things that I see represented here, may or may not be impressive, but sure as hell have never been the major blockers in achieving progress on complex tasks and software. I understand you're merely reporting, thank you for that, not criticizing you, but those tests are absolutely irrelevant.
- mnk47 1y ago
  In my experience, the model's performance in silly tasks like these is usually (not always) correlated with its performance in other areas except tool use/agent stuff.

by simonw 1y ago

ceejayoz 1y ago

> a very realistic page on Apple's website… Is this supposed to be a good example? It looks like something I'd put together, and you don't want me doing design work.

ajcp 1y ago

At this point until I see one run through the Pelican Benchmark I can't really take a new model seriously.

diggan 1y ago
Unfortunately, as every public benchmark, once it ends up in the training sets and/or the developers aware of it, it stops being effective, and I think we've started to reach that point. The only thing I've found to give me some sort of quantitative idea of how good a new model is, is my own private benchmarks. It doesn't cover everything I want to use LLMs for, and only has 20-30 tests per "category", but at least I'm 99% sure it isn't in the training datasets.
- simonw 1y ago
  I have a few "SVG of an X riding a Y" tests that I don't publish online which I run occasionally to see if a model is suspiciously better at drawing a pelican riding a bicycle than some other creature on some other form of transport. I would be so entertained if I found out an AI lab had wasted their time cheating on my dumb benchmark!
  - ajcp 1y ago
    -> I would be so entertained if I found out an AI lab had wasted their time cheating on my dumb benchmark! Que intro: "The gang wastes their time cheating on a dumb benchmark"
    - mcny 1y ago
      A shower thought I just had: there must be some AI training company somewhere that has injested all It is always sunny in Philadelphia, not just the text but all the video from all episodes somehow...
  - Imustaskforhelp 1y ago
    Please do let us know through your blog post if you ever find AI labs to cheat on your benchmark. But now I am worried that since you have shared that you do SVG of an X riding a Y thing, maybe these models will try to cheat on the whole SVG of X riding Y thing instead of hyper focusing the pelican. So now I suppose you might need to come up with an entirely new thing though :)
    - throwup238 1y ago
      There are so many X and Y combinations that I find it hard to believe they could realistically train for a even a small fraction of them. Someone has to generate the graphics output for the training. A duck billed platypus riding a unicycle? A man o' war riding a pyrosome? A chicken riding a Quetzalcoatlus? A tardigrade riding a surf board?

epolanski 1y ago

Jm2c I couldn't care less about this vibecode style benchmarks and I'm sick of them. The things that I see represented here, may or may not be impressive, but sure as hell have never been the major blockers in achieving progress on complex tasks and software. I understand you're merely reporting, thank you for that, not criticizing you, but those tests are absolutely irrelevant.

mnk47 1y ago
In my experience, the model's performance in silly tasks like these is usually (not always) correlated with its performance in other areas except tool use/agent stuff.