3 ms·
Only until they start incorporating this test into their training data.
by aqme28 1y ago
Only until they start incorporating this test into their training data.
- orbital-decay 1y agoDataset contamination alone won't get them good-looking SVG pelicans on bicycles though, they'll have to either cheat this particular question specifically or train it to make vector illustrations in general. At which point it can be easily swapped for another problem that wasn't in the data.
- jug 1y agoI like this one as an alternative, also requiring using a special representation to achieve a visual result: https://voxelbench.ai https://voxelbench.ai What's more, this doesn't benchmark a singular prompt.
- nwienert 1y agothey can have some cheap workers make about 10 pelicans by hand in svg, fuzz them to generate thousands of variations and throw it in their training pool. don't need to 'get good at svgs' by any means.