3 ms·
This Pelican benchmark has become irrelevant. SVG is already ubiquitous. We need a new, authentic scenario.
by solarized 8mo ago
This Pelican benchmark has become irrelevant. SVG is already ubiquitous.
We need a new, authentic scenario.
- viraptor 8mo agoLike identifying names of skateboard tricks from the description? https://skatebench.t3.gg/ https://skatebench.t3.gg/
- hmottestad 8mo agoo3-pro is better than 5.2 pro! And GPT 5 high is best. Really quite interesting.
- alargemoose 8mo agoI don’t care how practical it may or may not be, this is my new favorite LLM benchmark
- stevage 8mo agoI couldn't find an about page or similar?
- viraptor 8mo agoHere's the public sample https://github.com/T3-Content/skatebench/blob/main/bench/tests/skate-trick-test.json https://github.com/T3-Content/skatebench/blob/main/bench/tes... I don't think there's a good description anywhere. https://youtube.com/@t3dotgg https://youtube.com/@t3dotgg talks about it from time to time.
- echelon 8mo ago1. Take the top ten searches on Google Trends (on day of new model release) 2. Concatenate 3. SHA-1 hash them 4. Use this as a seed to perform random noun-verb lookup in an agreed upon large sized dictionary. 5. Construct a sentence using an agreed upon stable algorithm that generates reasonably coherent prompts from an immensely deep probability space. That's the prompt. Every existing model is given that prompt and compared side-by-side. You can generate a few such sentences for more samples. Alternatively, take the top ten F500 stock performers. Some easy signal that provides enough randomness but is easy to agree upon and doesn't provide enough time to game. It's also something teams can pre-generate candidate problems for to attempt improvement across the board. But they won't have the exact questions on test day.