3 ms·
I upgraded my llm-gemini plugin to handle this, and shared the results of my "Generate an SVG of a pelican riding a bicycle" benchmark here: https://simonwillis
by simonw 2y ago
I upgraded my llm-gemini plugin to handle this, and shared the results of my "Generate an SVG of a pelican riding a bicycle" benchmark here: https://simonwillison.net/2025/Feb/5/gemini-2/ https://simonwillison.net/2025/Feb/5/gemini-2/
The pricing is interesting: Gemini 2.0 Flash-Lite is 7.5c/million input tokens and 30c/million output tokens - half the price of OpenAI's GPT-4o mini (15c/60c).
Gemini 2.0 Flash isn't much more: 10c/million for text/image input, 70c/million for audio input, 40c/million for output. Again, cheaper than GPT-4o mini.
- zamadatix 2y agoIs there a way to see/compare the shared results for all of the LLMs you've tested this prompt on in one place? The 2.0 pro result seems decent but I don't have a baseline if that's because it is or if the other 2 are just "extremely bad" or something.
- nolist_policy 2y agoSearch by tag: https://simonwillison.net/tags/pelican-riding-a-bicycle/ https://simonwillison.net/tags/pelican-riding-a-bicycle/
- iimaginary 2y agoThe only benchmark worth paying attention to.
- qingcharles 2y agoNot a bad pelican from 2.0 Pro! The singularity is almost upon us :)
- deleted 2y ago[deleted]
- mattlondon 2y agoThe SVGs are starting to look actually recognisable! You'll need a new benchmark soon :)