6 ms·
Trying out Gemini 3 Pro with audio transcription and a new pelican benchmark
- simonw 11mo agoThe audio transcript exercise here is particularly interesting from a journalism perspective. Summarizing a 3.5 hour council meeting is something of a holy grail of AI-assisted reporting. There are a LOT of meetings like that, and newspapers (especially smaller ones) can no longer afford to have a human reporter sit through them all. I tried this prompt (against audio from https://www.youtube.com/watch?v=qgJ7x7R6gy0 https://www.youtube.com/watch?v=qgJ7x7R6gy0): Output a Markdown transcript of this meeting. Include speaker names and timestamps. Start with an outline of the key meeting sections, each with a title and summary and timestamp and list of participating names. Note in bold if anyone raised their voices, interrupted each other or had disagreements. Then follow with the full transcript. Here's the result: https://gist.github.com/simonw/0b7bc23adb6698f376aebfd700943314 https://gist.github.com/simonw/0b7bc23adb6698f376aebfd700943... I'm not sure quite how to grade it here, especially since I haven't sat through the whole 3.5 hour meeting video myself. It appears to have captured the gist of the meeting very well, but the fact that the transcript isn't close to an exact match to what was said - and the timestamps are incorrect - means it's very hard to trust the output. Could it have hallucinated things that didn't happen? Those can at least be spotted by digging into the video (or the YouTube transcript) to check that they occurred... but what about if there was a key point that Gemini 3 omitted entirely?
- WesleyLivesay 11mo agoI think it appears to have done a good job of summarizing the points that it summarize, at least judging from my quick watch of a few sections and from the YT Transcript (which seems quite accurate). Almost makes me wonder if it is behind the scenes doing something similar to: rough transcript -> Summaries -> transcript with timecodes (runs out of context) -> throws timestamps that it has on summaries. I would be very curious to see if it does better on something like an hour long chunk of audio, to see if it is just some sort of context issue. Or if this same audio was fed to it in say 45 minute chunks to see if the timestamps fix themselves.
- byt3bl33d3r 11mo agoI’ve been meaning to create & publish a structured extraction benchmark for a while. Using LLMs to extract info/entities/connections from large amounts of unstructured data is also a huge boon to AI-assisted reporting and has also a number of cybersecurity applications. Gemini 2.5 was pretty good but so far I have yet to see an LLM that can reliably , accurately and consistently do this
- simonw 11mo agoThis would be extremely useful. I think this is one of the most commercially valuable uses of these kinds of models, having more solid independent benchmarks would be great.
- mistercheph 11mo agoFor this use case I think best bet is still a toolchain with a transcription model like whisper fed into an LLM to summarize
- simonw 11mo agoYeah I agree. I ran Whisper (via MacWhisper) on the same video and got back accurate timestamps. The big benefit of Gemini for this is that it appears to do a great job of speaker recognition, plus it can identify when people interrupt each other or raise their voices. The best solution would likely include a mixture of both - Gemini for the speaker identification and tone-of-voice stuff, Whisper or NVIDIA Parakeet or similar for the transcription with timestamps.
- rahimnathwani 11mo agoFor this use case, why not use Whisper to transcribe the audio, and then an LLM to do a second step (summarization or answering questions or whatever)? If you need diarization, you can use something like https://github.com/m-bain/whisperX https://github.com/m-bain/whisperX
- pants2 11mo agoWhisper simply isn't very good compared to LLM audio transcription like gpt-4o-transcribe. If Gemini 3 is even better it's a game-changer.
- crazysim 11mo agoSince Gemini seems to be sucking at timestamps, perhaps Whisper can be used to help ground that as an additional input alongside the audio.
- ks2048 11mo agoDoes anyone benchmark these models for text-to-speech using traditional word-error-rates? It seems audio-input Gemini is a lot cheaper than Google Speech-to-text.
- simonw 11mo agoHere's one: https://voicewriter.io/speech-recognition-leaderboard https://voicewriter.io/speech-recognition-leaderboard "Real-World Speech-to-text API Leaderboard" - it includes scores for Gemini 2.5 Pro and Flash.
- Workaccount2 11mo agoMy assumption is that Gemini has no insight into the time stamps, and instead is ballparking it based on how much context has been analyzed up to that point. I wonder if you put the audio into a video that is nothing but a black screen with a timer running, it would be able to correctly timestamp.
- simonw 11mo agoThe Gemini documentation specifically mentions timestamp awareness here: https://ai.google.dev/gemini-api/docs/audio https://ai.google.dev/gemini-api/docs/audio
- minimaxir 11mo agoPer the docs, Gemini represents each second of audio as 32 tokens. Since it's a consistent amount, as long as the model is trained to understand the relation between timestamps and the number of tokens (which per Simon's link it does), it should be able to infer the correct amount of seconds.
- potatolicious 11mo agoYou really want to break a task like this down to constituent parts - especially because in this case the "end to end" way of doing it (i.e., raw audio to summary) doesn't actually get you anything. IMO the right way to do this is to feed the audio into a transcription model, specifically one that supports diarization (separation of multiple speakers). This will give you a high quality raw transcript that is pretty much exactly what was actually said. It would be rough in places (i.e., Speaker 1, Speaker 2, etc. rather than actual speaker names) Then you want to post-process with a LLM to re-annotate the transcript and clean it up (e.g., replace "Speaker 1" with "Mayor Bob"), and query against it. I see another post here complaining that direct-to-LLM beats a transcription model like Whisper - I would challenge that. Any modern ASR model will do a very, very good job with 95%+ accuracy.
- simonw 11mo agoWhich diarization models would you recommend, especially for running on macOS? (Update: I just updated MacWhisper and it can now run Parakeet which appears to have decent diarization built in, screenshot here: https://static.simonwillison.net/static/2025/macwhisper-parakeet.jpg https://static.simonwillison.net/static/2025/macwhisper-para... )
- atonse 11mo agoI use parakeet daily (with MacWhisper) to transcribe my meetings. It works really well, even with the speaker segmentation.
- darkwater 11mo agoWhy can't Gemini, the product, do that by itself? Isn't the point of all this AI hype to easily automate things with low effort?
- vlovich123 11mo agoMultimodal models are only now starting to come into the space and even then I don’t know they really support diarization yet (and often multimodal is thinking+speech/images, not sure about audio).
- stavros 11mo agoJust checking, but did you verify that the converted ffmpeg audio wasn't around an hour long? Maybe ffmpeg sped it up when converting?
- simonw 11mo agoHere is the file I used, it is the expected length https://static.simonwillison.net/static/2025/HMB-nov-4-2025.m4a https://static.simonwillison.net/static/2025/HMB-nov-4-2025....
- luke-stanley 11mo agoSimon, sorry I didn't get around to answering your question on post-t5 encoder-decoders from the Markdown Lethal Trifecta prompt injection post. (https://news.ycombinator.com/item?id=45724941 https://news.ycombinator.com/item?id=45724941) Since the plain decoder models stole the show, Google DeepMind demonstrated a way to adapt LLMs,adding a T5 encoder to an existing normal Gemma model to get the benefits of the more grounded text-to-text tasks WITHOUT instruction tuning (and the increased risk of prompt injection). They also have a few different kinds they shared on HuggingFace. I didn't get around to fine-tuning the weights of one for summarisation yet but it could well be a good way for more reliable summarisation. I did try out some models for inference though and made a Gist here, which is useful since I found the HF default code example a bit broken: https://gist.github.com/lukestanley/ee89758ea315b68fd66ba52cd2123c35 https://gist.github.com/lukestanley/ee89758ea315b68fd66ba52c... Google's minisite: https://deepmind.google/models/gemma/t5gemma/ https://deepmind.google/models/gemma/t5gemma/ Paper: https://arxiv.org/abs/2504.06225 https://arxiv.org/abs/2504.06225 Here is one such model that didn't hallucinate and actually did summarise on HF: https://huggingface.co/google/t5gemma-l-l-prefixlm https://huggingface.co/google/t5gemma-l-l-prefixlm
- luke-stanley 11mo agoI'd lower the temperature and try DSPy Refine loop on it (or similar). Using audio encoder-decoder models and segmentation are good things to try too. A length mismatch would be bad. DSPy has optimisers. It could probably optimise well with length match heuristic, there is probably a good Shannon entropy rule.
- barapa 11mo agoWe have done a lot of work solving this exact challenge at legaide.ai. I was both sad and relieved when gemini 3 didn't deprecate our startup.
- anilgulecha 11mo agoWe've done some transcription exercises. The way to get the timestamps to line is : 1) break up the audio into minutes, and pass it in, one after another, so the chat completions prompt looks like: Here's minute 1 of audio [ffmpeg 1st minute cut out.wav] Here's minute 2 of audio [ffmpeg 2nd minute cut out.wav] Here's minute 3 of audio [ffmpeg 3rd minute cut out.wav] and so on.. The cutting step is simple, and token count is pretty much the same, but the crucial additional detail allows for excellent transcription fidelity time wise. We've also experimented passing in regular TTS (non-llm) transcript for reference, which again helps the LLM do better.
- theshrike79 11mo agoThis is a Google notebook on multimodal transcription from a training session they do: https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/multimodal_video_transcription.ipynb https://colab.research.google.com/github/GoogleCloudPlatform... It goes through the video analysis process and terminology step by step. You need to say "timecode" instead of "timestamp" so the LLM aligns better to AV-stuff more than programming (weird, right?) Basically you need two to three passes to get a proper transcription 1. just listen and identify voices, give them some kind of ID 2. Maybe watch the video and see if the people have name tags on the table in front of them or there's an on-screen overlay (news or interviews) 3. Last pass, go through the transcription and map voices+ids to actual names
- londons_explore 11mo agoAnyone got a class full of students and able to get a human version of this pelican benchmark? Perhaps half with a web browser to view the results, and half working blind with the numbers alone?
- roxolotl 11mo agoIf a human was capable of doing it blind I’d be absolutely blown away. Many can’t even draw a bicycle from memory[0]. 0: https://www.wired.com/2016/04/can-draw-bikes-memory-definitely-cant/ https://www.wired.com/2016/04/can-draw-bikes-memory-definite...
- ethmarks 11mo agoI highly doubt that any human could manually write a pelican SVG in a single shot like LLMs do. With a lot of guessing and checking of coordinate positions in the web browser, maybe they could do it. I imagine that you could theoretically also guess and check without the web browser by manually rendering the SVG using some graph paper, a compass, a straightedge, and coloured pencils, but that sounds unbelievably tedious and also very error-prone.
- ZeroConcerns 11mo ago> so I shrunk the file down to a more manageable 38MB using ffmpeg Without having an LLM figure out the required command line parameters? Mad props!
- simonw 11mo agoHah, nope! I had Claude Code figure that one out.
- leetharris 11mo agoI used to work in ASR. Due to the nature of current multimodal architectures, it is unlikely we'll ever see accurate timestamps over a longer horizon. You're better off using encoder-decoder ASR architectures, then using traditional diarization using embedding clustering, then using a multimodal model to refine it, then use a forced alignment technique (maybe even something pre-NN) to get proper timestamps and reconciling it at the end. These things are getting really good at just regular transcription (as long as you don't care about verbatimicity), but every additional dimension you add (timestamps, speaker assignment, etc) will make the others worse. These work much better as independent processes that then get reconciled and refined by a multimodal LLM.
- nurumaik 11mo agoSeems like pelican benchmark is finally added to model training process
- red75prime 11mo agoIt's easy to test. Use "one-eyed octopus on a tricycle" or something.
- ethmarks 11mo agoMission accomplished for Simon: > Truth be told, I’m playing the long game here. All I’ve ever wanted from life is a genuinely great SVG vector illustration of a pelican riding a bicycle. My dastardly multi-year plan is to trick multiple AI labs into investing vast resources to cheat at my benchmark until I get one. https://simonwillison.net/2025/Nov/13/training-for-pelicans-riding-bicycles/ https://simonwillison.net/2025/Nov/13/training-for-pelicans-...
- Wowfunhappy 11mo agoAww, I don’t like the new pelican benchmark as much. I liked that the old prompt was vague and we could see how the AI interpreted it.
- ahmedfromtunis 11mo agoYeah. The new challenge seems easier to solve since it basically is hand-holding the LLMs into what the result should look like. I think a more challenging, well, challenge, would be to offer an even more absurd scenario and see how the model handles it. Example: generate an svg of a pelican and a mongoose eating popcorn inside a pyramid-shaped vehicle flying around Jupiter. Result: https://imgur.com/a/TBGYChc https://imgur.com/a/TBGYChc
- simonw 11mo agoI like the hand-holding because it's a better test of how well models can follow more detailed instructions. I was inspired by Max Woolf's nano banana test prompts: https://minimaxir.com/2025/11/nano-banana-prompts/ https://minimaxir.com/2025/11/nano-banana-prompts/
- ahmedfromtunis 11mo agoThat's a valid point but I'd argue the new test would be then interesting to couple with the original one, not to replace it. Do you think it would be reasonable to include both in future reviews, at least for the sake of back-compatibility (and comparability)?
- simonw 11mo agoYeah I'm going to keep on using the old one as well.
- ethmarks 11mo agoWhich model did you use in the example result? It looks fantastic.
- razodactyl 11mo agoI was waiting for this post. Love the pivot in pelican generation bench.
- scosman 11mo agoIf anyone enjoys cheeky mischief targeting LLM benchmark hackers: https://github.com/scosman/pelicans_riding_bicycles https://github.com/scosman/pelicans_riding_bicycles I'll need to update for V2!
- yberreby 11mo agoDelightfully evil.
- vessenes 11mo agoI'd like a long bet on when Simon will add animation to the pelican SVG benchmark. I'm thinking late 2026.
- Redster 11mo agoSo Gemini 3 Pro dropped today, which happens to be the day I proofread a historical timeline I'm assisting a PhD with. I do one pass and then realize I should try Gemini 3 Pro on it. I give the same exact prompt to 3 Pro as Claude 4.5 Sonnet. 3 pro finds 25 real errors, no hallucinations. Claude finds 7 errors, but only 2 of those are unique to Claude. (Claude was better at "wait, that reference doesn't match the content! It should be $corrected_citation!). But Gemini's visual understanding was top notch. It's biggest flaw was that it saw words that wrapped as having extra spaces. But it also correctly caught a typo where a wrapped word was misspelled, so something about it seemed to fixate on those line breaks, I think.
- Redster 11mo agoA better test would be 2.5 Pro vs 3 Pro. Google just has been doing better at vision for a while.
- hwasiti 11mo ago@simonw Can you try the transcription job again? It seems it does not work anymore. I noticed that Gemini 3 Pro can no longer recognize the audio files I upload. It just gives me information from old chats or random stuff that isn’t relevant. It can’t even grasp the topic of the class anymore, it just spits out nonsense. Something has changed. Just few days ago with Gemini 2.5 Pro it worked just fine! It's not just me: https://www.reddit.com/r/GeminiAI/comments/1p0givt/gemini_25_pro_can_no_longer_analyze_audio_files/ https://www.reddit.com/r/GeminiAI/comments/1p0givt/gemini_25... But I am trying Gemini 3 pro web (Pro subscription). aistudio is still working fine.
- lostmsu 11mo agoVote for the new pelicans at https://pelicans.borg.games/ https://pelicans.borg.games/