3 ms·
I’ll give it a shot and update the results. Unfortunately the bigger issue with Gemini is cost, which is too high for the scraping use case.
by marcell 2y ago
I’ll give it a shot and update the results.
Unfortunately the bigger issue with Gemini is cost, which is too high for the scraping use case.
- Shakahs 2y agoGemini Flash API costs 50% less than gpt-4o-mini. It's not clear why your benchmark recorded 13x more token usage for Flash for the same input.
- faangguyindia 2y agoIf you are based in US, you get 1 billion tokens per day for free from Gemini.
- deleted 2y ago[deleted]
- marcell 2y agoI checked the code and found the issue. It's a result of Gemini's larger context window. Basically, the Foxtrot scraping library sends the page in chunks. The chunk size is capped at the max context length of each model, which for Gemini is 1,000,000 input tokens for lite. That's compared to 128,000 for GPT-4o-mini. Typically, you won't need all the tokens in the page, and sending a million tokens when 100,000 will work is wasteful in terms of cost and runtime, and can also hurt accuracy. I'm going to re-run the benchmarks with a cap on the prompt size for models like Gemini.