3 ms·

It should be noted for some reason OpenAI prices GPT-4o-mini image requests at the same price as GPT-4o. I have a similar library but we found OpenAI has subtle

It should be noted for some reason OpenAI prices GPT-4o-mini image requests at the same price as GPT-4o. I have a similar library but we found OpenAI has subtle OCR inconsistencies with tables (numbers will be inaccurate). Gemini Flash, for all its faults, seems to do really well as a replacement while being significantly cheaper. Here’s our pricing comparison: *Gemini Pro* - $0.66 per 1k image inputs (batch) - $1.88 per text output (batch API, 1k tokens) - 395 pages per dollar *Gemini Flash* - $0.066 per 1k images (batch) - $0.53 per text output (batch API, 1k tokens) - 1693 pages per dollar *GPT-4o* - $1.91 per 1k images (batch) - $3.75 per text output (batch API, 1k tokens) - 177 pages per dollar *GPT-4o-mini* - $1.91 per 1k images (batch) - $0.30 per text output (batch API, 1k tokens) - 452 pages per dollar [1] https://community.openai.com/t/super-high-token-usage-with-gpt-4o-mini-and-image/873480 https://community.openai.com/t/super-high-token-usage-with-g... [2] https://github.com/Filimoa/open-parse https://github.com/Filimoa/open-parse

themanmaran 2y ago
Interesting. It didn't seem like gpt-4o-mini was priced the same as gpt-4o during our testing. We're relying on OpenAI usage page of course, which doesn't give as much request by request pricing. But we didn't see any huge usage spike after testing all weekend. For our testing we ran a 1000 page document set, all treated as images. We got to about 25M input / 0.4M output tokens for 1000 pages. Which would be a pretty noticeable difference based on the listed token prices. gpt-4o-mini => (24M/1M * $0.15) + (0.4M/1M * 0.60) = $4.10 gpt-4o => (24M/1M * $5.00) + (0.4M/1M * 15.00) = $126.00
- serjester 2y ago
  The pricing is strange because the same images will use up 30X more tokens with mini. They even show this in the pricing calculator. [1] https://openai.com/api/pricing/ https://openai.com/api/pricing/
  - elvennn 2y ago
    Indeed it does. But also the price for output tokens of the OCR is cheaper. So in total it's still much cheaper with gpt-4o-mini.
yfontana 2y ago
My testing was admitedly limited (but focused on cases where GPT-4o was struggling), but in my experience Claude 3.5 also produced fewer hallucinations than GPT when doing OCR with figures / tables recognition. It is of course quite pricey as well. However both GPT and Claude still struggle with documents that are very heavy on infographics, figures and tables.
raffraffraff 2y ago
That price compares favourably with AWS Textract. Has anyone compared their performance? Because a recent post about OCR had Textract at or near the top in terms of quality.
- aman2k4 2y ago
  I’m using AWS textract for scanning grocery receipts and i find it does it very well and fast. Can you say which performance metric you have in mind?
- ianhawes 2y ago
  Can you locate that post? In my own experience, Google Document AI has superior quality but I'm looking for something a bit more objective and scientific.

themanmaran 2y ago

Interesting. It didn't seem like gpt-4o-mini was priced the same as gpt-4o during our testing. We're relying on OpenAI usage page of course, which doesn't give as much request by request pricing. But we didn't see any huge usage spike after testing all weekend. For our testing we ran a 1000 page document set, all treated as images. We got to about 25M input / 0.4M output tokens for 1000 pages. Which would be a pretty noticeable difference based on the listed token prices. gpt-4o-mini => (24M/1M * $0.15) + (0.4M/1M * 0.60) = $4.10 gpt-4o => (24M/1M * $5.00) + (0.4M/1M * 15.00) = $126.00

serjester 2y ago
The pricing is strange because the same images will use up 30X more tokens with mini. They even show this in the pricing calculator. [1] https://openai.com/api/pricing/ https://openai.com/api/pricing/
- elvennn 2y ago
  Indeed it does. But also the price for output tokens of the OCR is cheaper. So in total it's still much cheaper with gpt-4o-mini.

yfontana 2y ago

My testing was admitedly limited (but focused on cases where GPT-4o was struggling), but in my experience Claude 3.5 also produced fewer hallucinations than GPT when doing OCR with figures / tables recognition. It is of course quite pricey as well. However both GPT and Claude still struggle with documents that are very heavy on infographics, figures and tables.

raffraffraff 2y ago

That price compares favourably with AWS Textract. Has anyone compared their performance? Because a recent post about OCR had Textract at or near the top in terms of quality.

aman2k4 2y ago
I’m using AWS textract for scanning grocery receipts and i find it does it very well and fast. Can you say which performance metric you have in mind?
ianhawes 2y ago
Can you locate that post? In my own experience, Google Document AI has superior quality but I'm looking for something a bit more objective and scientific.