3 ms·
For those who aren't aware, OpenAI has a very similar batch mode (50% discount if you wait up to 24 hours): https://platform.openai.com/docs/api-reference/batch
by tripplyons 1y ago
For those who aren't aware, OpenAI has a very similar batch mode (50% discount if you wait up to 24 hours): https://platform.openai.com/docs/api-reference/batch https://platform.openai.com/docs/api-reference/batch
It's nice to see competition in this space. AI is getting cheaper and cheaper!
- fantispug 1y agoYes, this seems to be a common capability - Anthropic and Mistral have something very similar as do resellers like AWS Bedrock. I guess it lets them better utilise their hardware in quiet times throughout the day. It's interesting they all picked 50% discount.
- qrian 1y agoBedrock has a batch mode but only for claude 3.5 which is like one year old, which isn't very useful.
- calaphos 1y agoInference throughout scales really well with larger batch sizes (at the cost of latency) due to rising arithmetic intensity and the fact that it's almost always memory BW limited.
- briangriffinfan 1y ago50% is my personal threshold for a discount going from not worth it to worth it.
- bayesianbot 1y agoDeepSeek has gone a bit different route - they give automatic 75% discount between UTC 16:30-00:30 https://api-docs.deepseek.com/quick_start/pricing https://api-docs.deepseek.com/quick_start/pricing
- dlvhdr 1y agoThe latest price increases beg to differ
- dmos62 1y agoWhat price increases?
- rvnx 1y agoI guess the Gemini price increase
- dmos62 1y agoAh, 2.5 flash non-thinking price was increased to match the price of 2.5 flash thinking.
- Workaccount2 1y agoNo, 2.5 flash non-thinking was replaced with 2.5 flash lite, and 2.5 flash thinking had it's cost rebalanced (input price increased/output price decreased) 2.5 flash non-thinking doesn't exist anymore. People call it a price increase but it's just confusion about what Google did.
- sunaookami 1y agoThey try to frame it as such but 2.5 Flash Lite is not the same as 2.5 Flash without thinking. It's worse.
- dist-epoch 1y agoOnly because Flash was mispriced to start with. It was set too cheap compared with its capabilities. They didn't raise the price of Pro.
- laborcontract 1y agoOne open secret is that batch mode generations often take much less than 24 hours. I've done a lot of generations where I get my results within 5ish minutes.
- ridgewell 1y agoIt can depend a lot on the shape of your batch to my understanding. A small batch job can be tasked out a lot quicker than a large batch job waiting for just the right moment where capacity fits.