4 ms·
There was an article on here a week or two ago on batch inference. Do you not think that batch inference gives at least a bit of a moat whereby unit costs fall
by joshuajooste05 1y ago
There was an article on here a week or two ago on batch inference.
Do you not think that batch inference gives at least a bit of a moat whereby unit costs fall with more prompts per unit of time, especially if models get more complicated and larger in the future?
- minimaxir 1y agoBatch inference is not exclusive to OpenAI.