4 ms·
So, bottom line, do you think it’s probable that either OpenAI or Anthropic are “losing money on inference?”
by Den_VR 1y ago
So, bottom line, do you think it’s probable that either OpenAI or Anthropic are “losing money on inference?”
- chillee 1y agoNo. In some sense, the article comes to the right conclusion haha. But it's probably >100x off on its central premise about output tokens costing more than input.
- doctorpangloss 1y agoI’m pretty sure input tokens are cheap because they want to ingest the data for training later no? They want huge contexts to slice up.
- awwaiid 1y agoAfaik all the large providers flipped the default to contractually NOT train on your data. So no, training data context size is not a factor.
- martinald 1y agoThanks for the correction (author here). I'll update the article - very fair point on compute on input tokens which I messed up. Tbh I'm pleased my napkin math was only 7x off the laws of physics :). Even rerunning the math on my use cases with way higher input token cost doesn't change much though.
- chillee 1y agoThe 32 parallel sequences is also arbitrary and significantly changes your conclusions. For example, if they run with 256 parallel sequences then that would result in a 8x cheaper factor in your calculations for both prefill and decode. The component about requiring long context lengths to be compute-bound for attention is also quite misleading.
- Barbing 1y agoAnyone up to publishing their own guess range?
- diamond559 1y agoEven if it is, ignoring the biggest costs going into the product and then claiming they are profitable would be actual fraud.