4 ms·
I really can’t stand when writers point to the difference in price per token on the api and subscription and use that as evidence that inference loses money. Th
by HDThoreaun 4mo ago
I really can’t stand when writers point to the difference in price per token on the api and subscription and use that as evidence that inference loses money. This author even says it’s implausible that the api charges 4x marginal cost when I think it’s very likely even higher than that. The entire rest of the post sits on this faulty assumption. Fixed costs don’t matter when marginal revenue is profitable and growing rapidly. The ai labs only have 2 questions. Can they prevent users from switching to open source models? Can they scale the number of users on enterprise plans the way they did for coding but in a more general way for all knowledge jobs?
- bcjdjsndon 4mo ago> Can they scale the number of users on enterprise plans the way they did for coding but in a more general way for all knowledge jobs? Do these knowledge jobs have a significant corpus of not only knowledge but discussion and problem solving, all conveniently labelled for the AI to train on? Probably not. Coding has stack overflow, what does, say, advertising use?
- warkdarrior 4mo ago> Coding has stack overflow, what does, say, advertising use? Advertising has centuries of print ads, 100 years of radio advertising, 70 years of TV commercials, etc. And modern AI does not necessarily need labeling.
- bcjdjsndon 4mo agoMy suspicion is coding agents didnt just train on source code but also on human discussions from message boards. There is no equivalent source for this data in advertising... That I know of. Any large scale question and answers boards pertaining to advertising, that have pretty much covered every common query of the past twenty years? If that exists, then I'm wrong.
- HDThoreaun 4mo agoI agree this is a hard problem for the labs. I would be hesitant about “probably not” though. There is just as much marketing copy floating around as there is coding training data. I struggle a bit in this question because I’ve only ever worked as a software engineer, so I can’t exactly make claims about all the work other jobs do. But, one example is I was talking to a doctor friend of mine the other day. He was talking about how he had to take his recertification exam recently and put the questions into chatGPT and thought it gave answers that were generally more thoughtful and correct than his own. Does that mean doctors are done? Of course not, but he’s now pushing hard for more ai tool use in his practice.
- jimbokun 4mo agoThen what are the real costs?
- martinald 4mo agoWrote this a while back. https://martinalderson.com/posts/no-it-doesnt-cost-anthropic-5k-per-claude-code-user/ https://martinalderson.com/posts/no-it-doesnt-cost-anthropic... OpenRouter is the best guide to real costs.
- jimbokun 4mo agoThanks, that’s exactly what I was looking for! And much more informative than the speculation and guessing in the article.
- anthonypasq 4mo agoagreed, this doesnt even account for prompt caching or the fact that anthropic has substantial proprietary efficiencies on their inference stack specific to their models and scale.