3 ms·
I think what we'll eventually see is frontier models getting priced dramatically more expensive (or rate limited), and more people getting pickier about what th
by jjfoooo4 1y ago
I think what we'll eventually see is frontier models getting priced dramatically more expensive (or rate limited), and more people getting pickier about what they send to frontier models vs cheaper, less powerful ones. This is already happening to some extent, with Opus being opt-in and much more restricted than Sonnet within Claude Code.
An unknown to me: are the less powerful models cheaper to serve, proportional to how much less capable they are than frontier models? One possible explanation for why e.g. OpenAI was eager to retire GPT 4 is that those older models are still money losers.
- simonw 1y agoEverything I've seen makes me suspect that models have continually got more efficient to serve. The strongest evidence is that the models I can run on my own laptop got massively better over the last three years, despite me keeping the same M2 64GB machine without upgrading it. Compare original LLaMA from 2023 to gpt-oss-20b from this year - same hardware, huge difference. The next clue is the continuing drop in API prices - at least prior to the reasoning rush of the last few months. One more clue: o3. OpenAI's o3 had a 80% price drop a few months ago which I believe was due to them finding further efficiencies in serving that model at the same quality. My hunch is that there are still efficiencies to be wrung out here. I think we'll be able to tell if that's not holding if API prices stop falling over time.
- realz 1y agoIf developers and enterprises can host their own OSS/fine-tuned models, why will they pay Anthropic or OpenAI?
- simonw 1y agoBecause hosting a really GOOD model requires hardware that costs tens of thousands of dollars. It's much cheaper to timeshare that hardware with other users than to buy and run it yourself.
- realz 1y agoThat may be true for independent devs or startups. Larger companies have enough demand to justify a few A/H100s.
- jjfoooo4 1y agoWhy do you think OpenAI wanted to get rid of GPT-4 etc so aggressively? I suppose there's a distinction between new less capable models, I can see why those would be more efficient. But maybe the older frontier models are less efficient to serve?
- simonw 1y agoDefinitely less efficient to serve. They used to charge $60/million input tokens for GPT-3 Da Vinci. They charge $1.25/million for GPT-5. Plus I believe they have to keep each model in GPU memory to serve it, which means that any GPU serving an older model is unavailable to serve the newer ones.