3 ms·

Very insightful. I think the payment model would have worked out just fine if the state of the art was the optimization of GPT-4 class models to bring down the

Very insightful. I think the payment model would have worked out just fine if the state of the art was the optimization of GPT-4 class models to bring down the cost over time, which would have made the services profitable eventually. Instead, newer models are getting larger and more resource heavy through reasoning, meaning costs per request are going up instead of down.

avianlyric 1y ago
I think we’ll start seeing people focus on optimisation, we already see companies like Apple focus on it. LLM are still to new, and still advancing to quickly for optimisation to take place. It’s like we’re back in the MHz wars of old between CPU manufacturers. The goal is just more performance, regardless of cost, because it was clear that even in the consumer space, people wanted more performance. Then we hit a kind of plateau in last 10 years, where basic compute is so powerful that your average consumer is not longer upgrading every year for better performance. A 5 year old machine has enough performance for most people. Then the focus on energy efficiency kicked in, because people didn’t want faster computers, they wanted battery life and cheaper computers. No doubt we’ll see the same with LLM, possibly quite soon. Claude Sonnet 4 and similar class models have enough reasoning performance, that agentic systems can be quite reliable. Which means we hit the base level of “reasoning” performance needed, and we can extend that “performance” in domain specific ways by lightly customising the agentic framework, with no need to fine tuning. The elimination of fine tuning to build domain specific agents is a huge game changer. But it also means that putting together a 10x or 100x efficient model, with “reasoning” performance equivalent to current gen LLM would also be a huge game changer. It opens up the possibility to apply this tech into spaces that currently require either lots of specialists knowledge to fine tune an LLM, or a huge amount of on tap compute to allow the agents to take enough turns to slowly “reason” they’re way through problems. But a Claude Sonnet 4 that runs on a iPhone for example. That would make Apple’s complete failure to improve Siri look like a genius level move. Why bother with small incremental improvements using current tech, when waiting a few years, and just stuffing a full fat LLM and agent system into an iPhone will basically give you the ultimate Siri.
- tom_m 1y ago
  Supply and demand says otherwise. I think costs are actually going to go up. Right now they are trying to make sure the AI you run on your phone can't be used for coding or other tasks. They want to keep it expensive. They have no incentive to make it more efficient. No need. The models aren't designed for efficiency either. There was a little scare with DeepSeek, but it turned out to be nothing. At the end of the day you need a lot of hardware and juice to run these things at scale. The costs aren't coming down any time soon. Nor is the impact to the environment unfortunately either.
  - willseth 1y ago
    Who is they? Even if Anthropic doesn’t have incentive to run their models on your phone (but really why wouldn’t they?), Apple, Google, and Samsung do. And DeepSeek was very much a something. It was a something that demonstrated the barrier to entry can be at least an order of magnitude lower than previously thought. LLM technology is easily commoditized, and moat building will be difficult. Supply and demand is doing just what you would expect.
- bobbob27 1y ago
  I truly believe you're right on that. Apple seems to be the most visible example of a company using LLMs in a way that will make long-term sense. The most logical future of AI is strictly on-device, capable enough to understand the user and making sense of the algorithmic-anarchy that is our phone. THAT is something exciting and practical, that doesn't require magical-thinking to desire.

avianlyric 1y ago

I think we’ll start seeing people focus on optimisation, we already see companies like Apple focus on it. LLM are still to new, and still advancing to quickly for optimisation to take place. It’s like we’re back in the MHz wars of old between CPU manufacturers. The goal is just more performance, regardless of cost, because it was clear that even in the consumer space, people wanted more performance. Then we hit a kind of plateau in last 10 years, where basic compute is so powerful that your average consumer is not longer upgrading every year for better performance. A 5 year old machine has enough performance for most people. Then the focus on energy efficiency kicked in, because people didn’t want faster computers, they wanted battery life and cheaper computers. No doubt we’ll see the same with LLM, possibly quite soon. Claude Sonnet 4 and similar class models have enough reasoning performance, that agentic systems can be quite reliable. Which means we hit the base level of “reasoning” performance needed, and we can extend that “performance” in domain specific ways by lightly customising the agentic framework, with no need to fine tuning. The elimination of fine tuning to build domain specific agents is a huge game changer. But it also means that putting together a 10x or 100x efficient model, with “reasoning” performance equivalent to current gen LLM would also be a huge game changer. It opens up the possibility to apply this tech into spaces that currently require either lots of specialists knowledge to fine tune an LLM, or a huge amount of on tap compute to allow the agents to take enough turns to slowly “reason” they’re way through problems. But a Claude Sonnet 4 that runs on a iPhone for example. That would make Apple’s complete failure to improve Siri look like a genius level move. Why bother with small incremental improvements using current tech, when waiting a few years, and just stuffing a full fat LLM and agent system into an iPhone will basically give you the ultimate Siri.

tom_m 1y ago
Supply and demand says otherwise. I think costs are actually going to go up. Right now they are trying to make sure the AI you run on your phone can't be used for coding or other tasks. They want to keep it expensive. They have no incentive to make it more efficient. No need. The models aren't designed for efficiency either. There was a little scare with DeepSeek, but it turned out to be nothing. At the end of the day you need a lot of hardware and juice to run these things at scale. The costs aren't coming down any time soon. Nor is the impact to the environment unfortunately either.
- willseth 1y ago
  Who is they? Even if Anthropic doesn’t have incentive to run their models on your phone (but really why wouldn’t they?), Apple, Google, and Samsung do. And DeepSeek was very much a something. It was a something that demonstrated the barrier to entry can be at least an order of magnitude lower than previously thought. LLM technology is easily commoditized, and moat building will be difficult. Supply and demand is doing just what you would expect.
bobbob27 1y ago
I truly believe you're right on that. Apple seems to be the most visible example of a company using LLMs in a way that will make long-term sense. The most logical future of AI is strictly on-device, capable enough to understand the user and making sense of the algorithmic-anarchy that is our phone. THAT is something exciting and practical, that doesn't require magical-thinking to desire.