5 ms·

This makes me wonder if AI companies even have a MOAT in the first place. All requests to an LLM are idempotent, for every API call you need to send it the ent

by me551ah 2mo ago

This makes me wonder if AI companies even have a MOAT in the first place. All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history so that it can process it. LLMs do not learn or remember anything, which makes it super easy for users to switch LLMs on the fly. Most popular AI frameworks, make this a one-liner change these days. And that makes me wonder if the trillion dollar valuations for OpenAI and Claude are even justified. Cause if that is justified, then Kimi, Qwen, Deepseek etc are also valued at a trillion dollars. Or all of them are worth a lot less. One of those statements is true. Also this makes me wonder if the next iteration of LLMs would be based on fine-tuning, where LLMs actually learn from your past behaviour so that it would grant some amount of stickiness to the product. OpenAI used to offer fine tuning runs for GPT-3.5, but they don't seem to do that anymore.

regularfry 2mo ago
With their current API approach they're essentially a commodity. They need to start moving parts of the harness behind the API, otherwise they'll remain a commodity. Recursive self-improvement changes the parameters a bit, especially for the market-leaders, and it's the one thing that makes me wonder if they'll be able to extend their lead faster than the smaller labs can keep up, but it's an option available to everyone.
- me551ah 2mo ago
  > They need to start moving parts of the harness behind the API This isn't without it's challenges however. 1. This will increase costs drastically, since they would need to a run a sandbox per use to ensure data isolation. 2. Increased latency, and this directly limits how much of the harness can be moved to the cloud before the users notice sluggishness
  - regularfry 2mo ago
    Oh, I'm sure. If it wasn't a bit of a tricky needle to thread we'd have seen them start to do it already.
  - owebmaster 2mo ago
    3. Users can still move to better open harnesses
jcoder 2mo ago
> All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history A more appropriate term is “stateless”. LLM responses are certainly not idempotent, as they are not even deterministic.
- msdz 2mo ago
  Which is why big labs have been working hard on making their harness not be stateless any longer: https://earendil.com/posts/session-portability/ https://earendil.com/posts/session-portability/ “Just take the session thread to another provider” might not be feasible anymore soon-ish.
  - jerf 2mo ago
    While that particular API might be nice, and people and companies should probably push back against the obfuscation, in the end it doesn't really matter. When I hand off between different models I already have the first model prepare a markdown file for the second rather than just importing the entire original thread wholesale, because that's expensive anyhow, and also rather unfocused. They can't get their models to stop generating that sort of checkpoint because that's a fundamental operation necessary for all the harnesses to work anyhow. The fundamental technology of LLMs and arguably AI in general strongly cuts against that sort of lockin. Handoff is a fundamental capability. There's no option to encrypt the docs or write it in some dialect only one model understands because humans need to understand it to, which stops that whole line dead in its tracks for at least the forseeable future. An AI can already today pick up such pieces, how much more easily will they do it tomorrow? If they want to lock me in, they're going to need to provide a feature that I need so badly I can't switch and nobody else has. It is hard to see what that would be, other than being a generally better model.
    - msdz 2mo ago
      Soooo… when can we expect an encrypted handoff.md to fully prevent session portability, then? (Only half-/s)
  - 2mo ago
chakintosh 2mo ago
> This makes me wonder if AI companies even have a MOAT in the first place. They don't. The moat will mainly be the tooling around AI, not the AI itself. You don't hear any company claiming their moat is the Internet.
ryanjshaw 2mo ago
> Cause if that is justified, then Kimi, Qwen, Deepseek etc are also valued at a trillion dollars It’s more like a bunch of people are placing different bets. Only a few bets are going to generate a return, possibly only one, but the profit on that one bet will make it all worthwhile. That’s the theory, anyway.
- owebmaster 2mo ago
  There's no mandate that says any of those bets are going to generate a return
torginus 2mo ago
> This makes me wonder if AI companies even have a MOAT in the first place. Generally speaking they do, at least from my experience when switching from one model to the other - their performance decreases, and they often do large refactors outside of the requested scope as they try to bring the code closer to 'their' style. Which makes sense imo - they'v been trained to iterate over the code they wrote, and not code that was modified by someone else in the interim.
satvikpendem 2mo ago
Google figured this out with their paper from 2023, We have no moat and neither does OpenAI. The moat now is the harness and being able to recursively self improve from RLHF, a great example is how Grok used to be pretty bad but since SpaceX bought Cursor, they used that data to train Grok 4.5 which is now very competent at coding and even exceeds frontier models in certain benchmarks. https://www.semianalysis.com/p/google-we-have-no-moat-and-neither https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
- gizmodo59 2mo ago
  Moat is not the harness. Harness itself is temporary until the models get better and slowly the code in harness will go down. Note that the biggest GPU providers in the world are the hyper scalers and even they couldn’t allocate more if you pay for it. Because the rich companies and well funded ones are gobbling them up to the point where if tomorrow a 5T model that smokes every other model in the world is released you just can’t afford inference.
  - vcryan 2mo ago
    Agree. Harness can not be a moat. There are many open harnesses and they are at least on par with the providers ones. It looks like Anthropic/OpenAI's approach to vendor lock-in is not so much the inference or the harness it is functional integration across the individuals and teams in a company. I don't think this will be a moat either, but I think it's all they have outside of compute.
  - satvikpendem 2mo ago
    I meant that companies like Anthropic are locking in users with proprietary formats in their harness where it's hard to leave.
- ListeningPie 2mo ago
  And yet investment is continuing. What are they counting on?
  - 2mo ago
gcr 2mo ago
This isn’t technically true. Most model providers don’t send the thinking tokens anymore, so if you switch from one provider to another, you will be missing large parts of the conversation.
credit_guy 2mo ago
They have 2 moats. The first is the compute. OpenAI and Anthropic secured huge amounts of compute, Google, Meta and xAI have their own huge datacenters. Now anyone can rent some cloud machines and start serving Kimi K3, but it's going to be impossible to get to a similar scale as the big 5 above. And inference has economies of scale: the more people you serve in parallel, the more efficient you are. The second is the data. By now (and maybe even by one year ago), all the data on the internet has been used for training. You need new data. The big AI companies sit on top of trillions or quadrillions of tokens that they have generated over the years. They can use that to train new models. That data is gold, and the proof is that SpaceX was happy to pay $60B to acquire Cursor. If you want to overtake the frontier labs, you have 2 options: use their models to generate synthetic data, and provide lots of (cheap, maybe below cost) inference to generate your own new data. The frontier labs know about the first, and I'm sure they try to limit how much others milk their models. As for the second, that's the "honest" way to compete, but it's not easy.
- owebmaster 2mo ago
  your post helped me realize a change Meta is pursuing on Instagram that is to give more weight to captions and long text posts so they can have more data to training that would usually go to websites/Google. Even AI slop is good for this.
- efficax 2mo ago
  compute is not a moat, it's a rapidly depreciating physical asset. buying up all the shovels in a gold rush does not give you a moat, it gives you a slight advantage for the time being. someone else will just start making shovels. and the data is clearly available, hence the number of open-weight models.
  - nemonemo 2mo ago
    Isn't a literal "moat" about temporary deterrence? I can imagine makeshift bridges could permanently make the moat useless.
    - 2mo ago
varispeed 2mo ago
From my experience these open source models are nowhere near the performance offered by Fable/Opus/GPT-5.6. Whenever I tried Qwen, Kimi, Deepseek, the results were much worse and it just took much more time to get something usable. When you consider that, the frontier offerings are still much cheaper.
- w4yai 2mo ago
  What provider did you use ? Synthetic's Kimi is a beast
- mtrovo 2mo ago
  That might be true right now, but how long until you have to move the goalposts? In my experience with DeepSeek and Kimi, they're as capable as the frontier was four months ago, which already solves a big chunk of the coding tasks that I'm interested in.
dangoodmanUT 2mo ago
The major labs don’t allow assistant prefill, so you have to “summarize”
randusername 2mo ago
Burdensome regulatory compliance is a moat. These companies have AI and enough money to lobby the Pope. They can afford to reanimate members of congress and push some tactical legislation through. But all the money in the world cannot move government too quickly. Other moats exist too. OS or browser can undermine performance and availability of alternatives.
efficax 2mo ago
fine tuning runs of models the size of gpt 5.6 are absurdly expensive. I'd guess at least $100k in cloud gpu time for a single run, and you have to do a few iterations to get things right
lenerdenator 2mo ago
They have a moat; they don't have $1 trillion valuations. Which anyone who hasn't been sitting in the SV echo chamber could have told you years ago after applying even the smallest bit of thought.
tootie 2mo ago
I think you're right and I think it's why Google have taken their pedal off the metal for model releases to focus on integrations and tools. And why Microsoft have backed off from the OpenAI partnership to do the same. Anthropic and OpenAI are going to massively struggle to maintain their pace and reach profitability just selling commodity tokens. Fine tunes are a possibility but I think it offers very little uplift for the vast majority of uses beyond just stuffing enough context.
clbrmbr 2mo ago
Mamba/SSMs could change this picture.
turing_complete 2mo ago
The moat is the US government.
notnullorvoid 2mo ago
> that makes me wonder if the trillion dollar valuations for OpenAI and Claude are even justified. They aren't, not even if we forget about the capable Chinese models. I suspect Anthropic will implode soon when employees are unable to get the cash-out that they expected. Having so much compensation locked up in company stock is risky on a good day.
crossroadsguy 2mo ago
China has the moat that they are cheap/free/open. The US corps have the moat that the other option is Chinese models. At least for some time.
kertoip_1 2mo ago
The actual moat is the same as in web services - data and user base. Why is Google a monopoly? Do they have so advanced software that no one can outperform? I doubt it. What they have is a giant user base that generate loads of real-time data, which make Google services more accurate. So how AI company can build a moat? Exactly the same way: by making a giant user base produce loads of real time data. Just imagine a service that will generate answers not only based on data they were trained on, but on all data from all user conversations. Imagine being at a concert, looking for a certain type of beer and instantly receiving an answer from an AI assistant about that only because some other guy in a crowd looking for exactly the same thing said to his agent "ah, here they are!". It is not happening just yet because of making it secure and private is not yet solved, but it's just a matter of time I think.
jamesrr39 2mo ago
I think Anthropic/OpenAI do have a moat in the (western) enterprise market. Chinese hosted models are a no-go, and my experience with enterprise IT departments is that they will not self-host. So far signing up with a known product (e.g. Claude) seems to be the way they will go and this is the moat that the AI companies have. Alternatively there is Copilot, but that for now seems to mostly be backed by Anthropic/OpenAI models[1]. Will this continue? The field is moving too fast to tell. Kimi, Qwen, Deepseek also produce very capable models but that doesn't automatically translate into trillion dollar valuations. However, trillion dollar valuations on Anthropic and OpenAI, such new companies, never publicly traded and such huge valuations decided just by investors. This is just asking for trouble. 1. https://docs.github.com/en/copilot/reference/ai-models/supported-models https://docs.github.com/en/copilot/reference/ai-models/suppo...

by me551ah 2mo ago

regularfry 2mo ago

With their current API approach they're essentially a commodity. They need to start moving parts of the harness behind the API, otherwise they'll remain a commodity. Recursive self-improvement changes the parameters a bit, especially for the market-leaders, and it's the one thing that makes me wonder if they'll be able to extend their lead faster than the smaller labs can keep up, but it's an option available to everyone.

me551ah 2mo ago
> They need to start moving parts of the harness behind the API This isn't without it's challenges however. 1. This will increase costs drastically, since they would need to a run a sandbox per use to ensure data isolation. 2. Increased latency, and this directly limits how much of the harness can be moved to the cloud before the users notice sluggishness
- regularfry 2mo ago
  Oh, I'm sure. If it wasn't a bit of a tricky needle to thread we'd have seen them start to do it already.
- owebmaster 2mo ago
  3. Users can still move to better open harnesses

jcoder 2mo ago

> All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history A more appropriate term is “stateless”. LLM responses are certainly not idempotent, as they are not even deterministic.

msdz 2mo ago
Which is why big labs have been working hard on making their harness not be stateless any longer: https://earendil.com/posts/session-portability/ https://earendil.com/posts/session-portability/ “Just take the session thread to another provider” might not be feasible anymore soon-ish.
- jerf 2mo ago
  While that particular API might be nice, and people and companies should probably push back against the obfuscation, in the end it doesn't really matter. When I hand off between different models I already have the first model prepare a markdown file for the second rather than just importing the entire original thread wholesale, because that's expensive anyhow, and also rather unfocused. They can't get their models to stop generating that sort of checkpoint because that's a fundamental operation necessary for all the harnesses to work anyhow. The fundamental technology of LLMs and arguably AI in general strongly cuts against that sort of lockin. Handoff is a fundamental capability. There's no option to encrypt the docs or write it in some dialect only one model understands because humans need to understand it to, which stops that whole line dead in its tracks for at least the forseeable future. An AI can already today pick up such pieces, how much more easily will they do it tomorrow? If they want to lock me in, they're going to need to provide a feature that I need so badly I can't switch and nobody else has. It is hard to see what that would be, other than being a generally better model.
  - msdz 2mo ago
    Soooo… when can we expect an encrypted handoff.md to fully prevent session portability, then? (Only half-/s)
- 2mo ago

chakintosh 2mo ago

> This makes me wonder if AI companies even have a MOAT in the first place. They don't. The moat will mainly be the tooling around AI, not the AI itself. You don't hear any company claiming their moat is the Internet.

ryanjshaw 2mo ago

> Cause if that is justified, then Kimi, Qwen, Deepseek etc are also valued at a trillion dollars It’s more like a bunch of people are placing different bets. Only a few bets are going to generate a return, possibly only one, but the profit on that one bet will make it all worthwhile. That’s the theory, anyway.

owebmaster 2mo ago
There's no mandate that says any of those bets are going to generate a return

torginus 2mo ago

> This makes me wonder if AI companies even have a MOAT in the first place. Generally speaking they do, at least from my experience when switching from one model to the other - their performance decreases, and they often do large refactors outside of the requested scope as they try to bring the code closer to 'their' style. Which makes sense imo - they'v been trained to iterate over the code they wrote, and not code that was modified by someone else in the interim.

satvikpendem 2mo ago

Google figured this out with their paper from 2023, We have no moat and neither does OpenAI. The moat now is the harness and being able to recursively self improve from RLHF, a great example is how Grok used to be pretty bad but since SpaceX bought Cursor, they used that data to train Grok 4.5 which is now very competent at coding and even exceeds frontier models in certain benchmarks. https://www.semianalysis.com/p/google-we-have-no-moat-and-neither https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...

gizmodo59 2mo ago
Moat is not the harness. Harness itself is temporary until the models get better and slowly the code in harness will go down. Note that the biggest GPU providers in the world are the hyper scalers and even they couldn’t allocate more if you pay for it. Because the rich companies and well funded ones are gobbling them up to the point where if tomorrow a 5T model that smokes every other model in the world is released you just can’t afford inference.
- vcryan 2mo ago
  Agree. Harness can not be a moat. There are many open harnesses and they are at least on par with the providers ones. It looks like Anthropic/OpenAI's approach to vendor lock-in is not so much the inference or the harness it is functional integration across the individuals and teams in a company. I don't think this will be a moat either, but I think it's all they have outside of compute.
- satvikpendem 2mo ago
  I meant that companies like Anthropic are locking in users with proprietary formats in their harness where it's hard to leave.
ListeningPie 2mo ago
And yet investment is continuing. What are they counting on?
- 2mo ago

gcr 2mo ago

This isn’t technically true. Most model providers don’t send the thinking tokens anymore, so if you switch from one provider to another, you will be missing large parts of the conversation.

credit_guy 2mo ago

They have 2 moats. The first is the compute. OpenAI and Anthropic secured huge amounts of compute, Google, Meta and xAI have their own huge datacenters. Now anyone can rent some cloud machines and start serving Kimi K3, but it's going to be impossible to get to a similar scale as the big 5 above. And inference has economies of scale: the more people you serve in parallel, the more efficient you are. The second is the data. By now (and maybe even by one year ago), all the data on the internet has been used for training. You need new data. The big AI companies sit on top of trillions or quadrillions of tokens that they have generated over the years. They can use that to train new models. That data is gold, and the proof is that SpaceX was happy to pay $60B to acquire Cursor. If you want to overtake the frontier labs, you have 2 options: use their models to generate synthetic data, and provide lots of (cheap, maybe below cost) inference to generate your own new data. The frontier labs know about the first, and I'm sure they try to limit how much others milk their models. As for the second, that's the "honest" way to compete, but it's not easy.

owebmaster 2mo ago
your post helped me realize a change Meta is pursuing on Instagram that is to give more weight to captions and long text posts so they can have more data to training that would usually go to websites/Google. Even AI slop is good for this.
efficax 2mo ago
compute is not a moat, it's a rapidly depreciating physical asset. buying up all the shovels in a gold rush does not give you a moat, it gives you a slight advantage for the time being. someone else will just start making shovels. and the data is clearly available, hence the number of open-weight models.
- nemonemo 2mo ago
  Isn't a literal "moat" about temporary deterrence? I can imagine makeshift bridges could permanently make the moat useless.
  - 2mo ago

varispeed 2mo ago

From my experience these open source models are nowhere near the performance offered by Fable/Opus/GPT-5.6. Whenever I tried Qwen, Kimi, Deepseek, the results were much worse and it just took much more time to get something usable. When you consider that, the frontier offerings are still much cheaper.

w4yai 2mo ago
What provider did you use ? Synthetic's Kimi is a beast
mtrovo 2mo ago
That might be true right now, but how long until you have to move the goalposts? In my experience with DeepSeek and Kimi, they're as capable as the frontier was four months ago, which already solves a big chunk of the coding tasks that I'm interested in.

dangoodmanUT 2mo ago

The major labs don’t allow assistant prefill, so you have to “summarize”

randusername 2mo ago

Burdensome regulatory compliance is a moat. These companies have AI and enough money to lobby the Pope. They can afford to reanimate members of congress and push some tactical legislation through. But all the money in the world cannot move government too quickly. Other moats exist too. OS or browser can undermine performance and availability of alternatives.

efficax 2mo ago

fine tuning runs of models the size of gpt 5.6 are absurdly expensive. I'd guess at least $100k in cloud gpu time for a single run, and you have to do a few iterations to get things right

lenerdenator 2mo ago

They have a moat; they don't have $1 trillion valuations. Which anyone who hasn't been sitting in the SV echo chamber could have told you years ago after applying even the smallest bit of thought.

tootie 2mo ago

I think you're right and I think it's why Google have taken their pedal off the metal for model releases to focus on integrations and tools. And why Microsoft have backed off from the OpenAI partnership to do the same. Anthropic and OpenAI are going to massively struggle to maintain their pace and reach profitability just selling commodity tokens. Fine tunes are a possibility but I think it offers very little uplift for the vast majority of uses beyond just stuffing enough context.

clbrmbr 2mo ago

Mamba/SSMs could change this picture.

turing_complete 2mo ago

The moat is the US government.

notnullorvoid 2mo ago

> that makes me wonder if the trillion dollar valuations for OpenAI and Claude are even justified. They aren't, not even if we forget about the capable Chinese models. I suspect Anthropic will implode soon when employees are unable to get the cash-out that they expected. Having so much compensation locked up in company stock is risky on a good day.

crossroadsguy 2mo ago

China has the moat that they are cheap/free/open. The US corps have the moat that the other option is Chinese models. At least for some time.

kertoip_1 2mo ago

The actual moat is the same as in web services - data and user base. Why is Google a monopoly? Do they have so advanced software that no one can outperform? I doubt it. What they have is a giant user base that generate loads of real-time data, which make Google services more accurate. So how AI company can build a moat? Exactly the same way: by making a giant user base produce loads of real time data. Just imagine a service that will generate answers not only based on data they were trained on, but on all data from all user conversations. Imagine being at a concert, looking for a certain type of beer and instantly receiving an answer from an AI assistant about that only because some other guy in a crowd looking for exactly the same thing said to his agent "ah, here they are!". It is not happening just yet because of making it secure and private is not yet solved, but it's just a matter of time I think.

jamesrr39 2mo ago

I think Anthropic/OpenAI do have a moat in the (western) enterprise market. Chinese hosted models are a no-go, and my experience with enterprise IT departments is that they will not self-host. So far signing up with a known product (e.g. Claude) seems to be the way they will go and this is the moat that the AI companies have. Alternatively there is Copilot, but that for now seems to mostly be backed by Anthropic/OpenAI models[1]. Will this continue? The field is moving too fast to tell. Kimi, Qwen, Deepseek also produce very capable models but that doesn't automatically translate into trillion dollar valuations. However, trillion dollar valuations on Anthropic and OpenAI, such new companies, never publicly traded and such huge valuations decided just by investors. This is just asking for trouble. 1. https://docs.github.com/en/copilot/reference/ai-models/supported-models https://docs.github.com/en/copilot/reference/ai-models/suppo...