6 ms·

The moat looks deep today but it's going to become more shallow every year. Training a new model from scratch takes serious resources. Post-training/fine-tunin

The moat looks deep today but it's going to become more shallow every year. Training a new model from scratch takes serious resources. Post-training/fine-tuning an existing model, dramatically less. The knowledge for the process was esoteric two years ago, now you can ask a current model (one of several) to walk you through it, while building the tools to do it as you go. Several of my recent weekend projects have been exactly that sort of thing, just so I understand it better. "Let's make a LoRA", "let's generate a corpus of training data for fine-tuning a model for X task", "how can I put my face in a text-to-image model?" stuff like that. All of this is do-able on kinda modest local hardware (a couple of old GPUs or a Strix Halo or DGX Spark or big Mac Studio), or for a few bucks or a few hundred bucks or a few thousand bucks of cloud compute, depending on scale. Scale that up to corporate or startup scale, with the money that's been flowing into AI for the past couple/few years, and it's obviously there's going to be a lot of competition just as the top model makers need to start ringing the cash register. That's a lot of opportunities for people to look at their ballooning Claude usage costs and find other ways to do the same thing for drastically less money. $100/month or $200/month is a no-brainer for Claude Code with probably the best model for coding, but they're pushing more users to usage-based billing which becomes cost-prohibitive real fast. So, they desperately need to continue to be among the only ways to solve the hardest problems, and they need the alternatives to cost a similar amount. They can count on OpenAI and Google to ratchet up prices, too. They probably can't count on everybody, especially the vendors in China with different economics, to do it. And, they can't count on companies to look at their own usage and not ask, "Can we train a smaller specialist model that does this one thing we're using the Anthropic API most heavily for?" I'm hoping they just mean stuff like using Claude for distillation by e.g. Chinese model makers, and not "how do I fine-tune Gemma 4 to write more like me?" or whatever.

hedora 5mo ago
What moat? There are multiple companies providing pareto-optimal frontier models, and it takes O(10) people to build one of these things. The rest is capital intensive, and the price will approach the cost of production over time. Thinking this is a profitable endeavor is equivalent to claiming coal plants have good margins because boilers are expensive.
- SwellJoe 5mo ago
  I think we agree? What moat? You answered yourself: "capital intensive" But, history says the supercomputer of today will fit in your pocket in a few years. They've bought up all the RAM and GPUs, which pushes the capital requirements upward for everyone else. But, they can't corner the market forever, there are too many competing interests. AMD and Intel keep making new GPUs and APUs. The memory makers can't just sell to only AI companies forever, if they do Chinese manufacturers will move in and eventually eat them from below (as has happened many times before). They have a moat today, and it's just that it's really expensive to train and host frontier models, especially at commercial scale. It used to be there was also some secret sauce to making it fast and efficient. But, secret sauce is being published daily by all sorts of researchers, folks are figuring out how to do more with less and it often finds its way into llama.cpp or vLLM or SGLang within days or weeks.
  - theLiminator 5mo ago
    > But, history says the supercomputer of today will fit in your pocket in a few years. I don't think this will be true in the same time span anymore. Each miniaturization is costing more and more money. Perhaps they'll come up with exotic fundamental improvements, but I don't think the rate of improvement of compute/watt will match the previous decades.
    - SwellJoe 5mo ago
      Yeah, that's probably true, but we're also seeing that there's still tons of inefficiencies in how LLMs are being run. Seems like every couple months there's some new technique to squeeze more performance out of less hardware. KV caching improvements, fast attention, speculative decoding, dynamic quantization, quantization aware training, etc. That said, I recently replaced my five year old self-built PC (with a top-of-the-line desktop CPU, chipset, memory, and GPU of the time) with a new everything-the-best build, and while it's clear we're not keeping up with Moore's Law anymore, it's still 4-5 times faster for compute-intensive stuff, especially parallelizable tasks. We're still getting faster/cheaper. So, the time scale is maybe ten years rather than five.
iplaymyowngames 5mo ago
> The moat looks deep today Does it? What can this model do that I both want and cannot already do? Anthropic made a nice little post saying how dangerous it is, because it is good enough to eat their own business. But I don't want to eat their business. They also said it was good at playing Slay the Spire, but I can't think of anything more insulting than have a machine do that in my place. That's MY comfort game, not something for a stupid Clanker to take away. They did not provide any other use case.
- cindyllm 5mo ago
  [dead]
Ferret7446 5mo ago
The moat is not the model, it's the harness. I wager that's one of the main reasons why Google made Antigravity closed source.
- dudisubekti 5mo ago
  But harness is relatively easy to code yourself? They're just system prompt composer, with some tool functions that the LLM can invoke. I've vibe coded my own in just one day.
  - Paradigma11 5mo ago
    But is there anything preventing them from putting their own proprietary wolfram alpha/prolog/super duper expert system in there?
    - SwellJoe 5mo ago
      Only that it would just slow down the model and make it dumber. You can't tool and harness a weak model into strength and you probably don't improve top models with boondoggles.
    - dudisubekti 5mo ago
      I guess... but I think, at its core, a good coding harness usually includes: - well-crafted system prompt that follows best practices - good contextual reminder prompts (when an llm got stuck in an infinite loop and times out, forgets how to use tools, or needs recurring best practice reminders, etc) - well-written ergonomic tools the llm can use (read/write files, read diffs, browse the internet, etc) I dont think these are anything special. The deepest moat I can think of is, proprietary models can be specifically trained to use their proprietary harnesses, so they are more token-efficient and make less tool call and file editing mistakes. However in my experience, I'm as comfortable working with my own homemade harness as with Claude Code, so I don't think it's a deep moat...
  - SOLAR_FIELDS 5mo ago
    I don't understand why this is being presented as an either/or thing. The moat is actually the harness AND the model, and one of the reasons that Claude works so well is because the model is actually trained with its usage in that specific harness in mind, and the harness is designed to deal with Claude model's idiosyncracies. Easy to validate, just run Claude through some other harness and compare, then just run some other model through Claude's harness and compare
ian_holt 5mo ago
<The memory makers can't just sell to only AI companies forever, if they do Chinese manufacturers will move in and eventually eat them from below (as has happened many times before)> Unfortunately this has been happening almost forever. You can spend 10s of thousands of $$ design, prototyping, building & marketing anything, whether a physical product or software & some company where the wages are lower are going to come along, build it cheaper (not necessarily better quality either) and ship it to the world. As a result, the other countries import more stuff "because it is cheaper" and eventually local manufacturing dwindles away to virtually nothing. That is the case here in Australia. Our manufacturing base has shrunk to stuff all compared to what we had 30 years ago & as a result we are poorer as a nation for it
- DeathArrow 5mo ago
  >Unfortunately this has been happening almost forever. You can spend 10s of thousands of $$ design, prototyping, building & marketing anything, whether a physical product or software & some company where the wages are lower are going to come along, build it cheaper (not necessarily better quality either) and ship it to the world. >As a result, the other countries import more stuff "because it is cheaper" and eventually local manufacturing dwindles away to virtually nothing. That is the case here in Australia. Our manufacturing base has shrunk to stuff all compared to what we had 30 years ago & as a result we are poorer as a nation for it Then the companies in that country need to learn how to be more competitive and governments need to learn how not to overregulate, overtax and raise barriers.
  - techdmn 5mo ago
    > overregulate, overtax and raise barriers Also known as labor and environmental protections. I am in favor of labor and environmental protections, but when producers are allowed to avoid them simply by moving production abroad, well, the incentives are clear.
- Joker_vD 5mo ago
  > some company where the wages are lower are going to come along, build it cheaper (not necessarily better quality either) and ship it to the world. Yeah, it's called competition. It existed even in the socialist countries (where is was called "socialist competition/emulation").
loeg 5mo ago
What makes you say usage billing is cost prohibitive? I use as much flagship model as I could possibly want and it's like four figures a month. That's totally doable compared to SWE pay.
mips_avatar 5mo ago
Given that Anthropic has never released anything open weights I wouldn’t count on the fact that they view finetuning Gemma 4 as something allowable. I think they think nobody other than Anthropic should have AI
psychoslave 5mo ago
There is no training from scratch though. It's kind of, "first create the universe" framing pretention. All models rely heavily on the large corpus that humanity built through large span of time. And of course humanity didn't create the condition of its emergence.
whiplash451 5mo ago
> The moat looks deep today but it's going to become more shallow every year. Unless the frontier labs start nerfing their models, which is exactly what seems to be happening. The counter-point to your argument is a future where less and less un-nerfed open-source frontier models exist. Sure, China/Meta might keep commoditizing their complement by releasing un-nerfed models, but these come with their own limitations too. I am worried that the door to great open-source frontier models might be closing by the day now.
- shaky-carrousel 5mo ago
  A nerfed model is a useless model which makes the moat shallower, not deeper.
  - whiplash451 5mo ago
    Not if the model is part of a broader product that happens to be very useful/relevant to private companies willing to pay a lot for it (e.g. a coding agent that can do many things but won't help you build a frontier LLM model). My intuition is that Claude and the likes are going gung-ho after this, along all the verticals that will generate money without threatening their moat.

hedora 5mo ago

What moat? There are multiple companies providing pareto-optimal frontier models, and it takes O(10) people to build one of these things. The rest is capital intensive, and the price will approach the cost of production over time. Thinking this is a profitable endeavor is equivalent to claiming coal plants have good margins because boilers are expensive.

SwellJoe 5mo ago
I think we agree? What moat? You answered yourself: "capital intensive" But, history says the supercomputer of today will fit in your pocket in a few years. They've bought up all the RAM and GPUs, which pushes the capital requirements upward for everyone else. But, they can't corner the market forever, there are too many competing interests. AMD and Intel keep making new GPUs and APUs. The memory makers can't just sell to only AI companies forever, if they do Chinese manufacturers will move in and eventually eat them from below (as has happened many times before). They have a moat today, and it's just that it's really expensive to train and host frontier models, especially at commercial scale. It used to be there was also some secret sauce to making it fast and efficient. But, secret sauce is being published daily by all sorts of researchers, folks are figuring out how to do more with less and it often finds its way into llama.cpp or vLLM or SGLang within days or weeks.
- theLiminator 5mo ago
  > But, history says the supercomputer of today will fit in your pocket in a few years. I don't think this will be true in the same time span anymore. Each miniaturization is costing more and more money. Perhaps they'll come up with exotic fundamental improvements, but I don't think the rate of improvement of compute/watt will match the previous decades.
  - SwellJoe 5mo ago
    Yeah, that's probably true, but we're also seeing that there's still tons of inefficiencies in how LLMs are being run. Seems like every couple months there's some new technique to squeeze more performance out of less hardware. KV caching improvements, fast attention, speculative decoding, dynamic quantization, quantization aware training, etc. That said, I recently replaced my five year old self-built PC (with a top-of-the-line desktop CPU, chipset, memory, and GPU of the time) with a new everything-the-best build, and while it's clear we're not keeping up with Moore's Law anymore, it's still 4-5 times faster for compute-intensive stuff, especially parallelizable tasks. We're still getting faster/cheaper. So, the time scale is maybe ten years rather than five.

iplaymyowngames 5mo ago

> The moat looks deep today Does it? What can this model do that I both want and cannot already do? Anthropic made a nice little post saying how dangerous it is, because it is good enough to eat their own business. But I don't want to eat their business. They also said it was good at playing Slay the Spire, but I can't think of anything more insulting than have a machine do that in my place. That's MY comfort game, not something for a stupid Clanker to take away. They did not provide any other use case.

cindyllm 5mo ago
[dead]

Ferret7446 5mo ago

The moat is not the model, it's the harness. I wager that's one of the main reasons why Google made Antigravity closed source.

dudisubekti 5mo ago
But harness is relatively easy to code yourself? They're just system prompt composer, with some tool functions that the LLM can invoke. I've vibe coded my own in just one day.
- Paradigma11 5mo ago
  But is there anything preventing them from putting their own proprietary wolfram alpha/prolog/super duper expert system in there?
  - SwellJoe 5mo ago
    Only that it would just slow down the model and make it dumber. You can't tool and harness a weak model into strength and you probably don't improve top models with boondoggles.
  - dudisubekti 5mo ago
    I guess... but I think, at its core, a good coding harness usually includes: - well-crafted system prompt that follows best practices - good contextual reminder prompts (when an llm got stuck in an infinite loop and times out, forgets how to use tools, or needs recurring best practice reminders, etc) - well-written ergonomic tools the llm can use (read/write files, read diffs, browse the internet, etc) I dont think these are anything special. The deepest moat I can think of is, proprietary models can be specifically trained to use their proprietary harnesses, so they are more token-efficient and make less tool call and file editing mistakes. However in my experience, I'm as comfortable working with my own homemade harness as with Claude Code, so I don't think it's a deep moat...
- SOLAR_FIELDS 5mo ago
  I don't understand why this is being presented as an either/or thing. The moat is actually the harness AND the model, and one of the reasons that Claude works so well is because the model is actually trained with its usage in that specific harness in mind, and the harness is designed to deal with Claude model's idiosyncracies. Easy to validate, just run Claude through some other harness and compare, then just run some other model through Claude's harness and compare

ian_holt 5mo ago

<The memory makers can't just sell to only AI companies forever, if they do Chinese manufacturers will move in and eventually eat them from below (as has happened many times before)> Unfortunately this has been happening almost forever. You can spend 10s of thousands of $$ design, prototyping, building & marketing anything, whether a physical product or software & some company where the wages are lower are going to come along, build it cheaper (not necessarily better quality either) and ship it to the world. As a result, the other countries import more stuff "because it is cheaper" and eventually local manufacturing dwindles away to virtually nothing. That is the case here in Australia. Our manufacturing base has shrunk to stuff all compared to what we had 30 years ago & as a result we are poorer as a nation for it

DeathArrow 5mo ago
>Unfortunately this has been happening almost forever. You can spend 10s of thousands of $$ design, prototyping, building & marketing anything, whether a physical product or software & some company where the wages are lower are going to come along, build it cheaper (not necessarily better quality either) and ship it to the world. >As a result, the other countries import more stuff "because it is cheaper" and eventually local manufacturing dwindles away to virtually nothing. That is the case here in Australia. Our manufacturing base has shrunk to stuff all compared to what we had 30 years ago & as a result we are poorer as a nation for it Then the companies in that country need to learn how to be more competitive and governments need to learn how not to overregulate, overtax and raise barriers.
- techdmn 5mo ago
  > overregulate, overtax and raise barriers Also known as labor and environmental protections. I am in favor of labor and environmental protections, but when producers are allowed to avoid them simply by moving production abroad, well, the incentives are clear.
Joker_vD 5mo ago
> some company where the wages are lower are going to come along, build it cheaper (not necessarily better quality either) and ship it to the world. Yeah, it's called competition. It existed even in the socialist countries (where is was called "socialist competition/emulation").

loeg 5mo ago

What makes you say usage billing is cost prohibitive? I use as much flagship model as I could possibly want and it's like four figures a month. That's totally doable compared to SWE pay.

mips_avatar 5mo ago

Given that Anthropic has never released anything open weights I wouldn’t count on the fact that they view finetuning Gemma 4 as something allowable. I think they think nobody other than Anthropic should have AI

psychoslave 5mo ago

There is no training from scratch though. It's kind of, "first create the universe" framing pretention. All models rely heavily on the large corpus that humanity built through large span of time. And of course humanity didn't create the condition of its emergence.

whiplash451 5mo ago

> The moat looks deep today but it's going to become more shallow every year. Unless the frontier labs start nerfing their models, which is exactly what seems to be happening. The counter-point to your argument is a future where less and less un-nerfed open-source frontier models exist. Sure, China/Meta might keep commoditizing their complement by releasing un-nerfed models, but these come with their own limitations too. I am worried that the door to great open-source frontier models might be closing by the day now.

shaky-carrousel 5mo ago
A nerfed model is a useless model which makes the moat shallower, not deeper.
- whiplash451 5mo ago
  Not if the model is part of a broader product that happens to be very useful/relevant to private companies willing to pay a lot for it (e.g. a coding agent that can do many things but won't help you build a frontier LLM model). My intuition is that Claude and the likes are going gung-ho after this, along all the verticals that will generate money without threatening their moat.