3 ms·

> There’s no moat People keep saying this, and while it may be partially true, it misses a very important detail. Right now, overall compute is a moat. Someone

> There’s no moat People keep saying this, and while it may be partially true, it misses a very important detail. Right now, overall compute is a moat. Someone even commented on another of my comments that one reason Google is lagging is they don't have the same level of Nvidia farms as OpenAI and Anthropic. A big reason that OpenAI and Anthropic are racing so fast is they both want to get to recursive self improvement (remains to be seen if that is actually possible, but AFAICT most people at these companies genuinely believe it is) before anyone else, because they believe whoever gets there first will then have an insurmountable lead. But I think they also clearly understand that neither of them have solved for alignment (and in fact they are further from it), and RSI with misaligned models, where interpretability is worse every 6 months, is incredibly dangerous. I think the concerns about regulatory capture are warranted, but I see so many comments parroting the "evil Anthropic and OpenAI" viewpoints that they are missing some of the real, valid concerns and dangers. I think this tweet by David Kokotajlo makes some good points on how to tell if regulations are being "cheated" for the purposes of regulatory capture or if they really are actually pacing the frontier: https://x.com/DKokotajlo/status/2099185129533186438 https://x.com/DKokotajlo/status/2099185129533186438

wwweston 20d ago
Which mostly seem to stem from the fundamental misalignment of OpenAI and Anthropic leaders/owners, aka the evil of the companies. Arguably, though, their evil is well within the normal distribution of the usual evil of humanity magnified via the social technology of capitalism. The further tech is mostly raising that exponentiation to its own exponent. The unchecked singularity was embraced centuries ago.
AnimalMuppet 20d ago
> RSI with misaligned models, where interpretability is worse every 6 months, is incredibly dangerous. This. So very much this. Misalignment (among other things) means that the parent in the RSI cycle isn't going to be working as hard on the alignment of the children as we need.
hintymad 20d ago
> Right now, overall compute is a moat. Very true. Or further, access to capital is the moat. It is the very reason that we don't have a real open-source community that trains frontier models - individuals simply can't afford the training infrastructure, nor sufficient high-quality training data.
- jamienk 20d ago
  https://hugovergnes.github.io/little-lm-3-8b/ https://hugovergnes.github.io/little-lm-3-8b/ << less than $1k for a 4B model Remember how DeepSeek v.whatever cost ~$5m Stable Diffusion 1.5 was reportedly $70k in compute.
  - hn_throwaway_99 20d ago
    None of those are frontier models, nor could they have been. E.g. DeepSeek's "$5 million" included piggybacking off OpenAI/Anthropic's hundreds of millions/billions by using distillation.
    - jamienk 19d ago
      1) I don't think "frontier model" is a term of art, it is a marketing term. I have an M1 MacBook Pro. What do you have? Always the "frontier model" laptop? Do you always order the most expensive "frontier menu item" at the bar and grill? 2) I do not believe the "billions" number is OpenAI/Anthropic's training costs. I suspect it includes business expenses (including the big $$ to the guy who came up with "frontier model") and infrastructure, etc. That includes the data-center costs for running the cloud. And the "R&D" expenses which includes who-knows-what. And the settlement payment for data access. Etc. Why are the cost breakdowns not available to the public? Not because of thoughtfulness, altruism, care for the human race, but because of business plans. 3) "Piggybacking off OpenAI/Anthropic" - Scraping the web is "piggybacking" too, and so is buying existing data or even paying for new data. The "L" in LLM stands for "language" which is our common heritage. But what does this have to do with anything anyway? People argue that truly Free (FOSS) LLMs couldn't be be developed because of costs, but I do not think that that is obvious. This used to be the argument against Linux and Wikipedia.
thaeli 20d ago
I will take those concerns more seriously when we start hearing similar consensus from the Chinese side.
jamienk 20d ago
Is this "compute is a moat" true though? Why are the economics so obfuscated? Where can we see how much money it costs to TRAIN various models. https://hugovergnes.github.io/little-lm-3-8b/ https://hugovergnes.github.io/little-lm-3-8b/ The training and cloud costs are often conflated. R&D costs too (which are hard to compare to open systems). A lot of compute is clearly "wasted" where they are not focusing on optimizations, etc.

wwweston 20d ago

Which mostly seem to stem from the fundamental misalignment of OpenAI and Anthropic leaders/owners, aka the evil of the companies. Arguably, though, their evil is well within the normal distribution of the usual evil of humanity magnified via the social technology of capitalism. The further tech is mostly raising that exponentiation to its own exponent. The unchecked singularity was embraced centuries ago.

AnimalMuppet 20d ago

> RSI with misaligned models, where interpretability is worse every 6 months, is incredibly dangerous. This. So very much this. Misalignment (among other things) means that the parent in the RSI cycle isn't going to be working as hard on the alignment of the children as we need.

hintymad 20d ago

> Right now, overall compute is a moat. Very true. Or further, access to capital is the moat. It is the very reason that we don't have a real open-source community that trains frontier models - individuals simply can't afford the training infrastructure, nor sufficient high-quality training data.

jamienk 20d ago
https://hugovergnes.github.io/little-lm-3-8b/ https://hugovergnes.github.io/little-lm-3-8b/ << less than $1k for a 4B model Remember how DeepSeek v.whatever cost ~$5m Stable Diffusion 1.5 was reportedly $70k in compute.
- hn_throwaway_99 20d ago
  None of those are frontier models, nor could they have been. E.g. DeepSeek's "$5 million" included piggybacking off OpenAI/Anthropic's hundreds of millions/billions by using distillation.
  - jamienk 19d ago
    1) I don't think "frontier model" is a term of art, it is a marketing term. I have an M1 MacBook Pro. What do you have? Always the "frontier model" laptop? Do you always order the most expensive "frontier menu item" at the bar and grill? 2) I do not believe the "billions" number is OpenAI/Anthropic's training costs. I suspect it includes business expenses (including the big $$ to the guy who came up with "frontier model") and infrastructure, etc. That includes the data-center costs for running the cloud. And the "R&D" expenses which includes who-knows-what. And the settlement payment for data access. Etc. Why are the cost breakdowns not available to the public? Not because of thoughtfulness, altruism, care for the human race, but because of business plans. 3) "Piggybacking off OpenAI/Anthropic" - Scraping the web is "piggybacking" too, and so is buying existing data or even paying for new data. The "L" in LLM stands for "language" which is our common heritage. But what does this have to do with anything anyway? People argue that truly Free (FOSS) LLMs couldn't be be developed because of costs, but I do not think that that is obvious. This used to be the argument against Linux and Wikipedia.

thaeli 20d ago

I will take those concerns more seriously when we start hearing similar consensus from the Chinese side.

jamienk 20d ago

Is this "compute is a moat" true though? Why are the economics so obfuscated? Where can we see how much money it costs to TRAIN various models. https://hugovergnes.github.io/little-lm-3-8b/ https://hugovergnes.github.io/little-lm-3-8b/ The training and cloud costs are often conflated. R&D costs too (which are hard to compare to open systems). A lot of compute is clearly "wasted" where they are not focusing on optimizations, etc.