4 ms·

It's fascinating how close these companies are to each other. Some company comes up with something clever/ground-breaking and everyone else has implemented it a

by lysace 2y ago

It's fascinating how close these companies are to each other. Some company comes up with something clever/ground-breaking and everyone else has implemented it a few weeks later. Hard not to think of Kurzweil's Law of Accelerating Returns.

mechagodzilla 2y ago
It does seem like it will be very, very hard for the companies training their own models to recoup their investment when the capabilities of open-weight models catch up so quickly - general purpose LLMs just seem destined to be a cheap commodity.
- szundi 2y ago
  [dead]
- jsheard 2y ago
  Well, the companies releasing open weights also need to recoup their investments at some point, they can't coast on VC hype forever. Huge models don't grow on trees.
  - mechagodzilla 2y ago
    Or, like Meta, they make their money elsewhere and just seem interested in wrecking the economics of LLMs. As soon as an open-weight model is released, it basically sets a global floor that says "Models with similar or worse performance effectively have zero value," and that floor has been rising incredibly quickly. I'd be surprised if the vast, vast majority of queries ChatGPT gets couldn't get equivalently good results from llama3/deepseek/qwen/mistral models, even for those paying for the pro versions.
    - Philpax 2y ago
      Eh, to some extent - there's still a pretty significant cost to actually running inference for those models. For example, no consumer can run DeepSeek v3/r1 - that requires tens, possibly hundreds, of thousands of dollars of hardware to run. There's still room for other models, especially if they have different performance characteristics that make them suitable to run under consumer constraints. Mistral has been doing quite well here.
      - mechagodzilla 2y ago
        If you don't need to pay for the model development costs, I think running inference will just be driven down to the underlying cloud computing costs. The actual requirement to passably (~4-bit quantization) run Deepseek v3/r1 at home is really just having 512GB or so of RAM - I bought a used dual-socket xeon for $2k that has 768GB of RAM, and can run Deepseek R1 at 1-1.5 tokens/sec, which is perfectly usable for "ask a complicated question, come back an hour or so later and check on the result".
    - riku_iki 2y ago
      I think Meta folks just don't know how to come to this market and build something potentially profitable, and doing random stuff, because need to report some results to management.
azinman2 2y ago
It’s extremely unlikely that everyone is copying in a few weeks for models that themselves take many weeks if not longer to train. Great minds think alike, and everyone is influencing everyone. The history of innovation is filled with examples of similar discoveries around the same time but totally disconnected in the world. Now with the rate of publishing and the openness of the internet, you’re only bound to get even more of that.
- lysace 2y ago
  Isn't the reasoning thing essentially a bolt-on to existing trained models? Like basically a meta-prompt?
  - pertymcpert 2y ago
    Somewhat but not exactly? I think the models need to be trained to think.
  - azinman2 2y ago
    No. DeepSeek and now related projects have shown it’s possible to add reasoning via SFT to existing models, but that’s not the same as a prompt. But if you look at R1 they do a blend of techniques to get reasoning. For Anthropic to have a hybrid model where you can control this, it will have to be built into the model directly in its training and probably architecture as well. If you’re a competent company filled with the best AI minds and a frontier model, you’re not just purely copying… you’re taking ideas while innovating and adapting.
  - Philpax 2y ago
    The fundamental innovation is training the model to reason through reinforcement learning; you can train existing models with traces from these reasoning models to get you within the same ballpark, but taking it further requires you to do RL yourself.
- KaoruAoiShiho 2y ago
  The copying here probably goes to strawberry from o1 which is like at least 6 months but maybe copying efforts started even earlier.
deleted 2y ago
[deleted]
luma 2y ago
Where RL can play into post training there's something of an anti-moat. Maybe a "tow rope"? Let's say OAI releases some great new model. The moment it becomes available via API, everyone else can make use of that model to create high-quality RL training data, which can then be used to make their models perform better. The very act of making an AI model commercially available is the same act which allows your competitors to pull themselves closer to you.

by lysace 2y ago

mechagodzilla 2y ago

It does seem like it will be very, very hard for the companies training their own models to recoup their investment when the capabilities of open-weight models catch up so quickly - general purpose LLMs just seem destined to be a cheap commodity.

szundi 2y ago
[dead]
jsheard 2y ago
Well, the companies releasing open weights also need to recoup their investments at some point, they can't coast on VC hype forever. Huge models don't grow on trees.
- mechagodzilla 2y ago
  Or, like Meta, they make their money elsewhere and just seem interested in wrecking the economics of LLMs. As soon as an open-weight model is released, it basically sets a global floor that says "Models with similar or worse performance effectively have zero value," and that floor has been rising incredibly quickly. I'd be surprised if the vast, vast majority of queries ChatGPT gets couldn't get equivalently good results from llama3/deepseek/qwen/mistral models, even for those paying for the pro versions.
  - Philpax 2y ago
    Eh, to some extent - there's still a pretty significant cost to actually running inference for those models. For example, no consumer can run DeepSeek v3/r1 - that requires tens, possibly hundreds, of thousands of dollars of hardware to run. There's still room for other models, especially if they have different performance characteristics that make them suitable to run under consumer constraints. Mistral has been doing quite well here.
    - mechagodzilla 2y ago
      If you don't need to pay for the model development costs, I think running inference will just be driven down to the underlying cloud computing costs. The actual requirement to passably (~4-bit quantization) run Deepseek v3/r1 at home is really just having 512GB or so of RAM - I bought a used dual-socket xeon for $2k that has 768GB of RAM, and can run Deepseek R1 at 1-1.5 tokens/sec, which is perfectly usable for "ask a complicated question, come back an hour or so later and check on the result".
  - riku_iki 2y ago
    I think Meta folks just don't know how to come to this market and build something potentially profitable, and doing random stuff, because need to report some results to management.

azinman2 2y ago

It’s extremely unlikely that everyone is copying in a few weeks for models that themselves take many weeks if not longer to train. Great minds think alike, and everyone is influencing everyone. The history of innovation is filled with examples of similar discoveries around the same time but totally disconnected in the world. Now with the rate of publishing and the openness of the internet, you’re only bound to get even more of that.

lysace 2y ago
Isn't the reasoning thing essentially a bolt-on to existing trained models? Like basically a meta-prompt?
- pertymcpert 2y ago
  Somewhat but not exactly? I think the models need to be trained to think.
- azinman2 2y ago
  No. DeepSeek and now related projects have shown it’s possible to add reasoning via SFT to existing models, but that’s not the same as a prompt. But if you look at R1 they do a blend of techniques to get reasoning. For Anthropic to have a hybrid model where you can control this, it will have to be built into the model directly in its training and probably architecture as well. If you’re a competent company filled with the best AI minds and a frontier model, you’re not just purely copying… you’re taking ideas while innovating and adapting.
- Philpax 2y ago
  The fundamental innovation is training the model to reason through reinforcement learning; you can train existing models with traces from these reasoning models to get you within the same ballpark, but taking it further requires you to do RL yourself.
KaoruAoiShiho 2y ago
The copying here probably goes to strawberry from o1 which is like at least 6 months but maybe copying efforts started even earlier.

deleted 2y ago

[deleted]

luma 2y ago

Where RL can play into post training there's something of an anti-moat. Maybe a "tow rope"? Let's say OAI releases some great new model. The moment it becomes available via API, everyone else can make use of that model to create high-quality RL training data, which can then be used to make their models perform better. The very act of making an AI model commercially available is the same act which allows your competitors to pull themselves closer to you.