4 ms·

There has never been much secret sauce in the model itself. The secret sauce or competitive advantage has always been in the engineering that goes into the data

There has never been much secret sauce in the model itself. The secret sauce or competitive advantage has always been in the engineering that goes into the data collection, model training infrastructure, and lifecycle/debugging management of model training. As well as in the access to GPUs. Yeah, with Deepseek the barrier to entry has become significantly lower now. That's good, and hopefully more competition will come. But it's not like it's a fundamental change of where the secret sauce is.

llm_trw 2y ago
The word you're looking for is copyright enfrignment. That's the secret sause that every good model uses.
- resters 2y ago
  True, and likely lots of other data that should not have been used to train a model. It is at this point hard to imagine a model that is good at reasoning that doesn't also have vast implicit "knowledge".
- silverliver 2y ago
  Humanity keeps running into copyright issues with every major leap in IT technology (photocopiers, tape cassettes, personal computers, internet, and now AI). I think it's about time for humanity to rethink their take on the unnatural restriction of information. I personally hope that countries recognize copyright and patents for what they really are and abolish them. Countries that refuse to do so can play catch up.
  - Mi2ko3 2y ago
    This is based on a flawed view of how we humans behave. Without incentive no effort. This is also the reason why socialism has and always will fail. People who put massive effort in creating original content need to be able to earn the rewards.
    - silverliver 2y ago
      The premise, that forgoing copyright would necessitate the forgoing of incentives and rewards, is one entirely of your own assertion and was not implied in my above comment. I agree that your assertion is flawed. There can be, and are, incentives and rewards associated with sharing information without flawed artificial constraints like copyright.
  - LincolnedList 2y ago
    Since all kinds of companies are getting a lot of money from the generative AI business, I think they can handle being sued for plagiarism if thats the content they produce.
- singularity2001 2y ago
  since all models are treating human knowledge as copyright free (as they should) no this is not at all what this new Chinese model is about
  - cluckindan 2y ago
    Oh. Does that ethics framework also extend to art such as music, movies and software? fires up BitTorrent
- fulafel 2y ago
  It will be interesting if a significant jurisdiction's copyright law is some day changed to treat LLM training as copying. In a lot of places, previous behaviour can't be retroactively outlawed[1]. So older LLMs will be much more capable than post-change ones. [1] https://en.wikipedia.org/wiki/Ex_post_facto_law https://en.wikipedia.org/wiki/Ex_post_facto_law
  - atq2119 2y ago
    Even if you can't be punished retroactively for previous behavior, continuing to benefit from it can be outlawed. In other words, it would be compatible from a legal perspective to ban the use of LLMs that were trained in violation of copyright law. Given the political landscape I doubt that's going to happen, though.
  - jazzyjackson 2y ago
    The part where a python script ingested the books is not the infringing step, it's when they downloaded the books in the first place.
resters 2y ago
I don't disagree, but the important point is that Deepseek showed that it's not just about CapEx, which is what the US firms were/are lining up to battle with. In my opinion there is something qualitatively better about Deepseek in spite of its small size, even compared to o1-pro, that suggests a door has been opened. GPUs are needed to rapidly iterate on ideas, train, evaluate, etc., but Deepseek has shown us that we are not yet in the phase where hardware CapEx guarantees victory. Imagine if Deeepseek hadn't been open sourced!
- govideo 2y ago
  Let me qualify your statement... CapEx is what EXISTING US firms were/are lining up to battle with. With R1 as inspiration/imperative, many new US startups will emerge who will be very strong. Can you feel a bunch of talent in limbo startups pivoting/re-energized now?
  - resters 2y ago
    > Can you feel a bunch of talent in limbo startups pivoting/re-energized now? True! It certainly should be, as there is a lot less reason to hitch one's wagon to one of the few big firms that can afford nation state scale GPU compute.
- EGreg 2y ago
  How do you know the CCP didn’t just help out with lots of compute and then tell the companies to lie about how much it cost to train the model? Reagan did the same with Star Wars, in order to throw the USSR into exactly the same kind of competition hysteria and try to bankrupt it. And USA today is very much in debt as it is… seems like a similar move: https://www.nytimes.com/1993/08/18/us/lies-and-rigged-star-wars-test-fooled-the-kremlin-and-congress.html https://www.nytimes.com/1993/08/18/us/lies-and-rigged-star-w...
- roenxi 2y ago
  Back in the day there were a lot of things that appeared not to be about capex because the quality of the capital was improving so quickly. Computers became obsolete after a year or two. Then the major exponential trends finished running their course and computers stayed useful for longer. At that point, suddenly AWS popped up and it turned out computing was all about massive capital investments. AI will be similar. In the fullness of time, for the major players it'll be all about capex. The question is really just what time horizon that equilibrium will form.

llm_trw 2y ago

The word you're looking for is copyright enfrignment. That's the secret sause that every good model uses.

resters 2y ago
True, and likely lots of other data that should not have been used to train a model. It is at this point hard to imagine a model that is good at reasoning that doesn't also have vast implicit "knowledge".
silverliver 2y ago
Humanity keeps running into copyright issues with every major leap in IT technology (photocopiers, tape cassettes, personal computers, internet, and now AI). I think it's about time for humanity to rethink their take on the unnatural restriction of information. I personally hope that countries recognize copyright and patents for what they really are and abolish them. Countries that refuse to do so can play catch up.
- Mi2ko3 2y ago
  This is based on a flawed view of how we humans behave. Without incentive no effort. This is also the reason why socialism has and always will fail. People who put massive effort in creating original content need to be able to earn the rewards.
  - silverliver 2y ago
    The premise, that forgoing copyright would necessitate the forgoing of incentives and rewards, is one entirely of your own assertion and was not implied in my above comment. I agree that your assertion is flawed. There can be, and are, incentives and rewards associated with sharing information without flawed artificial constraints like copyright.
- LincolnedList 2y ago
  Since all kinds of companies are getting a lot of money from the generative AI business, I think they can handle being sued for plagiarism if thats the content they produce.
singularity2001 2y ago
since all models are treating human knowledge as copyright free (as they should) no this is not at all what this new Chinese model is about
- cluckindan 2y ago
  Oh. Does that ethics framework also extend to art such as music, movies and software? fires up BitTorrent
fulafel 2y ago
It will be interesting if a significant jurisdiction's copyright law is some day changed to treat LLM training as copying. In a lot of places, previous behaviour can't be retroactively outlawed[1]. So older LLMs will be much more capable than post-change ones. [1] https://en.wikipedia.org/wiki/Ex_post_facto_law https://en.wikipedia.org/wiki/Ex_post_facto_law
- atq2119 2y ago
  Even if you can't be punished retroactively for previous behavior, continuing to benefit from it can be outlawed. In other words, it would be compatible from a legal perspective to ban the use of LLMs that were trained in violation of copyright law. Given the political landscape I doubt that's going to happen, though.
- jazzyjackson 2y ago
  The part where a python script ingested the books is not the infringing step, it's when they downloaded the books in the first place.

resters 2y ago

I don't disagree, but the important point is that Deepseek showed that it's not just about CapEx, which is what the US firms were/are lining up to battle with. In my opinion there is something qualitatively better about Deepseek in spite of its small size, even compared to o1-pro, that suggests a door has been opened. GPUs are needed to rapidly iterate on ideas, train, evaluate, etc., but Deepseek has shown us that we are not yet in the phase where hardware CapEx guarantees victory. Imagine if Deeepseek hadn't been open sourced!

govideo 2y ago
Let me qualify your statement... CapEx is what EXISTING US firms were/are lining up to battle with. With R1 as inspiration/imperative, many new US startups will emerge who will be very strong. Can you feel a bunch of talent in limbo startups pivoting/re-energized now?
- resters 2y ago
  > Can you feel a bunch of talent in limbo startups pivoting/re-energized now? True! It certainly should be, as there is a lot less reason to hitch one's wagon to one of the few big firms that can afford nation state scale GPU compute.
EGreg 2y ago
How do you know the CCP didn’t just help out with lots of compute and then tell the companies to lie about how much it cost to train the model? Reagan did the same with Star Wars, in order to throw the USSR into exactly the same kind of competition hysteria and try to bankrupt it. And USA today is very much in debt as it is… seems like a similar move: https://www.nytimes.com/1993/08/18/us/lies-and-rigged-star-wars-test-fooled-the-kremlin-and-congress.html https://www.nytimes.com/1993/08/18/us/lies-and-rigged-star-w...
roenxi 2y ago
Back in the day there were a lot of things that appeared not to be about capex because the quality of the capital was improving so quickly. Computers became obsolete after a year or two. Then the major exponential trends finished running their course and computers stayed useful for longer. At that point, suddenly AWS popped up and it turned out computing was all about massive capital investments. AI will be similar. In the fullness of time, for the major players it'll be all about capex. The question is really just what time horizon that equilibrium will form.