4 ms·

> That's the difference between innovating and copying/distilling someone else's innovation. Aren't the models from Anthropic and OpenAI simply the distilled w

> That's the difference between innovating and copying/distilling someone else's innovation. Aren't the models from Anthropic and OpenAI simply the distilled work of everyone else who ever put their work online, or in books? Why is their distillation okay, but other distillations not ok?

mike_hearn 3mo ago
Not at all. At this point, a large amount of the work is in reinforcement learning where they are effectively generating their own data.
- justacrow 3mo ago
  Ah, so it's okay to copy and distill all human-produced works evee, except for reinforcment-learning data generated by OpenAI/Anthropic etc
  - mike_hearn 3mo ago
    I didn't take any position on the morality of it, just pointing out that the data they're training on isn't all taken from the rest of the world.
rmunn 3mo ago
The difference I was referring to was economic; I was not making an ethical judgment. So many people seem to be reading my comment as making an ethical judgment (based on their reactions); maybe I should edit it to clarify. Nope, seems I'm past the edit window. Oh well.
- lelanthran 3mo ago
  > The difference I was referring to was economic; I was not making an ethical judgment What is the economic difference? LLMs have been trained on the results of billions of dollars worth of time, research, investment and expenditure. When you ask an LLM a question, they are giving you the results of those billions, or hundreds of billions, effort. Those things weren't free; they cost money to produce! If anything, the OpenAI and Anthropics of the world got more economic value for free than the people distilling them did.
  - rmunn 3mo ago
    It takes more computing power, and money, to pay for training an LLM, compared to distilling an LLM that someone else has already trained. That's the economic difference.
    - lelanthran 3mo ago
      > It takes more computing power, and money, to pay for training an LLM, compared to distilling an LLM that someone else has already trained. And that is still less money than it took to create that data in the first place, which the AI companies then gladly took to use for training.
      - rmunn 3mo ago
        Which is true, but also completely irrelevant to the point of this discussion. Which was about the U.S. companies spending trillions and the Chinese companies spending billions and matching them in quality. I made the point that the U.S. companies and Chinese companies were doing different things: training on raw data vs. distilling the model that someone else had trained. That difference explains the difference in spending. (The secondary point is that the Chinese companies would not have been able to get to the point they have, while spending as little as they have, without Claude, GPT, et al to copy from). This is probably the last reply I'll make to you. I'm getting a little tired of repeating myself. Whether you're just not getting it, or refusing to get it, either way it's starting to feel like a waste of time to try to rephrase the same thing again and again. Please read more carefully in the future.
  - Tadpole9181 3mo ago
    Can you please stop being coy and intentionally obtuse? Just have a discussion in good faith, I'm so beyond sick of this kind of rhetoric. Yes, AI training uses human data and a lot of it was not compensated. But that has absolutely nothing to do with the thing this thread is about. Distilling models costs less money than a really procuring quality data and training a model yourself. If you disagree, debate that.

mike_hearn 3mo ago

Not at all. At this point, a large amount of the work is in reinforcement learning where they are effectively generating their own data.

justacrow 3mo ago
Ah, so it's okay to copy and distill all human-produced works evee, except for reinforcment-learning data generated by OpenAI/Anthropic etc
- mike_hearn 3mo ago
  I didn't take any position on the morality of it, just pointing out that the data they're training on isn't all taken from the rest of the world.

rmunn 3mo ago

The difference I was referring to was economic; I was not making an ethical judgment. So many people seem to be reading my comment as making an ethical judgment (based on their reactions); maybe I should edit it to clarify. Nope, seems I'm past the edit window. Oh well.

lelanthran 3mo ago
> The difference I was referring to was economic; I was not making an ethical judgment What is the economic difference? LLMs have been trained on the results of billions of dollars worth of time, research, investment and expenditure. When you ask an LLM a question, they are giving you the results of those billions, or hundreds of billions, effort. Those things weren't free; they cost money to produce! If anything, the OpenAI and Anthropics of the world got more economic value for free than the people distilling them did.
- rmunn 3mo ago
  It takes more computing power, and money, to pay for training an LLM, compared to distilling an LLM that someone else has already trained. That's the economic difference.
  - lelanthran 3mo ago
    > It takes more computing power, and money, to pay for training an LLM, compared to distilling an LLM that someone else has already trained. And that is still less money than it took to create that data in the first place, which the AI companies then gladly took to use for training.
    - rmunn 3mo ago
      Which is true, but also completely irrelevant to the point of this discussion. Which was about the U.S. companies spending trillions and the Chinese companies spending billions and matching them in quality. I made the point that the U.S. companies and Chinese companies were doing different things: training on raw data vs. distilling the model that someone else had trained. That difference explains the difference in spending. (The secondary point is that the Chinese companies would not have been able to get to the point they have, while spending as little as they have, without Claude, GPT, et al to copy from). This is probably the last reply I'll make to you. I'm getting a little tired of repeating myself. Whether you're just not getting it, or refusing to get it, either way it's starting to feel like a waste of time to try to rephrase the same thing again and again. Please read more carefully in the future.
- Tadpole9181 3mo ago
  Can you please stop being coy and intentionally obtuse? Just have a discussion in good faith, I'm so beyond sick of this kind of rhetoric. Yes, AI training uses human data and a lot of it was not compensated. But that has absolutely nothing to do with the thing this thread is about. Distilling models costs less money than a really procuring quality data and training a model yourself. If you disagree, debate that.