3 ms·

False dichotomy right? Are Chinese labs impressively innovating? Clearly. However this doesn’t rule out possible gains due to distillation. I don’t know the

False dichotomy right? Are Chinese labs impressively innovating? Clearly. However this doesn’t rule out possible gains due to distillation. I don’t know the degree of the latter but both things could certainly be true.

api 2mo ago
“Distillation” is just indirectly pirating the largely pirated training data used to train the original model. “You stole my warez!”
- serial_dev 2mo ago
  If we do it, it's training a model. When they do it, it's distillation attack. - Anthropic
  - esafak 2mo ago
    "You are distilling what I have rightfully pirated."
    - Barbing 2mo ago
      Is there even a steelman against this?
    - trollbridge 2mo ago
      The current steelman argument against this is "China bad, west good".
SirHackalot 2mo ago
Didn't Anthropic train on our collective data just to sell it back to us for $100/month? On top of that, Apple is suing them over alleged IP and trade secret theft by ex-Apple employees. Hard to feel too sympathetic, and I’m not an Anthropic hater in particular…
- DashAnimal 2mo ago
  That Apple lawsuit is against OpenAi, just for clarity
  - SirHackalot 2mo ago
    Wow, I should never comment first thing in the morning... Thanks for the correction, you’re right. I will see if I can still edit my comment.
- hugopuybareau 2mo ago
  The only data-related lawsuit Anthropic got was the books nah ? And they paid only a minor part as paid agreement compared to what they would have paid losing the trial
  - SirHackalot 2mo ago
    Yep, that second part of my comment was an article I read about OpenAI and my mind mixed it up with Anthropic. My mistake.
  - joe_the_user 2mo ago
    In a sense, what eventually gets legalized through settlement or what doesn't provoke a lawsuit isn't that relevant. The process of creating an LLM involves taking and processing a massive amount of human generated data, roughly all the world's literature/thinking/etc. A large portion remains within these systems. Aside from the legality, ethically that shouldn't belong to any one company.
- linkregister 2mo ago
  Did Kimi or other open source models use a different corpus? Why is your animus directed specifically to Anthropic?
- hoihoi 2mo ago
  [dead]
cma 2mo ago
If they can distill fable into a full model post training run in ~15 days without the real thinking traces, yet we know Claude chats degraded with the thinking traces removed (chat resume bug from earlier in the year they reported stripping thinking to shed load as being the cause of degradation), how big can this degree be?
fnord123 2mo ago
Also possibly true: Anthropic is running Kimi locally in their hardware and "distilling" it.
- trollbridge 2mo ago
  If they have any sense, they should be. It would be permitted under the licence, too (unless I'm misreading the k3 licence).
  - logicallee 2mo ago
    Anthropic has more than $20 million in revenue so as per the k3 license they would need to enter a special commercial deal if they wanted to use it: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
    - fnord123 2mo ago
      No, keep reading: > The requirements set forth in Sections 2 and 3 do not apply to: (a) internal use of the Software, defined as any use that does not make the Software, its outputs, or its underlying capabilities available to third parties; or (b) any use of the Software accessed through Moonshot AI's official products or certified inference partners. "distilling" from their own hardware would be an internal use of the software.
      - logicallee 2mo ago
        In my personal opinion, distilling its output into training another model that is then provided to users does fall into the category of "making its output or its underlying capabilities available to third parties", but I could see the argument that it doesn't.
deleted 2mo ago
[deleted]
culi 2mo ago
Fable was available for a few weeks before Kimi K3 came out. If it was a distillation attack, then that's a truly groundbreaking technological feat to distill a model like Fable in 2 weeks
joe_the_user 2mo ago
All LLMs are based on distillation broadly defined. Western models began distilling texts. If Chinese models are distilling Western models, they are taking information that Western models don't own anyway - but that doesn't mean the Chinese models aren't also taking information from text as well (which they probably also don't own). And none of this means Western and Chinese companies aren't innovating by creating very elegant methods of distillation.

api 2mo ago

“Distillation” is just indirectly pirating the largely pirated training data used to train the original model. “You stole my warez!”

serial_dev 2mo ago
If we do it, it's training a model. When they do it, it's distillation attack. - Anthropic
- esafak 2mo ago
  "You are distilling what I have rightfully pirated."
  - Barbing 2mo ago
    Is there even a steelman against this?
  - trollbridge 2mo ago
    The current steelman argument against this is "China bad, west good".

SirHackalot 2mo ago

Didn't Anthropic train on our collective data just to sell it back to us for $100/month? On top of that, Apple is suing them over alleged IP and trade secret theft by ex-Apple employees. Hard to feel too sympathetic, and I’m not an Anthropic hater in particular…

DashAnimal 2mo ago
That Apple lawsuit is against OpenAi, just for clarity
- SirHackalot 2mo ago
  Wow, I should never comment first thing in the morning... Thanks for the correction, you’re right. I will see if I can still edit my comment.
hugopuybareau 2mo ago
The only data-related lawsuit Anthropic got was the books nah ? And they paid only a minor part as paid agreement compared to what they would have paid losing the trial
- SirHackalot 2mo ago
  Yep, that second part of my comment was an article I read about OpenAI and my mind mixed it up with Anthropic. My mistake.
- joe_the_user 2mo ago
  In a sense, what eventually gets legalized through settlement or what doesn't provoke a lawsuit isn't that relevant. The process of creating an LLM involves taking and processing a massive amount of human generated data, roughly all the world's literature/thinking/etc. A large portion remains within these systems. Aside from the legality, ethically that shouldn't belong to any one company.
linkregister 2mo ago
Did Kimi or other open source models use a different corpus? Why is your animus directed specifically to Anthropic?
hoihoi 2mo ago
[dead]

cma 2mo ago

If they can distill fable into a full model post training run in ~15 days without the real thinking traces, yet we know Claude chats degraded with the thinking traces removed (chat resume bug from earlier in the year they reported stripping thinking to shed load as being the cause of degradation), how big can this degree be?

fnord123 2mo ago

Also possibly true: Anthropic is running Kimi locally in their hardware and "distilling" it.

trollbridge 2mo ago
If they have any sense, they should be. It would be permitted under the licence, too (unless I'm misreading the k3 licence).
- logicallee 2mo ago
  Anthropic has more than $20 million in revenue so as per the k3 license they would need to enter a special commercial deal if they wanted to use it: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
  - fnord123 2mo ago
    No, keep reading: > The requirements set forth in Sections 2 and 3 do not apply to: (a) internal use of the Software, defined as any use that does not make the Software, its outputs, or its underlying capabilities available to third parties; or (b) any use of the Software accessed through Moonshot AI's official products or certified inference partners. "distilling" from their own hardware would be an internal use of the software.
    - logicallee 2mo ago
      In my personal opinion, distilling its output into training another model that is then provided to users does fall into the category of "making its output or its underlying capabilities available to third parties", but I could see the argument that it doesn't.

deleted 2mo ago

[deleted]

culi 2mo ago

Fable was available for a few weeks before Kimi K3 came out. If it was a distillation attack, then that's a truly groundbreaking technological feat to distill a model like Fable in 2 weeks

joe_the_user 2mo ago

All LLMs are based on distillation broadly defined. Western models began distilling texts. If Chinese models are distilling Western models, they are taking information that Western models don't own anyway - but that doesn't mean the Chinese models aren't also taking information from text as well (which they probably also don't own). And none of this means Western and Chinese companies aren't innovating by creating very elegant methods of distillation.