3 ms·
All frontier models have been trained without any regards for IP protection laws. I don't see how anyone can argue in good faith that distillation is not fair g
by pandoro 4mo ago
All frontier models have been trained without any regards for IP protection laws. I don't see how anyone can argue in good faith that distillation is not fair game and does not ultimately "benefit humanity™"
- skybrian 4mo agoDoes distilling actually violate any IP protection laws? Sure, it’s against their terms of service.
- tills13 4mo agoI get what you're saying and two wrongs do not make a right but the irony, and why people are even talking about this, is that the thing being distilled clearly, and knowingly, violated copyright & terms across the entire internet.
- skybrian 4mo agoYeah, everyone is saying that but I don't find the irony very interesting.
- Laurel1234 4mo agoIf it does then their original training did as well.
- nonethewiser 4mo agoTraining a model is very expensive and creates something no individual rights-holder could. Distilling a model copies this value add and captures it without bearing the cost that created it.
- mosura 4mo agoWhat of the costs for creating the data that was used to train the model being distilled?
- yandie 4mo agoAt least 1.5B by stealing books, per a recent ruling. I don’t feel sorry for the model companies
- mosura 4mo agoThey only had to pay for storing the books on a server for later possible use. They did not have to pay anything for the training which was declared fair use.
- deleted 4mo ago[deleted]
- deleted 4mo ago[deleted]
- nonethewiser 4mo agoThis actually undermines the argument that distilling is harmless because its founded on the idea that Anthropic did the same thing and didn’t have any repercussions.
- applicative 4mo agoDo you people seriously believe that Moonshot and the Chinese personal-cult-state dont possess and train on the same torrents?
- mosura 4mo agoThey aren’t hypocritically crying foul about it, so no we don’t care if they do.
- deleted 4mo ago[deleted]
- nonethewiser 4mo agoWell of course. But they dont hate Chinese model companies.
- titzer 4mo agoAn incredibly ironic comment.
- jayd16 4mo agoI'm going to hope this was sarcasm and if it is, it's great.
- 8note 4mo agothe outputs of the model have no property protections, and training a model on the outputs of another model does the exact same thing - its expensive and creates new value over what was in the input - a set of documents.
- pandoro 4mo agoCould you give an example of the value that only training a model can create but none of the rights-holder could? I feel like if you got a direct, instant communication channel to any of the rights-holder that created the content in the training set of those models, you'd get more value than what the LLM could ever give you on any specific subject.
- nonethewiser 4mo ago>Could you give an example of the value that only training a model can create but none of the rights-holder could? Yeah. LLMs
- shlewis 4mo agoYes. I bet Moonshot paid for API access as opposed to pirating like Anthropic did.
- deleted 4mo ago[deleted]
- nonethewiser 4mo agoI understand you think Anthropic should have paid for the information it trained the models on. But im talking about all the costs to build a model. Do you think Anthropic didnt spend money to build these models? Did you not know that its actually very expensive?
- shlewis 4mo agoMoonshot pay for access at the price point Anthropic sees fit, potentially more, since they likely had to jump through multiple hoops? It's hard to feel sorry about the breach of their ToS, which ultimately is all Anthropic can argue, when they are constantly being sued by countless IP owners
- rvba 4mo agoA lot of those arguments could be said about writing a book, or a decent forum guide.
- verdverm 4mo agoThere is a lot more one has to be good at to make a fable level model. Distillation won't get you there, it will help refine some at the end
- applicative 4mo agoOne mentions distillation to intimate that one is in fact still the leader. It has zero to do with fairness.
- parineum 4mo agoFor me, I don't really care about the theft aspect but when people are claiming that these open models are better value or going to overtake anthropic/openai models, the implication that the open models are training of distilled data means all the "progress" they are making is just mimiced from the closed models. It's a bit interesting how the open models are able to keep pace with the closed models except whole maintaining a steady following time.
- orbital-decay 4mo agoDistillation is absolutely not the reason they're good. It's not necessarily even done on a more capable model. It can even be done on itself and still bring improvement, or on a weaker model as well (see GLM and Gemini, which is definitely true because it repeats Deepmind's injections).
- parineum 4mo agoIf distillation doesn't make them better, why do it? If using Anthropic models for distillation doesn't make them better, why do it?
- orbital-decay 4mo agoIt does, it's just not the reason. Most of the work is done before that point, and as I said z.ai used a weaker model. Besides, this is all strictly one-sided, as nobody knows how much Anthropic and OpenAI borrowed from Chinese labs' open research and weights (and they innovated a lot, to put it mildly, starting with first reasoning models worth talking about long before OAI did the same). Chinese labs are also severely restricted on hardware. This entire story makes certain American AI shops look cartoonishly evil and Chinese ones relatively sane. Not only they want to grab without giving anything back, they also want to sabotage everyone else's AI research and do plenty of terrible things like media manipulation on the global scale and getting in bed with the government. This can't possibly end well, for the Americans in the first place.
- abdullahkhalids 4mo ago