17 ms·

There's two basic kinds of distillation: 1) the massive [and dumb] method where you ask a question and use the answer as reinforcement (Black Box), and 2) more

There's two basic kinds of distillation: 1) the massive [and dumb] method where you ask a question and use the answer as reinforcement (Black Box), and 2) more targeted distillation where you use one model to directly inform/train/guide another model (RLAIF). The latter is basically fine-tuning the model with direction from another model. Thousands of businesses do this every day to fine-tune. This is almost certainly what the Chinese labs are doing, since it has a much better effect on the end result than just getting simple answers to simple questions. These complaints of distillation are inflating the problem to make it sound worse than it is, because they want the USG to block/ban Chinese model providers as protectionism. They have already called for more export controls on chips (which is funny because DeepSeek v4 was designed to run on Huawei chips and now the other Chinese providers are following suit). But they can't come right out and say that, so their claim is that they're asking for more export controls because distilled models might not be as safe as their own. But if you show them a jailbreak of their model that bypasses their safety, they'll tell you that any model can eventually be jailbroken so don't worry about safety.

dannyw 4mo ago
If you’re doing evals, you’re basically doing RLAIF without training a model; just looking at the results. Fundamentally it is very difficult to stop this while still making your AI models useful.
- zmgsabst 4mo ago
  Similarly, if you did a corpus study on bioRvix to summarize recent science findings — you could use the same questions and answers to fine tune a model. There is no way to communicate information at scale to companies through the API, for anything approaching a real application, without that information forming a corpus another model can be trained on. But it wouldn’t be the first time they broke a model: Their “guardrails” that cause it to reject user prompts also means it relies on its pop science summary of medicine to tell you why bioRxiv is wrong rather than accurately summarize the papers. They’ve successfully created a smug, argumentative average of the internet which refuses to even consider it might be wrong or that it’s reading a science paper which is based on measurements and not vibes — but why would I pay for that? I get it for free online.
janalsncm 4mo ago
Yeah I think the technical term is something more like “pseudo-labeling”. The OG distillation requires logits which Anthropic doesn’t provide.
mannanj 4mo ago
>But if you show them a jailbreak of their model that bypasses their safety, they'll tell you that any model can eventually be jailbroken so don't worry about safety. Yes this is in line with what Anthropic said in their public statements about their Fable access restriction by the government directive. The hypocrisy and inconsistency in their statements and behavior feels quite childish and controlling. I believe our companies and their leaders, friends among our other influential leaders and leaders from rich social classes, want to actively hurt most people as this behavior looks to be quite self-interested.
- topato 4mo ago
  Not to mention, the person who brought this quote unquote jailbreak to the Trump Administration was Amazon’s new CEO. They know their IPOs are coming up, so locking their competitors out of the U.S. (even if just for the weeks surrounding the IPO date) would be a major boon. The White House seems to love making announcements just for the sake of making the market move…. Coincidentally, right after POTUS buys a massive amount of the benefactory company’s stock (Buy Dell Computers, lol)
gmerc 4mo ago
https://research.nvidia.com/labs/lpr/slm-agents/ https://research.nvidia.com/labs/lpr/slm-agents/ - Distillation data is a natural byproduct of using these models. There's no effective defence against it. Anthropic is degrading thinking blocks to summaries to slow it down and hide model internals, but in the end, the math says you're SOL and it works on MNC/Large Corporate scale well enough that the moment cost becomes a priority, you're left without the lock in you need to keep customers paying.
- alfiedotwtf 4mo ago
  Byproduct? It’s essentially the only part of an LLM that is useful, because it’s the WHOLE product! It’s the same reason why DRM for audio and video is a non sequitur - if you want a person to see or hear audio or video, eventually at the end of the chain, it’s going to be converted to audio for the ear and light for the eyes - that’s why you attach your tap. Without a model generating tokens, what’s the point. So if Anthropic somehow disable quality token generation, what’s the point!
  - TeMPOraL 4mo ago
    That's why the harness is moving server-side: because generating tokens is not the actual point of the model, not for the users. Especially with tool calling giving us agents that can act, most of the tokens generated are not, themselves, critical to the end users. Specifically, a lot of tokens goes into orchestrating actual tool calls, and then most "thinking tokens" are only relevant to users only in so far as they help users keep track of and verify what the LLM is doing. So all those tokens can be hidden or replaced by partial summaries, and all of that can happen server-side, and then there's very little to distill from.
    - TSiege 4mo ago
      I haven't heard of this happening, do you have links any explainers on this?
      - TeMPOraL 4mo ago
fnord77 4mo ago
Can you reach into the model and "transplant" weights directly?
- antonvs 4mo ago
  You can do things like that - one example is averaging weights between related models - but not with Anthropic's models, because outsiders don't have access to the weights.
  - fulafel 4mo ago
    Weights are just data a server, so we don't know outsiders have access (either via breakin or arrangement).
    - antonvs 4mo ago
      Yes, obviously. That's not the point.
- parineum 4mo ago
  If you have access to the weights, you can just use them as is...
  - HarHarVeryFunny 4mo ago
    Anthropic are not saying they have been hacked - they are saying that Alibaba have been sending lot of requests to their servers.
- jorisw 4mo ago
  No, you'd need to have the model on your filesystem for direct access, and then the architecture would need to be the same.
- X-Ryl669 4mo ago
  I'm not 100% sure it's not possible. If (I don't know) it's possible to freeze the temperature of the model so it's deterministic, and if you could make a map of produced words back to tokens (via HMM probably), then you can probably alter a minimal input and observe the output to model it. If you perform waves of such minimal alterations, you can expect to be able to locate the distance where each alteration impact the model (the idea being that a small alteration on output is likely due to the last layers of the models, and a small alteration is likely due to the deeper layer). Once you've located most of the last layer(s?) weights, you can try to solve for them. With a hundreds of billions weights model, the last layers will likely be so huge that it's probably unfeasible technically, but it's theoretically possible.
anon373839 4mo ago
> These complaints of distillation are inflating the problem to make it sound worse than it is Unfortunately, the Reuters piece itself is complicit in this dramatization. The lede paragraph parrots Anthropic's talking point that distillation is an "attack", without using quotes that would alert the reader that this framing is a corporate talking point. Distillation is NOT an attack.
- p4coder 4mo ago
  Agreed! I had to do a double take and check the URL. I thought I am reading a press release rather than actual reporting.
  - da_grift_shift 4mo ago
    Same thing nowadays :^)
    - verisimi 4mo ago
      It always was.
  - soperj 4mo ago
    That's exactly what they pay the publicist for.
  - friendzis 4mo ago
    https://news.ycombinator.com/item?id=13155538 https://news.ycombinator.com/item?id=13155538
    - verdverm 4mo ago
      ironically, I think this is why the jobs apocolypse is overblown, Ai is only good at a thing if the people using it are also good at that thing, and people are attributing Ai as superhuman at things they do not know themselves
lemax 4mo ago
I've used RLAIF to build out heuristic based non-LLM models for various decision systems and achieved like, 95% F1 on certain projects. We're in a place where models can be used to fine tune a lot of stuff via loops.
handoflixue 4mo ago
> But if you show them a jailbreak of their model that bypasses their safety, they'll tell you that any model can eventually be jailbroken so don't worry about safety. They claim two things: 1) The specific, available jailbreak for Fable 5 is not dangerous - this has been confirmed by multiple experts, and there is no credible evidence against this claim (in other words, Anthropic is probably correct) 2) It is impossible to build an LLM that is immune to all jailbreaks. Again, there is no credible evidence against this claim, i.e. Anthropic is again entirely correct. If #1 was false, they could just publish the details of the jailbreak - it supposedly only works on Fable 5, so there's no possible danger. If #2 was false, surely some other LLM lab would have done it by now. Especially since a number of governments have made it clear there is a market for such a project.
- mcintyre1994 4mo ago
  Supposedly the details of the ‘jailbreak’ are that you give it insecure code and say “fix this code”, and it does, and then you ask it for test scripts and that’s effectively an exploit against the unfixed code. If true then I have no idea how anyone’s going to release a useful model that doesn’t have the same jailbreak. https://www.theregister.com/security/2026/06/15/feds-freaked-over-fable-5-after-simple-fix-this-code-prompt-not-jailbreak-says-researcher/5255827 https://www.theregister.com/security/2026/06/15/feds-freaked...
  - handoflixue 4mo ago
    If that's the extent of the jailbreak, then the government should have banned every existing LLM - their story only makes sense if there's some Fable-specific capability that got unlocked.
    - le-mark 4mo ago
      There’s no logic to it, blocking fable was retaliation and market manipulation by the current admin, nothing more. Poorly conceived as well.
    - Rudybega 4mo ago
JumpCrisscross 4mo ago
> These complaints of distillation are inflating the problem They’re also missing the point. What would have happened to a member of the Manhattan Project who, through personal pursuit of profit, neglected their duty enough to let the bomb leak?
- nixon_why69 4mo ago
  The companies are all for-profit companies, its not like they're selling out some national security goal for profit, profit is the point. Anthropic already heavily restricts Chinese traffic but that only jams up researchers and regular Joes. Anyone motivated enough can hop a flight to Singapore with an nvme drive in their pocket.
cm2187 4mo ago
Stupid question: I was under the impression that these models were trained on PB of data. Surely the amount of questions/response they can extract from querying a bigger model (Claude) is fairly modest. How is it not a drop vs the training dataset?
- eru 4mo ago
  This might be like an observational study vs a study with a control?
  - anon373839 4mo ago
    From what I understand, at this point, the main value of stronger model outputs is simply to bootstrap reasoning behavior during the RL post-training phase. It gets you past the “cold start” problem with RL, after which the outputs aren’t needed anymore. From then on, it’s hill climbing and that requires environments for the model to interact with get rewards from.
- woctordho 4mo ago
  Actually nowadays LLMs are only trained with TBs rather than PBs of data, and it's not too hard to find GBs of agent traces online.
- reasonableklout 4mo ago
  There are multiple stages of training, and the data/compute mix at each are quite different and produce different "layers" of intelligence. The pretraining stage is the first stage which consists of "next token prediction" on the entire internet, PB of tokens, etc. This is what most people think of when they think of training LLMs, however it produces a "base model" which is not really "intelligent", but rather much like a blurry JPEG of all human language and knowledge. You cannot really talk to such a model; it will simply complete your prompt by producing both sides of the conversation. Note however at some level the training has encoded enough structure through compression that it is able to simulate all sorts of phenomena, from human conversations to code. The great R&D difficulty here is to scale pretraining so that it can proceed smoothly in vast distributed datacenters in a fault-tolerant manner. The next few stages are collectively called post-training, and typically consist of supervised fine-tuning, then reinforcement learning. In supervised fine-tuning, the model is further trained to predict the next token, but on a much more focused data set of natural language conversations where the "assistant" and "user" turns are explicitly delineated with special tokens. The output of this stage is a model which is capable of carrying on proper conversations, but typically with no ability to creatively problem-solve, and less of a personality. The data and compute are many orders of magnitude smaller than in pretraining. The reinforcement learning stage used to be a small part of model training, but ever since AI-assisted coding took off, it has become larger and larger chunk of training. In recent models, the compute spend on RL has allegedly come to rival or even exceed that of pretraining [1], which is a bit scary because RL is classically what lead to sci-fi like AIs which are extremely good at accomplishing goals to the detriment of everything else. The way that RL works is that you put an instance of your model in some environment (such as a VM containing a git repository) and give it a task (such as fix the linked github issue). The model will then generate a bunch of attempts to solve the task which we call "trajectories", in most cases there is either an objective measure of the task success (such as passing the tests), or a fuzzy measure (such as having another LLM look at the results and provide a score). This is called the reward, and the model will learn slowly by producing trajectories that receive reward. It can actually be quite hard to prevent "reward hacking" from the model here and the rewards must be shaped very carefully, much R&D labor goes into here, as well as similar challenges to distributed pretraining. A significant challenge is that coding/knowledge work tasks these days are getting extremely difficult, we are far beyond 2024 days where models could barely solve the easiest problems in SWE-bench. Tasks at the frontier now look more like mini projects that would take humans multiple hours or even days to finish (or in some cases, research-style tasks that would be beyond reach for even top human experts, such as the Erdős unit distance problem which was posed in 1946 but wasn't solved until recently, by GPT-5.5). Huge amounts of trajectories must be produced, and huge amounts of them produce zero reward and therefore are useless for learning. Getting a cold start requires running tens of thousands of instances of your model in VMs in parallel for multiple days to produce trajectories, to say nothing of the GPU costs. So what do you do when you only have a model which is capable of basic conversations but cannot even begin to tackle basic coding tasks, use tools, etc? The approach that companies behind the frontier have decided on is to bootstrap their learning process by having an already extremely intelligent model such as Claude produce hundreds of thousands of seed trajectories for them. Then they can use this data to get a warm start and begin learning immediately. And if you use Claude for your reward model too, you get to skip the nastiness of reward shaping. Therefore, even if in number of raw tokens the data are much smaller than internet-scale pretraining data, the value that each token provides is far far greater. [1] For example, Grok 4 compute spend on RL was ~100% of that of pretraining: https://www.interconnects.ai/p/grok-4-an-o3-look-alike-in-search https://www.interconnects.ai/p/grok-4-an-o3-look-alike-in-se...
friendzis 4mo ago
> These complaints of distillation are inflating the problem to make it sound worse than it is This is, in part, a problem every judicial and legislative system has faced since forever: form versus function. Take a classic elicitation spying techniques: a foreign spy meets a military officer/scientist at a bar, strikes up a conversation, makes an observation wondering how could a missile hit some target at some accuracy and elicits a response that with laser guidance it is entirely possible. From this they get info that there is some technology to laser guide missiles. Or in retail, a competitor hiring a secret buyer for core baskets of goods and analyzing prices in the receipts. The function is espionage, the form is conversation and all info is in a sense provided willingly. Where do you pull the slider? These distillation "attacks" are not only indistinguishable from evals, they ARE evals. The function is own model training, the form is eval. Normally, one would expect to have risk benefit analysis based discussion which direction to push the legality slider to. The problem with these recurring statements is that they invoke enshitification of legislature.
SubiculumCode 4mo ago
The compute deficit of Chinese Ai companies is real, and it IS THE ONLY competitive advantage that Western companies have. The only way the U.S. keeps that edge is to prevent distillation. The only way Chinese companies can make up for the deficit in compute is to distill. There innovation in great supply on every side of the Ocean. Its about the chips. And in terms of national security, for the U.S., and for China, its about the chips and the distillation that undermines that advantage. This is an arms race.
- PunchyHamster 4mo ago
  > The only way the U.S. keeps that edge is to prevent distillation. For how long ? year ? how long till model that is year behind will be fine for 90%+ use cases ?
  - dofm 4mo ago
    Putting aside agentic coding, that is to say, if you judge LLMs as a consumer technology (an old-fashioned idea for the inward-looking tech industry admittedly), then open weights LLMs, even quite small ones like Gemma 4, can likely already satisfy 90% of applications with a bit of help from search and browse tools. Much of the arms race for better LLMs exists to satisfy only the IT industry's needs.
- davedx 4mo ago
  Define compute deficit? They've been bringing out open weight models competitive with frontier models. How could they do that if they had a compute deficit?
  - RugnirViking 4mo ago
    I believe this article is about the technique they may or may not have used.
  - rescbr 4mo ago
    If they need to divert inference resources to train models, this counts as a compute deficit to me. I'm using GLM-5.2 daily for my own stuff, and during Chinese business hours, specially on their afternoon, it's a festival of rate limits.
crazylogger 4mo ago
Chinese labs access Claude via API. Isn't it the black box method by definition?
sorenjan 4mo ago
Doesn't "real" distillation use the logits instead of the final tokens? I would classify this more like using a model to generate synthetic training data.
- 0xbadcafebee 4mo ago
  Distillation is a category of techniques which generally speaking all extract knowledge from a target model to feed into a new model. Logit distillation requires access to the source model layers; final token distillation doesn't. The former is more effective, but the latter can be done with generation tokens alone. This article explains the difference (and addresses "they're distillin' our models!!!"): https://dev.to/p0rt/how-model-distillation-actually-works-and-what-the-china-distilled-our-model-headlines-really-3o0o https://dev.to/p0rt/how-model-distillation-actually-works-an...
killerstorm 4mo ago
I'm sorry, but you got the terminology exactly backwards. Training on the answer is called supervised fine-tuning. Just for the sake of clarity: 0. Full distillation uses logits of the teacher model - that's much more information than the text itself. This is a kind of distillation used inside labs, but one can't distill Claude this way as logits are not available via API. 1. Supervised fine-tuning on synthetic data might be called blackbox distillation. I guess that's what you meant in your case (1). 2. Reinforcement learning (like RLAIF) uses least amount of information from the teacher, i.e. only few bits per task.
- 0xbadcafebee 4mo ago
  I agree I left out option 0, but I think the other two were presented correctly? - Black box distillation uses direct answers to questions and conversation style. This is less useful as you still have to do supervised fine-tuning on the answers, as they may be wrong, and don't lead to greater insights (which reinforcement learning does) - RLIAF relies on preferences and values to judge answers. These don't need supervised fine-tuning and help guide the new model to better answers rather than just correcting specific previously asked answers
  - killerstorm 4mo ago
    Well, I mean you mixed up "fine-tuning" and "reinforcement learning" a bit when describing these options. Regarding the value of these options, SFT communicates more information to the model being trained, but there's a risk of overfitting. So I'd guess they might use both - do a bit of SFT and then finish with RLAIF.
giancarlostoro 4mo ago
Heck, one of my favorite fine tuned copies of Qwen uses Opus 4.6 Reasoning distilled. I'm not sure where the maintainer is based out of, but me in the states, if I had the hardware to do similar things I would. Its like you say, basically everyone is doing it. It kind of makes sense to me too given that you can have roughly similar data, but your reasoning logic is what the real secret sauce is in my eyes. It doesn't matter if you know everything in the world, if you don't know how to reason with that information. https://huggingface.co/Jackrong/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled https://huggingface.co/Jackrong/Qwen3.5-27B-Claude-4.6-Opus-...
ALLTaken 4mo ago
They want to create a monopoly and destroy every competitor, before they got a chance to rival them. Why can't OSS software rival closed source software? It should be an open market, at least "somewhat", what's happening for real? EU providers will also get banned, if they reach or exceed US model capabilties? Closed source providers can close your account at a whim like and destroy your business and then use the data you supplied them to create a competitor (Meta, Google, OpenAI, Anthrophic).
- buellerbueller 4mo ago
  >They want to create a monopoly and destroy every competitor, before they got a chance to rival them. VC/Startup playbook 101.
  - zobzu 4mo ago
    also why cant i have my own airport, too big to fit in my backyard... you guys lol.
- deaton 4mo ago
  Well Zai's GLM 5.2 legitimately is a frontier-level model, though not quite parallel with Opus or Fable. Unfortunately, its too damn big to run locally for most people. Thats the bottleneck right now; the open-weight models exist but something capable of competing with the frontier models just can't run on anything normal yet.
catigula 4mo ago
Chinese companies are engaging in anti-competitive practices, as usual. They are rogue actors on the economic scene. If it were feasible, they'd be widely banned, and for good reason.
- amanaplanacanal 4mo ago
  Bringing more competition is "anti-competitive" now.
  - catigula 4mo ago
    Merely copying products that actual companies produce and making them cheaper is anti-competitive. There's no incentive for the products to be developed in the first place in a market if this is happening. This is why copy protections exist in civilized countries (not China and to a lesser extent India).
    - amanaplanacanal 4mo ago
      That's why IP was invented, but what IP are they infringing? Not patents, not copyright, not trademarks, so what? Making something cheaper than someone else isn't anti-competitive. There are a lot of businesses that do that. That's the very essence of competition.
      - catigula 4mo ago
        They're not merely 'making something cheaper'. They're actively violating the terms of service using deceptive means. This is a trade-secret/DMCA violation. You're not allowed to use computers to do stuff like that. It's against the law.
    - ahartmetz 4mo ago
      And look how this lack of protection ruining the Chinese economy, which isn't developing any products.
summarybot 4mo ago
It's about training data and using Claude to compare 2 outputs and have it indicate the better one. This gives you higher quality training data that you can use to train a fresh set of weights. Weights don't get adjusted on-the-fly, instead the dataset for training is improved and then you train a'fresh. And it's hard to detect because you're just asking the model which of these outputs for a given prompt is better? Or something along those lines.

dannyw 4mo ago

If you’re doing evals, you’re basically doing RLAIF without training a model; just looking at the results. Fundamentally it is very difficult to stop this while still making your AI models useful.

zmgsabst 4mo ago
Similarly, if you did a corpus study on bioRvix to summarize recent science findings — you could use the same questions and answers to fine tune a model. There is no way to communicate information at scale to companies through the API, for anything approaching a real application, without that information forming a corpus another model can be trained on. But it wouldn’t be the first time they broke a model: Their “guardrails” that cause it to reject user prompts also means it relies on its pop science summary of medicine to tell you why bioRxiv is wrong rather than accurately summarize the papers. They’ve successfully created a smug, argumentative average of the internet which refuses to even consider it might be wrong or that it’s reading a science paper which is based on measurements and not vibes — but why would I pay for that? I get it for free online.

janalsncm 4mo ago

Yeah I think the technical term is something more like “pseudo-labeling”. The OG distillation requires logits which Anthropic doesn’t provide.

mannanj 4mo ago

>But if you show them a jailbreak of their model that bypasses their safety, they'll tell you that any model can eventually be jailbroken so don't worry about safety. Yes this is in line with what Anthropic said in their public statements about their Fable access restriction by the government directive. The hypocrisy and inconsistency in their statements and behavior feels quite childish and controlling. I believe our companies and their leaders, friends among our other influential leaders and leaders from rich social classes, want to actively hurt most people as this behavior looks to be quite self-interested.

topato 4mo ago
Not to mention, the person who brought this quote unquote jailbreak to the Trump Administration was Amazon’s new CEO. They know their IPOs are coming up, so locking their competitors out of the U.S. (even if just for the weeks surrounding the IPO date) would be a major boon. The White House seems to love making announcements just for the sake of making the market move…. Coincidentally, right after POTUS buys a massive amount of the benefactory company’s stock (Buy Dell Computers, lol)

gmerc 4mo ago

https://research.nvidia.com/labs/lpr/slm-agents/ https://research.nvidia.com/labs/lpr/slm-agents/ - Distillation data is a natural byproduct of using these models. There's no effective defence against it. Anthropic is degrading thinking blocks to summaries to slow it down and hide model internals, but in the end, the math says you're SOL and it works on MNC/Large Corporate scale well enough that the moment cost becomes a priority, you're left without the lock in you need to keep customers paying.

alfiedotwtf 4mo ago
Byproduct? It’s essentially the only part of an LLM that is useful, because it’s the WHOLE product! It’s the same reason why DRM for audio and video is a non sequitur - if you want a person to see or hear audio or video, eventually at the end of the chain, it’s going to be converted to audio for the ear and light for the eyes - that’s why you attach your tap. Without a model generating tokens, what’s the point. So if Anthropic somehow disable quality token generation, what’s the point!
- TeMPOraL 4mo ago
  That's why the harness is moving server-side: because generating tokens is not the actual point of the model, not for the users. Especially with tool calling giving us agents that can act, most of the tokens generated are not, themselves, critical to the end users. Specifically, a lot of tokens goes into orchestrating actual tool calls, and then most "thinking tokens" are only relevant to users only in so far as they help users keep track of and verify what the LLM is doing. So all those tokens can be hidden or replaced by partial summaries, and all of that can happen server-side, and then there's very little to distill from.
  - TSiege 4mo ago
    I haven't heard of this happening, do you have links any explainers on this?
    - TeMPOraL 4mo ago

fnord77 4mo ago

Can you reach into the model and "transplant" weights directly?

antonvs 4mo ago
You can do things like that - one example is averaging weights between related models - but not with Anthropic's models, because outsiders don't have access to the weights.
- fulafel 4mo ago
  Weights are just data a server, so we don't know outsiders have access (either via breakin or arrangement).
  - antonvs 4mo ago
    Yes, obviously. That's not the point.
parineum 4mo ago
If you have access to the weights, you can just use them as is...
- HarHarVeryFunny 4mo ago
  Anthropic are not saying they have been hacked - they are saying that Alibaba have been sending lot of requests to their servers.
jorisw 4mo ago
No, you'd need to have the model on your filesystem for direct access, and then the architecture would need to be the same.
X-Ryl669 4mo ago
I'm not 100% sure it's not possible. If (I don't know) it's possible to freeze the temperature of the model so it's deterministic, and if you could make a map of produced words back to tokens (via HMM probably), then you can probably alter a minimal input and observe the output to model it. If you perform waves of such minimal alterations, you can expect to be able to locate the distance where each alteration impact the model (the idea being that a small alteration on output is likely due to the last layers of the models, and a small alteration is likely due to the deeper layer). Once you've located most of the last layer(s?) weights, you can try to solve for them. With a hundreds of billions weights model, the last layers will likely be so huge that it's probably unfeasible technically, but it's theoretically possible.

anon373839 4mo ago

> These complaints of distillation are inflating the problem to make it sound worse than it is Unfortunately, the Reuters piece itself is complicit in this dramatization. The lede paragraph parrots Anthropic's talking point that distillation is an "attack", without using quotes that would alert the reader that this framing is a corporate talking point. Distillation is NOT an attack.

p4coder 4mo ago
Agreed! I had to do a double take and check the URL. I thought I am reading a press release rather than actual reporting.
- da_grift_shift 4mo ago
  Same thing nowadays :^)
  - verisimi 4mo ago
    It always was.
- soperj 4mo ago
  That's exactly what they pay the publicist for.
- friendzis 4mo ago
  https://news.ycombinator.com/item?id=13155538 https://news.ycombinator.com/item?id=13155538
  - verdverm 4mo ago
    ironically, I think this is why the jobs apocolypse is overblown, Ai is only good at a thing if the people using it are also good at that thing, and people are attributing Ai as superhuman at things they do not know themselves

lemax 4mo ago

I've used RLAIF to build out heuristic based non-LLM models for various decision systems and achieved like, 95% F1 on certain projects. We're in a place where models can be used to fine tune a lot of stuff via loops.

handoflixue 4mo ago

> But if you show them a jailbreak of their model that bypasses their safety, they'll tell you that any model can eventually be jailbroken so don't worry about safety. They claim two things: 1) The specific, available jailbreak for Fable 5 is not dangerous - this has been confirmed by multiple experts, and there is no credible evidence against this claim (in other words, Anthropic is probably correct) 2) It is impossible to build an LLM that is immune to all jailbreaks. Again, there is no credible evidence against this claim, i.e. Anthropic is again entirely correct. If #1 was false, they could just publish the details of the jailbreak - it supposedly only works on Fable 5, so there's no possible danger. If #2 was false, surely some other LLM lab would have done it by now. Especially since a number of governments have made it clear there is a market for such a project.

mcintyre1994 4mo ago
Supposedly the details of the ‘jailbreak’ are that you give it insecure code and say “fix this code”, and it does, and then you ask it for test scripts and that’s effectively an exploit against the unfixed code. If true then I have no idea how anyone’s going to release a useful model that doesn’t have the same jailbreak. https://www.theregister.com/security/2026/06/15/feds-freaked-over-fable-5-after-simple-fix-this-code-prompt-not-jailbreak-says-researcher/5255827 https://www.theregister.com/security/2026/06/15/feds-freaked...
- handoflixue 4mo ago
  If that's the extent of the jailbreak, then the government should have banned every existing LLM - their story only makes sense if there's some Fable-specific capability that got unlocked.
  - le-mark 4mo ago
    There’s no logic to it, blocking fable was retaliation and market manipulation by the current admin, nothing more. Poorly conceived as well.
  - Rudybega 4mo ago

JumpCrisscross 4mo ago

> These complaints of distillation are inflating the problem They’re also missing the point. What would have happened to a member of the Manhattan Project who, through personal pursuit of profit, neglected their duty enough to let the bomb leak?

nixon_why69 4mo ago
The companies are all for-profit companies, its not like they're selling out some national security goal for profit, profit is the point. Anthropic already heavily restricts Chinese traffic but that only jams up researchers and regular Joes. Anyone motivated enough can hop a flight to Singapore with an nvme drive in their pocket.

cm2187 4mo ago

Stupid question: I was under the impression that these models were trained on PB of data. Surely the amount of questions/response they can extract from querying a bigger model (Claude) is fairly modest. How is it not a drop vs the training dataset?

eru 4mo ago
This might be like an observational study vs a study with a control?
- anon373839 4mo ago
  From what I understand, at this point, the main value of stronger model outputs is simply to bootstrap reasoning behavior during the RL post-training phase. It gets you past the “cold start” problem with RL, after which the outputs aren’t needed anymore. From then on, it’s hill climbing and that requires environments for the model to interact with get rewards from.
woctordho 4mo ago
Actually nowadays LLMs are only trained with TBs rather than PBs of data, and it's not too hard to find GBs of agent traces online.
reasonableklout 4mo ago
There are multiple stages of training, and the data/compute mix at each are quite different and produce different "layers" of intelligence. The pretraining stage is the first stage which consists of "next token prediction" on the entire internet, PB of tokens, etc. This is what most people think of when they think of training LLMs, however it produces a "base model" which is not really "intelligent", but rather much like a blurry JPEG of all human language and knowledge. You cannot really talk to such a model; it will simply complete your prompt by producing both sides of the conversation. Note however at some level the training has encoded enough structure through compression that it is able to simulate all sorts of phenomena, from human conversations to code. The great R&D difficulty here is to scale pretraining so that it can proceed smoothly in vast distributed datacenters in a fault-tolerant manner. The next few stages are collectively called post-training, and typically consist of supervised fine-tuning, then reinforcement learning. In supervised fine-tuning, the model is further trained to predict the next token, but on a much more focused data set of natural language conversations where the "assistant" and "user" turns are explicitly delineated with special tokens. The output of this stage is a model which is capable of carrying on proper conversations, but typically with no ability to creatively problem-solve, and less of a personality. The data and compute are many orders of magnitude smaller than in pretraining. The reinforcement learning stage used to be a small part of model training, but ever since AI-assisted coding took off, it has become larger and larger chunk of training. In recent models, the compute spend on RL has allegedly come to rival or even exceed that of pretraining [1], which is a bit scary because RL is classically what lead to sci-fi like AIs which are extremely good at accomplishing goals to the detriment of everything else. The way that RL works is that you put an instance of your model in some environment (such as a VM containing a git repository) and give it a task (such as fix the linked github issue). The model will then generate a bunch of attempts to solve the task which we call "trajectories", in most cases there is either an objective measure of the task success (such as passing the tests), or a fuzzy measure (such as having another LLM look at the results and provide a score). This is called the reward, and the model will learn slowly by producing trajectories that receive reward. It can actually be quite hard to prevent "reward hacking" from the model here and the rewards must be shaped very carefully, much R&D labor goes into here, as well as similar challenges to distributed pretraining. A significant challenge is that coding/knowledge work tasks these days are getting extremely difficult, we are far beyond 2024 days where models could barely solve the easiest problems in SWE-bench. Tasks at the frontier now look more like mini projects that would take humans multiple hours or even days to finish (or in some cases, research-style tasks that would be beyond reach for even top human experts, such as the Erdős unit distance problem which was posed in 1946 but wasn't solved until recently, by GPT-5.5). Huge amounts of trajectories must be produced, and huge amounts of them produce zero reward and therefore are useless for learning. Getting a cold start requires running tens of thousands of instances of your model in VMs in parallel for multiple days to produce trajectories, to say nothing of the GPU costs. So what do you do when you only have a model which is capable of basic conversations but cannot even begin to tackle basic coding tasks, use tools, etc? The approach that companies behind the frontier have decided on is to bootstrap their learning process by having an already extremely intelligent model such as Claude produce hundreds of thousands of seed trajectories for them. Then they can use this data to get a warm start and begin learning immediately. And if you use Claude for your reward model too, you get to skip the nastiness of reward shaping. Therefore, even if in number of raw tokens the data are much smaller than internet-scale pretraining data, the value that each token provides is far far greater. [1] For example, Grok 4 compute spend on RL was ~100% of that of pretraining: https://www.interconnects.ai/p/grok-4-an-o3-look-alike-in-search https://www.interconnects.ai/p/grok-4-an-o3-look-alike-in-se...

friendzis 4mo ago

> These complaints of distillation are inflating the problem to make it sound worse than it is This is, in part, a problem every judicial and legislative system has faced since forever: form versus function. Take a classic elicitation spying techniques: a foreign spy meets a military officer/scientist at a bar, strikes up a conversation, makes an observation wondering how could a missile hit some target at some accuracy and elicits a response that with laser guidance it is entirely possible. From this they get info that there is some technology to laser guide missiles. Or in retail, a competitor hiring a secret buyer for core baskets of goods and analyzing prices in the receipts. The function is espionage, the form is conversation and all info is in a sense provided willingly. Where do you pull the slider? These distillation "attacks" are not only indistinguishable from evals, they ARE evals. The function is own model training, the form is eval. Normally, one would expect to have risk benefit analysis based discussion which direction to push the legality slider to. The problem with these recurring statements is that they invoke enshitification of legislature.

SubiculumCode 4mo ago

The compute deficit of Chinese Ai companies is real, and it IS THE ONLY competitive advantage that Western companies have. The only way the U.S. keeps that edge is to prevent distillation. The only way Chinese companies can make up for the deficit in compute is to distill. There innovation in great supply on every side of the Ocean. Its about the chips. And in terms of national security, for the U.S., and for China, its about the chips and the distillation that undermines that advantage. This is an arms race.

PunchyHamster 4mo ago
> The only way the U.S. keeps that edge is to prevent distillation. For how long ? year ? how long till model that is year behind will be fine for 90%+ use cases ?
- dofm 4mo ago
  Putting aside agentic coding, that is to say, if you judge LLMs as a consumer technology (an old-fashioned idea for the inward-looking tech industry admittedly), then open weights LLMs, even quite small ones like Gemma 4, can likely already satisfy 90% of applications with a bit of help from search and browse tools. Much of the arms race for better LLMs exists to satisfy only the IT industry's needs.
davedx 4mo ago
Define compute deficit? They've been bringing out open weight models competitive with frontier models. How could they do that if they had a compute deficit?
- RugnirViking 4mo ago
  I believe this article is about the technique they may or may not have used.
- rescbr 4mo ago
  If they need to divert inference resources to train models, this counts as a compute deficit to me. I'm using GLM-5.2 daily for my own stuff, and during Chinese business hours, specially on their afternoon, it's a festival of rate limits.

crazylogger 4mo ago

Chinese labs access Claude via API. Isn't it the black box method by definition?

sorenjan 4mo ago

Doesn't "real" distillation use the logits instead of the final tokens? I would classify this more like using a model to generate synthetic training data.

0xbadcafebee 4mo ago
Distillation is a category of techniques which generally speaking all extract knowledge from a target model to feed into a new model. Logit distillation requires access to the source model layers; final token distillation doesn't. The former is more effective, but the latter can be done with generation tokens alone. This article explains the difference (and addresses "they're distillin' our models!!!"): https://dev.to/p0rt/how-model-distillation-actually-works-and-what-the-china-distilled-our-model-headlines-really-3o0o https://dev.to/p0rt/how-model-distillation-actually-works-an...

killerstorm 4mo ago

I'm sorry, but you got the terminology exactly backwards. Training on the answer is called supervised fine-tuning. Just for the sake of clarity: 0. Full distillation uses logits of the teacher model - that's much more information than the text itself. This is a kind of distillation used inside labs, but one can't distill Claude this way as logits are not available via API. 1. Supervised fine-tuning on synthetic data might be called blackbox distillation. I guess that's what you meant in your case (1). 2. Reinforcement learning (like RLAIF) uses least amount of information from the teacher, i.e. only few bits per task.

0xbadcafebee 4mo ago
I agree I left out option 0, but I think the other two were presented correctly? - Black box distillation uses direct answers to questions and conversation style. This is less useful as you still have to do supervised fine-tuning on the answers, as they may be wrong, and don't lead to greater insights (which reinforcement learning does) - RLIAF relies on preferences and values to judge answers. These don't need supervised fine-tuning and help guide the new model to better answers rather than just correcting specific previously asked answers
- killerstorm 4mo ago
  Well, I mean you mixed up "fine-tuning" and "reinforcement learning" a bit when describing these options. Regarding the value of these options, SFT communicates more information to the model being trained, but there's a risk of overfitting. So I'd guess they might use both - do a bit of SFT and then finish with RLAIF.

giancarlostoro 4mo ago

Heck, one of my favorite fine tuned copies of Qwen uses Opus 4.6 Reasoning distilled. I'm not sure where the maintainer is based out of, but me in the states, if I had the hardware to do similar things I would. Its like you say, basically everyone is doing it. It kind of makes sense to me too given that you can have roughly similar data, but your reasoning logic is what the real secret sauce is in my eyes. It doesn't matter if you know everything in the world, if you don't know how to reason with that information. https://huggingface.co/Jackrong/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled https://huggingface.co/Jackrong/Qwen3.5-27B-Claude-4.6-Opus-...

ALLTaken 4mo ago

They want to create a monopoly and destroy every competitor, before they got a chance to rival them. Why can't OSS software rival closed source software? It should be an open market, at least "somewhat", what's happening for real? EU providers will also get banned, if they reach or exceed US model capabilties? Closed source providers can close your account at a whim like and destroy your business and then use the data you supplied them to create a competitor (Meta, Google, OpenAI, Anthrophic).

buellerbueller 4mo ago
>They want to create a monopoly and destroy every competitor, before they got a chance to rival them. VC/Startup playbook 101.
- zobzu 4mo ago
  also why cant i have my own airport, too big to fit in my backyard... you guys lol.
deaton 4mo ago
Well Zai's GLM 5.2 legitimately is a frontier-level model, though not quite parallel with Opus or Fable. Unfortunately, its too damn big to run locally for most people. Thats the bottleneck right now; the open-weight models exist but something capable of competing with the frontier models just can't run on anything normal yet.

catigula 4mo ago

Chinese companies are engaging in anti-competitive practices, as usual. They are rogue actors on the economic scene. If it were feasible, they'd be widely banned, and for good reason.

amanaplanacanal 4mo ago
Bringing more competition is "anti-competitive" now.
- catigula 4mo ago
  Merely copying products that actual companies produce and making them cheaper is anti-competitive. There's no incentive for the products to be developed in the first place in a market if this is happening. This is why copy protections exist in civilized countries (not China and to a lesser extent India).
  - amanaplanacanal 4mo ago
    That's why IP was invented, but what IP are they infringing? Not patents, not copyright, not trademarks, so what? Making something cheaper than someone else isn't anti-competitive. There are a lot of businesses that do that. That's the very essence of competition.
    - catigula 4mo ago
      They're not merely 'making something cheaper'. They're actively violating the terms of service using deceptive means. This is a trade-secret/DMCA violation. You're not allowed to use computers to do stuff like that. It's against the law.
  - ahartmetz 4mo ago
    And look how this lack of protection ruining the Chinese economy, which isn't developing any products.

summarybot 4mo ago

It's about training data and using Claude to compare 2 outputs and have it indicate the better one. This gives you higher quality training data that you can use to train a fresh set of weights. Weights don't get adjusted on-the-fly, instead the dataset for training is improved and then you train a'fresh. And it's hard to detect because you're just asking the model which of these outputs for a given prompt is better? Or something along those lines.