5 ms·

Garry Tan wants US open-weight AI labs to 'distill' frontier models, too

412 points by TheJCDenton 13d ago

TheJCDenton 13d ago
> He also notes that the proprietary AI labs didn’t ask permission when they vacuumed up as much human knowledge as they could to train their models. I think this should desactivate the moral high ground from which Anthropic is trying to speak. That they would want to make distillation orderly IMHO is fair, but to make it illegal is very rich from any AI frontier lab, really.
- toomuchtodo 13d ago
  YC does better if its startups get open weight frontier benefits. Garry’s just advocating for his book, which is his job. Consider how much capital YC portfolio companies would have to burn until liquidity if they have to pay OpenAI and Anthropic, versus relying on open weight frontier capabilities.
  - SOLAR_FIELDS 13d ago
    If someone proposes the right thing for selfish reasons, do we call that bad? Or do we call it proper incentive alignment?
    - deleted 13d ago
      [deleted]
    - dofm 13d ago
      We used to call it enlightened self-interest.
  - visarga 13d ago
    > versus relying on open weight frontier capabilities ahem.. it happens even today, you can use open weight models directly and even fine tune
- deleted 13d ago
  [deleted]
- sobellian 13d ago
  I reflected on this myself recently. Model distillation seems to be at least as fair a use as distilling a book.
  - causal 13d ago
    More than fair if you consider that the tokens are paid for.
    - dathery 13d ago
      Both labs even explicitly promise the customer owns the outputs. It feels like they want to have their cake (ensure enterprises don't get spooked away from using as many LLMs as possible) while eating it too (still arguing some level of control over the outputs). > Ownership of content. As between you and OpenAI, and to the extent permitted by applicable law, you (a) retain your ownership rights in Input and (b) own the Output. We hereby assign to you all our right, title, and interest, if any, in and to Output. https://openai.com/policies/terms-of-use/ https://openai.com/policies/terms-of-use/ > As between the parties and to the extent permitted by applicable law, Anthropic agrees that Customer (a) retains all rights to its Inputs, and (b) owns its Outputs. Anthropic disclaims any rights it receives to the Customer Content under these Terms. Subject to Customer’s compliance with these Terms, Anthropic hereby assigns to Customer its right, title and interest (if any) in and to Outputs. https://www.anthropic.com/legal/commercial-terms https://www.anthropic.com/legal/commercial-terms Obviously there is some bad behavior going on in the distillation scene with gray-market token resellers but that is "just" normal fraud.
      - zenoprax 13d ago
        > Both labs even explicitly promise the customer owns the outputs. > to the extent permitted by applicable law, you (a) retain your ownership rights in Input and (b) own the Output If the argument is that the model itself is under copyright protection then "as permitted by applicable law" would be doing some heavy lifting. Assuming that were true, given that locally-run LLMs exist, what would be illegal: the distillation itself or the provision of service of the distilled model?
- Bluestein 13d ago
  Also, as said elsewhere: "Lab" is rich here, for outfits that, facing these giant, energy swallowing black boxes have really no clue what's going on inside.- The moniker gives them an air of scientific, knowledgeable, tranquil, pro-social, pro bono work.- Of course they are entitled to kill off a few mice, or pillage the commons to forward their "lab" work.-
  - danny_codes 12d ago
    Obviously lab just means company. They use it because it confuses the public into thinking they are doing research primarily. Which of course is nonsense.
  - travisgriggs 13d ago
    We also associate laboratories with evil scientists and Frankenstein and the like. I can just hear Boris Karloff (er Bobby Picket) uttering “I was working in the lab late one night. When my eyes beheld an eerie sight… … … …the monster mash”. If anything, I associate _uncertainty_ with labs. The result is never known up front, they’re a place of discovery. But I get your meaning. What should they be called instead? AI Sausage Factories maybe (cue Upton Sinclair?)?
    - whatshisface 13d ago
      By that definition, wall street would be a lab, and so would be a casino. I guess we could call them, "data refineries."
      - Bluestein 13d ago
        Refinery makes a lot of sense. I like it particularly because it raises the question of whose (whose) "oil" (data) it is they are purloining.-
    - Avicebron 13d ago
      > What should they be called instead? AI Sausage Factories maybe (cue Upton Sinclair?)? That's actually great? Slaughterhouses killing off the collective genius of humanity and grinding it into a bland paste for mass consumption.
- deleted 13d ago
  [deleted]
kelnos 13d ago
I agree. The frontier models are based on training data from tons of copyrighted work. Some of that work was obtained illegally, even. They could not exist without strip-mining the commons. The labs have no moral or ethical ownership to the end result, and others should feel free to treat any company-imposed restrictions on their use as invalid. I don't expect Tan's position to be based on any kind of real moral high ground, but his conclusion is correct. I love the "illicit distillation attacks" framing from the incumbents. There's nothing illicit. There's no attack. You just don't like it because it threatens your market position and business model.
- knollimar 13d ago
  I'm sure they put some BS in their TOS
  - stymaar 13d ago
    I'm also certain that they violated countless ToS when they scrapped the internet for training purpose.
    - knollimar 13d ago
      Ethically sure but that doesn't mean taking from them is nothing "illicit".
      - deleted 13d ago
        [deleted]
- mobelkh 13d ago
  why can't I use the tokens i paid for anyway?
- giancarlostoro 13d ago
  Abolish copyright and make it less ridiculous. Sampling music was never a thing that required royalties until the 1990s when I guess someone got angry that rappers were making money off their sampled music. Its insane to me. Make it illegal to transfer ownership of copyrighted work too, only the spouse or one single inheritor who isnt a company can have the rights transferred, after both die, the work enters public domain. LLMs should just pay a flat fee to use a specific book and thats it. Fees should be reasonable (not a million dollars per book), so long as the model doesnt spit out the entire book.
brcmthrowaway 13d ago
[flagged]
quicklywilliam 13d ago
I see it as analogous to companies building fiber in the public ROW during the last big infrastructure bubble. Under the Telecoms Act, these companies had to allow competitors to use their fiber at a fair price. Similarly, AI companies should be required to allow distillation at a fair price. Fair Use doesn’t make sense as a social contract if it only cuts one way!
- jimnotgym 13d ago
  But if they tried to set a fair price they would have to report how much money they are losing on each token sold. This might be bad for the real business of ai firms, hoovering up as much capital as they can
re-thc 13d ago
There were comparisons and Muse Spark is so very similar to Fable / Opus... so...
- nijave 13d ago
  Knowing Meta, I'd be more surprised if they _didn't_ distill frontier models than if they did...
pton_xd 13d ago
Agreed! Allow US companies to innovate by creating an ecosystem of smaller, more efficient open weight models and it will be a net benefit for everyone. Distillation is a good thing. Preventing token-consumers from developing competing products should be litigated as anti-competitive behavior.
ViktorRay 13d ago
https://youtu.be/ZIaOBAjvc38 https://youtu.be/ZIaOBAjvc38 Garry Tan and Sam Altman recently did this interview together. They seemed pretty friendly with each other during it. Wonder what Sam Altman would say about Tan advocating for OpenAI’s models to be distilled. Then again this is the same OpenAI that has gotten into legal trouble recently regarding Apple’s IP so who knows
zombiwoof 13d ago
[dead]
9865322689965 13d ago
[dead]
okasaki 13d ago
Like Gates saying there should be UBI, or Musk saying... well, whatever. They know it won't happen, so arguing for it is 'effectively free' and purely personal marketing. A bullshit game played by politicians and wannabes.
- seanmcdirmid 13d ago
  Gates probably honestly believes in UBI; the guy is practical to a fault but evil misleading genius he is not. I actually don’t see any better options than UBI long term.
  - wannabe44 13d ago
    Only ways to rise in a UBI society where AI is supposed to replace intellectual work is crime and prostitution. Smart people who want better lives than the average will have to get into crime.
    - seanmcdirmid 13d ago
      A UBI society doesn't mean jobs aren’t available. There most certainly will be jobs. But with UBI and universal healthcare, the jobs can pay whatever the market really demands. People always complain about the government subsidizing low Walmart wages for example, but with UBI that argument is moot. Liberalizing the labor market wouldn’t mean less jobs, it would mean more (we would also have to lean more on corporate and consumption taxes rather than taxes around employment which would also make employment easier).
      - andriy_koval 13d ago
        > we would also have to lean more on corporate and consumption taxes rather than taxes around employment which would also make employment easier I think the only way forward is wealth tax. Rich accumulated so much wealth already, that they don't need to put it to profitable businesses.
        13d ago
fmnxl 13d ago
If it were so easy why aren't the frontier labs doing it themselves?
- layer8 13d ago
  Distilled models are worse than the original, so you can’t fully compete. Also, if all frontier labs did that, there would be nothing left to distill from.
Hikikomori 13d ago
Garry also goes to Thiels silicon valley church.
layer8 13d ago
https://archive.ph/BnceE https://archive.ph/BnceE
zetazzed 13d ago
Ok, but how do the economics of this work? Based on its settlement, Anthropic paid an average of $3000 per work they scanned based on their settlement (https://tech-insider.org/au/anthropic-copyright-settlement-2026/ https://tech-insider.org/au/anthropic-copyright-settlement-2...). They and OpenAI pay billions per year for a mix of experts and normal people to label or create data. Why would they continue doing this if the value of this is immediately copied by open models? If your goal is to end the economics of generating and buying data for AI (and I recognize for some people this is really the goal) then sure, but if you want AI for various subfields of interest to continue improving then it's not workable. Back when people made arguments for software privacy, the argument was usually "big business will still pay and consumers wouldn't have paid anyways so it's ok for us to pirate" - I actually think that was fine for business software but terrible for indie games, whose market was 0% businesses. But in the AI case, it's not like they get to keep some of the value of their investment - it all gets cloned into models that businesses and consumers alike are happy to use. If someone knows how labs could continue to fund data creation and acquisition in this model, please do share!
- etdznots 13d ago
  They can’t they’re literally fucked, and it’s not society’s problem! The whole world doesn't have to bend over to make sure a couple of lunatics who believe they are building a doomsday weapon also have a viable business model
  - dofm 13d ago
    This made me laugh out loud but ain't it the truth.
  - tmvphil 11d ago
    Somehow people who think open models will lead to infinite abundance and people who think open models will kill AI development are both liking this post.
- kadoban 13d ago
  > Anthropic paid an average of $3000 per work they scanned based on their settlement Not sure you get to count breaking the law and getting in trouble in your cost-of-doing-business. That's a little too on the nose. You're basically arguing that a criminal syndicate must be allowed to continue and we're required to make their business model make sense?
etdznots 13d ago
This is all based on the delusion that Chinese labs are mindlessly distilling the frontier. I would love for a US lab to be at or near the frontier with an open weight model, but it’s going to take some serious elbow grease, and yes some distillation (which btw OAI, anthropic et al, also use distillation of other’s outputs in their training)
dvt 13d ago
I think OpenAI and Anthropic will go bust, or at least be scrapped for parts in the next 5 years or so. It's clear that the extreme cost used up for training is impossible to recoup, as inference is already being subsidized. It's also clear that, as Tan indicates, open-weight models will be (and basically already are) just as good as frontier models. It's all about the harness, baby. We will have two main forks in the road, and two new industries created: - AI hardware (NVidia/Cerebras/etc.), the equivalent of Intel/AMD - AI software (harnesses, assistants, etc.) the equivalent of Microsoft/Apple We already saw a glimmer of this with popularity of OpenClaw—the problem is that it's janky, hard to set up, inconsistent, and very hacker-esque. Imo "AI labs" will be a dying breed because there's no real money in the actual models if they get commoditized, which they already kind of are.
- FanaHOVA 13d ago
  If harness is all that matters, a co-developed harness + model stack + large compute availability advantage + massive distribution advantage with data for post training will win the market.
  - willy_k 13d ago
    Inb4 Apple buys OAI in 10 years and gets 75% of the consumer market.
- Legend2440 13d ago
  >inference is already being subsidized. Inference is not being subsidized and in fact has pretty high margins. Similar-sized open weight models on openrouter are 15x cheaper per token than the big labs. This should reflect the isolated cost of inference, since 3rd party hosts have no reason to subsidize and no training costs to amortize. Only datacenter buildout costs are being subsidized.
  - dvt 13d ago
    > Inference is not being subsidized and in fact has pretty high margins. I was referring to the "AI labs" here. Sam Altman himself conceded that OpenAI is losing money on the $200 subscription. Using open-weight/open-source models is indeed cheaper (and no reason for inference to be subsidized).
Edwinat23 13d ago
Freefire
gr_norm 13d ago
Society as a whole has paid into this technology: through the theft of its intellectual property, through having to deal with the pillaging of so many commons (digital or otherwise) by it, through skyrocketing energy and computing device prices, and even just through ordinary investment. Democratize the technology! At the very least, don't step in legally to prevent this from happening.
dofm 13d ago
Controlling what users and customers do with API calls to closed weight models feels constraining, and there’s a role government can play here to normalize the fact that access to intelligence that was trained on broad public access data should itself also be more a form of a public good than something locked away behind restrictive terms of service I do not agree with this man all that often, but that is very concisely put.
consumer451 13d ago
> To him, the true AI doomer scenario is for all the immense power of frontier AI to wind up in the hands of a single powerful, proprietary provider. “The nightmare scenario, the doomer scenario for AI is that there’s just one company,” he said. “It has the best access to capital. It has the best AI researchers. It runs away with it and suddenly there’s one company that’s monolithic. And that would be bad. Well yes, as I think I said in a previous comment, on the current trajectory OpenAI and Anthropic will really stop releasing models due to distillation and regulatory pressures. Then, they would eat all knowledge work themselves, which would be the end of YC.
sick_of_slop 13d ago
Frontier labs trained their models on the entirety of human knowledge and didn't ask permission. It's a "want" or "should" it's a moral imperative to distill their models.
neilv 13d ago
Given the short-term pragmatic, conflicted way that AI tech adoption is happening... won't encouraging distillation effectively taint the entire space of open weights models, with the undisclosed biases of a few models that are under the influence of parties (certain billionaires and politicians) known for aggression and duplicity, and not for admirable ethics? Following news of companies and projects increasingly moving to open weights models. As AI gets more central to society, we really need to know how the weights were determined. Open weights isn't just "free as in beer"; it can be "free as in the mystery drug that creepy guy chatting you up at the bar offered you". And maybe even he doesn't even know everything that went into the tablets, since he too was being worked, by an organ-theft ring who will be harvesting both of you tonight. That's an analogy to get your attention. Your LLM probably isn't going to steal your organs. But in the current environment, it does and will have ideological biases determined by those with direct and indirect influence over it. And there will be a massive market for commercial influence biases (look at how previous generations of adtech invaded almost all technology companies). And there's incentive for military and spying capabilities to be buried in the models, perhaps as long-term sleepers. Maybe some organized crime trojans, too, depending which model you pick up. In this low-trust environment of the current real world, we need genuine open source models, not closed "open weights", and not mindlessly distilling black boxes gifted by sketchy powerful interests.
deleted 13d ago
[deleted]
amelius 13d ago
Governments should be more concerned about the _people's_ personal data instead. Ban data brokers before you ban distillation.
- Legend2440 13d ago
  Unfortunately, the government doesn't want to ban data brokers because the government wants to buy from data brokers.
gnarlouse 13d ago
you want smaller models with comparable capabilities. for resource efficiency, market efficiency, environmental conservation.
YuechenLi 13d ago
Distilling frontier models is a brute force approach that rapidly hits diminishing returns after bootstrap because of the unevenness of the data. The simpler and more effective method is to have dedicated "teacher" frontier LLMs to generate targeted training data sets specifically for training new models and adjust on the fly based on feedback from the student model.
Betelbuddy 13d ago
https://news.ycombinator.com/item?id=49655978 https://news.ycombinator.com/item?id=49655978
matt3210 13d ago
Distillation is fair use
matt3210 13d ago
Net neutrality anyone? If AI is critical to getting work done in the modern era, its access should be guaranteed. Anyone banned from accessing frontier AI is being forcibly left behind. This includes distillation.
artk42 13d ago
I can't believe to hear such a wisdom from Garry Tan.
seydor 13d ago
They should be called speakeasys
hintymad 13d ago
> To him, the true AI doomer scenario is for all the immense power of frontier AI to wind up in the hands of a single powerful, proprietary provider. Isn't this exactly what Dario wanted? He thought he knew what's best for the humanity...
- nullbio 12d ago
  It is. Dario the Book Burner will not good what he wants, the world sees through him.
davidguetta 13d ago
Yes and there's even a stronger argument that we could REQUIRE frontier model to be open weight / open source. At the end of the day they were built from data that did not belong to them. So it would be fair that humanity REQUIRES to give back the output of that. It's a bit like the free software thing: you can still make money from it and providing service to it, but if you build it based on another free stuff the derivative should be free. Why not do the same for intelligence ?
xlbuttplug2 13d ago
Eventually the top labs are going to collude and simply not release their best models to the public (if they aren't doing that already).
- jobs_throwaway 13d ago
  Then the next tier of labs will be even closer to the frontier than present, and the top labs will lose their pricing power
  - xlbuttplug2 13d ago
    So far the next tier has only demonstrated that they can catch up to, but not necessarily leapfrog, what the top tier has put out publicly. I suspect the top labs will come up with a business model that doesn't involve handing out their secret sauce for everyone else to reverse engineer. Perhaps restricting their top models to select high paying government/enterprise contracts. Or maybe a bespoke "describe the problem and we'll solve it for you" type service.
    - danny_codes 12d ago
      I think it’s laughable to think OpenAI has some special sauce that can’t be replicated easily. They simply have asymmetric access to compute and the dollars to power it. Thats the only moat here. OpenAI doesn’t even know how their model works. Nobody knows how LLMs work. So it’s not like it’s technically difficult to replicate, just costly.
darepublic 13d ago
I cannot feel anything but schadenfreude regarding anthropic having its IP stolen from it. Bravo Chinese labs, bravo
jimmydoe 13d ago
some people did bad things, now instead of punishing those people, we want rest of people all do bad things, because that's only fair.
nijave 13d ago
Not a lawyer but distillation sounds like a transformative work. Same thing as Cliff Notes imo. In every other area of manufacturering and tech I can use a machine to build a new machine that competes with the original machine. Should Milwaukee be able to prevent DeWalt from using their drill to make a competing drill? Should Jetbrains ban Eclipse contributors from using their IDE?
- credit_guy 12d ago
  My guess is that when you sign up for either Anthropic or OpenAI, the terms of use specify you can't use their model for purpose A, B, C, D. For example, you can't use their model to try to build biological weapons, or to try to extort people, etc. Most likely there is language there that you can't use their models to train other models. It's as simple as that. You agree to those terms of use, or you don't use their models.
  - danny_codes 12d ago
    How can we prove intent? I’m sure a clever actor can disguise their prompt and simply claim the LLM suggested such and such on its own. It’s not like Anthropic or OpenAI have the faintest idea how their models actually work.
TZubiri 13d ago
I disagree, I think chinese distillation relies on making multiple accounts at a provider, signing Terms of Services and breaking them repeatedly, in addition to using fraud patterns like IP proxies and networks of credit cards. I think that software execs should not incentivize users or other execs to break Terms of Services, or contracts of any kind. An executive or manager of a company that breaks contracts is worth 0, there's no incentive to do business with them, if you know they will agree to doing or not doing something and then breaking that promise. The word of a businessman is their most valuable asset, Tan is signalling that he is either misinformed on what Chinese distillation consists of, or that it's ok to do it. FAQ: - "But the frontier models do bad things too" - An argument worthy of a 5 year old, one civil issue doesn't negate the other, bring it to a court if you have an actual claim against OAI or Claude, etc... - "Companies have the right to reverse engineer" - Ok, do it, but the moment you are creating 10K accounts in a Distributed fashion (Distributed as in the first D of DDoS), using IP proxies and stolen credit cards or your employees and employee family credit cards, you are not doing it because you believe you have a right, you are doing it despite not having a right to it. EDIT: Re(actually)reading the article, Tan's take is a bit more nuanced, he seems to be advocating for regulation to restrict the capacity of Foundation models to restrict usage, on the basis (or to the extent) that it was trained on public data, and therefore it belongs or attributes its success to a wealth of the commons. My pre-existing quip is against those that want to solve this as-is by breaking the ToS. I think that's a weak version of Free Software position, it's very weak to complain that some software is proprietary and want to use it anyway, the strong FS position is that you don't even want to use it if it's proprietary, you won't catch a FS activist pirating proprietary software, they just don't use it and develop alternatives. Similarly it's not a FS position to distill a proprietary model (where you still wouldn't have source code at any rate).
- danny_codes 12d ago
  Plenty of contracts end up being unenforceable.If people think they have a strong case for breaking a contract, they are welcome to do so and see if a judge or jury agrees.
testfrequency 13d ago
OAI and Anthropic remain the biggest heist ever in our lifetime. Genuinely fucking crazy we pay money for fast access to autocomplete of stolen human remains.
mlazos 13d ago
I don’t think appeals to morality or ethics are required for this. You paid for the LLM’s output, you should be allowed to use it how you wish. The only reason distillation is a dirty word is the AI labs trying to spread FUD to protect their non-existent moat.
sabhiram 12d ago
If frontier labs can distill the internet and all of our data, then we should be able to distill their models further too. The fact that billions were spent on research to distill the internet should not preclude others from spending 10s of thousands to do the same to these frontier labs. Time to create a bigger moat than "but we spent so much money doing this ...".
andsoitis 12d ago
> He elaborated to TechCrunch that this means he wants smaller, American open-weight AI labs to use the same kind of training techniques on American frontier AI labs, giving the U.S. a more robust set of open-weight options that aren’t Chinese. Those market entrants would face commodity pricing power vs. high capital costs, no? Maybe there'd be ROI but I think there's another layer or competitive dimension that's neither frontier lab nor distilled model lab.
nullbio 12d ago
DaaS - Distillation as a Service. It's a good idea. It's not their data to begin with, anyway.
rudicjri27 12d ago
There are a couple of ideas that are very clearly being drip fed into the consciousness - theft / distilling (ie cheaper open models that are not from the soon to IPO US corps cheated rather than innovated) - danger / nat security (ie we can only trust the soon to IPO US corps to shepherd us)
thedougd 12d ago
Better hurry up. The frontier labs are making their latest coordinated push for regulatory capture.

412 points by TheJCDenton 13d ago

> He also notes that the proprietary AI labs didn’t ask permission when they vacuumed up as much human knowledge as they could to train their models. I think this should desactivate the moral high ground from which Anthropic is trying to speak. That they would want to make distillation orderly IMHO is fair, but to make it illegal is very rich from any AI frontier lab, really.

toomuchtodo 13d ago
YC does better if its startups get open weight frontier benefits. Garry’s just advocating for his book, which is his job. Consider how much capital YC portfolio companies would have to burn until liquidity if they have to pay OpenAI and Anthropic, versus relying on open weight frontier capabilities.
- SOLAR_FIELDS 13d ago
  If someone proposes the right thing for selfish reasons, do we call that bad? Or do we call it proper incentive alignment?
  - deleted 13d ago
    [deleted]
  - dofm 13d ago
    We used to call it enlightened self-interest.
- visarga 13d ago
  > versus relying on open weight frontier capabilities ahem.. it happens even today, you can use open weight models directly and even fine tune
deleted 13d ago
[deleted]
sobellian 13d ago
I reflected on this myself recently. Model distillation seems to be at least as fair a use as distilling a book.
- causal 13d ago
  More than fair if you consider that the tokens are paid for.
  - dathery 13d ago
    Both labs even explicitly promise the customer owns the outputs. It feels like they want to have their cake (ensure enterprises don't get spooked away from using as many LLMs as possible) while eating it too (still arguing some level of control over the outputs). > Ownership of content. As between you and OpenAI, and to the extent permitted by applicable law, you (a) retain your ownership rights in Input and (b) own the Output. We hereby assign to you all our right, title, and interest, if any, in and to Output. https://openai.com/policies/terms-of-use/ https://openai.com/policies/terms-of-use/ > As between the parties and to the extent permitted by applicable law, Anthropic agrees that Customer (a) retains all rights to its Inputs, and (b) owns its Outputs. Anthropic disclaims any rights it receives to the Customer Content under these Terms. Subject to Customer’s compliance with these Terms, Anthropic hereby assigns to Customer its right, title and interest (if any) in and to Outputs. https://www.anthropic.com/legal/commercial-terms https://www.anthropic.com/legal/commercial-terms Obviously there is some bad behavior going on in the distillation scene with gray-market token resellers but that is "just" normal fraud.
    - zenoprax 13d ago
      > Both labs even explicitly promise the customer owns the outputs. > to the extent permitted by applicable law, you (a) retain your ownership rights in Input and (b) own the Output If the argument is that the model itself is under copyright protection then "as permitted by applicable law" would be doing some heavy lifting. Assuming that were true, given that locally-run LLMs exist, what would be illegal: the distillation itself or the provision of service of the distilled model?
Bluestein 13d ago
Also, as said elsewhere: "Lab" is rich here, for outfits that, facing these giant, energy swallowing black boxes have really no clue what's going on inside.- The moniker gives them an air of scientific, knowledgeable, tranquil, pro-social, pro bono work.- Of course they are entitled to kill off a few mice, or pillage the commons to forward their "lab" work.-
- danny_codes 12d ago
  Obviously lab just means company. They use it because it confuses the public into thinking they are doing research primarily. Which of course is nonsense.
- travisgriggs 13d ago
  We also associate laboratories with evil scientists and Frankenstein and the like. I can just hear Boris Karloff (er Bobby Picket) uttering “I was working in the lab late one night. When my eyes beheld an eerie sight… … … …the monster mash”. If anything, I associate _uncertainty_ with labs. The result is never known up front, they’re a place of discovery. But I get your meaning. What should they be called instead? AI Sausage Factories maybe (cue Upton Sinclair?)?
  - whatshisface 13d ago
    By that definition, wall street would be a lab, and so would be a casino. I guess we could call them, "data refineries."
    - Bluestein 13d ago
      Refinery makes a lot of sense. I like it particularly because it raises the question of whose (whose) "oil" (data) it is they are purloining.-
  - Avicebron 13d ago
    > What should they be called instead? AI Sausage Factories maybe (cue Upton Sinclair?)? That's actually great? Slaughterhouses killing off the collective genius of humanity and grinding it into a bland paste for mass consumption.
deleted 13d ago
[deleted]

kelnos 13d ago

I agree. The frontier models are based on training data from tons of copyrighted work. Some of that work was obtained illegally, even. They could not exist without strip-mining the commons. The labs have no moral or ethical ownership to the end result, and others should feel free to treat any company-imposed restrictions on their use as invalid. I don't expect Tan's position to be based on any kind of real moral high ground, but his conclusion is correct. I love the "illicit distillation attacks" framing from the incumbents. There's nothing illicit. There's no attack. You just don't like it because it threatens your market position and business model.

knollimar 13d ago
I'm sure they put some BS in their TOS
- stymaar 13d ago
  I'm also certain that they violated countless ToS when they scrapped the internet for training purpose.
  - knollimar 13d ago
    Ethically sure but that doesn't mean taking from them is nothing "illicit".
    - deleted 13d ago
      [deleted]
mobelkh 13d ago
why can't I use the tokens i paid for anyway?
giancarlostoro 13d ago
Abolish copyright and make it less ridiculous. Sampling music was never a thing that required royalties until the 1990s when I guess someone got angry that rappers were making money off their sampled music. Its insane to me. Make it illegal to transfer ownership of copyrighted work too, only the spouse or one single inheritor who isnt a company can have the rights transferred, after both die, the work enters public domain. LLMs should just pay a flat fee to use a specific book and thats it. Fees should be reasonable (not a million dollars per book), so long as the model doesnt spit out the entire book.

brcmthrowaway 13d ago

[flagged]

quicklywilliam 13d ago

I see it as analogous to companies building fiber in the public ROW during the last big infrastructure bubble. Under the Telecoms Act, these companies had to allow competitors to use their fiber at a fair price. Similarly, AI companies should be required to allow distillation at a fair price. Fair Use doesn’t make sense as a social contract if it only cuts one way!

jimnotgym 13d ago
But if they tried to set a fair price they would have to report how much money they are losing on each token sold. This might be bad for the real business of ai firms, hoovering up as much capital as they can

re-thc 13d ago

There were comparisons and Muse Spark is so very similar to Fable / Opus... so...

nijave 13d ago
Knowing Meta, I'd be more surprised if they _didn't_ distill frontier models than if they did...

pton_xd 13d ago

Agreed! Allow US companies to innovate by creating an ecosystem of smaller, more efficient open weight models and it will be a net benefit for everyone. Distillation is a good thing. Preventing token-consumers from developing competing products should be litigated as anti-competitive behavior.

ViktorRay 13d ago

https://youtu.be/ZIaOBAjvc38 https://youtu.be/ZIaOBAjvc38 Garry Tan and Sam Altman recently did this interview together. They seemed pretty friendly with each other during it. Wonder what Sam Altman would say about Tan advocating for OpenAI’s models to be distilled. Then again this is the same OpenAI that has gotten into legal trouble recently regarding Apple’s IP so who knows

zombiwoof 13d ago

[dead]

9865322689965 13d ago

[dead]

okasaki 13d ago

Like Gates saying there should be UBI, or Musk saying... well, whatever. They know it won't happen, so arguing for it is 'effectively free' and purely personal marketing. A bullshit game played by politicians and wannabes.

seanmcdirmid 13d ago
Gates probably honestly believes in UBI; the guy is practical to a fault but evil misleading genius he is not. I actually don’t see any better options than UBI long term.
- wannabe44 13d ago
  Only ways to rise in a UBI society where AI is supposed to replace intellectual work is crime and prostitution. Smart people who want better lives than the average will have to get into crime.
  - seanmcdirmid 13d ago
    A UBI society doesn't mean jobs aren’t available. There most certainly will be jobs. But with UBI and universal healthcare, the jobs can pay whatever the market really demands. People always complain about the government subsidizing low Walmart wages for example, but with UBI that argument is moot. Liberalizing the labor market wouldn’t mean less jobs, it would mean more (we would also have to lean more on corporate and consumption taxes rather than taxes around employment which would also make employment easier).
    - andriy_koval 13d ago
      > we would also have to lean more on corporate and consumption taxes rather than taxes around employment which would also make employment easier I think the only way forward is wealth tax. Rich accumulated so much wealth already, that they don't need to put it to profitable businesses.
      - 13d ago

fmnxl 13d ago

If it were so easy why aren't the frontier labs doing it themselves?

layer8 13d ago
Distilled models are worse than the original, so you can’t fully compete. Also, if all frontier labs did that, there would be nothing left to distill from.

Hikikomori 13d ago

Garry also goes to Thiels silicon valley church.

layer8 13d ago

https://archive.ph/BnceE https://archive.ph/BnceE

zetazzed 13d ago

Ok, but how do the economics of this work? Based on its settlement, Anthropic paid an average of $3000 per work they scanned based on their settlement (https://tech-insider.org/au/anthropic-copyright-settlement-2026/ https://tech-insider.org/au/anthropic-copyright-settlement-2...). They and OpenAI pay billions per year for a mix of experts and normal people to label or create data. Why would they continue doing this if the value of this is immediately copied by open models? If your goal is to end the economics of generating and buying data for AI (and I recognize for some people this is really the goal) then sure, but if you want AI for various subfields of interest to continue improving then it's not workable. Back when people made arguments for software privacy, the argument was usually "big business will still pay and consumers wouldn't have paid anyways so it's ok for us to pirate" - I actually think that was fine for business software but terrible for indie games, whose market was 0% businesses. But in the AI case, it's not like they get to keep some of the value of their investment - it all gets cloned into models that businesses and consumers alike are happy to use. If someone knows how labs could continue to fund data creation and acquisition in this model, please do share!

etdznots 13d ago
They can’t they’re literally fucked, and it’s not society’s problem! The whole world doesn't have to bend over to make sure a couple of lunatics who believe they are building a doomsday weapon also have a viable business model
- dofm 13d ago
  This made me laugh out loud but ain't it the truth.
- tmvphil 11d ago
  Somehow people who think open models will lead to infinite abundance and people who think open models will kill AI development are both liking this post.
kadoban 13d ago
> Anthropic paid an average of $3000 per work they scanned based on their settlement Not sure you get to count breaking the law and getting in trouble in your cost-of-doing-business. That's a little too on the nose. You're basically arguing that a criminal syndicate must be allowed to continue and we're required to make their business model make sense?

etdznots 13d ago

This is all based on the delusion that Chinese labs are mindlessly distilling the frontier. I would love for a US lab to be at or near the frontier with an open weight model, but it’s going to take some serious elbow grease, and yes some distillation (which btw OAI, anthropic et al, also use distillation of other’s outputs in their training)

dvt 13d ago

I think OpenAI and Anthropic will go bust, or at least be scrapped for parts in the next 5 years or so. It's clear that the extreme cost used up for training is impossible to recoup, as inference is already being subsidized. It's also clear that, as Tan indicates, open-weight models will be (and basically already are) just as good as frontier models. It's all about the harness, baby. We will have two main forks in the road, and two new industries created: - AI hardware (NVidia/Cerebras/etc.), the equivalent of Intel/AMD - AI software (harnesses, assistants, etc.) the equivalent of Microsoft/Apple We already saw a glimmer of this with popularity of OpenClaw—the problem is that it's janky, hard to set up, inconsistent, and very hacker-esque. Imo "AI labs" will be a dying breed because there's no real money in the actual models if they get commoditized, which they already kind of are.

FanaHOVA 13d ago
If harness is all that matters, a co-developed harness + model stack + large compute availability advantage + massive distribution advantage with data for post training will win the market.
- willy_k 13d ago
  Inb4 Apple buys OAI in 10 years and gets 75% of the consumer market.
Legend2440 13d ago
>inference is already being subsidized. Inference is not being subsidized and in fact has pretty high margins. Similar-sized open weight models on openrouter are 15x cheaper per token than the big labs. This should reflect the isolated cost of inference, since 3rd party hosts have no reason to subsidize and no training costs to amortize. Only datacenter buildout costs are being subsidized.
- dvt 13d ago
  > Inference is not being subsidized and in fact has pretty high margins. I was referring to the "AI labs" here. Sam Altman himself conceded that OpenAI is losing money on the $200 subscription. Using open-weight/open-source models is indeed cheaper (and no reason for inference to be subsidized).

Edwinat23 13d ago

Freefire

gr_norm 13d ago

Society as a whole has paid into this technology: through the theft of its intellectual property, through having to deal with the pillaging of so many commons (digital or otherwise) by it, through skyrocketing energy and computing device prices, and even just through ordinary investment. Democratize the technology! At the very least, don't step in legally to prevent this from happening.

dofm 13d ago

Controlling what users and customers do with API calls to closed weight models feels constraining, and there’s a role government can play here to normalize the fact that access to intelligence that was trained on broad public access data should itself also be more a form of a public good than something locked away behind restrictive terms of service I do not agree with this man all that often, but that is very concisely put.

consumer451 13d ago

> To him, the true AI doomer scenario is for all the immense power of frontier AI to wind up in the hands of a single powerful, proprietary provider. “The nightmare scenario, the doomer scenario for AI is that there’s just one company,” he said. “It has the best access to capital. It has the best AI researchers. It runs away with it and suddenly there’s one company that’s monolithic. And that would be bad. Well yes, as I think I said in a previous comment, on the current trajectory OpenAI and Anthropic will really stop releasing models due to distillation and regulatory pressures. Then, they would eat all knowledge work themselves, which would be the end of YC.

sick_of_slop 13d ago

Frontier labs trained their models on the entirety of human knowledge and didn't ask permission. It's a "want" or "should" it's a moral imperative to distill their models.

neilv 13d ago

Given the short-term pragmatic, conflicted way that AI tech adoption is happening... won't encouraging distillation effectively taint the entire space of open weights models, with the undisclosed biases of a few models that are under the influence of parties (certain billionaires and politicians) known for aggression and duplicity, and not for admirable ethics? Following news of companies and projects increasingly moving to open weights models. As AI gets more central to society, we really need to know how the weights were determined. Open weights isn't just "free as in beer"; it can be "free as in the mystery drug that creepy guy chatting you up at the bar offered you". And maybe even he doesn't even know everything that went into the tablets, since he too was being worked, by an organ-theft ring who will be harvesting both of you tonight. That's an analogy to get your attention. Your LLM probably isn't going to steal your organs. But in the current environment, it does and will have ideological biases determined by those with direct and indirect influence over it. And there will be a massive market for commercial influence biases (look at how previous generations of adtech invaded almost all technology companies). And there's incentive for military and spying capabilities to be buried in the models, perhaps as long-term sleepers. Maybe some organized crime trojans, too, depending which model you pick up. In this low-trust environment of the current real world, we need genuine open source models, not closed "open weights", and not mindlessly distilling black boxes gifted by sketchy powerful interests.

deleted 13d ago

[deleted]

amelius 13d ago

Governments should be more concerned about the _people's_ personal data instead. Ban data brokers before you ban distillation.

Legend2440 13d ago
Unfortunately, the government doesn't want to ban data brokers because the government wants to buy from data brokers.

gnarlouse 13d ago

you want smaller models with comparable capabilities. for resource efficiency, market efficiency, environmental conservation.

YuechenLi 13d ago

Distilling frontier models is a brute force approach that rapidly hits diminishing returns after bootstrap because of the unevenness of the data. The simpler and more effective method is to have dedicated "teacher" frontier LLMs to generate targeted training data sets specifically for training new models and adjust on the fly based on feedback from the student model.

Betelbuddy 13d ago

https://news.ycombinator.com/item?id=49655978 https://news.ycombinator.com/item?id=49655978

matt3210 13d ago

Distillation is fair use

matt3210 13d ago

Net neutrality anyone? If AI is critical to getting work done in the modern era, its access should be guaranteed. Anyone banned from accessing frontier AI is being forcibly left behind. This includes distillation.

artk42 13d ago

I can't believe to hear such a wisdom from Garry Tan.

seydor 13d ago

They should be called speakeasys

hintymad 13d ago

> To him, the true AI doomer scenario is for all the immense power of frontier AI to wind up in the hands of a single powerful, proprietary provider. Isn't this exactly what Dario wanted? He thought he knew what's best for the humanity...

nullbio 12d ago
It is. Dario the Book Burner will not good what he wants, the world sees through him.

davidguetta 13d ago

Yes and there's even a stronger argument that we could REQUIRE frontier model to be open weight / open source. At the end of the day they were built from data that did not belong to them. So it would be fair that humanity REQUIRES to give back the output of that. It's a bit like the free software thing: you can still make money from it and providing service to it, but if you build it based on another free stuff the derivative should be free. Why not do the same for intelligence ?

xlbuttplug2 13d ago

Eventually the top labs are going to collude and simply not release their best models to the public (if they aren't doing that already).

jobs_throwaway 13d ago
Then the next tier of labs will be even closer to the frontier than present, and the top labs will lose their pricing power
- xlbuttplug2 13d ago
  So far the next tier has only demonstrated that they can catch up to, but not necessarily leapfrog, what the top tier has put out publicly. I suspect the top labs will come up with a business model that doesn't involve handing out their secret sauce for everyone else to reverse engineer. Perhaps restricting their top models to select high paying government/enterprise contracts. Or maybe a bespoke "describe the problem and we'll solve it for you" type service.
  - danny_codes 12d ago
    I think it’s laughable to think OpenAI has some special sauce that can’t be replicated easily. They simply have asymmetric access to compute and the dollars to power it. Thats the only moat here. OpenAI doesn’t even know how their model works. Nobody knows how LLMs work. So it’s not like it’s technically difficult to replicate, just costly.

darepublic 13d ago

I cannot feel anything but schadenfreude regarding anthropic having its IP stolen from it. Bravo Chinese labs, bravo

jimmydoe 13d ago

some people did bad things, now instead of punishing those people, we want rest of people all do bad things, because that's only fair.

nijave 13d ago

Not a lawyer but distillation sounds like a transformative work. Same thing as Cliff Notes imo. In every other area of manufacturering and tech I can use a machine to build a new machine that competes with the original machine. Should Milwaukee be able to prevent DeWalt from using their drill to make a competing drill? Should Jetbrains ban Eclipse contributors from using their IDE?

credit_guy 12d ago
My guess is that when you sign up for either Anthropic or OpenAI, the terms of use specify you can't use their model for purpose A, B, C, D. For example, you can't use their model to try to build biological weapons, or to try to extort people, etc. Most likely there is language there that you can't use their models to train other models. It's as simple as that. You agree to those terms of use, or you don't use their models.
- danny_codes 12d ago
  How can we prove intent? I’m sure a clever actor can disguise their prompt and simply claim the LLM suggested such and such on its own. It’s not like Anthropic or OpenAI have the faintest idea how their models actually work.

TZubiri 13d ago

I disagree, I think chinese distillation relies on making multiple accounts at a provider, signing Terms of Services and breaking them repeatedly, in addition to using fraud patterns like IP proxies and networks of credit cards. I think that software execs should not incentivize users or other execs to break Terms of Services, or contracts of any kind. An executive or manager of a company that breaks contracts is worth 0, there's no incentive to do business with them, if you know they will agree to doing or not doing something and then breaking that promise. The word of a businessman is their most valuable asset, Tan is signalling that he is either misinformed on what Chinese distillation consists of, or that it's ok to do it. FAQ: - "But the frontier models do bad things too" - An argument worthy of a 5 year old, one civil issue doesn't negate the other, bring it to a court if you have an actual claim against OAI or Claude, etc... - "Companies have the right to reverse engineer" - Ok, do it, but the moment you are creating 10K accounts in a Distributed fashion (Distributed as in the first D of DDoS), using IP proxies and stolen credit cards or your employees and employee family credit cards, you are not doing it because you believe you have a right, you are doing it despite not having a right to it. EDIT: Re(actually)reading the article, Tan's take is a bit more nuanced, he seems to be advocating for regulation to restrict the capacity of Foundation models to restrict usage, on the basis (or to the extent) that it was trained on public data, and therefore it belongs or attributes its success to a wealth of the commons. My pre-existing quip is against those that want to solve this as-is by breaking the ToS. I think that's a weak version of Free Software position, it's very weak to complain that some software is proprietary and want to use it anyway, the strong FS position is that you don't even want to use it if it's proprietary, you won't catch a FS activist pirating proprietary software, they just don't use it and develop alternatives. Similarly it's not a FS position to distill a proprietary model (where you still wouldn't have source code at any rate).

danny_codes 12d ago
Plenty of contracts end up being unenforceable.If people think they have a strong case for breaking a contract, they are welcome to do so and see if a judge or jury agrees.

testfrequency 13d ago

OAI and Anthropic remain the biggest heist ever in our lifetime. Genuinely fucking crazy we pay money for fast access to autocomplete of stolen human remains.

mlazos 13d ago

I don’t think appeals to morality or ethics are required for this. You paid for the LLM’s output, you should be allowed to use it how you wish. The only reason distillation is a dirty word is the AI labs trying to spread FUD to protect their non-existent moat.

sabhiram 12d ago

If frontier labs can distill the internet and all of our data, then we should be able to distill their models further too. The fact that billions were spent on research to distill the internet should not preclude others from spending 10s of thousands to do the same to these frontier labs. Time to create a bigger moat than "but we spent so much money doing this ...".

andsoitis 12d ago

> He elaborated to TechCrunch that this means he wants smaller, American open-weight AI labs to use the same kind of training techniques on American frontier AI labs, giving the U.S. a more robust set of open-weight options that aren’t Chinese. Those market entrants would face commodity pricing power vs. high capital costs, no? Maybe there'd be ROI but I think there's another layer or competitive dimension that's neither frontier lab nor distilled model lab.

nullbio 12d ago

DaaS - Distillation as a Service. It's a good idea. It's not their data to begin with, anyway.

rudicjri27 12d ago

There are a couple of ideas that are very clearly being drip fed into the consciousness - theft / distilling (ie cheaper open models that are not from the soon to IPO US corps cheated rather than innovated) - danger / nat security (ie we can only trust the soon to IPO US corps to shepherd us)

thedougd 12d ago

Better hurry up. The frontier labs are making their latest coordinated push for regulatory capture.