4 ms·

Who is going to fund it? Training is unfathomably expensive. You have either VC funded models looking for a return on investment, or CCP funded models looking

Who is going to fund it? Training is unfathomably expensive. You have either VC funded models looking for a return on investment, or CCP funded models looking to solidify authoritarian "model Chinese society". Maybe there are some university 4B models, but I doubt those will carry far.

brcmthrowaway 5mo ago
Who funds Semiconductor fabs
- nullbio 5mo ago
  When Jensen (Nvidia) was doing interviews at his recent public talks, he was asked something along the lines of: "Why release these new laptops which are a low margin market, if your other businesses are vastly more profitable?" and his answer was basically that if they can build the coolest and best technology and push the frontier, they will do it. It's not all about making tons of money. He seemed genuinely excited about the tech. It highlights the difference between companies like Nvidia and Anthropic to me, where one is clearly all about the money and power, and the other is doing it because they genuinely want to accelerate progress and make cool stuff as the driving factor. It's no surprise therefore, that Nvidia is the worlds largest open-source contributor to AI, with over 800 open-weight models. Of course, these models run on Nvidia hardware, so they benefit from it as a company. But with that healthy mindset, they found a way to contribute that not only benefits everyone, but also benefits themselves. Contrast to Anthropic, who has gone the complete opposite direction. Closed off everything, restricting everything, fearmongering progress, regulatory capture attempts, the list goes on. I mean, they won't even agree on using AGENTS.md as a standard because CLAUDE.md is free marketing for them. That's the level of disgusting greed we are dealing with... From a game theory perspective, the cooperative strategies tend to win. As a result, Nvidia has set themselves up for a lifetime. Anthropic however, is playing a strategy of winner takes all, and they're happy to see the world and the entire AI industry collapse in the process.
  - cwnyth 5mo ago
    That's not really the impression I get from Anthropic, but if you have the links to back it up, I'm always willing to change my mind. Compared to bizes like Oracle, Microsoft, or Facebook, I felt that Anthropic was more interested in progress (not to the neglect of business―AI training is expensive at the end of the day), but maybe I've just not seen what you've seen.
    - nullbio 5mo ago
      https://clawd.rip https://clawd.rip
  - SXX 5mo ago
    Nvidia and "open source" is like opposite things. Nvidia only ever opened stuff that helps their bottom line or improve vendor lock-in. But yeah they are good shovel seller and competitor to actually evil companies that literally wants to eat all the world chips and energy supply.
    - zozbot234 5mo ago
      In the open source space, the Nemotron models from nVidia are quite real. Including a Nemotron Ultra variety meant to be large enough for near-SOTA.
      - SXX 5mo ago
        Nvidia not doing it out of goodness of their hearts and love to open source. If at anynpoint their CUDA vendor lock-in moat will faik because Intel or AMD manage to get working software they'll return to keep everything locked and proprietary ASAP. Basically everything Nvidia does in open source is there to make sure their proprietary stack have a good moat and no competitor stack can catch up.
    - nullbio 5mo ago
      Strongly disagree: https://build.nvidia.com/models https://build.nvidia.com/models Their license terms are also incredibly generous and allow commercial use, modification, etc, at no cost.
cwnyth 5mo ago
Ever calculate the cost of a computer in the 1960s, adjusted for inflation? Training is unfathomably expensive right now. What if a bunch of universities pooled their money? Or a bunch of nations pooled their money? Breakthroughs will eventually happen, optimization will occur, etc. People questioned whether there could ever be a viable open source operating system, yet Linux has been a viable option for a desktop environment for decades now, and that's not to mention its ubiquitous use as a server or phone OS.
- kamaal 5mo ago
  Yes, You have to start some where. Im guessing, making progress also brings in new ideas how to move further.
- danaris 5mo ago
  Yes, but have you seen what's happened to hardware improvements over the past 20 years? From the 1960s to the mid-2000s, every 10 years you'd have a big enough improvement in computing power that you could basically throw out the old computers and replace them with two new ones that were each massive improvements for the same cost (this varied, of course, from hyperbole to massive understatement). We achieved this by shrinking transistors, so we could fit more onto the die. With that, we could dramatically increase clock speeds and the amount of RAM we could cram into a machine But then we hit the wall of physics. Things haven't stopped improving since ~2015, but they've slowed down so, so much. We've made transistors so small that there's very little more improvement we can get by continuing down that path—they're already seeing serious quantum tunneling effects that need to be adjusted for. We can no longer assume that we can just powerscale our way out of any computation-cost problem. And breakthroughs, by their very nature, cannot be relied upon—we have no guarantee that there's even a possible way to improve our silicon to scale the way we did before, let alone that it'll be something achievable this decade, or that it'll be cost-effective.
  - ajam1507 5mo ago
    Moore's law isn't as relevant with parallel workloads. If you can keep building more lanes you don't have to worry about making faster cars.
threethirtytwo 5mo ago
Maybe we do p2p compute?
- nullbio 5mo ago
  This is a good idea. I've been hoping that a large player with enough social reach would create an open-source fund that everyone can contribute to, to develop a company that trains and releases open-source models at the cutting edge. We can crowdfund the training costs, and the whole world benefits. It's the most logical solution for AI anyway, considering that it's training on humanities collective knowledge. It should be more of a public-funded and public-access resource, rather than something greedy tech companies distribute like crumbs while they use unlocked powers internally to clone all of our businesses and swallow the economy.
- yogthos 5mo ago
  there are already projects like Petals https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals
Grombobulous 5mo ago
I share your concerns, although we still see pretty similarly large and complex things that remain open source today. I am astonished on a daily basis that my Linux computer is so close to the same experience as two operating systems put out by trillion dollar companies. It even does things that those commercial alternatives don’t do. Also, if DeepSeek is truly putting out models with 1/10th the cost of Western competitors, and a fraction of the employee headcount, I think it implies that there will be a market for someone else to be in the space offering an alternative. I think about how companies like IBM are so willing to contribute to Linux and give away those contributions for free because they are part of group of corporate sponsors that need an alternative to more dominant commercial players in the market. Meta “gives away” React for similar reasons: it’s more beneficial for them to have it be a standard and be able to hire people who already know it. It’s definitely harder to imagine the same ecosystem benefits of an AI model, but maybe it’s out there somewhere. I could imagine some data center/VPS providers trying to sponsor something like that so that the big AI companies have less leverage over them. Or maybe all this optimism is a pipe dream?
- echelon 5mo ago
  > I am astonished on a daily basis that my Linux computer is so close to the same experience as two operating systems put out by trillion dollar companies. It even does things that those commercial alternatives don’t do. We live in a world where you can "port" open source software to a new language (Rust) and close it up. Linux will be ported to Rust and closed. It'll probably also be put under MIT/BSD because nobody cares anymore, but the companies will have their own internal private variants. And these will be the ones that see corporate development. The value in open source is that it was a lot of concentrated value that was hard to copy, clone, or rip off. Now you can one shot a replacement with a few hundred bucks in tokens. The economic value of Linux used to be billions of dollars. Soon it'll probably be closer to $0. It's over. > Meta “gives away” React for similar reasons: it’s more beneficial for them to have it be a standard and be able to hire people who already know it. Nah, now you just one shot your thing. And you do it fast enough and with distribution and you win. Eventually human devs can't afford to keep competing and launching startups slower than a hyperscaler's own massively funded efforts. This is the end of open source and the end of solo developers. And when the ruthlessly effective models that can one shot entire business functions cost $1,000,000 per invocation. Oracle can afford to press the button to create, say, a new smartphone. But you cannot. Just wait until devices start requiring trusted computing attestation. The ladder is going to be pulled up.
Fordec 5mo ago
Anyone who isn't currently own a piece of who is winning by the current model. Basic disruption theory, if the game isn't going your way, change the game.
nstart 5mo ago
Tbh, there really needs to be some legal precedent set that makes model distillation a legal activity. If the model makers can rip everyone else's work and launder information as if it's their own without giving credit back to the original creators, I don't see why it should be illegal to distill the models. It's the same thing the frontier model makers are doing to IP everywhere else.
- mewpmewp2 5mo ago
  And which leading country is going to go for allowing other countries to distill their models?
  - dimitar 5mo ago
    It doesn’t have to be the leading countries, if the EU allows it, it is good enough to create a market for distilled models
    - mewpmewp2 5mo ago
      But EU is way behind right?
  - vineyardmike 5mo ago
    If your country doesn't have any leading models, why not legalize distillation, either explicitly or implicitly? (Chinese labs famously distilled American models, and that seems to be going well for them. They now have a competitive industry, home-grown talent choosing not to leave, and they now can truly compete without distillation).
- boppo1 5mo ago
  I agree. But this won't happen in the US because Anthropic / OpenAI is a big ol economic recession risk because we levered ourselves to the tits and put our chips on them.
  - CuriouslyC 5mo ago
well_ackshually 5mo ago
You have an unhealthy and unreasonable obsession with the idea of CCP models, you should get that checked.
kristjansson 5mo ago
It’s expensive, but not unfathomably, esp in an open source setting where capable people might contribute high quality data for post training (worked problems, code reviews, feedback, …) gratis instead of at immense cost.
sschueller 5mo ago
The internet, the world wide web, etc. and much of the research into new medical tech. All public money. The fully open model Apertus (although not the frontier) was fully fundend by public Swiss institutions and a strategic national partners. I would not consider Switzerland to be a communist or totalitarian state...
rsanek 5mo ago
Perhaps an idea that could work is that if you're a lab that is releasing closed source models, you have to also release open source ones. gpt-oss is now old but was decent when it came out. Nemotron is solid, especially the recent ultra release. And Nvidia especially has a much better story vs Chinese models around releasing all parts (including pre and post training data), not just the model itself.
yogthos 5mo ago
I'll take these 'authoritarian' models from China any day over whatever you call this https://arxiv.org/abs/2406.17737 https://arxiv.org/abs/2406.17737
killerstorm 5mo ago
It's not only expensive, it's also wasteful - there's no value in using an obsolete model. Open source AI manifesto demand that "Opensource AI should remain ... economically viable". That's just wishful thinking.

brcmthrowaway 5mo ago

Who funds Semiconductor fabs

nullbio 5mo ago
When Jensen (Nvidia) was doing interviews at his recent public talks, he was asked something along the lines of: "Why release these new laptops which are a low margin market, if your other businesses are vastly more profitable?" and his answer was basically that if they can build the coolest and best technology and push the frontier, they will do it. It's not all about making tons of money. He seemed genuinely excited about the tech. It highlights the difference between companies like Nvidia and Anthropic to me, where one is clearly all about the money and power, and the other is doing it because they genuinely want to accelerate progress and make cool stuff as the driving factor. It's no surprise therefore, that Nvidia is the worlds largest open-source contributor to AI, with over 800 open-weight models. Of course, these models run on Nvidia hardware, so they benefit from it as a company. But with that healthy mindset, they found a way to contribute that not only benefits everyone, but also benefits themselves. Contrast to Anthropic, who has gone the complete opposite direction. Closed off everything, restricting everything, fearmongering progress, regulatory capture attempts, the list goes on. I mean, they won't even agree on using AGENTS.md as a standard because CLAUDE.md is free marketing for them. That's the level of disgusting greed we are dealing with... From a game theory perspective, the cooperative strategies tend to win. As a result, Nvidia has set themselves up for a lifetime. Anthropic however, is playing a strategy of winner takes all, and they're happy to see the world and the entire AI industry collapse in the process.
- cwnyth 5mo ago
  That's not really the impression I get from Anthropic, but if you have the links to back it up, I'm always willing to change my mind. Compared to bizes like Oracle, Microsoft, or Facebook, I felt that Anthropic was more interested in progress (not to the neglect of business―AI training is expensive at the end of the day), but maybe I've just not seen what you've seen.
  - nullbio 5mo ago
    https://clawd.rip https://clawd.rip
- SXX 5mo ago
  Nvidia and "open source" is like opposite things. Nvidia only ever opened stuff that helps their bottom line or improve vendor lock-in. But yeah they are good shovel seller and competitor to actually evil companies that literally wants to eat all the world chips and energy supply.
  - zozbot234 5mo ago
    In the open source space, the Nemotron models from nVidia are quite real. Including a Nemotron Ultra variety meant to be large enough for near-SOTA.
    - SXX 5mo ago
      Nvidia not doing it out of goodness of their hearts and love to open source. If at anynpoint their CUDA vendor lock-in moat will faik because Intel or AMD manage to get working software they'll return to keep everything locked and proprietary ASAP. Basically everything Nvidia does in open source is there to make sure their proprietary stack have a good moat and no competitor stack can catch up.
  - nullbio 5mo ago
    Strongly disagree: https://build.nvidia.com/models https://build.nvidia.com/models Their license terms are also incredibly generous and allow commercial use, modification, etc, at no cost.

cwnyth 5mo ago

Ever calculate the cost of a computer in the 1960s, adjusted for inflation? Training is unfathomably expensive right now. What if a bunch of universities pooled their money? Or a bunch of nations pooled their money? Breakthroughs will eventually happen, optimization will occur, etc. People questioned whether there could ever be a viable open source operating system, yet Linux has been a viable option for a desktop environment for decades now, and that's not to mention its ubiquitous use as a server or phone OS.

kamaal 5mo ago
Yes, You have to start some where. Im guessing, making progress also brings in new ideas how to move further.
danaris 5mo ago
Yes, but have you seen what's happened to hardware improvements over the past 20 years? From the 1960s to the mid-2000s, every 10 years you'd have a big enough improvement in computing power that you could basically throw out the old computers and replace them with two new ones that were each massive improvements for the same cost (this varied, of course, from hyperbole to massive understatement). We achieved this by shrinking transistors, so we could fit more onto the die. With that, we could dramatically increase clock speeds and the amount of RAM we could cram into a machine But then we hit the wall of physics. Things haven't stopped improving since ~2015, but they've slowed down so, so much. We've made transistors so small that there's very little more improvement we can get by continuing down that path—they're already seeing serious quantum tunneling effects that need to be adjusted for. We can no longer assume that we can just powerscale our way out of any computation-cost problem. And breakthroughs, by their very nature, cannot be relied upon—we have no guarantee that there's even a possible way to improve our silicon to scale the way we did before, let alone that it'll be something achievable this decade, or that it'll be cost-effective.
- ajam1507 5mo ago
  Moore's law isn't as relevant with parallel workloads. If you can keep building more lanes you don't have to worry about making faster cars.

threethirtytwo 5mo ago

Maybe we do p2p compute?

nullbio 5mo ago
This is a good idea. I've been hoping that a large player with enough social reach would create an open-source fund that everyone can contribute to, to develop a company that trains and releases open-source models at the cutting edge. We can crowdfund the training costs, and the whole world benefits. It's the most logical solution for AI anyway, considering that it's training on humanities collective knowledge. It should be more of a public-funded and public-access resource, rather than something greedy tech companies distribute like crumbs while they use unlocked powers internally to clone all of our businesses and swallow the economy.
yogthos 5mo ago
there are already projects like Petals https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals

Grombobulous 5mo ago

I share your concerns, although we still see pretty similarly large and complex things that remain open source today. I am astonished on a daily basis that my Linux computer is so close to the same experience as two operating systems put out by trillion dollar companies. It even does things that those commercial alternatives don’t do. Also, if DeepSeek is truly putting out models with 1/10th the cost of Western competitors, and a fraction of the employee headcount, I think it implies that there will be a market for someone else to be in the space offering an alternative. I think about how companies like IBM are so willing to contribute to Linux and give away those contributions for free because they are part of group of corporate sponsors that need an alternative to more dominant commercial players in the market. Meta “gives away” React for similar reasons: it’s more beneficial for them to have it be a standard and be able to hire people who already know it. It’s definitely harder to imagine the same ecosystem benefits of an AI model, but maybe it’s out there somewhere. I could imagine some data center/VPS providers trying to sponsor something like that so that the big AI companies have less leverage over them. Or maybe all this optimism is a pipe dream?

echelon 5mo ago
> I am astonished on a daily basis that my Linux computer is so close to the same experience as two operating systems put out by trillion dollar companies. It even does things that those commercial alternatives don’t do. We live in a world where you can "port" open source software to a new language (Rust) and close it up. Linux will be ported to Rust and closed. It'll probably also be put under MIT/BSD because nobody cares anymore, but the companies will have their own internal private variants. And these will be the ones that see corporate development. The value in open source is that it was a lot of concentrated value that was hard to copy, clone, or rip off. Now you can one shot a replacement with a few hundred bucks in tokens. The economic value of Linux used to be billions of dollars. Soon it'll probably be closer to $0. It's over. > Meta “gives away” React for similar reasons: it’s more beneficial for them to have it be a standard and be able to hire people who already know it. Nah, now you just one shot your thing. And you do it fast enough and with distribution and you win. Eventually human devs can't afford to keep competing and launching startups slower than a hyperscaler's own massively funded efforts. This is the end of open source and the end of solo developers. And when the ruthlessly effective models that can one shot entire business functions cost $1,000,000 per invocation. Oracle can afford to press the button to create, say, a new smartphone. But you cannot. Just wait until devices start requiring trusted computing attestation. The ladder is going to be pulled up.

Fordec 5mo ago

Anyone who isn't currently own a piece of who is winning by the current model. Basic disruption theory, if the game isn't going your way, change the game.

nstart 5mo ago

Tbh, there really needs to be some legal precedent set that makes model distillation a legal activity. If the model makers can rip everyone else's work and launder information as if it's their own without giving credit back to the original creators, I don't see why it should be illegal to distill the models. It's the same thing the frontier model makers are doing to IP everywhere else.

mewpmewp2 5mo ago
And which leading country is going to go for allowing other countries to distill their models?
- dimitar 5mo ago
  It doesn’t have to be the leading countries, if the EU allows it, it is good enough to create a market for distilled models
  - mewpmewp2 5mo ago
    But EU is way behind right?
- vineyardmike 5mo ago
  If your country doesn't have any leading models, why not legalize distillation, either explicitly or implicitly? (Chinese labs famously distilled American models, and that seems to be going well for them. They now have a competitive industry, home-grown talent choosing not to leave, and they now can truly compete without distillation).
boppo1 5mo ago
I agree. But this won't happen in the US because Anthropic / OpenAI is a big ol economic recession risk because we levered ourselves to the tits and put our chips on them.
- CuriouslyC 5mo ago

well_ackshually 5mo ago

You have an unhealthy and unreasonable obsession with the idea of CCP models, you should get that checked.

kristjansson 5mo ago

It’s expensive, but not unfathomably, esp in an open source setting where capable people might contribute high quality data for post training (worked problems, code reviews, feedback, …) gratis instead of at immense cost.

sschueller 5mo ago

The internet, the world wide web, etc. and much of the research into new medical tech. All public money. The fully open model Apertus (although not the frontier) was fully fundend by public Swiss institutions and a strategic national partners. I would not consider Switzerland to be a communist or totalitarian state...

rsanek 5mo ago

Perhaps an idea that could work is that if you're a lab that is releasing closed source models, you have to also release open source ones. gpt-oss is now old but was decent when it came out. Nemotron is solid, especially the recent ultra release. And Nvidia especially has a much better story vs Chinese models around releasing all parts (including pre and post training data), not just the model itself.

yogthos 5mo ago

I'll take these 'authoritarian' models from China any day over whatever you call this https://arxiv.org/abs/2406.17737 https://arxiv.org/abs/2406.17737

killerstorm 5mo ago

It's not only expensive, it's also wasteful - there's no value in using an obsolete model. Open source AI manifesto demand that "Opensource AI should remain ... economically viable". That's just wishful thinking.