3 ms·

> Do you mean e.g. small "companies trying to make money?" No they mean open source projects. The overwhelming majority of all generative AI related open sour

by stale2002 3y ago

> Do you mean e.g. small "companies trying to make money?" No they mean open source projects. The overwhelming majority of all generative AI related open source projects depend on, use, or are related to publicly available weights that were trained without permission on other people's data. The entire open source AI community wouldn't exist if those weights weren't available for everyone to use for basically any reason. One could argue that the existing weights aren't going to disappear as thats impossible to enforce. But what about the updated weights? If nobody is allowed to spend 50 million dollars training a new model and then releasing that model publicly for anyone to use, without getting permission from the entire internet to make that model, well then there aren't going to be anymore new models coming out. The government isn't going to be able to confiscate everyone's gamer PCs in order to ban inference, but they can stop someone from spending that 50 mil training the next cutting edge open source model. Which means that the existing models will be all that the open source community has. Yeah they could fine tune them, but cutting edge base models are still extremely important for open source AI progress.

zirgs 3y ago
But you won't need to spend 50M in the future. Hardware improves all the time. Remember all those "can it run crysis?" memes? Those GPUs are now slower than mobile phones.
- stale2002 3y ago
  Moore's law takes quite literally years to come into effect. And the AI progress time gap we are seeing right now, of "amount of time between major AI breakthroughs" is measured in weeks, not years. Going back to waiting years instead of weeks for entire technologies to be revolutionized is a major slowdown.
jrm4 3y ago
Right. I think you hit on the biggest question -- what is the likely MARGINAL improvements in weights that your hypothetical 50 million buys? -- even if you accept that we wouldn't be here without that first "50 million" that already went in? A reasonable argument, a possibility, is "very little." Perhaps compare to "so and so is going to put 50 million in improving the Linux Kernel?" I'm not sure this would improve anything at ALL, in fact, maybe make it worse -- given that someone will want to get something out of 50 million?
- stale2002 3y ago
  > what is the likely MARGINAL improvements in weights that your hypothetical 50 million buys Wrong question. The marginal improvements of a specific model might be marginal, but what matters is new models for different usecases. For example, we have great image generation models 6 months ago, and one coming out now isn't going to be that much better. But, what is starting to get good right now is generative video models. And the video models are a huge marginal improvement over what was available 6 months ago. Now do the same thing for any other use case. We have reasonable good videos now, but what about 3d models? Those mostly suck right now. Lots of improvement opportunities that wouldn't exist if training new models was banned. There might be a dozen such major usecases that don't exist yet, where we can't repurpose old models, and instead need to train an entirely new model with different data. Making sure we still get that innovation is still important.
  - jrm4 3y ago
    No one actually knows "what matters" yet, and also no one knows whether or not you actually need "new models." Everything you're saying right now is extremely hypothetical. The tech is very shiny, no doubt about that. But it's wildly conclusory to think "Yes, there will definitely be a demand for new use cases AND ALSO you definitely will be unable to create those new models without big money investment." Again, it sounds to me exactly like someone clamoring for "big investment in a new, revolutionary, operating system." Nothing at all inevitable about that for MANY reasons, most having to do with "incremental building on the past thing that already works is probably the most feasible."
    - deleted 3y ago
      [deleted]
    - stale2002 3y ago
      > and also no one knows whether or not you actually need "new models." All the more reason to not make it illegal to train these new models. There are massively improved models coming out once a month. There could be many more to come. As you said, we don't even know where this stops! and the current innovation is very large right now. Therefore, it is premature to just ban all of that, when we don't even know how much more innovation there is to go. So the point stands. The massive amount of innovation that has helped open source has all been driving by these new models being released. And there could be many more to come. > Everything you're saying right now is extremely hypothetical. Looking at the fact that we come out with a powerful new model every month is not hypothetical. I am referencing existing reality here.

zirgs 3y ago

But you won't need to spend 50M in the future. Hardware improves all the time. Remember all those "can it run crysis?" memes? Those GPUs are now slower than mobile phones.

stale2002 3y ago
Moore's law takes quite literally years to come into effect. And the AI progress time gap we are seeing right now, of "amount of time between major AI breakthroughs" is measured in weeks, not years. Going back to waiting years instead of weeks for entire technologies to be revolutionized is a major slowdown.

jrm4 3y ago

Right. I think you hit on the biggest question -- what is the likely MARGINAL improvements in weights that your hypothetical 50 million buys? -- even if you accept that we wouldn't be here without that first "50 million" that already went in? A reasonable argument, a possibility, is "very little." Perhaps compare to "so and so is going to put 50 million in improving the Linux Kernel?" I'm not sure this would improve anything at ALL, in fact, maybe make it worse -- given that someone will want to get something out of 50 million?

stale2002 3y ago
> what is the likely MARGINAL improvements in weights that your hypothetical 50 million buys Wrong question. The marginal improvements of a specific model might be marginal, but what matters is new models for different usecases. For example, we have great image generation models 6 months ago, and one coming out now isn't going to be that much better. But, what is starting to get good right now is generative video models. And the video models are a huge marginal improvement over what was available 6 months ago. Now do the same thing for any other use case. We have reasonable good videos now, but what about 3d models? Those mostly suck right now. Lots of improvement opportunities that wouldn't exist if training new models was banned. There might be a dozen such major usecases that don't exist yet, where we can't repurpose old models, and instead need to train an entirely new model with different data. Making sure we still get that innovation is still important.
- jrm4 3y ago
  No one actually knows "what matters" yet, and also no one knows whether or not you actually need "new models." Everything you're saying right now is extremely hypothetical. The tech is very shiny, no doubt about that. But it's wildly conclusory to think "Yes, there will definitely be a demand for new use cases AND ALSO you definitely will be unable to create those new models without big money investment." Again, it sounds to me exactly like someone clamoring for "big investment in a new, revolutionary, operating system." Nothing at all inevitable about that for MANY reasons, most having to do with "incremental building on the past thing that already works is probably the most feasible."
  - deleted 3y ago
    [deleted]
  - stale2002 3y ago
    > and also no one knows whether or not you actually need "new models." All the more reason to not make it illegal to train these new models. There are massively improved models coming out once a month. There could be many more to come. As you said, we don't even know where this stops! and the current innovation is very large right now. Therefore, it is premature to just ban all of that, when we don't even know how much more innovation there is to go. So the point stands. The massive amount of innovation that has helped open source has all been driving by these new models being released. And there could be many more to come. > Everything you're saying right now is extremely hypothetical. Looking at the fact that we come out with a powerful new model every month is not hypothetical. I am referencing existing reality here.