4 ms·

He would probably argue that future LLMs are less likely to have their weights released. It is remarkable and strange that we have so many open-weights models r

He would probably argue that future LLMs are less likely to have their weights released. It is remarkable and strange that we have so many open-weights models right now, considering how much they cost to make.

deleted 3y ago
[deleted]
AnthonyMouse 3y ago
It costs a lot of money to an individual, not to a corporation. Estimates are that it cost Facebook around $20 million for LLaMA, for example. This is less than 0.1% of their annual revenue. It's slightly weird that some of the companies doing this are VC-backed startups with no obvious business model, but that's hardly the only way to create them. Companies like Facebook could continue to release them purely for the PR. The cost is within the range of what could be done from donations by a non-profit organization. It's the sort of thing you might build a business model around in the same way that IBM or Intel does with Linux -- an obvious candidate here would be models released by Nvidia/AMD/Intel/Apple/etc. You might also build a business model out of consulting services rather than hardware sales, e.g. here's a free LLM and now you can pay us to integrate it with your company's systems. This is also assuming that nobody ever figures out how to train a model using distributed computing. The second that happens there will be arbitrarily many free models because there are a zillion people willing to contribute a day's worth of consumer GPU time to a free model project.
- rustcleaner 3y ago
  >It's slightly weird that some of the companies doing this are VC-backed startups with no obvious business model, but that's hardly the only way to create them. If you are a multi hundred-millionaire, do you privately train a model for fun to release for bragging rights and possibly expose yourself directly to some eyewateringly stupid high statutory damage liability? Or do you get a few friends together, funnel your fun money into a protective vehicle that can be sued instead of you, and go to town having fun and maybe making something of it?

deleted 3y ago

[deleted]

AnthonyMouse 3y ago

It costs a lot of money to an individual, not to a corporation. Estimates are that it cost Facebook around $20 million for LLaMA, for example. This is less than 0.1% of their annual revenue. It's slightly weird that some of the companies doing this are VC-backed startups with no obvious business model, but that's hardly the only way to create them. Companies like Facebook could continue to release them purely for the PR. The cost is within the range of what could be done from donations by a non-profit organization. It's the sort of thing you might build a business model around in the same way that IBM or Intel does with Linux -- an obvious candidate here would be models released by Nvidia/AMD/Intel/Apple/etc. You might also build a business model out of consulting services rather than hardware sales, e.g. here's a free LLM and now you can pay us to integrate it with your company's systems. This is also assuming that nobody ever figures out how to train a model using distributed computing. The second that happens there will be arbitrarily many free models because there are a zillion people willing to contribute a day's worth of consumer GPU time to a free model project.

rustcleaner 3y ago
>It's slightly weird that some of the companies doing this are VC-backed startups with no obvious business model, but that's hardly the only way to create them. If you are a multi hundred-millionaire, do you privately train a model for fun to release for bragging rights and possibly expose yourself directly to some eyewateringly stupid high statutory damage liability? Or do you get a few friends together, funnel your fun money into a protective vehicle that can be sued instead of you, and go to town having fun and maybe making something of it?