4 ms·
Open models are probably also comparatively astronomically expensive to train - just less so than the frontier models because they’re somewhat smaller, +/- the
by mft_ 3mo ago
Open models are probably also comparatively astronomically expensive to train - just less so than the frontier models because they’re somewhat smaller, +/- the creators are more incentivised to focus on getting more from less compute because they’re have to, +/- they rely on distillation of the frontier models and this is more efficient.
But efficiencies aside; creation of open models still requires a lot of money and compute from a large organisation which is willing to accept zero return for that spend. This largesse is unlikely to continue forever; so the question is which will crack first, the frontier models’ business model or the fast followers’ generosity?
- afavour 3mo agoI’m not exactly sure on the “how” but it only makes logical sense for (non-AI) companies to band together to fund the training of a shared model. Apple is a great example, AI is not their core business but they still require it. The only thing that took us down a different path is the vast sums of VC funding pumped into the AI companies.
- inigyou 3mo agoIf not for VC-funded LLMs there wouldn't be any LLMs.
- afavour 3mo ago[citation needed] Historically speaking a lot of inventions have come about without things like VC investment. Either way, there’s probably little point in debating it, just because VC funded companies control the market now doesn’t mean they should indefinitely.
- LunaSea 3mo agoMost of the innovations needed for LLMs came from people at Google.
- inigyou 3mo agoSo VC-funded.
- mrandish 3mo agoThe company was VC-funded as a search engine but by the time they made significant investments in AI (DeepMind etc) they'd been a publicly held company earning multiple billions a year from advertising for a decade.
- inigyou 3mo agoGoogle is a VC. Their side projects are VC projects.
- mrandish 3mo agoGoogle bought DeepMind and their other major AI acquisitions. Public companies make corporate venture investments for very different reasons than LP-backed VCs. They do early-stage investments to search for emerging players they can buy as soon as possible or to gain market intelligence on trends. They do later stage investments to help grow future vendors or customers and sometimes to foster ecosystems that form their competitive moat. But if they think it's important to their core business, corporations don't want to invest, they want to buy. Source: I used to be involved in corporate venture investing at a top 10 valley tech leader.
- stale2002 3mo agoIf your definition of VC is "Literally anything that requires a long term investment of money" then sure, but I think most people mean something different than that.
- echelon 3mo ago> came from people at Google Who had to leave to build anything.
- munk-a 3mo ago
- marcosdumay 3mo agoIt probably doesn't. There's a reason we let companies specialize in some kind of service and buy it from them. LLMs aren't looking like they'll be highly differentiated like software, so their market will probably be competitive. What negates the main reason Open Source software exists.
- munk-a 3mo agoLLM training doesn't carry the same NIH risks that normal internal software bloat does. They are relatively simple to setup training for and analysis of accuracy/recall can be automated. This leaves the price differential between a private third party and an internal initiative as barely more than the cost to train the model[1] - perhaps that's where we'll end up, a centrally trained model will represent an economy of scale that can leverage that difference into a margin it can profit off of but your business being purely profit driven by that training expenditure seems like a ridiculously thin margin. So where does that leave the AI companies? If their LLMs are off the shelf-once built products they have a strong advantage for casual low usage but enterprise customers will have a huge cost incentive to roll their own - if the LLMs require continuous retraining and the frontier keeps moving then enterprise customers will find a packaged service more attractive and likely continue to subscribe for more accuracy but casual low usage will likely shift towards "good enough" models. It seems inevitable that they'll lose half the market and it seems difficult to discern their long term profitability[2]. 1. Costs can, I think, reasonably be reduced to hardware depreciation and energy - if trends continue with cloud resource availability (it's possible this won't be the case if large compute providers start pulling resources offline to build a moat but I think they'd likely prefer the reliable compute income over model income which has several other competitive weaknesses). Hardware depreciation would normally be pretty negligible and equal across different training entities, right now we have a chip shortage but given the demand that can't last too long so I'd consider hardware to be fungible - and energy is entirely fungible - they're both hard to moat. 2. Outside of AGI, who knows if AGI will be or what even counts for it at this point - but I think if AGI isn't a doomsday scenario we fall back to one of the two above scenarios - either the frontier is ever moving and they can retain enterprise customers or the frontier seizes up and everyone can just use an off the shelf offering. In either scenario they don't have a lot of moat to deal with for their products unless they can restrict compute which is why Alphabet, AWS and MSFT are the only players I could see realistically coming out of this as an AI vendor winner and I'm not even certain if it'd be a good idea for them if it'd hamstring their cloud profitability.
- demosthanos 3mo agoYes, the problem with comparing open models to open source is that open source requires humans to volunteer their time. Open models requires humans to volunteer their money. These two types of contributions have very different behavioral profiles, and it doesn't obviously follow that the historical success of getting people to collaborate socially on building software for fun and for the benefit of the community will translate in any meaningful way to the necessity of being able to raise enormous amounts of money to pay for enormous amounts of electricity.
- connicpu 3mo agoTechnically open source requires some amount of monetary volunteering, it's just that the electricity to run a code editor and compile (most) open source code bases is within hobby budget for most people.
- enos_feedler 3mo agoThe biggest hurdle is whether humans volunteer their expertise. Not time or money. We need top talent to make the open models. Sponsorship is plentiful. Open source volunteers are less critical with LLM doing the grunt work. Its about talent contributing to the open
- calebkaiser 3mo agoThis is not my experience working in the field the last 8 years. There is not a dearth of talented researchers, engineers, etc. who are willing to contribute to open models. Just look at the ecosystem generally, from academia to industry. So much research still happens in the open, and the open source community in ML is still massive and energetic. Training a new frontier model just costs a lot of money and there are very few companies for whom it makes sense to do so, and even fewer orgs who are just going to gift those kinds of resources to an open source initiative.
- hobofan 3mo ago> open source requires humans to volunteer their time Your idealistic of open source may require that, but in practice a huge part of open source is commercial and a large chunk of that is low on collaboration (across vendor boundaries).
- sph 3mo agoHow does it work if people flock to open models but they're too expensive to train? What is the financial incentive to do so? I seem to understand open models are mostly coming from China, and the benefit of training and releasing them for 'free' is a powerful geopolitical weapon against the Western/US economy that at this point depends on OpenAI & co. to succeed. Will the West make open models illegal?
- echelon 3mo ago> Will the West make open models illegal? We better not. > What is the financial incentive to do so? If we'd been sharing all along (as we should have been), we probably would have gotten even further along in the development of the tech. Think of everything we could do if every researcher on the planet had first class access to the frontier. No academic fallback models. No crude API access. No limits, but direct access to the weights and the ability to lobotomize, splice, and dice. We could pour intelligence from one container to the next without paying a tax or wearing a blindfold. All without spilling a drop. *Open* *Must* *Win*
- inigyou 3mo ago> No X. No Y. No Z, but Q
- LorenDB 3mo ago"You wouldn't download an LLM"
- chaosharmonic 3mo agoI don't think it needs to be framed purely as generosity. You just need a sufficiently self-interested actor that sees open ecosystems as a necessary part of reducing their own risk profile, relative to the alternative of complete reliance of a third-party business that can take an exorbitant cut and/or Sherlock them at any time. Valve and SteamOS are a good example of what this idea looks like in practice. (Though they may also illustrate a third thing you need: a privately-run company, that has enough profit, and enough commitment from leadership to the company's vision, that they can make long-term bets without having to eventually bow to investors seeking short-term gains.)
- mft_ 3mo ago> You just need a sufficiently self-interested actor that sees open ecosystems as a necessary part of reducing their own risk profile, relative to the alternative of complete reliance of a third-party business that can take an exorbitant cut and/or Sherlock them at any time. This would be an argument for an organisation developing its own model; but not per se for releasing the weights openly. The possible explanations (I'm aware of, which overlap somewhat) for spending large amounts of money on models then releasing them for free (i.e. the current Chinese approach) are soft power, marketing for a future paid model business (i.e. competing with the US models for customers and mindshare during the time you can't compete directly at the bleeding edge), and/or a geopolitical move to diminish the value of the US's frontier model companies.
- chaosharmonic 3mo agoI acutally see the reduced burden that comes with actually sharing resources. To extend the previous analogy: Valve didn't make desktop Linux viable on their own. A lot of it is owed to another self-interested actor -- Google -- through the reduced need for dedicated desktop apps (largely pushed by Chrome), and various enhancements to wireless and power management that were necessary to make it a viable mobile platform (directly benefiting Android/ChromeOS, but then spreading out to laptops and mobile devices in general, including handhelds like the Steam Deck). You can see it on a smaller scale in ecosystems like Android -- where handset makers regularly contribute features from their UI skins upstream, so that they no longer need to spend engineering resources maintaining distinct versions of theming engines/notification badges/multiwindow/various other stuff. On a related note this is an argument for open source models, not just open weights. I think a lot of the diminishing returns relate to the opaque nature of most models.
- figassis 3mo agoI would definitely pay a monthly subscription to help fund a non profit compete with Anthropic and OpenAI. I already pay subscriptions for myself and 2+ other people. It's a non brainer to be able to pay for the training of better models that I can then run myself for many more. I hope someone starts this, I think this model would work. I'd start it today if I had the team and initial capital to bootstrap the infr. I know VCs won't fund it, but we definitely will, enthusiastically and continually.
- deleted 3mo ago[deleted]
- nijave 3mo agoMy (unverified) AI research claimed generally Chinese models are cheaper to train because Chinese data scientists are cheaper to hire and they're also under more pressure to optimize training cost due to limited hardware availability Seemed believable but not sure where that's true Chinese AI companies are generally smaller tho and the models they're releasing are also smaller (I think estimates put OpenAI and Anthropic SOTA into trillions of parameters)
- ubercore 3mo agoI can't imagine even with the crazy salaries at frontier labs, staff costs make much of a difference.
- holoduke 3mo agoIn the US and Europe its extremely expensive indeed. In China it is much lower and expected to be much lower in the future when China can bake their own high end chips. You know China is focussed on optimizing production and manufacturing. Unlike the west that primarily focus on regulations that make things super expensive.
- Danox 3mo agoEngineers at the top of the pyramid in China, Lawyers on top of the pyramid in America.
- slashdave 3mo agoExcept: distillation. It is not so strange that models from certain organizations are always a few months behind.