3 ms·
Courious question: how difficult and expensive is to train these models? Being the technology open source and assuming that we can collaborate on the training p
by wslh 4y ago
Courious question: how difficult and expensive is to train these models? Being the technology open source and assuming that we can collaborate on the training phase and have big open training sets, what will be the competitive advantage of companies such Google beyond the brand?
- criddell 4y agoGoogle scanned 25 million books. That has to help them at least a little.
- deleted 4y ago[deleted]
- wslh 4y agoBut it is possible to crawl Internet in a distributed way. Also Wikipedia is a treasure.
- criddell 4y agoThat's all true, but everybody has access to that. None of that is scarce so the value is basically 0. You asked for advantages that Google has and the text of 25 million books is one thing. A lot of the text is going to be scarce and so it's something of value Google has and others do not. Google also has built a massive GIS database from driving their cars around the world. Apple has a smaller database and there are some third parties (like ESRI) that will sell you access to their database. If you want to ask an AI something about your neighborhood, Google has more data than anybody on that front.
- lelanthran 4y ago> Courious question: how difficult and expensive is to train these models? Being the technology open source and assuming that we can collaborate on the training phase and have big open training sets, what will be the competitive advantage of companies such Google beyond the brand? Zero to none? I mean, it's much much much easier for collaborators to produce better (or even competitive) map data than training LLMs, and yet the open alternatives to google maps are hardly ever used, have spotty data for some locations and have made no compelling feature for user's to adopt.
- spacebanana7 4y agoTraining foundation models is very expensive. Easily a few million per model with no guarantee of success.
- wslh 4y agoAgain, we can assume a distribute training if that is feasible.
- spacebanana7 4y agoUnfortunately, I don’t think distributed training is feasible. My understanding is that these models are trained using some form of gradient descent. This requires knowledge of the model’s prior state and current training error to adjust parameter values. With distributed training it’s difficult, to point of impossible, to communicate the state of the model across large distances because these models are so large. Take my comment with a grain of salt because I’m not an expert in this area. I’d very much enjoy being wrong about this because distributed training would unlock so much potential.
- HDThoreaun 4y ago> what will be the competitive advantage of companies such Google beyond the brand? The advantage is that their AI will be integrated into products that already have billions of users. Android users will get used to the default google assistant model so theyre more likely to use it in other instances as well and so on.