4 ms·
Was it leaked or “leaked”? Google has a good, two part strategy available. First is to neutralize the threat: support the open source efforts covertly and/or
by gumby 4y ago
Was it leaked or “leaked”?
Google has a good, two part strategy available.
First is to neutralize the threat: support the open source efforts covertly and/or overtly, as those efforts undercut the revenue of OpenAI and other potential competitors for the base technology. This is what they did with Android: at a minimum all they had to do was ensure that Apple did not become a monopoly, which they accomplished (the secondary effect was to cut the underbrush of any other competitors). A name for this is to “commoditize your competition”.
The second half is to use their internal experience with LLMs and their other ecosystem to use supercharge their strengths. Side inventions can either be expanded in the hope of a (desperately needed, unrelated to LLMs) new revenue stream or given away to starve potential new entrants.
The big guys have good experience in weaponizing open source. But google has become flabby and seemingly lost any culture of execution so I don’t know if they can pull it off. But the path is there.
- sitkack 4y agoThe doc is too short and not puffy to be a normal google doc. There is a high likelihood this was placed to proactively shake off any regulation or scruitiny while they get the deck chairs in order. They might not appear to have a moat, but they definitely have an advantage that they can keep for 5+ years. They also have a "copy of the internet" that no one else does.
- akiselev 4y agoWhat are the chances that their copy is more useful than CommonCrawl? It’s been the target of SEO for decades - are they even capable of separating the good content from the crap anymore? Google may already be terminal
- sitkack 4y agoData pollution isn't the problem it is made out to be. You can even use a model trained on garbage data to relabel the original set and retrain. Google of all companies is sitting on exaflops of TPU capacity, they can retrain models larger than anyone else, faster than anyone else.
- visarga 4y agoI agree, but it would probably require to use LLMs to filter or augment the raw text. They can afford it.
- chaxor 4y agoTheir data may not be that much better; however Deepmind is still one of the best institutions that study AI fundamentals. OpenAI works hard and focused towards products, but the academic research is far better from Deepmind. So since Google hasn't been trying to put any AI into products really at all for the past 10 years, they're going to have to push a little bit to make the public facing interfaces and put polish on things, but the underlying models they have made are far better than OpenAI (just not the ones made publicly accessible to anyone).
- fomine3 4y agoFor websites that blocks all robots.txt-compliant bots except Google, Bing and some, Googlebots can crawl but CommonCrawl bots can't
- UncleMeat 4y ago> The doc is too short and not puffy to be a normal google doc. Loads of internal position papers read exactly like this. I've written docs that write like this.
- sitkack 4y agoThey also have "all of youtube" when it comes to video based learning. And GCS and Gmail, Google's moat is actually enormous.
- candiodari 4y agoThe points raised in the paper are that 1) algorithmic improvements beat processing capacity. One of the examples in the paper was that $100 in training compute budget was enough to get within 2% of Google Bard performance 2) data quality (even for "tiny" data sizes) beats dataset size in improvements Hence both aspects of the moat of large companies (dataset size and available compute) matter less than you'd expect. In contrast the advantage of "open source"/academics/"the market", the many warm bodies trying out new strategies, yields improvements at a rapid pace that Google, even with a huge team, cannot match.
- deleted 4y ago[deleted]