3 ms·

The doc is too short and not puffy to be a normal google doc. There is a high likelihood this was placed to proactively shake off any regulation or scruitiny wh

The doc is too short and not puffy to be a normal google doc. There is a high likelihood this was placed to proactively shake off any regulation or scruitiny while they get the deck chairs in order. They might not appear to have a moat, but they definitely have an advantage that they can keep for 5+ years. They also have a "copy of the internet" that no one else does.

akiselev 3y ago
What are the chances that their copy is more useful than CommonCrawl? It’s been the target of SEO for decades - are they even capable of separating the good content from the crap anymore? Google may already be terminal
- sitkack 3y ago
  Data pollution isn't the problem it is made out to be. You can even use a model trained on garbage data to relabel the original set and retrain. Google of all companies is sitting on exaflops of TPU capacity, they can retrain models larger than anyone else, faster than anyone else.
  - visarga 3y ago
    I agree, but it would probably require to use LLMs to filter or augment the raw text. They can afford it.
- chaxor 3y ago
  Their data may not be that much better; however Deepmind is still one of the best institutions that study AI fundamentals. OpenAI works hard and focused towards products, but the academic research is far better from Deepmind. So since Google hasn't been trying to put any AI into products really at all for the past 10 years, they're going to have to push a little bit to make the public facing interfaces and put polish on things, but the underlying models they have made are far better than OpenAI (just not the ones made publicly accessible to anyone).
  - visarga 3y ago
    They have no model on par with GPT-4.
    - chaxor 3y ago
      I would heavily disagree with that - the only reason that you think that is that GPT-4 is available to the public for cheap, whereas Deepminds models have not been released publicly.
- fomine3 3y ago
  For websites that blocks all robots.txt-compliant bots except Google, Bing and some, Googlebots can crawl but CommonCrawl bots can't
UncleMeat 3y ago
> The doc is too short and not puffy to be a normal google doc. Loads of internal position papers read exactly like this. I've written docs that write like this.
sitkack 3y ago
They also have "all of youtube" when it comes to video based learning. And GCS and Gmail, Google's moat is actually enormous.
- candiodari 3y ago
  The points raised in the paper are that 1) algorithmic improvements beat processing capacity. One of the examples in the paper was that $100 in training compute budget was enough to get within 2% of Google Bard performance 2) data quality (even for "tiny" data sizes) beats dataset size in improvements Hence both aspects of the moat of large companies (dataset size and available compute) matter less than you'd expect. In contrast the advantage of "open source"/academics/"the market", the many warm bodies trying out new strategies, yields improvements at a rapid pace that Google, even with a huge team, cannot match.

akiselev 3y ago

What are the chances that their copy is more useful than CommonCrawl? It’s been the target of SEO for decades - are they even capable of separating the good content from the crap anymore? Google may already be terminal

sitkack 3y ago
Data pollution isn't the problem it is made out to be. You can even use a model trained on garbage data to relabel the original set and retrain. Google of all companies is sitting on exaflops of TPU capacity, they can retrain models larger than anyone else, faster than anyone else.
- visarga 3y ago
  I agree, but it would probably require to use LLMs to filter or augment the raw text. They can afford it.
chaxor 3y ago
Their data may not be that much better; however Deepmind is still one of the best institutions that study AI fundamentals. OpenAI works hard and focused towards products, but the academic research is far better from Deepmind. So since Google hasn't been trying to put any AI into products really at all for the past 10 years, they're going to have to push a little bit to make the public facing interfaces and put polish on things, but the underlying models they have made are far better than OpenAI (just not the ones made publicly accessible to anyone).
- visarga 3y ago
  They have no model on par with GPT-4.
  - chaxor 3y ago
    I would heavily disagree with that - the only reason that you think that is that GPT-4 is available to the public for cheap, whereas Deepminds models have not been released publicly.
fomine3 3y ago
For websites that blocks all robots.txt-compliant bots except Google, Bing and some, Googlebots can crawl but CommonCrawl bots can't

UncleMeat 3y ago

> The doc is too short and not puffy to be a normal google doc. Loads of internal position papers read exactly like this. I've written docs that write like this.

sitkack 3y ago

They also have "all of youtube" when it comes to video based learning. And GCS and Gmail, Google's moat is actually enormous.

candiodari 3y ago
The points raised in the paper are that 1) algorithmic improvements beat processing capacity. One of the examples in the paper was that $100 in training compute budget was enough to get within 2% of Google Bard performance 2) data quality (even for "tiny" data sizes) beats dataset size in improvements Hence both aspects of the moat of large companies (dataset size and available compute) matter less than you'd expect. In contrast the advantage of "open source"/academics/"the market", the many warm bodies trying out new strategies, yields improvements at a rapid pace that Google, even with a huge team, cannot match.