3 ms·
Google just cut off 90% of the internet from AI – no one's talking about it
- fooker 1y agoWriting a web crawler is not too complicated. I predict every 'AI' company will have a homegrown search engine in a few months to account for this. The way this would be publicly usable is through the new generation of 'AI' browsers.
- toomuchtodo 1y agohttps://commoncrawl.org/ https://commoncrawl.org/
- aiauthoritydev 1y agoIt is not crawling but indexing is the problem. Google has over years learned the patterns and authority of different articles. It will be hard for others to replicate but not impossible. What Google should do is offer API based access to these providers but a lot of these providers might no adhere by contracts. So there is that.
- fooker 1y agoIndexing is a fairly well understood technology. You could hire one or experts for this to be doable with a pretty good amount of scalability.
- afavour 1y agoCrawling isn't complicated. But ranking? That was Google's reason for existence for a very long time. Remains to be seen if AI companies will be able to replicate that.
- fooker 1y agoHere's the interesting part - ranking matters when humans are looking at the results. For a bot with a large context window though, not so much.
- deleted 1y ago[deleted]
- barbazoo 1y ago> You can no longer view 100 results at once. The new hard limit is 10. Does Google not support lazy-loading more results or is that not supported via API or what's going on here?
- asdff 1y agoResults are certainly thinner than in years past when you could seemingly inspect the entire crawled corpus. You can search for a pretty broad topic and hit the wall pretty fast today. I think they are limiting the depth of queries these days probably owing to search volume and the size of cache they can sustain what with current webdev standards. It was a different story when websites were a few kb to mb 20 years ago even though storage is "cheaper" today.
- jdale27 1y agoClickbait title. They only cut off the AI that was using Google as their crawler, which was not a good idea in the first place. I’d love to ask the developers of these AIs: what exactly did you expect to happen here?
- pityJuke 1y agoi thought most of the major ai vendors (excl google) used their own crawlers and indexes, or licensed from a non-google company
- fooker 1y ago> used their own crawlers and indexes Not yet, but will eventually for sure. If you're an expert who has worked on Google search or something like that, this would b great time to start a company for this.
- aiauthoritydev 1y agoClickbait article.
- weinzierl 1y ago"Most large language models like OpenAI, Anthropic, and Perplexity rely directly or indirectly on Google's indexed results to feed their retrieval systems and crawlers." Is this true? I thought OpenAI was using Bing. Gemini obviously will use Google but to them the restriction does not apply. Claude says it uses Brave.
- jug 1y agoOr OAI-SearchBot for "web search" feature to augment queries and GPTBot for training? I swear I've even read how aggressive GPTBot is. Surely they aren't just googling stuff? https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots
- LargoLasskhyfv 1y agoAI too stupid to scroll, or what?