3 ms·
Writing a web crawler is not too complicated. I predict every 'AI' company will have a homegrown search engine in a few months to account for this. The way th
by fooker 1y ago
Writing a web crawler is not too complicated.
I predict every 'AI' company will have a homegrown search engine in a few months to account for this.
The way this would be publicly usable is through the new generation of 'AI' browsers.
- toomuchtodo 1y agohttps://commoncrawl.org/ https://commoncrawl.org/
- aiauthoritydev 1y agoIt is not crawling but indexing is the problem. Google has over years learned the patterns and authority of different articles. It will be hard for others to replicate but not impossible. What Google should do is offer API based access to these providers but a lot of these providers might no adhere by contracts. So there is that.
- fooker 1y agoIndexing is a fairly well understood technology. You could hire one or experts for this to be doable with a pretty good amount of scalability.
- afavour 1y agoCrawling isn't complicated. But ranking? That was Google's reason for existence for a very long time. Remains to be seen if AI companies will be able to replicate that.
- fooker 1y agoHere's the interesting part - ranking matters when humans are looking at the results. For a bot with a large context window though, not so much.
- deleted 1y ago[deleted]