4 ms·

> The experts, who chose to answer, suggested that we should first start with crawling the whole web. We were told that this would take between 1 and 2 years to

by dsunku 7y ago

> The experts, who chose to answer, suggested that we should first start with crawling the whole web. We were told that this would take between 1 and 2 years to complete, and would cost a minimum of $1 billion Why are costs so high for crawling?

shriphani 7y ago
I did a large-scale crawl of the web some years ago and we put together a billion pages. The bulk of the content ended up being index pages - i.e. large list of links taking you to the content - pagination, other breadcrumbs etc. You can exhaust a lot of resources without getting anything useful. It is no longer a point and go kind of thing unfortunately, you need a good understanding of page structure, estimate what kinds of links are vital, etc. else there's a ton of crap you'll pick up. Or maybe I was doing something wrong.
philippclassen 7y ago
(Disclaimer: I work at Clizq) I don't work on the search, but did some work recently on the crawling part. What I know is that crawling is far more difficult if you are not a big player. Sites will quickly block you once you hit a rate limit. We have to be very careful, since when we get blocked there is normally no way to get unblocked again. You can try to send them an email to unblock you, but it is unlikely that you get a response. This is one part of the explanation why crawling is slow. The other part is more obvious: the internet is large. The blocking part is hard to overcome as a small player, while for Google it is the opposite as sites simply cannot afford being exclude from the index. If we would not have to care about rate limits, it would simplify the problem.
55555 7y ago
There is no way it costs near a billion dollars to do an effective crawl of the web. That's laughable. I do wonder how much Googlebot spends on bandwidth and servers though.
theblackcat2004 7y ago
Wouldn’t common crawl content be enough? If not what are the issues?
- trisch 7y ago
  No, it's not enough and had poor coverage outside of USA. We have also answered this question (it appears to be popular) in today's post about technical details of our search https://0x65.dev/blog/2019-12-06/building-a-search-engine-from-scratch.html https://0x65.dev/blog/2019-12-06/building-a-search-engine-fr...

by dsunku 7y ago

shriphani 7y ago

I did a large-scale crawl of the web some years ago and we put together a billion pages. The bulk of the content ended up being index pages - i.e. large list of links taking you to the content - pagination, other breadcrumbs etc. You can exhaust a lot of resources without getting anything useful. It is no longer a point and go kind of thing unfortunately, you need a good understanding of page structure, estimate what kinds of links are vital, etc. else there's a ton of crap you'll pick up. Or maybe I was doing something wrong.

philippclassen 7y ago

(Disclaimer: I work at Clizq) I don't work on the search, but did some work recently on the crawling part. What I know is that crawling is far more difficult if you are not a big player. Sites will quickly block you once you hit a rate limit. We have to be very careful, since when we get blocked there is normally no way to get unblocked again. You can try to send them an email to unblock you, but it is unlikely that you get a response. This is one part of the explanation why crawling is slow. The other part is more obvious: the internet is large. The blocking part is hard to overcome as a small player, while for Google it is the opposite as sites simply cannot afford being exclude from the index. If we would not have to care about rate limits, it would simplify the problem.

55555 7y ago

There is no way it costs near a billion dollars to do an effective crawl of the web. That's laughable. I do wonder how much Googlebot spends on bandwidth and servers though.

theblackcat2004 7y ago

Wouldn’t common crawl content be enough? If not what are the issues?

trisch 7y ago
No, it's not enough and had poor coverage outside of USA. We have also answered this question (it appears to be popular) in today's post about technical details of our search https://0x65.dev/blog/2019-12-06/building-a-search-engine-from-scratch.html https://0x65.dev/blog/2019-12-06/building-a-search-engine-fr...