4 ms·

Thanks, corrected it

• ozh 13y ago
  I had the same "fun" idea 6 years ago. In 3 months Yahoo Site Explorer reported 150 million pages indexed. As of today Google still shows 55 million pages (in comparison, https://www.google.com/search?q=site:http://en.wikipedia.org/ https://www.google.com/search?q=site:http://en.wikipedia.org... reports 34 million pages). I had to kill the experiment (no more new "pages" crawled) because of the CPU load and bandwidth costs, even throttling robots