4 ms·
http://pastebin.com/m54b7d354 http://pastebin.com/m54b7d354 Looks like it just hit a bunch of links from the first page.
by bdr 18y ago
http://pastebin.com/m54b7d354 http://pastebin.com/m54b7d354
Looks like it just hit a bunch of links from the first page.
- snprbob86 18y agoSo it looks like Google grabbed about 40 links before giving up? I wonder what a good "score" is? At first, I'd guess less is better, but too few might be running the risk of throwing out potentially good pages. Too many, and the bot is just wasting effort. The 40 score could vary as well based on parallel conditions assuming many bot instances are sharing a task pool. Be sure to post the Microsoft results if/when they crawl you.
- brianr 18y agoLooks like it hit all of the links on the first page (there are 8x5 boxes = 40) and didn't find anything interesting, so it didn't crawl any deeper. If the second-level pages had more interesting/unique content, I bet it would've kept going.
- hhm 18y agoThanks a lot for sharing this. Very interesting indeed.