4 ms·
This is interesting. On a related note, how come I've never seen britannica.com in my Google search results? Their robots.txt file doesn't scream crawlers go
by jgalt212 11y ago
This is interesting. On a related note, how come I've never seen britannica.com in my Google search results?
Their robots.txt file doesn't scream crawlers go away.
http://www.britannica.com/robots.txt http://www.britannica.com/robots.txt
- dazc 11y agoInteresting point. Their first article is about 100th Anniversary of the Battle of Verdun so I search "Battle of Verdun" and their page, in the UK, is #15. Looking at site overall I think this is a fair result? Maybe demonstrates that results on page 2 are generally overlooked/rarely seen?
- dpc59 11y agoI wonder what would happen if google added 50% more results per page.