3 ms·
robots this size aren't that unusual http://www.bbc.co.uk/robots.txt http://www.bbc.co.uk/robots.txt
by zalew 14y ago
robots this size aren't that unusual http://www.bbc.co.uk/robots.txt http://www.bbc.co.uk/robots.txt
- chayesfss 14y agowow, just realized this is a great way to find information about websites
- deleted 14y ago[deleted]
- thenextcorner 14y agoI think you just proved the point here: SEO can be very valuable. Any good SEO would point out to the BBC that having similar rules for User-agent: Googlebot-Mobile and User-agent: * is not very efficient.