3 ms·
http://digg.com/robots.txt http://digg.com/robots.txt <-- 404 Not Found I know a robots.txt 404 shouldn't de-list you, but I would have expected digg to have o
by kiallmacinnes 14y ago
http://digg.com/robots.txt http://digg.com/robots.txt <-- 404 Not Found
I know a robots.txt 404 shouldn't de-list you, but I would have expected digg to have one?
Maybe they requested de-listing and removed their robots.txt? Who knows!
- davenseo 14y agohttp://digg.com/robots.txt http://digg.com/robots.txt looks fine too me
- randomdata 14y agoThere is one there now: User-agent: * Disallow:
- kiallmacinnes 14y agoYup - It's back now.. But it 100% didn't exist when I posted earlier :)