24 ms·

Apple allows robots access to everything! http://www.apple.com/robots.txt http://www.apple.com/robots.txt

• oneeyedpigeon 13y ago
  If you're big enough, doesn't this just make sense? Why waste time maintaining a robots.txt policy when it must represent a tiny fraction of your traffic, which your servers can surely handle? And the really 'bad' guys are going to ignore it anyway. And if you really care, you'll have some much more sophisticated bandwidth throttling in place. For the smaller guys, sure it makes sense to have some kind of simple robots.txt policy.
  • keule 13y ago
    Thats not quite true. Apple has multiple subdomains, each for some part of their site. www.apple.com hosts most of the marketing stuff, but have a look at other pages, e.g.: http://store.apple.com/robots.txt http://store.apple.com/robots.txt
  • datawander 13y ago
    Actually, robot traffic accounts for a much larger share of traffic for any of the big websites than you think.