3 ms·

A few years ago I talked to an IA engineer, who said they were planning on dealing with this by not crawling sites whose nameservers were known to point to a do

by jleader 12y ago

A few years ago I talked to an IA engineer, who said they were planning on dealing with this by not crawling sites whose nameservers were known to point to a domain parking company. The idea was that if they never retrieved the robots.txt, they wouldn't retroactively apply it. I don't know if that filtering out of parking nameservers ever happened, and it wouldn't help for parked domains whose robots.txt they'd already retrieved, but but it would help with domains that lapse in the future.

StavrosK 12y ago
But why retroactively remove the data? The original owner was fine with holding it, why should the snapshot be deleted because a completely different person wants his completely different website to not be crawled?
- bjt 12y ago
  It's hard for a bot to understand concepts of 'owner' and 'completely different person' based on the data they have available. Companies can use this robots.txt feature to un-index old marketing content after a re-branding, for example. Or after an acquisition.
  - StavrosK 12y ago
    Sure, but, surely, the bot has timestamps saying "robots.txt allowed me to keep these documents last time I spidered them". Why do they have to be retroactively removed? robots.txt only disallows spidering, it doesn't mandate that you should delete all the data you've already spidered.
    - db48x 12y ago
      Because most of the problems come from people who want to hide old material that they didn't realize was being indexed. The automatic behavior is simple and easy to implement, and doesn't require any human intervention.
- deleted 12y ago
  [deleted]
im3w1l 12y ago
Imo, a better approach than nuking old data, could be to keep the data, but not show it.
- psykovsky 12y ago
  I may be mistaken, but I think I read somewhere that's exactly what they do. But don't take my word for it.
  - ersii 12y ago
    You are not mistaken. The Internet Archive does not delete or "nuke" the data that is blocked by a robots.txt. Even though cough some people believe so (see parent thread). Source: IA staffers.

by jleader 12y ago

StavrosK 12y ago

But why retroactively remove the data? The original owner was fine with holding it, why should the snapshot be deleted because a completely different person wants his completely different website to not be crawled?

bjt 12y ago
It's hard for a bot to understand concepts of 'owner' and 'completely different person' based on the data they have available. Companies can use this robots.txt feature to un-index old marketing content after a re-branding, for example. Or after an acquisition.
- StavrosK 12y ago
  Sure, but, surely, the bot has timestamps saying "robots.txt allowed me to keep these documents last time I spidered them". Why do they have to be retroactively removed? robots.txt only disallows spidering, it doesn't mandate that you should delete all the data you've already spidered.
  - db48x 12y ago
    Because most of the problems come from people who want to hide old material that they didn't realize was being indexed. The automatic behavior is simple and easy to implement, and doesn't require any human intervention.
deleted 12y ago
[deleted]

im3w1l 12y ago

Imo, a better approach than nuking old data, could be to keep the data, but not show it.

psykovsky 12y ago
I may be mistaken, but I think I read somewhere that's exactly what they do. But don't take my word for it.
- ersii 12y ago
  You are not mistaken. The Internet Archive does not delete or "nuke" the data that is blocked by a robots.txt. Even though cough some people believe so (see parent thread). Source: IA staffers.