5 ms·

I'll look into implementing this, nice tip :) archive.org is the worst offender for me; not only do they ignore robots.txt, there is absolutely no way to get s

I'll look into implementing this, nice tip :) archive.org is the worst offender for me; not only do they ignore robots.txt, there is absolutely no way to get something removed once they archived it (despite the data including accidentally leaked PII for example - which can cause actual harm to someone).

ShakataGaNai 4y ago
Just send them a DMCA request, that's their takedown mechanism. Is it a good one? No, but that's how they do it. You see it posted about all over in their forums. ex: https://archive.org/post/1022869/site-removal-request https://archive.org/post/1022869/site-removal-request
- gigel82 4y ago
  They ignored all emails sent to info@archive.org from the actual domain in question which I owned, with a link to a URL on the domain asking them to remove it. Don't know what the DMCA process is, I presume it involves lawyers and such.
Analemma_ 4y ago
I want archive.org to ignore robots.txt and make it as difficult as possible to remove pages from it; it would be a broken archive tool if this were not the case.
- anonymousiam 4y ago
  Sometimes there's a legitimate need to remove archived content. Back in the early days of running an ISP, I had some users hide links to copyrighted content with "map name" in a hosted image. Eventually the search engines began indexing these links and my net traffic exploded. I was not happy.
- gigel82 4y ago
  I hope you never accidentally publish private information anywhere on the web.
MereInterest 4y ago
Interesting, I've heard of the exact opposite issue with archive.org. That they take down archived pages at the request of the domain owner, even if registration lapsed and was picked up by somebody else since the snapshot was taken.
gwillen 4y ago
Are you mixing up archive.org (the Internet Archive, a nonprofit company headquartered in San Francisco) with archive.is/archive.ph/archive.today, which ignores robots.txt, and is a for-profit company with unknown owners and an unknown location?
- gigel82 4y ago
  No, I'm talking about archive.org, specifically their "wayback machine".
  - snoopy_telex 4y ago
    I had to have content removed from the wayback machine and it was pretty simple and painless all said. Perhaps it's gotten better over time?
hackernewds 4y ago
which is the point of archive.org - if there were human discretion it would turn into modern day Twitter

ShakataGaNai 4y ago

Just send them a DMCA request, that's their takedown mechanism. Is it a good one? No, but that's how they do it. You see it posted about all over in their forums. ex: https://archive.org/post/1022869/site-removal-request https://archive.org/post/1022869/site-removal-request

gigel82 4y ago
They ignored all emails sent to info@archive.org from the actual domain in question which I owned, with a link to a URL on the domain asking them to remove it. Don't know what the DMCA process is, I presume it involves lawyers and such.

Analemma_ 4y ago

I want archive.org to ignore robots.txt and make it as difficult as possible to remove pages from it; it would be a broken archive tool if this were not the case.

anonymousiam 4y ago
Sometimes there's a legitimate need to remove archived content. Back in the early days of running an ISP, I had some users hide links to copyrighted content with "map name" in a hosted image. Eventually the search engines began indexing these links and my net traffic exploded. I was not happy.
gigel82 4y ago
I hope you never accidentally publish private information anywhere on the web.

MereInterest 4y ago

Interesting, I've heard of the exact opposite issue with archive.org. That they take down archived pages at the request of the domain owner, even if registration lapsed and was picked up by somebody else since the snapshot was taken.

gwillen 4y ago

Are you mixing up archive.org (the Internet Archive, a nonprofit company headquartered in San Francisco) with archive.is/archive.ph/archive.today, which ignores robots.txt, and is a for-profit company with unknown owners and an unknown location?

gigel82 4y ago
No, I'm talking about archive.org, specifically their "wayback machine".
- snoopy_telex 4y ago
  I had to have content removed from the wayback machine and it was pretty simple and painless all said. Perhaps it's gotten better over time?

hackernewds 4y ago

which is the point of archive.org - if there were human discretion it would turn into modern day Twitter