13 ms·

More than 9M broken links on Wikipedia are now rescued

689 points by infodocket 8y ago

ravenstine 8y ago
Archive.org is such a wonderful institution. The other day, I discovered that the Wayback Machine has been archiving YouTube videos in full HD. Most videos aren't on there, of course, and it seems to only go back as far as ~2012 (HTML5 video switchover?), but some of them are there. Y'all will be getting more donations from me. :)
- themodelplumber 8y ago
  Thanks, I had no idea about that. I found their Game Books Collection today, thought that was pretty neat. https://archive.org/details/gamebooks https://archive.org/details/gamebooks
- jxramos 8y ago
  I was wondering if something like that existed. A lot of times I add to my YouTube WatchLater list, which can often times be much later in tim that by the time I get around to it it’s not unusual for videos to be removed by the user or deleted for unknown reasons. And there’s no text to even see what the video was.
  - pymai 8y ago
    that does my head in. I have so many videos on most of my playlists that im hardly ever able to guess what the missing video was.
    - ravenstine 8y ago
      This is why I backup videos with youtube-dl. Internet history can be completely erased at a moment's notice, and some things ought to belong to the public if they're to never see the light of day again otherwise.
      - dylan604 8y ago
        I'd suggest a slight revision to your theorem: > Internet history can be completely erased at a momement's notice (unless it would embarrassing for you later in life)...
        ravenstine 8y ago
        Yes, indeed.
  - hoppelhase 8y ago
    In these cases, I just Google the video I'd. Enough to find its title somewhere.
twelvechairs 8y ago
The internet archive is great for static pages but what will happen for today's interactive content with complex data stored across different domains?
- voltagex_ 8y ago
  The Archive's crawlers will improve.
- tomatotomato37 8y ago
  They save JavaScript, flash apps, and even some downloads too. Just recently I used them to get an old flash game of a studio that went bust a couple years ago
- superkuh 8y ago
  Developers like using those time saving frameworks because it's easier. The management likes them because it makes linking to, downloading, or otherwise inspecting page content so much harder. For people that make those kind of sites it is a feature that you can't archive them easily. They are a cancer.
anon7429 8y ago
It's no suprise wikipedia is not as good as it could be, given dedicated corporate editors adding propaganda and stripping out criticism, and random editors with taglines like "independent blues artist" lacking credentials and education in subject-matter seemly deleting and alterting articles almost at random because they don't know the facts or what's important.
fermienrico 8y ago
I've always wondered this - How does Archive.org work in terms of storage? Internet is massive and caching every single site periodically for years on, isn't that unreasonably huge amount of data? Edit: I just checked Wikipedia, it says they're using about 15 PB of storage. Edit 2: 15 PB cost => 15,000 TB x $30/TB = $450,000. Ofcourse, back of the napkin cost (no maintenance, power, etc). That's not too bad actually.
- albertgoeswoof 8y ago
  Blockchain
  - antoineMoPa 8y ago
    I see this is downvoted. Actually, if a blockchain used proof of actual useful storage or computation, that'd be great. (instead of wasting energy by computing "useless" hashes)
    - QuercusMax 8y ago
      It's probably downvoted because replying 'blockchain' to a question about how Archive.org stores their data is not even a useful answer. There may well be fascinating ways one can use blockchain for archives of this sort, but the grandparent comment was essentially useless noise.
    - contravariant 8y ago
      I suppose that would be kind of interesting. Although if it's conceivable that someone could just store all the useful information then you can't just request a random piece of information as a proof of 'work', so you'd need something clever. Anyway the comment was probably downvoted because it consisted of a single buzzword.
    - X6S1x6Okd1st 8y ago
      Proof of storage is used in siacoin, but not to secure the network. I haven't seen a scheme where proof of storage can be used to secure the network. In terms of making sure the work that is done is useful it's fairly hard to do that because if you let someone dictate what work should be done then you potentially allow them to do the work upfront and a 51% attack becomes pretty easy.
jonah-archive 8y ago
If you're curious to learn more about us, we're hosting our big annual event in SF this Wednesday (Oct 3)! Details: https://blog.archive.org/2018/08/20/save-the-date-building-a-better-web-internet-archives-annual-bash/ https://blog.archive.org/2018/08/20/save-the-date-building-a...
- deleted 8y ago
  [deleted]
- jasonmunro 8y ago
  This will be my first year attending the annual event, and I'm super excited! I'm a long time fan of the Wayback Machine, even though my early work is terribly embarrassing :) Love what the archive does, which is so much more than snapshots of old websites. If you somehow have not read the mission statement, it's worth a minute: https://archive.org/about/ https://archive.org/about/
  - Intermernet 8y ago
    I'm now almost weirdly proud of the crap work of mine that I can find on the wayback machine :-) For instance: 2nd April 2004 [1] was my first crap Flash animation for my consultancy company at the time. It's terrible, but watching it now gives me so much nostalgia! [1]: WARNING: REQUIRES FLASH: https://web.archive.org/web/20040402230914/http://noisiadetrotsid.com:80/ https://web.archive.org/web/20040402230914/http://noisiadetr...
- jhabdas 8y ago
  If you're not hosting that site P2P yet you're just centralizing the distributed Web.
  - jonah-archive 8y ago
    http://dweb.archive.org http://dweb.archive.org -- previously: https://news.ycombinator.com/item?id=17685682 https://news.ycombinator.com/item?id=17685682
  - 8y ago
marknadal 8y ago
And, for those who don't know, you can help host the Internet Archive now by running a P2P/decentralized backup of it: https://news.ycombinator.com/item?id=17685682 https://news.ycombinator.com/item?id=17685682
- jacquesm 8y ago
  Thank you for pointing this out, I will see what I can do to help, this is the sort of thing I'm more than happy to dedicate resources to. I still have a 36 drive enclosure lying around that would make a nice bit of storage if I can get it to be silent enough for the home.
  - marknadal 8y ago
    This is the repo Mitra has been working on for this: https://github.com/internetarchive/dweb-mirror https://github.com/internetarchive/dweb-mirror
LoSboccacc 8y ago
And they are one robots away from cancellation. For all the good they do, retroactively applying robots exclusions to their crawler is a terrible thing. Luckily there are alternatives for going forward.
- slededit 8y ago
  They keep the data and just don’t display it. The last thing they need is a court order demanding they delete it. Sure the archive is useful today, but it’s primary purpose is retaining information for future generations. If that means placating copyright holders it’s worth the cost.
  - rovr138 8y ago
    > They keep the data and just don’t display it. I’ve read about the robots.txt mentioned before but hadn’t seen this mentioned. Any idea if they have this somewhere on the site?
    - slededit 8y ago
      Sure see here: https://archive.org/post/133690/robotstxt-only-gives-temporary-removal-of-pages https://archive.org/post/133690/robotstxt-only-gives-tempora...
  - dingaling 8y ago
    > If that means placating copyright holders it’s worth the cost. In many cases it's nothing to do with the copyright holder. In fact, the complete opposite: someone who subsequently bought the domain and unwittingly stuck a generic robots.txt on the site. If the IA hid content upon receiving a request from webmaster@domain then that'd make sense. But doing it automatically and retrospectively from robots.txt is bizarre and shows again the dangers of centralisation on the Internet. I now block IA from my sites as a protest. We need more competition and fresh ideas in digital archiving.
- dredmorbius 8y ago
  IA are ignoring robots.txt as of earlier this year.
solarkraft 8y ago
I spend quite some time on archive.org. Of course the wayback machine is great, but I am mostly interested in old digitized media. There really is some great stuff on there - But what is really missing is organization (and a less broken-seeming website, I guess). It doesn't help you much to have a great archive if noone will find anything. User-curation would help a lot with this.
- movedx 8y ago
  Found any gems you want to share? What you're doing is essentially digital archeology, which is super cool. In 50-100 years, if not sooner, people will be digging through digital "graveyards" for evidence of this and that. That's so intriguing.
  - sixdimensional 8y ago
    I like this term “digital archaeology”. I find myself saying it a lot when dealing with “only” 20 year old database data in my day job. Apparently, but not surprisingly, it’s a real thing[1]! [1] https://en.m.wikipedia.org/wiki/Digital_archaeology https://en.m.wikipedia.org/wiki/Digital_archaeology
    - InternetUser 8y ago
      There's long been a saying that "Once it's out there (on the Internet), it's forever," but I used to save links in a Microsoft Word document, and I went through them a few years later and almost none of them worked anymore. The years in which they were saved was 2006 to 2009, and the year I went through them was 2012. The links were from MySpace (which totally overhauled the entire site and all content), Facebook (where users had deleted their profiles or pictures), Tumblr (where bloggers rename their blogs, which change the URL, or they wipe them clean, or delete their blogs), YouTube (tons of videos and whole accounts have been deleted because of copyright infringement, whether by the account holder or by YouTube itself), Blogspot (same, but also that some bloggers made their blogs private, perhaps to prevent spam-comments or trolling), Yahoo articles (which I see Yahoo deletes after some time), Style.com (Vogue magazine's website of all runway shows, which are now on Vogue.com instead, with a different URL structre), and dozens of other websites that don't exist anymore. I think the statement about "stuff that's out there" really only applies to famous or public people, where leaked and/or damning photos or videos are quickly copied, saved, and rehosted by websites all over the world, including Twitter, Pinterest, and other platforms. For instead, while Google Images fastidiously won't show you the hacked photos of "Jennifer Lawrence naked," as Google sought to avoid a $100M lawsuit [0], Bing Images, once you turn off Safe Search, shows plenty of sites that host the pictures, with most frequent such site being a German-based one called "OhFree," but there are at least 3 Blogspot sites as well, I suppose ironically. [0] > "We've removed tens of thousands of pictures," says the web giant - https://www.hollywoodreporter.com/thr-esq/google-responds-jennifer-lawrence-attorneys-737656 https://www.hollywoodreporter.com/thr-esq/google-responds-je...
Hard_Space 8y ago
It's an invaluable resource. In the past week I realized it is quite likely that a site I used to work at which hosts a lot of my portfolio might disappear or be heavily amended. So I located and indexed a complete list of my articles there -- and I was even able to click a button and create an archive for the few pages that IA hadn't bothered to index (I wish I had known about this before, since it was a shock to find that IA can be quite selective, and to find a page you were hoping was there simply isn't, and is now irretrievable). But as one other commenter here has mentioned, you're only a robots.txt amend away from the oblivion that the entire IMDB comments section fell into [1], so a good archiving system is essential. I use (no affiliation)Save Page WE on Waterfox: https://addons.mozilla.org/en-US/firefox/addon/save-page-we/ https://addons.mozilla.org/en-US/firefox/addon/save-page-we/ [1] https://news.ycombinator.com/item?id=13571893 https://news.ycombinator.com/item?id=13571893
- zmw 8y ago
  Wayback Machine isn’t Googlebot, it doesn’t crawl the web, so there’s no such thing as “hadn’t bother to index”... Someone, be it a human or a bot, needs to submit a page for archival. Programmatically submitting to Wayback Machine is trivial enough, so I have cron jobs backing up most of my static sites (in their entirety) periodically.
- duckington 8y ago
  "bothered to index" and "IA can be quite selective" implies that there's a human overseer to the Wayback operation, not true! Pages are archived automatically, and sites are crawled by robots, not humans.
modeless 8y ago
This just reminded me to donate. I've used the archive several times just in the past few days to resolve 404s on old gamedev blogs. I'm amazed how often what I'm looking for is in the archive considering how big the internet is and how niche the content I'm looking for. Truly an amazing resource comparable to Wikipedia in value.
- ahmedalsudani 8y ago
  To everyone considering donating, please set up a monthly donation if it’s within your means! I used to donate every time something reminded me of the value of the archive. Now I just think “that’s why I have a monthly pledge!”
  - toomuchtodo 8y ago
    Also, your employer may match charitable contributions (up to a predefined amount). Check if they do! It’s effectively free money for the Internet Archive.
    - modeless 8y ago
      Googlers can donate with two clicks on G-Give, with Google matching. I highly recommend it.
agumonkey 8y ago
a worthy donation candidate ;)
bobochan 8y ago
I got a call a few years ago from a member of a humanitarian organization that had accidentally lost a significant percentage of their web site detailing projects that they had completed over many years with no backups. The people that had completed the work had moved on and they were frantic that the work was gone forever, but the Wayback Machine had almost perfect records to restore everything.
- toomuchtodo 8y ago
  There is a github project out there where you can specify the site, and it will rebuild the content locally from wayback content. Something to consider for last resort recovery. EDIT: https://github.com/oduwsdl/warrick https://github.com/oduwsdl/warrick
ta3216 8y ago
If a tree falls in the forest and nobody is there to hear it, does it make a noise? If IA is storing copyrighted (noarchive) content but not displaying it, does that make it acceptable?
- deleted 8y ago
  [deleted]
- pixelpoet 8y ago
  Yes, because then it's saved. One day we'll have a little memory stick that contains the collected and organised works of mankind (all movies, all music, all websites, digitised books, ...) and it'll be thanks to the archivists.
- Benjamin_Dobell 8y ago
  Yes, because copyright only lasts for a finite duration. In 100+ years when the original right-holders have disappeared, or when copyright expires and the original right-holders have no incentive to keep the originals; IA archive and similar archiving efforts will be able to make their copies available as a matter of historical importance.
- testvox 8y ago
  Yes at least in the United State copyright law has exemptions for archival purposes. http://www.ala.org/advocacy/copyright/dmca/section108 http://www.ala.org/advocacy/copyright/dmca/section108
Cogito 8y ago
They mention at the end of the article that 'content drift' may be a bigger issue than link rot; when the content of the post is simply changed rather than missing, it is much harder to notice. Is there a scalable way to monitor Wikipedia links to see if the content is changed after originally being posted? They are already storing every link in the Internet Archive when it gets added, so there should be a reference point to compare against. One easy option would be to make Internet Archive links available for every single link on Wikipedia, even if it hasn't rotted yet. So a 'live' link to the current content, and an archive link for what it was at the time of linking.
- deleted 8y ago
  [deleted]
- dredmorbius 8y ago
  You might start with Wikipedia article's history pages.
  - Cogito 8y ago
    Does the article's history page keep a record of what the content of the link was when it was updated? I guess it doesn't, but assuming that IA records every link as it gets added you can use the history page to see which snapshot on the wayback machine corresponds to the link at the time it was added. You could use that to compare to now, but unless you made sure to snapshot the content regularly (whenever the page is edited?) you wouldn't necessarily know when the content drift happened, or if multiple changes have happened to the content over time. Using the history page would likely give you enough information to, for example, do a study on links to find out how many have different content now vs when the link was added.
    - dredmorbius 8y ago
      Ah, no, that would be a separate problem. Though IA's archival-on-creation mechanism should at least mean that the orighinal reference is preserved. An explicit Wikipedia link to reference-at-time-of-archive, or better, a diff flag or listing, would be useful.
lulu90 8y ago
I'm living my best life, my business partner who we worked together few months referred me to deepwebservice01 AT gmail DOT com when I was very broke because of some bad business decisions and i got a transfers of $75,000 with their help. I was about going homeless but now i can pay my bills comfortably, they also help with credit repairs, carding and so many hacks
noufalibrahim 8y ago
I worked at the Archive for a few years remotely. It permanently altered my view of the tech. world. Here are the notable differences. I think these would apply to several non-profits but this is my experience 1. There was no rush to pick the latest technologies. Tried and tested was much better than new and shiny. Archive.org was mostly old PHP and shell scripts (atleast the parts I worked on). 2. The software was just a necessity. The data was what was valuable. Archive.org itself had tons of kluges and several crude bits of code to keep it going but the aim was the keep the data secure and it did that. Someone (maybe Brewster himself) likened it to a ship traveling through time. Several repairs with limited resources have permanently scarred the ship but the cargo is safe and pristine. When it finally arrives, the ship itself will be dismantled or might just crumble but the cargo will be there for the future. 3. Everything was super simple. Some of the techniques to run things etc. were absurdly simple and purposely so to help keep the thing manageable. Storage formats were straightforward so that even if a hard disk from the archive were found in a landfill a century from now, the contents would be usable (unlike if it were some kind of complex filesystem across multiple disks). 4. Brewster, and consequently the crew, were all dedicated to protecting the user. e.g. https://blog.archive.org/2011/01/04/brewster-kahle-receives-the-zoia-horn-intellectual-freedom-award/ https://blog.archive.org/2011/01/04/brewster-kahle-receives-.... There was code and stuff in place to not even accidentally collect data so that even if everything was confiscated, the user identities would be safe. 5. There was a mission. A serious social mission. Not just, "make money" or "build cool stuff" or anything. There was a buzz that made you feel like you were playing your role in mankinds intellectual history. That's an amazing feeling that I've never been able to replicate. Archive.org is truly only of the most underappreciated corners of the world wide web. Gives me faith in the positive potential of the internet.
- tannhaeuser 8y ago
  > There was a mission. A serious social mission. Not just, "make money" or "build cool stuff" or anything. There was a buzz that made you feel like you were playing your role in mankinds intellectual history. That's an amazing feeling that I've never been able to replicate. This resonates with me. Sometimes we developers need to get off the "move fast and break stuff" bandwagon (which has been ongoing for over decades now), and consider we're the ones responsible for preserving almost all human digital heritage of our epoch. There's a simple and obvious method to implement preservation-friendly content implicit in the web architecture: emit/materialize everything as plain HTML, even dynamic content. This is of course antithetical to most of this decade's SPA web development trends, but I think it's worth drawing a line between web content (worth preserving in the first place) and web apps (which have highly volatile content not worth preserving). I feel like this distinction isn't considered sufficiently in our staged web app architecture dicussions which are all about your latest JS MVw framework, to the degree that newby web devs really don't learn the fundamentals of HTML etc. anymore, and are lead to use React, Vue, etc. for content-based web sites.
iso-8859-1 8y ago
There was a big drama in 2012. http://Archive.is http://Archive.is was proactively archiving Wikipedia links. An unauthorized bot (RotlinkBot) was linking to Archive.is. The bot was banned. I liked how Archive.is was so fast at archiving, its UI more clean. And since it proactively archived links, it still happens today that a dead reference link will be archive in Archive.is, but not in the Wayback Machine. See https://en.wikipedia.org/wiki/User:RotlinkBot https://en.wikipedia.org/wiki/User:RotlinkBot
- duckington 8y ago
  I'm guessing Archive.is will probably disappear within the next 5 years, taking all data down with it. Nobody knows who owns or maintains the site, and recently the mysterious owner started taking donations to keep the site running. It's a commercial enterprise. Slick UI or not, Archive.org's longevity is probably more feasible.

689 points by infodocket 8y ago

Archive.org is such a wonderful institution. The other day, I discovered that the Wayback Machine has been archiving YouTube videos in full HD. Most videos aren't on there, of course, and it seems to only go back as far as ~2012 (HTML5 video switchover?), but some of them are there. Y'all will be getting more donations from me. :)

themodelplumber 8y ago
Thanks, I had no idea about that. I found their Game Books Collection today, thought that was pretty neat. https://archive.org/details/gamebooks https://archive.org/details/gamebooks
jxramos 8y ago
I was wondering if something like that existed. A lot of times I add to my YouTube WatchLater list, which can often times be much later in tim that by the time I get around to it it’s not unusual for videos to be removed by the user or deleted for unknown reasons. And there’s no text to even see what the video was.
- pymai 8y ago
  that does my head in. I have so many videos on most of my playlists that im hardly ever able to guess what the missing video was.
  - ravenstine 8y ago
    This is why I backup videos with youtube-dl. Internet history can be completely erased at a moment's notice, and some things ought to belong to the public if they're to never see the light of day again otherwise.
    - dylan604 8y ago
      I'd suggest a slight revision to your theorem: > Internet history can be completely erased at a momement's notice (unless it would embarrassing for you later in life)...
      - ravenstine 8y ago
        Yes, indeed.
- hoppelhase 8y ago
  In these cases, I just Google the video I'd. Enough to find its title somewhere.

twelvechairs 8y ago

The internet archive is great for static pages but what will happen for today's interactive content with complex data stored across different domains?

voltagex_ 8y ago
The Archive's crawlers will improve.
tomatotomato37 8y ago
They save JavaScript, flash apps, and even some downloads too. Just recently I used them to get an old flash game of a studio that went bust a couple years ago
superkuh 8y ago
Developers like using those time saving frameworks because it's easier. The management likes them because it makes linking to, downloading, or otherwise inspecting page content so much harder. For people that make those kind of sites it is a feature that you can't archive them easily. They are a cancer.

anon7429 8y ago

It's no suprise wikipedia is not as good as it could be, given dedicated corporate editors adding propaganda and stripping out criticism, and random editors with taglines like "independent blues artist" lacking credentials and education in subject-matter seemly deleting and alterting articles almost at random because they don't know the facts or what's important.

fermienrico 8y ago

I've always wondered this - How does Archive.org work in terms of storage? Internet is massive and caching every single site periodically for years on, isn't that unreasonably huge amount of data? Edit: I just checked Wikipedia, it says they're using about 15 PB of storage. Edit 2: 15 PB cost => 15,000 TB x $30/TB = $450,000. Ofcourse, back of the napkin cost (no maintenance, power, etc). That's not too bad actually.

albertgoeswoof 8y ago
Blockchain
- antoineMoPa 8y ago
  I see this is downvoted. Actually, if a blockchain used proof of actual useful storage or computation, that'd be great. (instead of wasting energy by computing "useless" hashes)
  - QuercusMax 8y ago
    It's probably downvoted because replying 'blockchain' to a question about how Archive.org stores their data is not even a useful answer. There may well be fascinating ways one can use blockchain for archives of this sort, but the grandparent comment was essentially useless noise.
  - contravariant 8y ago
    I suppose that would be kind of interesting. Although if it's conceivable that someone could just store all the useful information then you can't just request a random piece of information as a proof of 'work', so you'd need something clever. Anyway the comment was probably downvoted because it consisted of a single buzzword.
  - X6S1x6Okd1st 8y ago
    Proof of storage is used in siacoin, but not to secure the network. I haven't seen a scheme where proof of storage can be used to secure the network. In terms of making sure the work that is done is useful it's fairly hard to do that because if you let someone dictate what work should be done then you potentially allow them to do the work upfront and a 51% attack becomes pretty easy.

jonah-archive 8y ago

If you're curious to learn more about us, we're hosting our big annual event in SF this Wednesday (Oct 3)! Details: https://blog.archive.org/2018/08/20/save-the-date-building-a-better-web-internet-archives-annual-bash/ https://blog.archive.org/2018/08/20/save-the-date-building-a...

deleted 8y ago
[deleted]
jasonmunro 8y ago
This will be my first year attending the annual event, and I'm super excited! I'm a long time fan of the Wayback Machine, even though my early work is terribly embarrassing :) Love what the archive does, which is so much more than snapshots of old websites. If you somehow have not read the mission statement, it's worth a minute: https://archive.org/about/ https://archive.org/about/
- Intermernet 8y ago
  I'm now almost weirdly proud of the crap work of mine that I can find on the wayback machine :-) For instance: 2nd April 2004 [1] was my first crap Flash animation for my consultancy company at the time. It's terrible, but watching it now gives me so much nostalgia! [1]: WARNING: REQUIRES FLASH: https://web.archive.org/web/20040402230914/http://noisiadetrotsid.com:80/ https://web.archive.org/web/20040402230914/http://noisiadetr...
jhabdas 8y ago
If you're not hosting that site P2P yet you're just centralizing the distributed Web.
- jonah-archive 8y ago
  http://dweb.archive.org http://dweb.archive.org -- previously: https://news.ycombinator.com/item?id=17685682 https://news.ycombinator.com/item?id=17685682
- 8y ago

marknadal 8y ago

And, for those who don't know, you can help host the Internet Archive now by running a P2P/decentralized backup of it: https://news.ycombinator.com/item?id=17685682 https://news.ycombinator.com/item?id=17685682

jacquesm 8y ago
Thank you for pointing this out, I will see what I can do to help, this is the sort of thing I'm more than happy to dedicate resources to. I still have a 36 drive enclosure lying around that would make a nice bit of storage if I can get it to be silent enough for the home.
- marknadal 8y ago
  This is the repo Mitra has been working on for this: https://github.com/internetarchive/dweb-mirror https://github.com/internetarchive/dweb-mirror

LoSboccacc 8y ago

And they are one robots away from cancellation. For all the good they do, retroactively applying robots exclusions to their crawler is a terrible thing. Luckily there are alternatives for going forward.

slededit 8y ago
They keep the data and just don’t display it. The last thing they need is a court order demanding they delete it. Sure the archive is useful today, but it’s primary purpose is retaining information for future generations. If that means placating copyright holders it’s worth the cost.
- rovr138 8y ago
  > They keep the data and just don’t display it. I’ve read about the robots.txt mentioned before but hadn’t seen this mentioned. Any idea if they have this somewhere on the site?
  - slededit 8y ago
    Sure see here: https://archive.org/post/133690/robotstxt-only-gives-temporary-removal-of-pages https://archive.org/post/133690/robotstxt-only-gives-tempora...
- dingaling 8y ago
  > If that means placating copyright holders it’s worth the cost. In many cases it's nothing to do with the copyright holder. In fact, the complete opposite: someone who subsequently bought the domain and unwittingly stuck a generic robots.txt on the site. If the IA hid content upon receiving a request from webmaster@domain then that'd make sense. But doing it automatically and retrospectively from robots.txt is bizarre and shows again the dangers of centralisation on the Internet. I now block IA from my sites as a protest. We need more competition and fresh ideas in digital archiving.
dredmorbius 8y ago
IA are ignoring robots.txt as of earlier this year.

solarkraft 8y ago

I spend quite some time on archive.org. Of course the wayback machine is great, but I am mostly interested in old digitized media. There really is some great stuff on there - But what is really missing is organization (and a less broken-seeming website, I guess). It doesn't help you much to have a great archive if noone will find anything. User-curation would help a lot with this.

movedx 8y ago
Found any gems you want to share? What you're doing is essentially digital archeology, which is super cool. In 50-100 years, if not sooner, people will be digging through digital "graveyards" for evidence of this and that. That's so intriguing.
- sixdimensional 8y ago
  I like this term “digital archaeology”. I find myself saying it a lot when dealing with “only” 20 year old database data in my day job. Apparently, but not surprisingly, it’s a real thing[1]! [1] https://en.m.wikipedia.org/wiki/Digital_archaeology https://en.m.wikipedia.org/wiki/Digital_archaeology
  - InternetUser 8y ago
    There's long been a saying that "Once it's out there (on the Internet), it's forever," but I used to save links in a Microsoft Word document, and I went through them a few years later and almost none of them worked anymore. The years in which they were saved was 2006 to 2009, and the year I went through them was 2012. The links were from MySpace (which totally overhauled the entire site and all content), Facebook (where users had deleted their profiles or pictures), Tumblr (where bloggers rename their blogs, which change the URL, or they wipe them clean, or delete their blogs), YouTube (tons of videos and whole accounts have been deleted because of copyright infringement, whether by the account holder or by YouTube itself), Blogspot (same, but also that some bloggers made their blogs private, perhaps to prevent spam-comments or trolling), Yahoo articles (which I see Yahoo deletes after some time), Style.com (Vogue magazine's website of all runway shows, which are now on Vogue.com instead, with a different URL structre), and dozens of other websites that don't exist anymore. I think the statement about "stuff that's out there" really only applies to famous or public people, where leaked and/or damning photos or videos are quickly copied, saved, and rehosted by websites all over the world, including Twitter, Pinterest, and other platforms. For instead, while Google Images fastidiously won't show you the hacked photos of "Jennifer Lawrence naked," as Google sought to avoid a $100M lawsuit [0], Bing Images, once you turn off Safe Search, shows plenty of sites that host the pictures, with most frequent such site being a German-based one called "OhFree," but there are at least 3 Blogspot sites as well, I suppose ironically. [0] > "We've removed tens of thousands of pictures," says the web giant - https://www.hollywoodreporter.com/thr-esq/google-responds-jennifer-lawrence-attorneys-737656 https://www.hollywoodreporter.com/thr-esq/google-responds-je...

Hard_Space 8y ago

It's an invaluable resource. In the past week I realized it is quite likely that a site I used to work at which hosts a lot of my portfolio might disappear or be heavily amended. So I located and indexed a complete list of my articles there -- and I was even able to click a button and create an archive for the few pages that IA hadn't bothered to index (I wish I had known about this before, since it was a shock to find that IA can be quite selective, and to find a page you were hoping was there simply isn't, and is now irretrievable). But as one other commenter here has mentioned, you're only a robots.txt amend away from the oblivion that the entire IMDB comments section fell into [1], so a good archiving system is essential. I use (no affiliation)Save Page WE on Waterfox: https://addons.mozilla.org/en-US/firefox/addon/save-page-we/ https://addons.mozilla.org/en-US/firefox/addon/save-page-we/ [1] https://news.ycombinator.com/item?id=13571893 https://news.ycombinator.com/item?id=13571893

zmw 8y ago
Wayback Machine isn’t Googlebot, it doesn’t crawl the web, so there’s no such thing as “hadn’t bother to index”... Someone, be it a human or a bot, needs to submit a page for archival. Programmatically submitting to Wayback Machine is trivial enough, so I have cron jobs backing up most of my static sites (in their entirety) periodically.
duckington 8y ago
"bothered to index" and "IA can be quite selective" implies that there's a human overseer to the Wayback operation, not true! Pages are archived automatically, and sites are crawled by robots, not humans.

modeless 8y ago

This just reminded me to donate. I've used the archive several times just in the past few days to resolve 404s on old gamedev blogs. I'm amazed how often what I'm looking for is in the archive considering how big the internet is and how niche the content I'm looking for. Truly an amazing resource comparable to Wikipedia in value.

ahmedalsudani 8y ago
To everyone considering donating, please set up a monthly donation if it’s within your means! I used to donate every time something reminded me of the value of the archive. Now I just think “that’s why I have a monthly pledge!”
- toomuchtodo 8y ago
  Also, your employer may match charitable contributions (up to a predefined amount). Check if they do! It’s effectively free money for the Internet Archive.
  - modeless 8y ago
    Googlers can donate with two clicks on G-Give, with Google matching. I highly recommend it.

agumonkey 8y ago

a worthy donation candidate ;)

bobochan 8y ago

I got a call a few years ago from a member of a humanitarian organization that had accidentally lost a significant percentage of their web site detailing projects that they had completed over many years with no backups. The people that had completed the work had moved on and they were frantic that the work was gone forever, but the Wayback Machine had almost perfect records to restore everything.

toomuchtodo 8y ago
There is a github project out there where you can specify the site, and it will rebuild the content locally from wayback content. Something to consider for last resort recovery. EDIT: https://github.com/oduwsdl/warrick https://github.com/oduwsdl/warrick

ta3216 8y ago

If a tree falls in the forest and nobody is there to hear it, does it make a noise? If IA is storing copyrighted (noarchive) content but not displaying it, does that make it acceptable?

deleted 8y ago
[deleted]
pixelpoet 8y ago
Yes, because then it's saved. One day we'll have a little memory stick that contains the collected and organised works of mankind (all movies, all music, all websites, digitised books, ...) and it'll be thanks to the archivists.
Benjamin_Dobell 8y ago
Yes, because copyright only lasts for a finite duration. In 100+ years when the original right-holders have disappeared, or when copyright expires and the original right-holders have no incentive to keep the originals; IA archive and similar archiving efforts will be able to make their copies available as a matter of historical importance.
testvox 8y ago
Yes at least in the United State copyright law has exemptions for archival purposes. http://www.ala.org/advocacy/copyright/dmca/section108 http://www.ala.org/advocacy/copyright/dmca/section108

Cogito 8y ago

They mention at the end of the article that 'content drift' may be a bigger issue than link rot; when the content of the post is simply changed rather than missing, it is much harder to notice. Is there a scalable way to monitor Wikipedia links to see if the content is changed after originally being posted? They are already storing every link in the Internet Archive when it gets added, so there should be a reference point to compare against. One easy option would be to make Internet Archive links available for every single link on Wikipedia, even if it hasn't rotted yet. So a 'live' link to the current content, and an archive link for what it was at the time of linking.

deleted 8y ago
[deleted]
dredmorbius 8y ago
You might start with Wikipedia article's history pages.
- Cogito 8y ago
  Does the article's history page keep a record of what the content of the link was when it was updated? I guess it doesn't, but assuming that IA records every link as it gets added you can use the history page to see which snapshot on the wayback machine corresponds to the link at the time it was added. You could use that to compare to now, but unless you made sure to snapshot the content regularly (whenever the page is edited?) you wouldn't necessarily know when the content drift happened, or if multiple changes have happened to the content over time. Using the history page would likely give you enough information to, for example, do a study on links to find out how many have different content now vs when the link was added.
  - dredmorbius 8y ago
    Ah, no, that would be a separate problem. Though IA's archival-on-creation mechanism should at least mean that the orighinal reference is preserved. An explicit Wikipedia link to reference-at-time-of-archive, or better, a diff flag or listing, would be useful.

lulu90 8y ago

I'm living my best life, my business partner who we worked together few months referred me to deepwebservice01 AT gmail DOT com when I was very broke because of some bad business decisions and i got a transfers of $75,000 with their help. I was about going homeless but now i can pay my bills comfortably, they also help with credit repairs, carding and so many hacks

noufalibrahim 8y ago

I worked at the Archive for a few years remotely. It permanently altered my view of the tech. world. Here are the notable differences. I think these would apply to several non-profits but this is my experience 1. There was no rush to pick the latest technologies. Tried and tested was much better than new and shiny. Archive.org was mostly old PHP and shell scripts (atleast the parts I worked on). 2. The software was just a necessity. The data was what was valuable. Archive.org itself had tons of kluges and several crude bits of code to keep it going but the aim was the keep the data secure and it did that. Someone (maybe Brewster himself) likened it to a ship traveling through time. Several repairs with limited resources have permanently scarred the ship but the cargo is safe and pristine. When it finally arrives, the ship itself will be dismantled or might just crumble but the cargo will be there for the future. 3. Everything was super simple. Some of the techniques to run things etc. were absurdly simple and purposely so to help keep the thing manageable. Storage formats were straightforward so that even if a hard disk from the archive were found in a landfill a century from now, the contents would be usable (unlike if it were some kind of complex filesystem across multiple disks). 4. Brewster, and consequently the crew, were all dedicated to protecting the user. e.g. https://blog.archive.org/2011/01/04/brewster-kahle-receives-the-zoia-horn-intellectual-freedom-award/ https://blog.archive.org/2011/01/04/brewster-kahle-receives-.... There was code and stuff in place to not even accidentally collect data so that even if everything was confiscated, the user identities would be safe. 5. There was a mission. A serious social mission. Not just, "make money" or "build cool stuff" or anything. There was a buzz that made you feel like you were playing your role in mankinds intellectual history. That's an amazing feeling that I've never been able to replicate. Archive.org is truly only of the most underappreciated corners of the world wide web. Gives me faith in the positive potential of the internet.

tannhaeuser 8y ago
> There was a mission. A serious social mission. Not just, "make money" or "build cool stuff" or anything. There was a buzz that made you feel like you were playing your role in mankinds intellectual history. That's an amazing feeling that I've never been able to replicate. This resonates with me. Sometimes we developers need to get off the "move fast and break stuff" bandwagon (which has been ongoing for over decades now), and consider we're the ones responsible for preserving almost all human digital heritage of our epoch. There's a simple and obvious method to implement preservation-friendly content implicit in the web architecture: emit/materialize everything as plain HTML, even dynamic content. This is of course antithetical to most of this decade's SPA web development trends, but I think it's worth drawing a line between web content (worth preserving in the first place) and web apps (which have highly volatile content not worth preserving). I feel like this distinction isn't considered sufficiently in our staged web app architecture dicussions which are all about your latest JS MVw framework, to the degree that newby web devs really don't learn the fundamentals of HTML etc. anymore, and are lead to use React, Vue, etc. for content-based web sites.

iso-8859-1 8y ago

There was a big drama in 2012. http://Archive.is http://Archive.is was proactively archiving Wikipedia links. An unauthorized bot (RotlinkBot) was linking to Archive.is. The bot was banned. I liked how Archive.is was so fast at archiving, its UI more clean. And since it proactively archived links, it still happens today that a dead reference link will be archive in Archive.is, but not in the Wayback Machine. See https://en.wikipedia.org/wiki/User:RotlinkBot https://en.wikipedia.org/wiki/User:RotlinkBot

duckington 8y ago
I'm guessing Archive.is will probably disappear within the next 5 years, taking all data down with it. Nobody knows who owns or maintains the site, and recently the mysterious owner started taking donations to keep the site running. It's a commercial enterprise. Slick UI or not, Archive.org's longevity is probably more feasible.