5 ms·

I've been an editor on Wikipedia for years, and it's simply amazing how many web pages I referenced in 2008–09 have disappeared. Digital archivists have their h

I've been an editor on Wikipedia for years, and it's simply amazing how many web pages I referenced in 2008–09 have disappeared. Digital archivists have their hands full.

dredmorbius 10y ago
I had to migrate my archived articles off Readability prior to the OEL at the end of September. I'd only used the service for a couple of years, and hit about a 5% bitrot rate, this on fairly significant articles. One of the more curious cases was CSIRO (Australia's national science and research organisation) which seems to have not only deliberately purged a fair amount of data (Graham Turner's work specifically), but has a robots.txt in place which blocks archival by TIA. That strikes me as ... downright curious.
- caf 10y ago
  Which CSIRO site is that with the robots.txt? (it's not their main site). I'm thinking that could be something to have an Opposition Senator bring up in Senate Estimates.
- toomuchtodo 10y ago
  Could you reply with CSIRO links blocked by robots.txt? I run my own instance of ArchiveTeam's ArchiveBot (which archives links provided regardless of robots.txt), and would be happy to put the content into cold storage.
  - dredmorbius 10y ago
    I'll need to check my Readability dump.
    - toomuchtodo 10y ago
      No rush; I'll bookmark this thread to check two weeks from now.
      - dredmorbius 10y ago
        Found it. Apparently the robots.txt has been fixed: 404: http://www.csiro.au/en/Portals/Multimedia/CSIROpod/Growth-Limits.aspx http://www.csiro.au/en/Portals/Multimedia/CSIROpod/Growth-Li... Now available: http://web.archive.org/web/20120508210658/http://www.csiro.au/en/Portals/Multimedia/CSIROpod/Growth-Limits.aspx http://web.archive.org/web/20120508210658/http://www.csiro.a... That's among the specific links which wasn't being served by TIA earlier.
        toomuchtodo 10y ago
        Awesome. I've emailed CSIRO to try to track down that podcast included in the article that was not archived.
        dredmorbius 10y ago
        Thanks, I really appreciate this. dredmorbius@gmail.com if you happen to track that down.
        toomuchtodo 10y ago
        Emailed. Also, archiving all of the current version of csiro.au, just in case.
        dredmorbius 10y ago
        It seems to be climate, CO2, and limits-related work that is most prone to being censored.
        toomuchtodo 10y ago
        Not surprising based on the political climate in AU currently.
colejohnson66 10y ago
Whenever I cite a webpage, I try to archive the page at that moment and include the archive link in the citation. It's a bit more effort than just `<ref>[url]</ref>`, but it really is necessary.
noufalibrahim 10y ago
A little anecdote that might be interesting here. I worked at The Archive for a little while and one of the projects I worked on was to unpack about 300 TB of crawl data from the defunct search company cuil.com. It was mostly decent quality data in a standard format and after some grinding, the whole thing was converted into warc files which the wayback machine could use to show the URLs. The end result was that about 60 billion URLs came "back onto the web". During the work, I was looking at the stones rather than cathedral but after I left and thought about it in detail, it was very satisfying. I was reading the book "A Canticle For Leibowitz" at the time and the general theme of cycles of history was in my head. That dovetailed very well with the work I had done. If you're interested, you can download the dumps over here https://archive.org/details/cuilcrawl https://archive.org/details/cuilcrawl
- toomuchtodo 10y ago
  Upvote for "A Canticle For Leibowitz" alone. Thanks for your service at the Internet Archive!
  - noufalibrahim 10y ago
    It was a privilege to work there and definitely the best period of my career. So many wonderful people and experiences in such a short time. I personally consider Brewster Kahle a severely underrated hero of our age. The Archive's work is extremely valuable but I think the value will be appreciated only by a future generation. As for the book, I think if The Archive had novel for a totem, it'd be "A Canticle For Leibowitz". Very much affected my world view when I read it and this thread has just kindled my interest again. :)
    - toomuchtodo 10y ago
      If you don't have a paperback copy of A Canticle, I'd be happy to send you mine!
      - noufalibrahim 10y ago
noufalibrahim 10y ago
Here's another piece of irony for you. There was an article by Brewster about preserving the Internet in the scientific American which put the average lifespan of a URL at around 40 days. The said article now 404s and the only way to get it is through the wayback machine. :) http://web.archive.org/web/19970504212157/http://www.sciam.com/0397issue/0397kahle.html http://web.archive.org/web/19970504212157/http://www.sciam.c...
- jacquesm 10y ago
  Of course. The page is much older than 40 days...

I've been an editor on Wikipedia for years, and it's simply amazing how many web pages I referenced in 2008–09 have disappeared. Digital archivists have their hands full.

dredmorbius 10y ago

I had to migrate my archived articles off Readability prior to the OEL at the end of September. I'd only used the service for a couple of years, and hit about a 5% bitrot rate, this on fairly significant articles. One of the more curious cases was CSIRO (Australia's national science and research organisation) which seems to have not only deliberately purged a fair amount of data (Graham Turner's work specifically), but has a robots.txt in place which blocks archival by TIA. That strikes me as ... downright curious.

caf 10y ago
Which CSIRO site is that with the robots.txt? (it's not their main site). I'm thinking that could be something to have an Opposition Senator bring up in Senate Estimates.
toomuchtodo 10y ago
Could you reply with CSIRO links blocked by robots.txt? I run my own instance of ArchiveTeam's ArchiveBot (which archives links provided regardless of robots.txt), and would be happy to put the content into cold storage.
- dredmorbius 10y ago
  I'll need to check my Readability dump.
  - toomuchtodo 10y ago
    No rush; I'll bookmark this thread to check two weeks from now.
    - dredmorbius 10y ago
      Found it. Apparently the robots.txt has been fixed: 404: http://www.csiro.au/en/Portals/Multimedia/CSIROpod/Growth-Limits.aspx http://www.csiro.au/en/Portals/Multimedia/CSIROpod/Growth-Li... Now available: http://web.archive.org/web/20120508210658/http://www.csiro.au/en/Portals/Multimedia/CSIROpod/Growth-Limits.aspx http://web.archive.org/web/20120508210658/http://www.csiro.a... That's among the specific links which wasn't being served by TIA earlier.
      - toomuchtodo 10y ago
        Awesome. I've emailed CSIRO to try to track down that podcast included in the article that was not archived.
        dredmorbius 10y ago
        Thanks, I really appreciate this. dredmorbius@gmail.com if you happen to track that down.
        toomuchtodo 10y ago
        Emailed. Also, archiving all of the current version of csiro.au, just in case.
        dredmorbius 10y ago
        It seems to be climate, CO2, and limits-related work that is most prone to being censored.
        toomuchtodo 10y ago
        Not surprising based on the political climate in AU currently.

colejohnson66 10y ago

Whenever I cite a webpage, I try to archive the page at that moment and include the archive link in the citation. It's a bit more effort than just `<ref>[url]</ref>`, but it really is necessary.

noufalibrahim 10y ago

A little anecdote that might be interesting here. I worked at The Archive for a little while and one of the projects I worked on was to unpack about 300 TB of crawl data from the defunct search company cuil.com. It was mostly decent quality data in a standard format and after some grinding, the whole thing was converted into warc files which the wayback machine could use to show the URLs. The end result was that about 60 billion URLs came "back onto the web". During the work, I was looking at the stones rather than cathedral but after I left and thought about it in detail, it was very satisfying. I was reading the book "A Canticle For Leibowitz" at the time and the general theme of cycles of history was in my head. That dovetailed very well with the work I had done. If you're interested, you can download the dumps over here https://archive.org/details/cuilcrawl https://archive.org/details/cuilcrawl

toomuchtodo 10y ago
Upvote for "A Canticle For Leibowitz" alone. Thanks for your service at the Internet Archive!
- noufalibrahim 10y ago
  It was a privilege to work there and definitely the best period of my career. So many wonderful people and experiences in such a short time. I personally consider Brewster Kahle a severely underrated hero of our age. The Archive's work is extremely valuable but I think the value will be appreciated only by a future generation. As for the book, I think if The Archive had novel for a totem, it'd be "A Canticle For Leibowitz". Very much affected my world view when I read it and this thread has just kindled my interest again. :)
  - toomuchtodo 10y ago
    If you don't have a paperback copy of A Canticle, I'd be happy to send you mine!
    - noufalibrahim 10y ago

noufalibrahim 10y ago

Here's another piece of irony for you. There was an article by Brewster about preserving the Internet in the scientific American which put the average lifespan of a URL at around 40 days. The said article now 404s and the only way to get it is through the wayback machine. :) http://web.archive.org/web/19970504212157/http://www.sciam.com/0397issue/0397kahle.html http://web.archive.org/web/19970504212157/http://www.sciam.c...

jacquesm 10y ago
Of course. The page is much older than 40 days...