4 ms·

Where did the web archive go?

94 points by Hard_Space 5y ago

sysadm1n 5y ago
If you don't want to click: > Abstract. To perform a longitudinal investigation of web archives and detecting variations and changes replaying individual archived pages, or mementos, we created a sample of 16,627 mementos from 17 public web archives. Over the course of our 14-month study (November, 2017– January, 2019), we found that four web archives changed their baseURIs and did not leave a machine-readable method of locating their new base URIs, necessitating manual rediscovery. Of the 1,981 mementos in our sample from these four web archives, 537 were impacted: 517 mementos were rediscovered but with changes in their time of archiving(or Memento-Datetime), HTTP status code, or the string comprising their original URI (or URI-R), and 20 of the mementos could not be found at all.
- bfirsh 5y ago
  If you don't want to read a PDF: https://www.arxiv-vanity.com/papers/2108.05939/ https://www.arxiv-vanity.com/papers/2108.05939/
webmaven 5y ago
Relevant: "Cool URIs Don't Change": https://www.w3.org/Provider/Style/URI https://www.w3.org/Provider/Style/URI
- martindale 5y ago
  First thing that came to mind as well.
  - Cullinet 5y ago
    I honestly thought that Jakob Nielson earned his keep by having gotten the basics enough mind share to make the situation better than fond memories for turn of the century web designers. Being surrounded by designers at the time, all who became developers soon after, the loss of elementary design and UI knowledge I believe is due to the vacuum left behind when the on ramp to get a Internet job became programmatic instead of visual arts. However possibly the same effect can't happen in the same way again because early in the century coding for the web meant mainly fancy graphics and Flash, nothing related to let alone connected with the plumbing. (ActionSctipt was good enough for a friend who was a novice but desiccated and earnest programmer to write a Flash demo of a insulin regulator device that was so good his simulation from the specifications was used as a reference through to certification of the C and assembly embedded implementation. A ESP8266 for ActionSctipt would be awesome. This guy lost windows for Linux the instant the penny dropped that the greatest most important barrier to this profession, for the majority, is voodoo FUD and a matter of self confidence and reading the right books. This most recent generation doesn't seem to have a equivalent reading culture similar to how we assembled libraries of O'Reilley titles in our offices and homes - everyone who I remember purchased additional titles in fields that they didn't know about, every single time they bought required references something additional got smuggled into patiently awaiting brains. I copied this at my office letting anyone add a extra title every ten references. This almost immediately became a proper budget for a library but suddenly the requests dried up unexpectedly. We realised the attraction was having snuck the expense onto the job budget people took them home to read only afterwards bringing them back into circulation out of unnecessary sensitivity. This is why I passionately hate hermetically sealed corporate systems. Creativity needs inspiration even from a little lisp (or whatever) learning larceny.
- Cullinet 5y ago
  HTTP/1.1 Status Code Definitions : https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html It has always bugged me, e.g. to get 403: Forbidden when the intended message and situation is 401: Unauthorised. I am mesmerised to this day why HTTP, maybe a rapidly enforced required HTTP/1.2 spec, didn't make it mandatory for servers to enforce strict management of URI errors especially, together with a crud layer of enforcement and even FTP protocol extensions for providing a UI to enable intelligent use as standard. For example : if you follow the standard date formats including the date in the filename, it's trivial to implement 300: Multiple Choices whenever you confirm that you really do want to upload _National_News_Headlines_for_1995.12.25.htm after _National_News_Headlines_for_1995.12.24.htm
Chinjut 5y ago
A bit funny to have a paper on arxiv.org investigating archive.org (and europarchive.org and archive-it.org and so on).
- sp332 5y ago
  It does not investigate archive.org (although it did use it to track changes in the archives that were studied).
  - Cullinet 5y ago
    I will pay for or sponsor any project that reports on the existence of archive efforts, historical and current and planned. The existence of Archive.Org has led me to conclude that everyone has assumed that the problem is solved. We all know - here - how far that is from any consistent definition of reality.
    - someguy101010 5y ago
      If your are serious I'm down to do something like this.
4oo4 5y ago
Who archives the archive?
- scelerat 5y ago
  Needs more latin
  - MaxBarraclough 5y ago
    Assuming Roman numbers count, I suggest arx4.
    - Y_Y 5y ago
      ar0xE
  - edoceo 5y ago
    Isn't "archive" already Latin? Quis archivet ipsos archives?
  - Y_Y 5y ago
    Quis tuetur ipse tuendum. (Loose amateur translation, note that verbs for tabularium and archivum were not attested in my brief check.)
    - Y_Y 5y ago
      (that should be ipsum and I noticed too late to edit)
- Cullinet 5y ago
  My father used to hum or whistle this song, EDIT : [Who Looks After] The Caretaker's Daughter [When The Caretaker's Busy Taking Care?] whenever appropriate : https://www.google.com/search?q=caretakers+daughter+song https://www.google.com/search?q=caretakers+daughter+song
- thinkmassive 5y ago
  Everyone who runs ArchiveBox at home :) https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox
sam0x17 5y ago
If you don't want to read the title: "Where did the web archive go?"
- Bilal_io 5y ago
  If you don't want to read:
  - justinator 5y ago
    If:
    - deleted 5y ago
      [deleted]
    - midwestemo 5y ago
      ⠀‎
  - junon 5y ago
    This isn't reddit. Not only will this go over the heads of many HN users here, it's against the site's guidelines.
    - sam0x17 5y ago
      So you honestly believe that most users on HN are less intelligent than most users on Reddit? Now that's a hot take.
    - 0xdeadb00f 5y ago
      A bit of fun is against guidelines huh? That's news to me.
      - warent 5y ago
        The guidelines aren't really comprehensive. A lot of the content in HN is curated by the community, which is guided by tribal knowledge and unspoken rules. Fun and jokes are notoriously hard to get across in HN, which actually makes it all the more rewarding when it works. When the "fun" is just completely absent of originality and personality--just another regurgitated meme pattern of zero effort--many people just downvote and flag because the community doesn't want it.

94 points by Hard_Space 5y ago

If you don't want to click: > Abstract. To perform a longitudinal investigation of web archives and detecting variations and changes replaying individual archived pages, or mementos, we created a sample of 16,627 mementos from 17 public web archives. Over the course of our 14-month study (November, 2017– January, 2019), we found that four web archives changed their baseURIs and did not leave a machine-readable method of locating their new base URIs, necessitating manual rediscovery. Of the 1,981 mementos in our sample from these four web archives, 537 were impacted: 517 mementos were rediscovered but with changes in their time of archiving(or Memento-Datetime), HTTP status code, or the string comprising their original URI (or URI-R), and 20 of the mementos could not be found at all.

bfirsh 5y ago
If you don't want to read a PDF: https://www.arxiv-vanity.com/papers/2108.05939/ https://www.arxiv-vanity.com/papers/2108.05939/

webmaven 5y ago

Relevant: "Cool URIs Don't Change": https://www.w3.org/Provider/Style/URI https://www.w3.org/Provider/Style/URI

martindale 5y ago
First thing that came to mind as well.
- Cullinet 5y ago
  I honestly thought that Jakob Nielson earned his keep by having gotten the basics enough mind share to make the situation better than fond memories for turn of the century web designers. Being surrounded by designers at the time, all who became developers soon after, the loss of elementary design and UI knowledge I believe is due to the vacuum left behind when the on ramp to get a Internet job became programmatic instead of visual arts. However possibly the same effect can't happen in the same way again because early in the century coding for the web meant mainly fancy graphics and Flash, nothing related to let alone connected with the plumbing. (ActionSctipt was good enough for a friend who was a novice but desiccated and earnest programmer to write a Flash demo of a insulin regulator device that was so good his simulation from the specifications was used as a reference through to certification of the C and assembly embedded implementation. A ESP8266 for ActionSctipt would be awesome. This guy lost windows for Linux the instant the penny dropped that the greatest most important barrier to this profession, for the majority, is voodoo FUD and a matter of self confidence and reading the right books. This most recent generation doesn't seem to have a equivalent reading culture similar to how we assembled libraries of O'Reilley titles in our offices and homes - everyone who I remember purchased additional titles in fields that they didn't know about, every single time they bought required references something additional got smuggled into patiently awaiting brains. I copied this at my office letting anyone add a extra title every ten references. This almost immediately became a proper budget for a library but suddenly the requests dried up unexpectedly. We realised the attraction was having snuck the expense onto the job budget people took them home to read only afterwards bringing them back into circulation out of unnecessary sensitivity. This is why I passionately hate hermetically sealed corporate systems. Creativity needs inspiration even from a little lisp (or whatever) learning larceny.
Cullinet 5y ago
HTTP/1.1 Status Code Definitions : https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html It has always bugged me, e.g. to get 403: Forbidden when the intended message and situation is 401: Unauthorised. I am mesmerised to this day why HTTP, maybe a rapidly enforced required HTTP/1.2 spec, didn't make it mandatory for servers to enforce strict management of URI errors especially, together with a crud layer of enforcement and even FTP protocol extensions for providing a UI to enable intelligent use as standard. For example : if you follow the standard date formats including the date in the filename, it's trivial to implement 300: Multiple Choices whenever you confirm that you really do want to upload _National_News_Headlines_for_1995.12.25.htm after _National_News_Headlines_for_1995.12.24.htm

Chinjut 5y ago

A bit funny to have a paper on arxiv.org investigating archive.org (and europarchive.org and archive-it.org and so on).

sp332 5y ago
It does not investigate archive.org (although it did use it to track changes in the archives that were studied).
- Cullinet 5y ago
  I will pay for or sponsor any project that reports on the existence of archive efforts, historical and current and planned. The existence of Archive.Org has led me to conclude that everyone has assumed that the problem is solved. We all know - here - how far that is from any consistent definition of reality.
  - someguy101010 5y ago
    If your are serious I'm down to do something like this.

4oo4 5y ago

Who archives the archive?

scelerat 5y ago
Needs more latin
- MaxBarraclough 5y ago
  Assuming Roman numbers count, I suggest arx4.
  - Y_Y 5y ago
    ar0xE
- edoceo 5y ago
  Isn't "archive" already Latin? Quis archivet ipsos archives?
- Y_Y 5y ago
  Quis tuetur ipse tuendum. (Loose amateur translation, note that verbs for tabularium and archivum were not attested in my brief check.)
  - Y_Y 5y ago
    (that should be ipsum and I noticed too late to edit)
Cullinet 5y ago
My father used to hum or whistle this song, EDIT : [Who Looks After] The Caretaker's Daughter [When The Caretaker's Busy Taking Care?] whenever appropriate : https://www.google.com/search?q=caretakers+daughter+song https://www.google.com/search?q=caretakers+daughter+song
thinkmassive 5y ago
Everyone who runs ArchiveBox at home :) https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox

sam0x17 5y ago

If you don't want to read the title: "Where did the web archive go?"

Bilal_io 5y ago
If you don't want to read:
- justinator 5y ago
  If:
  - deleted 5y ago
    [deleted]
  - midwestemo 5y ago
    ⠀‎
- junon 5y ago
  This isn't reddit. Not only will this go over the heads of many HN users here, it's against the site's guidelines.
  - sam0x17 5y ago
    So you honestly believe that most users on HN are less intelligent than most users on Reddit? Now that's a hot take.
  - 0xdeadb00f 5y ago
    A bit of fun is against guidelines huh? That's news to me.
    - warent 5y ago
      The guidelines aren't really comprehensive. A lot of the content in HN is curated by the community, which is guided by tribal knowledge and unspoken rules. Fun and jokes are notoriously hard to get across in HN, which actually makes it all the more rewarding when it works. When the "fun" is just completely absent of originality and personality--just another regurgitated meme pattern of zero effort--many people just downvote and flag because the community doesn't want it.