5 ms·

Tbf, if you're lucky, there is web.archive.org and others like it, but yes, situations like those are common and disappointing.

Roark66 2y ago
It has became very obvious how reliant we are on Archive.org last time they had an extended outage (remember?, they got hacked,then they couldn't bring the system back up for weeks and weeks). Huge amounts of reference material suddenly dissappeared. Personally I think it is a huge error we don't force archive.org to allow others to mirror their data easily.
- trilbyglens 2y ago
  Probably the real error is in not having a publically funded mirror.
  - 3eb7988a1663 2y ago
    Certainly seems like something that the Library of Congress should be handling.
    - jacobgkau 2y ago
      The problem with this is that there's all kinds of user-generated web content that certain people wouldn't want tax dollars going towards preserving. I can already look at e.g. the National Film Registry and say I don't care about movies so my tax dollars shouldn't go towards preserving them, regardless of what some bureaucrat thinks is "culturally significant." The film industry has a pretty massive network of artists who are able to go to bat for that stuff, though. But then take pornography, or hate speech (and/or straight-up misinformation), or content that's illegal in the US but not elsewhere (like drug recipes or weapons schematics). It's already tricky for a private entity to handle convincing people of the value of preserving everything physically and legally possible to preserve. Making an argument for doing it as a public function, even in good faith, could easily turn into a mess-- especially when we can barely agree what should be allowed on the internet now.
- __jonas 2y ago
  > Personally I think it is a huge error we don't force archive.org to allow others to mirror their data easily. What do you mean by this? I thought they definitely were open to mirrors, why would ’we’ need to ‘force’ them to anything? IIRC the Bibliotheca Alexandrina in Egypt had a mirror of the web archive up, though they may have failed to maintain it
  - deleted 2y ago
    [deleted]
- Idesmi 2y ago
  It's not like the forbid mirroring. The problem is simply the size of it: around 100 petabytes.
johnfernow 2y ago
Right, archive.org is very useful if you have a URL, but if you're searching for a question that was answered in a forum, and that forum post no longer exists and no longer shows up in search results, then it's effectively undiscoverable as far as I'm aware. It amazes me how companies will have free volunteers help people to use their (often expensive) paid subscription products, and then delete all that info those volunteers wrote up. Don't they want people to use their products?! They're less likely to renew their subscription if they struggle or are unable to use the product for their particular use case. Unaffiliated forums not ran by the company are better in that the company can't decide to just delete all old posts one day (and while the owner could, certain types of unaffiliated forums are usually a bit easier to clone and republish.) The downside is you don't get assistance from people who work for that company, but often you rarely get that in official forums. The usual reason to use official forums is just that they have significantly more users asking and answering questions than unofficial ones.
- krackers 2y ago
  >that forum post no longer exists and no longer shows up in search results I dream of someone taking the internet archive data, capping it at 2010 or so, then making a search engine out of it. I mean if AI companies are looking to gobble all the data they can get, then surely they'd jump at the chance to train on (higher quality) data from the past that simply no longer exists on the web. So it'd seem like a win-win situation if IA gave them a copy of the data on the condition that they maintain a permanent backup and provide some sort of searchable index on the data (maybe even via LLM), and in turn the AI companies got access to high quality data on obscure topics that simply no longer exists.
  - jraph 2y ago
    Yup, let's not tie such an important endeavor to AI and AI startups though, we need something robust and lasting :-)
  - sangnoir 2y ago
    > I dream of someone taking the internet archive data, capping it at 2010 or so, then making a search engine out of it It sounds like you're describing CommonCrawl.org, and yes, it's already popular with AI companies.

Roark66 2y ago

It has became very obvious how reliant we are on Archive.org last time they had an extended outage (remember?, they got hacked,then they couldn't bring the system back up for weeks and weeks). Huge amounts of reference material suddenly dissappeared. Personally I think it is a huge error we don't force archive.org to allow others to mirror their data easily.

trilbyglens 2y ago
Probably the real error is in not having a publically funded mirror.
- 3eb7988a1663 2y ago
  Certainly seems like something that the Library of Congress should be handling.
  - jacobgkau 2y ago
    The problem with this is that there's all kinds of user-generated web content that certain people wouldn't want tax dollars going towards preserving. I can already look at e.g. the National Film Registry and say I don't care about movies so my tax dollars shouldn't go towards preserving them, regardless of what some bureaucrat thinks is "culturally significant." The film industry has a pretty massive network of artists who are able to go to bat for that stuff, though. But then take pornography, or hate speech (and/or straight-up misinformation), or content that's illegal in the US but not elsewhere (like drug recipes or weapons schematics). It's already tricky for a private entity to handle convincing people of the value of preserving everything physically and legally possible to preserve. Making an argument for doing it as a public function, even in good faith, could easily turn into a mess-- especially when we can barely agree what should be allowed on the internet now.
__jonas 2y ago
> Personally I think it is a huge error we don't force archive.org to allow others to mirror their data easily. What do you mean by this? I thought they definitely were open to mirrors, why would ’we’ need to ‘force’ them to anything? IIRC the Bibliotheca Alexandrina in Egypt had a mirror of the web archive up, though they may have failed to maintain it
- deleted 2y ago
  [deleted]
Idesmi 2y ago
It's not like the forbid mirroring. The problem is simply the size of it: around 100 petabytes.

johnfernow 2y ago

Right, archive.org is very useful if you have a URL, but if you're searching for a question that was answered in a forum, and that forum post no longer exists and no longer shows up in search results, then it's effectively undiscoverable as far as I'm aware. It amazes me how companies will have free volunteers help people to use their (often expensive) paid subscription products, and then delete all that info those volunteers wrote up. Don't they want people to use their products?! They're less likely to renew their subscription if they struggle or are unable to use the product for their particular use case. Unaffiliated forums not ran by the company are better in that the company can't decide to just delete all old posts one day (and while the owner could, certain types of unaffiliated forums are usually a bit easier to clone and republish.) The downside is you don't get assistance from people who work for that company, but often you rarely get that in official forums. The usual reason to use official forums is just that they have significantly more users asking and answering questions than unofficial ones.

krackers 2y ago
>that forum post no longer exists and no longer shows up in search results I dream of someone taking the internet archive data, capping it at 2010 or so, then making a search engine out of it. I mean if AI companies are looking to gobble all the data they can get, then surely they'd jump at the chance to train on (higher quality) data from the past that simply no longer exists on the web. So it'd seem like a win-win situation if IA gave them a copy of the data on the condition that they maintain a permanent backup and provide some sort of searchable index on the data (maybe even via LLM), and in turn the AI companies got access to high quality data on obscure topics that simply no longer exists.
- jraph 2y ago
  Yup, let's not tie such an important endeavor to AI and AI startups though, we need something robust and lasting :-)
- sangnoir 2y ago
  > I dream of someone taking the internet archive data, capping it at 2010 or so, then making a search engine out of it It sounds like you're describing CommonCrawl.org, and yes, it's already popular with AI companies.