6 ms·

Most projects output WARC files which are batched, uploaded to archive.org, and go into the Wayback Machine: https://web.archive.org/ https://web.archive.org/

Most projects output WARC files which are batched, uploaded to archive.org, and go into the Wayback Machine: https://web.archive.org/ https://web.archive.org/ (Note that Archive Team is separate from Internet Archive)

lopkeny12ko 2y ago
That still doesn't solve the discoverability problem. The Wayback Machine is only useful if you happen to know the URL where a piece of content used to live.
- rglullis 2y ago
  That, and the fact that the data is "frozen". There is no way to add a comment to a Reddit thread that has been archived without going to Reddit first. I'd like to have a two-way mirror.
  - userbinator 2y ago
    Isn't read-only one of the main points of being "archived"?
    - rglullis 2y ago
      Yeah, but I am saying that I don't want it merely "archived". I want it "copied and available outside of Reddit's control".
      - gaius_baltar 2y ago
        That's lemmy.ml . See this comment: https://news.ycombinator.com/item?id=41143632 https://news.ycombinator.com/item?id=41143632 edit: they already did.
        rglullis 2y ago
        Did you link to my own comment? :)
- userbinator 2y ago
  Indeed, what's really needed is a search engine that indexes all the content on the Wayback Machine and makes it accessible in a similar fashion as e.g. Google.
  - card_zero 2y ago
    The Wayback Machine does that, to some extent. It tends to prefer to return results where the query is in the URL, but sometimes it picks up on the word being in the page. I don't know how much indexing it does, but it does some.
- hexage1814 2y ago
  >That still doesn't solve the discoverability problem I have a somewhat of a conspiracy theory that deep down they don't implement a search feature on all their content on purpose. Essentially if WB made easy to discover stuff, you end up having to deal more and more with all those shenanigans of people requesting information to be removed. By making the information there, but somewhat unfindable or at least very hard to find, they essentially preserve the information, without having to deal with such problem (I know this happens even nowadays, but if it was easier to find information, it would happen even more).
  - Xen9 2y ago
    Raw genius, the site even looks "old" ... On related note, Internet Archive backup would probably cost between 20M and 60M USD. Many EU countries would have incentive to do this as public culture preservation projects. The archive is roughly 70 PB. Decentralized storage projects have achieved 7PB already. Thus attempting a decentralized backup would ALSO work.
    - mellosouls 2y ago
      Many EU countries would have incentive to do this as public culture preservation projects There are multiple archive projects around the world, which often isn't understood when the Internet Archive (the biggest and original) is discussed. Various countries including some EU already have those as "public culture preservation projects", targeting their own nation's web presence. In that context, and with scarce funding already, there is not really an incentive to back up a load of irrelevant (in the sense it's not their country's) archive material.
      - Xen9 2y ago
        Targeting a nation's own web presence cannot capture as complete picture as IA. For example, discussions about minor languages that are vanishing are often English. In addition, IA does not only do archiving of the inernet; it's rather an archive ON the internet that scans, digitalizes & organizes tons of material. Now, notice that the budget of many of these projects are X billions: https://www.ne-mo.org/cooperation-funding/funding-opportunities https://www.ne-mo.org/cooperation-funding/funding-opportunit... Putting a 100 million EUR into an European IA backup would be more cost effective than any of these projects. Alternatively or additionally: https://en.m.wikipedia.org/wiki/Wikipedia:Fundraising_statistics https://en.m.wikipedia.org/wiki/Wikipedia:Fundraising_statis... Wikipedia could actually also probably – being dependent on IA – invest 50M into the project. In fact, this would probably do more what the donations were meant to do than anything else they could do with the ("excess") funds. Truth to be spoken, NSA probably has an IA backup. But it still sort of drives me insane to know that political change or natural catastrophes could lead to loss of public access to the IA. No one seems to care about IA enough except IA itself.
rglullis 2y ago
Right, but AFAIK there is no way to query a WARC file, is there? Let me explain where I am coming from. I'm working on an (open source, self-hosted) service to help people migrate from Reddit to Lemmy, called Fediverser [0]. It offers the following: - A crowdsourced map of "reddit-to-lemmy" alternatives. - Use the list of subreddits and some preferences to find a Lemmy instance that is suitable for you. - Lets people sign up to a "fediversed" Lemmy instance directly via Reddit OAuth. Simplifies the registration process, can let an admin skip the verification process (e.g, reject redditors whose account are less than 3 years old) - Using the crowdsourced data, automatically subscribe the user to the communities that correspond to their favorite subreddits. - If the admin of the Lemmy instance so chooses, it can also set up mirror bots, which will create "shadow accounts" for each reddit author. This shadow account can then be "taken over" by the real redditor if/when they sign up to the instance. I believe that these features together would lead to a credible threat to Reddit's dominance. My remaining "problem" to solve is, simply put, that I need more people running this, because it's just too much data for a single node. I set up an instance that was mirroring ~100 reddits (posts and comments). In three months, my database was already recording ~3 million "shadow" users and ~10 million posts + submissions. For this to work, I either need to have more instance admins willing to run the Fediverser software, or I need to move the "shadow" users and the mirrored content straight to the client and only bring to the Lemmy server the content from users who actually migrated. [0]: https://fediverser.io https://fediverser.io
- abracadaniel 2y ago
  It’s an iso standard. There’s warc libraries around. Ex: https://github.com/webrecorder/warcio https://github.com/webrecorder/warcio

lopkeny12ko 2y ago

That still doesn't solve the discoverability problem. The Wayback Machine is only useful if you happen to know the URL where a piece of content used to live.

rglullis 2y ago
That, and the fact that the data is "frozen". There is no way to add a comment to a Reddit thread that has been archived without going to Reddit first. I'd like to have a two-way mirror.
- userbinator 2y ago
  Isn't read-only one of the main points of being "archived"?
  - rglullis 2y ago
    Yeah, but I am saying that I don't want it merely "archived". I want it "copied and available outside of Reddit's control".
    - gaius_baltar 2y ago
      That's lemmy.ml . See this comment: https://news.ycombinator.com/item?id=41143632 https://news.ycombinator.com/item?id=41143632 edit: they already did.
      - rglullis 2y ago
        Did you link to my own comment? :)
userbinator 2y ago
Indeed, what's really needed is a search engine that indexes all the content on the Wayback Machine and makes it accessible in a similar fashion as e.g. Google.
- card_zero 2y ago
  The Wayback Machine does that, to some extent. It tends to prefer to return results where the query is in the URL, but sometimes it picks up on the word being in the page. I don't know how much indexing it does, but it does some.
hexage1814 2y ago
>That still doesn't solve the discoverability problem I have a somewhat of a conspiracy theory that deep down they don't implement a search feature on all their content on purpose. Essentially if WB made easy to discover stuff, you end up having to deal more and more with all those shenanigans of people requesting information to be removed. By making the information there, but somewhat unfindable or at least very hard to find, they essentially preserve the information, without having to deal with such problem (I know this happens even nowadays, but if it was easier to find information, it would happen even more).
- Xen9 2y ago
  Raw genius, the site even looks "old" ... On related note, Internet Archive backup would probably cost between 20M and 60M USD. Many EU countries would have incentive to do this as public culture preservation projects. The archive is roughly 70 PB. Decentralized storage projects have achieved 7PB already. Thus attempting a decentralized backup would ALSO work.
  - mellosouls 2y ago
    Many EU countries would have incentive to do this as public culture preservation projects There are multiple archive projects around the world, which often isn't understood when the Internet Archive (the biggest and original) is discussed. Various countries including some EU already have those as "public culture preservation projects", targeting their own nation's web presence. In that context, and with scarce funding already, there is not really an incentive to back up a load of irrelevant (in the sense it's not their country's) archive material.
    - Xen9 2y ago
      Targeting a nation's own web presence cannot capture as complete picture as IA. For example, discussions about minor languages that are vanishing are often English. In addition, IA does not only do archiving of the inernet; it's rather an archive ON the internet that scans, digitalizes & organizes tons of material. Now, notice that the budget of many of these projects are X billions: https://www.ne-mo.org/cooperation-funding/funding-opportunities https://www.ne-mo.org/cooperation-funding/funding-opportunit... Putting a 100 million EUR into an European IA backup would be more cost effective than any of these projects. Alternatively or additionally: https://en.m.wikipedia.org/wiki/Wikipedia:Fundraising_statistics https://en.m.wikipedia.org/wiki/Wikipedia:Fundraising_statis... Wikipedia could actually also probably – being dependent on IA – invest 50M into the project. In fact, this would probably do more what the donations were meant to do than anything else they could do with the ("excess") funds. Truth to be spoken, NSA probably has an IA backup. But it still sort of drives me insane to know that political change or natural catastrophes could lead to loss of public access to the IA. No one seems to care about IA enough except IA itself.

rglullis 2y ago

Right, but AFAIK there is no way to query a WARC file, is there? Let me explain where I am coming from. I'm working on an (open source, self-hosted) service to help people migrate from Reddit to Lemmy, called Fediverser [0]. It offers the following: - A crowdsourced map of "reddit-to-lemmy" alternatives. - Use the list of subreddits and some preferences to find a Lemmy instance that is suitable for you. - Lets people sign up to a "fediversed" Lemmy instance directly via Reddit OAuth. Simplifies the registration process, can let an admin skip the verification process (e.g, reject redditors whose account are less than 3 years old) - Using the crowdsourced data, automatically subscribe the user to the communities that correspond to their favorite subreddits. - If the admin of the Lemmy instance so chooses, it can also set up mirror bots, which will create "shadow accounts" for each reddit author. This shadow account can then be "taken over" by the real redditor if/when they sign up to the instance. I believe that these features together would lead to a credible threat to Reddit's dominance. My remaining "problem" to solve is, simply put, that I need more people running this, because it's just too much data for a single node. I set up an instance that was mirroring ~100 reddits (posts and comments). In three months, my database was already recording ~3 million "shadow" users and ~10 million posts + submissions. For this to work, I either need to have more instance admins willing to run the Fediverser software, or I need to move the "shadow" users and the mirrored content straight to the client and only bring to the Lemmy server the content from users who actually migrated. [0]: https://fediverser.io https://fediverser.io

abracadaniel 2y ago
It’s an iso standard. There’s warc libraries around. Ex: https://github.com/webrecorder/warcio https://github.com/webrecorder/warcio