6 ms·

These projects are very much needed, but I also believe that we need to go beyond simple "archiving" and have a way to make this data actually available to the

These projects are very much needed, but I also believe that we need to go beyond simple "archiving" and have a way to make this data actually available to the public. For example: I don't think that Reddit ever felt threatened by the fact that a bunch of people are pulling their data and creating a backup that can not be easily accessible. But I think that Reddit would be very much afraid of a distributed network of nodes running partial copies of Reddit and making it available for local-first clients and/or Lemmy mirrors.

tech234a 2y ago
Most projects output WARC files which are batched, uploaded to archive.org, and go into the Wayback Machine: https://web.archive.org/ https://web.archive.org/ (Note that Archive Team is separate from Internet Archive)
- lopkeny12ko 2y ago
  That still doesn't solve the discoverability problem. The Wayback Machine is only useful if you happen to know the URL where a piece of content used to live.
  - rglullis 2y ago
    That, and the fact that the data is "frozen". There is no way to add a comment to a Reddit thread that has been archived without going to Reddit first. I'd like to have a two-way mirror.
    - userbinator 2y ago
      Isn't read-only one of the main points of being "archived"?
      - rglullis 2y ago
        Yeah, but I am saying that I don't want it merely "archived". I want it "copied and available outside of Reddit's control".
        gaius_baltar 2y ago
        That's lemmy.ml . See this comment: https://news.ycombinator.com/item?id=41143632 https://news.ycombinator.com/item?id=41143632 edit: they already did.
        rglullis 2y ago
        Did you link to my own comment? :)
  - userbinator 2y ago
    Indeed, what's really needed is a search engine that indexes all the content on the Wayback Machine and makes it accessible in a similar fashion as e.g. Google.
    - card_zero 2y ago
      The Wayback Machine does that, to some extent. It tends to prefer to return results where the query is in the URL, but sometimes it picks up on the word being in the page. I don't know how much indexing it does, but it does some.
  - hexage1814 2y ago
    >That still doesn't solve the discoverability problem I have a somewhat of a conspiracy theory that deep down they don't implement a search feature on all their content on purpose. Essentially if WB made easy to discover stuff, you end up having to deal more and more with all those shenanigans of people requesting information to be removed. By making the information there, but somewhat unfindable or at least very hard to find, they essentially preserve the information, without having to deal with such problem (I know this happens even nowadays, but if it was easier to find information, it would happen even more).
    - Xen9 2y ago
      Raw genius, the site even looks "old" ... On related note, Internet Archive backup would probably cost between 20M and 60M USD. Many EU countries would have incentive to do this as public culture preservation projects. The archive is roughly 70 PB. Decentralized storage projects have achieved 7PB already. Thus attempting a decentralized backup would ALSO work.
      - mellosouls 2y ago
        Many EU countries would have incentive to do this as public culture preservation projects There are multiple archive projects around the world, which often isn't understood when the Internet Archive (the biggest and original) is discussed. Various countries including some EU already have those as "public culture preservation projects", targeting their own nation's web presence. In that context, and with scarce funding already, there is not really an incentive to back up a load of irrelevant (in the sense it's not their country's) archive material.
        Xen9 2y ago
        Targeting a nation's own web presence cannot capture as complete picture as IA. For example, discussions about minor languages that are vanishing are often English. In addition, IA does not only do archiving of the inernet; it's rather an archive ON the internet that scans, digitalizes & organizes tons of material. Now, notice that the budget of many of these projects are X billions: https://www.ne-mo.org/cooperation-funding/funding-opportunities https://www.ne-mo.org/cooperation-funding/funding-opportunit... Putting a 100 million EUR into an European IA backup would be more cost effective than any of these projects. Alternatively or additionally: https://en.m.wikipedia.org/wiki/Wikipedia:Fundraising_statistics https://en.m.wikipedia.org/wiki/Wikipedia:Fundraising_statis... Wikipedia could actually also probably – being dependent on IA – invest 50M into the project. In fact, this would probably do more what the donations were meant to do than anything else they could do with the ("excess") funds. Truth to be spoken, NSA probably has an IA backup. But it still sort of drives me insane to know that political change or natural catastrophes could lead to loss of public access to the IA. No one seems to care about IA enough except IA itself.
- rglullis 2y ago
  Right, but AFAIK there is no way to query a WARC file, is there? Let me explain where I am coming from. I'm working on an (open source, self-hosted) service to help people migrate from Reddit to Lemmy, called Fediverser [0]. It offers the following: - A crowdsourced map of "reddit-to-lemmy" alternatives. - Use the list of subreddits and some preferences to find a Lemmy instance that is suitable for you. - Lets people sign up to a "fediversed" Lemmy instance directly via Reddit OAuth. Simplifies the registration process, can let an admin skip the verification process (e.g, reject redditors whose account are less than 3 years old) - Using the crowdsourced data, automatically subscribe the user to the communities that correspond to their favorite subreddits. - If the admin of the Lemmy instance so chooses, it can also set up mirror bots, which will create "shadow accounts" for each reddit author. This shadow account can then be "taken over" by the real redditor if/when they sign up to the instance. I believe that these features together would lead to a credible threat to Reddit's dominance. My remaining "problem" to solve is, simply put, that I need more people running this, because it's just too much data for a single node. I set up an instance that was mirroring ~100 reddits (posts and comments). In three months, my database was already recording ~3 million "shadow" users and ~10 million posts + submissions. For this to work, I either need to have more instance admins willing to run the Fediverser software, or I need to move the "shadow" users and the mirrored content straight to the client and only bring to the Lemmy server the content from users who actually migrated. [0]: https://fediverser.io https://fediverser.io
  - abracadaniel 2y ago
    It’s an iso standard. There’s warc libraries around. Ex: https://github.com/webrecorder/warcio https://github.com/webrecorder/warcio
Centigonal 2y ago
I'm not sure if threatening a corporate entity who can change their API at any time is a feature here. See: PushShift, Nitter, Youtube Vanced
- rglullis 2y ago
  There are levels of survival we are prepared to accept... 1) A lot of Reddit's usefulness comes from the bots. If they shut down the API entirely, they would lose a lot of value which would accelerate their demise. 2) A more cynical person would say that without bots, Reddit would lose 30-40% of its "traffic", and that they can not afford to do that. This is why the current API is still quite generous. It's enough for most bots, but just too expensive for third-party clients. 3) Even if Reddit shut down its API tomorrow, the majority of interesting content has already been copied/archived. 4) Scraping old.reddit is quite easy, and getting rid of it altogether is not something that they are willing to do. All in all, I'd say that if we ever get to a point where Reddit is cutting down the API, it will be a time to celebrate.
  - gary_0 2y ago
    > Scraping old.reddit is quite easy, and getting rid of it altogether is not something that they are willing to do. Really? Unless there's an ironclad public statement to the contrary, the vibe I get from old.reddit is that it won't be long until the axe falls, especially since they are fine with "new" reddit features that malfunction in old.reddit. And I'd imagine their attitude towards unhappy users once they kill old.reddit will be the same as when they killed the API (which was "go fuck yourselves", to put it plainly).
    - rglullis 2y ago
      Maybe it was just my corner of Reddit, but last year when I was trying to convince people to move away. There were a good number of people who were just saying "I don't care about the API pricing changes because I just use old.reddit. If they do get rid of it, then I'll leave it right away." My feeling is that Reddit will continue to push and nudge people to new UI, but will not fully retire old. I think that if they ever do it it will be the final straw and mass migration will be inevitable.
notinmykernel 2y ago
Reddit was quite unhappy with services that mirrored their data and made it publicly available. So much so they updated TOS for the API, and forced services to delete data and shutter.
- rglullis 2y ago
  Yet, it doesn't take a motivated actor to do it. They certainly don't want to have other Big Tech companies exploring "their" data, but I'd argue they would only after someone who had a clear commercial interest. They are not going to chase a few dozen people on /r/DataHoarder running an Archive Warrior.
- immibis 2y ago
  So much so, they've now blocked all search engines except for Google. (Google paid a lot of money for this) Reddit's become low quality now anyway. All the good old high quality information that people search Reddit for is in the dumps - they're not generating much more of it.
deleted 2y ago
[deleted]
pyinstallwoes 2y ago
P2P ad hoc offline first internet that deprecates bgp and uses a globally memoized shared content/compute addressable memory via DHT/virtual machine. Identity for any thing is its composition. I can dream.

tech234a 2y ago

Most projects output WARC files which are batched, uploaded to archive.org, and go into the Wayback Machine: https://web.archive.org/ https://web.archive.org/ (Note that Archive Team is separate from Internet Archive)

lopkeny12ko 2y ago
That still doesn't solve the discoverability problem. The Wayback Machine is only useful if you happen to know the URL where a piece of content used to live.
- rglullis 2y ago
  That, and the fact that the data is "frozen". There is no way to add a comment to a Reddit thread that has been archived without going to Reddit first. I'd like to have a two-way mirror.
  - userbinator 2y ago
    Isn't read-only one of the main points of being "archived"?
    - rglullis 2y ago
      Yeah, but I am saying that I don't want it merely "archived". I want it "copied and available outside of Reddit's control".
      - gaius_baltar 2y ago
        That's lemmy.ml . See this comment: https://news.ycombinator.com/item?id=41143632 https://news.ycombinator.com/item?id=41143632 edit: they already did.
        rglullis 2y ago
        Did you link to my own comment? :)
- userbinator 2y ago
  Indeed, what's really needed is a search engine that indexes all the content on the Wayback Machine and makes it accessible in a similar fashion as e.g. Google.
  - card_zero 2y ago
    The Wayback Machine does that, to some extent. It tends to prefer to return results where the query is in the URL, but sometimes it picks up on the word being in the page. I don't know how much indexing it does, but it does some.
- hexage1814 2y ago
  >That still doesn't solve the discoverability problem I have a somewhat of a conspiracy theory that deep down they don't implement a search feature on all their content on purpose. Essentially if WB made easy to discover stuff, you end up having to deal more and more with all those shenanigans of people requesting information to be removed. By making the information there, but somewhat unfindable or at least very hard to find, they essentially preserve the information, without having to deal with such problem (I know this happens even nowadays, but if it was easier to find information, it would happen even more).
  - Xen9 2y ago
    Raw genius, the site even looks "old" ... On related note, Internet Archive backup would probably cost between 20M and 60M USD. Many EU countries would have incentive to do this as public culture preservation projects. The archive is roughly 70 PB. Decentralized storage projects have achieved 7PB already. Thus attempting a decentralized backup would ALSO work.
    - mellosouls 2y ago
      Many EU countries would have incentive to do this as public culture preservation projects There are multiple archive projects around the world, which often isn't understood when the Internet Archive (the biggest and original) is discussed. Various countries including some EU already have those as "public culture preservation projects", targeting their own nation's web presence. In that context, and with scarce funding already, there is not really an incentive to back up a load of irrelevant (in the sense it's not their country's) archive material.
      - Xen9 2y ago
        Targeting a nation's own web presence cannot capture as complete picture as IA. For example, discussions about minor languages that are vanishing are often English. In addition, IA does not only do archiving of the inernet; it's rather an archive ON the internet that scans, digitalizes & organizes tons of material. Now, notice that the budget of many of these projects are X billions: https://www.ne-mo.org/cooperation-funding/funding-opportunities https://www.ne-mo.org/cooperation-funding/funding-opportunit... Putting a 100 million EUR into an European IA backup would be more cost effective than any of these projects. Alternatively or additionally: https://en.m.wikipedia.org/wiki/Wikipedia:Fundraising_statistics https://en.m.wikipedia.org/wiki/Wikipedia:Fundraising_statis... Wikipedia could actually also probably – being dependent on IA – invest 50M into the project. In fact, this would probably do more what the donations were meant to do than anything else they could do with the ("excess") funds. Truth to be spoken, NSA probably has an IA backup. But it still sort of drives me insane to know that political change or natural catastrophes could lead to loss of public access to the IA. No one seems to care about IA enough except IA itself.
rglullis 2y ago
Right, but AFAIK there is no way to query a WARC file, is there? Let me explain where I am coming from. I'm working on an (open source, self-hosted) service to help people migrate from Reddit to Lemmy, called Fediverser [0]. It offers the following: - A crowdsourced map of "reddit-to-lemmy" alternatives. - Use the list of subreddits and some preferences to find a Lemmy instance that is suitable for you. - Lets people sign up to a "fediversed" Lemmy instance directly via Reddit OAuth. Simplifies the registration process, can let an admin skip the verification process (e.g, reject redditors whose account are less than 3 years old) - Using the crowdsourced data, automatically subscribe the user to the communities that correspond to their favorite subreddits. - If the admin of the Lemmy instance so chooses, it can also set up mirror bots, which will create "shadow accounts" for each reddit author. This shadow account can then be "taken over" by the real redditor if/when they sign up to the instance. I believe that these features together would lead to a credible threat to Reddit's dominance. My remaining "problem" to solve is, simply put, that I need more people running this, because it's just too much data for a single node. I set up an instance that was mirroring ~100 reddits (posts and comments). In three months, my database was already recording ~3 million "shadow" users and ~10 million posts + submissions. For this to work, I either need to have more instance admins willing to run the Fediverser software, or I need to move the "shadow" users and the mirrored content straight to the client and only bring to the Lemmy server the content from users who actually migrated. [0]: https://fediverser.io https://fediverser.io
- abracadaniel 2y ago
  It’s an iso standard. There’s warc libraries around. Ex: https://github.com/webrecorder/warcio https://github.com/webrecorder/warcio

Centigonal 2y ago

I'm not sure if threatening a corporate entity who can change their API at any time is a feature here. See: PushShift, Nitter, Youtube Vanced

rglullis 2y ago
There are levels of survival we are prepared to accept... 1) A lot of Reddit's usefulness comes from the bots. If they shut down the API entirely, they would lose a lot of value which would accelerate their demise. 2) A more cynical person would say that without bots, Reddit would lose 30-40% of its "traffic", and that they can not afford to do that. This is why the current API is still quite generous. It's enough for most bots, but just too expensive for third-party clients. 3) Even if Reddit shut down its API tomorrow, the majority of interesting content has already been copied/archived. 4) Scraping old.reddit is quite easy, and getting rid of it altogether is not something that they are willing to do. All in all, I'd say that if we ever get to a point where Reddit is cutting down the API, it will be a time to celebrate.
- gary_0 2y ago
  > Scraping old.reddit is quite easy, and getting rid of it altogether is not something that they are willing to do. Really? Unless there's an ironclad public statement to the contrary, the vibe I get from old.reddit is that it won't be long until the axe falls, especially since they are fine with "new" reddit features that malfunction in old.reddit. And I'd imagine their attitude towards unhappy users once they kill old.reddit will be the same as when they killed the API (which was "go fuck yourselves", to put it plainly).
  - rglullis 2y ago
    Maybe it was just my corner of Reddit, but last year when I was trying to convince people to move away. There were a good number of people who were just saying "I don't care about the API pricing changes because I just use old.reddit. If they do get rid of it, then I'll leave it right away." My feeling is that Reddit will continue to push and nudge people to new UI, but will not fully retire old. I think that if they ever do it it will be the final straw and mass migration will be inevitable.

notinmykernel 2y ago

Reddit was quite unhappy with services that mirrored their data and made it publicly available. So much so they updated TOS for the API, and forced services to delete data and shutter.

rglullis 2y ago
Yet, it doesn't take a motivated actor to do it. They certainly don't want to have other Big Tech companies exploring "their" data, but I'd argue they would only after someone who had a clear commercial interest. They are not going to chase a few dozen people on /r/DataHoarder running an Archive Warrior.
immibis 2y ago
So much so, they've now blocked all search engines except for Google. (Google paid a lot of money for this) Reddit's become low quality now anyway. All the good old high quality information that people search Reddit for is in the dumps - they're not generating much more of it.

deleted 2y ago

[deleted]

pyinstallwoes 2y ago

P2P ad hoc offline first internet that deprecates bgp and uses a globally memoized shared content/compute addressable memory via DHT/virtual machine. Identity for any thing is its composition. I can dream.