12 ms·

We need archives built on decentralized storage. Don't get me wrong, I really like and support the work Internet Archive is doing, but preserving history is too

We need archives built on decentralized storage. Don't get me wrong, I really like and support the work Internet Archive is doing, but preserving history is too important to entrust it solely to singular entities, which means singular points of failure.

oytis 2y ago
We'll need to find even more people willing to expose themselves to legal threats and cyberattacks then.
- trompetenaccoun 2y ago
  The legal side is a big issue, true. The simplest and best workaround that I'm aware of is how the Arweave network handles it. They leave it up to the individual what parts of the data they want to host, but they're financially incentivized to take on rare data that others aren't hosting, because the rarer it is the more they get rewarded. Since it's decentralized and globally distributed, if something is risky to host in one jurisdiction, people in another can take that job and vice versa. The data also can not be altered after it's uploaded, and that's verifiable through hashes and sampling. Main downside in its current form is that decentralized storage isn't as fast as having central servers. And the experience can vary of course, depending on the host you connect to. As for technical attacks, I'm not an expert but I'd assume it's more difficult for bad actors to bring down decentralized networks. Has the BitTorrent network ever gone offline because it was hacked for example? That seems like it would be extremely hard to do, not even the movie industry managed to take them down.
  - Aachen 2y ago
    > decentralized storage isn't as fast as having central servers. With the 30-second "time to first byte" speed we all know and love from IA, I'm pretty sure it'd only get faster when you're the only person accessing an obscure document on a random person's shoebox in Korea as compared to trying to fetch it from a centralised server that has a few thousand other clients to attend to simultaneously
  - jmb99 2y ago
    > decentralized storage isn't as fast as having central servers. Depending on scale that’s not necessarily true. I find even today there are many services that cannot keep up with my residential fiber connection (3Gbps symmetrical), whereas torrents frequently can. IA in particular is notoriously slow when downloading from their servers, and even taking into account DHT time torrents can be much faster. Now if all of their PBs of data were cached in a CDN, yeah that’s probably faster than any decentralized solution. But that will take a heck of a lot more money to maintain than I think is possible for IA.
- Aachen 2y ago
  I collect, archive, and host data. Haven't gotten any threats or attacks. Not one. The average r/selfhosted user hiding their personal OwnCloud behind the DDoS maffia seems more afraid than one needs to be even for hosting all sorts of things publicly. I guess this fearmongering comes from tech news about breaches and DDoS attacks on organisations, similar to regular news impacting your regular worldview regardless of how it's actually going in the world or how things personally affect you
  - trod123 2y ago
    Its not a problem until it suddenly is, and by the time it becomes a problem its too late. Its not fear mongering, its risk management and the laws are draconian and fail fundamental basis for a "rule of law", we have a "rule by law".
    - Aachen 2y ago
      And that's how you get preppers: it's a remote problem until it's too late, let's prepare for every eventuality before it's suddenly too late! Risk management is a balance, not fearmongering as you say. That's why I'd rather use advice from people with daily experience than look at the newsworthy experiences ("nothing happened today, again; regular security patches working fine" you'll never see) and conclude you'd attract threats and cyber attacks just by hosting backup copies of parts of the Internet Archive
MattPalmer1086 2y ago
Lots of Copies Keeps Stuff Safe https://www.lockss.org/ https://www.lockss.org/ This is a brilliant system relying on a randomised consensus protocol. I wanted to do my info sec dissertation on it, but its security model is extremely well thought out. There wasn't anything I felt I could add to it.
- TZubiri 2y ago
  High Costs Makes Lots of Copies Unfeasible
  - MattPalmer1086 2y ago
    That was actually one of the key constraints in the LOCKSS system, since it was designed to be run by libraries that don't have big budgets. The design is really very good.
- ChadNauseam 2y ago
  I wish IPFS wasn't so wasteful with respect to storage. I tried pinning a 200mb PDF on IPFS and doing so ended up taking almost a gigabyte of disk space altogether. It's also relatively slow. However its implementation of global deduplication is super cool – it means that I can host 5 pages and you can host 50, and any overlap between them means we can both help one another keep them available even if we don't know about one another beforehand. For a large-scale archival project, it might not be ideal. Maybe something based on erasure coding would be better. Do you know how LOCKSS compares?
  - diggan 2y ago
    > I tried pinning a 200mb PDF on IPFS and doing so ended up taking almost a gigabyte of disk space altogether Was that any file in particular? I just tried it myself with a 257mb PDF (as reported by `ls -lrth`) and doesn't seem to add that much overhead: $ du -sh ~/.ipfs 84K /home/user/.ipfs $ ipfs add ~/Downloads/large\ PDF\ File.pdf added QmSvbEgCuRNZpkKyQm6nA5vz5RTHW1nxb6MJdR4cZUrnDj large PDF File.pdf 256.58 MiB / 256.58 MiB [============] 100.00% $ du -sh ~/.ipfs 264M /home/user/.ipfs
jdiff 2y ago
This seems to get brought at least once in the comments for every one of these articles that pops up. The IA has tried distributing their stores, but nowhere near enough people actually put their storage where their mouths are.
- immibis 2y ago
  Keep in mind the IA archives a lot of garbage. If it could be more focused it would be more likely to work.
  - db48x 2y ago
    The attempts have actually been focused on specific types of content, such as historical videos.
  - Blackthorn 2y ago
    The IA only works because it archives everything. You don't know what you need until you need it.
  - Spooky23 2y ago
    Archives generally purposefully don’t have a strong editorial streak. My trash is your treasure.
    - immibis 2y ago
      They have to if they don't want to use infinite space.
  - unleaded 2y ago
    personally I love all the random crap on IA!
- WarOnPrivacy 2y ago
  > nowhere near enough people actually put their storage where their mouths are. Typically because most people who have the upload, don't know that they can. And if they come to the notion on their own, they won't know how. If they put the notion to a search engine, the keywords they come up with probably don't return the needed ELI5 page. As in: How do I [?] for the Internet Archive?, most folks won't know what [?] needs to be.
sksxihve 2y ago
There's no real financial incentive for people to archive the data as a singular entity so even less for a distributed collection. Also it's probably easier to fund a single entity sufficiently so they can have security/code audits than a bunch of entities all trying to work together.
- riiii 2y ago
  Some people are motivated by more than just financial incentive.
  - sksxihve 2y ago
    That's true, but something like archiving the internet is very costly, IA has an annual budget in the tens of millions.
    - trompetenaccoun 2y ago
      Yes, it's a good point. Though they could take that money and reward people for hosting the data as well, couldn't they? They don't have to be in charge of hosting.
      - deleted 2y ago
        [deleted]
      - sksxihve 2y ago
        Yes, they could, that's not much different than a single company distributing the archive to multiple storage centers though. My original comment was about it being more cost effective for a single company to do that than coordinating with a bunch of disjoint entities.
        trompetenaccoun 2y ago
        Our digital memory shouldn't be in the hands of a small number of organizations in my view. You're right about cost effectiveness. There are pros and cons to both but it's not just external threats that have to be considered. History has always gotten rewritten throughout time. If you have a giant library it's easier for bad actors to gain influence and alter certain books, or remove them. This isn't just theoretical, under external pressure IA has already removed sites from its archive for copyright and political reasons. There are also threats that are generally not even considered because they happen with rare frequency, but when they happen they're devastating. The library of Alexandria was burned by Julius Caesar during a war. Likewise, if all your servers are in one country that geographic risk, they can get destroyed in the event of a war or such. No one expects this to happen today in the US, but archives should be robust long term, for decades, ideally even centuries.
__MatrixMan__ 2y ago
To make the web distributed-archive-friendly I think we need to start referencing things by hash and not by a path which some server has implied it will serve consistently but which actually shows you different data at different times for a million different reasons. If different data always gets a different reference, it's easy to know if you have enough backups of it. If the same name gets you a pile of snapshots taken under different conditions, it's hard to be sure which of those are the thing that we'd want to back up for that particular name.
- Cheer2171 2y ago
  Done. It is called IPFS. The IA already supports it. https://github.com/internetarchive/dweb-archive/blob/master/docs/archive_architecture_ipfs.md https://github.com/internetarchive/dweb-archive/blob/master/...
  - __MatrixMan__ 2y ago
    Right, what I'm saying is that now we need to get the rest of the web (or at least the parts we want to keep) on board.
  - majorchord 2y ago
    IPFS has shown that the protocol is fundamentally broken at the level of growth they want to achieve and it is already extremely slow as it is. It often takes several minutes to locate a single file.
    - diggan 2y ago
      The beauty is that IA could offer their own distribution of IPFS that uses their own DHT for example, and they could allow only public read access to it. This would solve the slow part of finding a file, for IA specifically. Then the actual transfers tend to be pretty quick with IPFS. What's the point of using IPFS then? Others can still spread the file elsewhere and verify it's the correct one, by using the exact same ID of the file, although on two different networks. The beauty of content-addressing I guess.
TechSquidTV 2y ago
This has really shown that the be true. I am stuck in a situation right now where I have some lost media I want to upload but they have been down for over a week. I plan to create a torrent in the meantime but that means relying on my personal network connection for the vast majority of downloads up front. I looked into CloudFlare R2, not terrible but not free either. I was looking into using R2 as a web seed for the torrent but I don't _really_ want to spend much to upload content that is going to get "stolen" and reuploaded by content farms anyway you know?
- tourmalinetaco 2y ago
  Why not subscribe to a seedbox? They’re about $5/2TB/mo. It protects your IP, you can buy for only the month, and since seedboxes are hosted in DMCA-resistant data centers you can download riskier torrents lightning fast, meaning you’re not just spending money for others, you can get something out of it too.
  - bigiain 2y ago
    Any hints or recommendations on how to find a decent seedbox vendor? (working email in profile if you'd rather not name any in public)
    - tourmalinetaco 2y ago
      I’ve only used r/Seedboxes on Reddit, and that’s yet to fail me. The specific one I mentioned is EvoSeedBox’s $5/mo tier with 130GB HDD + 2TB bandwidth which is all I’ve needed so far.
  - gruez 2y ago
    2TB of bandwidth or storage?
    - tourmalinetaco 2y ago
      Bandwidth, though some provide multi-TB storage (I assume you pay out the nose however).
Cheer2171 2y ago
You say this as if the IA is not already deeply invested in the DWeb movement. If you go to a DWeb event in the Bay Area, there is a good chance it will be held at the IA.
sschueller 2y ago
Yes, I was quite shocked when I found out that all their DCs are within driving distance.
- absence5875 2y ago
  [dead]
delfinom 2y ago
Yea so, who pays for the decentralized storage long term? What happens when someone storing decentralized data decides to exit? Will data be copied to multiple places, who is going to pay for doubling, tripling or more the storage costs for backups? Centralized entities emerge to absorb costs because nobody else can do it as efficiently alone.
- jmb99 2y ago
  At the moment, IA stores everything, and I imagine that most people are picturing a scenario where the decentralized data is in addition to IA's current servers. At least, that's the easiest bootstrapping path. >What happens when someone storing decentralized data decides to exit? They exit, and they no longer store decentralized data. At the very least, IA would still have their copy(s), and that data can be spread to other decentralized nodes once it has been determined (through timeouts, etc) that the person has exited. > Will data be copied to multiple places[...]? Ideally, yes. It is fairly trivial to determine the reliability of each member (uptime + hash checks), and reliable members (a few nines of uptime and hash matches) can be trusted to store data with fewer copies while unreliable members can store data with more copies. Could also balance that idea with data that's in higher demand, by storing hot data lots of times on less reliable members while storing cold data on more reliable members. > who pays for the decentralized storage long term? [...] who is going to pay for doubling, tripling or more the storage costs for backups? This is unanswered for pretty much any decentralized storage project, and is probably the only important question left. There are people who would likely contribute to some degree without a financial incentive, but ideally there would be some sort of reward. This in theory could be a good use for crypto, but I'd be concerned about the possible perverse incentives and the general disdain the average person has for crypto these days. Funding in general could come from donations received by IA, whatever excess they have beyond their operating costs and reserve requirements - likely would be nowhere near enough to make something like this "financially viable" (i.e. profitable) but it might be enough to convince people who were on the fence to chip in few hundred GB and some bandwidth. This is an open question though, and probably the main reason no decentralized storage project has really taken off.
NelsonMinar 2y ago
Is anyone using ArchiveBox regularly? It's a self-hosted archiving solution. Not the ambitious decentralized system I think this comment is thinking of but a practical way for someone to run an archive for themselves. https://archivebox.io/ https://archivebox.io/
- bigiain 2y ago
  @nikisweeting the dev of archivebox was active in a thread about out here last week. https://news.ycombinator.com/item?id=41860909 https://news.ycombinator.com/item?id=41860909 I'd never heard of it, but their responses to question and comments in that thread were really really good (and I now have "install and configure archivebox on the media server" on my upcoming weekend projects list).
- jumpingscript 2y ago
  I am self-hosting ArchiveBox through yunohost, for the odd blog article I come across and like. Not a heavy user per se, but it's doing its thing reliably.
johndhi 2y ago
Ipfs
throwaway984393 2y ago
[dead]
patcon 2y ago
The internet archive shepherded the early https://getdweb.net/ https://getdweb.net/ community, and works with groups like IPFS, so they're well aware and offering operational support to decentralized storage projects. This has been going since at least 2016 when I was involved in some projects involving environmental data archiving during the Trump transition
stavros 2y ago
I designed a system where you could say "donate this spare 2 TB of my disk space to the Internet Archive" and the IA would push 2 TB of data to you. This system also has the property that it can be reconstructed if the IA (or whatever provider) goes away. Unfortunately, when I talked to a few archival teams (including the IA) about whether they'd be interested in using it, I either got no response or a negative one.
- 4gotunameagain 2y ago
  Why reinvent the wheel ? There are so many proven distributed archiving systems, a lot of which are mentioned in these comments.
  - stavros 2y ago
    What system of these allows me to donate a bunch of disk space to a provider of my choosing, without thinking about it afterwards?
- whywhywhywhy 2y ago
  Because the incentive will be archiving things they believe should be archived, you need the process to begin with what urls do you want to be archiving, then people will be incentivized for archiving the juicy stuff IA is used for and you just throw some stuff they didn't ask to archive in the remit of them storing what they want.
  - stavros 2y ago
    I don't think that's necessarily true, I have a spare TB that I'd be glad to donate to the IA to store whatever they want in.
- boramalper 2y ago
  Is this open source or do you have any design docs? I love the idea and would love to learn more about it.
rootsudo 2y ago
I watched a hbo series about this once, I think it was called Pied Piper.
scirob 2y ago
LigGen / SciHub are model citizens in this regard. Use the best tech for the job . Torrents + IPFS + simple http mirrors all over . While their data is big its not as big as Archive.org https://libgen.is/repository_torrent/ https://libgen.is/repository_torrent/ so I guess one still needs the funding to go to these decentralized nodes and they are there mostly for resiliancy
- TZubiri 2y ago
  If so, why do they keep jumping from one dubious domain and TLD from another?
  - skeaker 2y ago
    That's mentioned in the comment you're replying to as "simple HTTP mirrors." Those are vulnerable to takedowns but are kept up for the sake of easy accessibility to the public that doesn't know how to access any of the other options. Their availability has no real bearing on the availability of the other options beyond the tech savviness of the general public.
sersi 2y ago
If there was a way, I could dedicate 10 TB to IA and have the system automatically use those 10TB to store whatever is best I'd love that. Right now there are torrents and I do keep any torrents I download from IA in my client for years but torrents means I only get to contribute by sharing the things I downloaded in the past.
deleted 2y ago
[deleted]
hacknewslogin 2y ago
This sounds like a task that could be taken care of by public libraries. That, and supporting Tor. I just don't see it happening anytime soon, at least in the US.

oytis 2y ago

We'll need to find even more people willing to expose themselves to legal threats and cyberattacks then.

trompetenaccoun 2y ago
The legal side is a big issue, true. The simplest and best workaround that I'm aware of is how the Arweave network handles it. They leave it up to the individual what parts of the data they want to host, but they're financially incentivized to take on rare data that others aren't hosting, because the rarer it is the more they get rewarded. Since it's decentralized and globally distributed, if something is risky to host in one jurisdiction, people in another can take that job and vice versa. The data also can not be altered after it's uploaded, and that's verifiable through hashes and sampling. Main downside in its current form is that decentralized storage isn't as fast as having central servers. And the experience can vary of course, depending on the host you connect to. As for technical attacks, I'm not an expert but I'd assume it's more difficult for bad actors to bring down decentralized networks. Has the BitTorrent network ever gone offline because it was hacked for example? That seems like it would be extremely hard to do, not even the movie industry managed to take them down.
- Aachen 2y ago
  > decentralized storage isn't as fast as having central servers. With the 30-second "time to first byte" speed we all know and love from IA, I'm pretty sure it'd only get faster when you're the only person accessing an obscure document on a random person's shoebox in Korea as compared to trying to fetch it from a centralised server that has a few thousand other clients to attend to simultaneously
- jmb99 2y ago
  > decentralized storage isn't as fast as having central servers. Depending on scale that’s not necessarily true. I find even today there are many services that cannot keep up with my residential fiber connection (3Gbps symmetrical), whereas torrents frequently can. IA in particular is notoriously slow when downloading from their servers, and even taking into account DHT time torrents can be much faster. Now if all of their PBs of data were cached in a CDN, yeah that’s probably faster than any decentralized solution. But that will take a heck of a lot more money to maintain than I think is possible for IA.
Aachen 2y ago
I collect, archive, and host data. Haven't gotten any threats or attacks. Not one. The average r/selfhosted user hiding their personal OwnCloud behind the DDoS maffia seems more afraid than one needs to be even for hosting all sorts of things publicly. I guess this fearmongering comes from tech news about breaches and DDoS attacks on organisations, similar to regular news impacting your regular worldview regardless of how it's actually going in the world or how things personally affect you
- trod123 2y ago
  Its not a problem until it suddenly is, and by the time it becomes a problem its too late. Its not fear mongering, its risk management and the laws are draconian and fail fundamental basis for a "rule of law", we have a "rule by law".
  - Aachen 2y ago
    And that's how you get preppers: it's a remote problem until it's too late, let's prepare for every eventuality before it's suddenly too late! Risk management is a balance, not fearmongering as you say. That's why I'd rather use advice from people with daily experience than look at the newsworthy experiences ("nothing happened today, again; regular security patches working fine" you'll never see) and conclude you'd attract threats and cyber attacks just by hosting backup copies of parts of the Internet Archive

MattPalmer1086 2y ago

Lots of Copies Keeps Stuff Safe https://www.lockss.org/ https://www.lockss.org/ This is a brilliant system relying on a randomised consensus protocol. I wanted to do my info sec dissertation on it, but its security model is extremely well thought out. There wasn't anything I felt I could add to it.

TZubiri 2y ago
High Costs Makes Lots of Copies Unfeasible
- MattPalmer1086 2y ago
  That was actually one of the key constraints in the LOCKSS system, since it was designed to be run by libraries that don't have big budgets. The design is really very good.
ChadNauseam 2y ago
I wish IPFS wasn't so wasteful with respect to storage. I tried pinning a 200mb PDF on IPFS and doing so ended up taking almost a gigabyte of disk space altogether. It's also relatively slow. However its implementation of global deduplication is super cool – it means that I can host 5 pages and you can host 50, and any overlap between them means we can both help one another keep them available even if we don't know about one another beforehand. For a large-scale archival project, it might not be ideal. Maybe something based on erasure coding would be better. Do you know how LOCKSS compares?
- diggan 2y ago
  > I tried pinning a 200mb PDF on IPFS and doing so ended up taking almost a gigabyte of disk space altogether Was that any file in particular? I just tried it myself with a 257mb PDF (as reported by `ls -lrth`) and doesn't seem to add that much overhead: $ du -sh ~/.ipfs 84K /home/user/.ipfs $ ipfs add ~/Downloads/large\ PDF\ File.pdf added QmSvbEgCuRNZpkKyQm6nA5vz5RTHW1nxb6MJdR4cZUrnDj large PDF File.pdf 256.58 MiB / 256.58 MiB [============] 100.00% $ du -sh ~/.ipfs 264M /home/user/.ipfs

jdiff 2y ago

This seems to get brought at least once in the comments for every one of these articles that pops up. The IA has tried distributing their stores, but nowhere near enough people actually put their storage where their mouths are.

immibis 2y ago
Keep in mind the IA archives a lot of garbage. If it could be more focused it would be more likely to work.
- db48x 2y ago
  The attempts have actually been focused on specific types of content, such as historical videos.
- Blackthorn 2y ago
  The IA only works because it archives everything. You don't know what you need until you need it.
- Spooky23 2y ago
  Archives generally purposefully don’t have a strong editorial streak. My trash is your treasure.
  - immibis 2y ago
    They have to if they don't want to use infinite space.
- unleaded 2y ago
  personally I love all the random crap on IA!
WarOnPrivacy 2y ago
> nowhere near enough people actually put their storage where their mouths are. Typically because most people who have the upload, don't know that they can. And if they come to the notion on their own, they won't know how. If they put the notion to a search engine, the keywords they come up with probably don't return the needed ELI5 page. As in: How do I [?] for the Internet Archive?, most folks won't know what [?] needs to be.

sksxihve 2y ago

There's no real financial incentive for people to archive the data as a singular entity so even less for a distributed collection. Also it's probably easier to fund a single entity sufficiently so they can have security/code audits than a bunch of entities all trying to work together.

riiii 2y ago
Some people are motivated by more than just financial incentive.
- sksxihve 2y ago
  That's true, but something like archiving the internet is very costly, IA has an annual budget in the tens of millions.
  - trompetenaccoun 2y ago
    Yes, it's a good point. Though they could take that money and reward people for hosting the data as well, couldn't they? They don't have to be in charge of hosting.
    - deleted 2y ago
      [deleted]
    - sksxihve 2y ago
      Yes, they could, that's not much different than a single company distributing the archive to multiple storage centers though. My original comment was about it being more cost effective for a single company to do that than coordinating with a bunch of disjoint entities.
      - trompetenaccoun 2y ago
        Our digital memory shouldn't be in the hands of a small number of organizations in my view. You're right about cost effectiveness. There are pros and cons to both but it's not just external threats that have to be considered. History has always gotten rewritten throughout time. If you have a giant library it's easier for bad actors to gain influence and alter certain books, or remove them. This isn't just theoretical, under external pressure IA has already removed sites from its archive for copyright and political reasons. There are also threats that are generally not even considered because they happen with rare frequency, but when they happen they're devastating. The library of Alexandria was burned by Julius Caesar during a war. Likewise, if all your servers are in one country that geographic risk, they can get destroyed in the event of a war or such. No one expects this to happen today in the US, but archives should be robust long term, for decades, ideally even centuries.

__MatrixMan__ 2y ago

To make the web distributed-archive-friendly I think we need to start referencing things by hash and not by a path which some server has implied it will serve consistently but which actually shows you different data at different times for a million different reasons. If different data always gets a different reference, it's easy to know if you have enough backups of it. If the same name gets you a pile of snapshots taken under different conditions, it's hard to be sure which of those are the thing that we'd want to back up for that particular name.

Cheer2171 2y ago
Done. It is called IPFS. The IA already supports it. https://github.com/internetarchive/dweb-archive/blob/master/docs/archive_architecture_ipfs.md https://github.com/internetarchive/dweb-archive/blob/master/...
- __MatrixMan__ 2y ago
  Right, what I'm saying is that now we need to get the rest of the web (or at least the parts we want to keep) on board.
- majorchord 2y ago
  IPFS has shown that the protocol is fundamentally broken at the level of growth they want to achieve and it is already extremely slow as it is. It often takes several minutes to locate a single file.
  - diggan 2y ago
    The beauty is that IA could offer their own distribution of IPFS that uses their own DHT for example, and they could allow only public read access to it. This would solve the slow part of finding a file, for IA specifically. Then the actual transfers tend to be pretty quick with IPFS. What's the point of using IPFS then? Others can still spread the file elsewhere and verify it's the correct one, by using the exact same ID of the file, although on two different networks. The beauty of content-addressing I guess.

TechSquidTV 2y ago

This has really shown that the be true. I am stuck in a situation right now where I have some lost media I want to upload but they have been down for over a week. I plan to create a torrent in the meantime but that means relying on my personal network connection for the vast majority of downloads up front. I looked into CloudFlare R2, not terrible but not free either. I was looking into using R2 as a web seed for the torrent but I don't _really_ want to spend much to upload content that is going to get "stolen" and reuploaded by content farms anyway you know?

tourmalinetaco 2y ago
Why not subscribe to a seedbox? They’re about $5/2TB/mo. It protects your IP, you can buy for only the month, and since seedboxes are hosted in DMCA-resistant data centers you can download riskier torrents lightning fast, meaning you’re not just spending money for others, you can get something out of it too.
- bigiain 2y ago
  Any hints or recommendations on how to find a decent seedbox vendor? (working email in profile if you'd rather not name any in public)
  - tourmalinetaco 2y ago
    I’ve only used r/Seedboxes on Reddit, and that’s yet to fail me. The specific one I mentioned is EvoSeedBox’s $5/mo tier with 130GB HDD + 2TB bandwidth which is all I’ve needed so far.
- gruez 2y ago
  2TB of bandwidth or storage?
  - tourmalinetaco 2y ago
    Bandwidth, though some provide multi-TB storage (I assume you pay out the nose however).

Cheer2171 2y ago

You say this as if the IA is not already deeply invested in the DWeb movement. If you go to a DWeb event in the Bay Area, there is a good chance it will be held at the IA.

sschueller 2y ago

Yes, I was quite shocked when I found out that all their DCs are within driving distance.

absence5875 2y ago
[dead]

delfinom 2y ago

Yea so, who pays for the decentralized storage long term? What happens when someone storing decentralized data decides to exit? Will data be copied to multiple places, who is going to pay for doubling, tripling or more the storage costs for backups? Centralized entities emerge to absorb costs because nobody else can do it as efficiently alone.

jmb99 2y ago
At the moment, IA stores everything, and I imagine that most people are picturing a scenario where the decentralized data is in addition to IA's current servers. At least, that's the easiest bootstrapping path. >What happens when someone storing decentralized data decides to exit? They exit, and they no longer store decentralized data. At the very least, IA would still have their copy(s), and that data can be spread to other decentralized nodes once it has been determined (through timeouts, etc) that the person has exited. > Will data be copied to multiple places[...]? Ideally, yes. It is fairly trivial to determine the reliability of each member (uptime + hash checks), and reliable members (a few nines of uptime and hash matches) can be trusted to store data with fewer copies while unreliable members can store data with more copies. Could also balance that idea with data that's in higher demand, by storing hot data lots of times on less reliable members while storing cold data on more reliable members. > who pays for the decentralized storage long term? [...] who is going to pay for doubling, tripling or more the storage costs for backups? This is unanswered for pretty much any decentralized storage project, and is probably the only important question left. There are people who would likely contribute to some degree without a financial incentive, but ideally there would be some sort of reward. This in theory could be a good use for crypto, but I'd be concerned about the possible perverse incentives and the general disdain the average person has for crypto these days. Funding in general could come from donations received by IA, whatever excess they have beyond their operating costs and reserve requirements - likely would be nowhere near enough to make something like this "financially viable" (i.e. profitable) but it might be enough to convince people who were on the fence to chip in few hundred GB and some bandwidth. This is an open question though, and probably the main reason no decentralized storage project has really taken off.

NelsonMinar 2y ago

Is anyone using ArchiveBox regularly? It's a self-hosted archiving solution. Not the ambitious decentralized system I think this comment is thinking of but a practical way for someone to run an archive for themselves. https://archivebox.io/ https://archivebox.io/

bigiain 2y ago
@nikisweeting the dev of archivebox was active in a thread about out here last week. https://news.ycombinator.com/item?id=41860909 https://news.ycombinator.com/item?id=41860909 I'd never heard of it, but their responses to question and comments in that thread were really really good (and I now have "install and configure archivebox on the media server" on my upcoming weekend projects list).
jumpingscript 2y ago
I am self-hosting ArchiveBox through yunohost, for the odd blog article I come across and like. Not a heavy user per se, but it's doing its thing reliably.

johndhi 2y ago

Ipfs

throwaway984393 2y ago

[dead]

patcon 2y ago

The internet archive shepherded the early https://getdweb.net/ https://getdweb.net/ community, and works with groups like IPFS, so they're well aware and offering operational support to decentralized storage projects. This has been going since at least 2016 when I was involved in some projects involving environmental data archiving during the Trump transition

stavros 2y ago

I designed a system where you could say "donate this spare 2 TB of my disk space to the Internet Archive" and the IA would push 2 TB of data to you. This system also has the property that it can be reconstructed if the IA (or whatever provider) goes away. Unfortunately, when I talked to a few archival teams (including the IA) about whether they'd be interested in using it, I either got no response or a negative one.

4gotunameagain 2y ago
Why reinvent the wheel ? There are so many proven distributed archiving systems, a lot of which are mentioned in these comments.
- stavros 2y ago
  What system of these allows me to donate a bunch of disk space to a provider of my choosing, without thinking about it afterwards?
whywhywhywhy 2y ago
Because the incentive will be archiving things they believe should be archived, you need the process to begin with what urls do you want to be archiving, then people will be incentivized for archiving the juicy stuff IA is used for and you just throw some stuff they didn't ask to archive in the remit of them storing what they want.
- stavros 2y ago
  I don't think that's necessarily true, I have a spare TB that I'd be glad to donate to the IA to store whatever they want in.
boramalper 2y ago
Is this open source or do you have any design docs? I love the idea and would love to learn more about it.

rootsudo 2y ago

I watched a hbo series about this once, I think it was called Pied Piper.

scirob 2y ago

LigGen / SciHub are model citizens in this regard. Use the best tech for the job . Torrents + IPFS + simple http mirrors all over . While their data is big its not as big as Archive.org https://libgen.is/repository_torrent/ https://libgen.is/repository_torrent/ so I guess one still needs the funding to go to these decentralized nodes and they are there mostly for resiliancy

TZubiri 2y ago
If so, why do they keep jumping from one dubious domain and TLD from another?
- skeaker 2y ago
  That's mentioned in the comment you're replying to as "simple HTTP mirrors." Those are vulnerable to takedowns but are kept up for the sake of easy accessibility to the public that doesn't know how to access any of the other options. Their availability has no real bearing on the availability of the other options beyond the tech savviness of the general public.

sersi 2y ago

If there was a way, I could dedicate 10 TB to IA and have the system automatically use those 10TB to store whatever is best I'd love that. Right now there are torrents and I do keep any torrents I download from IA in my client for years but torrents means I only get to contribute by sharing the things I downloaded in the past.

deleted 2y ago

[deleted]

hacknewslogin 2y ago

This sounds like a task that could be taken care of by public libraries. That, and supporting Tor. I just don't see it happening anytime soon, at least in the US.