6 ms·

It would be better if the Internet Archive was decentralized without a central point of failure, maybe run on something like bittorent.

dusted 2y ago
I kind of agree, but the way the internet is going, with everyone being behind carrier-grade nat, it's not much of a decentralized network of computers anymore, not to mention all the kids with their laptops and tablets not even hosting anything :(
- Kuinox 2y ago
  UPnP exists and allow devices to ask the router to open a port to them.
  - cosarara 2y ago
    That doesnt help with CGNAT.
  - Dalewyn 2y ago
    UPnP is just automating the process of forwarding ports, CGNAT will still screw you sideways because you're behind a router you can't access or order around.
  - Fidelix 2y ago
    UPnP is useless with CGNAT (Carrier Grade NAT), which is what the op is talking about. There are other ways to get seeding working, though, including IPV6, which is gaining adoption, so I don't agree with the OP.
  - deleted 2y ago
    [deleted]
- nikisweeting 2y ago
  There are ways around this, I've experimented with setting up a cluster of ArchiveBox instances that share snapshots over Tailscale. Tailscale lets users sign up for free accounts, and you can share machines between separate accounts. A (CGNAT-compatible) decentralized invite-only network could concievably spread that way.
uniqueuid 2y ago
The problem is that it's hard to do this in a way that ensures good archival of ALL resources. Bittorrent works well for popular things but fails for marginal content (unless some really dedicated individuals step in.) What the internet archive provides is a way to have access to many many resources which you didn't know you needed in advance.
- deleted 2y ago
  [deleted]
- TZubiri 2y ago
  Lol, in a way the decentralized version is actually the internet.
  - vwkd 2y ago
    Or a brain.
maire 2y ago
I don't know if bittorrent has improved - but 20 years ago I had a personal issue with it. At that time our son was using it for games. He goes away to college and came home for the first school break. I get a phone call from our internet provider asking if our son was home. I was so shocked and handed the phone to our son. Apparently at that time bittorrent was optimizing for the most efficient path to a host. Since we had relatively good connection, the mighty weight of the internet was funnelling through our tiny internet provider to our son's computer. The provider (without our knowing it) had made a deal with our son that he would only turn on bittorrent between midnight and 6 AM. I doubt other providers would be so generous. I have been sceptical of bittorrent since that day.
- jetrink 2y ago
  All clients today (and probably back then) have options to limit bandwidth consumption including throttling, scheduling, and total data transfer caps. For serving mostly HTML and images, dedicating even 10% of a home broadband connection to serving content would allow many, many people per day to access archived pages.
Cheer2171 2y ago
You say this as if it is an original idea. Of course the IA is working on this and have been for over 6 years. There already is a DWeb version. They have been advancing DWeb infrastructure. The IA hosts all kinds of DWeb developer events. But it is over 50 petabytes and the IA gets a huge amount of traffic through the regular web that they need to serve quickly and efficiently to their users. Guess what has happened over 6 years of decentralization of 50 TB? People only seed what they want or care about and there aren't enough seeders to host. They set all this up and nobody volunteers. You're a DWeb advocate and you haven't been seeding. That's a recipe for disaster if they rely on the goodness of volunteer seeders. The IA's mission is broader. DWeb will ever only compliment the IAs mission. https://blog.archive.org/2021/02/18/behind-the-scenes-of-the-decentralized-web-principles/ https://blog.archive.org/2021/02/18/behind-the-scenes-of-the... https://www.bleepingcomputer.com/news/technology/archiveorg-has-created-a-decentralized-or-dweb-version-of-their-site/ https://www.bleepingcomputer.com/news/technology/archiveorg-...
- seestem 2y ago
  It is not my original idea, but it is an obvious idea to anyone who knows how the internet works, I just added it to get the discussion going.
  - Cheer2171 2y ago
    It already exists. IA has had a DWeb for 6 years. Nobody seeds it.
    - seestem 2y ago
      Incentivising seeding is hard. Maybe cryptocurrencies can be useful here, but I understand not everyone likes them especially here on HN. In retrospect the ideal setup would have been if archiving was included into the core HTTP protocol.
      - Cheer2171 2y ago
        [flagged]
mrtksn 2y ago
That would be an awful lot of replication or very shitty archive. Decentralization works when each node can serve all the functions and content alone or when you don't care about completeness. Unless I'm missing something, an archive is not something small or something that's just as good when part of it is missing.
nikisweeting 2y ago
I'm working on this, ArchiveBox v0.8 adds the beginnings of a content addressable store, with plans for bittorrent-backed instance-to-instance sharing in a later version. I think Archive.org should still exist too (and ArchiveBox donates + submits URLs to Archive.org too), but having a self-hosted option where you can archive personal stuff that requires a login, and do P2P sharing with with fine grained permissions is a gap that should be filled. Aiming to archive the entire internet is Archive.org's goal, aiming to archive the part of the internet YOU care about is our goal.
sourcepluck 2y ago
I'm hoping that Autonomi (formerly The Safe Network) is up for the job when (if) it makes it out into the real world one day https://forum.autonomi.community/t/the-internet-archive-a-perfect-partner/40479 https://forum.autonomi.community/t/the-internet-archive-a-pe... [I know that some percentage between 95 and 100 of crypto projects are a scam. I personally believe this one isn't, after much diligent reading. Whether it gets released or does what it claims it will do is another question, but please do spare me the kneejerk anti-crypto reactions, if you can. Just because they're almost all money-making scams, doesn't mean they're all money-making scams.]
thrownaway561 2y ago
that will never happen. no one is going to be able to seed the amount of data that IA has. The only thing they can hope for is that a company like Google or CF provides another data center for them.

dusted 2y ago

I kind of agree, but the way the internet is going, with everyone being behind carrier-grade nat, it's not much of a decentralized network of computers anymore, not to mention all the kids with their laptops and tablets not even hosting anything :(

Kuinox 2y ago
UPnP exists and allow devices to ask the router to open a port to them.
- cosarara 2y ago
  That doesnt help with CGNAT.
- Dalewyn 2y ago
  UPnP is just automating the process of forwarding ports, CGNAT will still screw you sideways because you're behind a router you can't access or order around.
- Fidelix 2y ago
  UPnP is useless with CGNAT (Carrier Grade NAT), which is what the op is talking about. There are other ways to get seeding working, though, including IPV6, which is gaining adoption, so I don't agree with the OP.
- deleted 2y ago
  [deleted]
nikisweeting 2y ago
There are ways around this, I've experimented with setting up a cluster of ArchiveBox instances that share snapshots over Tailscale. Tailscale lets users sign up for free accounts, and you can share machines between separate accounts. A (CGNAT-compatible) decentralized invite-only network could concievably spread that way.

uniqueuid 2y ago

The problem is that it's hard to do this in a way that ensures good archival of ALL resources. Bittorrent works well for popular things but fails for marginal content (unless some really dedicated individuals step in.) What the internet archive provides is a way to have access to many many resources which you didn't know you needed in advance.

deleted 2y ago
[deleted]
TZubiri 2y ago
Lol, in a way the decentralized version is actually the internet.
- vwkd 2y ago
  Or a brain.

maire 2y ago

I don't know if bittorrent has improved - but 20 years ago I had a personal issue with it. At that time our son was using it for games. He goes away to college and came home for the first school break. I get a phone call from our internet provider asking if our son was home. I was so shocked and handed the phone to our son. Apparently at that time bittorrent was optimizing for the most efficient path to a host. Since we had relatively good connection, the mighty weight of the internet was funnelling through our tiny internet provider to our son's computer. The provider (without our knowing it) had made a deal with our son that he would only turn on bittorrent between midnight and 6 AM. I doubt other providers would be so generous. I have been sceptical of bittorrent since that day.

jetrink 2y ago
All clients today (and probably back then) have options to limit bandwidth consumption including throttling, scheduling, and total data transfer caps. For serving mostly HTML and images, dedicating even 10% of a home broadband connection to serving content would allow many, many people per day to access archived pages.

Cheer2171 2y ago

You say this as if it is an original idea. Of course the IA is working on this and have been for over 6 years. There already is a DWeb version. They have been advancing DWeb infrastructure. The IA hosts all kinds of DWeb developer events. But it is over 50 petabytes and the IA gets a huge amount of traffic through the regular web that they need to serve quickly and efficiently to their users. Guess what has happened over 6 years of decentralization of 50 TB? People only seed what they want or care about and there aren't enough seeders to host. They set all this up and nobody volunteers. You're a DWeb advocate and you haven't been seeding. That's a recipe for disaster if they rely on the goodness of volunteer seeders. The IA's mission is broader. DWeb will ever only compliment the IAs mission. https://blog.archive.org/2021/02/18/behind-the-scenes-of-the-decentralized-web-principles/ https://blog.archive.org/2021/02/18/behind-the-scenes-of-the... https://www.bleepingcomputer.com/news/technology/archiveorg-has-created-a-decentralized-or-dweb-version-of-their-site/ https://www.bleepingcomputer.com/news/technology/archiveorg-...

seestem 2y ago
It is not my original idea, but it is an obvious idea to anyone who knows how the internet works, I just added it to get the discussion going.
- Cheer2171 2y ago
  It already exists. IA has had a DWeb for 6 years. Nobody seeds it.
  - seestem 2y ago
    Incentivising seeding is hard. Maybe cryptocurrencies can be useful here, but I understand not everyone likes them especially here on HN. In retrospect the ideal setup would have been if archiving was included into the core HTTP protocol.
    - Cheer2171 2y ago
      [flagged]

mrtksn 2y ago

That would be an awful lot of replication or very shitty archive. Decentralization works when each node can serve all the functions and content alone or when you don't care about completeness. Unless I'm missing something, an archive is not something small or something that's just as good when part of it is missing.

nikisweeting 2y ago

I'm working on this, ArchiveBox v0.8 adds the beginnings of a content addressable store, with plans for bittorrent-backed instance-to-instance sharing in a later version. I think Archive.org should still exist too (and ArchiveBox donates + submits URLs to Archive.org too), but having a self-hosted option where you can archive personal stuff that requires a login, and do P2P sharing with with fine grained permissions is a gap that should be filled. Aiming to archive the entire internet is Archive.org's goal, aiming to archive the part of the internet YOU care about is our goal.

sourcepluck 2y ago

I'm hoping that Autonomi (formerly The Safe Network) is up for the job when (if) it makes it out into the real world one day https://forum.autonomi.community/t/the-internet-archive-a-perfect-partner/40479 https://forum.autonomi.community/t/the-internet-archive-a-pe... [I know that some percentage between 95 and 100 of crypto projects are a scam. I personally believe this one isn't, after much diligent reading. Whether it gets released or does what it claims it will do is another question, but please do spare me the kneejerk anti-crypto reactions, if you can. Just because they're almost all money-making scams, doesn't mean they're all money-making scams.]

thrownaway561 2y ago

that will never happen. no one is going to be able to seed the amount of data that IA has. The only thing they can hope for is that a company like Google or CF provides another data center for them.