5 ms·

Open source plus distributed p2p file sharing is the killer combo. I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as

Open source plus distributed p2p file sharing is the killer combo. I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution. P2P, such as IPFS and recently BitTorrent 2.0 with its hash tree per file, is the only free (as beer and speech) reliable way to host things online forever - at least as long there is the last of the veterans who keeps a copy and seeds it.

Avamander 6y ago
I don't get why machines running the same Linux distribution couldn't actually do the same. BT2 would indeed work very well for this.
- e12e 6y ago
  Nixos/guix might be a good fit for rootfs on ipfs? Comes with a local cache, and you could overlay local changes (eg: /etc/passwd)?
  - takeda 6y ago
    Actually not exactly rootfs on ipfs, but with NixOS is fairly easy to make it wipe data on reboot, here's an example article[1]. The idea is to only keep state we care about and rebuild the system on every boot. [1] https://grahamc.com/blog/erase-your-darlings https://grahamc.com/blog/erase-your-darlings
solarkraft 6y ago
> I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution 2 of them are run by commercial entities. I don't think those are all that interested in reducing dependency on them.
- whyrusleeping 6y ago
  Some links without comment: - https://www.npmjs.com/package/ipfs-npm https://www.npmjs.com/package/ipfs-npm - https://github.com/ipfs-shipyard/apt-on-ipfs https://github.com/ipfs-shipyard/apt-on-ipfs - https://blog.ipfs.io/2020-02-14-improved-bitswap-for-container-distribution/ https://blog.ipfs.io/2020-02-14-improved-bitswap-for-contain...
- akerro 6y ago
  >2 of them are run by commercial entities. I don't think those are all that interested in reducing dependency on them. Docker is also removing some old images and putting limits on existing images, it's too expensive for them to host.
  - m463 6y ago
    or it could be a strategy to move more people to a pay model.
na85 6y ago
In a lot of the world, upload speeds are very limited and ISPs charge an arm and a leg if you exceed your upstream data caps. On top of that, Bit torrent has a tendency to overwhelm routers and make them slow to a crawl. Those are a lot of downsides to ask your users to accept just so you can cheap out on download servers.
- Avamander 6y ago
  With LPD, we could have two PCs upload to each other, it would be even beneficial to ISPs if everyone didn't have to download updates at a similar time from the furthest-away server. Especially in developing countries.
- takeda 6y ago
  Perhaps there are routers that don't handle well large amounts of data, but from my experience what typically is happening are two things: 1. the fairness in TCP is done per TCP connection, bittorrent opens many connections so they take over the whole bandwidth. You can limit number of connections or throttle the bandwidth used by BT (of course you will get slower speeds), maybe there are some other ways. 2. when you maximize your throughput ACKs might get dropped which slows everything down (I suspect that might be the issue you're talking about) again you could throttle or enable traffic shaping and give highest priority to ACKs. This worked well for me.
  - linsomniac 6y ago
    3. Bufferbloat. It's still a thing. dslreports has a bufferbloat score, one of my coworkers has been really impacted by it. http://www.dslreports.com/speedtest http://www.dslreports.com/speedtest
    - takeda 6y ago
      I heard about bufferbloat, but I didn't think it was related to bittorrent maximizing throughput
      - 6y ago
cpuguy83 6y ago
Because changing things out is hard and the benefit of pulling in that much more code and complexity is dubious. It seems like you could implement a p2p solution as a pull through registry... as in host a registry (even per node if you like) that Docker will use as a pull through cache, still speak the same distribution protocol, but have the registry use p2p to get the content.
shp0ngle 6y ago
BitTorrent 2.0 is not recent, no. The current bittorrent company is interested in chasing ICOs, I don’t think they work on new versions of the protocol anymore? edit: yeah it’s from 2008
- capableweb 6y ago
  People are working on BitTorrent yes! libtorrents blog made a blog post quite recently (September 7th, 2020) - https://blog.libtorrent.org/2020/09/bittorrent-v2/ https://blog.libtorrent.org/2020/09/bittorrent-v2/
  - shp0ngle 6y ago
    That is libtorrent, unrelated to bittorrent, no? (organizationally)
    - capableweb 6y ago
      Yeah, maybe you were explicitly talking about the company BitTorrent Inc. "work on new versions of the protocol anymore?" made me think of the BitTorrent protocol in general, sorry for the misunderstanding. I think you're right that the BitTorrent company aren't directly involved in the protocol work, they seem to be focused on trying to push their plagiarized cryptocurrency project.
- leppr 6y ago
  Going on a tangent but this article that came out today is a good read for anyone that didn't follow what happened to BitTorrent the company: https://www.theverge.com/21459906/bittorrent-tron-acquisition-justin-sun-us-china https://www.theverge.com/21459906/bittorrent-tron-acquisitio...

Avamander 6y ago

I don't get why machines running the same Linux distribution couldn't actually do the same. BT2 would indeed work very well for this.

e12e 6y ago
Nixos/guix might be a good fit for rootfs on ipfs? Comes with a local cache, and you could overlay local changes (eg: /etc/passwd)?
- takeda 6y ago
  Actually not exactly rootfs on ipfs, but with NixOS is fairly easy to make it wipe data on reboot, here's an example article[1]. The idea is to only keep state we care about and rebuild the system on every boot. [1] https://grahamc.com/blog/erase-your-darlings https://grahamc.com/blog/erase-your-darlings

solarkraft 6y ago

> I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution 2 of them are run by commercial entities. I don't think those are all that interested in reducing dependency on them.

whyrusleeping 6y ago
Some links without comment: - https://www.npmjs.com/package/ipfs-npm https://www.npmjs.com/package/ipfs-npm - https://github.com/ipfs-shipyard/apt-on-ipfs https://github.com/ipfs-shipyard/apt-on-ipfs - https://blog.ipfs.io/2020-02-14-improved-bitswap-for-container-distribution/ https://blog.ipfs.io/2020-02-14-improved-bitswap-for-contain...
akerro 6y ago
>2 of them are run by commercial entities. I don't think those are all that interested in reducing dependency on them. Docker is also removing some old images and putting limits on existing images, it's too expensive for them to host.
- m463 6y ago
  or it could be a strategy to move more people to a pay model.

na85 6y ago

In a lot of the world, upload speeds are very limited and ISPs charge an arm and a leg if you exceed your upstream data caps. On top of that, Bit torrent has a tendency to overwhelm routers and make them slow to a crawl. Those are a lot of downsides to ask your users to accept just so you can cheap out on download servers.

Avamander 6y ago
With LPD, we could have two PCs upload to each other, it would be even beneficial to ISPs if everyone didn't have to download updates at a similar time from the furthest-away server. Especially in developing countries.
takeda 6y ago
Perhaps there are routers that don't handle well large amounts of data, but from my experience what typically is happening are two things: 1. the fairness in TCP is done per TCP connection, bittorrent opens many connections so they take over the whole bandwidth. You can limit number of connections or throttle the bandwidth used by BT (of course you will get slower speeds), maybe there are some other ways. 2. when you maximize your throughput ACKs might get dropped which slows everything down (I suspect that might be the issue you're talking about) again you could throttle or enable traffic shaping and give highest priority to ACKs. This worked well for me.
- linsomniac 6y ago
  3. Bufferbloat. It's still a thing. dslreports has a bufferbloat score, one of my coworkers has been really impacted by it. http://www.dslreports.com/speedtest http://www.dslreports.com/speedtest
  - takeda 6y ago
    I heard about bufferbloat, but I didn't think it was related to bittorrent maximizing throughput
    - 6y ago

cpuguy83 6y ago

Because changing things out is hard and the benefit of pulling in that much more code and complexity is dubious. It seems like you could implement a p2p solution as a pull through registry... as in host a registry (even per node if you like) that Docker will use as a pull through cache, still speak the same distribution protocol, but have the registry use p2p to get the content.

shp0ngle 6y ago

BitTorrent 2.0 is not recent, no. The current bittorrent company is interested in chasing ICOs, I don’t think they work on new versions of the protocol anymore? edit: yeah it’s from 2008

capableweb 6y ago
People are working on BitTorrent yes! libtorrents blog made a blog post quite recently (September 7th, 2020) - https://blog.libtorrent.org/2020/09/bittorrent-v2/ https://blog.libtorrent.org/2020/09/bittorrent-v2/
- shp0ngle 6y ago
  That is libtorrent, unrelated to bittorrent, no? (organizationally)
  - capableweb 6y ago
    Yeah, maybe you were explicitly talking about the company BitTorrent Inc. "work on new versions of the protocol anymore?" made me think of the BitTorrent protocol in general, sorry for the misunderstanding. I think you're right that the BitTorrent company aren't directly involved in the protocol work, they seem to be focused on trying to push their plagiarized cryptocurrency project.
leppr 6y ago
Going on a tangent but this article that came out today is a good read for anyone that didn't follow what happened to BitTorrent the company: https://www.theverge.com/21459906/bittorrent-tron-acquisition-justin-sun-us-china https://www.theverge.com/21459906/bittorrent-tron-acquisitio...