5 ms·

I’ve said this at other companies I’ve worked at — the datacenter / enterprise software distribution is underserved by P2P. I wish more companies used BitTorre

I’ve said this at other companies I’ve worked at — the datacenter / enterprise software distribution is underserved by P2P. I wish more companies used BitTorrent internally for distribution of tools and software. I feel like homogenous environments like those that benefit from centralized config management like puppet would probably see performance gains with P2P technologies introduced. This looks like a great contribution from Uber engineering and I look forward to playing with it!

jzelinskie 8y ago
This is partially why I built chihaya[0]. The idea is by having a powerful middleware model for traditional BitTorrent software, devs can easily extend the protocol for whatever they need internally. As a sibling comment mentions, an older fork of Chihaya is one of the tools used at FB for distribution in their orchestration system, Tupperware. In practice, internal networks are such a mess that leveraging p2p is often more trouble than it's worth. [0]: https://chihaya.io https://chihaya.io
- dman 8y ago
  Could you elaborate on the issues with local networks?
  - deleted 8y ago
    [deleted]
- steeve 8y ago
  I've used it for a side project and it worked great, thank you for it!
dymk 8y ago
Facebook uses BitTorrent for binary and artifact distribution within their DCs
- kosinus 8y ago
  This is absolutely great if it fits your case, but otherwise maybe not the best time investment. We happen to do event-based stuff at $work, with short running servers. Deploys go to several AWS regions, coordinated from servers in Europe. Bittorrent is really amazing for that sort of thing.
52-6F-62 8y ago
I just had flashbacks to Bearshare on U of T's campus
stingraycharles 8y ago
On this note, we’ve been happily using Resilio (formerly BTSync) here at QuasarDB, as an alternative to Dropbox and the likes. It’s P2P and without a central cloud, and serves us very well.
JMTQp8lwXL 8y ago
Enterprises want to consume software dependencies they can trust. In a peer-to-peer model, anybody can publish anything. The value in having some sort of intermediary (e.g., Artifactory, etc) is they can act as a gateway before vulnerable dependencies make it into your build pipeline.
- ses1984 8y ago
  But torrent has built in integrity checking. You should still check signatures on packages. Your distribution and storage media should be totally independent of the packages you choose to use and how you validate them.
  - JMTQp8lwXL 8y ago
    I think the idea is that enterprises would still prefer to offload that responsibility to a third-party provider. I agree with what you're saying, from a risk management perspective, you shouldn't be trusting that your dev's will be pulling in safe third party code (e.g., from the NPM public registry) instead of forcing them to go through your dependency gateway (e.g., artifactory). Under the hood, a tool like Artifactory could take advantage of this means of distribution, be responsible for signature checking, etc.
    - baroffoos 8y ago
      I didn't think anyone was suggesting to pull random stuff out of a p2p network but instead that you build your docker image and then use p2p to share it to thousands of servers. There are no trust issues here because you are the creator of the data being downloaded.
- tlrobinson 8y ago
  > In a peer-to-peer model, anybody can publish anything. That doesn't have to be the case. You can certainly have private peer-to-peer networks that are authenticated and/or restricted to your internal network. Facebook did this back in 2010: https://torrentfreak.com/facebook-uses-bittorrent-and-they-love-it-100625/ https://torrentfreak.com/facebook-uses-bittorrent-and-they-l...

jzelinskie 8y ago

This is partially why I built chihaya[0]. The idea is by having a powerful middleware model for traditional BitTorrent software, devs can easily extend the protocol for whatever they need internally. As a sibling comment mentions, an older fork of Chihaya is one of the tools used at FB for distribution in their orchestration system, Tupperware. In practice, internal networks are such a mess that leveraging p2p is often more trouble than it's worth. [0]: https://chihaya.io https://chihaya.io

dman 8y ago
Could you elaborate on the issues with local networks?
- deleted 8y ago
  [deleted]
steeve 8y ago
I've used it for a side project and it worked great, thank you for it!

dymk 8y ago

Facebook uses BitTorrent for binary and artifact distribution within their DCs

kosinus 8y ago
This is absolutely great if it fits your case, but otherwise maybe not the best time investment. We happen to do event-based stuff at $work, with short running servers. Deploys go to several AWS regions, coordinated from servers in Europe. Bittorrent is really amazing for that sort of thing.

52-6F-62 8y ago

I just had flashbacks to Bearshare on U of T's campus

stingraycharles 8y ago

On this note, we’ve been happily using Resilio (formerly BTSync) here at QuasarDB, as an alternative to Dropbox and the likes. It’s P2P and without a central cloud, and serves us very well.

JMTQp8lwXL 8y ago

Enterprises want to consume software dependencies they can trust. In a peer-to-peer model, anybody can publish anything. The value in having some sort of intermediary (e.g., Artifactory, etc) is they can act as a gateway before vulnerable dependencies make it into your build pipeline.

ses1984 8y ago
But torrent has built in integrity checking. You should still check signatures on packages. Your distribution and storage media should be totally independent of the packages you choose to use and how you validate them.
- JMTQp8lwXL 8y ago
  I think the idea is that enterprises would still prefer to offload that responsibility to a third-party provider. I agree with what you're saying, from a risk management perspective, you shouldn't be trusting that your dev's will be pulling in safe third party code (e.g., from the NPM public registry) instead of forcing them to go through your dependency gateway (e.g., artifactory). Under the hood, a tool like Artifactory could take advantage of this means of distribution, be responsible for signature checking, etc.
  - baroffoos 8y ago
    I didn't think anyone was suggesting to pull random stuff out of a p2p network but instead that you build your docker image and then use p2p to share it to thousands of servers. There are no trust issues here because you are the creator of the data being downloaded.
tlrobinson 8y ago
> In a peer-to-peer model, anybody can publish anything. That doesn't have to be the case. You can certainly have private peer-to-peer networks that are authenticated and/or restricted to your internal network. Facebook did this back in 2010: https://torrentfreak.com/facebook-uses-bittorrent-and-they-love-it-100625/ https://torrentfreak.com/facebook-uses-bittorrent-and-they-l...