8 ms·
Dragonfly: Alibaba P2P file distribution system
- ofrzeta 8y agoJesus, is it so hard to google at least once before naming a project? Just recently that Sonar thing now this. What's next? A Windows editor called Linux?
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- justicezyx 8y agoFWIW, they can just call it Alibaba File Distribution System (AliFDS), which is: - Easier to recoganize - More branding influence (see the g... staff from Google) - Easier to search BTW: A windows edit called linux: not a bad idea!
- Moseman23 8y agoI think Alibaba Integrated file Distribution System is a more accurate representation of this project in acronym form.
- ofrzeta 8y agoTo be fair, I've to admit that Dragonfly BSD doesn't show up on the first result page when you search for "dragonfly". What does show up is the malware by the same name. Side note: this also shows how simplistic Google search really is. No way to search for "Dragonfly /computers/" opposed to "Dragonfly /nature/" with the terms in slashes denoting a concept or domain instead of a syntactic element.
- agentd00nut 8y agoeh? dragonfly +nature dragonfly +computer Though i don't blame you for not knowing, out of 4-5 operator cheat sheets and guides i only see mention of the "-" to exclude terms.
- ofrzeta 8y ago"+" is for synonyms. That might lead in the right direction but it's not the same.
- star-castle 8y agoGoogle manipulates its results in a variety of ways. The end result is that if you and your networked peers aren't already searching for things related to Dragonfly BSD, then you're less likely to find it, because Google will be 'helpfully' biasing your search towards other stuff.
- cat199 8y agohttp://lmgtfy.com/?t=sc&q=site%3A+github.com+dragonfly http://lmgtfy.com/?t=sc&q=site%3A+github.com+dragonfly http://lmgtfy.com/?q=dragonfly+open+source http://lmgtfy.com/?q=dragonfly+open+source
- xellisx 8y agohttps://www.reddit.com/r/javascript/comments/8gsx36/i_used_nodejs_to_create_samba_a_simple_and/ https://www.reddit.com/r/javascript/comments/8gsx36/i_used_n... The developer has since changed the name from Samba to Battlecry....
- ksec 8y agoI used to think that too, but at the rate JS developers are creating package it wouldn't take very long before all decent, easy to remember names to be used up. Surely you don't want Alibaba to name a product in Chinese sounded English which no one could spell or remember right? ( I still cant spell many of the Chinese Companies' English name )
- SEJeff 8y agoAlso a bit confusing as Dragonfly BSD is known for it's custom filesystem, HAMMER: https://www.dragonflybsd.org/hammer/ https://www.dragonflybsd.org/hammer/
- RIMR 8y agoIf you get confused between a filesystem called HAMMER, and a file distribution system called Firefly, I don't know what to tell you. They are different things with different names.
- dpark 8y agoWell, this one is called Dragonfly, not Firefly, so the naming overlap is relevant.
- wccrawford 8y agoDo you also get confused by Dragon Naturally Speaking and Firefox and their similarity to those 2 names?
- cat199 8y agoDo you also realize that calling the browser Firefox was a rename from Firebird due to confusion with an unrelated RDBMS system?
- baybal2 8y agoThe amount of p2p-ness in it is not any much more than that of any DFS of previous decade. The P2P slang though is freaking everywhere here. There are P2P bank and P2P brand sausages.
- justinsaccount 8y agoNot quite, this is not a "distributed file system" this is a "file distribution system". This is more along the lines of https://github.com/lg/murder https://github.com/lg/murder
- the_arun 8y agoIs Dragonfly a server or client? Cause it is compared with wget which is a client. Am I missing something?
- RIMR 8y agoIt's P2P, so I would wager it is both.
- synctext 8y agoIt duplicates most features of libtorrent.org; but requires a server. Therefore, a comparison to wget misses the 17 year old Bittorrent protocol.
- nickpsecurity 8y agoExactly. They should be comparing their P2P protocol to the whatever is currently the best of open-source, P2P libraries.
- ksec 8y agoOk, this is huge. >At Alibaba, the system transfers 2 billion times and distributes 3.4PB of data every month, it has become one of the most important piece of infrastructure at Alibaba. The reliability is up to 99.9999%. I took at look at their repo and it turns out there are surprisingly lots of good stuffs in it which never gets much spotlight or attentions.
- justinsaccount 8y agoIt's a cool project, but I'm not sure how huge that is. If you have 5000 nodes running this, that could be as small as distributing 23.8 gigs/day to each node over the course of the month.
- justicezyx 8y ago> The reliability is up to 99.9999% "up to"... Certainly looks like a wrong wording.
- zild3d 8y agoI read it in the "we have now gotten it up to X" sense, such as "I'm now up to my 5th beer"
- unicornporn 8y agoThat's the way to read it.
- gcbw2 8y agoexcept reliability reduces a total, not sum integers like beer count. if I write a shell script with a scp line right now I can tell you it has 100% reliability. up to X% reliability means that "in the very nice and controlled environment, with the own devs attacking every production problem, trying to get 100% got us up to X%"
- quixoticelixer- 8y agoExcept that is clearly not what they where trying to communicate.
- netvarun 8y agoContainer image distribution seems to be one of the primary problems this tackles: "DevOps ... brings a lot of challenges: the efficiency of image distribution, especially when you have a lot of applications and require image distribution at the same time. Dragonfly works extremely well with both Docker and Pouch, and actually we are compatible with any other container technologies without any modifications of container engine." FWIW, this was a similar problem that I tackled for Golang gopher gala hackathon 2015 - a custom bittorrent based docker image registry POC. https://github.com/netvarun/docket https://github.com/netvarun/docket Interestingly my problem statement was somewhat similar: "Large scale deploys are going to choke your docker registry. Imagine pulling/deploying a 800mb base image (for example the official perl image) across 200 machines in one go. That's 800*200 = 160GB [EDIT: Correction thanks to kingbirdy] of data that's going to be deployed and it'll definitely choke your private docker registry (and will take a while pulling in from the public image)."
- kingbirdy 8y agoIt would be 160GB, not 1.6TB
- netvarun 8y agoThanks, fixed.
- ithkuil 8y agoany thoughts on https://github.com/jvassev/image2ipfs/ https://github.com/jvassev/image2ipfs/ ?
- oscargrouch 8y agoDevs need to be aware that bittorrent now(for sometime) have a DHT solution that allows to have "mutable" slots with the crypto public address tied up to it. So if you own the private key you can write the payload, and just share your public-key address to the people you want to share the payload with. In the payload you can write a traditional immutable torrent manifest for instance, which is in essence a public-key crypto based update system. For a lot of cases i think it's a better approach than what IPFS and DAT provides, because you dont care about it being a global address. All you want is to share with a group of people, more in the p2p social/organic way. I was playing with it once, using the libtorrent library and the main bittorrent DHT, and it was a very nice experience.. it finds the payload pretty fast when you think its a DHT, and you are working in pure p2p fashion. The only single point of failure here is the DHT bootstrap peer. Im planing to use this feature to distribute binary images for clients that have my public key.
- rb2k_ 8y agoFrom: https://github.com/alibaba/Dragonfly/blob/master/src/README.md https://github.com/alibaba/Dragonfly/blob/master/src/README.... - supernode(Java) - dfdaemon(GoLang) - getter(Python) Interesting distribution of languages in what seems to be a somewhat self-contained project. Oddly, https://github.com/alibaba/Dragonfly/tree/master/src https://github.com/alibaba/Dragonfly/tree/master/src only contains the getter and the supernode at the moment
- timdorr 8y agoLooks like dfdaemon is at the root: https://github.com/alibaba/Dragonfly/tree/master/dfdaemon https://github.com/alibaba/Dragonfly/tree/master/dfdaemon
- lowzj 8y agoThere is a plan to use one language(GoLang) to refactor this project. Now the project's directory structure has been restructured to meet the GoLang project style, and the entire CI process has been built. The next step is to migrate the 'getter' & 'supernode' from '/src' to the root and reconstruct them with GoLang.
- betolink 8y agoThis is very similar to what BitTorrent does or am I missing something?
- cabalamat 8y agoHow does this compare to IPFS or BitTorrent?
- noja 8y agoor Syncthing
- pushrax 8y agoQuay has supported Docker image pulls over BitTorrent for a couple years now https://coreos.com/blog/torrent-pulls https://coreos.com/blog/torrent-pulls. Docker themselves have discussed making the official registry extensible enough to support BitTorrent pulls, but I don't know if anything ever happened there. Facebook has been using BitTorrent for deploys for something like 9 years now. They configured the tracker to prefer sharing peers with longer matching subnet prefixes, to keep bandwidth off the backbone as much as possible.
- fizwhiz 8y agoIs there a paper or talk you could link to that describes their system?
- jzelinskie 8y agoJustin linked the blog post[0] which is probably the best written description. The short of it is that when you upload layers to Quay, it stream calculates the BitTorrent pieces. Private layers are given unique swarms isolated by namespace and peer discovery is protected by a tracker[1] that contains middleware validating JWTs passed in the announce URL of the torrent. A custom client[2] can be used to simplify downloading and importing of images into the local docker CAS. Honestly, most organizations don't have sophisticated enough networks that the benefits outweigh the complexity of p2p orchestration. This is why it's popular at Alibaba, Facebook, Twitter, but most people are still just using the OCI distribution protocol[3]. Feel free to contact me (Keybase is in my profile) if interested. I'd love to get more people on the path to p2p, but it's often a solution looking for a problem. [0]: https://coreos.com/blog/torrent-pulls https://coreos.com/blog/torrent-pulls [1]: http://chihaya.io http://chihaya.io [2]: https://github.com/coreos/quayctl https://github.com/coreos/quayctl [3]: https://github.com/opencontainers/distribution-spec https://github.com/opencontainers/distribution-spec
- akavel 8y agoAnyone knows if it does NAT hole punching? I'm interested in such a tool for deploying to remote machines.
- namibj 8y agoIf you only need such a tool, other comments on this submission linked ways to use bittorrent with docker, and µTP [0] seems to be reasonably good at punching through the good old style of NAT, where ports are sequential and on the same IP, with something that can coordinate accessible to both. It also enables gentle use of your bandwidth, in the sense of playing reasonably well even if you don't have fq-codel or similar in use on the router. With somewhat nice networks it can be pretty gentle on other users of the networks, without wasting any part of it. Do consider QoS though, it is preferable to send other, important traffic first, as µTP is good at backing off in these cases. The latency in backing off is just a little high to be stealthy towards concurrent TCP connections. Packet loss is rare, but lag spikes are still a nuisance. [0]: https://en.wikipedia.org/wiki/Micro_Transport_Protocol https://en.wikipedia.org/wiki/Micro_Transport_Protocol
- spullara 8y agoSounds like Twitter's Murder tool from 2010: https://blog.twitter.com/engineering/en_us/a/2010/murder-fast-datacenter-code-deploys-using-bittorrent.html https://blog.twitter.com/engineering/en_us/a/2010/murder-fas...
- pm90 8y agoLooks like its not maintained anymore