6 ms·
Murder: Fast datacenter code deploys using BitTorrent
- maukdaddy 16y agono NAT/Firewall issues, no ISP traffic shaping, only trusted peers Sounds like an easy-to-exploit datacenter to me.
- winthrowe 16y agoThey'd have those protecting the perimenter, but this is only for deploys in a single datacenter right now. They're only doing it on their network, so they don't have to worry about all the hoops torrents normally go through; they can drop the timeouts to nothing, they don't need encryption, etc. The video states that they will be looking into how best to do deploys across centers in the future, but they are not going to have peers connect across their wan links. One tracker and seeder per location is a first musing. [Edit: paragraphs]
- koenigdavidmj 16y ago>The video states that they will be looking into how best to do deploys across centers in the future, but they are not going to have peers connect across their wan links. One tracker and seeder per location is a first musing. Heck, use a VPN to accomplish this if you're lazy.
- naner 16y agoSometimes I wish geeks wouldn't worry about giving something a cool sounding name and instead just picked something conventional or unremarkable. I don't want to search Google for help with Murder. I don't want to discuss Murder at the office.
- jcapote 16y agoYou guys know a group of crows is called a Murder, right? I'd hate to work somewhere where I could only use stuffy, politically correct software names. Call me crazy, but quirky software names are half the reason I release a project.
- jolan 16y agoIndeed. I called my asset manager 'assman'. :)
- mrduncan 16y agoContext: http://www.youtube.com/watch?v=tujqM2u-BVo http://www.youtube.com/watch?v=tujqM2u-BVo
- mhd 16y agoIt's a neat idea to use collective nouns at a bird-themed company, there's only one problem: crows aren't known for their tweeting. Crows, well, crow. Flight, flock, bevy would be pretty generic, pick a songbird and you'd get something more interesting.
- gxti 16y agoAs long as it doesn't end in -ly or -ing, I'm pretty much happy.
- Jim_Neath 16y agoI remember the uproar when I released a plugin called Shemail.
- alec 16y agoI wonder if they tried multicast. It would be strictly more efficient in terms of bandwidth (one copy per router/hop rather than all the different pieces flying back and forth) and likely less CPU. There's premade software like uftp (http://www.tcnj.edu/~bush/uftp.html http://www.tcnj.edu/~bush/uftp.html) to do this.
- moe 16y agoAmen. The NIH syndrome at twitter seems to be going strong. They just don't stop reinventing stuff, and usually poorly. In this case uftp is the time-tested solution, made precisely for this purpose (mass deployment) and provably better on all metrics (speed, bandwidth, security via SSL, complexity). The mind remains boggled...
- nullstyle 16y agoYou do realize that twitter didn't invent bittorrent, right? ;)
- alec 16y agoThey appear to have invested substantial time into integrating it with their deployment system and improving its performance for their particular network situation rather than using off-the-shelf software like uftp.
- deleted 16y ago[deleted]
- DannoHung 16y agoMaybe this is too easy, but if you're using Git, why not have servers that get a new release be added to a list of servers that can distribute a new release?
- derwiki 16y agoAdded complexity to continue to scale, and one deploy replica high up in the chain that breaks will break everything downstream. As you start building a system to address those issues, you're adding more complexity. This would lead me toward a Murder based solution from the start.
- bonzoesc 16y agoCouldn't they use a tree-like distribution system where each node only receives pulls from a small number of child nodes? You could organize this by giving each server an incrementing number and using math to deduce their parent node, much like an implicit binary tree stored as an array: http://en.wikipedia.org/wiki/Binary_tree#Ahnentafel_list http://en.wikipedia.org/wiki/Binary_tree#Ahnentafel_list That way you can still use the http or git protocol without having to invent a special flavor of BitTorrent.
- bjg 16y agoHe covered why they didn't want to use a tree like distribution method in his presentation. It introduces a possible single point of failure for all nodes which are children of a failed node.
- metamemetics 16y agocss text-shadow on body content is over-rated and actually makes it harder to read.