3 ms·
As you say Google were one of the pioneers of the Chaos Monkey concept; they simply run at a scale where the Chaos Monkey occurs through normal failure rates. F
by Smerity 16y ago
As you say Google were one of the pioneers of the Chaos Monkey concept; they simply run at a scale where the Chaos Monkey occurs through normal failure rates. For sufficiently large MapReduce jobs you can expect one of the compute nodes to fail during the task. If the MapReduce jobs restarted any time this occurred the jobs would never actually complete[1]!
As we're allowed to comment on anything public, I'll focus on a paper about disaster recovery at Google which focuses on the Perforce version control system[2]. As Perforce is centralized and proprietary it even raises a few novel issues. As they can't modify the code themselves it's in fact one of the few vertically scaled pieces of software at Google (Perforce instances run on machines with 256GB of RAM[3]).
Of particular interest to me is the Annual Disaster Recovery Test that Google runs. They assume that the admins/engineers at Mountain View are entirely unavailable and that the fail-overs happen with no advance notice. The idea is that during an actual disaster your staff won't have time to answer queries as to which folder that documentation was in or the order that commands need to be run.
[1] - This is in one of the official Google MapReduce papers, I'll try and hunt it down
[2] - http://www.perforce.com/perforce/conferences/us/2009/Presentations/Wright-Disaster_Recovery-paper.pdf http://www.perforce.com/perforce/conferences/us/2009/Present...
[3] - http://www.perforce.com/perforce/conferences/us/2009/Presentations/Wright-Disaster_Recovery-slides.pdf http://www.perforce.com/perforce/conferences/us/2009/Present...
- robin_reala 16y agoWhy are Google using a centralised and closed piece of software? Does it bring many benefits that haven’t been replicated in open alternatives? Or is it just that the cost of switching is high enough to become prohibitive?
- brown9-2 16y agoPerforce was arguably the best VCS for large groups when Google first started over a decade ago. It's difficult to move all that code and history and the accumulated tools to a new system.
- arethuza 16y agoI suspect it would be pretty difficult to run a company the size of Google without any closed software (e.g. Oracle Hyperion).
- Lewisham 16y agoThe short answer is that when Google started, Perforce was the best kid on the block. Once you get as far down the road as Google, it's hard to change that incumbent, even if culture dictates the use of open-source software. The other problem is that Google just has a whole lot of code. They've got engineers cranking out code all day all over the world. Working at that sort of scale rules out other alternatives (note that Google hired the Subversion guys, and Google isn't using Subversion... this should tell you something).