4 ms·
I'll share my experience, which may differ from other people's. The largest system that I've worked on was Amazon S3. At the time that I worked there we were do
by SpikeGronim 16y ago
I'll share my experience, which may differ from other people's. The largest system that I've worked on was Amazon S3. At the time that I worked there we were doing 100,000+ requests per second (peak), storing 100+ billion objects (aka files), and growing our stored object count by more than double every year. The most important skills for that job were distributed system theory, managing complexity, and operations. I can't explain all of these skills in depth, but I will try to give you enough pointers to learn on your own.
For distributed systems there are two main things to learn from: good papers and good deployed systems. A researcher named Leslie Lamport invented a number of key ideas such as Lamport timestamps and Byzantine failure models. Some other basic ideas include quorums for replicated data storage and the linearizability consistency model. Google has published some good papers about their systems like MapReduce, BigTable, Dapper, and Percolator. Amazon's Dynamo paper was very influential. The Facebook engineering "notes" blog also has good content. Netflix has been blogging about their move to AWS.
Every software engineer needs to manage complexity, but there are some kinds of complexity that only show up in big systems. First, your system's modules wil be running on many different machines. The most important advice I can give is to have your modules separated by very simple APIs. Joshua Bloch has written a great presentation on how to do that. Think about what happens when you do a rolling upgrade of a 1,000 node system. It might take days to complete. All the systems have to interoperate correctly during the upgrade. The fewer, simpler interactions between components the better.
The best advice I know of about operating a big distributed system is this paper[1] by James Hamilton. I won't repeat its contents, but I can tell you that every time that we didn't follow its guidelines we ended up regretting it. The other important thing is to get really good with the Unix command line. You'll need to run ad-hoc commands on many machines, slice and dice log files, etc.
How did I learn these skills? The usual mix of how people learn anything - independent study, school, and building both experimental and production systems.
1. http://www.usenix.org/event/lisa07/tech/full_papers/hamilton/hamilton_html/ http://www.usenix.org/event/lisa07/tech/full_papers/hamilton...
- YuriNiyazov 16y agohi spike!
- SpikeGronim 16y agoHey Yuri long time no see. Facebook me if you're ever in Seattle.
- tptacek 16y agoLogical timestamps are an extremely simple idea that knocked me on my ass when I first worked them into a system. Also a great thing to look up to get a "flavor" of how distributed systems work. I feel like if you walk into a job interview knowing the corner-cases of a two-phase commit and being able to solve a problem using Lamport timestamps, you're probably in the top 90th percentile of dev applicants.
- palish 16y ago90th percentile? Maybe ~99th. The author has been developing software for 20 years. He is likely a fine applicant for a significant number of software dev positions, since he can learn and apply many different technologies very quickly, from what he has said. And also, from what he has said... he doesn't come close to what you described. I've been developing software professionally for five years. I've been programming C/C++ for 10. (I'm 23; my passion has been for gamedev.) And I don't come close to what you described. If I devoted myself to learning what you just described, I could probably achieve a thorough understanding (deep knowledge, an important distinction from superficial knowledge) inside a month. But at the end of that, it seems doubtful I'd be much closer to accomplishing the author's stated goal... I would only know two essentially random cornercases. All of that said, thank you (and SpikeGronim) for mentioning Lamport timestamps; time to go a-wikipedia'n. http://en.wikipedia.org/wiki/Lamport_timestamps http://en.wikipedia.org/wiki/Lamport_timestamps
- SpikeGronim 16y agoMaybe tptacek just has a really great applicant pool ;). You also have to account for specialization. I have no clue how to do optimized gamedev. If you want more Lamport goodies: "Paxos Made Simple" (distributed transactions done right): http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.69.3093&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.69.... "The Byzantine Generals Problem" (harshest failure model known and how to cope with it): http://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html#byz http://research.microsoft.com/en-us/um/people/lamport/pubs/p... An interview where he talks about his approach to systems, particularly formal reasoning and specification: http://www.budiu.info/blog/2007/05/03/an-interview-with-leslie-lamport/ http://www.budiu.info/blog/2007/05/03/an-interview-with-lesl... His publication list - http://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html http://research.microsoft.com/en-us/um/people/lamport/pubs/p...
- Zeu5 16y agoHi Spike, I am asking a question on continuous deployment here. http://news.ycombinator.com/item?id=2288382 http://news.ycombinator.com/item?id=2288382 Was wondering if you can spare some advice for me? Thank you.
- SpikeGronim 16y agoHere you go. http://news.ycombinator.com/item?id=2290675 http://news.ycombinator.com/item?id=2290675
- SpikeGronim 16y agoOne follow up re: how I personally learned distributed systems. This was a great course, and the lecture notes are public: http://www.andrew.cmu.edu/course/15-440-sp11/index/lecture_index.html http://www.andrew.cmu.edu/course/15-440-sp11/index/lecture_i... .
- code_duck 16y ago> The usual mix of how people learn anything - independent study, school, and building both experimental and production systems. Exactly, that along with being in the company of people who know more than you. Studying and experience. One learns things like this incrementally, like anything else, and nothing is better than a good teacher.