11 ms·
Site Reliability Engineering
- riteshkpr 10y agothis book is my first intro to SRE. lots of relevant concepts. very well written.
- lgierth 10y agoAdded it to IPFS: https://ipfs.io/ipfs/QmTfeaEwMSKzoA4TFS6G7Qz2p6XZ9pP89VVod42FhFQgih https://ipfs.io/ipfs/QmTfeaEwMSKzoA4TFS6G7Qz2p6XZ9pP89VVod42... Command: wget --mirror --convert-links --no-parent --no-verbose https://landing.google.com/sre/book/ https://landing.google.com/sre/book/
- alinspired 10y agosame, but to the current directory: wget -c -nv -r -nH -np --cut-dirs 2 -k "https://landing.google.com/sre/book/index.html" https://landing.google.com/sre/book/index.html"
- kevan 10y agoLots of great concepts in this book, highly recommended to anyone, not just people going down the SRE track.
- pram 10y agoIt's really a shame that "SRE" came to mean entry level server janitor at so many companies when it strikes me as a very senior role.
- brian-armstrong 10y agoIsn't there always a fundamental tension like this between people who build and people who maintain?
- donut 10y agoThis essay talks about the difference between thinkers, doers and janitors. http://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl/ http://multithreaded.stitchfix.com/blog/2016/03/16/engineers... In it the difference is erased and people are reorganized in a, I think, more productive way.
- jimbokun 10y agoAs a person who builds, I am awfully grateful for the people who maintain, as my stuff would never work without them.
- brian-armstrong 10y agoI think devs should be responsible for maintaining what they build. You get your incentives aligned better that way, and it removes the tension.
- packetslave 10y agoThis is actually covered pretty well (imho) in Chapter 32. Many services start out wholly supported by their developers, and are only onboarded to SRE after they reach a certain level of maturity (and/or criticality to the business) Even after SRE takes over operations of a service, developers remain closely involved, and in many cases have their own pager rotation for "something's really broken, SRE has stopped the user-visible bleeding, but we need a code owner to jump in and help solve the root issue."
- jimbokun 10y agoWell, I haven't read the book yet, but the commentary seems to suggest the SRE's role is more to teach the devs how to make their software maintainable, scalable and reliable, and then verify it really meets all of those traits. As a developer, that is a service I greatly value, and greatly respect those who have that knowledge and experience.
- user5994461 10y agoNever understood why that should be separate people. Actually, dont understand how it could be separate people either.
- jmspring 10y agoI don't know if it has changed, but one thing I enjoyed about living in Finland in the early 2000s was that a janitor and a director at a company were given the same respect. If was more about contributing to the society. Your comment on SRE -> server janitor is interesting. Especially in the land of DevOps. Companies are looking for ways out, to have people do multiple things and not necessarily acknowledge complexity, experience when things go sideways, etc. Being able to get things to run smoothly, having the background to know when X happens and where to look, being able to plan and intelligently navigate future capacity, etc. is incredibly important. The ops roles that used to exist, places like Google keep them, but a lot of companies do not.
- _d8fd 10y agoThe only problem with being a server janitor is there being a communication mismatch. I've worked with a sys admin type who was told, before being hired, he'd be a space janitor. He loved the reference, because that's what he wanted to do. If you need a dog crap collector, sell the job as a poop scooper, not a pre-compost collections technician.
- dannypgh 10y agoSREs with the time and ability (and you don't want to hire those who lack the ability) to do so will automate away whatever operations tasks being a "server janitor" entails. Hence the need of management to ensure they have the time, and that bureaucracy doesn't stand in their way. I once had a "software engineering" gig where I found a way to reduce the labor of the tasks being asked of me by a factor of N (where N equaled their number of clients, which was around 10) for very little work (would've paid for itself with one, possibly two, of the six or so tasks they gave to me my first week). I was told not to do it, because those were client-billable hours when being done per client, and not when theybwere being done to help all clients. I stayed their a very small number of days after hearing that. Point is, there are businesses that value operations work more than engineering, and that they should not.
- deleted 10y ago[deleted]
- general_ai 10y agoAnyone who has ever seen the deployment diagram of Google's ad serving will vouch that Google simply cannot exist without great SREs. If you like both dev ops and software engineering, and have found that your affinity to dev ops makes you a black sheep, I encourage you to apply to an SRE position at Google. I can state unequivocally, that SRE's are held in great regard at Google, and they receive a tremendous amount of respect. This is helped somewhat by the fact that you have to actually earn their support. Until your service is considered maintainable and observable enough to not cause pain, you'll be doing your own DevOps. It's only when you pass the PRR (production readiness review) that you _might_ get _some_ SRE help. Disclosure: I'm a former Google employee
- packetslave 10y agoyep, and if your service gets significantly less reliable over time after SRE takes it on... well, either that will get fixed or you'll be taking the pager back until it does.
- tehlike 10y agoGoogle SREs are pretty good, but there is disconnect between product and sre (somewhat understandably). SREs will always take steps to mitigate issue. rolling back binaries, mlts, gdps, you name it. Sometimes, it's actually easier to fix the problem, but SRE doesn't know it. It's understandable, they manage many many projects, and they can't know all the products. It's just something i found very interesting. Obvious PS: Google employee.
- geofft 10y agoWhat's the right role for someone who wants to deeply know some products enough to fix the code the right way, but doesn't want to be a dev? (Be a dev somewhere where maintenance is as valued as creation?)
- rumcajz 10y agoI think SRE is what you are looking for. They typically know the product pretty well. The reason why they rollback rather than fix the bug immediately is that they want the outage to by fixed in minutes. Even if you spotted the bug immediately you would not have enough time to do the build, let alone run the tests.
- HarrisonFisk 10y agoIf you find this book interesting, a good video which talks about Production Engineering at Facebook and talks a bit about Google SRE: https://www.youtube.com/watch?v=ugkkza3vKbc https://www.youtube.com/watch?v=ugkkza3vKbc
- mmckeen 10y agoI think Production Engineering is especially interesting to me as I think we do a very good job of lowering the wall between ops and dev, mostly we build tools and do evangelism to help SWEs own their own services, but above all we do whatever is necessary to keep the site up in the most scalable way we possibly can. This includes engaging with our product engineers, not just infrastructure, and helping them understand the impact of product on infra and vice versa. That's one thing I especially like about PE, there is very little of the ops/dev divide, it's more of a partnership with PE and SWE both helping to own the service in production. (disclosure, I work at FB as a PE)
- nunez 10y agoI read halfway through the book. I was also on a SRE team at Google. Some of my ex teammates co-authored or contributed to this book. I think that much of what Google is espousing is only applicable to companies like Google, I.e. Technology companies with billions in the bank to spend on extra nine's. The "problem" is much more fundamental. Most businesses still feel that technology is a cost to debit against the business. As long as those in charge feel this way, the issues that necessitate a book like this will continue to persist.
- yeukhon 10y agoI have a different perspective. The fundamental issue with many IT departments out there is the people managing the IT department out there or the people working in the department. The cultural difference between the young and the very traditional IT are in conflicts. Finding the right talents (sigh every time someone said let's look for someone with scripting experience - no...please look for someone with software engineering experience).
- nunez 10y agoI agree that disconnects between management and the "boots on the ground" are part of the problem here.
- bogomipz 10y agoCan I ask what is the level of software engineering vs systems engineering they want or expect for the SRE role?
- cdjones 10y agohttps://www.usenix.org/publications/login/june15/hiring-site-reliability-engineers https://www.usenix.org/publications/login/june15/hiring-site... might be interesting on that point.
- bogomipz 10y agoThanks, this is a good read in and of itself. It's rather vague about the software engineering bar unfortunately. Basically it just says that one of the interviews they do is coding.
- ucaetano 10y agoA cool quote: "And taking the historical view, who, then, looking back, might be the first SRE? We like to think that Margaret Hamilton, working on the Apollo program on loan from MIT, had all of the significant traits of the first SRE."
- fuck_google 10y agoI wish Google would focus more on providing even basic technical support for their services and listening to the users. When there is something wrong with a service, it seems almost impossible to get in touch with technical support staff. Its also very hard to have your voice heard as a customer/user. Ignoring the customers/users is not the way to offer reliable services.
- peatfreak 10y agoCan anybody recommend good books (other than the SRE Book), blogs, mailing lists, IRC channels, articles, videos, etc, that have SRE as the focus and go into it deeply? For example, I'm looking for forums where you can engage in serious discussion about the role, or other books/blogs/articles that aren't simply regurgitating what the SRE Book says.
- peterwwillis 10y agoI hope someone writes (has written?) a book about operations engineering that isn't from the perspective of an "internet company". Google's approach is that of some software devs who were tasked with maintaining infrastructure. Which is markedly different than typical Enterprise-scale engineering, even if Google is bigger than most enterprise orgs.
- gtirloni 10y agohttps://www.amazon.com/Practice-System-Network-Administration-Second/dp/0321492668 https://www.amazon.com/Practice-System-Network-Administratio...
- packetslave 10y agoco-authored by a former Google SRE :)
- peterwwillis 10y agoYeah I skimmed this book, it's basically a For Dummies version of what I would hope to find.
- packetslave 10y agoThere's also the follow-on book, The Practice of Cloud System Administration, which is focused more on distributed systems. https://www.amazon.com/Practice-Cloud-System-Administration-Practices/dp/032194318X https://www.amazon.com/Practice-Cloud-System-Administration-...
- partycoder 10y agoMonitoring is the last line of defense and is by nature reactive, rather than preventive. On the other hand, testing (e.g: unit testing, load testing, etc.) is the preventive counterpart. Both are important and necessary and should not be neglected.
- jpgvm 10y agoMonitoring is also proactive if you are including distributed tracing and metrics. A lot of behaviour in large distributed systems is emergent and synthetic load tests etc often aren't enough to reveal what is going to happen under hundreds of thousands QPS. Metrics and tracing are how you get a handle on this and make fixes before emergent behaviour boils over and causes an outage.
- bostik 10y agoApplication monitoring, beyond "is it live?", based on anything except metrics is IMO flawed. Performance, round-trip times, requests processed per $timeunit, error rate for both the application in question AND all other services it uses, ... - the list is nearly endless. But for every time-series dimension you collect, you really also want their value distributions. Increased error rate or spiking tail latency are the first symptoms of an oncoming problem. Incidentally they tend to go hand in hand, because error handling is by definition outside the happy path and as such often more expensive. On a longer timespan, 30-day, 60-day or even 90-day windows can give very nice insights on peak resource use trends. Spotting trends is important in capacity planning.
- eeZah7Ux 10y agoThere's even more. Good metric analysis give valuable data to drive development. Why did the last 5-lines code change increase GC time by 3%? Why are traffic and memory having correlation of .7 instead of .5 as usual? Why is 10% of the fleet is logging more lines than the others hosts during high network congestion events? Questions like this lead to a much better understanding on how your system work and how to improve them.
- always_learning 10y agoAhh yes finally a book! Now I can learn more about the role to see if I'd like to apply. I've always liked writing code and devops. Perhaps this is a perfect role.
- omegote 10y agoI interviewed for an SRE position and unfortunately, after all the interview process, I wasn't offered a job. However I was impressed by how wide the knowledge of the SRE team is. One would think that a devops just needs a shallow understanding of programming (for example), but the interviews were as varied as they were deep. Too bad I didn't make it.
- murtnowski 10y agoDoes Amazon have an equivalent position as an SRE?
- oncallthrowaway 10y agoYes, but Amazon uses the job title "Software Development Engineer" instead of SRE.
- kevan 10y agoPretty sure you're being facetious, but yes, each team of SDEs is responsible for their own operations. Reliability Engineer positions do exist, but there doesn't seem to be a company-wide standard job title. There is a group (Operational Excellence) that focuses on things you'd expect SREs to focus on, but I think they focus more on building the tools than actual operational support. Source: Am an SDE at Amazon
- sh_tinh_hair 10y agoFirstly: I've administered and designed large CI/CD real world installations for what would be single project scope at Google and those efforts are challenging enough for me. The book was informative as it contains true to life episodes in a huge (and formative) devops environment. But ,in general, there was nothing that I took away from the Google SRE 'way'...except that I have no desire to work in a huge and hugely rigorous devops environment like the one at Google (though I see it's necessity at that scale). Under the guise of being creative and solving unique problems you eventually drill down to the reality of a pseudo-religious approach to building,maintaining and administering rapidly changing large systems. I'd argue that the truly valuable parts of the book for most folks are snippets on the evolution of Google infra, component reuse, design philosophy and lessons learned. These are valuable for any size environment doing any sort of computing.
- packetslave 10y agoThank you, anonymous HN editor, for editing the title of my post from a useful description of the link to a completely meaningless one. Yes, it's now a copy/paste of the <TITLE> tag of the site, but it's now entirely information-free. Good job. You should be proud.
- imcoconut 10y agoThis is great.