7 ms·
Google's SRE Book (2017)
- overbytecode 3y agoCan any SREs tell us how applicable this book is today? Is it still a useful read?
- oooyay 3y agoYeah, it is, but there's also a lot more to being a SRE than this book. This book more or less tells you how to stand up a reliability program, what it doesn't really indicate is what SREs do. A lot of people I meet think SRE is just the new title for "operator" which can't be farther from the truth. Whether you're doing an embedding model, like is referenced in the book, or you have a central org - both are made up of software and systems software engineers that are focused on performance and reliability. They build software, do analysis, and write policy that improve the bottom line reliability of the organization.
- pm90 3y agoNot an SRE, but I think the main contribution from this book was to popularize terminology of operations (eg SLAs) and to give an opinionated perspective on how to handle operations at scale. More practically, I don’t think the book is as useful, as it generally only makes sense when you reach a certain scale that few organizations ever do (imo). However, we are heading into a future where computing will be everywhere and sensors in everything so in maybe a decade even the “smallest” of organizations may be responsible for large scale distributed systems and operating that would require concepts that are provided in the book.
- arter4 3y agoAs a non Googler myself, it still is if you want to know how to set up an SRE team and introduce SRE (ie good sysadmin, for lack of a better word) best practices. The focus on actual indicators such as SLI and SLO, the importance of reducing "toil" (boring repetitive tasks) and automating,... these are all valid concerns. If you want more about system design and how to design reliability, I suggest reading https://google.github.io/building-secure-and-reliable-systems/raw/toc.html https://google.github.io/building-secure-and-reliable-system...
- bananapub 3y agoyes, but not as a checklist of things you have to do, instead it's a valuable discussion of lots of problems and how they were solved in specific circumstances. learn from it, don't copy from it.
- Rapzid 3y agoThe front half is for introducing ideas. The back chapters where never that great IMHO. They get both too in the weeds and at the same time missing actionable advice.
- jupp0r 3y agoRead this, even if you are far away from any actual operation of the systems you work on. Read it especially then. Learning the princples and philosophy conveyed in that book helped me tremendously in my career (as a software engineer). Thanks people at Google for writing and open sourcing it.
- hrpnk 3y agoThe important bit to remember when reading is to understand the origin (why) of the concepts. I've seen Engineers being too dogmatic about the book, saying that "Google does it this way" and not being able to apply the concepts to one's own organizational context. Even at Google, there will be different teams who will deviate from the "described process" given their business context, setup, or stakeholders.
- jeffbee 3y ago> Even at Google, there will be different teams who will deviate from the "described process" given their business context, setup, or stakeholders. Seriously. A lot of the book was influenced by Social SRE who had opinions all out of proportion to their own importance and success. At the time, there was some doubt about whether Social's pet theories belonged in the book, considering the varying practices and beliefs of other SRE groups supporting products that people actually use. This is related to my rule that anybody can title their doc "Best Practices" even if nobody subscribes to them.
- esafak 3y agoCan you name a better book?
- jeffbee 3y agoNone off the top of my head but for the engineering topics in Part III I think it pays to read them as historical background material, then read the last decade of conference papers, talks, and articles. Example: the section on backend subsetting in distributed systems is not current. If you wanted the current Google practice you need to read "Reinventing Backend Subsetting at Google"[1], and there are other interesting publications from other organizations. 1: https://queue.acm.org/detail.cfm?id=3570937 https://queue.acm.org/detail.cfm?id=3570937
- rozenmd 3y agoThere's still good advice in the book, but be aware it was published in 2016, with folks likely having started writing it around 2014. Both Google and SRE/DevOps have advanced greatly since then, and following the book blindly would be cargo culting. Edit: apparently this is a controversial opinion?
- esafak 3y agoWhat contemporary book do you recommend?
- Rapzid 3y agoHow has it advanced greatly since then?
- lrem 3y agoMost of the tools I’ve been using when my colleagues were writing the book are either gone, or half-forgotten abandonware. The new tools were built for different processes, system layout and organisational structures.
- Rapzid 3y agoBook barely talks about tools. It wasn't about tools. The epiphany for many was the concept of an error budget and establishing SLOs. Then, basing investment in reliability on data. That's as applicable today as it was then.
- malwarebytess 3y agoSo, what exactly do you propose as an alternative?
- softirq 3y agoMost companies completely missed the point of SRE/PE/DevOps and keep them on separate teams doing sysadmin toil work and oncall thrown over the wall by engineers who are only concerned with feature deadlines. They regress them back to sysadmin duties and get none of the value of a true SRE program. SRE should always be a subtitle for a SWE and not a separate position, and they should always be embedded with SWEs into one team either building products of infrastructure. The shared ownership and toil reduction only works if you have these two things. All this said, I think the regression is also due to the fact that real SREs are rare. A solid SWE that also has deep systems domain knowledge, understanding how to sift through dashboards and live data, and root cause complex performance problems is a master of many domains and is hard to find.
- snowfield 3y agoThe regression is also due to that a real SRE is expensive. It's cheaper to just get some newly grads to react to alarms following a set runbook of what to do if that alarm triggers. VERY few companies operate at googles scale. For 99.99% of companies it makes sense to investigate single machine issues.
- bananapub 3y agoGoogle SREs also end up investigating single machine issues, fyi.
- jeffbee 3y agoYes, but At Scale® It's a totally different experience when you have the people who technically own the hardware side of the operations taking no responsibility for the well-being of it, and the people who own the software developing elaborate workarounds for bad machines, and the SREs maintaining blacklists of individual nodes.
- dpbriggs 3y agoIn my experience it's fun to do that but only worth it when SLOs are on the line (so a significant number of bad machines).
- amelius 3y agoDoes it have a chapter on how to deal with end-user support?
- dijit 3y agoWhat I found most fascinating is that Google essentially rediscovered what is important in a sysadmin and codified the contract between feature developers and reliability roles. Instead of having feature developers feeling like they have no say in operational requirements, and instead of having reliability staff fighting unstable mess: properly making the contract means everyone gets heard. Contrasted to devops which despite coming out later was in vogue when this book came out; which caused the muddying of the role of sysadmin to meaning either: * a sysadmin practicing agile (the original definition btw) * a software engineer with enough OS skills to carry the pager (the popular one), or * a team consisting of sysadmins and software developers with no barrier between them (10+ deploys per day style). Everyone had their own definition of DevOps. So when SRE came clear: sysadmins are needed, stop trying to push everything into one person, heres how we fix the tension between teams: it was a breath of fresh air. The only revisionist history (that even google seems to forget) is that sysadmins could indeed write code, though it wasnt pretty and didnt have the nice things like mocks and tests. This has changed a little since 2010 at least but it is still dire, even with Cloud making things much easier. *EDIT:* I've gone from +4 to 0 points in a very, very rapid amount of time. If I have offended you; how?
- alexey-salmin 3y ago* SRE are sysadmins with Go * DevOps are sysadmins with Python * Sysadmins are sysadmins with Perl (stolen from twitter, can't find the source)
- jimbokun 3y agoI’m a software engineer that also does operations work. But I really love Go for operations type tasks. Simple, direct, code that’s not too verbose, with predictable performance, and fast compilation to a single executable. I’ve even taken to baking an editor and the compiler into an image with a small library of code for operational tasks, that I can quickly edit and run in the deployed environment for ad how operational tasks.
- oblio 3y ago
- ChrisArchitect 3y agoRelated bunch of discussion from a few months ago: Lessons Learned from Twenty Years of Site Reliability Engineering https://news.ycombinator.com/item?id=38037141 https://news.ycombinator.com/item?id=38037141
- dang 3y agoThanks! Macroexpanded: Lessons Learned from Twenty Years of Site Reliability Engineering - https://news.ycombinator.com/item?id=38037141 https://news.ycombinator.com/item?id=38037141 - Oct 2023 (124 comments) Google Online SRE Books - https://news.ycombinator.com/item?id=31373170 https://news.ycombinator.com/item?id=31373170 - May 2022 (11 comments) What Is ‘Site Reliability Engineering’? - https://news.ycombinator.com/item?id=14153545 https://news.ycombinator.com/item?id=14153545 - April 2017 (86 comments) Site Reliability Engineering - https://news.ycombinator.com/item?id=13503161 https://news.ycombinator.com/item?id=13503161 - Jan 2017 (111 comments) Notes on Google's Site Reliability Engineering Book - https://news.ycombinator.com/item?id=11474002 https://news.ycombinator.com/item?id=11474002 - April 2016 (93 comments)
- poisonta 3y agoI'm curious whether the success of Google in launching software that seems not fully developed can be attributed to their Site Reliability Engineering (SRE) practices.
- bananapub 3y agothis is a dumb comment, but yes, part of the role of SREs was helping people make (and then implement) trade-offs around system deployment while deploying things that basically worked as intended.
- eichin 3y agoAs I understand it (from friends who were SREs in the 2010s) the really clever bit was that projects basically had a budget for "how much SRE attention your deployment needed" - so there was payoff for getting more deployment details right the first time, and structural pushback for just throwing things over the wall. Sounded like an interesting way to connect up the levers...
- deleted 3y ago[deleted]
- poisonta 3y agoIt seems that there may be issues with accountability within their development teams. The reliability of Google Cloud is in question, as encountering 500 errors appears to be a frequent problem. It has been observed that if one persists in retrying a request, it may eventually succeed. This suggests that their teams may have an error budget and might not take action until the issue is flagged by their Site Reliability Engineering (SRE) team.
- dpbriggs 3y agoNot really, the company is massive and until recently very motivated (promo) to launch new things. SREs probably helped get things across the finish line but likely didn't start those projects.
- ChrisArchitect 3y agoIn looking at the Book Updates section (https://sre.google/resources/book-update/ https://sre.google/resources/book-update/) there's a bunch of companion articles and resources but has there been any actual updates to the book since 2017?
- Rapzid 3y agoThe other books.
- yashness 3y agoGoogle is anyway planning to shut down SRE role & transitioning them to SWE role predominantly. A few months back there has been announcement already & one of the reflections is to start with reducing the numbers - https://archive.ph/YWp4O https://archive.ph/YWp4O
- danpalmer 3y agoThis doesn't say SRE is shutting down, it says that they're changing the ratio of SRE to SWE. One thing to realise about Google is that the technology is increasingly unified across the company. 10 years ago everything worked in different ways, but now there are very standard technologies and paths, and naturally this requires fewer SREs to the SWEs developing the products. I don't think this is a bad thing, and in the layoffs SREs have not to my knowledge been hit any harder than SWEs.
- rwiggins 3y agoIt has indeed been a strange time for Google SRE recently. However, they're definitely not planning on shutting down SRE - at least, if you can trust what Google leadership's actual explanation of what that meant. Supposedly, the ratio of SRE to product eng had been growing slowly over the years. The KR to "readjust" that ratio was to bring it back in line with historical norms, i.e., to ensure that SRE continued to scale sub-linearly with SWE/systems. This had (primarily) two facets. First, it gave SRE teams an effectively-blank check to reevaluate their existing dev engagements and jettison the ones that weren't working well. Second, it pushed to eliminate old tools/systems/platforms and converge onto the more modern stuff, like Annealing [1]. Fewer crufty platforms means fewer teams needed to run them, and improvements in those platforms have broad impact. Anecdotally, my own sub-org (within SRE) is growing at the moment. Not by a huge amount, but growing nonetheless. [1]: https://www.usenix.org/publications/loginonline/prodspec-and-annealing-intent-based-actuation-google-production https://www.usenix.org/publications/loginonline/prodspec-and...
- Unfrozen0688 3y agoOfc everyone downsizing... smh
- dpbriggs 3y ago
- yashness 3y agoGoogle is anyway reducing SRE task force & is planning to completely eliminate SRE role. There has been recent announcement already & have already started the move - https://archive.ph/YWp4O https://archive.ph/YWp4O
- hedora 3y agoThe archive link is busted for me, but that sounds like a bad move. 90+% of SWEs are bad at and hate SRE work, and vice versa. The rare ones that can do a mediocre job at both (and that won't burn out and switch jobs if told to do both) are usually not capable of doing an excellent job at either. Using analogies from pretty much any other field shows how dumb it is to combine SRE and SWE, or fuse DevOps (or, god forbid DevSecOps) into one rule: - Would you have a surgeon drive an ambulance? - An expert car mechanic manage fleet scheduling and logistics? - Tell a salesman to design marketing graphics, and have your graphic designer manage high-value customer accounts?
- VirusNewbie 3y agoThat’s not what your link says, did you read what you posted?
- habitue 3y agoThe SRE book is a little more advertisement of Google's internal systems than real actionable advice outside of Google for SREs. There is some generally useful stuff in there, but it probably fits in a few pages vs a full book.
- lrem 3y agoThe second book is actually actionable, according to coauthors.
- kyrra 3y agoBy second book, do you mean the SWE book? https://abseil.io/resources/swe-book https://abseil.io/resources/swe-book It was written by titans with the SWE ladder at Google, fairly disconnected from the SRE book.
- arccy 3y agono, the sre workbook https://sre.google/workbook/table-of-contents/ https://sre.google/workbook/table-of-contents/
- darksim905 3y agoThis seems easier to digest, but it's a long way off from The Practice of System and Network Administration of yesterday.
- lulznews 3y agoIt’s really an incredible marketing piece
- pquki4 3y agoOnly a few paragraphs into the book: > One continual challenge Google faces is hiring SREs: not only does SRE compete for the same candidates as the product development hiring pipeline, but the fact that we set the hiring bar so high in terms of both coding and system engineering skills means that our hiring pool is necessarily small. I was thinking, ok so does this mean the book is completely useless for most companies in the world, since they don't have such standards for hiring people or run DevOps this way? How much of the rest of the book is still applicable?