17 ms·
New Google SRE book: Building Secure and Reliable Systems
- dijit 7y agoI know it can happen to anyone and that every system will eventually go down no matter how many resources are spent or how smart you are. Heck, it might even be financially prudent to not chase those last 9s of uptime. But r̶e̶l̶e̶a̶s̶i̶n̶g̶ posting this hours after a huge outage that affected most services for over an hour and also less than 12 days after a similar multi-hour outage seems somewhat ironic. EDIT: guess I hurt someone’s feelings.
- DevKoala 7y agoUpvoted since I was going to make the same comment. I have been dealing with some downfall from that issue today. As a user of their cloud services, my perception of their reliability is pretty low compared to competitors. I still like GCP the best though.
- giovannibonetti 7y ago> As a user of their cloud services, my perception of their reliability is pretty low compared to competitors. I still like GCP the best though. I guess we tend to notice more the flaws of the services we use the most
- DevKoala 7y agoNot in this case. At work, we use AWS and GCP, everything that runs on top of Kubernetes is deployed on both clouds. If I isolate the number of service stopping incidents this year for that vertical, I can find 3 on GCP's side, and zero on AWS.
- dijit 7y agoAs a user of GCP I agree. And anyway, bad things happen, a few bad things if handled honestly and transparently do not affect my trust.
- mangatmodi 7y agoI guess people didn't like that - "it might even be financially prudent to not chase those last 9s of uptime." It just promotes bad mindset
- namdnay 7y agoMaybe they were too busy writing the book :)
- dan_quixote 7y ago> it might even be financially prudent to not chase those last 9s of uptime That's the basic premise of an "error budget".
- deleted 7y ago[deleted]
- jonathanoliver 7y agoI'm wondering if this has floated to the top of HN because of the recent GCP outages (both a few days ago and from this morning). I'm trying to figure out if this is coincidental or ironic. NOTE: As a heavy user of GCP we we're affected by the three most recent outages (GCIC20005, GCIC20004, GCIC20003), but I definitely feel for those that were impacted.
- danso 7y agoCoincidence and irony aren't mutually exclusive. In any case, it's not surprising that free books about SRE have been noticed by HN users during a time of widescale quarantine.
- jonathanoliver 7y agoI'm surprised by the down votes. I still find it intriguing that during an outage the SRE handbook floats to the top.
- rossjudson 7y agoThe outages of sufficiently large and complex systems always have a kind of "are you frickin' kidding me?" aspect to them. It comes from crazy things nobody ever thought of, or the probabilities of single-throwing a dart through three separate keyholes in three different doors, or the unintended consequences of seemingly unrelated actions.
- brightball 7y agoIs there a more digestible version of SRE concepts somewhere? I'm just looking for an easier way to communicate core principles and concepts to my team without asking them to sink into 500 pages?
- dijit 7y agoThe books read quite easy. The first book is just stories from google; it doesn’t really prescribe anything- it’s a collection of people talking about what SRE means to them and also how it fits together with “devops”. The second book (the SRE workbook) is more prescriptive, walks through practical ways of implementing it. The most base description of SRE principles is simply that: 1) You automate aggressively and develop or use self-service tools as much as possible (over ops work) 2) you define what “availability” really means; institute an allowance of errors based on budget. Highly reliable systems should get much more attention and budget than lower requirement systems. Make an SLO dashboard; alert based on your “error budget” being eaten too quickly. 3) try to avoid allowing your staff to work more than 50% on operations work; that’s your indicator for being overloaded.
- lrem 7y agoAnecdata: some people disbelieve this ≤50% ops time figure. At the moment my SRE team is getting some flak for exceeding 30%, which exceeds the PA's guideline of 20% for the wrong reasons.
- gentleman11 7y agoIf somebody was in a hurry, which sections of which book should they start with?
- atmosx 7y agoCheck out the YouTube videos on SRE by the Seth Vargo and friends at Google.
- peterwwillis 7y agoSRE Book (https://landing.google.com/sre/sre-book/toc/index.html https://landing.google.com/sre/sre-book/toc/index.html): Chapters 4, 5, 6, 17, 28, 29, 30, 31, 33, All the Appendixes SRE Workbook (https://landing.google.com/sre/workbook/toc/ https://landing.google.com/sre/workbook/toc/): Chapters 1, 2, 5, 6, 8, 16, 17, 19, 20, 21, All the Appendixes
- kerng 7y agoA little ironic, but aren't parts of GCP down right now? https://www.theguardian.com/technology/2020/apr/08/google-outage-hits-gmail-snapchat-and-nest https://www.theguardian.com/technology/2020/apr/08/google-ou... EDIT: looks like this comment didn't resonate well with some readers.
- jldugger 7y agoYou might argue that the Google SRE books are part of a recruiting strategy, and the GCP service creates a massive need to recruit more SRE.
- deleted 7y ago[deleted]
- neonate 7y agoThe pdf of the book is https://static.googleusercontent.com/media/landing.google.com/en//sre/static/pdf/SRS.pdf https://static.googleusercontent.com/media/landing.google.co...
- xyst 7y agoWhat does a lizard have to do with SRE?
- sethvargo 7y agoGreat question. As an O'Reilly author myself, I can tell you that we have no control over the animal selected. There's a fun animal selection process, but the publisher's decide. Disclaimer - I work for Google and worked on this book.
- pkaye 7y agoJust curious deals like this work. Does Google pay O'Reilly for the editing and typography or does O'Reilly make money by selling the book themselves.
- dielectrikboog 7y agoHey, at least you didn’t get Cthulhu on the cover, like Andrew Lombardi’s WebSocket.
- sethvargo 7y agoHey everyone - Seth from Google here. Thank you for all the positive comments about the book. I'll be around to answer any questions you might have. As noted, the book can be downloaded for free in digital formats. PDF: https://landing.google.com/sre/static/pdf/SRS.pdf https://landing.google.com/sre/static/pdf/SRS.pdf EPUB: https://landing.google.com/sre/static/pdf/srs-epub.epub https://landing.google.com/sre/static/pdf/srs-epub.epub MOBI: https://landing.google.com/sre/static/pdf/srs-mobi.mobi https://landing.google.com/sre/static/pdf/srs-mobi.mobi
- athenot 7y agoNo questions, just THANK YOU for this effort and for making it public. We use it here to help expand people's minds, shifting their thinking from just writing applications to designing for large-scale HA systems, with all the fun pitfalls that lurk in a cloud.
- ignoramous 7y agoI really liked that there were HTML versions of the previous two books. Any chance that'll be up for this one? A bit far-fetched but: Have you (or anyone else at Google) looked at Amazon Builder's Library [0] and/or various re:Invent / re:Inforce talks from 2018/19 [1][2] that focus on similar topics as in this book and other SRE books? If so, what are some ideas (infrastructure, blast radius, incident management, resilience, recovery, deployment strategies, crisis management, disaster planning, aftermath etc) you folks think that contrast / complement Google's approach to building hyperscale systems? Thanks. [0] https://news.ycombinator.com/item?id=21714209 https://news.ycombinator.com/item?id=21714209 [1] https://news.ycombinator.com/item?id=22347694 https://news.ycombinator.com/item?id=22347694 [2] https://news.ycombinator.com/item?id=19291163 https://news.ycombinator.com/item?id=19291163
- sethvargo 7y agoI'll flag the HTML pages with the team. I'm honestly not sure. I know we're able to offer epub and mobi this time, something not previously possible with the other books. I haven't looked at those other resources, but I'll ask if others have.
- armitron 7y agoGenuinely curious: What secure and reliable systems has Google built? Nothing really springs to mind. Android, their most popular end-user product, is a security disaster [1]. Chrome, "the most secure browser in the world", has a huge list of serious vulnerabilities [2]. [1] https://www.cl.cam.ac.uk/~drt24/papers/spsm-scoring.pdf https://www.cl.cam.ac.uk/~drt24/papers/spsm-scoring.pdf [2] https://www.cvedetails.com/vulnerability-list/vendor_id-1224/product_id-15031/opec-1/Google-Chrome.html https://www.cvedetails.com/vulnerability-list/vendor_id-1224...
- danso 7y agoYou believe that Google Accounts, GMail, and GDrive are fundamentally insecure and unreliable? Compared to what?
- armitron 7y agoYou mean the same Google services running on top of data centers that the NSA had infiltrated and was monitoring for years?
- tekno45 7y agoif it's so secure that only state sponsored actors can force their way in, thats pretty damn secure.
- danso 7y agoYou mean the same program that also "infiltrated" (I guess you take those companies at their word that they weren't cooperating) every other tech giant? Again, that's why I asked "Compared to what?" What is your favorite tech company, uninfiltrated by the NSA while also serving billions of users, whose SRE books would be more worthwhile?
- armitron 7y agoAssuming that an answer to your question is even quantifiable (it isn't since you're asking me to prove nonexistence), how is it relevant to my original question? The security failures that allowed the NSA to come in were comical. Deliberate choices that Google made are for the most part responsible for the Android fiasco. In fact, Google is one of the behemoths that put us all at -ever increasing- risk in the name of profit and so far have done precious little to reverse course [1]. [1] https://seclists.org/dailydave/2020/q2/1 https://seclists.org/dailydave/2020/q2/1
- danso 7y agoOff-topic: It made me chuckle to see this well-designed page, with great+free content, and also pulling in angular.js, doesn't follow Google's recommended practices for SERP, e.g. meta tags so that pasting the link into Slack, etc. displays some info rather than just the bare URL
- Scottopherson 7y agoIs there a Slack setting for this? I personally prefer bare URLs and always have to manually edit my messages to remove the URL info snippets.
- spurgu 7y agoYeah as long as you're admin: https://slack.com/intl/en-fi/help/articles/360001502048-Manage-link-previews-for-your-workspace https://slack.com/intl/en-fi/help/articles/360001502048-Mana...
- muro 7y agoLooking at the authors, this book would be more about security, rather than reliability.
- UncleMeat 7y agoIt is. You can just take a look at the table of contents.
- sethvargo 7y agoThis is correct. This is the third book in our SRE series. It's reliability through the lens of security. Disclaimer: I work for Google
- gentleman11 7y agoWhat is the focus of the others?
- cameronbrown 7y agoMuch more general. The first book is a solid overview of how Google does SRE and outlining each of the various concepts (error budgets, blameless culture, etc..). The second is more of a practical guide on deploying SRE into an organisation, a lessons learned type of book. (I work for Google but not SRE, just enjoyed reading the books)
- pleddy 7y agoWhy are 51% of your people contractors?
- l9i 7y agoWe did our best to highlight the intersection of security and reliability. I admit there might be more emphasis on security in some chapters, especially those that overlap with the previous two SRE books (that were exclusively about reliability). We wanted to avoid repeating ourselves. (disclaimer: I work at Google)
- containrh4x0r 7y ago"use containers" - no no no - containers and kubernetes are horribly insecure and in a multi-tenant situation not even an option great marketing google!
- colinrand 7y agoThe most challenging part of this series is not in the material itself. These insights are learnings through hard experience and scale from Google are invaluable. No, the hard thing for everyone is to recognize is that most companies are not Google and don't have Google's problems, resources, or time to follow these practices. Definitely read the material, I will thoroughly, but don't apply this blindly. Solve YOUR problems, not theirs.
- l9i 7y agoWe were very much aware that not all companies can afford to staff a dedicated security team. We tried to do our best to make sure that the book is applicable to a wider audience: from startups, to big corporations. (disclaimer: I work at Google)
- pathseeker 7y agoIt's not as applicable to startups as you would think. The real calculation startups are making all of the time that this book doesn't mention is "is it worth making this particular piece scale/secure/robust before we run out of money?" While it's technically true that the advice would apply to startups in the sense that it would improve their reliability, the elephant in the room is that it doesn't matter. The engineering skill at a startup is understanding what's actually critical, and this book doesn't speak to that.
- delroth 7y agoHow do you measure "is it worth" without a good idea of the risks and costs involved in the decision making? Just because it doesn't directly answer your question doesn't mean it's not applicable. In fact, I'd argue the risk factor is significantly different across startups, so exploring the tradeoffs is the only way to approach the problem generically. (Disclaimer: I work at Google and was involved with some aspects of the new book.)
- pdeuchler 7y agoThe "is it doing X before we run out of money?" question is way overblown in startup land, usually by product people to skew developer time towards more features instead of much needed foundational work. In reality, this question is almost always instantly answerable. You're either still building out your MVP and desperately need customers to validate your idea, in which case the answer is "No", or you're an established startup with runway and a growing customer base, in which case the answer is "Yes".
- nerpderp82 7y agoThe book has no value unless the goals and the lessons are internalized.
- Veserv 7y agoI scanned the introduction, but failed to see any concrete information on the expertise of the authors on security. Can anybody speak to the expertise of the authors on security? In particular, I am interested in specific projects or initiatives they directed or lead. The state of systems before and after these projects. If there were any long-term regressions after their involvement. To be even more concrete if possible: 1. What was the project and what would occur in the event of unmitigated compromise? 2. What was the threat model? 2a. Why was that the appropriate threat model given the possible outcomes? 3. How did they validate that the project met its goals in mitigating the threats in the threat model? 4. What level of resources would be necessary to compromise the systems they were trying to protect? 4a. Would the system prevent compromise by a red team with a $1 Billion, $1 Million, $1000, $1 budget? 4b. What resources did the red teams have? Personal questions for the responder: 1. Would you feel comfortable using the processes you have used in the past to develop a system where compromise would result in the loss of human life? 2. If you answered yes, what project and process and why do you believe that it sufficient? 3. If you answered no, do you have any first hand knowledge of systems that achieve that standard? 4. What is the best system that you have first hand knowledge of that has achieved at least that standard? Is there a non-theoretical gold standard?
- sethvargo 7y ago> Can anybody speak to the expertise of the authors on security? I think a cursory LinkedIn or social media search for any of the title authors or chapter authors will demonstrate their credentials. There were many people involved in this book, all of whom carry the necessary credentials and experience. > Personal questions for the responder: Guidelines help us scale, but at the end of the day, some services are unique and require additional review. I recommend reading the bits on threat modeling for more information.
- Veserv 7y agoGeneric credentials and experience provide very little information to me on their expertise. The CSO of JP Morgan, James Cummings, was highly experienced and credentialed when JP Morgan was breached in 2014 in one of the largest data breaches in history. The CSO of Equifax, Susan Mauldin, was highly experienced when Equifax was breached. The head of security for Windows at Microsoft is probably highly credentialed and experienced, but we all make fun of the insecurity of Windows. This is why I am interested in the specific projects they worked on and how they stack up. It is much harder to game the system if there is concrete, auditable evidence backing their expertise. Yes, guidelines are not the end-all-be-all and you can never be sure, but when a civil engineer approves a bridge, they assert that they are confident that human lives can be trusted to the bridge (in certain configurations). They can do this with reasonable confidence because they have seen systems that have stood the test of time that prove out the techniques that they are applying. That is what I am interested in, do you/they have that level of confidence? What justifies that confidence? What systems prove out the techniques that were used? Did any techniques they invent stand the test of time (this provides evidence they can invent new techniques)?
- containrh4x0r 7y agoanother gem "ptrace sandboxing" ala gvisor which is horribly slow and shouldn't be used for production systems https://news.ycombinator.com/item?id=19924036 https://news.ycombinator.com/item?id=19924036 seriously - what is going on at google nowadays? has it always been like this?
- rb808 7y agoAny tips for the vast majority of SRE groups where people are paid a fraction of google employees and never given any time to fix things?
- superflit 7y agoDouble the caffeine intake and "RUN FOREST" :) I feel your pain.
- BrandonY 7y agoTech debt is most easily measurable in repetitive operational work. Measure it. Bring a story to your leaders: "We are spending 60 hours per week doing repetitive task X. With 200 hours of work, we could eliminate this work. It would pay for itself in a month." If your leadership declines to take you up on this, escalate. If that fails, you must choose between continuing to do the repetitive operational work as instructed or leaving.
- Aeolun 7y agoFix shit anyway. You are never going to get time to fix your operational procedures, so just be working on project A or B, but really consider fixing your procedures part of those projects.
- johngalt 7y agoYou would be surprised how far you can get with consistent pressure and time. Small iterative changes. It works for practices/culture as well as technology. There is never some big meeting where everyone decides that 'operations are going to change! From now on we are going to X!" Well sometimes there are such meetings, but often means merely adding another demand. Like a new years resolution, it is a strongly voiced command for better results, with no real changes behind it. Real improvement is about adding practices and habits. Starting with the most needed/highest payoff items and gradually building on them.
- lonelappde 7y agoHigh pay is great, but doesn't make better software. Do Googlers get time to fix things?
- sosilkj 7y agoWhat would be the equivalent trio of books for the SWE?
- UncleMeat 7y agoI know some of the content contributors for the book and some of my work is discussed it it’s text. I’m a SWE and a huge amount of the material in the book is directly relevant to development.
- aoeuid 7y agoGoogle recently published the SWE Book: https://www.amazon.com/dp/1492082791 https://www.amazon.com/dp/1492082791
- laddng 7y agoHave you read it? Just curious if you found it to be good because the only review for it wasn't very hopeful.
- cloakedarbiter 7y agoI'm looking for reviews as well. FWIW, there's a list of table of contents available here: https://www.oreilly.com/library/view/software-engineering-at/9781492082781/ https://www.oreilly.com/library/view/software-engineering-at...
- brlebtag 7y agoNOOOOOOOOOOO!!!!!!!! I just bought the first one. Just kidding. In my 'to buy' queue.
- lainga 7y agoPiotr and Anthony: any relations?
- l9i 7y agoLe*v*andowski != Le*w*andowski Seriously though, there is no relation that I am aware of. It's a very common surname in Poland (source: https://en.wikipedia.org/wiki/Lewandowski https://en.wikipedia.org/wiki/Lewandowski). (I'm Piotr Lewandowski.)
- deleted 7y ago[deleted]
- zimmertr 7y agoHi, I want a physical copy. I went to see how much they were to ask HR to buy one for me and found it was $52 on Amazon! What's up with that?
- TheGeminon 7y agoThat seems pretty reasonable to me for something like this, I just purchased it myself for $90 CAD, and generally I see these types books around the $100 CAD mark.
- stevofolife 7y agoJust looked through the ToC. There's quite a bunch of topics being covered here. Are there any recommended sections to focus on for someone with limited time that comes from a software engineering background? Thanks!
- anaoprea 7y agoThe Introduction chapters, then chapters from the Design and Implementation parts (depending on what your current focus is, one section might be more relevant than the other). Copy/paste from the preface: "We recommend you start with Chapters 1 and 2, and then read the chapters that most interest you. Most chapters begin with a boxed preface or executive summary that outlines the following: • The problem statement • When in the software development lifecycle you should apply these principles and practices • The intersections of and/or tradeoffs between reliability and security to consider Within each chapter, topics are generally ordered from the most fundamental to the most sophisticated. We also call out deep dives and specialized subjects with an alligator icon." (Book author here)
- billfruit 7y agoThe title is very generic though, not indicating what type of systems. Like if I am working in embedded systems, should I read the book? I skimmed through a few pages, still no idea..
- anaoprea 7y agoWe define systems in the Preface: "In this book we talk generally about systems, which is a conceptual way of thinking about the groups of components that cooperate to perform some function. In our context of systems engineering, these components typically include pieces of software running on the processors of various computers. They may also include the hardware itself, as well as the processes by which people design, implement, and maintain the systems. Reasoning about the behavior of systems can be difficult, since they’re prone to complex emergent behaviors." (Book author here)
- tinco 7y agoIt's written by Google SRE's.
- l9i 7y agoMost of the design principles covered by the first part of the book are quite generic and can be applied to a broad range of systems. If you are working on embedded systems, the Design for Recovery chapter might be a good start since it contains examples related to constrained enviornments. (disclaimer: I worked on the book)
- DeathArrow 7y ago> In our experience, when you use a hardened data library such as TrustedSqlString (see “SQL Injection Vulnerabilities: TrustedSqlString” on page 252) That is not my experience. Yes, the most simple SQL injection a newbie attacker would try, is running a query directly on your database using stuff like" ' OR '1' =='1' " However, one can do a lot of other things like getting the schema, table names and the actual data in the tables by observing the answers and timing. When I did my master's degree, on the course about database security the teacher said there isn't any mean to 100% prevent SQL injection. There are other means to protect data, like not using a single app user to access the database, use security rules at database level together with security rules at app level. One clever trick is to return fake data if you detect a smart ass is trying to access data he shouldn't, rather then tell him he is forbidden. Let him enjoy his fake data. :)
- onion2k 7y agoOne clever trick is to return fake data if you detect a smart ass is trying to access data he shouldn't, rather then tell him he is forbidden. Let him enjoy his fake data. :) That sounds like a lot of effort for something that should never happen if your real security systems are working, and a huge problem if something breaks and returns fake data to real users. It would look like their accounts have been compromised which is far worse for the business than any amount of enjoyment you might get messing with an attacker. Honeypots are useful in some very specific situations, but you need to be really careful where and how you implement them. Generally, leave them to the network security team. In my experience anything you do that tries to be 'clever' is a bad idea. Implement the simplest possible solution that solves the problem, otherwise it's going to blow up in your face one day.
- tziki 7y agoIt's also a big problem if the hack gained any publicity. Who's actually going to believe "no no, it was actually fake data that got stolen"?
- DeathArrow 7y ago
- LeonM 7y agoThe HN account of Ana Oprea (anaoprea), one of the authors of the book seems to be blocked. All comments below are marked dead. Probably because it is a new account with low karma rating or such. Can any mod here restore the comments?
- gwern 7y agoSeems visible now.
- jxub 7y agoIn a somewhat cynic remark (and having nothing against the authors), I cannot think of a better timing to assuage someone about Google's competency after the numerous outgages in GCP these weeks. The "Compliments of Google Cloud" sticker on the cover makes sure to reinforce the association.
- nomanlagharipak 7y agoplease visit below link https://www.yoururdupoetry.com/2020/04/poetry-for-love-in-urdu-poetry-urdu.html https://www.yoururdupoetry.com/2020/04/poetry-for-love-in-ur...
- MaxBarraclough 7y agoLooks like a valuable resource. A bit surprised to see that in Building Secure and Reliable Systems, as far as I can tell the word reliable isn't given a precise definition, even when it's contrasted with security. The preface opens with "Can a system ever truly be considered reliable if it isn’t fundamentally secure?" but the terms don't seem to be clarified anywhere. It appears not to mean the same thing as service availability, given the section on the CIA triad. Am I missing something terribly obvious?
- bumblebritches5 7y agostep 1: Use seL4
- deleted 7y ago[deleted]
- pleddy 7y agoLet me guess. Now trying to destroy careers of security folks and replace with bad practices from the mouths of managers at Google. Wow! Gee. I'll buy a paper copy and burn it. 99% of people here are not in Google's use case, so this info can not apply. Brag brag. If you didn't learn the lesson from last book, enjoy. I'm just amazed by Google's ability to run huge kubernetes clusters all on windows, with zero networking or Linux skills, I'm impressed.
- forlorn 7y agoBy the way the story in chapter one about a smard-card was very amusing to read!
- LordCres 7y agoOk
- classified 7y agoFascinating. No matter how badly Google fucks up, some people still need to worship them and lick their boots. Somebody should do a study about this phenomenon.