11 ms·
Simplicity – Google SRE Handbook (2017)
- OutOfHere 2y agoDoes this include instructions on accidentally deleting a customer's account? Because that's what Google does. I don't think I want to take any advice from Google on anything.
- postepowanieadm 2y agoIt's from 2016 when google was less trash.
- infinityplus1 2y agoCloud computers are just someone's else computer. Amazon and Microsoft engineers can make the same mistake too. Take backups and test them regularly and you'll be OK.
- dieortin 2y agoI challenge you to find an organization that has never made a mistake. Truth is the uptime and reliability of Google services is very good, while operating at huge scale. And I have no association with Google whatsoever.
- gtirloni 2y ago> Because that's what Google does Your argument would be stronger if you could list a few cases like that latest high profile one where GCP deleted some enterprise customer's account. A single one won't cut it for "that's what Google does".
- OutOfHere 2y agoWith Google, the deletions almost always are intentional, not accidental, and this is a huge problem with it. Google (not GCP) deleted ten years of my data without warning or notification or remorse or recourse even though I was doing nothing illegal. Amazon would never do something like it. To Google, once a customer or service becomes just 1% inconvenient, it's time to get rid of the customer or service. It's a very valid concern.
- elktown 2y agoJust remember that what google writes in these kind of things is not universal. It's written from their very unusual circumstances. You can certainly pick nuggets that are more universal than others but, like in many other instances, too much unnecessary work is spent trying to imitate Google and others when it's not really needed. And no, you won't turn into Google over night, you will have time to adapt if fortune hits you. Some things are not even necessarily good advice at all, but rather a product of incentives within Google (and perhaps most tech corps) rewarding the aesthetics of "innovation".
- fmbb 2y agoI read the whole text (granted, a bit quickly) looking for weird or unnecessary advice but I cannot see any. This is a great text about considerations everyone operating software services should take to heart. It applies regardless of if you deploy a monolith or several smaller servers. If you are only one developer, it might apply in a smaller context.
- elktown 2y ago"If you are only one developer" suggests zero interests in being nuanced. To be clear, this linked specifically to simplicity which I'm certainly in favor of emphasizing the importance of. But IME the exact opposite happens when people try to imitate Google overall in a smaller setting, where instead too much resources are spent on meta-issues instead of the product being developed.
- zbentley 2y agoI think you’re arguing with someone who isn’t here. Nobody is endorsing the practices in TFA “because it’s Google”/in order to be like Google. Sure, people elsewhere make those claims all the time, and they’re wrong, but that’s not in evidence here that I can see. The article does seem to come pretty close to universally applicable good ideas. Not because of where its author works, but because of the content.
- 2y ago
- userbinator 2y agoA lot of preaching but bears little resemblance to what Google is actually doing in reality. IMHO those who actually understand what "simplicity" means in software are only those who have tried to do anything in highly-resource-constrained environments.
- hiAndrewQuinn 2y agoA taxonomy of what we mean when we talk about "resource-constrained" might be helpful for those seeking to gain this knowledge. Limited CPU, RAM, etc are the obvious contenders - but then there's also "resource-constrained" as in "I'm the solo dev of this project and have 5 hours in a good week to work on it", or "this runs in a weird place without Internet that I only get access to twice a year". I've been in all of these situations, sometimes multiple at the same time, and they've been great forcing functions to find new paths towards simplicity.
- gtirloni 2y agoYou also have to keep in mind the scope and timeline of where these principles apply. I'm sure someone would be able to apply them to their own work most of the time but if you look at a company as a whole, unless someone at the top is really pushing for global simplicity, things are pretty messy most of the time. I'm just saying this because Google might be doing this in little islands, not as a company strategy. I don't really know and can only guess from the outside.
- sumeno 2y ago> bears little resemblance to what Google is actually doing in reality What is Google doing in reality?
- quintes 2y agoYeah look. This may be the throw it over the wall problem. sRE says No. You build it you run it but may work at their scale
- burakemir 2y agoWhile the text touches on many points I would immediately sign, the paragraph starting with "Because engineers are human beings who often form an emotional attachment to their creations, ..." is really out of place. The cause of complexity is not emotional attachment, these are decisions being made. The decision to add feature after feature and punt on maintenance for example is something that has little to do with emotions. There is a lot of agency that engineers, SWE and SRE alike have in shaping how things are. However there can be good reasons to abandon simplicity. The real trouble here is not psychology but that as a profession we are really bad at measuring and estimating the effective cost of maintenance. Part of that is considering measures to improve simplicity and maintainability as cost that comes without gain and somehow less important than features, and then just accept giant rewrite a few years later. A continuous portion of upkeep would likely be more economical and real engineering has always included an aspect of economy - cost vs benefit. IMHO the loaded accusation of emotional attachment might be rooted in an "us vs them" attitude (SRE vs software engineering) that should have no place in a sober discussion on the value of simplicity and it diminishes an otherwise great text.
- CraigJPerry 2y ago>> Because engineers are human beings who often form an emotional attachment to their creations, confrontations over large-scale purges of the source tree are not uncommon. Some might protest, "What if we need that code later?" > the paragraph starting with "Because engineers are human beings who often form an emotional attachment to their creations, ..." is really out of place. FWIW I’ve definitely encountered developers clinging to things when the business context has completely changed. I totally recognise the scenario in the original text.
- burakemir 2y agoSure, but if we argue that these values and principles should be applicable, then it should also be possible to make an argument why and not blame the irrationality on emotions. It seems more likely that bounded rationality is at play here, where different parties only know part of the picture (and fail to bring these together and find out what would be best globally.)
- kryptonomist 2y agoGlad to see those valuable principles written, even if it seems we are heading in the complete opposite. At least we can try to apply them on our side business. These were also true in the early ages of aviation: “Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.” ― Antoine de Saint-Exupéry
- deleted 2y ago[deleted]
- kubb 2y agoSRE has got to be one of the organisations that have done the most damage in the big G. They were given a license to mandate things based on philosophical musings backed with no science, and they can decide what's best and should be done without any data, just based on feels. They also have a culture of misanthropy, patronization and contempt towards devs. From what I can tell anyway.
- alienchow 2y agoWhy don't you give Mission Control a try for 6 months?
- bru 2y ago[citation needed]
- sgarland 2y ago> culture of misanthropy, patronization and contempt towards devs. When you’re being paged for the Nth time because of an idiotic problem that you’ve pointed out repeatedly, you too might exhibit these traits.
- makerofthings 2y agoBe google SRE. Elite software engineer. Cool under pressure. Pager goes off! Grab pixel. Press finger print reader until it lets me enter my passcode. Ack page. Put down whisky. Shake self. 5 minutes to be logged in and dealing with the problem. Password. gnubby. password. gnubby. gnubby. gnubby. Check alert, see playbook, ignore playbook. Check which cell the problem is in. Correlate with rollouts. See a match. Roll back poorly tested dev promo project. Charts recover. Alert not firing. Log out. Back to whisky.
- kubb 2y agoThe drinking culture is also pretty weird.
- joshuamorton 2y ago
- ChrisArchitect 2y agoSome more recent discussion: https://news.ycombinator.com/item?id=39580346 https://news.ycombinator.com/item?id=39580346
- YokoZar 2y agoSee also the Simplicity chapter in the followup Google SRE Workbook: https://sre.google/workbook/simplicity/ https://sre.google/workbook/simplicity/
- randmeerkat 2y agoGoogle’s “best practices” lead them to deleting an entire customer’s $135 billion pension account [1]. I’m surprised anyone is still reading anything Google writes. 1. https://arstechnica.com/gadgets/2024/05/google-cloud-accidentally-nukes-customer-account-causes-two-weeks-of-downtime/ https://arstechnica.com/gadgets/2024/05/google-cloud-acciden...
- klabb3 2y agoYou’re assuming that those systems were all implemented to the letter of that guide. That’s never the case. Often these type of guidelines are written to address recurring problems found in an organization.
- thirteenfingers 2y agoThat was seven years later. Maybe the problem is that Google stopped reading what Google wrote.
- passion__desire 2y agoGoogle : We will breach the rules we preach.
- wiseowise 2y agoRight, because every line of code written by tens of thousands of Google engineers is being validated against every guidebook.
- randmeerkat 2y ago> That was seven years later. Maybe the problem is that Google stopped reading what Google wrote. The problem is that it was never that good. Anyone who has used K8s at scale will tell you at length how it doesn’t scale. People should stop focusing on tech companies like celebrities and focus instead on domain problems related to their business.
- 2y ago
- maximinus_thrax 2y agoMaybe an unpopular opinion, but this type of content is useless and serves no other purpose than feeding the already bloated Google cargo-culting machine.
- dieortin 2y agoCould you elaborate on why you think that? Just stating that does not really add to the conversation
- maximinus_thrax 2y agoBecause is just useless. I mean seriously, what valuable insight does anyone get from that? It's some sort of truism wrapped in a word sandwich, ready for linkedin lunatics to pat themselves on the back sharing it. Do you feel you've gained something by reading it? Is this a valuable piece of intelligence which would guide your future decisions? Will you bring this to the team during an argument to push your agenda? This feels like the same type of 'feel good' content which people read and then feel like they did something productive. But I would argue that every piece of insight coming for a mega corp, valuable inside the mega corp is actually dangerous outside when people take it as dogma and try to apply it. SRE in general is something which IMHO of working in the industry for decades has poisoned the industry with half assed cargo cult implementations. But it has Google branding, so it must be valuable, hits hard for the fanboys and obviously can and should be applied in every company and every context. I also find it ironic to see 'Simplicity' touted from the same people who let Kubernetes lose in the wild, but that's a different story for a different time
- gtirloni 2y agoThat's circular reasoning ("it's useless because it's useless"). If you haven't gained any insights from reading that content, maybe it doesn't apply to you or you don't know what you don't know. > valuable inside the mega corp is actually dangerous outside when people take it as dogma and try to apply it. mega corp or not, dogmatic principles are usually bad coming from anywhere. The SRE book contains insights that apply to startup, medium-sized companies and mega corps. It's not prescriptive for a reason.
- zbentley 2y agoMany commenters here are rightly pointing out Google’s hypocrisy in actually following the principles in this article. Fair enough. But others are throwing the baby out with the bathwater: it’s a little silly to read comment after comment saying that the advice in TFA must be bad because Google does dumb/bad stuff on the regular. Companies aren’t homogenous. Even misguided companies may employ people who can teach others important things. Boeing is a perfect example of this. I would absolutely read an article proposing principles of engineering reliability from a Boeing eng/QA greybeard. Even as the rest of the company spiraled due to horrible leadership and management practices, many people in engineering and quality control did their damnedest to keep those failures from causing even more harm and loss of life. Those people probably have very valuable lessons to share about how to maintain what quality you can in a deeply hostile environment.
- dangus 2y agoI don’t see any validity to the alleged hypocrisy. End users making that criticism are confusing the products with the reliability practices.
- Tao3300 2y agoIndeed, allegations of hypocrisy are a class of ad hominem. They don't necessarily weigh in on the validity. It just... feels good? I guess? People LOVE to feel like they caught a hypocrite. It's probably in the Top 5 most sought after dopamine kicks.
- tbrownaw 2y ago>here's why doing <thing> is dumb >regularly does <thing> I think that might be a reason to suspect that the person doing that is hiding some holes in their argument.
- water-your-self 2y agoOn the contrary, they are likely very familiar with <thing> and their judgement comes from experience. Imagine a smoker telling you to never touch a cigarette
- wouldbecouldbe 2y ago"Why don’t we gate the code with a flag instead of deleting it?" These are all terrible suggestions. Source control systems make it easy to reverse changes, whereas hundreds of lines of commented code create distractions and confusion." In most cases to delete code would be a good idea, but to say that source control systems make reverting easier. After a few months most developers will have forgot about those lines and at times uncommenting code & explaining it explicitly might be a better way to preserve knowledge then to rely on digging through GIT.
- lloydatkinson 2y agoFirst time I’m hearing that feature flags and commented out code are the same thing.
- wouldbecouldbe 2y agoI've seen it be a company culture thing where every discussion was resolved with we'll put it behind a config/flag. It's an easy way to avoid hard choices. It's probably something like that the author refers to.
- wiseowise 2y agoNothing like supporting dead code forever just because you might need it some day.
- gnuser 2y agoAt the last “real” job I tried to help implement this as part of and later the manager of the ops team. It’s a great start, but in that case management wanted the idea of devops/sre but didn’t actually support it, and it really was a shit show. If you have a bad CTO and leadership on the board level, no amount of re-tooling will paper over their lack of support for the real principles.
- YZF 2y agoThis reads for me as a reflection of Google politics/org structure. The SRE org positioning itself as the guardian of system design vs. the SWEs who are agents of complexity. Doesn't feel healthy to me. The principles are fine but it's the SWEs that should be talking and applying them because they are "closer" to the decisions.
- MichaelMug 2y ago> Source control systems make it easy to reverse changes I have not observed this to be the case. After a few revisions there are so much changes that the code cannot be reversed without loosing a lot. A mech aims to cut out the soon-to-be-dead code like a flag is better. But perhaps maybe I’m doing something wrong.