8 ms·
> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected
by tarruda 6y ago
> I do not envy Google engineers right now.
A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release.
I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.
- ignoramous 6y agoSame. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).
- robertlagrant 6y agoWho automates the automators? :)
- papito 6y agoJesus. One would think you'd have some safeguards for that. Even Dropbox will give you an alert if you try to nuke over 1,000 files. More reasons to COLOR CODE your work environments, if possible.
- ignoramous 6y agoYes but that was eons ago. The safeguards are well and truly in-place now. Not just one, several in fact.
- qz2 6y agoApart from the ones that they haven't worked out yet :)
- zentiggr 6y agoWhen I meet the engineer who can design for the unknown unknowns, I will bow to them.
- papito 6y agoThe trick is to be paranoid. You literally sit down and think exclusively about what COULD go wrong.
- GreenWatermelon 6y agoAnxiety is a bitch.
- sitkack 6y agoFormal methods for your formal methods. And never shipping on Friday.
- marcus_holmes 6y agocolorblind (red/green) person here - 5% of the male population just don't see color enough for it to be an important visual clue. So sure, color-code your environments, but if you find someone about to do something to a red environment that they clearly should only be doing to a green environment, just check if they're seeing what you're seeing before you sack them ;)
- warrenm 6y agoMy primary customer right now color codes both with "actual color", and with words - ie the RED environment is tagged with red color bars, and also big [black] letters in the [red] bars reading "RED"
- deleted 6y ago[deleted]
- ahachete 6y agoDoesn't AWS (and every big cloud/enterprise) follow best-practices for production operation like FIT-ACER? https://pythian.com/uncategorized/fit-acer-dba-checklist/ https://pythian.com/uncategorized/fit-acer-dba-checklist/ That's even more surprising to me.
- ayoubElk 6y agoIrrelevant to the discussion, but I just wanted to say thank you for the categorized list of users I can follow on your profile!
- rptr 6y agowow, that list is crazy. thanks OP.
- laddng 6y agoWhich tool do you use to follow users on HN?
- ignoramous 6y agoThere used to be hystry.com [0] but it isn't functional anymore. Another workflow, though cumbersome, is: Search for a username on hn.algolia, select "comments" and "past months" as filters, then press enter. Ex: https://hn.algolia.com/?dateRange=pastMonth&query=nostrademons&page=0&prefix=true&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=pastMonth&query=nostrademo... [0] https://news.ycombinator.com/item?id=71827 https://news.ycombinator.com/item?id=71827
- laddng 6y agoThanks for sharing those tips!
- znpy 6y agoIt reminds me of this: https://www.youtube.com/watch?v=30jNsCVLpAE https://www.youtube.com/watch?v=30jNsCVLpAE -- "GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill"
- ardy42 6y ago> At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases). It seems like a design flaw for actions like that to be so easy. E.g. > Hey, we detected you want to delete an AWS region. Please have an authorized coworker enter their credentials to second your command.
- ignoramous 6y agoIf it was indeed in-production, I'd never in a million years have had access rights to delete the stack. Those are gated just the way one would imagine they should be. The service stack for the region (and not an entire region itself) looked like prod, but wasn't. It made me feel like shit anyway.
- solids 6y agoWell, it's something that can happen to anyone, take it easy. When I made the transition from developer to manager and become responsible for this situations, at first every problem made me feel as you describe. Eventually what helped me to be free is the understanding that how we feel about a fact does not change anything about that fact.
- domano 6y agoI remember how one of our engineers had his docker daemon connected to production instead of his local one and casually did a docker rm -f $(docker ps -aq) . Same thing happened to me but with CI, which felt bad enough already.
- alias_neo 6y agoNo Engineer should have production access from their workstation. Period. source: am Engineer =).
- tasuki 6y agoWhy not? (I think I can find some cases where production access from an engineer's workstation is a good idea)
- alias_neo 6y agoIt can be efficient, particularly in smaller companies, but that's where exactly this rule should be applied. In some industries, security and customer requirements will at times mandate that developer workstations have no access to production. Deployments must even be carried out using different accounts than those used to access internal services, for security and auditing purposes. There are of course good reasons for this; accidents, malicious engineers, overzealous engineers, lost/stolen equipment, risk avoidance, etc. When you apply this rule, it makes for more process and perhaps slower response times to problems, but accidents or other internal-related issues mentioned above drop to zero. Given how easy it is to destroy things these days with a single misplaced Kubernetes or Docker command, safeguards need to be put in place. Let me tell you a little story from my experience; I built myself a custom keyboard from a Numpad kit. I had gotten tired of typing so many docker commands in every day and I had the desire to build something. I built this little numpad into a full blown Docker control centre using QMK. A single key-press could deploy or destroy entire systems. One day, something slid off of something else on my desk, onto said keyboard, pressing several of the keys while I happened to have an SSH session to a remote server in focus. Suffice it to say, that little keyboard has never been seen since. On an unrelated topic, I don't have SSH access to production systems.
- cube00 6y agoDon't be too hard on yourself, no dev works in a silo, there is usually user acceptance testing and product owner sign offs involved so they also have to wear some of this too.
- rplnt 6y agoIf you alone were able to do it, then the system was designed badly. The bigger the impact, the more robust it has to be to prevent accidents.
- deleted 6y ago[deleted]
- a3_nm 6y agoThe big mistake in the system is that everyone in the world is relying on Google services... These problems would have less impact with a more diverse ecosystem.
- lawrjone 6y agoWould they have less impact? Or would it have the same impact, just distributed across many more outages? You can rely on Google outages being very few and far between, and recovering pretty fast. For the benefits you get from such a connected ecosystem, I'm not sure anyone is net positive from using a variety of different tools rather than Google supplying many of them.
- rightbyte 6y agoCompare closing down one road for repair a day per year to closing down all roads one day a year.
- lawrjone 6y agoI'm not sure I see that as a fair comparison. I think it's best to use the same durations for this, as an entire day changes the level of impact. <1hr outages have a fraction of the impact that an entire days outage might have. It's obviously subjective, but even with our entire work leaning on Google– from GMail, GDrive and Google Docs, through to all our infrastructure being in GCP– todays outage just meant everyone took an hour break. History suggests we won't see another of these for another year, so everyone taking a collective 60m break has been minimally impactful vs many smaller, isolated outages spread over the year.
- papito 6y agoWhen I was interviewing at Morgan Stanley, I asked "how do you do this job if a mistake can cost people money?". The answer was "well, if you don't do anything, you make NO money".
- cvrjk 6y agoWelp, as a new grad there, I had brought down one very important database server on a Sunday night (a series of really unfortunate events). Multiple senior DBAs had to be involved to resuscitate it. It started functioning normally just a few hours before market open in HK. If it was any later, it would have been some serious monetary loss. Needless to say, I was sweating bullets. Couldn't eat anything the entire day lol. Took me like 2 days to calm down. And this was after I was fully shielded cuz I was a junior. God knows what would've happened if someone more experienced had done that.
- b0afc375b5 6y agoI'm so glad I'm not the only one feeling deployment anxiety. The project I'm involved in doesn't really have serious money involved, but when there's a regression found only after production deployment my stress levels go up a notch.
- FloatingUp 6y agoWhen I was working at a pretty big IT provider in the electronic banking sector, we (management and senior devs) made it an unspoken rule, that: - Juniors shall also handle production deployments regularly. - A senior person is always on call (even if only unofficially / off the clock). - Junior devs are never blamed for fuckups, irrespective of the damage they caused. That was the only way to help people develop routine regarding big production deployments.
- awesomemonster 6y agoSame thing -- used to work at a very large hosting provider. One of our big internal infra management teams wouldn't consider newhires fully "part of the team" until they had caused a significant outage. It was genuinely a right of passage, as one person put it, "to cause a measurable part of the internet to disappear". I got to see a lot of people pass through this right of passage, and it was always fun to watch. Everyone would take it incredibly seriously, some VP would invariably yell at them, but at the end of the day their managers and all their peers were smiling and clapping them on the back.
- zeven7 6y agoSeveral years back when I was working at Google I made a mistake that caused some of the special results in the knowledge cards to become unclickable for a small subset of queries for about an hour. As part of the postmortem I had to calculate how many people likely tried to interact with it while it was broken. It was a lot and really made me realize the magnitude of an otherwise seemingly small production failure. My boss didn't give me a hard time, just pointed me toward documentation about how to write the report. And crunching the numbers is what really made me feel the weight of it. It was a good process. I feel for the engineer who has to calculate the cost of this bug.
- zymhan 6y agoPresumably there were more failures than a single engineer could've been responsible for here.
- fjcp 6y agoIts absolutely possible, the worst AWS outage was caused by one engineer running the wrong command [0]. "This past Tuesday morning Pacific Time an Amazon Web Services engineer was debugging an issue with the billing system for the company’s popular cloud storage service S3 and accidentally mistyped a command. What followed was a several hours’ long cloud outage that wreaked havoc across the internet and resulted in hundreds of millions of dollars in losses for AWS customers and others who rely on third-party services hosted by AWS." [0] - https://www.datacenterknowledge.com/archives/2017/03/02/aws-outage-that-broke-the-internet-caused-by-mistyped-command https://www.datacenterknowledge.com/archives/2017/03/02/aws-...
- robotnikman 6y agoThis sounds like a good practice and hopefully something they still do. Calculating the exact numbers would definitely help cement the experience and its consequences into your mind.