9 ms·
23:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a seco
by ams6110 10y ago
23:00-ish
YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com
Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.
- eric_h 10y agoGood lesson on making command prompts on machines always tell you exactly what machine you're working on.
- m_fam_wa_k 10y agoI like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.
- joshaidan 10y agoHow do you go about colour coding your terminal?
- TimWolla 10y agoI assume he color coded the prompt. You can use ANSI color escape codes in there to e.g. color your hostname. Here's a generator for Bash: http://bashrcgenerator.com/ http://bashrcgenerator.com/, the prompt's format string is stored in the $PS1 variable.
- TimWolla 10y agoIn this case it looks like it has been a confusion between two different replicated Production databases. So this would not have helped.
- deleted 10y ago[deleted]
- niftich 10y agoI do this too, but in this case both machines were production, so this alone would not have sufficed. The system-default prompts on the other hand are universally garbage.
- tokenizerrr 10y agoHow exactly do you color code it?
- x0x0 10y agofor example, in .bashrc on osx: RED=$(tput setaf 1) NORMAL=$(tput sgr0) PS1="\[${RED}\]PROD \[${NORMAL}\]\W\$ " produces prompt PROD ~$ <-- prod in red, directory ~
- sytse 10y agoYep, good idea. The same thing has been suggested by team members http://imgur.com/a/TPt7O http://imgur.com/a/TPt7O
- Perihelion 10y agoAll of my non-production machines have emojis in PS1 somewhere. It sounds ridiculous, but I know that if I see a cheeseburger or a burrito I'm not about to completely mess everything up. Silly terminal = silly data that I can obliterate.
- StavrosK 10y agoI think I'd rather make the production systems stand out, and add the emojis there. My prompts have a red background, but emoji prompts just tickle me, somehow.
- kofejnik 10y agoI've been color-coding my PS1 for years, but this is seriously brilliant, thanks!
- scrame 10y agoha! I do the same thing with figlet and cowsay, it prints a big dragon saying "welcome to shell!", if I see that, then I know I'm on a box I own/have sudo/am me. it's a good visual reminder. I don't fuss with prompts much, but this is a pretty good idea!
- marcosdumay 10y agoI got the idea of an emoji prompty that is happy or sad depending on the $? value from somebody here a few years ago. I run it on any dev environment since then.
- cabargas 10y agoIs a really good idea, and is one of the improvements that are likely to be put in place as soon as possible. Its already listed on the document.
- gknoy 10y agoI use iterm2's "badging" to set a large text badge on the terminal of the name of the system as part of my SSH-into-ec2-systems alias: i2-badge () { printf "\e]1337;SetBadgeFormat=%s\a" $(echo -n "$1" | base64) } It's not quite as good as having a separate terminal theme, but then I haven't been able to use that feature properly. :(
- kevindong 10y agoIt seems Gitlab has noticed your comment. Recovery item 3f currently says: > Create issue to change terminal PS1 format/colours to make it clear whether you’re using production or staging (red production, yellow staging)
- karmicthreat 10y agoI've done this exact thing my my servers. Also like GL I prepend PRODUCTION and STAGING on PS1. I should probably make the PRODUCTION flash just in case.
- hobarrera 10y agoWe need a <blink> tag for our PS1.
- spydum 10y agoGood news! Enable ANSI in your terminal and: ESC[5m http://wiki.bash-hackers.org/scripting/terminalcodes http://wiki.bash-hackers.org/scripting/terminalcodes
- eric_h 10y agoYou can do that - I actually tried it and everyone on my team agreed that it was bloody annoying, so I made it stop blinking (still red, though).
- Ezhik 10y agoNot as serious, but I always set root terminals on my system to be this scary red.
- NamTaf 10y agoI have a user and a local admin account at work on our Windows SOE. I have made my PC's cmd.exe show green on black in large font, but intentionally did not change the local admin. When I run a command prompt in admin it's visible different. This is a great tip and it's caught me many times.
- simooooo 10y agoMy dev is yellow, staging orange, and live red
- nodesocket 10y agoIf using iterm you can set a "badge" text on a terminal window that shows up as an overlay. Super useful when you have lots of SSH sessions open to different servers.
- corecoder 10y agoiterm badges have saved me many times.
- echelon 10y agoThis doesn't really help if there are multiple production databases. It could be sharded, replicated, multi-tenant, etc.
- gnufied 10y agoWhy would it matter? In my last job we had user home directories synced via puppet (I am overly simplifying this) which enabled any ops guy to have same set of shell and vim configuration settings on production machines too. I daresay - having hostname as part of prompt saves lot of trouble.
- echelon 10y agoHaving the hostname on the prompt is a good idea, but I don't think it would have helped with this process failure. I work at a company where we have hundreds of database machines. Running this kind of command _anywhere_ without some kind of plan would be foolish. (It's one of the reasons why we have a datastore team that handles database administration.) But the same lesson applies to application servers as well. Don't run deleterious commands out of curiosity. Have a peer-reviewed roll plan to act on when doing things like this. A role plan would have called for verifying the host before running the command. But even before that, the issue should have been investigated more! All of these things contributed to the failure. There should ideally be better ownership through dedicated roles, peer-reviewed processes for dangerous activities, and a better process for investigation that does not involve deleting things haphazardly.
- dorfsmay 10y agoUh no! Don't rely on command prompt, there are hardcoded ones out there, and clonining scripts have duplicated them. uname -n Takes seconds.
- vacri 10y agocolour is more noticeable than words. I do both, though.
- wtbob 10y ago'db1' vs. 'db2' is still insufficiently clear, though. Even better would be e.g. to name development systems after planets and production systems after superheroes. Very few people would mistake 'superman' for either 'green-lantern' or 'pluto,' but it's really easy to mistake 'sfnypudb13' for 'sfnydudb13.'
- grzm 10y agoOr skip the middleman: prod-db1, prod-db2, dev-db1, dev-db2.
- wtbob 10y agoVisually there's not a whole lot of difference. Ideally, you want something where the shape of each name is reasonably distinct from all the rest — otherwise folks will just ignore that odd blot before the prompt. And then, once you've started naming things as $ENV-$TYPE, someone will want to cram in the location, and the OS, and the team which maintains it, and the customer, and and and. Then someone will reduce all of those identifiers into single characters … and you'll be in the situation I mentioned. Clearly clpudb1 is CharlesCorp's first London production database system!
- grzm 10y agoI strongly argue against those types of abbreviations in naming, for exactly the points you make. Here's how I view it: The distinction between prod and dev is pretty clear cut. The words "dev" and "prod" have significantly different shapes and are immediately unambiguous. There's no need to remember a superhero vs astronomical object distinction. As the number of database server instances grows, you've got another level of naming issues that arises when needing to distinguish between hosts and db server instances—and perhaps even database/schema independence if necessary. Staying consistent and easy-to-use with an arbitrary naming scheme becomes increasingly unwieldy, in my opinion.
- sratner 10y agoAdditionally, I have taken up a habit of not abbreviating "production" and "staging", whenever possible. The fact that len("production") > len("staging") > len("dev") is a feature when you find yourself typing it into a terminal or db shell.
- theptip 10y agoAlso, as a safety net, sometimes you don't need to run `rm -rf` (a command which should always be prefaced with 5 minutes of contemplation on a production system). In this case, `rmdir` would have been much safer, as it errors on non-empty directories.
- bpchaps 10y agoThese days, I've been very implicit in how I run rm. To the extent that I don't do rm -rf or rmdir (edit: immediately), but in separate lines as something like: pushd dir ; find . -type f -ls | less ; find . -type f -exec rm '{}' \; ; popd ; rm -rf dir It takes a lot longer to do, but I've seen and made enough mistakes over the years that the forced extra time spent feels necessary. It's worked pretty well so far -- knock knock.
- gizmo 10y agoGood advice. I would add a step where you dump the output of find (after filtering) into a textfile, so you have a record of exactly what you deleted. Especially when deleting files recursively based on a regular expression that extra step is very worthwhile. It's also a good practice to rename instead of delete whenever possible. Rename first, and the next day when you're fresh walk through the list of files you've renamed and only then nuke them for good.
- kstrauser 10y agoBTW, find ... -delete avoids any potential shell escaping weirdness and saves you a fork() per file.
- wodny 10y agoThis seems to be the best here. As a side note: if someone does something more complicated and uses piping find output to xargs, there are very important arguments to find and xargs to delimit names with binary zero -- -print0 and -0 respectively. Very interesting article: https://www.dwheeler.com/essays/fixing-unix-linux-filenames.html https://www.dwheeler.com/essays/fixing-unix-linux-filenames.....
- sbuttgereit 10y agoAlso a good lesson for testing your availability and disaster recovery measures for effectiveness. Far, far too many companies get production going and then just check to see that certain things "completed successfully" or didn't throw an overt alert in terms of their safety nets. Just because things seem to be working doesn't mean they are or that they are working in a way that is recoverable.
- cesarb 10y ago> Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated. In these situations, I always keep the following xkcd in mind: https://xkcd.com/349/ https://xkcd.com/349/
- anamoulous 10y agoI audibly gasped when I hit this part of the document.
- craigkerstiens 10y agoI'm not sure there is a "late" at night or tired in the incident report. All times are UTC and it's unclear where all the team is located but if in SF then this is at 4pm which is when the incident just occurred. It doesn't necessarily change that you shouldn't be firefighting for extremely long cycles in hero-mode for that long, but not exactly the same as exhausted and powering away for hours.
- sbuttgereit 10y agoActually, there is a brief comment: "At this point frustration begins to kick in. Earlier this night YP explicitly mentioned he was going to sign off as it was getting late (23:00 or so local time), but didn’t due to the replication problems popping up all of a sudden."
- Scriptor 10y agoGitlab's team is worldwide. YP seems to be someone in EU time, which right now is UTC+1. That would mean that the incident happened around 11pm or midnight in YP's local time.
- YorickPeterse 10y agoCorrect, I'm based in Europe/Amsterdam so this happened mostly during the evening.
- 1_2__3 10y agoI think I can count on one hand the number of times I've run an rm command on a production server. I'll move it at worst, and only delete anything if I'm critically low on disk space. But even then I don't even like typing those characters if I can avoid it, regardless of if I'm running as root or a normal user.
- mbrain 10y agoJust a thought, but is it possible to override destructive commands to confirm the hostname by typing it before running the command?