40 ms·
This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far mo
by gizmo 10y ago
This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed.
When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize the steps. If there is no coworker, talk to your rubber ducky or stapler on your desk. This will help you catch mistakes. Then when the entire plan looks sensible, go through the steps one by one. Don't deviate from the plan. Don't get distracted and start switching between terminal windows. While making the checklist ask yourself if what you're doing is A) absolutely necessary and B) risks making things worse. Even when the angry emails are piling up you can't allow that pressure to cloud your judgment.
Every startup has moments when last-minute panic-patching of a critical part of the server infrastructure is needed, but if you use a checklist you're not likely to mess up badly, even when tired.
- connorshea 10y agoYup, it's never the fault of a person, always of the system. Once we get this resolved we'll definitely look at ways to prevent anything like it in the future.
- deleted 10y ago[deleted]
- duncan_bayne 10y agoWishing you the best of luck partly out of sympathy, and partly as an active user wanting my account operative again :)
- wereHamster 10y agoAt some point you have to blame the person. If the person did it wilfully, deliberately, ignoring the checks and safeguards. Or because there's already so much system that having any more of it would place severe restrictions on everyday tasks. I'm not saying that's the case here, because it does seem that GitLab has systemic deficiencies. But "never" and "always" are such strong statements. Prosecutor: Mr. Accused, here is evidence that you murdered that other person. Accused: It's not my fault, but the systems'. Judge: Oh, ok. You are a free man.
- DanHulton 10y agoThe law and a "no fault" postmortem are entirely different things and you conflating the two doesn't help the discussion. Let's assume a bad actor in a company. It still doesn't help improve the situation to allow blame to rest with the bad actor. Definitely, there should be penalties applied (likely the termination of their position), but it doesn't help your company at all to stop there. Did they delete data? Why is there no secure backup system in place to recover that data? Why was there such lax security in place to allow them to delete the data in the first place? Why are we hiring people who will go rogue and delete data? Did they "turn" after working here for a while because of toxic culture, processes, etc? Hell, if the law worked this way, we might actually have less crime because we'd look further into the causes of crime and work to address them instead of simply punishing the offenders.
- kefka 10y agoThe reasoning for that was to prevent reducto ad absurdum and "It's YOUR fault". "You should have KNOWN that erudite command was going to fail." "You should have known that our one-off program had issues you did not account for." "You should have known that the backups were not properly tested elsewhere for known good state." "You should have ....." In reality, some disasters were caused by idiotic things like "rm -rf /opt/somedir ." You just hosed the system, or a large part of it quickly. And we could say that your malfeasance of including the "." started wiping / immediately. But we can also say that rm should be aliased to prevent accidents like that, or that rm should do some minimalistic sanity checking on critical directories before executing them. People can, and will mess up. These computers are nice, in that they can have logic that can self-correct, or at least loudly alert errors.
- click170 10y agoBBC's Horizon has a really good episode about checklists and how they're used to prevent mistakes in hospitals, and how they're being adopted in other environments in light of that success. It's called How To Avoid Mistakes In Surgery for the interested.
- virtuabhi 10y agoHere's the book on this topic- The Checklist Manifesto https://smile.amazon.com/Checklist-Manifesto-How-Things-Right/dp/0312430000?sa-no-redirect=1 https://smile.amazon.com/Checklist-Manifesto-How-Things-Righ...
- rgawdzik 10y agoA must read if you don't want to make mistakes, it's given to all new Square employees.
- ndh2 10y agoThere's the article that this book is based on The Checklist http://www.newyorker.com/magazine/2007/12/10/the-checklist http://www.newyorker.com/magazine/2007/12/10/the-checklist
- pgambling 10y agoSo glad someone brought up this book. It a good read and full of sound advice.
- FireBeyond 10y agoThere are very, very few situations in life where there's not enough time to take a time-out, sitrep, or checklist. I work in EMS when not in IT, and even bringing a trauma or cardiac arrest patient into the Emergency Room, there is still time to review and consider. If there's time in Emergency Medicine, there's time in IT.
- flukus 10y agoI worked in software to help manage this for a while. There are still checklists but they are produced ahead of schedule. Every instrument a nurse takes off the tray is counted and then checked at the end for instance.
- batbomb 10y agoAnd a logbook
- nowarninglabel 10y agoYes, good tip from "Turn the Ship Around" by David Marquet is to use the "I intend to" model. For every action you are going to undertake that is critical, first announce your intentions and give enough time for reactions from others before following through.
- secfirstmd 10y ago"I intend to" is actually a phrased deliberately used a lot during military mission briefings. I always wondered why, I guess it's deliberate.
- cmdkeen 10y agoThe military uses "intention" rather than "I will" because they understand that no plan survives contact with the enemy. It is also higher level so that when exhausted, stressed subordinates find themselves in life threatening circumstances the most important thing they need to remember is the intent. If they forget steps 1-5 of the plan but recall the intent they can't go that badly wrong in using their initiative. There's also the distinction between specific orders - "you are to". Knowing the commander's intent, and his commander's intent (known as 1 Up and 2 Up) enables Mission Command, the concept of giving latitude to subordinates to achieve the mission in the best way possible within the confines and direction given to them. Finally there's the ritual of it, NATO forces expect to operate within multi-national structures where English won't be a first language. There is a "NATO sequence of orders" which should be roughly followed. It means everyone knows the structure of what is coming up and when in the brief - so you don't get people asking questions about equipment when the limitations are being explained, they know that comes later. Opening, especially from a junior officer, with "my intention is" is essentially like having a schema definition at the start of a document - it defines the structure of what is coming for those who are going to be parsing it.
- secfirstmd 10y agoMuch better answer! I recognise a lot of that.
- 10y ago
- MattRogish 10y agoWe have 2 people on rotation and anything like this you pair with someone else and talk through it. As we like to say "It's only a mistake if we both make it."
- chrissnell 10y agoAbsolutely. Same goes for any ops response. You need two people: one to triage the issue and another one to communicate with external stakeholders and to help the one doing the triage. The military does a very similar thing. An Army company commander usually has a RTO (radiotelephone operator) to handle taking on the radio. This frees the commander to make real-time decisions and response quickly to the situation on the ground, and frees him/her from having to spend time explaining things. A really good RTO will function as the voice of the commander, anticipating what he/she needs to get from the person on the other end of the radio. This is a great characteristic of a good operations engineer, too. While the person doing triage is addressing the problem, the communicator is roping in other resources that might be needed and communicating the current situation out to the rest of the team/company
- deleted 10y ago[deleted]
- throwaway8124 10y agoIndeed, and a very good way to proceed. Another thing to note, is that the RTO will occasionally state someone and "ACTUAL". That means that whomever they are speaking for is actually speaking and not the RTO on behalf of the CO.
- atmosx 10y agoThat is interesting: You're saying that at any given weekend you have 2 ppl on rotation which work on parallel when/if there's a crisis?
- bluedino 10y ago"What? You told me to do that!" "I meant the other server."
- ebbv 10y agoAnother lesson is this one which I also learned the hard way: don't work long hours or late at night unless it is absolutely necessary. It doesn't sound like it was totally necessary for YP to be pushing this hard, and pushing that hard is what leads to these kinds of errors. Take your time and work when you're at your best.
- hobarrera 10y agoOn a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.
- wtbob 10y agoEven better, move stuff to /trash/$(date -I), and have your cleanup script regularly clean out any directory from before a week ago.
- namepros 10y agoThis sounds good until you consider that many systems are utilizing multiple drives. When someone is expecting to delete a large file and it ends up on a different drive, problems could arise.
- yorwba 10y agoRenaming a file should not move its data, right? So rename the file into .file.$(date -Iseconds).trash (but make sure that no legitimate files are ever named in this pattern). Then put that file path into a global /var/trashlist. To cleanup, you just check that file for expired trash and make the final deletion.
- deadbunny 10y agoIt will when moving across disks.
- traviscj 10y agoBeware race conditions when writing to /var/trashlist (assuming you mean "a file with one path per line.") Proposed tweaks: symbolic link into /var/trashlist directory, where the name of the symbolic link is "<timestamp>-<random stub>-<original basename>". Timestamp first so we can stop once we hit the first too-recent timestamp, random stub to unique the original base name if two different files In different directories are deleted at the same timestamp, original file name for inspection.
- 10y ago
- FireBeyond 10y agoThe amount of times describing something to a co-worker or manager out loud has allowed me to catch a problem or caveat is innumerable. Even more so if you tag-team things, so much better.
- ryandrake 10y agoIf you get the chance to observe pilots operating in the cockpit, I'd recommend it. Every important procedure (even though the pilot has it memorized) is done with a checklist. Important actions are verbally announced and confirmed: "You have the controls" "I have the controls". Much of flight training deals with situational awareness and eliminating distractions in the cockpit. Crew Resource Management[1]. 1: https://en.wikipedia.org/wiki/Crew_resource_management https://en.wikipedia.org/wiki/Crew_resource_management
- nihonde 10y agoThis reminds me of Japanese train crews and factory workers who use hand signals and audible call-outs to reinforce checklists with muscle memory.
- misiti3780 10y agoAs documented in this great book:https://www.amazon.com/Checklist-Manifesto-How-Things-Right/dp/0312430000 https://www.amazon.com/Checklist-Manifesto-How-Things-Right/... not just pilots ... doctors, nurses, etc.
- sschueller 10y agoThere is a neat video[1] where a Swiss flight has to make an emergency landing and just happens to have a film crew in the cockpit. [1] https://www.youtube.com/watch?v=rEf35NtlBLg https://www.youtube.com/watch?v=rEf35NtlBLg
- schleppy_oc 10y agoThe best Ops people I have worked with (looking at you Dennis and Alan) repeat everything back that I say. More than once I have caught mistakes in my approach simply by hearing someone else repeat back exactly what I just said.
- late2part 10y agoThe best ops people have made all the mistakes they're helping you avoid.
- imsodrunklol 10y agoAmen brother. My worst nightmares are nothing but a blimp on a developers mind. I've have lost it all. I have lost it all on multiple servers. On multiple volumes. No one should ever experience that. Ugh, developing, sometimes, is just so frustrating.
- on_and_off 10y agoI think this applies to pretty much any field : http://thecodelesscode.com/case/100 http://thecodelesscode.com/case/100
- deleted 10y ago[deleted]
- atmosx 10y ago> Don't get distracted and start switching between terminal windows. I got bitten by this in the past, luckily nothing that could not be reversed, just 2-3 hours lost. I can imagine how YP's stomach must have felt when he realised what happened. Still, I had no idea about checklists and so many people here seem to be pretty familiar with the concept :-)
- NamTaf 10y agoTwo points: 1) Patio11 touches on a very good lesson, in passing, in an article about Japanese business[1]: While raw programming ability might not be highly valued at many Japanese companies, and engineers are often not in positions of authority, there is nonetheless a commitment to excellence in the practice of engineering. I am an enormously better engineer for having had three years to learn under the more senior engineers at my former employer. We had binders upon binders full of checklists for doing things like e.g. server maintenance, and despite how chafing the process-for-the-sake-of-process sometimes felt, I stole much of it for running my own company. (For example, one simple rule is “One is not allowed to execute commands on production which one has not written into a procedural document, executed on the staging environment, and recorded the expected output of each command into the procedural document, with a defined fallback plan to terminate the procedure if the results of the command do not match expectations.” This feels crazy to a lot of engineers who think “I’ll just SSH in and fix that in a jiffy” and yet that level of care radically reduces the number of self-inflicted outages you’ll have.) 2) I once heard organisational 'red tape' described as 'the scar tissue of process failures' and it is absolutely true and I deeply regret not recording the source of it. Whenever you wonder why there's some tiresome, overly onerous process in place that is slowing you down, consider why it may have been put in place - chances are, there was a process failure that resulted in Bad Things. When you wonder why big orgs are glacially slow compared to more nimble startup competitors, understand that those startups have yet to experience the Bad Things that the big org has probably already endured. Like scar tissue, the processes they develop reduce their agility and performance but also serve to protect the wounds they experienced. [1] http://www.kalzumeus.com/2014/11/07/doing-business-in-japan/ http://www.kalzumeus.com/2014/11/07/doing-business-in-japan/
- zwp 10y ago> regret not recording the source of it The googlable nugget is actually "organizational scar tissue" (it caught my attention too). It's from Jason Fried. On twitter: https://twitter.com/jasonfried/status/2758624714 https://twitter.com/jasonfried/status/2758624714 and also apparently in "Rework", quoted with more context here: https://www.goodreads.com/quotes/1012423-policies-are-organizational-scar-tissue-they-are-codified-overreactions-to https://www.goodreads.com/quotes/1012423-policies-are-organi...
- fnord123 10y agoYou missed the most important part: writ down the backout plan.
- discordianfish 10y agoChecklists are great if you have them and often you should create one even during an outage but other times you waste time. It's really hard to balance fixing things quickly and not breaking more. There isn't a silver bullet anyway, it's layer on layer of operational best practices what makes you resilient against such issues.