7 ms·
A few ops lessons we all learn the hard way (2020)
- crazylifetwist 4y agoCache: https://web.archive.org/web/20220817133026/https://www.netmeister.org/blog/ops-lessons.html https://web.archive.org/web/20220817133026/https://www.netme...
- dsr_ 4y agoI can point to examples for at least 80% of these. Mostly not in public, though.
- gonzo41 4y agoNo, 11. Took the last place I worked about 5 years to rollout a pretty good solution to a massive legacy set of servers. Though I gotta say, the super privileged automated certificate renewer thingy does seem like a real honey pot.
- mcqueenjordan 4y agoReally good article. Some of these are subtle, and really must be learned the hard way. The only one I found myself thinking I disagreed with was "85. Multithreading is rarely worth the added complexity." Maybe I simply have yet to learn it the hard way, but of all the ways to add complexity, I have tended to find multithreading as one of the more legitimate. That being said, it has to be done in a simple, easy to reason about way. Usually for me, this means fork-joining homogeneous tasks.
- twh270 4y agoYes. If you're going to do multi-threading, let the framework/language handle the hard parts[1]. [1] It's _all_ hard parts.
- eschneider 4y agoA. However well you understand multithreading, you only need one coworker who doesn't understand multithreading to make your life an unending hell. B. You always have at least one coworker who doesn't completely understand multithreading. :/
- cpurdy 4y agomultithreading to A. understand coworker who multithreading, you only need one multithreading. :/ doesn't coworker who However understand well you life an unending hell. B. You always have at least one doesn't make your completely understand
- eschneider 4y ago"But I only need to lock when I write!" O_o
- gwillen 4y agoI need this on a shirt.
- naasking 4y ago> Usually for me, this means fork-joining homogeneous tasks I think the article doesn't make the right distinction: parallelism is often worth it, concurrency is what causes the headaches. Fork-join is parallelism and generally safe and relatively easy.
- bravetraveler 4y agoQuite interesting but I don't really understand the framing with self-signed certs It can and (often does) go poorly, but the only thing you really gain from an external CA is a [minor] reduction in responsibility
- number6 4y agoOk lets spin up the ca and create a cert for this service. Validity 15 years. <15 years later> Why did this thingy break? How do I get a new cert? CA was on a Virtul Machine? The old server? What is ESXi? Ok we will just spin up this docker container, create a new cert and make it last 20 years...
- bravetraveler 4y agoI'm not doubting it can be done poorly I'm just curious why an org would do that when they have a domain controller/login infrastructure providing a CA, or one of the many secret-storing-engines backing the vast majority of their sensitive data (eg: Vault)? Policies can be defined and enforced to serve as guard rails -- we choose the nightmares we accept
- zaphar 4y agoAhhh, yes I see where you got lost. You are assuming that they either: * Have a team willing to commit to managing CAs as part of the DC/Login infrastructure. or * Are actually using something like Vault in their infrastructure. Common mistake, really. In all seriousness you would be amazed how many places neither of those things are true for, and additionally how much sheer effort it would take to make them true at those orgs.
- 0xbadcafebee 4y agoIf you have a team that manages a domain, it will take a month to get anything done with them, if they even allow you to get what you need done. If there is a way to make it easier, they aren't interested.
- 0xbadcafebee 4y agoIf a post-mortem follow-up task is not picked up within a week, it's unlikely to be completed at all. This one is literally a law of physics. People give talks at conferences not to convince others that their work is awesome and totally worth the time and effort they put in, but themselves. I would add: "If you see a big name company give a talk about some cool thing they made, it's probably already been abandoned by that company." Turning things off permanently is surprisingly difficult. If you don't have a plan to sunset whatever you're building, you're basically telling your future self to go fuck himself. Unless you plan to quit first, in which case you're telling your successor to go fuck himself. The source you're looking at is not the code running in production. ((cries)) Mandatory code reviews do not automatically improve code quality nor reduce the frequency of incidents. The primary purpose of mandatory code reviews - without a sensible plan of who, when, why or how - is just for people to nitpick your code.
- powerhour 4y ago> I would add: "If you see a big name company give a talk about some cool thing they made, it's probably already been abandoned by that company." Alternatively, the talk should be viewed as the company recruiting someone to replace said cool thing.
- marcosdumay 4y agoA "plan to sunset" something looks like an incredibly alien concept to me. How do you do it when people keep finding new uses for whatever it is?
- WJW 4y agoWith careful change management and access control. It's easy to turn off a system if you know for sure that nobody is using it anymore. Large companies and militaries do it all the time just fine so there is no reason a dynamic young startup shouldn't be able to do it. :) (As a non-sarcastic response: back when I was an officer in the navy, how we would get rid of systems was most definitely taken into account from the very start. Even before we started building ships or radars or whatever, budgets and dock space would be reserved ~30 years into the future to do the decommissioning work. I do realize that this works much better for established organizations that can be reasonably sure that they will exists in 30 years, after all most startups are hard pressed to last even two. That said, completely disregarding any planning on how to grow out of your current systems does seem to have been the case at all the startups I have consulted for and was a major part of technical debt. IMO a technical leader should know where the skeletons are hidden in the setup for their organization and know roughly in how many months/years they will no longer suffice. Then they can plan the replacement and/or upgrading of said skeletons accordingly)
- andreareina 4y agoTAI > UTC. Except for the fact that no-one uses TAI.
- marcosdumay 4y agoAlmost everyone is quite ok with whatever, will treat whatever they get as UTC, will be happy to ignore the difference. Everybody that is not on the above paragraph uses TAI.
- a1369209993 4y ago> Everybody that is not on the above paragraph uses TAI. To be fair, even many of us TAI users are quite ok with whatever, will treat whatever we get as TAI, will be happy to ignore the difference. After all, if you use UTC, you're wrong, and anything that goes wrong as a result is your fault.
- marcosdumay 4y ago> and anything that goes wrong as a result is your fault Honestly, that doesn't give me the "not care about it" vibes.
- a1369209993 4y ago> "not care about it" Never said that. Rather, some of us are commited to attributing blame to what actually caused the problem (namely, leap seconds and anyone who listens to the IAU).
- belfalas 4y ago> Your network team has a way into the network that your security team doesn't know about. This one is true, and even if you right now go and try to make it not true for your company...it will become true again later.
- WFHRenaissance 4y agoThis, along with the obvious intersection of skillsets and interests, is why I often now see either network ops and security bundled together on the same team/org, or DevOps people doing the work of both teams.
- dsr_ 4y agoDevNetSecOps is much more sensible than most of the other ways. The audit team needs to be independent, of course.
- avidphantasm 4y ago“CAPEX budget always increases, OPEX budget always decreases.” is a great synopsis of how capitalism works.
- teddyh 4y agoAs per #53, #37 could probably use a link to https://xkcd.com/1597/ https://xkcd.com/1597/
- olddustytrail 4y agoMany that are absolutely true here. But also one that's flat out wrong: 37. Nobody knows how git works; everybody simply rm -fr && git checkout's periodically. I know how it works. Because the first time I ever heard of git was when I had started a new job and was told the dev team were switching to it. So I spent a couple of days reading up on it and learned exactly how it works. If you work in Ops, I suggest you do your job properly and do likewise.
- kerblang 4y agoI learned how git works but without my cheat sheet and scripts I am a helpless angry baby. (guess where I keep them? git) Still I happily award the Cryptic Weirdo Savant of the Year Trophy to anyone who can convincingly lie about memorizing that gibberish.
- olddustytrail 4y agoYou clearly didn't learn how it works since you're helpless without a cheat sheet. Sounds like you memorized some stuff. I didn't memorize it, I learned how it worked. These are very different things.
- kerblang 4y agoThat was mildly convincing but it's been a fairly competitive year
- olddustytrail 4y agoIt's not because I'm super smart, you're just super lazy.
- AnimalMuppet 4y agoPersonal attacks are against the site rules here.
- 4y ago
- pid-1 4y ago> The source you're looking at is not the code running in production. My first boss favorite words were: check check check check check. That's also the first thing I teach to new engineers: most of your assumptions are completly wrong, double check everything.
- d23 4y agoMost debugging is simply a series of validations of which of your assumptions is incorrect.
- TheDauthi 4y agoI do a lot of mentoring with younger engineers. They're surprised the first few times that when we start debugging a problem, the first thing we do is check all of the things that "have to be working" instead of just trying to dig for a bug right where the error looks like it's coming from.
- encoderer 4y agoAbsence of a signal is itself a signal. I’ve built a business around this one!
- dvtrn 4y agoOnce got a job offer around it (I declined).
- sokoloff 4y ago> 3. The severity of an incident is measured by the number of rules broken in resolving it. We got around this general truth by enumerating the list of people (some by name, some by role) who could declare that a prod issue was emergent enough to authorize any action. We based it on the concept in the aviation law 14 CFR 91.3, in relevant part: “In an in-flight emergency requiring immediate action, the pilot in command may deviate from any rule of this part to the extent required to meet that emergency.“ https://www.law.cornell.edu/cfr/text/14/91.3 https://www.law.cornell.edu/cfr/text/14/91.3 The intent of adding our own version of part a and b of this law (including their origin) was to drive home the sense of responsibility and trust we placed in those employees. It also allows us to pass SOX control audits because our stated policies have a “these people can authorize anything they think is needed”, which means when we do that, we’re still following our controls.
- hackernewds 4y ago> was to drive home the sense of responsibility and trust we placed in those employees. This seems like the opposite of a blameless culture, putting the onus on individuals vs the system that caused/can prevent it
- sokoloff 4y agoDisagree, but it might be because I phrased/explained it badly. There is a set of employees E, who might be the cause of the production problem. There is a set of employees P, who might be in a position to use this emergency authorization. E contains about 1000 employees. P contains about 5 employees (including me, as it happens). Members of E can do anything the normal rules allow during an emergency. Members of P can authorize anything they think is appropriate during an emergency. Only in that limited sense is the onus on them, but that's clearly time-bound to the emergency. P's responsibility is to restore service to our customers, not to worry during the incident about how service was originally lost. Later, in figuring out what happened, we do get to the point of who did what, when, and if we're able, why they thought that was the best course of action. In 19 years here and 17 since I designed this process, I've never seen a production incident nor a mistake in handling one held against a member of E nor P, even though we often learn something from our mistakes leading up and during prod incidents.
- noisy_boy 4y ago> That janky script you put together during the outage -- the one that uses expect(1) and 'ssh -t -t' -- now is the foundation of the entire team's toolchest. The only thing that separates tactical from strategic in an enterprise setting is time. Due to my frustrations with various constraints with running things in parallel (and my complete ignorance of GNU parallel), I wrote a script that does similar work in Perl (with some added custom features we needed). Against all my expectations, 16 years later, that script is still being used (unmodified). > There's nothing wrong with Perl Indeed.
- prosaole 4y ago> and my complete ignorance of GNU parallel [...] 16 years later Parallel was created 2002-01-06: https://www.gnu.org/software/parallel/20th-birthday.html https://www.gnu.org/software/parallel/20th-birthday.html It did, however, only become a GNU tool in 2011, so I think it is completely fair that you were ignorant about it 16 years ago.
- noisy_boy 4y agoAh that is interesting - yeah, we were on Sun Solaris at that time (which AFAIK remember, didn't have the GNU toolkit on it). Thanks for making me feel better about it :)