10 ms·
Trying to sneak in a sketchy .so over the weekend
- nojvek 7y agoIt’s about process + team and not about individuals. Far too many times I’ve seen companies over praise/reward certain individuals at the expense of demotivating the team around them that makes things possible. I’ve found a lot of things solved by a decent review and deploy process. My precious company just went through SOC2 certification. SOC2 is mostly about change management. Who can make changes to prod? Is every change approved by the right personnel? Is every change auditable? Etc Mostly boils down to using git(hub) properly and having a sane deploy/monitoring/revert system. Doesn’t matter if you’re the CEO or a VP or a yolo manager. You can’t deploy things without oversight. If you want to deploy on the weekend you need approval from the opslead for that system who is responsible for keeping things alive. On the weekend, no one should be working, even opsleads should only work if they are paged. Only broken things that really affect customers are fixed on weekends. No new features. Period.
- PragmaticPulp 7y agoI expect this post will resonate with a lot of HN readers. Every tech company I’ve worked for has had at least 1 manager who tries to ship features over the weekend with a ragtag team of developers who don’t understand the system or why that’s a problem. If they succeeded, the manager would have pointed to the feature as an example of their “hustle” and ability to get things done where others couldn’t. If they shipped the feature and it crashed the website, the manager would blame the front end team for making a fragile system that couldn’t handle a simple feature. If they failed or were blocked, they’d point out their working proof-of-concept and blame the front end team for onerous process or politics. The real litmus test is how the company reacts to that manager after this stunt. If the company sides with the hustle manager, it doesn’t bode well for engineering in the long term. When management rewards shows of effort instead of long-term results and fails to acknowledge negative externalities or selfish managers, you breed more of that behavior. However, if management sides with engineering and shuts the hustle manager down, you’ve found a good workplace.
- closeparen 7y agoMaking it easy and self service to do things the right way also helps create a cultural expectation that if you think you need to leave the happy path, you’re probably holding it wrong. Or at least, you brought the blockage/expensive coordination problem on yourself.
- raz32dust 7y agoIt's not that black and white though. A startup will need to have a different culture than an established company with thousands of users. What is right for a consumer company is not necessarily right for an enterprise company. Companies need to evolve all the time, and such stories are tipping points where such evolution should happen.
- PragmaticPulp 7y agoStartups aren’t an excuse to get sloppy and break other team’s production environments. If a startup needed to move quickly, they’d ping the relevant parties at the planning phase and get everyone on the same page. I was referring to the archetypal “hustle” managers who deliberately try to do an end-run around other teams for their own personal gain. Startup or enterprise, doesn’t matter. You can’t have management that rewards asymmetrical games that benefit rule-breakers at the expense of everyone else, including the customers.
- raz32dust 7y agoAgree with that. Rules should be agreed and then followed by all. I was talking about the general rule of whether "ship features over the weekend with a ragtag team of developers" is bad.
- brongondwana 7y agoHere's an example of an end-run. I had a 6 month secondment to the USA at a previous job, and I needed a server to do data analysis on. It would have taken over 6 months to get one provisioned for me because of "process". I had a laptop which I brought with me from Australia. It wasn't in the asset register in the USA, so I was entitled to a computer. I ordered the most highly specced desktop build available, put it into the empty cubicle next to mine, and spun it up as a development server. It didn't have backups, but that was OK because I never worked with primary data on it and all my work got committed back to Perforce daily. Strictly it was very much against policy, but policy would have meant I spent 6 months sitting on my hands. My manager "hustled" for me and did an end run around process.
- iamleppert 7y agoJust reading that, the tone of this person made me cringe. I definitely would never want to work with him/her. I can feel the negativity and attitude in almost every sentence. The right thing to do would be to help this person achieve their goals while maintaining reliability of the operation. Calling a co-worker who is just trying to get something done a rando is the epitome of toxicity. She wouldn't last 5 minutes with that attitude in my organization.
- tonyhb 7y agoYeah. The author has a good point about _the specific way the co-worker went about things_ being wrong. But there was also a lack of understanding to explain things. I'm assuming we, the blog audience, had a better explanation for "no, just no" than the co-worker did. See another post by this author for a similar tone: https://news.ycombinator.com/item?id=22277582 https://news.ycombinator.com/item?id=22277582. Maybe work right now is tough. It sounds that way. All that said, a solid takeaway is that sometimes people are naive and you need to say no. The next step is: alongside saying no, ask what their aim is. You're saying no to the solution and you don't know the problem.
- bluedino 7y agoIt sounds like the person was told to stop and directed to the people to discuss the correct solution with. And then it sounds like the person did it anyway, at a later date, and broke shit. If that person repeated that behavior it is someone I wouldn't want on the team, or at least having access to certain systems.
- HarryHirsch 7y agoDon't know about the tone. What worries me is the company. There's no procedure for updates that have far-reaching ramifications? Yeah, can see why that would wear you down.
- nbm 7y agoThere usually are procedures. Doesn’t stop people trying to work around them anywhere I’ve heard of. And I’ve yet to find any set of procedures that cover all possible cases. So you still need people to apply good judgement when the procedures don’t cover situations, or if the procedures currently would allow something that shouldn’t be allowed. Like: “no, even though we have a way to safely roll new libraries out, we won’t allow you to add this library to all front-ends, you should follow best practice and use ...”
- frequentnapper 7y agowho is this person and why am I seeing blog posts of theirs on hn front page two days in a row?
- NikolaeVarius 7y agolmgtfy https://news.ycombinator.com/item?id=13401293 https://news.ycombinator.com/item?id=13401293
- EdwardDiego 7y agoShe's an experienced SRE/sysadmin who writes a really interesting blog, I highly recommend you check the archives. :)
- sdan 7y agoWhat just happened? I tried reading it but felt like I wasn’t reading English... when this is clearly in English. Something about a customer coming into the chat room on a Sunday asking to implement a new transport feature... what?
- monkeydreams 7y agoYes. This post relies on the reader knowing some background to the services/business RbtB supports. She is in the top comments on HN relatively often so it probably made more sense to others more familiar.
- rcfox 7y agoI don't know anything about the author, but it read just fine for me. Someone else criticized the style of the writing, but I found that it was like telling a story in the way you would naturally do it face to face. I would invite anyone who is confused about a part of the story to ask for clarification on jargon or situations.
- stordoff 7y agoFWIW I don't know anything about the author, other than having scanned a couple of articles from time to time (I don't recall anything I couldn't glean from reading this article), and it read fine (and was interesting to boot).
- deleted 7y ago[deleted]
- edoceo 7y agoNot a customer. A junior dev trying to ship untested code to production on a day off. Then later that low quality code borked an internal system.
- furyofantares 7y agoWe can certainly hope they were junior but I don't believe the post ever says so. Also I believe there were two separate people trying to ship the thing.
- azernik 7y ago"There's a time and a place to dig in your heels and say NO to something. It usually comes when someone else has been reckless. If you do it for everything, then either you're the problem, or you're surrounded by recklessness. I think you can figure out what to do then." The magic is in telling the difference.
- __blockcipher__ 7y agoWasn’t her implicit point that you wouldn’t need to know which it is, because the answer to both (“you’re the problem” vs widespread recklessness) is to leave?
- azernik 7y agoIn the former case, the answer is to change your own behavior. The sentence structure is a bit confusing, but I interpreted that last clause as referring only to the latter case (surrounded by recklessness).
- HarryHirsch 7y agoFrom her other posts on her blog, it's clear that she means that her managers consider her a problem because when she predicts trouble she is usually right. When you continuously play Cassandra it gets tiring for everyone involved, Cassandra, her manager and the audience viewing the spectacle. Time to find a new job.
- marcinzm 7y agoThe flip question is always, how often was Cassandra wrong? We're after all seeing one side here. A perfect Cassandra is potentially useful, a trigger happy Cassandra is just annoying. Even a perfect Cassandra can be annoying if the issues in the end cause minimal monetary fallout.
- opless 7y agoThere is a rule of thumb to not ship anything on a Friday. This was a Sunday! And it didn’t seem to have been tested, so where was the Dev/UAT/Prod separation? So many red flags, and I think anyone who has been in development for a while has had some product owner try to break procedures to force whatever pet project that is falling behind to hit some target to get their bonus/gain brownie points. It almost never goes well. But by this time the PO has a scapegoat because it’s no longer their fault it’s not working, it’s production’s fault. (Also it’s not always a PO, it can be a developer or anyone up to c-suite level) Sigh
- techntoke 7y agoI don't understand the writing style. The situations seem to be made up with a level of detail that lacks any real technical depth. The whole story could have been summed up into a single paragraph and would have been equally as informative and enjoyable.
- SpicyLemonZest 7y agoThe article goes into quite a bit of technical depth. I don’t mean this in a negative way, but you may be lacking experience in the areas of work that would contextualize what she’s saying.
- guug 7y agoI would say that I am adequately experienced, and I also share GP's opinion. I don't think the lack of experience is the issue here.
- SpicyLemonZest 7y agoThe article describes in detail the actions the rollout team was trying to perform, the new communication protocol they were introducing, and the timing characteristics of their backend which caused the new protocol to be problematic. I'm not sure how you could get more deeply technical. I guess in principle you could say something like "at 9:52 our monitoring showed error rates above 1%", but who remembers things that specifically?
- GavinMcG 7y agoI think most people would find a single paragraph less enjoyable. Most people enjoy stories, and find them more memorable and impactful than data.
- guug 7y agoMost isn't the word I would choose when I consider the audience of this article.
- 7ewis 7y agoWas half expecting these people to be rogue actors trying to install some malware into prod. Glad I was wrong!
- cortesoft 7y agoIt is really better if they are coworkers trying to install malware?
- lmkg 7y ago"Sufficiently advanced incompetence is indistinguishable from malware."
- NikolaeVarius 7y agoI remember an instance where someone updated and deployed the strong_passwords ruby gem since "it had an update". This was more or less right after it was found out that the v0.7 gem was compromised. Cue rushing at the commit and double checking that it was using the "fixed" 0.8 and not the 0.7. Developer had no idea that 0.7 was compromised, and just updated the gem because they could without looking at patch notes.
- toyg 7y agoIs this yours? Sounds perfect. Sounds even better (and more malicious) reversed: "Sufficiently advanced malware is indistinguishable from incompetence". All those debug features "mistakenly" shipped in hardware...
- opless 7y agoSketchy .so acted like a denial of service. That’s malware in my book.
- AnimalMuppet 7y agoIt's accidental malware. That's better, if you're an ethicist, and no better at all, if you're the one having to clean up the mess.
- snlnspc 7y agoThis comment thread originally consisted of one single reply by someone who, apparently, was not aware of Rachel by the bay and her wonderful blog. That post concluded that calling a fellow worker a "rando" was toxic, and that they wouldn't want to work with the author. While I agree with Rachel here at a high level, and have been a dedicated follower of her blog, I completely agreed with that comment. You shouldn't be shipping things in the manner described in this post, and you shouldn't be considering your coworker "some rando" and looking down on them for not having the same schedule as you.
- GavinMcG 7y agoPeople are free to project "looking down on" or contempt onto the phrase "some random person" and the (subsequent) use of "rando" but I felt like it communicated precisely what was going on: this is a large organization, and the person is totally unknown to this sysadmin. Even aside from whether "rando" is contemptuous, the issue isn't that they don't have the same schedule: it's that they are not respecting the company-wide schedule, nor are they respecting fairly obvious norms of professional software development. I'm a teacher, and I very much believe in educating people rather than putting them down, but jumping on a single phrase/term when there's nothing else to suggest contempt here strikes me as odd. It's especially odd when the entire culture of sysadminship has a reputation of eye-rolling and begrudging wizardry to protect users from themselves.
- SpicyLemonZest 7y agoI think there's a legitimate development culture gap. I've had good friends tell me that, at their company, it's totally acceptable to roll things out on a day's notice and you're expected to help make that happen unless you have something more urgent to do. I doubt that's a functional culture for any but the smallest companies, but that still means there are some places where it'd be unacceptably rude to shove someone off and not help them release their code.
- snlnspc 7y agoI think this is a fair point, and I cannot disagree with what you've said. But, I must reiterate, the tone of this piece changes dramatically if "some random person" becomes "a junior developer". As far as the company wide/normal schedule goes, at my previous place of employment, major changes were routinely performed (by me) off hours on a Sunday with only relevant personnel on hand. This was primarily for B2B reasons where the vast majority of our clients were doing mission critical things from Monday to Friday. I don't feel that this was the case in these circumstances, which is why I completely agree with what was really said here, and with your reply was well. I suppose my personal experience in this industry leads me to believe that small snipes like that uncover much deeper contempt than is revealed on the surface.
- enchiridion 7y agoWhy are the comments here so hostile? Seems awfully critical, even for hn.
- zmj 7y agoMisogyny.
- dijit 7y agoI (cynically) believe it's due to very low attention span. At least, I'm guilty of that, I skim articles and read comments. But this post is not skim-able(?) and thus feels a lot more dense than usual. However, I went back and read the whole thing and it's certainly worth the read.
- rachelbythebay 7y agoPretty sure it’s the domain name. Also the time of day/day of week. But mostly the domain name.
- edoceo 7y agoWhy? Are you not by The Bay anymore?
- wgerard 7y agoFor what it's worth, I personally thoroughly enjoy reading your articles. I wouldn't consider myself adept at operations by any means, but your articles are extremely approachable and I always enjoy reading them. I always click on the article when I see it's from you.
- thequux 7y agoThat's really a shame. Of all the colleagues I've had over the years, you still stand out as a beacon of competence, and I learned a ton from both working with you and reading your blog in the years since. Your haters have no idea what they're missing out on :-(
- xenadu02 7y ago
- z3t4 7y agoThe interesting question is why the ODBC driver needed to be updated and why it was such a hurry. Probably because he/she wanted to see if it worked before business opening... And when he/she was not allow to do that maintenance on off hours, then he/she tested it on a less important internal system. The thing with ODBC drivers is that they deal with things like bigin support, which cover edge cases that you only see in production.
- rachelbythebay 7y agoODBC was not used anywhere prior to this point. The company had its own hardened transports. The frontend systems, taken as a whole, amounted to a bitchin’ botnet. They’d destroy ordinary services without even working up a sweat. Using the right mechanisms for comms between them was in everyone’s best interest.
- asdfasgasdgasdg 7y agoOdbc/SQL newb. I know what begin does, but I don't know why you'd only see it in prod. Surely you should test the same codepaths in staging that you plan to execute in prod?
- NikolaeVarius 7y agoGenerally speaking from experience, most non-production facing testing does not accurately reflect what is actually happening in production in terms of actual queries run and volume of it. A codepath which is perfectly fine in testing/staging could kneel over immediately against a production workload,
- mbreese 7y ago"Quantity has a quality all its own." - Someone (I heard it attributed to Napoleon, but have seen it attributed to others) In this case, it's very true -- workloads are really difficult to properly test with respect to production load. A code path that works well in one test can easily buckle under the pressure of a production workload. Stress testing is hard.
- arctangent 7y agoThis is why we have release management processes https://en.wikipedia.org/wiki/Release_management https://en.wikipedia.org/wiki/Release_management
- chapium 7y ago> Several hours later, I guess I wandered by the monitor, and since nobody else had said anything, I decided to jump in. > no, just no. post to (group) for a proper conversation, but I'll warn you right now: adding to stuff to [the entire frontend of the whole business] is rarely the answer. Looks like a bit of seagull management to me.
- rhizome 7y agoNot seagull, because the effect is dump-avoidance. Outside of electrical-fire-in-hair moments, going slow is a virtue. There was no apparent reason to do this specific thing at that time, nor even (evidently) within an identifiable timeframe in the future.
- mateo411 7y agoIsn't seagull management where you fly in, shit over everything, then leave. You kind of need to be VP level to pull that off.
- chapium 7y agoI suppose my view of it is, I see a Sysadmin admittedly monitoring a situation unfolding that they have no take in then dumps on the situation. I've never really cared for this style of tech management. What I feel really should have gone down is a call to discuss what is going down (by all means, slay the dragons). Outright stating that this "isn't happening on my watch" or whatever is really counter productive because the employees trying to execute this change surely didn't volunteer on a weekend to be doing it. Process has already been breached and they are in a sticky situation. Now they have a hostile sysadmin pushing on them rather than whoever conceived this half-cocked plan.
- hyperpape 7y agoFunny that everyone leapt to the conclusion that it was a junior developer. I immediately thought of a senior developer who "gets things done" and gets rewarded for that, but leaves a flaming pile of rubble in their wake. I've seen bad bugs written by junior developers, but I've spent more time cleaning up issues introduced by experienced people for whom time had just reinforced bad habits. I don't know which is more common in general, that's just my personal history. Who knows which it is. The article says they haven't touched the frontend servers yet, not that they haven't done anything substantial yet, so I don't think we can know unless Rachel decides to comment on that point.
- PragmaticPulp 7y agoThat was my first assumption. These dangerous situations come up when a manager or developer think they have an asymmetric opportunity: Ship this feature quickly and collect all the credit, or crash the website in the process and blame the website team.
- ObsoleteNerd 7y agoThe worst experiences I ever had at a job were due to the most senior developer there. He was basically given this god-like status from management where he could do no wrong, and had free reign of all systems, because he'd been there for so long.. Yes, he was an incredibly capable software developer, but that had led to an ego issue where he thought he knew EVERYTHING, not just developing in his particular stack... so he'd find his way into various systems (software and hardware) he had no experience in, and assume he could just figure it out. We had half a dozen full-on-disaster level situations in the couple years I was there thanks to this 1 person. Each time it was shrugged off as "an accident" because heaven forbid you actually upset him or hold him accountable.
- pnako 7y agoI've observed that pattern with developers who are relatively senior, and have been in the same, smallish place since they graduated. Basically they had no technical feedback ever in their career. Some developers put their crazy ideas and experiments on github; some developers put those in production.
- ATsch 7y ago> That's why some clever person had added a client-side "gate" to the transport. If it saw that too many requests were timing out to some service, they'd just start failing all of them without waiting I believe this is commonly called the "circuit breaker pattern"
- Rapzid 7y agoSi. If you already have a distributed rate-limiting service, it could be used to create a more intelligent circuit-breaking service as well I would imagine..
- texthompson 7y agoOne of the funniest stories I've ever heard was about how a junior developer asked a more senior developer a creatively terrifying question: How do I install half of an RPM?
- WrtCdEvrydy 7y ago> half of an RPM Well, I'm stumped... I wonder if he just needed one tool out of suite?
- Const-me 7y agoMaybe trying to workaround a dependency issue. Not sure what's in rpms, but in deb the manifest may require libsomething version exactly 1.1, will fail if something else requires libsomething >= 1.2.
- ars 7y ago1 revolution per 30 seconds maybe?
- jwilk 7y agoThat would be 2 RPMs. Still easy to install!
- oblio 7y agoFrom what I can tell, with cpio :-D
- deleted 7y ago[deleted]
- hyfgfh 7y agoHelooo 1990 ``` body { width: 600px; ... } ```
- iamleppert 7y agoCalling your co-worker a rando? Very unprofessional. Why not teach people the proper way to release things to production and avoid the name calling and negativity?
- JasonFruit 7y agoFor once, she's not being unprofessional. Give her credit for that much.
- shrimpx 7y agoI agree that it might be offensive to the actual person who is being singled out in this article, if they’re reading the article. It also occurred to me that this person is fictitious or semi fictitious in which case it’s not just inoffensive, but accurate, to call them a rando.
- technion 7y agoI need blogs like this to remind myself that there in fact are actually sane organisations in the world. It's a glimmering ray of hope against my own experiences, which more and more would suggest the writers pushback in that chat room would just get them considered a problem employee.
- saber6 7y agoCan we please stop spamming rachelbythebay stories?
- EdwardDiego 7y agoWhy? She writes good content.
- tialaramex 7y agoWeek patterns have a long history, and humans have to rest some point, so I respect the "Nothing ships on Friday [and implicitly Saturday or Sunday]" plan outside of very young (often self-funded) startups where basically all that the half a dozen people doing it are thinking about 24/7 is that one project so there aren't really "weekends" anyway at first. However, bigger organisations can get really out of hand with this. They start out with small carve outs, oh, nobody really works from the start of Christmas week until the first full week of the New Year. Soon it's from Black Friday although good luck scheduling anything the week before that, and so before you know it they're declaring that nothing can even be planned for the months of November, December and January each year. Now, if you really must have this habit, which I don't recommend, what you need to understand, just as much as the "No shipping on Friday" folks is what this does to your perception of time. If you have "Don't ship Friday" then the question in the Tuesday meeting "Will it be live on Monday?" must be re-phrased "Is it ready to ship by Thursday?" and not every manager seems to understand that. But when you have "Nothing can be scheduled for our 90 day Winter Risk period" as policy then the question in that meeting "Will we be ready for the January 31st legal deadline?" is actually "Are we 100% sure we'll have fixed this by the end of October this year?" and I have yet to see anyone who recognises that despite having a policy which makes it so.
- ineedasername 7y agoIn the early 00's I had to work with a company that setup payment portals. My workplace already used them for existing portals. We needed another for a separate service. It took a few days to get the paperwork signed for the banking end of things, and then we were told by the portal people "sorry, yesterday was Tuesday. We only go live with changes on the first Tuesday of every month."
- tim58 7y agoI'd like to provide a counter-argument. Every software release includes risk. There's risk that there are bugs. There's risk that it will introduce regressions. There's risk it will cause a major crash. We can do things that reduce the risk. We can have recovery plans for negative outcomes. We cannot eliminate all risk. The reason businesses have black-out deploy dates is so that they can control the risk. Deploying code 24 hours before a critical business day, like black friday is for some businesses, is an entirely different risk/reward ratio and should be treated different. Even 90 day risk periods can be valid for some businesses, for example seasonal businesses during peak season.
- neya 7y agoOn another note, I hate posts from Rachel. My opinion is from reading a dozen posts from her blog by now. It's usually about complaining of fellow colleagues and her employers. I would really hate to be her colleague. I've never seen a post where she admitted she fucked up. It's always about how everyone else is awful and how she is the victim in all this. I really hate this toxic attitude and I would really hate to have a toxic colleague who instead of grabbing her teammates for coffee and giving them constructive feedback (and keeping company events confidential), shames her colleagues publicly.
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- quadrifoliate 7y agoI disagree 100%. I have read quite a few posts on her blog as well. Most of them are purely technical; and the failures described there are mostly "Hey cool, this failed" and could be attributed to any person including herself. In the ones where she specifically blames other people or companies, I mostly see a competent, reasonable person who is fighting an uphill battle against unreasonable management, which is a trope among software developers for a reason. Like this one for example -- do you really think adding (effectively) backdoors for your special service on a Sunday evening constitutes sustainable software development? I'd rather have that "toxic attitude" than some meaningless sunshine-out-the-bottom screed about "team players who get things done".
- ThrowawayR2 7y ago> "On another note, I hate posts from Rachel." I don't but then again I spent years on call for a couple of critical services. SREs / sysadmins are the people who have to work overtime on evenings or weekends to restore services when an incident causes downtime, whether accidental or caused by foul-ups by the development team. After a few years of that, having a rather low opinion of colleagues whose routine bad practices or behavior risks downtime for critical services is more than forgivable in my book.
- jrockway 7y agoSort of tangential to the article, but... Setting aggressive HTTP timeouts can be quite informative in finding "interesting" HTTP endpoints. Set them, enable tracing, and search for timeouts every morning. At my last job, I found an interesting "charge_cards" endpoint and learned that someone logs in every day, clicks "charge cards" and all billing is handled in the frontend web process instead of the event queue. It was idempotent so it wasn't a problem (just click reload until it returns), but was an interesting bit of visibility into a large system. Knowing that that's how it works makes it less surprising when someone wonders "I noticed cards didn't get charged". Is the person who normally does that on vacation? Aha, that's why. This never caused us any problems so we didn't rewrite it, but it's always nice to have weirdness exposed front and center so you are less surprised when it starts causing an issue. It was also interesting to read about a wide deployment of circuit breaking. That's a relatively new concept to me (never ran into it at Google, for whatever reason), but it's another way to prevent weird outages. Last week I upgraded something in my k8s cluster and the API server stopped responding. With the API server unresponsive, I couldn't kill the workload (or even determine if that's what was causing it). I had to kill all the nodes before I could get back in. If the job were communicating to the API server through a circuit breaker, it would have popped after requests started taking too long, and I would have been able to just kill the job instead of having downtime. But, I guess it's still a relatively esoteric concept. I know this article is about not doing dumb things in production. I just assume those are a given. I'm more interested in the nuts and bolts of day-to-day operation ;)
- rawoke083600 7y agoMy ex-boss use to say... "Don't you like your weekends ?" whenever somebody wanted to deploy something on a Friday :D
- crtlaltdel 7y agoha! last year i was at a shop that had a dev manager who snuck in a stored procedure into a prod db. the arch called for an instance per customer, with each using the same sprocs. when one customer suddenly had prod fails it caused all the teams who had pushed code to burn cycles tracing the issue (the effects were system-wide and this was the biggest customer). we diffed the sprocs and saw what was up. the correct code was restored and the problem got fixed. in the end the dev manager in question didn’t have much of a reason for making this change without telling anyone.
- klausjensen 7y ago...I'm probably missing something obvious, but what is ".so"? I have read the post, btw (liked it - I usually like Rachel's posts) - but still don't understand what .so is.
- wickedwiesel 7y agoIn Unix these are shared object libraries. In this context, I believe .so is the file extension for extensions that can be loaded by web servers during runtime. I think the person in the story was trying to change the Apache2 configuration of all FE servers to load the wanted client library / .so file. - https://httpd.apache.org/docs/2.4/mod/mod_so.html https://httpd.apache.org/docs/2.4/mod/mod_so.html
- quibono 7y agoI think it's the shared library file that the person was trying to add to the frontend
- fock 7y ago`find /usr -name '*.so'` that's enough for you ;)
- mrunkel 7y agoit's a shared object file. Basically a pre-compiled set of functions. For example libc is a .so file. In windows, think .DLL.
- deleted 7y ago[deleted]
- darepublic 7y agoBringing up problems only shortly before launch sounds damn familiar