8 ms·
I expect this post will resonate with a lot of HN readers. Every tech company I’ve worked for has had at least 1 manager who tries to ship features over the wee
by PragmaticPulp 7y ago
I expect this post will resonate with a lot of HN readers. Every tech company I’ve worked for has had at least 1 manager who tries to ship features over the weekend with a ragtag team of developers who don’t understand the system or why that’s a problem.
If they succeeded, the manager would have pointed to the feature as an example of their “hustle” and ability to get things done where others couldn’t.
If they shipped the feature and it crashed the website, the manager would blame the front end team for making a fragile system that couldn’t handle a simple feature.
If they failed or were blocked, they’d point out their working proof-of-concept and blame the front end team for onerous process or politics.
The real litmus test is how the company reacts to that manager after this stunt. If the company sides with the hustle manager, it doesn’t bode well for engineering in the long term. When management rewards shows of effort instead of long-term results and fails to acknowledge negative externalities or selfish managers, you breed more of that behavior.
However, if management sides with engineering and shuts the hustle manager down, you’ve found a good workplace.
- closeparen 7y agoMaking it easy and self service to do things the right way also helps create a cultural expectation that if you think you need to leave the happy path, you’re probably holding it wrong. Or at least, you brought the blockage/expensive coordination problem on yourself.
- raz32dust 7y agoIt's not that black and white though. A startup will need to have a different culture than an established company with thousands of users. What is right for a consumer company is not necessarily right for an enterprise company. Companies need to evolve all the time, and such stories are tipping points where such evolution should happen.
- PragmaticPulp 7y agoStartups aren’t an excuse to get sloppy and break other team’s production environments. If a startup needed to move quickly, they’d ping the relevant parties at the planning phase and get everyone on the same page. I was referring to the archetypal “hustle” managers who deliberately try to do an end-run around other teams for their own personal gain. Startup or enterprise, doesn’t matter. You can’t have management that rewards asymmetrical games that benefit rule-breakers at the expense of everyone else, including the customers.
- raz32dust 7y agoAgree with that. Rules should be agreed and then followed by all. I was talking about the general rule of whether "ship features over the weekend with a ragtag team of developers" is bad.
- brongondwana 7y agoHere's an example of an end-run. I had a 6 month secondment to the USA at a previous job, and I needed a server to do data analysis on. It would have taken over 6 months to get one provisioned for me because of "process". I had a laptop which I brought with me from Australia. It wasn't in the asset register in the USA, so I was entitled to a computer. I ordered the most highly specced desktop build available, put it into the empty cubicle next to mine, and spun it up as a development server. It didn't have backups, but that was OK because I never worked with primary data on it and all my work got committed back to Perforce daily. Strictly it was very much against policy, but policy would have meant I spent 6 months sitting on my hands. My manager "hustled" for me and did an end run around process.
- snovv_crash 7y agoThat's fine because it isn't contagious. If your rulebreaking bites you, the entire company's customer base won't even know.
- marcus_holmes 7y agono no no no. I've seen this a hundred times[0]. That service becomes the source for a management report. That management report contains useful data that the CEO looks at weekly and uses to build his board report. The original Aussie guy leaves, but leaves the laptop behind because it's not his. He also doesn't document it (because that would get him in trouble for running a server on a laptop). ...Years pass... The laptop finally dies. The CEO is furious because he can't create his report. He leans on the IT manager. The IT manager has no freaking idea where this report is coming from or who makes it. They lean on the Support team to find out which server produces this report. The Support team drop everything to work out wtf is going on, because this suddenly became their #1 priority. Eventually, someone finds the decaying husk of the laptop, and works out what's going on. They put together a plan for creating a supported server to do the same thing. It'll take weeks, because they have to provision a server properly through the usual channels. CEO has a rant at the entire IT department for not supporting critical business processes, and not being agile enough to support the business. IT manager takes a beating in the next management reviews. No-one is happy. [0]usually a rogue spreadsheet rather than server. The worst case I saw was an Excel spreadsheet in a business-critical department running on a user machine with a post-it note on it saying "don't turn this machine off". If the logged-in user name wasn't the same as the temp who had originally built it, the spreadsheet refused to work and the department ground to a halt.
- protomyth 7y agoAt the exact moment you take money or personal information from users, you really better have a safety / security first approach and have a defined, repeatable, scheduled, abortable deploy procedure. Its not number of users, its what its going to cost each user when you screw up.
- zild3d 7y ago> Every tech company I’ve worked for has had at least 1 manager who tries to ship features over the weekend with a ragtag team of developers who don’t understand why that’s a bad idea. Maybe I've been in startup land for too long, but seems super normal and fine to ship a feature over the weekend if it goes through the regular CI gates - it's tested, peer reviewed, been QAd in staging, etc. Is this not accepted outside of startups?
- closeparen 7y agoThis handy flowchart hangs on the wall near my company's core infrastructure group [0]. [0] https://www.amazon.com/Should-Deploy-Friday-T-Shirt-Sizes/dp/B07HDTHPMF https://www.amazon.com/Should-Deploy-Friday-T-Shirt-Sizes/dp...
- brongondwana 7y agoThat's almost complete... there's another three question flow which gets to yes. 1) Is stuff really badly broken? 2) Was the really bad breakage introduced recently (this could either be an earlier bad rollout, or it could be external factors changing)? 3) Is this requested deploy either a revert of a recently-made bad change or the minimal possible fix/bandaid? If all three of these are true, then you can do a deploy RIGHT NOW regardless of the calendar. (recently - because if this is something which has been broken for years then it's unlikely that it suddenly became urgent absent external change - which is already carved out above)
- michaelt 7y agoSo what you're saying is, it's acceptable to deploy on Friday at 5pm if and only if you're undoing the damage caused by some chump deploying on Friday at 4:55pm?
- Piskvorrr 7y agoYes. And then to find out why a DEFCON-1 level revert was necessary at all: that's a deploy that you want to be prepared for, but you never want to use.
- ChuckMcM 7y agoEvery company over a certain size has a manager who explicitly puts together a rag tag team of cluelessness in order to "get stuff done" because they don't understand the complexity of the stuff. They interpret push back as a lack of co-operation, rather than the requirement for sanity that it entails. Ideally, once they identify by trying to pull the trigger you can move them out of the company.
- tgsovlerkhgsel 7y agoThe probability of such "get stuff done" goes up the more seemingly-pointless and painful, slow and bureaucratic processes are introduced. Many of these processes may be necessary, but it's also necessary to explain why, and to make them as fast and painless/frictionless as possible - especially as each single process in isolation may seem reasonable, but when stacked on top of each other, the "get stuff done" approach becomes a lot more tempting.
- ChuckMcM 7y agoAbsolutely agree. And it's up to a company to insure that process isn't pointless or obfuscating the reason for its existence. The less clarity there is on the "why" the more creative the management will be. Of course, managers who say "I don't believe that will happen so I'm going to skip this part." should be walked out of the door to their car immediately. :-)
- carlmr 7y ago>especially as each single process in isolation may seem reasonable, but when stacked on top of each other, the "get stuff done" approach becomes a lot more tempting. Process stacking for me is one of the reasons why it's super painful to work in big companies if you want to get something done. As soon as somebody makes a mistake, they will add a little bit of process to ensure that never happens again. Individually, as you point out, that makes sense. But if you have to go through a 1000 item review checklist for a single line of code, then I can assure you that no human will be able to actually think through those 1000 items. But they will go through the motions to satisfy the process. Then because they have the checklist they don't think they have to think about it anymore. They make a mistake. It gets added to the checklist. I experienced situations where a single code change would take at least a month. This lead to people trying to save time on a) tests, b) any kind of refactoring, c) adapting libraries instead of writing your own implementation (because fixing the library would be 2 code changes and not just twice the process effort, but an actual committee had to decide about the library change first.) So a lot of process IMHO is the worst thing you can do for your code quality. Checklists are good, but they should be limited to a manageable number (e.g. 10 items, if you want to add something, you have to remove something less important first). It should also not be harder to do the right thing, e.g. centralizing functionality in libraries should be easier than.
- sitkack 7y agoBTW, the night crew pulling a stunt is how the Chernobyl Disaster happened. https://en.wikipedia.org/wiki/Chernobyl_disaster https://en.wikipedia.org/wiki/Chernobyl_disaster
- corndoge 7y agoNever heard of this before - thanks!
- nothrabannosir 7y agoIf this interests you, there's a show on HBO by the same name that you should give a go. https://www.hbo.com/chernobyl https://www.hbo.com/chernobyl
- rcxdude 7y agoThey weren't exactly pulling a stunt. They were carrying out a test as planned, just off schedule (because of other operational demands) and the key thing is that neither the test planners nor the operators were aware of the design flaw in the reactor (because the government had declared it a state secret).
- rofrol 7y agohttps://en.m.wikipedia.org/wiki/RBMK#High_positive_void_coefficient https://en.m.wikipedia.org/wiki/RBMK#High_positive_void_coef... These analysts say that Soviet authorities appear to recognize that operator errors at the Chernobyl plant on the night of April 25-26 were not the sole cause of the accident, and that technical flaws in the reactor’s design contributed to the worst accident in the 44-year history of nuclear energy. In particular, they said, a distinctive feature of the Chernobyl design, which sets it apart from conventional nuclear power plants in most of the world, is its tendency to generate a sudden and uncontrollable burst of power if large steam bubbles, or “voids,” are allowed to form in the reactor core, as they did before the accident. This peculiarity of the Chernobyl type of graphite reactor, called a positive void effect, is now seen as a decisive factor in the accident, one that transformed successive blunders on the part of Soviet operators over a period of hours into a catastrophe. https://www.latimes.com/archives/la-xpm-1986-08-23-mn-15781-story.html https://www.latimes.com/archives/la-xpm-1986-08-23-mn-15781-...
- ineedasername 7y agoYes, you are correct. But, to play devil's advocate to this, I will also say that there are operations where resistance to any change that will cause more than a trivial amount of work are roadblocked endlessly for no substantive reason except that it will disrupt a cushy routine of just keeping the lights on.
- Cthulhu_ 7y agoYup, and developers looking for shortcuts or how to do things the way they are used to (e.g. calling a database directly). I can recall one instance where a guy was asking for SSH access to the production environments, just so he could look at some env variables and logs. We had the best ops team (IMO) who worked with Principles (they were working towards 99,99999% uptime), which simply excluded SSHing into production servers. The developer was told to add something to his application that logged the environment variable, if that's what he really needed. It's a bit more work but at the same time he had no excuse because new deploys were a matter of minutes. But anyway, the main lesson I learned there is that as an ops team (or broader, as an IT department) you need to have Principles, capital P. A short set of rules and goals which you can always point to. Like the uptime goal, which excludes / includes a LOT of things right off the bat - access controls (nobody can touch production directly), testing practices, application architecture (stateless), etc.
- mnd999 7y agoI don't think asking what environment variables are available for application developers to use on the servers is an unreasonable request. It should probably be part of the platform documentation though. Logs, again there should be a documented safe way to do this (ELK etc.). SSH to production is not the answer though.
- ljm 7y agoI’ve faced this a lot, particularly when the management is a couple of layers away from the team doing the work. It makes me wonder how these organisations don’t collapse under the weight of their ineptitude. Most of the bugs or issues I have to fix are from problems we created by short term hacks. Way beyond simple tech debt. The engineers are as much at fault as the managers, particularly when it comes to introducing insane complexity to the stack to solve simple problems (how many startups seriously need to invest in tech like gRPC or graphql except to gain cool points?). Management, on the other hand, have no empathy for the people doing the work, so quality dips as we are pressured by both self-imposed and external deadlines which are decided with zero input from engineers. Half the time we web engineers are building glorified content management systems with some nice design over the top. It’s boring but it’s not a burn out.
- jariel 7y agoIn my experience at all levels, I would say Engineers generally don't have a very good sense of the big picture and of what really matters, are generally caught up in a lot of details unneeded complexity that doesn't create value in the manner they think it does. I remember as a young Eng. getting caught up in the platform holy wars, and then sitting as a PM looking back on it all like I must have been in some cult. There's truth to the notion that 'it's complicated' and rarely does anything get done in a weekend, but if there is focus, a decent dynamic process, things can move faster. I worked at one company that had a messaging product, it has a big team of Engineers and things were at a snail's pace. I suggested bringing in a few talented people and starting from scratch as a re-factor, they thought I was crazy. A young intern left the team, did it on his own with one other person and met with enormous success. The company, even after literally watching an intern out-do them never changed. Both the old company and the new company are big enough names you've all heard of them I wish I was at liberty to share. In another project, we were opening up some basic APIs. We did some work with Facebook and they were able to give us a custom API in literally a few days. Our own, simple APIs took 18 months to deliver. The weekly product teams consisted of 10 people rambling on - and the two most important people, the dudes actually writing the code, were not ever present. It was a colossal and shameful waste. Even though getting a rag-tag bunch of Engineers over the weekend is usually not a good sign (it might actually work for some marathon bug fixes or something), I'm usually sympathetic to the cause.