11 ms·

It's Not Sabotage, They're Drowning

koz_ 7y ago
This allergy to reality is surely the biggest trap not just for startups but people in general. It's surprising but I feel like the norm is for people to care far more about feeling that everything's okay, rather than everything actually being okay. That said, one problem I see with the anecdotes provided is that the problem solver is doing two things: discovering a problem and solving it. Conflating these two things is a great way to encourage pushback: every change comes with an associated risk, so the general rule is to say no (using whatever flimsy reason) if it's not well understood and agreed that the problem being addressed is a real problem worth solving.
- conradp 7y ago
  > ... the problem solver is doing two things: discovering a problem and solving it. Conflating these two things is a great way to encourage pushback This squares with my experience strongly. At a prior job, I remember two or three times where a set of changes was almost rejected for production because despite solving real bugs. The bugs hadn't affected anything in production yet, but the fixes carried the risk of maybe breaking something. This resulted in some fun times, like a particular client running something like three major versions behind on our software, precisely because management didn't want to accept any disruption risks there, and the old version grumbled along well enough.
Ensorceled 7y ago
Reminds me of this scene from Frank Herbert's Dune: “Because of an observation made by my father at the time. He said the drowning man who climbs on your shoulders to save himself is understandable– except when you see it happen in the drawing room.” Paul hesitated just long enough for the banker to see the point coming, then: “And, I should add, except when you see it at the dinner table.”
ltbarcly3 7y ago
I think saying 'they're drowning' is still giving them too much credit. People who built terrible broken systems don't know they are bad at building things, and they probably don't have a clear idea of what would be better or worse to begin with. They don't know it's better to have a system that can be debugged, they don't even know what the difference is. If you give them examples of good designs and bad designs, they can't tell you which is which. When they say 'this is good' or 'this is bad' they will astound you with the reasoning. Software is an industry with no standards for employment. All it takes to be a Software Engineer is for an incompetent manager to hire you. More than half the people I interview barely have a basic familiarity with writing software. These are people who have been employed as software engineers for years and will certainly find another job. They struggle to write a nested for loop. They write functions with thousands of lines. They just don't know what they are doing. The author here claims they don't want the visibility of the situation to increase, but I think they don't know it's a problem to begin with. Their stupid excuses for not moving forward are just their stupid thoughts said out loud.
- vanniv 7y ago
  My team has built garbage software at times. It isn't because we don't know how to build good software, and it isn't because we don't care, and it isn't because we don't want to build good software. It is usually because of a continuous sequence of emergencies to which we are forced to react. An emergency occurs, and we have requirements to do something in 1/3 the necessary time. We get 80% of the way though that and, before launch, there's a new emergency, and we have to rush to that. So we end up with crap software that looks like shit and only almost works. But it almost works enough to keep our very large business humming along. Sure, we hate it, and we're making more technical debt, and over time, we get less effective. But sometimes we go through cycles of a year or two when the business either can't afford to do it right or isn't willing to pay the price to do it right. Sometimes, they really are willing to accept paying 10x as much so long as they can pay in a year or two. (I hate it, but it is true)
  - AstralStorm 7y ago
    Make it 100x as much with 5 times as many people in 10 times as long. Efficiency is never important when there is money to burn and it's not yours, and you won't be blamed when it breaks. It does not even take any emergency - often it is conscious or subconscious project management strategy. It's related to observations about Peter principle, Dilbert principle, Gervais principle and office politics and dynamics. Avoiding this tendency is essentially only possible with small teams with full ownership of the problem they're working on and direct stakes.
lowercased 7y ago
> I refactored the code from untested and untestable, to testable with 40% test coverage. The senior architect is refusing to merge because the test coverage is to low. what's likely even more frustrating is that other merges went in for years with no tests or testing in place at all.
- umvi 7y ago
  Seems like unit tests are only really valuable for library code. Application/glue code... not so much, especially if it is something being actively developed and you are constantly throwing away tons of unit tests.
  - AstralStorm 7y ago
    Totally untrue. Unit tests in strict sense, perhaps, but nobody writes true unit tests - there's a high chance these are actually short range functional or integration tests. Such tests are of high value.
    - umvi 7y ago
      Maybe it just depends on the context. I write embedded linux applications. One application I wrote talks to a Qualcomm modem using their QMI framework and then passes on messages to an antenna subsystem. I wrote some library functions, which were easily unit testable, then I wrote the glue code, which I found wasn't very testable at all. I found myself having to stub out the entire QMI Framework/modem and the entire Antenna subsystem API since obviously I don't have access to those things at build time. So basically now my application/glue code is being testing in a fake vacuum that may not even reflect the real behaviors of the modem or antenna. So I waste tons of time trying to predict how the modem and antenna might behave, and test my application code accordingly. Lastly, I end up scrapping a bunch of the tests because the actual behavior of the modem/antenna is different than I predicted and changes slightly with each update.
  - larzang 7y ago
    The weakest link in your software is always you. Automation that can help compensate for human failings always has value. I can't count the number of times writing tests has led me to catch simple mistakes in code that I'd stared at for hours and declared good. Not to even get into the ways that tests can help guide sensible architecture, especially for junior devs, because "this is hard to test" is the single most straightforward code smell for "this probably needs to be simplified or broken up".
mirimir 7y ago
When I was trained in lifesaving, one strategy for panicked victims, who might drown the rescuer, was just to give them some space to drown enough to lose consciousness. Then they were much easier to rescue.
- wheelerwj 7y ago
  it doesn’t usually apply to drowning victims, but more broadly its applied when someone is choking in general. if someone is refusing help, you wait until they pass out, and it becomes implied consent. I assume in this example, that would be firing or otherwise being told by your supervisor to take the help.
  - kevingadd 7y ago
    Consent isn't necessarily the issue in drowning individuals, since the instinctive drowning response can make it dangerous to save them even if they're not about to tell you 'let me drown'. Maybe actually a good comparison for software teams that refuse to make incremental improvements out of fear or panic... it's not that they don't want things to be better or refusing help, necessarily - they're just terrified of the consequences if the improvements go wrong.
Aeolun 7y ago
So being the person that’s manning the lifeboat, what do you do? Kick them down? If you do nothing everyone will surely drown.
stickfigure 7y ago
I can perhaps understand where this one is coming from: I refactored the code from untested and untestable, to testable with 40% test coverage. The senior architect is refusing to merge because the test coverage is to low. You have a piece of software running in production; even if it's poorly written and bereft of tests, your customers are satisfied. Customers (unwittingly) have already vetted this version! A rewrite, even with brilliant design and fantastic test coverage, is still more of a risk than fixing what isn't broken. This isn't to say that you shouldn't rewrite code, especially if it makes other improvements possible. But don't discount how high a bar new code has to reach to measure up with "it already works". Maybe 40% is still too risky.
- AstralStorm 7y ago
  It's always a lie, about running in production. I've personally seen untested code allegedly running but it isn't actually working. Defect numbers through the roof and test harness literally not being merged exactly because of this issue. You will get visibility, either by customer complaints, slowing down development speed and ultimately projects being shut down. It's not sabotage and it's not exactly drowning. It's willful attempt to ignore reality and cheat on metrics, prolonging the inevitable and defrauding stakeholders.
  - BigJono 7y ago
    It's always a lie? You're basically arguing that code with bad test coverage never works. Production code is copping constant manual end-to-end tests. Those tests are more valuable than most unit testing, and some automated black box testing. I'll take stable production system over the unit tested new code any day of the week.
    - peterkelly 7y ago
      That's the kind of thinking that leads to situations like this: https://www.digitaltrends.com/mobile/samsung-galaxy-s10-ultrasonic-fingerprint-scanner-fooled-by-screen-protector/ https://www.digitaltrends.com/mobile/samsung-galaxy-s10-ultr... A lot of the time it's just as important to make sure certain things can't work as making sure that certain other things will.
CoolGuySteve 7y ago
Nobody’s better at letting better be the enemy of good than software engineers. For some reason our culture likes to argue and nitpick over the most mundane stuff, usually missing the forest for the trees. I’ve had jobs where the manpower spent just arguing outweighed the cost of the change significantly. One dickhead manager I had spent weeks arguing over the acquisition of a $500 SSD. It made me want to slap $500 down on the table just so he’d shut the fuck up already. The only way I know to counteract it is by saying “don’t let better be the enemy of good”.
- brain5ide 7y ago
  A few points: 1. Computing and engineering in general is such a discipine where one nitpicky detail can make all the difference. One bit flip, one keystone, one badly selected material. However, the conduct of negotiation, could be improved as always, but it's a general listening problem most of the time. 2. From the dickheads point or view, everything can be solved by spending more money on HW. Each unexpected spend like this is a risk to open the floodgates and lose both financial and engineering discipline by setting a bad precedent. Not sure you want easy handed decisions there as well.
  - biggestdecision 7y ago
    It makes no sense to cheap out on hardware though. It's such a tiny cost compared to the overall cost of the employee, you are wasting money if inadequate hardware is making your employees less productive.
    - retSava 7y ago
      In one sense, it doesn't make sense. The productivity boost would likely be worth it alone. From another perspective, this is a +500$ post in the cost column of that unit (and perhaps n*500 if the whole unit then comes asking for the same thing). The cost of spending time arguing is baked into the cost of salary, meaning it doesn't show since it's the same regardless. It's a form of optimization, but the optimized unit is not productivity. A friend of mine was working as a recruiter where the whole company was sitting all day long at their laptops. He was denied buying an external screen to be more efficient since he then could have more information on the screen (eg an application next to a browser), "because then everybody wants one", even buying it himself. edit: nb I'm not arguing for not buying the ssd/screen.
dredmorbius 7y ago
A possible way to sell instrumentation / metering is to build your own scaffolding, find a problem using it, and sell both the scaffolding and the solution it's provided. As the article notes, metering by itself simply illuminates problems. For the drowning team (or individual), that simply delivers bad news. If you can show that the effort actually delivers results, you've got a way to sell the instrumentation as well. In more pathological cases, you don't reveal the scaffolding, but start cranking out code fixes based on it. You may have the opportunity to sell the metering later, when you're asked how you're finding and fixing the problems. Rescuing projets from antipatterns is painful at best. Enlightenment is a painful process. It peels away comforting lies to reveal unpleasant truths.
deboflo 7y ago
This is one of those things you don’t truly understand until you’ve experienced it first hand. I have. Very appropriate metaphors.
peter_retief 7y ago
Could it be a competence barrier to protect incompetence? I have experienced so many similar experiences, normally with middle managers who are terrified they will get caught out for not knowing what they are doing. I noticed they always stick with brands like Microsoft or SAP are deeply suspicious of open source. Oh and they dont drown, they get promoted. Could this be an opportunity for a startup to identify this risk "as a service"
jacquesm 7y ago
The way to establish change is to do it gradually, with buy-in from all parties prior to introducing the change and to make small steps in the beginning. If you showed up with a huge chunk of code that has been refactored I'd probably not merge your change either. The test cases have been written by the same person that did the refactoring, there is a very good chance that any wrong assumptions you made about the code will be present in both.
inertiatic 7y ago
>I refactored the code from untested and untestable, to testable with 40% test coverage. This tells me you think "coverage" is a good metric, which also tells me you're you're not covering a whole lot of cases (edge or not) where the "untested code" functioned as it should. You're asking to replace something which has potentially some known issues, with something that has a potentially unlimited amount of unknown issues. And the more experienced person at your job refused? What a tyrant!
- poutrathor 7y ago
  I strongly disagree. The said refactoring was probably not a huge task since he is reporting it done. Larger tasks are most often planned and monitored. Splitting some class and decoupling some resources + adding the testing framework and implementing a first batch of tests. Of course it might have introduced few regressions. But > has potentially some known issues Read again that. Exactly, none knows how many bugs there are, only how many were found so far. Bug tracking is often lacking, behind schedule, wont fix, etc. On the contrary modern code analyse tools are frightening good at spotting untested conditions every where (sonarcube comes to my mind). Much better and much reliable than human developer and human tester. Sonarcube does not care if the case can not happen because user interface does not allow the input. The bug is here, and the bug is spotted. The more experienced person is plain wrong : more tests is better. It's not panacea yet, but it's better.
  - inertiatic 7y ago
    function int untested_sum(int a, int b) { return a + b; } function int tested_sum(int a, int b) { return a * b; } function void test_providing_total_coverage() { assertEqual(tested_sum(2, 2), 4, 'Expected sum to be 4!'); }
    - Doxin 7y ago
      Which is why I'm a big fan of testing tools like hypothesis: from hypothesis import given, assume from hypothesis.strategies import floats from math import isnan, isfinite def tested_sum(a, b): return a * b @given(a=floats(), b=floats()) def test_sum(a, b): assume(not isnan(a)) assume(not isnan(b)) assume(isfinite(a) or isfinite(b)) assert tested_sum(a, b) == a + b Running that test gives Falsifying example: test_sum(a=0.0, b=1.0) Note how it also forces me to put a bunch of assumptions in there explicitly, hypothesis is very good at finding weird edge cases. For this example I don't really care that it goes wrong when NaN gets involved, but if I did I discover through testing that that case even exists.
steve_gh 7y ago
So many years ago I was involved in an issue with a BTS (a cellular radio base station) for a telecoms equipment manufacurer. The software was written in C, and had a message parsing architecture between processes, with fixed length queues, which discarded messages when they overloaded. There were various problems (the subject of a number of war-stories, which I won't repeat now) - but basically anything that wasn't using a properly designed state machine could die horribly when messages started getting discarded. So we did some work around the fundamental design of the queue handling mechanism. We came up with a software modification which would improve the efficiency of the queues handling, which would stop the queues overloading (which we saw happening occasionally at peak loads on the biggest BTSs). We could measure various aspects of the queue handling performance, and use queuing theory to prove that the performance would be better at very high loads. We had a test rig and we could test the relative performance of the new software vs the old software at normal loads. What we didn't have, was a test rig which could run at the highest loads, so we couldn't test the behaviour of the new algorithms at very high loads. So the improvements were binned. The existing software was known to fail (badly) at extreme loads. The new software could be tested at reasonable loads, and mathematically be shown to improve matters at very high loads, but because it couldn't be tested, the process said that it must not be deployed!
- Aeolun 7y ago
  How was the original version deployed? The definition of ‘very high load’ was different at the time.
- edwhitesell 7y ago
  I've seen this with people I've worked with that came from those environments. Usually it's a balance between knowing how the current implementation fails is better than not knowing when the new one will.
pvaldes 7y ago
"I think that is not sabotage, but I don't have any real data that support or dismiss my statement" would be a better title. Your mileage may vary.
marcus_holmes 7y ago
I think it's kinda hilarious that the article talks about a psychological effect that hurts software teams, and HN gets into a full-on debate about software testing strategies ;) I've seen this stuff before. Developers making obviously-bad decisions is usually a symptom of a dysfunctional work culture and a code base that's both vital and fragile. It's not fun to work in such places, but it is possible to fix them.
- hvidgaard 7y ago
  The main issue is for management to accept that velocity will be significantly reduced and that many issues will pop up along the way and require a substantial QA effort to keep the quality from dipping too low. It's still the best way forward as a complete rewrite is rarely a good idea.
  - marcus_holmes 7y ago
    yeah, I had to explain to a CEO that we weren't going to be shipping any new features for a few months while we upgraded core systems and eliminated tech debt. It was a tough conversation. But about 6 months later we were able to start developing new features again, and everything was much quicker and smoother. The lead dev had regained his composure, too, and wasn't being actively obstructive to everyone who tried to do anything to "his" code base.
    - deleted 7y ago
      [deleted]
merrill_g 7y ago
In my world (marketing, not dev) the saying is "you cannot optimise something you do not measure".
stakhanov 7y ago
Between sabotage and people wanting to sink the lifeboat, I have a third option: These people have a warped perception of what's happening and priorities around what's happening. Warped by their own irrational beliefs which have come about as a result of cognitive dissonance reduction. Initially, they can see how messy & bad things are but since they, themselves, are responsible for things having gotten to that point, they start bending their own beliefs in a way that will make themselves and their past actions look good. In the final phase of this psychological process, some kind of Stockholm syndrome kicks in in relation to the poor standard of engineering that is making their lives miserable: They start to actively seek out poor engineering because it's the only mode of existence that their psyche will allow them to fathom. Warstory time. A tech company I worked at was being encumbered by a codebase with poor code quality. Bugs were being introduced all the time, with no chance of preventing it because nothing is testable/tested. Bugs were introduced at a faster rate than they were fixed. The whole system was a timebomb because it was built in a dying ecosystem. For certain kinds of code changes, their effect couldn't even be made visible at development-time at all. You had to release them into the production system and look at metrics to see their effect, hoping that nothing would blow up, so the system was layers and layers of code that was trying to be minimally-invasive with respect to existing code. Whenever an engineer would be newly hired, they would, within their first week of working there, suggest "Let's take a month to refactor this/that part of the system". When an oldtimer would hear the new guy say "refactor", their warped perception would hear "accomplish nothing" or "threaten our mode of existence". They would give the following speech: "Hey, get it in your head, noob: We don't do that here. This company is not focused on code quality, it's focused on a mission. Every week we come to work and we accomplish a part of a mission. I KNOW the code is probably the worst piece of shit you've ever seen in your 20 years. [This company only hires extremely experienced people]. The engineers who work here are all on board with our mission-driven culture and pride themselves about their skill at dealing with the shitty codebase. Your suggestion tells me that you can't deal with it, and you are unwilling to absorb our mission-focused culture. You will not get a career out of working here, unless you properly absorb that element of our culture". So, in their mind, they'd taken "shitty codebase" which is a negative, and turned it around and made it into a religion about a "mission-driven" culture which is a positive. [Cognitive dissonance reduction at work, right there]. The reality was, of course, the poor code quality was a huge drag on this company's ability to accomplish a mission of any kind: Because changes that took their competitors a day, took them a week. Changes that their competitors did in a month, would have been a suicide mission for them and they therefore rationalized their way out of even wanting to do them. ...but I wouldn't recommend to the newly hired engineer in the story to mention that particular elephant that was in the room. Now, here comes the really sick part, where the picture of the drowning man falls apart: From time to time, due to a rare alignment of circumstances, this company might be forced into a situation of starting an implementation of a new element of the system with a clean slate, i.e. completely unencumbered by the current codebase. They would intentionally implement the new codebase to be just as shitty as the legacy code. They would implement it in the dying ecosystem. They would generously insert unreachable code. They would pursue the fastest path towards making it untestable. Now, under normal circumstances, if I hear someone telling that story, I would suppose there might be a motive for this. For example, I've seen developers in hedge funds intentionally rig a system to explode if they should ever be forced out of the company and be as hard as possible for anyone else to fix, for obvious reasons having to do with their incentivization structure. But that's not what was happening here. They had a culture of openly sharing knowledge of the system, were hiring new people all the time, and working hard to bring them up to speed about the system's working etc, so it wasn't that. My guess is that it must be the "Stockholm syndrome" element I mentioned: They started to develop a psychological attachment to a poor standard of engineering. It was making their lives miserable. But it was also the only existence they knew and they felt was available to them, so cognitive-dissonance reduction was maneuvering their psyches into actually favoring that mode of existence. Some might have said things like: "One day, after we've grown enough, we will do a complete rewrite." Or "Any day now, we might be acquired by another company and will leave all of this behind". But it was more in the vein as when a religious persons speaks of a glorious afterlife. There is a lot of really sick shit that the human psyche is capable of, and sometimes it surfaces in engineering. I think that's my takeaway from all this.
taneq 7y ago
> When you add visibility to a system, the numbers are always bad. That’s why you’re putting in the effort to add visibility to an existing system. This is something everyone needs to internalize. Don't be upset if the numbers are bad when you first instrument. That's WHY you instrumented. Instrumentation is a ratchet to help you improve your product.

koz_ 7y ago

This allergy to reality is surely the biggest trap not just for startups but people in general. It's surprising but I feel like the norm is for people to care far more about feeling that everything's okay, rather than everything actually being okay. That said, one problem I see with the anecdotes provided is that the problem solver is doing two things: discovering a problem and solving it. Conflating these two things is a great way to encourage pushback: every change comes with an associated risk, so the general rule is to say no (using whatever flimsy reason) if it's not well understood and agreed that the problem being addressed is a real problem worth solving.

conradp 7y ago
> ... the problem solver is doing two things: discovering a problem and solving it. Conflating these two things is a great way to encourage pushback This squares with my experience strongly. At a prior job, I remember two or three times where a set of changes was almost rejected for production because despite solving real bugs. The bugs hadn't affected anything in production yet, but the fixes carried the risk of maybe breaking something. This resulted in some fun times, like a particular client running something like three major versions behind on our software, precisely because management didn't want to accept any disruption risks there, and the old version grumbled along well enough.

Ensorceled 7y ago

Reminds me of this scene from Frank Herbert's Dune: “Because of an observation made by my father at the time. He said the drowning man who climbs on your shoulders to save himself is understandable– except when you see it happen in the drawing room.” Paul hesitated just long enough for the banker to see the point coming, then: “And, I should add, except when you see it at the dinner table.”

ltbarcly3 7y ago

I think saying 'they're drowning' is still giving them too much credit. People who built terrible broken systems don't know they are bad at building things, and they probably don't have a clear idea of what would be better or worse to begin with. They don't know it's better to have a system that can be debugged, they don't even know what the difference is. If you give them examples of good designs and bad designs, they can't tell you which is which. When they say 'this is good' or 'this is bad' they will astound you with the reasoning. Software is an industry with no standards for employment. All it takes to be a Software Engineer is for an incompetent manager to hire you. More than half the people I interview barely have a basic familiarity with writing software. These are people who have been employed as software engineers for years and will certainly find another job. They struggle to write a nested for loop. They write functions with thousands of lines. They just don't know what they are doing. The author here claims they don't want the visibility of the situation to increase, but I think they don't know it's a problem to begin with. Their stupid excuses for not moving forward are just their stupid thoughts said out loud.

vanniv 7y ago
My team has built garbage software at times. It isn't because we don't know how to build good software, and it isn't because we don't care, and it isn't because we don't want to build good software. It is usually because of a continuous sequence of emergencies to which we are forced to react. An emergency occurs, and we have requirements to do something in 1/3 the necessary time. We get 80% of the way though that and, before launch, there's a new emergency, and we have to rush to that. So we end up with crap software that looks like shit and only almost works. But it almost works enough to keep our very large business humming along. Sure, we hate it, and we're making more technical debt, and over time, we get less effective. But sometimes we go through cycles of a year or two when the business either can't afford to do it right or isn't willing to pay the price to do it right. Sometimes, they really are willing to accept paying 10x as much so long as they can pay in a year or two. (I hate it, but it is true)
- AstralStorm 7y ago
  Make it 100x as much with 5 times as many people in 10 times as long. Efficiency is never important when there is money to burn and it's not yours, and you won't be blamed when it breaks. It does not even take any emergency - often it is conscious or subconscious project management strategy. It's related to observations about Peter principle, Dilbert principle, Gervais principle and office politics and dynamics. Avoiding this tendency is essentially only possible with small teams with full ownership of the problem they're working on and direct stakes.

lowercased 7y ago

> I refactored the code from untested and untestable, to testable with 40% test coverage. The senior architect is refusing to merge because the test coverage is to low. what's likely even more frustrating is that other merges went in for years with no tests or testing in place at all.

umvi 7y ago
Seems like unit tests are only really valuable for library code. Application/glue code... not so much, especially if it is something being actively developed and you are constantly throwing away tons of unit tests.
- AstralStorm 7y ago
  Totally untrue. Unit tests in strict sense, perhaps, but nobody writes true unit tests - there's a high chance these are actually short range functional or integration tests. Such tests are of high value.
  - umvi 7y ago
    Maybe it just depends on the context. I write embedded linux applications. One application I wrote talks to a Qualcomm modem using their QMI framework and then passes on messages to an antenna subsystem. I wrote some library functions, which were easily unit testable, then I wrote the glue code, which I found wasn't very testable at all. I found myself having to stub out the entire QMI Framework/modem and the entire Antenna subsystem API since obviously I don't have access to those things at build time. So basically now my application/glue code is being testing in a fake vacuum that may not even reflect the real behaviors of the modem or antenna. So I waste tons of time trying to predict how the modem and antenna might behave, and test my application code accordingly. Lastly, I end up scrapping a bunch of the tests because the actual behavior of the modem/antenna is different than I predicted and changes slightly with each update.
- larzang 7y ago
  The weakest link in your software is always you. Automation that can help compensate for human failings always has value. I can't count the number of times writing tests has led me to catch simple mistakes in code that I'd stared at for hours and declared good. Not to even get into the ways that tests can help guide sensible architecture, especially for junior devs, because "this is hard to test" is the single most straightforward code smell for "this probably needs to be simplified or broken up".

mirimir 7y ago

When I was trained in lifesaving, one strategy for panicked victims, who might drown the rescuer, was just to give them some space to drown enough to lose consciousness. Then they were much easier to rescue.

wheelerwj 7y ago
it doesn’t usually apply to drowning victims, but more broadly its applied when someone is choking in general. if someone is refusing help, you wait until they pass out, and it becomes implied consent. I assume in this example, that would be firing or otherwise being told by your supervisor to take the help.
- kevingadd 7y ago
  Consent isn't necessarily the issue in drowning individuals, since the instinctive drowning response can make it dangerous to save them even if they're not about to tell you 'let me drown'. Maybe actually a good comparison for software teams that refuse to make incremental improvements out of fear or panic... it's not that they don't want things to be better or refusing help, necessarily - they're just terrified of the consequences if the improvements go wrong.

Aeolun 7y ago

So being the person that’s manning the lifeboat, what do you do? Kick them down? If you do nothing everyone will surely drown.

stickfigure 7y ago

I can perhaps understand where this one is coming from: I refactored the code from untested and untestable, to testable with 40% test coverage. The senior architect is refusing to merge because the test coverage is to low. You have a piece of software running in production; even if it's poorly written and bereft of tests, your customers are satisfied. Customers (unwittingly) have already vetted this version! A rewrite, even with brilliant design and fantastic test coverage, is still more of a risk than fixing what isn't broken. This isn't to say that you shouldn't rewrite code, especially if it makes other improvements possible. But don't discount how high a bar new code has to reach to measure up with "it already works". Maybe 40% is still too risky.

AstralStorm 7y ago
It's always a lie, about running in production. I've personally seen untested code allegedly running but it isn't actually working. Defect numbers through the roof and test harness literally not being merged exactly because of this issue. You will get visibility, either by customer complaints, slowing down development speed and ultimately projects being shut down. It's not sabotage and it's not exactly drowning. It's willful attempt to ignore reality and cheat on metrics, prolonging the inevitable and defrauding stakeholders.
- BigJono 7y ago
  It's always a lie? You're basically arguing that code with bad test coverage never works. Production code is copping constant manual end-to-end tests. Those tests are more valuable than most unit testing, and some automated black box testing. I'll take stable production system over the unit tested new code any day of the week.
  - peterkelly 7y ago
    That's the kind of thinking that leads to situations like this: https://www.digitaltrends.com/mobile/samsung-galaxy-s10-ultrasonic-fingerprint-scanner-fooled-by-screen-protector/ https://www.digitaltrends.com/mobile/samsung-galaxy-s10-ultr... A lot of the time it's just as important to make sure certain things can't work as making sure that certain other things will.

CoolGuySteve 7y ago

Nobody’s better at letting better be the enemy of good than software engineers. For some reason our culture likes to argue and nitpick over the most mundane stuff, usually missing the forest for the trees. I’ve had jobs where the manpower spent just arguing outweighed the cost of the change significantly. One dickhead manager I had spent weeks arguing over the acquisition of a $500 SSD. It made me want to slap $500 down on the table just so he’d shut the fuck up already. The only way I know to counteract it is by saying “don’t let better be the enemy of good”.

brain5ide 7y ago
A few points: 1. Computing and engineering in general is such a discipine where one nitpicky detail can make all the difference. One bit flip, one keystone, one badly selected material. However, the conduct of negotiation, could be improved as always, but it's a general listening problem most of the time. 2. From the dickheads point or view, everything can be solved by spending more money on HW. Each unexpected spend like this is a risk to open the floodgates and lose both financial and engineering discipline by setting a bad precedent. Not sure you want easy handed decisions there as well.
- biggestdecision 7y ago
  It makes no sense to cheap out on hardware though. It's such a tiny cost compared to the overall cost of the employee, you are wasting money if inadequate hardware is making your employees less productive.
  - retSava 7y ago
    In one sense, it doesn't make sense. The productivity boost would likely be worth it alone. From another perspective, this is a +500$ post in the cost column of that unit (and perhaps n*500 if the whole unit then comes asking for the same thing). The cost of spending time arguing is baked into the cost of salary, meaning it doesn't show since it's the same regardless. It's a form of optimization, but the optimized unit is not productivity. A friend of mine was working as a recruiter where the whole company was sitting all day long at their laptops. He was denied buying an external screen to be more efficient since he then could have more information on the screen (eg an application next to a browser), "because then everybody wants one", even buying it himself. edit: nb I'm not arguing for not buying the ssd/screen.

dredmorbius 7y ago

A possible way to sell instrumentation / metering is to build your own scaffolding, find a problem using it, and sell both the scaffolding and the solution it's provided. As the article notes, metering by itself simply illuminates problems. For the drowning team (or individual), that simply delivers bad news. If you can show that the effort actually delivers results, you've got a way to sell the instrumentation as well. In more pathological cases, you don't reveal the scaffolding, but start cranking out code fixes based on it. You may have the opportunity to sell the metering later, when you're asked how you're finding and fixing the problems. Rescuing projets from antipatterns is painful at best. Enlightenment is a painful process. It peels away comforting lies to reveal unpleasant truths.

deboflo 7y ago

This is one of those things you don’t truly understand until you’ve experienced it first hand. I have. Very appropriate metaphors.

peter_retief 7y ago

Could it be a competence barrier to protect incompetence? I have experienced so many similar experiences, normally with middle managers who are terrified they will get caught out for not knowing what they are doing. I noticed they always stick with brands like Microsoft or SAP are deeply suspicious of open source. Oh and they dont drown, they get promoted. Could this be an opportunity for a startup to identify this risk "as a service"

jacquesm 7y ago

The way to establish change is to do it gradually, with buy-in from all parties prior to introducing the change and to make small steps in the beginning. If you showed up with a huge chunk of code that has been refactored I'd probably not merge your change either. The test cases have been written by the same person that did the refactoring, there is a very good chance that any wrong assumptions you made about the code will be present in both.

inertiatic 7y ago

>I refactored the code from untested and untestable, to testable with 40% test coverage. This tells me you think "coverage" is a good metric, which also tells me you're you're not covering a whole lot of cases (edge or not) where the "untested code" functioned as it should. You're asking to replace something which has potentially some known issues, with something that has a potentially unlimited amount of unknown issues. And the more experienced person at your job refused? What a tyrant!

poutrathor 7y ago
I strongly disagree. The said refactoring was probably not a huge task since he is reporting it done. Larger tasks are most often planned and monitored. Splitting some class and decoupling some resources + adding the testing framework and implementing a first batch of tests. Of course it might have introduced few regressions. But > has potentially some known issues Read again that. Exactly, none knows how many bugs there are, only how many were found so far. Bug tracking is often lacking, behind schedule, wont fix, etc. On the contrary modern code analyse tools are frightening good at spotting untested conditions every where (sonarcube comes to my mind). Much better and much reliable than human developer and human tester. Sonarcube does not care if the case can not happen because user interface does not allow the input. The bug is here, and the bug is spotted. The more experienced person is plain wrong : more tests is better. It's not panacea yet, but it's better.
- inertiatic 7y ago
  function int untested_sum(int a, int b) { return a + b; } function int tested_sum(int a, int b) { return a * b; } function void test_providing_total_coverage() { assertEqual(tested_sum(2, 2), 4, 'Expected sum to be 4!'); }
  - Doxin 7y ago
    Which is why I'm a big fan of testing tools like hypothesis: from hypothesis import given, assume from hypothesis.strategies import floats from math import isnan, isfinite def tested_sum(a, b): return a * b @given(a=floats(), b=floats()) def test_sum(a, b): assume(not isnan(a)) assume(not isnan(b)) assume(isfinite(a) or isfinite(b)) assert tested_sum(a, b) == a + b Running that test gives Falsifying example: test_sum(a=0.0, b=1.0) Note how it also forces me to put a bunch of assumptions in there explicitly, hypothesis is very good at finding weird edge cases. For this example I don't really care that it goes wrong when NaN gets involved, but if I did I discover through testing that that case even exists.

steve_gh 7y ago

So many years ago I was involved in an issue with a BTS (a cellular radio base station) for a telecoms equipment manufacurer. The software was written in C, and had a message parsing architecture between processes, with fixed length queues, which discarded messages when they overloaded. There were various problems (the subject of a number of war-stories, which I won't repeat now) - but basically anything that wasn't using a properly designed state machine could die horribly when messages started getting discarded. So we did some work around the fundamental design of the queue handling mechanism. We came up with a software modification which would improve the efficiency of the queues handling, which would stop the queues overloading (which we saw happening occasionally at peak loads on the biggest BTSs). We could measure various aspects of the queue handling performance, and use queuing theory to prove that the performance would be better at very high loads. We had a test rig and we could test the relative performance of the new software vs the old software at normal loads. What we didn't have, was a test rig which could run at the highest loads, so we couldn't test the behaviour of the new algorithms at very high loads. So the improvements were binned. The existing software was known to fail (badly) at extreme loads. The new software could be tested at reasonable loads, and mathematically be shown to improve matters at very high loads, but because it couldn't be tested, the process said that it must not be deployed!

Aeolun 7y ago
How was the original version deployed? The definition of ‘very high load’ was different at the time.
edwhitesell 7y ago
I've seen this with people I've worked with that came from those environments. Usually it's a balance between knowing how the current implementation fails is better than not knowing when the new one will.

pvaldes 7y ago

"I think that is not sabotage, but I don't have any real data that support or dismiss my statement" would be a better title. Your mileage may vary.

marcus_holmes 7y ago

I think it's kinda hilarious that the article talks about a psychological effect that hurts software teams, and HN gets into a full-on debate about software testing strategies ;) I've seen this stuff before. Developers making obviously-bad decisions is usually a symptom of a dysfunctional work culture and a code base that's both vital and fragile. It's not fun to work in such places, but it is possible to fix them.

hvidgaard 7y ago
The main issue is for management to accept that velocity will be significantly reduced and that many issues will pop up along the way and require a substantial QA effort to keep the quality from dipping too low. It's still the best way forward as a complete rewrite is rarely a good idea.
- marcus_holmes 7y ago
  yeah, I had to explain to a CEO that we weren't going to be shipping any new features for a few months while we upgraded core systems and eliminated tech debt. It was a tough conversation. But about 6 months later we were able to start developing new features again, and everything was much quicker and smoother. The lead dev had regained his composure, too, and wasn't being actively obstructive to everyone who tried to do anything to "his" code base.
  - deleted 7y ago
    [deleted]

merrill_g 7y ago

In my world (marketing, not dev) the saying is "you cannot optimise something you do not measure".

stakhanov 7y ago

Between sabotage and people wanting to sink the lifeboat, I have a third option: These people have a warped perception of what's happening and priorities around what's happening. Warped by their own irrational beliefs which have come about as a result of cognitive dissonance reduction. Initially, they can see how messy & bad things are but since they, themselves, are responsible for things having gotten to that point, they start bending their own beliefs in a way that will make themselves and their past actions look good. In the final phase of this psychological process, some kind of Stockholm syndrome kicks in in relation to the poor standard of engineering that is making their lives miserable: They start to actively seek out poor engineering because it's the only mode of existence that their psyche will allow them to fathom. Warstory time. A tech company I worked at was being encumbered by a codebase with poor code quality. Bugs were being introduced all the time, with no chance of preventing it because nothing is testable/tested. Bugs were introduced at a faster rate than they were fixed. The whole system was a timebomb because it was built in a dying ecosystem. For certain kinds of code changes, their effect couldn't even be made visible at development-time at all. You had to release them into the production system and look at metrics to see their effect, hoping that nothing would blow up, so the system was layers and layers of code that was trying to be minimally-invasive with respect to existing code. Whenever an engineer would be newly hired, they would, within their first week of working there, suggest "Let's take a month to refactor this/that part of the system". When an oldtimer would hear the new guy say "refactor", their warped perception would hear "accomplish nothing" or "threaten our mode of existence". They would give the following speech: "Hey, get it in your head, noob: We don't do that here. This company is not focused on code quality, it's focused on a mission. Every week we come to work and we accomplish a part of a mission. I KNOW the code is probably the worst piece of shit you've ever seen in your 20 years. [This company only hires extremely experienced people]. The engineers who work here are all on board with our mission-driven culture and pride themselves about their skill at dealing with the shitty codebase. Your suggestion tells me that you can't deal with it, and you are unwilling to absorb our mission-focused culture. You will not get a career out of working here, unless you properly absorb that element of our culture". So, in their mind, they'd taken "shitty codebase" which is a negative, and turned it around and made it into a religion about a "mission-driven" culture which is a positive. [Cognitive dissonance reduction at work, right there]. The reality was, of course, the poor code quality was a huge drag on this company's ability to accomplish a mission of any kind: Because changes that took their competitors a day, took them a week. Changes that their competitors did in a month, would have been a suicide mission for them and they therefore rationalized their way out of even wanting to do them. ...but I wouldn't recommend to the newly hired engineer in the story to mention that particular elephant that was in the room. Now, here comes the really sick part, where the picture of the drowning man falls apart: From time to time, due to a rare alignment of circumstances, this company might be forced into a situation of starting an implementation of a new element of the system with a clean slate, i.e. completely unencumbered by the current codebase. They would intentionally implement the new codebase to be just as shitty as the legacy code. They would implement it in the dying ecosystem. They would generously insert unreachable code. They would pursue the fastest path towards making it untestable. Now, under normal circumstances, if I hear someone telling that story, I would suppose there might be a motive for this. For example, I've seen developers in hedge funds intentionally rig a system to explode if they should ever be forced out of the company and be as hard as possible for anyone else to fix, for obvious reasons having to do with their incentivization structure. But that's not what was happening here. They had a culture of openly sharing knowledge of the system, were hiring new people all the time, and working hard to bring them up to speed about the system's working etc, so it wasn't that. My guess is that it must be the "Stockholm syndrome" element I mentioned: They started to develop a psychological attachment to a poor standard of engineering. It was making their lives miserable. But it was also the only existence they knew and they felt was available to them, so cognitive-dissonance reduction was maneuvering their psyches into actually favoring that mode of existence. Some might have said things like: "One day, after we've grown enough, we will do a complete rewrite." Or "Any day now, we might be acquired by another company and will leave all of this behind". But it was more in the vein as when a religious persons speaks of a glorious afterlife. There is a lot of really sick shit that the human psyche is capable of, and sometimes it surfaces in engineering. I think that's my takeaway from all this.

taneq 7y ago

> When you add visibility to a system, the numbers are always bad. That’s why you’re putting in the effort to add visibility to an existing system. This is something everyone needs to internalize. Don't be upset if the numbers are bad when you first instrument. That's WHY you instrumented. Instrumentation is a ratchet to help you improve your product.