13 ms·

Having been involved in meetings where "stop ship" was the phrase of the day, I'd bet money that the following at least vaguely resembles a real conversation:

Having been involved in meetings where "stop ship" was the phrase of the day, I'd bet money that the following at least vaguely resembles a real conversation: Engineer Alice: We should really fix this properly. Manager: How sure are you that the proper fix won't break something else for $BIG_CUSTOMERS who are responsible for $OBSCENE percent of this product line's revenue? Engineer Bob: Uh, ten percent on a good day? Do you remember the last time we applied a "simple" hotfix to this function? I almost had to sleep under my desk. Manager: And we can just prevent it with an extra regex rule in the front end code? Engineer Alice: ... Yes. irrepressibly existential sigh Manager: How much QA effort to call this good? QA Engineer: We can run through a cut-down version of the acceptance test suite in four or five hours. Manager: And for the proper fix? QA Engineer: Oh, hell, at least a week of stress testing to really be sure. Manager: Add the regex rule.

coldacid 8y ago
Bob is either a brown-noser or a complete turncoat. Don't be like Bob.
- philpem 8y ago
  Bob is probably an engineer with twenty years experience, who's been thoroughly ground down by management and peer power-plays and has realised that the best solution is to keep your head below the parapets!
  - opportune 8y ago
    Lol, this is too true. Though I think I learned 80% of this in my first couple months at my first real full time job
anyfoo 8y ago
> Manager: And we can just prevent it with an extra regex rule in the front end code? > Engineer Alice: ... Yes. irrepressibly existential sigh The answer is "no", though. This exchange sounds like the decision was between "fixing it cleanly" and "fixing it with a hack", which is roughly equivalent with fixing the problem at the expense of technical debt. But this change does not fix the problem at all.
- fosco 8y ago
  I have to agree that the answer is no. curl -A "anything but Curl" x.x.x.x the problem has not been fixed by a user agent string.
amalcon 8y ago
I've been Engineer Alice before (not with a security issue, but a pretty bad systems issue). In case anyone is ever in that situation, here are some more productive replacements for that existential sigh (in that they sometimes work). Choose the one most suited to manager's biases. The cliches are important: think of it as giving manager an easy way to explain it to their manager. > Yes, but that would only help with this specific case. The proper fix would ensure that we don't see a similar issue tomorrow. We don't want to prepare for yesterday's battle. > Yes, but the extra regex rule would be new code, which would need its own maintenance in perpetuity. Fixing it properly would let us remove the existing code. Removing code is the most efficient thing an engineer can do, and will save man-hours in the long run. > Yes, but the extra regex rule has worse runtime performance than the proper fix. [Performance is our big competitive advantage | We don't want to fall behind in performance | We are already behind in performance]. We don't want to create a customer-visible performance regression.
- anyfoo 8y ago
  No! The answer is an unequivocal “No“, without any „but“s or anything like that. The fix does not fix the problem, it is not even a fix, just a wrong code change that sets out to do something but does not achieve it. Answering „yes“ is a lie here. This is different from a fix that fixes the problem in an ugly way, where „yes, but...“ is applicable.
  - Quekid5 8y ago
    Indeed, and what's worse is that many managers[1] will stop listening after "Yes, ...". [1] The incompetent ones, of which there are many.
  - amalcon 8y ago
    In this specific case (I have never encountered a proposal quite this bad), the correct answer is probably "Give me the proof of concept code and about ten minutes, and I'll give you a version that works despite the patch." You're right about that. "No" is likely to bog down the discussion with "but Bob says it will work", and it's better to skip to the end. I'm envisioning more of a middle-ground case, where the code is still obviously wrong but it's harder to cleanly demonstrate why.
throwawaymath 8y ago
Ouch, that hits close to home. It's like you've sat in on all the same meetings I have, word for word.
neandrake 8y ago
Manager: Bob get to work on the quick fix that will mitigate some of the attacks based on the published exploit to go out within a day, Alice get to work on the correct fix to address the underlying issue to go out in a few weeks when it's fully tested and ready.
- witten 8y ago
  Manager: Alice, yeah, I know I said we'd carve out time this sprint for you to work on the long-term fix. But this other hot feature request came in, and there's potentially a lot of money riding on getting it done quickly. So can you put off the long-term fix, maybe a week? Two tops. You can just put the ticket.. yeah that one.. back on the backlog. There you go. Thanks so much!
admax88q 8y ago
On HN its always big bad management who is the cause of every security problem or shoddy piece of engineering. If only that pesky management would screw off then we could do things "properly". You'd be suprieed at how many incompetent engineers there are out there. If "engineer" Alice in your story was actually competent they would never agree to implement the proposed "fix". Its not a fix. To pass it off as one would be malpractice for a real Engineer.
- raverbashing 8y ago
  And I might add: especially at big companies where being yes men is a higher valued characteristic than being technical
  - philpem 8y ago
    Oh, so much this. "I'm sick of you bringing me problems, bring me solutions! And I want that blockchain based big-data cloud IoT solution by the end of the week!"
- astrobe_ 8y ago
  You live in a nice world, because usually the actual power (and duty) of a dev team member is to advise, not to agree or refuse.
  - admax88q 8y ago
    You can always quit. Or force them to fire you for refusing to implement a non fix. In reality though I doubt this narrative even occurred. Some incompetent engineer likely proposed this fix thinking that it was actually a fix. Edit: I see I've been downvoted for this comment. If we were real engineers working on things like cars and bridges we'd actually be held accountable. Take some pride in your work people, this is one of the most in demand professions in today's economy.
    - pjmlp 8y ago
      Quitting assumes the possibility of finding a job around the corner and not having people that depend on you.
skrebbel 8y ago
Engineer Alice is the only person incompetent at their job in this conversation. If "irrepressibly existential sigh" is how you argue security with your bosses then maybe you're not senior enough to be in meetings like these. The Manager and QA Engineers here depend on the expertise of the engineers. If the engineers fail to communicate key details of the situation, then that's on them. Sure, the boss is still at fault for depending on a shitty engineer for decision making, but that's all.
- consp 8y ago
  > The Manager and QA Engineers here depend on the expertise of the engineers. Both should be sufficiently competent and knowledgeable to know this is a terrible idea, especially in the long run. Otherwise they do not understand the task they have been given. So no, Alice is not the only incompetent one here.
- Aeolun 8y ago
  If you answer with the existential sigh, you’ve had at least 20 conversations with a similar outcome before. Nobody arrives at a new company that jaded.
  - skrebbel 8y ago
    I see your point and I suspect that that's what the author meant. At the same time, I know lots of, well, arrogant nerds who get into "existential sigh" mode as soon as someone with less technical skill than them says something stupid. That's more how I read it.
    - Aeolun 8y ago
      Ah, yes, I see how that is another valid interpretation. I guess for me the word existential is a bit different. For an arrogant developer it would have been something like ‘long-suffering sigh’. Unless, I guess, you feel you are so smart that your entire life is defined by interacting with morons :P
gdfasfklshg4 8y ago
In similar situations I 1. Explain that the 'hack' does not actually fix the problem. This is absolutely the engineers responsibility. In your example Alice does not say that. This has always been sufficient. 2. I make a habit of sending emails or keeping minutes confirming such meetings and the details of who said what.
- robertcope 8y ago
  This is about all you can do. But you're right. You have to explain it and document it: "It's trivial to change User-Agent, and any decent hacker would probably not be using 'curl' as their User-Agent. We can look at logs from past events to validate this assumption."
supermatt 8y ago
Ironically, this is the change most likely to break things for customers that use curl for automation.
m463 8y ago
This reminds me of the old BOFH posts from days past. The one where the "appeasement engineer" shows up to fulfill the guaranteed response time requirement.
CM30 8y ago
Then someone realises one of their enterprise clients is using curl for a critical system, and said patch has completely broken it. Or more likely, they realise after customer/company calls complaining about how their systems are now busted...
stebann 8y ago
Wow, You got this one!

coldacid 8y ago

Bob is either a brown-noser or a complete turncoat. Don't be like Bob.

philpem 8y ago
Bob is probably an engineer with twenty years experience, who's been thoroughly ground down by management and peer power-plays and has realised that the best solution is to keep your head below the parapets!
- opportune 8y ago
  Lol, this is too true. Though I think I learned 80% of this in my first couple months at my first real full time job

anyfoo 8y ago

> Manager: And we can just prevent it with an extra regex rule in the front end code? > Engineer Alice: ... Yes. irrepressibly existential sigh The answer is "no", though. This exchange sounds like the decision was between "fixing it cleanly" and "fixing it with a hack", which is roughly equivalent with fixing the problem at the expense of technical debt. But this change does not fix the problem at all.

fosco 8y ago
I have to agree that the answer is no. curl -A "anything but Curl" x.x.x.x the problem has not been fixed by a user agent string.

amalcon 8y ago

I've been Engineer Alice before (not with a security issue, but a pretty bad systems issue). In case anyone is ever in that situation, here are some more productive replacements for that existential sigh (in that they sometimes work). Choose the one most suited to manager's biases. The cliches are important: think of it as giving manager an easy way to explain it to their manager. > Yes, but that would only help with this specific case. The proper fix would ensure that we don't see a similar issue tomorrow. We don't want to prepare for yesterday's battle. > Yes, but the extra regex rule would be new code, which would need its own maintenance in perpetuity. Fixing it properly would let us remove the existing code. Removing code is the most efficient thing an engineer can do, and will save man-hours in the long run. > Yes, but the extra regex rule has worse runtime performance than the proper fix. [Performance is our big competitive advantage | We don't want to fall behind in performance | We are already behind in performance]. We don't want to create a customer-visible performance regression.

anyfoo 8y ago
No! The answer is an unequivocal “No“, without any „but“s or anything like that. The fix does not fix the problem, it is not even a fix, just a wrong code change that sets out to do something but does not achieve it. Answering „yes“ is a lie here. This is different from a fix that fixes the problem in an ugly way, where „yes, but...“ is applicable.
- Quekid5 8y ago
  Indeed, and what's worse is that many managers[1] will stop listening after "Yes, ...". [1] The incompetent ones, of which there are many.
- amalcon 8y ago
  In this specific case (I have never encountered a proposal quite this bad), the correct answer is probably "Give me the proof of concept code and about ten minutes, and I'll give you a version that works despite the patch." You're right about that. "No" is likely to bog down the discussion with "but Bob says it will work", and it's better to skip to the end. I'm envisioning more of a middle-ground case, where the code is still obviously wrong but it's harder to cleanly demonstrate why.

throwawaymath 8y ago

Ouch, that hits close to home. It's like you've sat in on all the same meetings I have, word for word.

neandrake 8y ago

Manager: Bob get to work on the quick fix that will mitigate some of the attacks based on the published exploit to go out within a day, Alice get to work on the correct fix to address the underlying issue to go out in a few weeks when it's fully tested and ready.

witten 8y ago
Manager: Alice, yeah, I know I said we'd carve out time this sprint for you to work on the long-term fix. But this other hot feature request came in, and there's potentially a lot of money riding on getting it done quickly. So can you put off the long-term fix, maybe a week? Two tops. You can just put the ticket.. yeah that one.. back on the backlog. There you go. Thanks so much!

admax88q 8y ago

On HN its always big bad management who is the cause of every security problem or shoddy piece of engineering. If only that pesky management would screw off then we could do things "properly". You'd be suprieed at how many incompetent engineers there are out there. If "engineer" Alice in your story was actually competent they would never agree to implement the proposed "fix". Its not a fix. To pass it off as one would be malpractice for a real Engineer.

raverbashing 8y ago
And I might add: especially at big companies where being yes men is a higher valued characteristic than being technical
- philpem 8y ago
  Oh, so much this. "I'm sick of you bringing me problems, bring me solutions! And I want that blockchain based big-data cloud IoT solution by the end of the week!"
astrobe_ 8y ago
You live in a nice world, because usually the actual power (and duty) of a dev team member is to advise, not to agree or refuse.
- admax88q 8y ago
  You can always quit. Or force them to fire you for refusing to implement a non fix. In reality though I doubt this narrative even occurred. Some incompetent engineer likely proposed this fix thinking that it was actually a fix. Edit: I see I've been downvoted for this comment. If we were real engineers working on things like cars and bridges we'd actually be held accountable. Take some pride in your work people, this is one of the most in demand professions in today's economy.
  - pjmlp 8y ago
    Quitting assumes the possibility of finding a job around the corner and not having people that depend on you.

skrebbel 8y ago

Engineer Alice is the only person incompetent at their job in this conversation. If "irrepressibly existential sigh" is how you argue security with your bosses then maybe you're not senior enough to be in meetings like these. The Manager and QA Engineers here depend on the expertise of the engineers. If the engineers fail to communicate key details of the situation, then that's on them. Sure, the boss is still at fault for depending on a shitty engineer for decision making, but that's all.

consp 8y ago
> The Manager and QA Engineers here depend on the expertise of the engineers. Both should be sufficiently competent and knowledgeable to know this is a terrible idea, especially in the long run. Otherwise they do not understand the task they have been given. So no, Alice is not the only incompetent one here.
Aeolun 8y ago
If you answer with the existential sigh, you’ve had at least 20 conversations with a similar outcome before. Nobody arrives at a new company that jaded.
- skrebbel 8y ago
  I see your point and I suspect that that's what the author meant. At the same time, I know lots of, well, arrogant nerds who get into "existential sigh" mode as soon as someone with less technical skill than them says something stupid. That's more how I read it.
  - Aeolun 8y ago
    Ah, yes, I see how that is another valid interpretation. I guess for me the word existential is a bit different. For an arrogant developer it would have been something like ‘long-suffering sigh’. Unless, I guess, you feel you are so smart that your entire life is defined by interacting with morons :P

gdfasfklshg4 8y ago

In similar situations I 1. Explain that the 'hack' does not actually fix the problem. This is absolutely the engineers responsibility. In your example Alice does not say that. This has always been sufficient. 2. I make a habit of sending emails or keeping minutes confirming such meetings and the details of who said what.

robertcope 8y ago
This is about all you can do. But you're right. You have to explain it and document it: "It's trivial to change User-Agent, and any decent hacker would probably not be using 'curl' as their User-Agent. We can look at logs from past events to validate this assumption."

supermatt 8y ago

Ironically, this is the change most likely to break things for customers that use curl for automation.

m463 8y ago

This reminds me of the old BOFH posts from days past. The one where the "appeasement engineer" shows up to fulfill the guaranteed response time requirement.

CM30 8y ago

Then someone realises one of their enterprise clients is using curl for a critical system, and said patch has completely broken it. Or more likely, they realise after customer/company calls complaining about how their systems are now busted...

stebann 8y ago

Wow, You got this one!