8 ms·

At my last job I made a feature to deploy arbitrary user code (most likely a trained model in Python) to a subdomain running a JS server that could take input p

At my last job I made a feature to deploy arbitrary user code (most likely a trained model in Python) to a subdomain running a JS server that could take input parameters when called at that URL and would respond with the answer from the model. This effectively cut down DS deployments from potentially days to about 45 seconds. It was seen that be hard and I had a good amount of help from our awesome devops person and worked through a number of issues with the lead dev. When this was delivered everyone loved it, lots of kudos and compliments because it was seen as hard. I felt like a bit of a fraud as I felt I had a lot of help and it really was a team effort, I made efforts to highlight this. I also worked on my own (with the exception of some nginx issues) on a feature to keep an in browser file browse up to date for all users via web sockets. This was really a view on a git repo accessed through the Gitlab api, it was a nightmare to deal with because of the way Git tracks (or doesn’t really) folders. This feature was seen to be easy and I slaved on it for days and eventually weeks because of edge cases and feature creep to no kudos or compliments. Which is fine, it’s my job. It matters, I think, not if the task is easy or hard but if management/product view something as easy or hard.

version_five 5y ago
It's sort of like how it's better to be a firefighter than a maintainer, which is guess is the point. The ease may be similar, but coming in to save everyone from an explosion gets you more points than oiling the valve every day so pressure doesn't build up in the first place. It's a cultural issue that is very hard to combat.
- passivate 5y ago
  Indeed, thank your (figurative and literal) janitors and maintenance staff! :)
- beebeepka 5y ago
  Indeed. Sometimes the saviour is the one responsible for the disaster but companies also need their heroes or at least that's how I interpret certain management decisions. Been on both sides of the fence.
- klyrs 5y ago
  God, this hurts. I pride myself in avoiding deadlines by looking to the future and getting things done before they're needed, so when the time comes, I can focus on polish and integration. But then some dick in management notices that I did this thing in a day, doesn't count the prep-work, and hands me an impossible deadline that's already promised to a customer...
  - fridif 5y ago
    In my fantasy world, the developers are the ones who make promises to the users and thats it
    - jjk166 5y ago
      Hell, I'd settle for managers simply being held responsible when the impossible deadline they set wasn't met.
- ip26 5y ago
  Perhaps you can staff your firefighting crew with the effective maintainers. They've been oiling their valves, so they have time to spare helping fight the fire. This becomes doubly visible.
- ineedasername 5y ago
  it's better to be a firefighter than a maintainer Oh this is very true. I've achieved a pretty good reputation in my org getting called in when things have gone wrong somewhere. Not so much for typical day-to-day work.
- chestervonwinch 5y ago
  Probably has to do with our brains preferring instant gratification over long term payoffs.
- ravenstine 5y ago
  Reminds me of moments I had early in my career where I was the hero for fixing problems I caused. :) That company didn't have code reviews and my bosses weren't very technical, so it was pretty easy to pretend like I wasn't at fault for anything. Sometimes they'd figure out that I caused the problems and even then it didn't seem to really matter. If I had the skills I have now 7 years later, my bosses would probably think they're overpaying me because it wouldn't seem to them like I'm doing anything more of the time.
  - toast0 5y ago
    If you don't get enough praise for not breaking things, get a days since a workplace accident sign, and update it every day. Set it to zero when you break something. Edit to add: you should be able to get some praise (or at least self-praise) at milestones like 30/60/90/180/365 days without an incident.
  - dreadlordbone 5y ago
    Recently one of my clients noticed something we had broken on their website. When they noticed it, we fixed it in a matter of minutes. They were so pumped on the response time they then left us a very glowing public review.
- casey_lang 5y ago
  Thanks for connecting these dots! You've put words to a situation I've experienced before but never fully grokked
- ngc248 5y ago
  true, the problem is when the same person is both the arsonist and the fire fighter. It becomes a habit for some people to show "heroic" effort. No effort is done to see who caused the issue, the spotlight is usually on who fixed it. These "arsonists" also have higher velocity in delivering features since they don't care about quality, so they get into the good books of upper management.
- Lio 5y ago
  I remember John Sterman doing some system dynamics modelling on Oil Refinery maintenance vs actual fire fighting[1]. The results are just as you suggest. Personally speaking I'd like to work at the software equivalent of Toyota. Where the whole organisation is constantly looking for feedback about development processes and how barriers to quality can be removed. I think the problem is that how to get credit for the not having problems in the first place. 1. http://web.mit.edu/jsterman/www/SDG/maint.pdf http://web.mit.edu/jsterman/www/SDG/maint.pdf
- hutzlibu 5y ago
  "but coming in to save everyone from an explosion gets you more points than oiling the valve every day so pressure doesn't build up in the first place. It's a cultural issue that is very hard to combat." The "hero" part of the firefighter comes from the fact, that they actually go towards real danger as part of their job, they often do unpaid and voluntarily. (A big fire out of control, is a very scary thing) While the maintainance worker is usually not in danger, if he does his job right. " It's a cultural issue that is very hard to combat." So I do not really see a issue here to combat (when using this metaphor). The actual firefighters deserve their praise. And if some companies are too shortsighted and praise the people heroically fixing their own mess, then I would use a different metaphor there, to adress this.
  - Dudeman112 5y ago
    What metaphor would you use though? Clearly there are companies praising heroic fixes, while not praising too much regular maintenance.
    - hutzlibu 5y ago
      I guess firefighting is the right metaphor, to fixing big errors in live production environments: it takes a different skill and mindset, to do things under pressure, nothing wrong to praise that. But I think it is wrong, to talk down the job of actual firefighters, because: "Clearly there are companies praising heroic fixes, while not praising too much regular maintenance" that is a seperate issue.
    - jbn 5y ago
      the fire metaphor still works: we have arsonists (that build up tech debt or just bad practices) that turn firefighters (by fixing in emergency mode). Of course not all tech debt is bad, the trick is know when to use it to good effect, not building it up everywhere only to pay it back at the costliest moment. It's all a balancing act.
- booleandilemma 5y ago
  When trouble is solved before it forms, who calls that clever? When there is victory without battle, who talks about bravery? - Sun Tzu
civilized 5y ago
> It matters, I think, not if the task is easy or hard but if management/product view something as easy or hard. I think that's why he's putting scare quotes around "easy", because it's referring to what managers or stakeholders think rather than the real level of challenge.
- AQuantized 5y ago
  He explicitly says this in the article. I think the majority of commenters only read the title.
  - civilized 5y ago
    > I think the majority of commenters only read the title. I've lost count of how many times this has happened. In one case, I went through and read several dozen comments on an article. No exaggeration: all commenters (100%) said nothing whatsoever about the article's text. They were all just holding forth on their own opinions and experiences related to the headline.
    - iamcurious 5y ago
      Maybe if the article was right here instead of a click away, more people would read it.
      - chillfox 5y ago
        I don't think so. Just look at YouTube, lots of people comment on videos without actually watching the video despite it being right there.
    - Shugarl 5y ago
      I've gotta ask : How do you read these quickly ? It'd take me so much time to read even half of the articles from threads I'm somewhat interested in, I can't help but feel that trying to get the gist of it by reading the comments is the most efficient way for me. Well, it would've been, if the people commenting actually read the article.
thehesiod 5y ago
this is what the article talks about, you want to work on things you know are easy, but others think is hard
ppati000 5y ago
Those two problems sound like they could differ substantially in perceived business value. Maybe it’s not how hard management think it is, but how valuable?
- lazide 5y ago
  If management thinks it’s easy and it has big business value - they’re going to understaff it and expect it done tomorrow. Which is gonna hurt someone, and it usually isn’t the execs.
sam0x17 5y ago
I will say as someone who has to wrangle a lot of junior devs -- don't have imposter syndrome on this. The hard part is sticking with the issue and seeing it through to the logical conclusion. Many people have difficulty even conceptualizing the steps that need to be taken in order to solve a problem. If you are smart enough to get help from someone in devops and understand the advice they gave, you're doing the same stuff senior devs and CTOs do on a daily basis. We're all frauds lol
- danuker 5y ago
  computing is a fraud, a CPU is literally a rock that we tricked into thinking
  - vvilliamperez 5y ago
    This gave me a great belly laugh
  - teddyh 5y ago
    If a CPU is thinking, then all rocks are constantly thinking their rocky thoughts, contemplating the immeasurable eons of their existence. A CPU is, then, merely a rock which we have made to think some thoughts which are to our benefit.
  - Ancapistani 5y ago
    You’re oversimplifying. We didn’t just trick a rock into thinking - we beat on it until it was really thin, shoved a little bit of lightning inside of it, then tricked it into thinking.
- 8ytecoder 5y ago
  That's pretty much it. There's a certain category of people with a certain attitude towards debugging. You have to be able to break it down into steps - sometimes from "first principles". Then work up and identify where the issue is and how to resolve it. Sometimes it requires looking at logs and simulating production conditions and other times you may have to understand low level network stack. I still remember the time trying to reproduce and fix a certain issue. We had a 3-node cluster mysql database setup with one main and two replicas. The replicas are read-only. If one replica goes down, reads shouldn't be affected at all. We had an application layer on top of this to coordinate and enforce these requirements. Every which way you look at the code, there's no reason why this shouldn't work. And yet, when one node went down in production under a very particular circumstance with the network, it took down the entire cluster - not that the cluster was not reachable - it wouldn't allow reads and just hang. Tracing it took me down the hole of understanding the precise network issue (fading memory - I think the machine wasn't responding back and tcp kept retrying) + linux tcp setting (I think tcp_delack_min) + a java networking setting + how thread safety was setup in the application (do not use synchronized at the method level). It took a week to diagnose. Recently, there was a minor issue with adapting a sample Spark UDF to work with our setup and it took the developers 3 weeks and they kept giving up after every try.
irrational 5y ago
> if management/product view something as easy or hard. This applies not just to work done, but work to be done. Once the business people asked me to do something that turned out to be extraordinarily difficult. Afterwards we were talking about it and they mentioned that they considered asking me to do it a different way that would have achieved the same goal, but they thought the second option would be harder, so they only asked me to do the first option. The second option turned out to be extremely easy and I could have finished it in 5% of the time option 1 took. This applies outside of programming as well. One time I was building kitchen cabinets for a remodel. I gave my wife 2 options for the doors. She went with the more difficult option. After they were done she admitted that she preferred the other option, but thought it would be harder so she went with the less desirable option to save me work. Facepalm.
- wikibob 5y ago
  I think this is a perfect example of why it is necessary to effectively communicate the relative tradeoffs and costs of option A and B when someone asks you to do something. And to also also questions to dig into what they are really trying to accomplish, and why they are trying to do it that way.
ineedasername 5y ago
but if management/product view something as easy or hard Yep. My job involves a fair amount of coding but it's geared toward data analysis. I might create something in a day with dozens of data points and look like a hero, while another project might take a week and the output is a single number. Actually, yesterday it was two numbers, X & Y. (Upper & lower bounds on something). Deceptively simple in appearance to anyone consuming the information, but behind the scenes it took an order of magnitude more time to make it a dynamic model based on latest available data instead of a static hand-crafted analysis.
cbushko 5y ago
I feel your pain and I have been/am in your shoes. Unfortunately, you will not get the kudos for things you build unless you spend time selling those projects. I personally hate this as I want to spend my energy on building things that make a difference.
yarky 5y ago
> It matters, I think, not if the task is easy or hard but if management/product view something as easy or hard. That's eye opening, thank you! I've once worked on a "hard" statistical model and all I got from my boss' boss was a "who cares" face. Then he was so happy to see our (to my view) dumb web app, he didn't even realized it was the model behind that app that consumed 80% of my time/energy :/
- phist_mcgee 5y ago
  I've had this plenty of times, where praise and accoaldes come down tot he person who made the prettiest or shiniest thing. It's kind of why I moved into frontend development, I always have something so show every week, and management are always happy, since I always look like I have made amazing progress. It's gaming the system, but I would rather this, than what you experienced, where the developer looks like they are wasting time in the eyes of management.
dexwiz 5y ago
In the last 5 years I have probably written 100k+ lines of production code. My most celebrated feature was about 100 lines to store a single flag in a per user store.
- newsyyswen 5y ago
  Way to describe the engineer/business disconnect. You are proud of the complex problems which you have solved over the years. The business is proud of the little widget that made everyone's jobs a bit easier. I've seen people take this lesson to heart and go full Wally.
  - gjvc 5y ago
    >I've seen people take this lesson to heart and go full Wally. Who is Wally?
    - benhurmarcel 5y ago
      https://en.wikipedia.org/wiki/Wally_(Dilbert) https://en.wikipedia.org/wiki/Wally_(Dilbert)
  - dexwiz 5y ago
    It was a little widget, just needed a bit of state persisted across sessions. The pain point was at page load, so about as far as most business pays attention. Also I do feel like it caused me to go full Wally without realizing it.

version_five 5y ago

It's sort of like how it's better to be a firefighter than a maintainer, which is guess is the point. The ease may be similar, but coming in to save everyone from an explosion gets you more points than oiling the valve every day so pressure doesn't build up in the first place. It's a cultural issue that is very hard to combat.

passivate 5y ago
Indeed, thank your (figurative and literal) janitors and maintenance staff! :)
beebeepka 5y ago
Indeed. Sometimes the saviour is the one responsible for the disaster but companies also need their heroes or at least that's how I interpret certain management decisions. Been on both sides of the fence.
klyrs 5y ago
God, this hurts. I pride myself in avoiding deadlines by looking to the future and getting things done before they're needed, so when the time comes, I can focus on polish and integration. But then some dick in management notices that I did this thing in a day, doesn't count the prep-work, and hands me an impossible deadline that's already promised to a customer...
- fridif 5y ago
  In my fantasy world, the developers are the ones who make promises to the users and thats it
  - jjk166 5y ago
    Hell, I'd settle for managers simply being held responsible when the impossible deadline they set wasn't met.
ip26 5y ago
Perhaps you can staff your firefighting crew with the effective maintainers. They've been oiling their valves, so they have time to spare helping fight the fire. This becomes doubly visible.
ineedasername 5y ago
it's better to be a firefighter than a maintainer Oh this is very true. I've achieved a pretty good reputation in my org getting called in when things have gone wrong somewhere. Not so much for typical day-to-day work.
chestervonwinch 5y ago
Probably has to do with our brains preferring instant gratification over long term payoffs.
ravenstine 5y ago
Reminds me of moments I had early in my career where I was the hero for fixing problems I caused. :) That company didn't have code reviews and my bosses weren't very technical, so it was pretty easy to pretend like I wasn't at fault for anything. Sometimes they'd figure out that I caused the problems and even then it didn't seem to really matter. If I had the skills I have now 7 years later, my bosses would probably think they're overpaying me because it wouldn't seem to them like I'm doing anything more of the time.
- toast0 5y ago
  If you don't get enough praise for not breaking things, get a days since a workplace accident sign, and update it every day. Set it to zero when you break something. Edit to add: you should be able to get some praise (or at least self-praise) at milestones like 30/60/90/180/365 days without an incident.
- dreadlordbone 5y ago
  Recently one of my clients noticed something we had broken on their website. When they noticed it, we fixed it in a matter of minutes. They were so pumped on the response time they then left us a very glowing public review.
casey_lang 5y ago
Thanks for connecting these dots! You've put words to a situation I've experienced before but never fully grokked
ngc248 5y ago
true, the problem is when the same person is both the arsonist and the fire fighter. It becomes a habit for some people to show "heroic" effort. No effort is done to see who caused the issue, the spotlight is usually on who fixed it. These "arsonists" also have higher velocity in delivering features since they don't care about quality, so they get into the good books of upper management.
Lio 5y ago
I remember John Sterman doing some system dynamics modelling on Oil Refinery maintenance vs actual fire fighting[1]. The results are just as you suggest. Personally speaking I'd like to work at the software equivalent of Toyota. Where the whole organisation is constantly looking for feedback about development processes and how barriers to quality can be removed. I think the problem is that how to get credit for the not having problems in the first place. 1. http://web.mit.edu/jsterman/www/SDG/maint.pdf http://web.mit.edu/jsterman/www/SDG/maint.pdf
hutzlibu 5y ago
"but coming in to save everyone from an explosion gets you more points than oiling the valve every day so pressure doesn't build up in the first place. It's a cultural issue that is very hard to combat." The "hero" part of the firefighter comes from the fact, that they actually go towards real danger as part of their job, they often do unpaid and voluntarily. (A big fire out of control, is a very scary thing) While the maintainance worker is usually not in danger, if he does his job right. " It's a cultural issue that is very hard to combat." So I do not really see a issue here to combat (when using this metaphor). The actual firefighters deserve their praise. And if some companies are too shortsighted and praise the people heroically fixing their own mess, then I would use a different metaphor there, to adress this.
- Dudeman112 5y ago
  What metaphor would you use though? Clearly there are companies praising heroic fixes, while not praising too much regular maintenance.
  - hutzlibu 5y ago
    I guess firefighting is the right metaphor, to fixing big errors in live production environments: it takes a different skill and mindset, to do things under pressure, nothing wrong to praise that. But I think it is wrong, to talk down the job of actual firefighters, because: "Clearly there are companies praising heroic fixes, while not praising too much regular maintenance" that is a seperate issue.
  - jbn 5y ago
    the fire metaphor still works: we have arsonists (that build up tech debt or just bad practices) that turn firefighters (by fixing in emergency mode). Of course not all tech debt is bad, the trick is know when to use it to good effect, not building it up everywhere only to pay it back at the costliest moment. It's all a balancing act.
booleandilemma 5y ago
When trouble is solved before it forms, who calls that clever? When there is victory without battle, who talks about bravery? - Sun Tzu

civilized 5y ago

> It matters, I think, not if the task is easy or hard but if management/product view something as easy or hard. I think that's why he's putting scare quotes around "easy", because it's referring to what managers or stakeholders think rather than the real level of challenge.

AQuantized 5y ago
He explicitly says this in the article. I think the majority of commenters only read the title.
- civilized 5y ago
  > I think the majority of commenters only read the title. I've lost count of how many times this has happened. In one case, I went through and read several dozen comments on an article. No exaggeration: all commenters (100%) said nothing whatsoever about the article's text. They were all just holding forth on their own opinions and experiences related to the headline.
  - iamcurious 5y ago
    Maybe if the article was right here instead of a click away, more people would read it.
    - chillfox 5y ago
      I don't think so. Just look at YouTube, lots of people comment on videos without actually watching the video despite it being right there.
  - Shugarl 5y ago
    I've gotta ask : How do you read these quickly ? It'd take me so much time to read even half of the articles from threads I'm somewhat interested in, I can't help but feel that trying to get the gist of it by reading the comments is the most efficient way for me. Well, it would've been, if the people commenting actually read the article.

thehesiod 5y ago

this is what the article talks about, you want to work on things you know are easy, but others think is hard

ppati000 5y ago

Those two problems sound like they could differ substantially in perceived business value. Maybe it’s not how hard management think it is, but how valuable?

lazide 5y ago
If management thinks it’s easy and it has big business value - they’re going to understaff it and expect it done tomorrow. Which is gonna hurt someone, and it usually isn’t the execs.

sam0x17 5y ago

I will say as someone who has to wrangle a lot of junior devs -- don't have imposter syndrome on this. The hard part is sticking with the issue and seeing it through to the logical conclusion. Many people have difficulty even conceptualizing the steps that need to be taken in order to solve a problem. If you are smart enough to get help from someone in devops and understand the advice they gave, you're doing the same stuff senior devs and CTOs do on a daily basis. We're all frauds lol

danuker 5y ago
computing is a fraud, a CPU is literally a rock that we tricked into thinking
- vvilliamperez 5y ago
  This gave me a great belly laugh
- teddyh 5y ago
  If a CPU is thinking, then all rocks are constantly thinking their rocky thoughts, contemplating the immeasurable eons of their existence. A CPU is, then, merely a rock which we have made to think some thoughts which are to our benefit.
- Ancapistani 5y ago
  You’re oversimplifying. We didn’t just trick a rock into thinking - we beat on it until it was really thin, shoved a little bit of lightning inside of it, then tricked it into thinking.
8ytecoder 5y ago
That's pretty much it. There's a certain category of people with a certain attitude towards debugging. You have to be able to break it down into steps - sometimes from "first principles". Then work up and identify where the issue is and how to resolve it. Sometimes it requires looking at logs and simulating production conditions and other times you may have to understand low level network stack. I still remember the time trying to reproduce and fix a certain issue. We had a 3-node cluster mysql database setup with one main and two replicas. The replicas are read-only. If one replica goes down, reads shouldn't be affected at all. We had an application layer on top of this to coordinate and enforce these requirements. Every which way you look at the code, there's no reason why this shouldn't work. And yet, when one node went down in production under a very particular circumstance with the network, it took down the entire cluster - not that the cluster was not reachable - it wouldn't allow reads and just hang. Tracing it took me down the hole of understanding the precise network issue (fading memory - I think the machine wasn't responding back and tcp kept retrying) + linux tcp setting (I think tcp_delack_min) + a java networking setting + how thread safety was setup in the application (do not use synchronized at the method level). It took a week to diagnose. Recently, there was a minor issue with adapting a sample Spark UDF to work with our setup and it took the developers 3 weeks and they kept giving up after every try.

irrational 5y ago

> if management/product view something as easy or hard. This applies not just to work done, but work to be done. Once the business people asked me to do something that turned out to be extraordinarily difficult. Afterwards we were talking about it and they mentioned that they considered asking me to do it a different way that would have achieved the same goal, but they thought the second option would be harder, so they only asked me to do the first option. The second option turned out to be extremely easy and I could have finished it in 5% of the time option 1 took. This applies outside of programming as well. One time I was building kitchen cabinets for a remodel. I gave my wife 2 options for the doors. She went with the more difficult option. After they were done she admitted that she preferred the other option, but thought it would be harder so she went with the less desirable option to save me work. Facepalm.

wikibob 5y ago
I think this is a perfect example of why it is necessary to effectively communicate the relative tradeoffs and costs of option A and B when someone asks you to do something. And to also also questions to dig into what they are really trying to accomplish, and why they are trying to do it that way.

ineedasername 5y ago

but if management/product view something as easy or hard Yep. My job involves a fair amount of coding but it's geared toward data analysis. I might create something in a day with dozens of data points and look like a hero, while another project might take a week and the output is a single number. Actually, yesterday it was two numbers, X & Y. (Upper & lower bounds on something). Deceptively simple in appearance to anyone consuming the information, but behind the scenes it took an order of magnitude more time to make it a dynamic model based on latest available data instead of a static hand-crafted analysis.

cbushko 5y ago

I feel your pain and I have been/am in your shoes. Unfortunately, you will not get the kudos for things you build unless you spend time selling those projects. I personally hate this as I want to spend my energy on building things that make a difference.

yarky 5y ago

> It matters, I think, not if the task is easy or hard but if management/product view something as easy or hard. That's eye opening, thank you! I've once worked on a "hard" statistical model and all I got from my boss' boss was a "who cares" face. Then he was so happy to see our (to my view) dumb web app, he didn't even realized it was the model behind that app that consumed 80% of my time/energy :/

phist_mcgee 5y ago
I've had this plenty of times, where praise and accoaldes come down tot he person who made the prettiest or shiniest thing. It's kind of why I moved into frontend development, I always have something so show every week, and management are always happy, since I always look like I have made amazing progress. It's gaming the system, but I would rather this, than what you experienced, where the developer looks like they are wasting time in the eyes of management.

dexwiz 5y ago

In the last 5 years I have probably written 100k+ lines of production code. My most celebrated feature was about 100 lines to store a single flag in a per user store.

newsyyswen 5y ago
Way to describe the engineer/business disconnect. You are proud of the complex problems which you have solved over the years. The business is proud of the little widget that made everyone's jobs a bit easier. I've seen people take this lesson to heart and go full Wally.
- gjvc 5y ago
  >I've seen people take this lesson to heart and go full Wally. Who is Wally?
  - benhurmarcel 5y ago
    https://en.wikipedia.org/wiki/Wally_(Dilbert) https://en.wikipedia.org/wiki/Wally_(Dilbert)
- dexwiz 5y ago
  It was a little widget, just needed a bit of state persisted across sessions. The pain point was at page load, so about as far as most business pays attention. Also I do feel like it caused me to go full Wally without realizing it.