9 ms·

As a member of a core infra/"foundation" team, the biggest drain on my soul is the number of other engineers that are helpless, or never learned how to find sol

by jreese 5y ago

As a member of a core infra/"foundation" team, the biggest drain on my soul is the number of other engineers that are helpless, or never learned how to find solutions on their own. They never search wikis, look for similar posts on internal groups, or even read the error message from the tool that tells them exactly how to fix the problem they're asking about. When the culture has become "google everything", but you can't google for internal tools/tech problems, suddenly folks have no idea how to read error messages, debug a stack trace, or solve any problem without hand-holding from a senior engineer. I've been at BigCo for nine years now, and it has only gotten worse as the size of the company has grown exponentially.

tppiotrowski 5y ago
I agree with this and would add Slack to the equation. It's just faster to ping a senior engineer on Slack than to spend 5 minutes looking into an issue. I've also seen new hires burn an entire week because they were afraid to ask a question. But I think the balance has definitely shifted towards asking too much help...
- jreese 5y ago
  When I started at BigCo, the recommended policy was to spend two hours trying to solve it yourself, including searching the wiki, debugging code, etc. If you hadn't made any progress in that time period, then ask someone from your own team or make a post in a related internal support group. Only after exhausting that route, and getting no help, should you escalate to the team/oncall that owns the tool/service you are having problems with. It seems that culture has been lost.
  - noisy_boy 5y ago
    I still do this and I don't think this is entirely about culture. Some people do the legwork and some take the laziest route. Sure if the technical tools at hand make it that much hard to do the due diligence, maybe promptly reaching out for help can be (somewhat) justified, but I don't think that is usually the case.
- PikachuEXE 5y ago
  Here is my balance: Study a bit, WRITE my discoveries down Craft a message with issues and background clear to readers (that includes myself when I forgot about this) Send message to selected recipients, continue studying depends on free time available and issue importance
ptmcc 5y ago
I know your frustration and pain, but as someone frequently on the other side of the equation trying to seek out answers, it is often difficult-to-impossible. Just this past week I was working through some infra setup challenges, I had to piece together half a dozen different documents, several README files, and a few archived slack threads. These sources often are outdated or even contradicted each other. Even after all that and some earnest trial and error I still had to ask for help. It's fine to want to point to the docs, but when you do that you better have good docs you're pointing to. As an infra team, often your "API" is your docs and templates. Searching slack threads and piecing together bad docs isn't a scalable solution for every dev to repeat individually. But neither is pinging an individual with every question. Teams need to invest in good documentation.
- Aeolun 5y ago
  No, no. This sounds completely different. You searched out all the information you could, and still had no idea. That’s great (I mean, not great, ideally the necessary information would have been available, but that’s on the people providing the information). The problem is people that have not searched or tried at all, and then ask their senior to please do their job for them.
  - draugadrotten 5y ago
    There is a classic article about this issue which is from Harvard Business Review in 1974 and is about monkeys. a Must Read. https://www.academia.edu/36290372/Monkey https://www.academia.edu/36290372/Monkey https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxhNTE4NDU2Nzg0NHxneDoyOGM1OTE1NDc2OWM5Yzkz https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsd...
    - Aeolun 5y ago
      Hah, this is great. I think I’ll start telling people to keep their monkeys :)
yodsanklai 5y ago
> I've been at BigCo for nine years now, and it has only gotten worse as the size of the company has grown exponentially. You should try to change team or company then. Don't believe all SWEs suck. There are pretty good coders out there :)
notenoughbeans 5y ago
I am working at company that just won a contract with a BigCo and it has to be one of the most frustrating experiences I have ever had. I was asked to test a configuration change and I was able to demonstrate that it worked locally in 15 minutes. Unfortunately, testing locally isn't enough so now I need to test it in the lab, but BigCo is so compartmentalised that it can weeks even to organise a meeting where all the stakeholder can get together and talk about what needed to be done to get the configuration changed in the lab. We've had working sessions where the moment there was an issue, the developers at BigCo said that they need to talk to someone on some team and they wrap up the meeting for the day -- even if we had just got started. It's gotten to the point where we will have project managers in the working session -- essentially babysitting -- to stop the devs from just giving up. The amount of stonewalling I've seen is insane. We were trying to address an issue with our integration and I was told that a particular configuration on the client side was impossible, or maybe that we needed a new license, or perhaps we needed to contract to another vendor and it's going to cost half a million dollars, or whatever other excuses they could come up with. So I asked them what product BigCo used, I looked up the documentation, I sent an email with 12 steps on how to make the configuration change, I got on a video call with my contact at BigCo (and his boss to make sure he showed up) and I literally walked him through the setting up the configuration. I just don't understand how you get to the point where that's business as usual.
- Aeolun 5y ago
  > I just don't understand how you get to the point where that's business as usual. Simple, just get 100 junior developers working on the same product, and make the Project Managers’s manage them.
hsn915 5y ago
Think about it from the other side: Dealing with the infrastructure is your job. You know the ins and outs. It doesn't seem complicated to you. For someone who is dealing with the application logic, having to context switch to infrastructure is painful. I have no idea how you have set things up. When I read your documentation I'm just baffled at the sheer amount of complexity. I don't want to deal with it. It's not my full time job. I have other things to worry about. I have no mental space to deal with this headache. I will just ask you so I can move on to do my actual job - which is already taking up over 90% of my mental capacity. When I setup my own infrastructure, I make it as simple and stupid as possible, precisely because I have no mental slack to deal with all the complexity that I see devops people deal with on a daily basis. At most jobs I've been to, the complexity I've seen in infrastructure is mind boggling. My intuition is that this complexity is not actually needed if you remove all the cruft and think from first principles about what the actual problem is you are trying to solve. However, people in devops seem to enjoy building and maintaining this kind of complexity, and even take pride in how everything is orchestrated. To me it's just a headache.
- jreese 5y ago
  The drain I am talking about is a complete lack of due diligence on behalf of those asking for my help (and my limited time/attention). When the error message contains a clear message and a URL with step-by-step instructions, and the person in question just pastes the full error message including the URL, and asks "what do I do?" When a problem can be solved by responding with "please follow the instructions in the error message you sent me". When someone makes an internal group post asking "how do I do X?" and the pinned post that was right below the submit button contains a summary and link to detailed step-by-step instructions titled "how to X".
  - deleted 5y ago
    [deleted]
  - draugadrotten 5y ago
    I've been in your shoes and I've accepted this fate. I am the expert. I know things. I choose to change my viewpoint to taking pride in making the best possible explanation that serves the users' need. When they ask me "what do I do?" I no longer snicker or sigh that they don't know or that they are too lazy to find out. Instead I copy-paste the answer they need. It takes me seconds, perhaps it saves them thinking. Thinking seems hard to some people. I choose to believe I save the company some time when I think for them. Yes, they do not learn when they are spoon-fed the answers. Yes, they could have figured it out. So what if they didn't? They are producing whatever they are supposed to produce. When I get too tired of the same questions, I change jobs. I am in a C-level position now, and somehow the questions from other C-level managers are similar. They have no idea how they should deal with things from my business area, and that's okey. I'm here to help. I help. When I get too tired of the same questions, I will change jobs.
pif 5y ago
> engineers that are helpless, or never ... or even read the error message from the tool If they don't read the error messages from their tools, they are not engineers: they are idiots.

by jreese 5y ago

tppiotrowski 5y ago

I agree with this and would add Slack to the equation. It's just faster to ping a senior engineer on Slack than to spend 5 minutes looking into an issue. I've also seen new hires burn an entire week because they were afraid to ask a question. But I think the balance has definitely shifted towards asking too much help...

jreese 5y ago
When I started at BigCo, the recommended policy was to spend two hours trying to solve it yourself, including searching the wiki, debugging code, etc. If you hadn't made any progress in that time period, then ask someone from your own team or make a post in a related internal support group. Only after exhausting that route, and getting no help, should you escalate to the team/oncall that owns the tool/service you are having problems with. It seems that culture has been lost.
- noisy_boy 5y ago
  I still do this and I don't think this is entirely about culture. Some people do the legwork and some take the laziest route. Sure if the technical tools at hand make it that much hard to do the due diligence, maybe promptly reaching out for help can be (somewhat) justified, but I don't think that is usually the case.
PikachuEXE 5y ago
Here is my balance: Study a bit, WRITE my discoveries down Craft a message with issues and background clear to readers (that includes myself when I forgot about this) Send message to selected recipients, continue studying depends on free time available and issue importance

ptmcc 5y ago

I know your frustration and pain, but as someone frequently on the other side of the equation trying to seek out answers, it is often difficult-to-impossible. Just this past week I was working through some infra setup challenges, I had to piece together half a dozen different documents, several README files, and a few archived slack threads. These sources often are outdated or even contradicted each other. Even after all that and some earnest trial and error I still had to ask for help. It's fine to want to point to the docs, but when you do that you better have good docs you're pointing to. As an infra team, often your "API" is your docs and templates. Searching slack threads and piecing together bad docs isn't a scalable solution for every dev to repeat individually. But neither is pinging an individual with every question. Teams need to invest in good documentation.

Aeolun 5y ago
No, no. This sounds completely different. You searched out all the information you could, and still had no idea. That’s great (I mean, not great, ideally the necessary information would have been available, but that’s on the people providing the information). The problem is people that have not searched or tried at all, and then ask their senior to please do their job for them.
- draugadrotten 5y ago
  There is a classic article about this issue which is from Harvard Business Review in 1974 and is about monkeys. a Must Read. https://www.academia.edu/36290372/Monkey https://www.academia.edu/36290372/Monkey https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxhNTE4NDU2Nzg0NHxneDoyOGM1OTE1NDc2OWM5Yzkz https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsd...
  - Aeolun 5y ago
    Hah, this is great. I think I’ll start telling people to keep their monkeys :)

yodsanklai 5y ago

> I've been at BigCo for nine years now, and it has only gotten worse as the size of the company has grown exponentially. You should try to change team or company then. Don't believe all SWEs suck. There are pretty good coders out there :)

notenoughbeans 5y ago

I am working at company that just won a contract with a BigCo and it has to be one of the most frustrating experiences I have ever had. I was asked to test a configuration change and I was able to demonstrate that it worked locally in 15 minutes. Unfortunately, testing locally isn't enough so now I need to test it in the lab, but BigCo is so compartmentalised that it can weeks even to organise a meeting where all the stakeholder can get together and talk about what needed to be done to get the configuration changed in the lab. We've had working sessions where the moment there was an issue, the developers at BigCo said that they need to talk to someone on some team and they wrap up the meeting for the day -- even if we had just got started. It's gotten to the point where we will have project managers in the working session -- essentially babysitting -- to stop the devs from just giving up. The amount of stonewalling I've seen is insane. We were trying to address an issue with our integration and I was told that a particular configuration on the client side was impossible, or maybe that we needed a new license, or perhaps we needed to contract to another vendor and it's going to cost half a million dollars, or whatever other excuses they could come up with. So I asked them what product BigCo used, I looked up the documentation, I sent an email with 12 steps on how to make the configuration change, I got on a video call with my contact at BigCo (and his boss to make sure he showed up) and I literally walked him through the setting up the configuration. I just don't understand how you get to the point where that's business as usual.

Aeolun 5y ago
> I just don't understand how you get to the point where that's business as usual. Simple, just get 100 junior developers working on the same product, and make the Project Managers’s manage them.

hsn915 5y ago

Think about it from the other side: Dealing with the infrastructure is your job. You know the ins and outs. It doesn't seem complicated to you. For someone who is dealing with the application logic, having to context switch to infrastructure is painful. I have no idea how you have set things up. When I read your documentation I'm just baffled at the sheer amount of complexity. I don't want to deal with it. It's not my full time job. I have other things to worry about. I have no mental space to deal with this headache. I will just ask you so I can move on to do my actual job - which is already taking up over 90% of my mental capacity. When I setup my own infrastructure, I make it as simple and stupid as possible, precisely because I have no mental slack to deal with all the complexity that I see devops people deal with on a daily basis. At most jobs I've been to, the complexity I've seen in infrastructure is mind boggling. My intuition is that this complexity is not actually needed if you remove all the cruft and think from first principles about what the actual problem is you are trying to solve. However, people in devops seem to enjoy building and maintaining this kind of complexity, and even take pride in how everything is orchestrated. To me it's just a headache.

jreese 5y ago
The drain I am talking about is a complete lack of due diligence on behalf of those asking for my help (and my limited time/attention). When the error message contains a clear message and a URL with step-by-step instructions, and the person in question just pastes the full error message including the URL, and asks "what do I do?" When a problem can be solved by responding with "please follow the instructions in the error message you sent me". When someone makes an internal group post asking "how do I do X?" and the pinned post that was right below the submit button contains a summary and link to detailed step-by-step instructions titled "how to X".
- deleted 5y ago
  [deleted]
- draugadrotten 5y ago
  I've been in your shoes and I've accepted this fate. I am the expert. I know things. I choose to change my viewpoint to taking pride in making the best possible explanation that serves the users' need. When they ask me "what do I do?" I no longer snicker or sigh that they don't know or that they are too lazy to find out. Instead I copy-paste the answer they need. It takes me seconds, perhaps it saves them thinking. Thinking seems hard to some people. I choose to believe I save the company some time when I think for them. Yes, they do not learn when they are spoon-fed the answers. Yes, they could have figured it out. So what if they didn't? They are producing whatever they are supposed to produce. When I get too tired of the same questions, I change jobs. I am in a C-level position now, and somehow the questions from other C-level managers are similar. They have no idea how they should deal with things from my business area, and that's okey. I'm here to help. I help. When I get too tired of the same questions, I will change jobs.

pif 5y ago

> engineers that are helpless, or never ... or even read the error message from the tool If they don't read the error messages from their tools, they are not engineers: they are idiots.