9 ms·

Being a devops guy these last 2 years I have noticed that besides doing what you just mentioned, I am basically end user support for developers. And through th

Being a devops guy these last 2 years I have noticed that besides doing what you just mentioned, I am basically end user support for developers. And through this I've noticed that developers can be just as bad at communicating issues as any end user I've ever dealt with. Just because they've written the system end users use, does not mean they're better at reporting problems to me, because their problems are about the pipelines, the kubernetes clusters, the containers, anything outside of their programming code basically. So I still need to use all the communication skills I learned decades ago when I was doing end user support for people who didn't even work in IT.

codeduck 5y ago
> I am basically end user support for developers. Pull up a seat at the bar with us.
- anotherhue 5y ago
  And drink whenever the dev works out that it was their fault the entire time.
  - codeduck 5y ago
    enlightenment comes seldom and without thanks, unfortunately.
pm90 5y ago
I’ve seen this and I empathize. I’ve seen it lead to a sort of learned helplessness in developers who are supposed to own their whole stack. I do think it’s unavoidable though. No company will have a truly open stack. There will be customized versions of everything, with caveats only the infrastructure team or developer-tools team are aware of. Naturally these teams become the go to folks to ask questions and resolve issues. I don’t know if a good way around it. Documentation works to an extent. But at some point you have to decide what parts of the system you want to expose to developers, and whenever that happens you become the layer they communicate with rather than the underlying technology.
- jsperson 5y ago
  I’m a consultant data engineer and am on my third engagement in a row where devops has been the hardest part of my job. Every org has their pet stack with their own poorly documented process. Having devops as developer support is critical to getting solutions out the door.
- sjtindell 5y ago
  Agreed. In my experience there are only a handful of companies (Netflix for example) where the pipeline is truly hands off and any developer can deploy to production at any time. We’re talking about the people who literally built Spinnaker, and most teams aren’t that.
  - spion 5y ago
    In my experience thats not true, but it does take education and cultural change to get there.
capn_duck 5y ago
In my experience, its because the tools that are given to the dev team are opaque, poorly documented, made for a general case that rarely is sufficient, and usually locked down with some role based access control to the point where, even if I can figure out whats wrong with some pipeline, I rarely have the access to change something. I have an AWS Codepipeline that I can see running, but the logs from each step are sent to a Cloudwatch instance running in another AWS account, which only the devops folks can view. I have a Jenkins pipeline that was cookie cutter from what the devops team offers. If I wanted to actually grok whats occurring during the pipeline, I'd have to jump between 3 or 4 github repos, and also account for the versions/branches on each one. It really feels worse than back in my websphere days, where in order to see app logs, we had to submit a ticket to the hosting teams and wait 2 days for a .log file to be sent back through email. I'm consistently left wondering if we've actually gained anything.
- ethbr0 5y ago
  I think the above is what happens when a devops tools team doesn't do discovery and user interviews. We've been at a point of software and development where we can build pretty much anything. But people are still making the same mistakes in figuring out what we should build. Fix the cause, not the symptom. Edit: And because I know it's coming... if IT security (or insert other org here) is the reason things can't be better, then it becomes the devops tools team's job to cajole them into doing it anyway, on behalf of their users.
  - 2fast4you 5y ago
    Do you have good resource in how to discovery/user interviews? I’m on a DevOps team working to mature the first few tools we made
    - ethbr0 5y ago
      I'd look at UX books. They've got more formal approaches. Personally? Make a list of all the teams that use your product. Schedule interviews (30 minutes to an hour for initial) with a few that use it the most, and a few that use it less. Ask them what the best and worst parts are. Ask them details about the things they bring up, and prompt about related areas they might be struggling with too. If you get any repeats from team to team, assume those are systemic issues and try and find a solution. The hard part usually isn't figuring out the nuances, but rather realizing that someone wants to do something at all and/or can't because of some restriction. Assume you know nothing about how they actually do what they do and listen closely.
Pxtl 5y ago
> And through this I've noticed that developers can be just as bad at communicating issues I swear to god the next guy who sends me the screenshot of an error log...
- datavirtue 5y ago
  Blame management. They thought it would be a good idea to go into the DevOps PaaS business.
  - xxpor 5y ago
    At a certain point you have to ask how can someone not think about what they're doing at all. If someone asks for a log, what would go through your brain that says, should I give it to the person asking in the most inconvenient format possible? What's the first thing they're going to do with it?
- testplzignore 5y ago
  You haven't known hell until you've been sent a video of someone scrolling through a log file.
- bcrosby95 5y ago
  My boss will text me screenshots of error logs and ask me to help debug his problem via text message. UGH.
- INTPenis 5y ago
  Or another example from last week, a java developer sends me 50 lines of exception text when even I know that ONE of those lines is all you need to explain the problem. And this particular developer is my main problem in that project. So it's not all developers who are bad, it's about the same percentage as regular end users I'd say.
- mrweasel 5y ago
  The number of developers who believe that operations is a support channel for development is truly amazing. How am I suppose to know why your software is broken? You wrote it. In reality it's easy enough most of the time, if you can read logs. Still, there's a weird expectation that operations is a bunch of elite developers, knowledgeable about all programming language, who just choose to do operations because they enjoy getting calls at 3AM

codeduck 5y ago

> I am basically end user support for developers. Pull up a seat at the bar with us.

anotherhue 5y ago
And drink whenever the dev works out that it was their fault the entire time.
- codeduck 5y ago
  enlightenment comes seldom and without thanks, unfortunately.

pm90 5y ago

I’ve seen this and I empathize. I’ve seen it lead to a sort of learned helplessness in developers who are supposed to own their whole stack. I do think it’s unavoidable though. No company will have a truly open stack. There will be customized versions of everything, with caveats only the infrastructure team or developer-tools team are aware of. Naturally these teams become the go to folks to ask questions and resolve issues. I don’t know if a good way around it. Documentation works to an extent. But at some point you have to decide what parts of the system you want to expose to developers, and whenever that happens you become the layer they communicate with rather than the underlying technology.

jsperson 5y ago
I’m a consultant data engineer and am on my third engagement in a row where devops has been the hardest part of my job. Every org has their pet stack with their own poorly documented process. Having devops as developer support is critical to getting solutions out the door.
sjtindell 5y ago
Agreed. In my experience there are only a handful of companies (Netflix for example) where the pipeline is truly hands off and any developer can deploy to production at any time. We’re talking about the people who literally built Spinnaker, and most teams aren’t that.
- spion 5y ago
  In my experience thats not true, but it does take education and cultural change to get there.

capn_duck 5y ago

In my experience, its because the tools that are given to the dev team are opaque, poorly documented, made for a general case that rarely is sufficient, and usually locked down with some role based access control to the point where, even if I can figure out whats wrong with some pipeline, I rarely have the access to change something. I have an AWS Codepipeline that I can see running, but the logs from each step are sent to a Cloudwatch instance running in another AWS account, which only the devops folks can view. I have a Jenkins pipeline that was cookie cutter from what the devops team offers. If I wanted to actually grok whats occurring during the pipeline, I'd have to jump between 3 or 4 github repos, and also account for the versions/branches on each one. It really feels worse than back in my websphere days, where in order to see app logs, we had to submit a ticket to the hosting teams and wait 2 days for a .log file to be sent back through email. I'm consistently left wondering if we've actually gained anything.

ethbr0 5y ago
I think the above is what happens when a devops tools team doesn't do discovery and user interviews. We've been at a point of software and development where we can build pretty much anything. But people are still making the same mistakes in figuring out what we should build. Fix the cause, not the symptom. Edit: And because I know it's coming... if IT security (or insert other org here) is the reason things can't be better, then it becomes the devops tools team's job to cajole them into doing it anyway, on behalf of their users.
- 2fast4you 5y ago
  Do you have good resource in how to discovery/user interviews? I’m on a DevOps team working to mature the first few tools we made
  - ethbr0 5y ago
    I'd look at UX books. They've got more formal approaches. Personally? Make a list of all the teams that use your product. Schedule interviews (30 minutes to an hour for initial) with a few that use it the most, and a few that use it less. Ask them what the best and worst parts are. Ask them details about the things they bring up, and prompt about related areas they might be struggling with too. If you get any repeats from team to team, assume those are systemic issues and try and find a solution. The hard part usually isn't figuring out the nuances, but rather realizing that someone wants to do something at all and/or can't because of some restriction. Assume you know nothing about how they actually do what they do and listen closely.

Pxtl 5y ago

> And through this I've noticed that developers can be just as bad at communicating issues I swear to god the next guy who sends me the screenshot of an error log...

datavirtue 5y ago
Blame management. They thought it would be a good idea to go into the DevOps PaaS business.
- xxpor 5y ago
  At a certain point you have to ask how can someone not think about what they're doing at all. If someone asks for a log, what would go through your brain that says, should I give it to the person asking in the most inconvenient format possible? What's the first thing they're going to do with it?
testplzignore 5y ago
You haven't known hell until you've been sent a video of someone scrolling through a log file.
bcrosby95 5y ago
My boss will text me screenshots of error logs and ask me to help debug his problem via text message. UGH.
INTPenis 5y ago
Or another example from last week, a java developer sends me 50 lines of exception text when even I know that ONE of those lines is all you need to explain the problem. And this particular developer is my main problem in that project. So it's not all developers who are bad, it's about the same percentage as regular end users I'd say.
mrweasel 5y ago
The number of developers who believe that operations is a support channel for development is truly amazing. How am I suppose to know why your software is broken? You wrote it. In reality it's easy enough most of the time, if you can read logs. Still, there's a weird expectation that operations is a bunch of elite developers, knowledgeable about all programming language, who just choose to do operations because they enjoy getting calls at 3AM