4 ms·

> I think another good one is when you are fixing a broken production system, and end up having to revert to running a debugger on the code (involving various h

by tetha 10y ago

> I think another good one is when you are fixing a broken production system, and end up having to revert to running a debugger on the code (involving various hacks to get connected to the production DB/etc), only to realize the entire problem was a simple configuration setting -- which would have been totally obvious and fixed in seconds if there was a bit more logging and/or a better error message. This is one of the reasons why my current shop has added a new rule. When dev gives my ops team a new release, one of the devs comes to our office and stays here during the initial deployment and test. It's a good feeling to see a dev equally confused about weird crap the application does because it doesn't log and it doesn't work and it's just weird. And for bugs in on-premise systems outside our network, the support devs have even more fun. Overall, this is a pretty funny development. Both teams grew a silly amount relatively speaking, and due to stress and other issues we had a couple of tensions between teams. But by now both teams are learning. Ops is annoyed because of some weird config, and dev hates this weird config because it's a huge pain in the code. Oh, teams talked, config was ripped out and simplified. Dev is desperate about a hard bug regarding request headers, because this might need a hotfix and logging and changes. The admin just shrugged and changed the apache access log with said header, and the problem was resolved 15 minutes later. I think this is one of the bigger lessons for me. Never assume something to be hard or easy for another team. My huge issue might be a 2 line change while removing a huge pain. Your issue might be something my config management already does.

gregmac 10y ago
I've had similar experiences, where I learned about something ops was doing -- such as a really complex script to configure something -- that was easily solved by a simple code change. I always pushed for feedback from ops, saying we'd make whatever changes they needed, but they never asked for much. What was happening is they just accepted the software the way it is, and because they were capable of writing this complex script, they didn't see the need to ask for help or changes. I've spent some time recently learning more about the specifics of their deployment and have some changes to make as a result (though since deployment is working, it'll be a slow change since it's more like technical debt and not an immediate problem). However, I think I'm actually going to suggest we do what you're doing, and actually make a developer go sit with ops whenever they deploy a new release. That's a great idea, thanks.
- tetha 10y ago
  Look at it from my perspective. I deploy elasticsearch, hazelcast, mysql, apache, haproxy, maxscale, diamond, and a lot of other stuff. All of these applications except diamond are out of my reach to really change. Thus, it becomes natural to accept the software as it is and build workarounds, because you cannot change these applications. And then there's that one in-house application. It's easy to just lump it into the pile of crap we need to deploy and maintain and handle it in the same way we handle the other services. And at least at this shop, this was reinforced further for some time. We in ops had two choices: Either, we block a release for hard reasons, no matter how close we'd have to look for them, then we'd get some bandaid quickly. Or we could attempt to explain a highly technical issue or a really weird configuration glitch to some entirely non-technical product management, who put it on the latter half of the "never happens" backlog inside the round ticket storage, which is groomed daily by the room services. At this point, just scripting around bullshit costs me like 2 hours to do, while all that other nonsense easily costs days and days. But yeah, all of this is being mediated because devs now see the amount of nonsense we have to do, and they can relate this nonsense with their own testing problems, their own dev setup problems and overall, both teams start to support each other against sales and PM.

by tetha 10y ago

gregmac 10y ago

I've had similar experiences, where I learned about something ops was doing -- such as a really complex script to configure something -- that was easily solved by a simple code change. I always pushed for feedback from ops, saying we'd make whatever changes they needed, but they never asked for much. What was happening is they just accepted the software the way it is, and because they were capable of writing this complex script, they didn't see the need to ask for help or changes. I've spent some time recently learning more about the specifics of their deployment and have some changes to make as a result (though since deployment is working, it'll be a slow change since it's more like technical debt and not an immediate problem). However, I think I'm actually going to suggest we do what you're doing, and actually make a developer go sit with ops whenever they deploy a new release. That's a great idea, thanks.

tetha 10y ago
Look at it from my perspective. I deploy elasticsearch, hazelcast, mysql, apache, haproxy, maxscale, diamond, and a lot of other stuff. All of these applications except diamond are out of my reach to really change. Thus, it becomes natural to accept the software as it is and build workarounds, because you cannot change these applications. And then there's that one in-house application. It's easy to just lump it into the pile of crap we need to deploy and maintain and handle it in the same way we handle the other services. And at least at this shop, this was reinforced further for some time. We in ops had two choices: Either, we block a release for hard reasons, no matter how close we'd have to look for them, then we'd get some bandaid quickly. Or we could attempt to explain a highly technical issue or a really weird configuration glitch to some entirely non-technical product management, who put it on the latter half of the "never happens" backlog inside the round ticket storage, which is groomed daily by the room services. At this point, just scripting around bullshit costs me like 2 hours to do, while all that other nonsense easily costs days and days. But yeah, all of this is being mediated because devs now see the amount of nonsense we have to do, and they can relate this nonsense with their own testing problems, their own dev setup problems and overall, both teams start to support each other against sales and PM.