3 ms·

I'm currently on-call, essentially, for code I didn't write. (Not by choice, by necessity.) We have a subsystem that is currently emitted millions of logs per

I'm currently on-call, essentially, for code I didn't write. (Not by choice, by necessity.) We have a subsystem that is currently emitted millions of logs per hour, It's eating up most of the available compute. In a separate incident, it racked up a few thousand dollar bill by making millions of API calls that all failed. It clearly has issues. But I'm not the primary dev: I have no familiarity with the code base, I have little idea what it is doing (and yes, I've asked). As I'm not a dev of the code (and have no time to become one — our agile sprint planning will never allow time for that, and, since I'm not one of the devs) I'm not able to add the information I need to the code to get the insights into answering "why is it eating up most of the compute?". > The people who designed your car don’t have to change the oil in it. No, but when the car fails to operate as designed, those people need to figure out the why. Also, a mechanic has an understanding of how the car is built, and how it functions. In software, the only people that have that are the devs.

yuribro 6y ago
> But I'm not the primary dev: I have no familiarity with the code base, I have little idea what it is doing (and yes, I've asked). As I'm not a dev of the code (and have no time to become one — our agile sprint planning will never allow time for that, and, since I'm not one of the devs) I'm not able to add the information I need to the code to get the insights into answering "why is it eating up most of the compute?". It looks like you identified the root causes of the problem here: the fact that you're not the dev doesn't have to be a problem. It's the fact that it's under-documented & has bad metrics. While everyone understands that we will always have bugs and issues (at least while we keep working in the current paradigm for software development), having good designs, documentation and metrics is attainable. It just has to be prioritized by management. > "our agile sprint planning will never allow time for that" Sounds like those who call the shots either don't understand the cost of not doing these things, or believe that it's more cost effective not to do them.
hyko 6y ago
I'm currently on-call[...]Not by choice, by necessity. I have no familiarity with the code base, I have little idea what it is doing (and yes, I've asked) It sounds like you’re being asked to do a job without the tools you need to do it (i.e. supporting documentation, a runbook etc.). I obviously don’t know the circumstances, but the organisation needs to resolve those issues so you can do an effective job. when the car fails to operate as designed, those people need to figure out why I agree completely. The team responsible for the codebase should be fixing the bugs.
rualca 6y ago
> As I'm not a dev of the code (and have no time to become one — our agile sprint planning will never allow time for that, and, since I'm not one of the devs) I'm not able to add the information I need to the code to get the insights into answering "why is it eating up most of the compute?". But as oncall, that's not your job, is it? Being oncall means you are the primary point of contact for your team regarding any issue involving how your product reaches the public. You take the lead identifying issues and finding ways to mitigate how problems impact users. Yet, that doesn't mean you should be attaching debuggers to running processes and adding breakpoints here and there. You are expected to avoid downtime, meet service levels, and coordinate with all teams to fix operational issues and increase code quality. If you're not the primary dev and you stumble on an issue, you are expected to file a ticket and bring it to the attention of anyone who is in a position to address the issue.

yuribro 6y ago

> But I'm not the primary dev: I have no familiarity with the code base, I have little idea what it is doing (and yes, I've asked). As I'm not a dev of the code (and have no time to become one — our agile sprint planning will never allow time for that, and, since I'm not one of the devs) I'm not able to add the information I need to the code to get the insights into answering "why is it eating up most of the compute?". It looks like you identified the root causes of the problem here: the fact that you're not the dev doesn't have to be a problem. It's the fact that it's under-documented & has bad metrics. While everyone understands that we will always have bugs and issues (at least while we keep working in the current paradigm for software development), having good designs, documentation and metrics is attainable. It just has to be prioritized by management. > "our agile sprint planning will never allow time for that" Sounds like those who call the shots either don't understand the cost of not doing these things, or believe that it's more cost effective not to do them.

hyko 6y ago

I'm currently on-call[...]Not by choice, by necessity. I have no familiarity with the code base, I have little idea what it is doing (and yes, I've asked) It sounds like you’re being asked to do a job without the tools you need to do it (i.e. supporting documentation, a runbook etc.). I obviously don’t know the circumstances, but the organisation needs to resolve those issues so you can do an effective job. when the car fails to operate as designed, those people need to figure out why I agree completely. The team responsible for the codebase should be fixing the bugs.

rualca 6y ago

> As I'm not a dev of the code (and have no time to become one — our agile sprint planning will never allow time for that, and, since I'm not one of the devs) I'm not able to add the information I need to the code to get the insights into answering "why is it eating up most of the compute?". But as oncall, that's not your job, is it? Being oncall means you are the primary point of contact for your team regarding any issue involving how your product reaches the public. You take the lead identifying issues and finding ways to mitigate how problems impact users. Yet, that doesn't mean you should be attaching debuggers to running processes and adding breakpoints here and there. You are expected to avoid downtime, meet service levels, and coordinate with all teams to fix operational issues and increase code quality. If you're not the primary dev and you stumble on an issue, you are expected to file a ticket and bring it to the attention of anyone who is in a position to address the issue.