3 ms·

A requirement for 24/7 on-call programmers demonstrates a systemic organizational failure in the design and implementation of robust, well-architected software.

by nupark2 14y ago

A requirement for 24/7 on-call programmers demonstrates a systemic organizational failure in the design and implementation of robust, well-architected software. 37Signals would see significant savings in development and maintenance costs -- and increased customer satisfaction -- if they approached this staffing requirement as a band-aid, not as a final solution, and took a long, considered look at the root cause of this systemic failure.

gchpaco 14y ago
What they're actually doing is conflating programmer and sysadmin here; the tasks they're assigning to the 'on-call programmer' are very similar to the ones I would expect as a sysadmin to get. Things like monitor the service, act as first responder, coordinate fixes; there's precious little that is actual programming there until you get into fixing it, and even there the goal is minimizing downtime, not debugging the change into working on the live system; typically "revert whatever changed" is one of the first tools for this work. An actual need to do programming instantly and with no warning is quite a different proposition, not one that I'm aware of having ever been needed in any of the companies I've worked for.
- sciurus 14y ago
  It sounds like they have a dedicated customer support team and the on-call programmers are the second-level support. Take a look at the examples DHH gives of the work on-call programmers do- "We spend time trying to figure out why emails weren’t delivered (often because they get caught in the client’s spam filter or their inbox is over capacity), or why an import of contacts from Excel is broken (because some formatting isn’t right), or any of the myriad of other issues that arises from having variable input and output from an application that’s been used by millions of people."
doktrin 14y ago
Someone made this point in a direct response to the post, and was quickly dismissed as a troll (as well as being treated to a particularly snarky response from DHH). While I certainly have to think that 37signals knows what they are doing, the need for a 24/7 programmer does sound a little strange. Perhaps it's just a question of semantics, as the post describes dev/ops duties more than anything else.
zavulon 14y ago
Couldn't agree more. For a long time I was a developer at a financial company, where there was a similar rotation for level 2 support programmer. For a week every 5-6 weeks, you could expect phone calls at 3 AM when European users had a problem. An absolute majority of these issues were caused by bad system architecture issues (which, to be fair, in financial company is usually not up to the technical people to solve). And also, today a much better solution is available: instead of requiring people to be on call at nighttime, why not hire people in different time zones, across the world, specifically for the purpose of Level 2 support when your main devs are asleep?
joedev 14y ago
Couldn't agree more. I left an otherwise good job just for this reason: I was hired as a programmer, built a 15+ year career as a programmer, and while I love programming, I hate late-night systems support. Most companies do not staff this way so it was easy to find other jobs where programmers are not expected to be at the company's beck-and-call 24/7.
wvenable 14y ago
I disagree. All software has bugs. I do this sort of support in my company although I don't have to: a ticket will come in or an automatic bug report and if it's not to onerous I'll fix it immediately and update the ticket. Could it wait till tomorrow? Perhaps. But I prefer the customer get back to their work as soon as possible. Most problems of this sort are not very serious -- if you've got serious problems all night then that's a systemic organizational failure)

by nupark2 14y ago

gchpaco 14y ago

What they're actually doing is conflating programmer and sysadmin here; the tasks they're assigning to the 'on-call programmer' are very similar to the ones I would expect as a sysadmin to get. Things like monitor the service, act as first responder, coordinate fixes; there's precious little that is actual programming there until you get into fixing it, and even there the goal is minimizing downtime, not debugging the change into working on the live system; typically "revert whatever changed" is one of the first tools for this work. An actual need to do programming instantly and with no warning is quite a different proposition, not one that I'm aware of having ever been needed in any of the companies I've worked for.

sciurus 14y ago
It sounds like they have a dedicated customer support team and the on-call programmers are the second-level support. Take a look at the examples DHH gives of the work on-call programmers do- "We spend time trying to figure out why emails weren’t delivered (often because they get caught in the client’s spam filter or their inbox is over capacity), or why an import of contacts from Excel is broken (because some formatting isn’t right), or any of the myriad of other issues that arises from having variable input and output from an application that’s been used by millions of people."

doktrin 14y ago

Someone made this point in a direct response to the post, and was quickly dismissed as a troll (as well as being treated to a particularly snarky response from DHH). While I certainly have to think that 37signals knows what they are doing, the need for a 24/7 programmer does sound a little strange. Perhaps it's just a question of semantics, as the post describes dev/ops duties more than anything else.

zavulon 14y ago

Couldn't agree more. For a long time I was a developer at a financial company, where there was a similar rotation for level 2 support programmer. For a week every 5-6 weeks, you could expect phone calls at 3 AM when European users had a problem. An absolute majority of these issues were caused by bad system architecture issues (which, to be fair, in financial company is usually not up to the technical people to solve). And also, today a much better solution is available: instead of requiring people to be on call at nighttime, why not hire people in different time zones, across the world, specifically for the purpose of Level 2 support when your main devs are asleep?

joedev 14y ago

Couldn't agree more. I left an otherwise good job just for this reason: I was hired as a programmer, built a 15+ year career as a programmer, and while I love programming, I hate late-night systems support. Most companies do not staff this way so it was easy to find other jobs where programmers are not expected to be at the company's beck-and-call 24/7.

wvenable 14y ago

I disagree. All software has bugs. I do this sort of support in my company although I don't have to: a ticket will come in or an automatic bug report and if it's not to onerous I'll fix it immediately and update the ticket. Could it wait till tomorrow? Perhaps. But I prefer the customer get back to their work as soon as possible. Most problems of this sort are not very serious -- if you've got serious problems all night then that's a systemic organizational failure)