3 ms·

Having your application reviewed by SREs who are going to support it is a legendary experience. They have no motivation to be gentle. It changes the mindset fr

by praptak 12y ago

Having your application reviewed by SREs who are going to support it is a legendary experience. They have no motivation to be gentle. It changes the mindset from "Failure? Just log an error, restore some 'good'-ish state and move on to the next cool feature." towards "New cool feature? What possible failures will it cause? How about improving logging and monitoring on our existing code instead?"

lamontcg 12y ago
I like making the devs who write the application the SREs who are going to be supporting it and making them carry a pager. If its worth it to the team to write code with edge conditions that will errantly wake them up in the middle of night occasionally they can make that decision to put that tax on their own lives, not someone else's. Then the SREs are in every single code review.
- quanticle 12y ago
  This works only as long as you have a single application, or a bunch of independent applications, each with their own team. With any kind of "platform" or "framework" or shared service architecture, the incentive will be for the devs on the application teams to do as little logging as possible, because what will inevitably happen is that failures and errors with "unknown" causes will be marked as platform failures and will wake the platform team in the middle of the night. At that point, you're back to where you started. I've been in that scenario multiple times, and believe me, there are few things worse than trying to debug someone else's code at three in the morning to try to determine if the page that woke me up was a legitimate platform issue or if it's due to application code that's misbehaving or misusing the platform. Making devs carry pagers certainly helps, but it's a mistake to think that it's a panacea, and that forcing devs to carry pagers will suddenly make them write code with perfect logging and perfect error handling.
  - dredmorbius 12y ago
    Nobody's arguing that it's a panacea. Only that raising awareness of operational issues, reliability, and lost sleep does tend to put feet to the fire.
philsnow 12y ago
In a past life, we had good luck with obligating dev teams to be (co-)oncall for their own systems until it conformed to to an agreed-upon SLA for N months, before taking on oncall for the system. That aligned everybody's interests: devs came to SREs for design review and code review, and were incentivized to not just throw code over the wall.

by praptak 12y ago

lamontcg 12y ago

I like making the devs who write the application the SREs who are going to be supporting it and making them carry a pager. If its worth it to the team to write code with edge conditions that will errantly wake them up in the middle of night occasionally they can make that decision to put that tax on their own lives, not someone else's. Then the SREs are in every single code review.

quanticle 12y ago
This works only as long as you have a single application, or a bunch of independent applications, each with their own team. With any kind of "platform" or "framework" or shared service architecture, the incentive will be for the devs on the application teams to do as little logging as possible, because what will inevitably happen is that failures and errors with "unknown" causes will be marked as platform failures and will wake the platform team in the middle of the night. At that point, you're back to where you started. I've been in that scenario multiple times, and believe me, there are few things worse than trying to debug someone else's code at three in the morning to try to determine if the page that woke me up was a legitimate platform issue or if it's due to application code that's misbehaving or misusing the platform. Making devs carry pagers certainly helps, but it's a mistake to think that it's a panacea, and that forcing devs to carry pagers will suddenly make them write code with perfect logging and perfect error handling.
- dredmorbius 12y ago
  Nobody's arguing that it's a panacea. Only that raising awareness of operational issues, reliability, and lost sleep does tend to put feet to the fire.

philsnow 12y ago

In a past life, we had good luck with obligating dev teams to be (co-)oncall for their own systems until it conformed to to an agreed-upon SLA for N months, before taking on oncall for the system. That aligned everybody's interests: devs came to SREs for design review and code review, and were incentivized to not just throw code over the wall.