4 ms·

Some companies, ones large enough to have serious SRE staffs, have 24 hour staffing of an operations center, and a War Room where people can huddle until a prob

by JJMcJ 5y ago

Some companies, ones large enough to have serious SRE staffs, have 24 hour staffing of an operations center, and a War Room where people can huddle until a problem is fixed. If they are smart, they have ways to roll back offending code. But any company smaller than say a regional bank is going to have trouble affording this. Now, if you using a cloud provider, you can design in rollback. But of course there are surprises. A scenario I just thought up, when you discover on Black Friday that your order system can't handle more than 1048575 (that's 2 to the 20th minus 1, just to pick a slightly plausible number) orders in one day, and this limit is deep in your code and architecture.

cactus2093 5y ago
The trend I've seen in big tech is that microservices architectures kind of negate a lot of this benefit though. Yes, there might be SRE's who are on-call for existential company-wide outages, and ideally they have some general knowledge about some of the biggest systems and common causes of issues. But for any issues that are due to bugs in the code or configuration of one particular service (which is usually most issues), there is a separate on-call rotation for each service. These folks are mostly just regular engineers that are primarily responsible for building features and delivering under time pressure and don't have the luxury of a lot of training & practice dealing with incidents until they happen for real.
lamontcg 5y ago
So after a year of two of oncall at Amazon (it was long enough ago that we were oncall for the whole thing, long before SRE roles were ever invented) the biggest thing I learned was that nobody was dying and it wasn't actually fire-fighting and we shouldn't be using analogies like War Rooms and all these other War/EMS/fire analogies. I've never worked in one of those jobs, but I've actually dragged dead bodies out of the water due to scuba diving accidents and analogizing IT outages to real life fatalities is almost sort of like how we think about holocaust/nazi comparisons and shouldn't use those casually. The industry really needs to chill out a bit about the impact, nobody is generally dying for the kinds of things we're talking about where the website is just down (obviously some IT really is life-affecting, and in finance people can be hauled off to jail by the SEC, but we're generally not talking about those industries here). There are some processes such as accident analysis from aviation and aerospace which are useful, but could be used a bit more dispassionately. Blameless postmortems are also better than blamestorming, although I think there is a bit of a limit (at some point you may really have a bad team producing poor architecture which is constantly knocking the website over and you need to kick someone out, that isn't the purpose of an engineering postmortem though, that should happen at a management level). I'm also weird and I don't really feel a lot of guilt (never raised religious might have something to do with it) and I just naturally understand how "shoulds" are bad and find it mildly appalling how Millennials love to throw the word "shame" around for situations where it really doesn't apply (your sportsball team shouldn't really be "shamed" if they lose). The broader population could do with a reassessment of shame and guilt and perfectionism and "shoulds".

by JJMcJ 5y ago

cactus2093 5y ago

The trend I've seen in big tech is that microservices architectures kind of negate a lot of this benefit though. Yes, there might be SRE's who are on-call for existential company-wide outages, and ideally they have some general knowledge about some of the biggest systems and common causes of issues. But for any issues that are due to bugs in the code or configuration of one particular service (which is usually most issues), there is a separate on-call rotation for each service. These folks are mostly just regular engineers that are primarily responsible for building features and delivering under time pressure and don't have the luxury of a lot of training & practice dealing with incidents until they happen for real.

lamontcg 5y ago

So after a year of two of oncall at Amazon (it was long enough ago that we were oncall for the whole thing, long before SRE roles were ever invented) the biggest thing I learned was that nobody was dying and it wasn't actually fire-fighting and we shouldn't be using analogies like War Rooms and all these other War/EMS/fire analogies. I've never worked in one of those jobs, but I've actually dragged dead bodies out of the water due to scuba diving accidents and analogizing IT outages to real life fatalities is almost sort of like how we think about holocaust/nazi comparisons and shouldn't use those casually. The industry really needs to chill out a bit about the impact, nobody is generally dying for the kinds of things we're talking about where the website is just down (obviously some IT really is life-affecting, and in finance people can be hauled off to jail by the SEC, but we're generally not talking about those industries here). There are some processes such as accident analysis from aviation and aerospace which are useful, but could be used a bit more dispassionately. Blameless postmortems are also better than blamestorming, although I think there is a bit of a limit (at some point you may really have a bad team producing poor architecture which is constantly knocking the website over and you need to kick someone out, that isn't the purpose of an engineering postmortem though, that should happen at a management level). I'm also weird and I don't really feel a lot of guilt (never raised religious might have something to do with it) and I just naturally understand how "shoulds" are bad and find it mildly appalling how Millennials love to throw the word "shame" around for situations where it really doesn't apply (your sportsball team shouldn't really be "shamed" if they lose). The broader population could do with a reassessment of shame and guilt and perfectionism and "shoulds".