5 ms·

I'm a freelancer who always winds up as "the ops person", and I'm just waiting for my copy of this book to come from Amazon. I watched a video of a talk about t

I'm a freelancer who always winds up as "the ops person", and I'm just waiting for my copy of this book to come from Amazon. I watched a video of a talk about this book---sadly I can't find it on Youtube anymore. A HN comment linked to it last week. It was interesting but I'm hoping the book will have a lot more meat. A few thoughts: > Google places a 50% cap on the amount of “ops” work for SREs: Upper bound. Actual amount of ops work is expected to be much lower I didn't catch the "upper bound" part from the talk. Good to know! I really enjoy being a developer-who-does-ops. I wouldn't want to be a sysadmin, and 50% ops is probably my limit for happiness. > I don’t really understand how this is an example of circumventing the dev/ops split I felt the same way from the Youtube talk. I think there must be a lot behind the SRE role that makes it successful or not: culture, policies, who you hire, how you train, etc. Also I feel like the best sysadmins have been encouraging coding and automation for a long time, e.g. Thomas Limoncelli. But I've certainly been on the "dev" side of the dev-vs-sysadmin fight before, and it makes sense to be seeking ways to improve things. > Error budget. 100% is the wrong reliability target for basically everything I think I saw just this month that Google Apps uptime is 99.95%? Some major Google service. I remember in the early 2000s everyone cared about "5 9s", and I feel like for most of us that is just not worth the effort. > Chubby was so reliable that teams were incorrectly assuming that it would never be down This reminds me of Nygard's point in Release It! that your theoretical best SLA is the product of your dependencies' SLAs, e.g. 0.999 * 0.999 = 0.998. But in the world of microservices, this logic seems likely to make you underestimate your uptime. Also I think Feynman's remarks about the Challenger accident apply here: if you are building a new product with, say, 5 microservices, you don't know the reliability of any of them yet. It's dubious to estimate low-frequency events based on "it's hasn't happened yet." Thanks for sharing your notes. I'm envious you've already got a copy. :-)

packetslave 11y ago
Ben's talk from SREcon 2014 is pretty good at giving the underlying philosophy behind SRE: https://www.usenix.org/conference/srecon14/technical-sessions/presentation/keys-sre https://www.usenix.org/conference/srecon14/technical-session...
cabacon 11y ago
Regarding "theoretical best", I think that is "in the absence of mitigations". I think you can build a service with a higher SLA than one of its dependencies, but only if you recognize that impedance mismatch and build in defenses. As a contrived example, if you've got a microservice that provides data FOO about a request that isn't actually end-user critical, you can mitigate your dependency on it by allowing your top-level request to succeed even if the FOO data is missing. Or maybe you can paper over blips of unavailability with cached data. But, yes, know what you depend on and how reliable they are, then see if you need to take more action than that if your target is higher than the computed target.
- asuffield 11y ago
  (Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google) Building reliable services out of unreliable dependencies is a part of what we do. At the lowest level, we're building services out of individual machines that have a relatively high rate of failure, and the same basic principles can be applied at every layer of the stack: make a bunch of copies, and make sure their failure modes are uncorrelated.
pjungwir 11y ago
Here is the comment linking to the talk I mentioned: https://news.ycombinator.com/item?id=11448482 https://news.ycombinator.com/item?id=11448482
geocar 11y ago
> I feel like for most of us that is just not worth the effort. I feel like the exact opposite. What exactly is going to happen when your page doesn't load? If you're Google or Amazon, the user will try again. Otherwise the user is going to hit the back button.
- tamana 11y ago
  What is the bottom line cost of losing 0.05% of your customers?
rhizome 11y ago
In the early 2000s all the cool people were running Sun E450s, mod_* and Oracle, which makes 5 9s a little more reasonable than we see today with multiple layers of things to go wrong on less-managed platforms. I'm not throwing shade on any particular technologies, I'm just saying that the deployment and configuration management side of things these days is more complicated.

packetslave 11y ago

Ben's talk from SREcon 2014 is pretty good at giving the underlying philosophy behind SRE: https://www.usenix.org/conference/srecon14/technical-sessions/presentation/keys-sre https://www.usenix.org/conference/srecon14/technical-session...

cabacon 11y ago

Regarding "theoretical best", I think that is "in the absence of mitigations". I think you can build a service with a higher SLA than one of its dependencies, but only if you recognize that impedance mismatch and build in defenses. As a contrived example, if you've got a microservice that provides data FOO about a request that isn't actually end-user critical, you can mitigate your dependency on it by allowing your top-level request to succeed even if the FOO data is missing. Or maybe you can paper over blips of unavailability with cached data. But, yes, know what you depend on and how reliable they are, then see if you need to take more action than that if your target is higher than the computed target.

asuffield 11y ago
(Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google) Building reliable services out of unreliable dependencies is a part of what we do. At the lowest level, we're building services out of individual machines that have a relatively high rate of failure, and the same basic principles can be applied at every layer of the stack: make a bunch of copies, and make sure their failure modes are uncorrelated.

pjungwir 11y ago

Here is the comment linking to the talk I mentioned: https://news.ycombinator.com/item?id=11448482 https://news.ycombinator.com/item?id=11448482

geocar 11y ago

> I feel like for most of us that is just not worth the effort. I feel like the exact opposite. What exactly is going to happen when your page doesn't load? If you're Google or Amazon, the user will try again. Otherwise the user is going to hit the back button.

tamana 11y ago
What is the bottom line cost of losing 0.05% of your customers?

rhizome 11y ago

In the early 2000s all the cool people were running Sun E450s, mod_* and Oracle, which makes 5 9s a little more reasonable than we see today with multiple layers of things to go wrong on less-managed platforms. I'm not throwing shade on any particular technologies, I'm just saying that the deployment and configuration management side of things these days is more complicated.