5 ms·

This is a great article for defining terms. For some reason though, this quote made me laugh out loud: "Excessive availability can become a problem because now

This is a great article for defining terms. For some reason though, this quote made me laugh out loud: "Excessive availability can become a problem because now it’s the expectation. Don’t make your system overly reliable if you don’t intend to commit to it to being that reliable."

sdhgaiojfsa 8y ago
This is a semi-variant of Hyrum's law.
sgift 8y ago
Isn't that one of the reasons for Netflix' chaos monkey? To make sure no one thinks "my dependency will always be there"?
- sargun 8y ago
  That's more part of CHaP: https://medium.com/netflix-techblog/chap-chaos-automation-platform-53e6d528371f https://medium.com/netflix-techblog/chap-chaos-automation-pl... and FIT: https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2 https://medium.com/netflix-techblog/fit-failure-injection-te... -- it's a mechanism to artificially inject errors to understand how upstream dependencies effect your availability.
anonacct37 8y ago
It's funny but true. All observable properties of a system will eventually become hard dependencies for someone.
frowawayz 8y ago
This is actually a serious point, not a joke. New services may be launched with provisional technology to establish or evaluate a market or pricing model. The underlying technology in the initial implementation may have different performance or availability characteristics to what's actually envisioned for the full-scale product, and care has to be taken to actually compensate for this - i.e. introducing synthetic delay/jitter/faults to avoid setting the wrong expectation for the product.
ssdd 8y ago
I guess what meant here is that one should never make mistake of assuming that a highly reliable system can be built. As you start to approach near 100% reliable system, you start experiencing failures that are caused by minute disturbances/flaws in underlying dependencies(hardware, physical location) which can't be controlled. This is what they realized while trying to push the limits to build highly reliable system.
- antpls 8y ago
  > I guess what meant here is that one should never make mistake of assuming that a highly reliable system can be built. Wrong guess imho. It means building highly reliable systems requires knowledge and experiences. Trying to build them and solving the problems step-by-step is one way to understand how it can be achieved.
ksmith14 8y ago
The Google SREs mentioned this in their book; the Chubby locking service had uptime that was so high that folks started to neglect making their own services resilient to Chubby failures: https://landing.google.com/sre/book/chapters/service-level-objectives.html#xref_risk-management_global-chubby-planned-outage https://landing.google.com/sre/book/chapters/service-level-o...
- mav3rick 8y ago
  Still that's bad design on the clients' part. E.g. - Just because malloc "never" fails doesn't mean it can't fail :) so better error check for it.
  - Filligree 8y ago
    Doesn't matter. Engineering around human failure is part of the profession.
    - kjeetgill 8y ago
      That's a beautiful way to put it. I'd read that book.
      - Filligree 8y ago
        Well, I'm a Google SRE so...
  - smcameron 8y ago
    Failure of malloc() might be a bad example to pick because on linux, by default, most distros overcommit, so malloc won't fail, generally. Instead, malloc will succeed allocating the address space just fine, but the RAM will get allocated upon first use, meaning that even though malloc gave you a supposedly valid pointer rather than NULL, actually using that pointer will crash your program.
    - tannhaeuser 8y ago

sdhgaiojfsa 8y ago

This is a semi-variant of Hyrum's law.

sgift 8y ago

Isn't that one of the reasons for Netflix' chaos monkey? To make sure no one thinks "my dependency will always be there"?

sargun 8y ago
That's more part of CHaP: https://medium.com/netflix-techblog/chap-chaos-automation-platform-53e6d528371f https://medium.com/netflix-techblog/chap-chaos-automation-pl... and FIT: https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2 https://medium.com/netflix-techblog/fit-failure-injection-te... -- it's a mechanism to artificially inject errors to understand how upstream dependencies effect your availability.

anonacct37 8y ago

It's funny but true. All observable properties of a system will eventually become hard dependencies for someone.

frowawayz 8y ago

This is actually a serious point, not a joke. New services may be launched with provisional technology to establish or evaluate a market or pricing model. The underlying technology in the initial implementation may have different performance or availability characteristics to what's actually envisioned for the full-scale product, and care has to be taken to actually compensate for this - i.e. introducing synthetic delay/jitter/faults to avoid setting the wrong expectation for the product.

ssdd 8y ago

I guess what meant here is that one should never make mistake of assuming that a highly reliable system can be built. As you start to approach near 100% reliable system, you start experiencing failures that are caused by minute disturbances/flaws in underlying dependencies(hardware, physical location) which can't be controlled. This is what they realized while trying to push the limits to build highly reliable system.

antpls 8y ago
> I guess what meant here is that one should never make mistake of assuming that a highly reliable system can be built. Wrong guess imho. It means building highly reliable systems requires knowledge and experiences. Trying to build them and solving the problems step-by-step is one way to understand how it can be achieved.

ksmith14 8y ago

The Google SREs mentioned this in their book; the Chubby locking service had uptime that was so high that folks started to neglect making their own services resilient to Chubby failures: https://landing.google.com/sre/book/chapters/service-level-objectives.html#xref_risk-management_global-chubby-planned-outage https://landing.google.com/sre/book/chapters/service-level-o...

mav3rick 8y ago
Still that's bad design on the clients' part. E.g. - Just because malloc "never" fails doesn't mean it can't fail :) so better error check for it.
- Filligree 8y ago
  Doesn't matter. Engineering around human failure is part of the profession.
  - kjeetgill 8y ago
    That's a beautiful way to put it. I'd read that book.
    - Filligree 8y ago
      Well, I'm a Google SRE so...
- smcameron 8y ago
  Failure of malloc() might be a bad example to pick because on linux, by default, most distros overcommit, so malloc won't fail, generally. Instead, malloc will succeed allocating the address space just fine, but the RAM will get allocated upon first use, meaning that even though malloc gave you a supposedly valid pointer rather than NULL, actually using that pointer will crash your program.
  - tannhaeuser 8y ago