3 ms·

I tend to agree with the green dude :| It's normal to have a production service replicated on 2 availability regions. The green guy is annoying, because reali

by rvnx 5y ago

I tend to agree with the green dude :| It's normal to have a production service replicated on 2 availability regions. The green guy is annoying, because reality is annoying, and reliability is not about luck, but is about a properly calibrated and tested process. Yes, you need to write monitoring, you cannot run only with "hope". Yes it sucks that a DC can go down. Your particular service is not important if it's down, but having a copy of the production data is essential in case of a catastrophe. Except for the tests that are probably unnecessary, everything else seems to make sense. The peer bonuses are an issue though.

novok 5y ago
Green guy should be making a all of that a one click process to start up a service shell that does all of that for you although. Then as you write it up an automatic linting & rules engine will highlight what is missing before you make a final pull request to get the necessary human approvals, ONCE.
ridaj 5y ago
I think the challenge is you'd expect a company like Google to have more of the setup be automated. If replication and monitoring are such universally good ideas, then why don't they come out of the box?
midasuni 5y ago
Depends what the problem you’re trying to solve is. In my experience the vast majority of business problems do not need that kind of reliability, and if they do they don’t need it deployed in such a Byzantian way.
- zaphar 5y ago
  You say that but then the system goes down and the CTO is walking up to your desk asking why it's down and exactly when can they expect it to be up and don't you know we are bleeding money right now? What you call byzantine an SRE calls necessary complexity to meet the needs of your business.
  - midasuni 5y ago
    No it’s not. There are of course some systems like that, but refit the majority of systems for the majority of companies that’s not the case. If an invoice can’t be paid for 6 hours the world doesn’t collapse. If the CFO can’t get the statistics for his quarterly powerpoint at 3am, it’s not a major problem.
gopher_space 5y ago
A lot of it feels like premature optimization. Like I'm laying down a heavy infrastructure to support change but it's already locking me into certain ways of looking at problems.
- loxias 5y ago
  Great observation. Perhaps the term "premature infrastructure architecture optimization".
- Jensson 5y ago
  It isn't premature though, a service is as robust as its weakest link, so if you let people write crappy services that easily goes down and are hard to get back when they do then you will get a huge amount of outages in major services since they depend on so many small ones.
loxias 5y ago
All of this is true, but I'd wager there's 10, at MOST, entities on the planet that are large enough to warrant this level of ... "architectural overkill". The other 99.99% of us don't need it. I CERTAINLY don't debate that Google, or Amazon, or Facebook, or Netflix, or the phone system, or anything else that touches a noticeable percentage of the human race needs architecture like this to provide "5 9s". But, just like when "big data" became a buzzword, and many people thought their problem needed "big data" approaches to solve, the thought that all but a small minority of entities need this is Simply Wrong. I am reminded of a client doing something with genomics about 9 years ago. They had some over-complicated "new tool" infested approach to solve their "big data" problem, but the run times were taking too long. I was brought in as a consultant to improve it. After I was done, a data processing run that took hours (causing employees to run them overnight) before took minutes, or seconds. What did I do? I got rid of all the complexity. I replaced their expensive cluster with one studly provisioned machine. I replaced their collection of networked Java microservices with 1 non-networked multithreaded C program. I replaced their XML based format for data at rest with something I whipped up, tuned to what they actually needed. Once their "we need big data!" >10TB data set could fit in a single machine's memory, the rest was easy. What used to "require" a cluster of machines and overnight processing could be done interactively, and quickly enough for the scientists to get into a much more productive "flow", doing dozens of runs per day. tl;dr: unless you're google (or google scale) you don't need all this crap. :)

by rvnx 5y ago

novok 5y ago

Green guy should be making a all of that a one click process to start up a service shell that does all of that for you although. Then as you write it up an automatic linting & rules engine will highlight what is missing before you make a final pull request to get the necessary human approvals, ONCE.

ridaj 5y ago

I think the challenge is you'd expect a company like Google to have more of the setup be automated. If replication and monitoring are such universally good ideas, then why don't they come out of the box?

midasuni 5y ago

Depends what the problem you’re trying to solve is. In my experience the vast majority of business problems do not need that kind of reliability, and if they do they don’t need it deployed in such a Byzantian way.

zaphar 5y ago
You say that but then the system goes down and the CTO is walking up to your desk asking why it's down and exactly when can they expect it to be up and don't you know we are bleeding money right now? What you call byzantine an SRE calls necessary complexity to meet the needs of your business.
- midasuni 5y ago
  No it’s not. There are of course some systems like that, but refit the majority of systems for the majority of companies that’s not the case. If an invoice can’t be paid for 6 hours the world doesn’t collapse. If the CFO can’t get the statistics for his quarterly powerpoint at 3am, it’s not a major problem.

gopher_space 5y ago

A lot of it feels like premature optimization. Like I'm laying down a heavy infrastructure to support change but it's already locking me into certain ways of looking at problems.

loxias 5y ago
Great observation. Perhaps the term "premature infrastructure architecture optimization".
Jensson 5y ago
It isn't premature though, a service is as robust as its weakest link, so if you let people write crappy services that easily goes down and are hard to get back when they do then you will get a huge amount of outages in major services since they depend on so many small ones.

loxias 5y ago

All of this is true, but I'd wager there's 10, at MOST, entities on the planet that are large enough to warrant this level of ... "architectural overkill". The other 99.99% of us don't need it. I CERTAINLY don't debate that Google, or Amazon, or Facebook, or Netflix, or the phone system, or anything else that touches a noticeable percentage of the human race needs architecture like this to provide "5 9s". But, just like when "big data" became a buzzword, and many people thought their problem needed "big data" approaches to solve, the thought that all but a small minority of entities need this is Simply Wrong. I am reminded of a client doing something with genomics about 9 years ago. They had some over-complicated "new tool" infested approach to solve their "big data" problem, but the run times were taking too long. I was brought in as a consultant to improve it. After I was done, a data processing run that took hours (causing employees to run them overnight) before took minutes, or seconds. What did I do? I got rid of all the complexity. I replaced their expensive cluster with one studly provisioned machine. I replaced their collection of networked Java microservices with 1 non-networked multithreaded C program. I replaced their XML based format for data at rest with something I whipped up, tuned to what they actually needed. Once their "we need big data!" >10TB data set could fit in a single machine's memory, the rest was easy. What used to "require" a cluster of machines and overnight processing could be done interactively, and quickly enough for the scientists to get into a much more productive "flow", doing dozens of runs per day. tl;dr: unless you're google (or google scale) you don't need all this crap. :)