4 ms·

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS

This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one we used was unavailable. And I think Google has a backup IRC server on AWS, but that might just apocryphal. Always good to make sure you have a backup side channel that has as little to do with your infrastructure as possible.

dilyevsky 3y ago
Afair Google just ran irc on their corp network which was completely separate from prod so I wouldn’t be surprised if it was in a small server room in the office somewhere. > very broadly applicable. I don't see any "this would only apply at Google" in here. One thing I haven’t even heard of anyone else doing was production panic rooms - a secure room with backup vpn to prod
- DaiPlusPlus 3y ago
  > Afair Google just ran irc on their corp network which was completely separate from prod I thought Google didn't have a "corp" network because of their embrace of zero-trust in BeyondCorp?
  - dilyevsky 3y ago
    I don't think zero-trust prohibits network segmentation for redundancy or due to geographical constraints etc. It's mainly about how you gain access.
    - Moto7451 3y ago
      Correct. At an old job we did zero trust corp on a different AWS region and account. The admin site was a different zero trust zone in prod region/account and was supposed to eventually become another AWS account in another region (for cost purposes). I can’t say if any of this was ideal but it did work unobtrusively.
  - shanemhansen 3y ago
    They do. But I'd say most employees go their whole career without needing to do anything that requires a VPN. It's basically all web based access through what is, at the end of the day, a http proxy. SREs need to be ready for stuff like "hey, what if the big proxy we all use to access internal resources is down?".
  - foobazgt 3y ago
    Way back when, for a while, our local (Google) office's internet access ran off the same physical lines as the local prod datacenter traffic. So, any time there was a datacenter traffic outage of any kind, our office was also out. There weren't a lot of outages of that variant, but we knew immediately when one was happening. It's not particularly fun to have all of your access go out concurrently with a prod outage.
- jeffbee 3y ago
  When I was there, the main IRC ran in prod. But it was intentionally a low-dependency system, an actual IRC server instead of something ridiculous like gIRC or IRC-over-stubby.
  - dilyevsky 3y ago
    i think it had a corp dns label but i'm fuzzy on that. yes it could've been a prod instance which would mean you'd need to go to panic room in case corp was down but maybe that was the intention.
gaogao 3y ago
Also make sure your backup channel can scale. Getting a flood of 10,000+ folks over to a dinky IRC server can knock it over easy. Throttling new joinees isn't a panacea either, since there might be someone critical to get on a channel which throttling can complicate.
- londons_explore 3y ago
  Maybe I'm naive, but I would imagine that any raspberry pi could run an IRC server with 10,000 users... Surely 10,000 Users each on average receiving perhaps five 50 byte messages per second (that's a very busy chat room!) is a total bandwidth of 2.5 megabytes per second. And the CPU time to shuffle messages from one TCP connection to another, or encrypt/decrypt 2.5 megabytes per second should be small. There is no complex parsing involved - it is literally just "forward this 50 byte message into this list of TCP connections". If they're all internal/authed users, you can hopefully assume nobody is deliberately DoSing the server with millions of super long messages into busy channels too.
  - xena 3y ago
    IRC servers are single threaded. You have contention at that point.
    - goalieca 3y ago
      Some of the bigger irc channels have thousand users and this was in the day of 128MB of ram.
      - pinkgolem 3y ago
        Sure it's possible. But you need to test... For example there was/is a 1k file descriptor limit per process in a lot of distributions..
  - pests 3y ago
    Let's not forget IRC channels can be split between servers too - don't wanna complicate the backup system too much but this is in the original design of IRC.
strangattractor 3y ago
Brings back memories. I started getting phone calls at 5AM regarding servers going down. I immediately went to email to connect with other members of the team and found my inbox full of thousands of emails from the monitoring service notifying me of the outage. Those were days:)
fragmede 3y ago
The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent. While on call as an SRE for Google on the Internet traffic team, we would naturally default to Google Meet for communications during an indent, but the questions is, what would we do if Google Meet is down? It's a critical-path team, which mean that if I got paged because my team's system is down, it's not improbable that Google Meet (along with Google.com) was down because of it. We needed to have extra layers of backup communications because the first three layers systems were all also Google properties. That is to say, email wouldn't work because Gmail is Google, our work cell phones wouldn't work because they're Google Fi, and my home's ISP was Google Fiber/Webpass. All of which is to confirm that, yes, Google has a backup IRC server for communication. I won't say where, but it's explicitly totally off Google infrastructure for that very reason.
- jeffrallen 3y ago
  Hurricane Electric? He, he, he.
- kortilla 3y ago
  It’s still batshit insane to me that Google Fi had any infrastructure overlap with Gmail. When I was there these grand unified base layer systems everything else was built on top of were a point of pride. From a risk perspective though it was sheer stupidity in retrospect. As Google continues to atrophy and suffers attrition of the original people that built those systems, the probability creeps up more and more that someone will one day cause a catastrophic global outage of systems that should share absolutely no dependencies but do because “it scales”. If Google made auto pilot for airplanes, it would all be on a centralized SaaS with white papers published in academic conferences about the elaborate systems designed to ensure server crashes don’t impact it. Nobody will ever complain about how all of those guarantees depend on chubby, big table, etc.
  - Guvante 3y ago
    At my not Google job we talk about "what happens if a meteor hits a DC". We agree that that is so rare that as long as there are buttons we can push to recover after a reasonable timeframe that is an acceptable risk, we don't need a fully automatic way to recover from that. However your SRE teams needs a way to recover without intervention which is why there is talk of backups. BTW even using different cloud providers isn't enough to avoid a DC outage necessarily. No amount of redundancy can protect you from it beyond a ton of services which intentionally slice off access to the DC leading to the risk of that happening accidentally which is its own risk.
hinkley 3y ago
Before divestiture we had two groups splitting responsibility for IT, so our DevOps team (bleh) only had partial control of things. We were then running in heterogenous mode - one on prem data center and cloud services. One day a production SAN went wonky, taking out our on prem data center. ... and also Atlassian with it. No Jira, no Confluence. Possibly no CI/CD as well. No carefully curated runbooks to recover. Just tribal knowledge. People. Were. Furious. And rightfully so. The 'IT' team lost control of a bunch of things in that little incident. Who puts customer facing and infrastructure eggs into the same basket like that?
l9i 3y ago
> And I think Google has a backup IRC server on AWS, but that might just apocryphal. They (>1) do exist but not on AWS or any other major cloud provider. (Or at least that was the case two years ago when I still worked there.)
cortesoft 3y ago
Yeah, I work for a CDN and we have to make sure our status page provider is not one of our customers.
riwsky 3y ago
God, I feel like such an idiot. All this time I’ve been making fun of Google for having Google Talk, Hangouts, Allo, Duo, Messages, Spaces, Wave, Buzz, Plus, and Meet, I’d never realized that it’s simply a necessary SRE measure at their scale.
- Jensson 3y ago
  Imagine if they accidentally deprecated their last messaging service? Can't let that happen so better build some extra just in case.
- deleted 3y ago
  [deleted]
pulsetracker 3y ago
For small startups, WhatsApp group or phone call can act as a backup.

dilyevsky 3y ago

Afair Google just ran irc on their corp network which was completely separate from prod so I wouldn’t be surprised if it was in a small server room in the office somewhere. > very broadly applicable. I don't see any "this would only apply at Google" in here. One thing I haven’t even heard of anyone else doing was production panic rooms - a secure room with backup vpn to prod

DaiPlusPlus 3y ago
> Afair Google just ran irc on their corp network which was completely separate from prod I thought Google didn't have a "corp" network because of their embrace of zero-trust in BeyondCorp?
- dilyevsky 3y ago
  I don't think zero-trust prohibits network segmentation for redundancy or due to geographical constraints etc. It's mainly about how you gain access.
  - Moto7451 3y ago
    Correct. At an old job we did zero trust corp on a different AWS region and account. The admin site was a different zero trust zone in prod region/account and was supposed to eventually become another AWS account in another region (for cost purposes). I can’t say if any of this was ideal but it did work unobtrusively.
- shanemhansen 3y ago
  They do. But I'd say most employees go their whole career without needing to do anything that requires a VPN. It's basically all web based access through what is, at the end of the day, a http proxy. SREs need to be ready for stuff like "hey, what if the big proxy we all use to access internal resources is down?".
- foobazgt 3y ago
  Way back when, for a while, our local (Google) office's internet access ran off the same physical lines as the local prod datacenter traffic. So, any time there was a datacenter traffic outage of any kind, our office was also out. There weren't a lot of outages of that variant, but we knew immediately when one was happening. It's not particularly fun to have all of your access go out concurrently with a prod outage.
jeffbee 3y ago
When I was there, the main IRC ran in prod. But it was intentionally a low-dependency system, an actual IRC server instead of something ridiculous like gIRC or IRC-over-stubby.
- dilyevsky 3y ago
  i think it had a corp dns label but i'm fuzzy on that. yes it could've been a prod instance which would mean you'd need to go to panic room in case corp was down but maybe that was the intention.

gaogao 3y ago

Also make sure your backup channel can scale. Getting a flood of 10,000+ folks over to a dinky IRC server can knock it over easy. Throttling new joinees isn't a panacea either, since there might be someone critical to get on a channel which throttling can complicate.

londons_explore 3y ago
Maybe I'm naive, but I would imagine that any raspberry pi could run an IRC server with 10,000 users... Surely 10,000 Users each on average receiving perhaps five 50 byte messages per second (that's a very busy chat room!) is a total bandwidth of 2.5 megabytes per second. And the CPU time to shuffle messages from one TCP connection to another, or encrypt/decrypt 2.5 megabytes per second should be small. There is no complex parsing involved - it is literally just "forward this 50 byte message into this list of TCP connections". If they're all internal/authed users, you can hopefully assume nobody is deliberately DoSing the server with millions of super long messages into busy channels too.
- xena 3y ago
  IRC servers are single threaded. You have contention at that point.
  - goalieca 3y ago
    Some of the bigger irc channels have thousand users and this was in the day of 128MB of ram.
    - pinkgolem 3y ago
      Sure it's possible. But you need to test... For example there was/is a 1k file descriptor limit per process in a lot of distributions..
- pests 3y ago
  Let's not forget IRC channels can be split between servers too - don't wanna complicate the backup system too much but this is in the original design of IRC.

strangattractor 3y ago

Brings back memories. I started getting phone calls at 5AM regarding servers going down. I immediately went to email to connect with other members of the team and found my inbox full of thousands of emails from the monitoring service notifying me of the outage. Those were days:)

fragmede 3y ago

The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent. While on call as an SRE for Google on the Internet traffic team, we would naturally default to Google Meet for communications during an indent, but the questions is, what would we do if Google Meet is down? It's a critical-path team, which mean that if I got paged because my team's system is down, it's not improbable that Google Meet (along with Google.com) was down because of it. We needed to have extra layers of backup communications because the first three layers systems were all also Google properties. That is to say, email wouldn't work because Gmail is Google, our work cell phones wouldn't work because they're Google Fi, and my home's ISP was Google Fiber/Webpass. All of which is to confirm that, yes, Google has a backup IRC server for communication. I won't say where, but it's explicitly totally off Google infrastructure for that very reason.

jeffrallen 3y ago
Hurricane Electric? He, he, he.
kortilla 3y ago
It’s still batshit insane to me that Google Fi had any infrastructure overlap with Gmail. When I was there these grand unified base layer systems everything else was built on top of were a point of pride. From a risk perspective though it was sheer stupidity in retrospect. As Google continues to atrophy and suffers attrition of the original people that built those systems, the probability creeps up more and more that someone will one day cause a catastrophic global outage of systems that should share absolutely no dependencies but do because “it scales”. If Google made auto pilot for airplanes, it would all be on a centralized SaaS with white papers published in academic conferences about the elaborate systems designed to ensure server crashes don’t impact it. Nobody will ever complain about how all of those guarantees depend on chubby, big table, etc.
- Guvante 3y ago
  At my not Google job we talk about "what happens if a meteor hits a DC". We agree that that is so rare that as long as there are buttons we can push to recover after a reasonable timeframe that is an acceptable risk, we don't need a fully automatic way to recover from that. However your SRE teams needs a way to recover without intervention which is why there is talk of backups. BTW even using different cloud providers isn't enough to avoid a DC outage necessarily. No amount of redundancy can protect you from it beyond a ton of services which intentionally slice off access to the DC leading to the risk of that happening accidentally which is its own risk.

hinkley 3y ago

Before divestiture we had two groups splitting responsibility for IT, so our DevOps team (bleh) only had partial control of things. We were then running in heterogenous mode - one on prem data center and cloud services. One day a production SAN went wonky, taking out our on prem data center. ... and also Atlassian with it. No Jira, no Confluence. Possibly no CI/CD as well. No carefully curated runbooks to recover. Just tribal knowledge. People. Were. Furious. And rightfully so. The 'IT' team lost control of a bunch of things in that little incident. Who puts customer facing and infrastructure eggs into the same basket like that?

l9i 3y ago

> And I think Google has a backup IRC server on AWS, but that might just apocryphal. They (>1) do exist but not on AWS or any other major cloud provider. (Or at least that was the case two years ago when I still worked there.)

cortesoft 3y ago

Yeah, I work for a CDN and we have to make sure our status page provider is not one of our customers.

riwsky 3y ago

God, I feel like such an idiot. All this time I’ve been making fun of Google for having Google Talk, Hangouts, Allo, Duo, Messages, Spaces, Wave, Buzz, Plus, and Meet, I’d never realized that it’s simply a necessary SRE measure at their scale.

Jensson 3y ago
Imagine if they accidentally deprecated their last messaging service? Can't let that happen so better build some extra just in case.
deleted 3y ago
[deleted]

pulsetracker 3y ago

For small startups, WhatsApp group or phone call can act as a backup.