3 ms·

Something I hope to eventually hear is the solution to the full cold start problem. Most giant custom-stack companies have circular dependencies on core infras

Something I hope to eventually hear is the solution to the full cold start problem. Most giant custom-stack companies have circular dependencies on core infrastructure. Software-defined networking needs some software running to start routing packets again, diskless machines need some storage to boot from, authentication services need access to storage to start handing out service credentials to bootstrap secure authz, etc. It's currently handled by running many independent regions so that data centers can be brought up from fully dark by bootstrapping them from existing infra. I haven't heard of anyone bringing the stack up from a full power-off situation. Even when Facebook completely broke its production network a couple years ago the machines stayed on and running and had some internal connectivity. This matters to everyone because while cloud resources are great at automatic restarts and fault recovery there's no guarantee that AWS, GCP, and friends would come back up after, e.g., a massive solar storm that knocks out the grid worldwide for long enough to run the generators down. My guess is that there are some dedicated small DCs with exceptional backup power and the ability to be fully isolated from grid surges (flywheel transformers or similar).

Gh0stRAT 3y ago
Azure has procedures in place to prevent circular dependencies, and regularly exercises them when bringing new regions online. IIRC some of the information about their approach is considered sensitive so I won't elaborate further.
- jeremyjh 3y ago
  Are you saying they can bring a new data center online without any connectivity to the rest of their infrastructure? GP isn't concerned about turning on one data center, they are concerned about turning them all on at the same time, and that can never be tested.
  - Gh0stRAT 3y ago
    If you practice bringing a new datacenter online without any connectivity on the existing deployment, and you practice then joining two disjoint "clouds", then you've pretty much covered your bases. Are you making a rate limiting/ddos argument about "turning them all on at the same time"?
throwawaaarrgh 3y ago
The solution to this has to be done earlier, but it's simple: start a habit of destroying and recreating everything. If you wait to start doing this, it's very painful. If you start doing it at the very beginning, you quickly get used to it, and breaking changes and weird dependencies are caught early. You can even do this with hardware. It changes how you architect things, to deal with shit getting unplugged or reset. You end up requiring more automation, version control and change management, which speeds up and simplifies overall work, in addition to preventing and quickly fixing outages. It's a big culture shift.
- ahmedfromtunis 3y ago
  My SaaS app is small even compared to some desktop apps. But at least once a year, I try to reboot it from scratch. Of course this is way easier for a nano-scale app, but I love the feeling of knowing that it can be started from any server in the world in less than 10 minutes (including copying the data). I also make sure there are 0 errors even with a clean slate database. For some reason I can't understand, this gives me some joy.
- jeremyjh 3y ago
  That really doesn't address GP's concerns at all, which are only concerns for the hyperscale cloud providers. Because they use their global infrastructure to turn on a data center that has been down, we don't know if they have a way to restart the entire planet. It is impossible to test this, because that would cause a global outage that could possibly be permanent.
- kevincox 3y ago
  The problem is that you can never really destroy and recreate everything. At the very least you want to preserve your user data but you probably also don't want to regularly go offline. IIRC Google used to have a dependency on starting up a new datacenter where it needed a DNS request to have some core services find each other and start up. This could be served by any other Google datacenter. But if all datacenters went offline at the same time it wouldn't be able to start. So just destroying and recreating full datacenters won't find this. You literally would have to take all of Google offline at the same time to find this by testing.
jeffbee 3y ago
When I was in Google SRE we had monitoring and enforcement of permitted and forbidden RPC peers, such that a system that attempted to use another system would fail or send alerts. This was useful at the top of the stack to keep track of dependencies silently added by library authors, and at the low levels to ensure the things at the bottom of the stack were really at the bottom. We also did virtual automated cluster turn-up and turn-down, to make sure our documented procedures did not get out of date, and in my 6 years in SRE I saw that procedure fall from 90 days to under an hour. We also regularly exercised the scratch restarts of things like global encryption key management, which involves a physical object. The annual DiRT exercise also tried to make sure that no person, team, or office was necessary to the continuing function of systems.
jeffrallen 3y ago
The power grid guys claim to have cold start plans locked and loaded, but I'm not convinced they would work. Anyone seen an after-action report saying how well a real grid cold start went? It would also be interesting to know which grid has had the most cold starts: in a perfect world, they'd be good at it by now. Bet it's in the Caribbean or Africa. But it's funny: small grid cold starts (i.e. an isolated island with one diesel generator and some solar) are so easy they probably wouldn't make good case studies. It's clear that the Internet itself could not be cold started like that AC grids, there's simply too many AS's. (Think about what AS means for a second to understand why a coordinated, rehearsed cold start is not possible.)
plygrnd 3y ago
AWS learned this lesson in 2017 when S3 fell over. Lots of internal changes after that.
Too 3y ago
Not just cold starts. Similar struggles if your Infra-as-code deployments depend on your CI/CD pipelines, conveniently running in the same environment as the one having an outage that you need to push new configs to.
az226 3y ago
Hyperscalers have several days’ worth of diesel that power generators after the batteries are used up. I’m pretty sure fueling trucks would be routed there should there be a longer power outage than 1-2 days.

Gh0stRAT 3y ago

Azure has procedures in place to prevent circular dependencies, and regularly exercises them when bringing new regions online. IIRC some of the information about their approach is considered sensitive so I won't elaborate further.

jeremyjh 3y ago
Are you saying they can bring a new data center online without any connectivity to the rest of their infrastructure? GP isn't concerned about turning on one data center, they are concerned about turning them all on at the same time, and that can never be tested.
- Gh0stRAT 3y ago
  If you practice bringing a new datacenter online without any connectivity on the existing deployment, and you practice then joining two disjoint "clouds", then you've pretty much covered your bases. Are you making a rate limiting/ddos argument about "turning them all on at the same time"?

throwawaaarrgh 3y ago

The solution to this has to be done earlier, but it's simple: start a habit of destroying and recreating everything. If you wait to start doing this, it's very painful. If you start doing it at the very beginning, you quickly get used to it, and breaking changes and weird dependencies are caught early. You can even do this with hardware. It changes how you architect things, to deal with shit getting unplugged or reset. You end up requiring more automation, version control and change management, which speeds up and simplifies overall work, in addition to preventing and quickly fixing outages. It's a big culture shift.

ahmedfromtunis 3y ago
My SaaS app is small even compared to some desktop apps. But at least once a year, I try to reboot it from scratch. Of course this is way easier for a nano-scale app, but I love the feeling of knowing that it can be started from any server in the world in less than 10 minutes (including copying the data). I also make sure there are 0 errors even with a clean slate database. For some reason I can't understand, this gives me some joy.
jeremyjh 3y ago
That really doesn't address GP's concerns at all, which are only concerns for the hyperscale cloud providers. Because they use their global infrastructure to turn on a data center that has been down, we don't know if they have a way to restart the entire planet. It is impossible to test this, because that would cause a global outage that could possibly be permanent.
kevincox 3y ago
The problem is that you can never really destroy and recreate everything. At the very least you want to preserve your user data but you probably also don't want to regularly go offline. IIRC Google used to have a dependency on starting up a new datacenter where it needed a DNS request to have some core services find each other and start up. This could be served by any other Google datacenter. But if all datacenters went offline at the same time it wouldn't be able to start. So just destroying and recreating full datacenters won't find this. You literally would have to take all of Google offline at the same time to find this by testing.

jeffbee 3y ago

When I was in Google SRE we had monitoring and enforcement of permitted and forbidden RPC peers, such that a system that attempted to use another system would fail or send alerts. This was useful at the top of the stack to keep track of dependencies silently added by library authors, and at the low levels to ensure the things at the bottom of the stack were really at the bottom. We also did virtual automated cluster turn-up and turn-down, to make sure our documented procedures did not get out of date, and in my 6 years in SRE I saw that procedure fall from 90 days to under an hour. We also regularly exercised the scratch restarts of things like global encryption key management, which involves a physical object. The annual DiRT exercise also tried to make sure that no person, team, or office was necessary to the continuing function of systems.

jeffrallen 3y ago

The power grid guys claim to have cold start plans locked and loaded, but I'm not convinced they would work. Anyone seen an after-action report saying how well a real grid cold start went? It would also be interesting to know which grid has had the most cold starts: in a perfect world, they'd be good at it by now. Bet it's in the Caribbean or Africa. But it's funny: small grid cold starts (i.e. an isolated island with one diesel generator and some solar) are so easy they probably wouldn't make good case studies. It's clear that the Internet itself could not be cold started like that AC grids, there's simply too many AS's. (Think about what AS means for a second to understand why a coordinated, rehearsed cold start is not possible.)

plygrnd 3y ago

AWS learned this lesson in 2017 when S3 fell over. Lots of internal changes after that.

Too 3y ago

Not just cold starts. Similar struggles if your Infra-as-code deployments depend on your CI/CD pipelines, conveniently running in the same environment as the one having an outage that you need to push new configs to.

az226 3y ago

Hyperscalers have several days’ worth of diesel that power generators after the batteries are used up. I’m pretty sure fueling trucks would be routed there should there be a longer power outage than 1-2 days.