4 ms·

I have a hard time understanding why they needed humans babysitting the restart of servers and services. Servers and services are supposed to automatically rest

I have a hard time understanding why they needed humans babysitting the restart of servers and services. Servers and services are supposed to automatically restart after the power came back. This implies OVH doesn't regularly test server restarts with something like Netflix Chaos Monkey. Moreover, the power lines were not really redundant in SBG, and regarding the network downtime in RBX, it looks like the network core is not really redundant either. I'd be curious to know how competitors like AWS and Google Cloud mitigate this kind of problems.

icebraining 9y ago
OVH runs servers for their clients, do you really expect them to randomly restart their clients' servers? Netflix is not a hosting provider, the situations are not comparable.
- ngrilly 9y ago
  The comment in the OVH post-mortem suggests it is about internal servers and services. You're right about customer bare-metal servers. They can't restart them randomly.
- wereHamster 9y ago
  I would be pissed if my hosting provider randomly restarted my servers, taking them offline for minutes each time.
  - drdebug 9y ago
    So would I, but that actually makes me realize that I should be doing more of what netflix is doing too and ensure I can randomly restart servers without affecting service.
    - tete 9y ago
      Isn't choasmonkey just randomly restarting virtual servers on Amazon? Kudos to Netflix, but restarting a virtual server vs a physical or a whole data center are different things. I think every company that cares a bit about high availability knocks out stuff randomly or at least in different ways, does stuff like introducing packet loss, etc. It's another layer and another thing to test that on service/virtual server layer than on close to physical layers. Of course, one should test that too and it's nowhere near impossible, but Chaosmonkey is for a somewhat different use case. Also the "article" mentions that tests are done.
pmontra 9y ago
I have some VPSes there. All of them run independent services, no redundancy. They can restart all the servers they want if they manage not to kill any of my processes, break network connections, change IP addresses. If they can't (it's probably hard) it's up to me to implement a reduntant architecture. Then maybe I'll also implement some random kill switch to proof that the system is resilient to failures.
vbernat 9y ago
Restarting a whole datacenter from scratch is different from restarting individual servers. If a service needs another service running on another server, it may fail to start. For example, many servers may need the DHCP server. Some servers may need the DNS. Some others may need Kafka to be up and running, etc. You may have some documentation on which servers to start first, but things are moving so fast that it's not unlikely you'll need to rely on supervision to know which service was able to start correctly.
- ngrilly 9y ago
  I don't buy this argument. If a service needs another service running on another server which is not already available, it should retry with an exponential backoff. Doing anything else is a bug.
  - snom380 9y ago
    I agree it's a bug, but this is far, far more difficult to test against with a chaos monkey than one might think. With a whole datacenter down, not to speak of multiple availability zones, problems and dependencies will crop up that are extremely difficult to anticipate. Maybe you expected one other AZ to be up so you could provision servers from there, but now you have to somehow get one of them bootstrapped. Or maybe during the outage power was cycled on some equipment one too many times, causing some failsafe to trip, or maybe some of the core infrastructure like your switches and routers expect some core service to be up, but that core service can't get online without working network.
    - ngrilly 9y ago
      Thanks for the great explanation. I agree that a Chaos Monkey test something a lot more easier to recover from than a whole datacenter getting down. I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure, which made believe it is possible to automatically recover from this. Are you saying that even the companies I mentioned have the same issues as OVH when they recover from a complete power failure?
ngrilly 9y ago
Some of you are commenting that asking OVH to randomly restart customer servers would be silly. I agree. My comment was about servers managed by OVH. The post-mortem specifically says: "Since then, we have been working on restarting services. Powering the site with energy allows the servers to be restarted, but the services running on the servers still need to be restarted. That's why each service has been coming back gradually since 10:30 am. Our monitoring system allows us to know the list of servers that have successfully started up and those that still have a problem. We intervene on each of these servers to identify and solve the problem that prevents it from restarting." The language used suggests that these are servers monitored and operated by OVH, not by random customers.
- Feld0 9y ago
  OVH monitors customers' servers and investigates if they stop responding to pings, unless you opt out of this. I'm guessing they were looking at this system to make sure all their customers' machines came online again after the incident. Source: am an OVH customer
  - ngrilly 9y ago
    Yes, I'm aware of this monitoring service. Is OVH able to login in your server to investigate what is the issue?
dboreham 9y ago
Meanwhile in the real world...

icebraining 9y ago

OVH runs servers for their clients, do you really expect them to randomly restart their clients' servers? Netflix is not a hosting provider, the situations are not comparable.

ngrilly 9y ago
The comment in the OVH post-mortem suggests it is about internal servers and services. You're right about customer bare-metal servers. They can't restart them randomly.
wereHamster 9y ago
I would be pissed if my hosting provider randomly restarted my servers, taking them offline for minutes each time.
- drdebug 9y ago
  So would I, but that actually makes me realize that I should be doing more of what netflix is doing too and ensure I can randomly restart servers without affecting service.
  - tete 9y ago
    Isn't choasmonkey just randomly restarting virtual servers on Amazon? Kudos to Netflix, but restarting a virtual server vs a physical or a whole data center are different things. I think every company that cares a bit about high availability knocks out stuff randomly or at least in different ways, does stuff like introducing packet loss, etc. It's another layer and another thing to test that on service/virtual server layer than on close to physical layers. Of course, one should test that too and it's nowhere near impossible, but Chaosmonkey is for a somewhat different use case. Also the "article" mentions that tests are done.

pmontra 9y ago

I have some VPSes there. All of them run independent services, no redundancy. They can restart all the servers they want if they manage not to kill any of my processes, break network connections, change IP addresses. If they can't (it's probably hard) it's up to me to implement a reduntant architecture. Then maybe I'll also implement some random kill switch to proof that the system is resilient to failures.

vbernat 9y ago

Restarting a whole datacenter from scratch is different from restarting individual servers. If a service needs another service running on another server, it may fail to start. For example, many servers may need the DHCP server. Some servers may need the DNS. Some others may need Kafka to be up and running, etc. You may have some documentation on which servers to start first, but things are moving so fast that it's not unlikely you'll need to rely on supervision to know which service was able to start correctly.

ngrilly 9y ago
I don't buy this argument. If a service needs another service running on another server which is not already available, it should retry with an exponential backoff. Doing anything else is a bug.
- snom380 9y ago
  I agree it's a bug, but this is far, far more difficult to test against with a chaos monkey than one might think. With a whole datacenter down, not to speak of multiple availability zones, problems and dependencies will crop up that are extremely difficult to anticipate. Maybe you expected one other AZ to be up so you could provision servers from there, but now you have to somehow get one of them bootstrapped. Or maybe during the outage power was cycled on some equipment one too many times, causing some failsafe to trip, or maybe some of the core infrastructure like your switches and routers expect some core service to be up, but that core service can't get online without working network.
  - ngrilly 9y ago
    Thanks for the great explanation. I agree that a Chaos Monkey test something a lot more easier to recover from than a whole datacenter getting down. I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure, which made believe it is possible to automatically recover from this. Are you saying that even the companies I mentioned have the same issues as OVH when they recover from a complete power failure?

ngrilly 9y ago

Some of you are commenting that asking OVH to randomly restart customer servers would be silly. I agree. My comment was about servers managed by OVH. The post-mortem specifically says: "Since then, we have been working on restarting services. Powering the site with energy allows the servers to be restarted, but the services running on the servers still need to be restarted. That's why each service has been coming back gradually since 10:30 am. Our monitoring system allows us to know the list of servers that have successfully started up and those that still have a problem. We intervene on each of these servers to identify and solve the problem that prevents it from restarting." The language used suggests that these are servers monitored and operated by OVH, not by random customers.

Feld0 9y ago
OVH monitors customers' servers and investigates if they stop responding to pings, unless you opt out of this. I'm guessing they were looking at this system to make sure all their customers' machines came online again after the incident. Source: am an OVH customer
- ngrilly 9y ago
  Yes, I'm aware of this monitoring service. Is OVH able to login in your server to investigate what is the issue?

dboreham 9y ago

Meanwhile in the real world...