5 ms·

I don't have a lot of experience with datacenters, and am trying to understand this sentence: "Because there was a simultaneous reboot of every system in the d

I don't have a lot of experience with datacenters, and am trying to understand this sentence: "Because there was a simultaneous reboot of every system in the datacenter, there was extremely high contention on the TFTP boot infrastructure, which like all of our infrastructure, normally has throttles in place to ensure that it cannot run away with a machine." What does "cannot run away with a machine" mean? Why would you want to by default restrict the speed at which that system runs?

jacquesm 12y ago
I think they're trying to say that that machine has other duties except for TFTP and that the process is running at a priority low enough that it can be pre-empted by other processes. So if there is a lot of contention on the TFTP boot infrastructure the throttles limit the amount of work a single machine can do, which in turn can cause starvation. The reason why is probably related to what it does besides TFTP, which under normal circumstances is probably more important than new nodes joining the network. During a mass reboot that will bite you because then the chances of starvation will go up quite a bit as the throttles cause one machine after another to time-out and retry their boot sequence.
ChargingWookie 12y ago
They wanted to prevent all of the servers trying to contact the PXE server at the same time to avoid overloading it. Its basically to prevent an accidently self DDOS.
CHY872 12y ago
I also have no experience, but my guess is that when you are running normally, you want to share the bandwidth of a server fairly between all clients. When you've just accidentally rebooted all servers, you want to do it completely unfairly - so that instead of 10 computers taking an hour to download something, the first takes 6 minutes, the second takes 12 minutes etc. This allows at least some of the servers to get started and your infrastructure to start getting back online. I.e. you want to change from round robin scheduling to shortest job first scheduling (which gives the minimum average waiting time).
deleted 12y ago
[deleted]
bcantrill 12y ago
The throttle that we're referring to there is a CPU throttle. When we provision an OS-virtualized instance, there is a default throttle to prevent it from consuming a disproportionate amount of CPU on the box. The instance that runs TFTP on the headnode was provisioned as a relatively small instance (it needs very little DRAM), which also gave it (by default) a CPU throttle that restricted its CPU utilization. Normally, of course, this isn't an issue -- but normally we don't try to TFTP boot the entire datacenter at once. This issue was obvious immediately (thanks, DTrace!), and we resolved it on the headnode by manually raising the throttle, and will be making the fix in SmartDataCenter itself as well. Does that answer your question?
- adamfeldman 12y ago
  Ah got it! The control plane has instance restrictions similar to customer nodes, since it uses some of the same infrastructure.
  - bcantrill 12y ago
    Exactly; it uses all of the same infrastructure, actually -- it's just provisioned on a different network (namely, the admin network). It's also worth noting that OS-based virtualization helped us here not only because of the global visibility we get with DTrace (which immediately indicated that TFTP was waiting for CPU), but also because we could dynamically adjust the throttle and simply give it more CPU without having to bounce the box and interrupt all of the TFTP booting in progress. It was a small amount of solace on what was easily the worst day we've had in a while, if not ever...
- sokoloff 12y ago
  dtrace reminds me a lot of awk...
  - bcantrill 12y ago
    Not an accident -- I'm a well-known awk lover, and it was very much an inspiration for DTrace. (Even down to the documentation: if you haven't read it recently, re-read "The AWK Programming Language"[1] -- it's beautifully concise.) [1] http://www.amazon.com/The-AWK-Programming-Language-Alfred/dp/020107981X http://www.amazon.com/The-AWK-Programming-Language-Alfred/dp...
deleted 12y ago
[deleted]

jacquesm 12y ago

I think they're trying to say that that machine has other duties except for TFTP and that the process is running at a priority low enough that it can be pre-empted by other processes. So if there is a lot of contention on the TFTP boot infrastructure the throttles limit the amount of work a single machine can do, which in turn can cause starvation. The reason why is probably related to what it does besides TFTP, which under normal circumstances is probably more important than new nodes joining the network. During a mass reboot that will bite you because then the chances of starvation will go up quite a bit as the throttles cause one machine after another to time-out and retry their boot sequence.

ChargingWookie 12y ago

They wanted to prevent all of the servers trying to contact the PXE server at the same time to avoid overloading it. Its basically to prevent an accidently self DDOS.

CHY872 12y ago

I also have no experience, but my guess is that when you are running normally, you want to share the bandwidth of a server fairly between all clients. When you've just accidentally rebooted all servers, you want to do it completely unfairly - so that instead of 10 computers taking an hour to download something, the first takes 6 minutes, the second takes 12 minutes etc. This allows at least some of the servers to get started and your infrastructure to start getting back online. I.e. you want to change from round robin scheduling to shortest job first scheduling (which gives the minimum average waiting time).

deleted 12y ago

[deleted]

bcantrill 12y ago

The throttle that we're referring to there is a CPU throttle. When we provision an OS-virtualized instance, there is a default throttle to prevent it from consuming a disproportionate amount of CPU on the box. The instance that runs TFTP on the headnode was provisioned as a relatively small instance (it needs very little DRAM), which also gave it (by default) a CPU throttle that restricted its CPU utilization. Normally, of course, this isn't an issue -- but normally we don't try to TFTP boot the entire datacenter at once. This issue was obvious immediately (thanks, DTrace!), and we resolved it on the headnode by manually raising the throttle, and will be making the fix in SmartDataCenter itself as well. Does that answer your question?

adamfeldman 12y ago
Ah got it! The control plane has instance restrictions similar to customer nodes, since it uses some of the same infrastructure.
- bcantrill 12y ago
  Exactly; it uses all of the same infrastructure, actually -- it's just provisioned on a different network (namely, the admin network). It's also worth noting that OS-based virtualization helped us here not only because of the global visibility we get with DTrace (which immediately indicated that TFTP was waiting for CPU), but also because we could dynamically adjust the throttle and simply give it more CPU without having to bounce the box and interrupt all of the TFTP booting in progress. It was a small amount of solace on what was easily the worst day we've had in a while, if not ever...
sokoloff 12y ago
dtrace reminds me a lot of awk...
- bcantrill 12y ago
  Not an accident -- I'm a well-known awk lover, and it was very much an inspiration for DTrace. (Even down to the documentation: if you haven't read it recently, re-read "The AWK Programming Language"[1] -- it's beautifully concise.) [1] http://www.amazon.com/The-AWK-Programming-Language-Alfred/dp/020107981X http://www.amazon.com/The-AWK-Programming-Language-Alfred/dp...

deleted 12y ago

[deleted]