4 ms·
The throttle that we're referring to there is a CPU throttle. When we provision an OS-virtualized instance, there is a default throttle to prevent it from consu
by bcantrill 12y ago
The throttle that we're referring to there is a CPU throttle. When we provision an OS-virtualized instance, there is a default throttle to prevent it from consuming a disproportionate amount of CPU on the box. The instance that runs TFTP on the headnode was provisioned as a relatively small instance (it needs very little DRAM), which also gave it (by default) a CPU throttle that restricted its CPU utilization. Normally, of course, this isn't an issue -- but normally we don't try to TFTP boot the entire datacenter at once. This issue was obvious immediately (thanks, DTrace!), and we resolved it on the headnode by manually raising the throttle, and will be making the fix in SmartDataCenter itself as well. Does that answer your question?
- adamfeldman 12y agoAh got it! The control plane has instance restrictions similar to customer nodes, since it uses some of the same infrastructure.
- bcantrill 12y agoExactly; it uses all of the same infrastructure, actually -- it's just provisioned on a different network (namely, the admin network). It's also worth noting that OS-based virtualization helped us here not only because of the global visibility we get with DTrace (which immediately indicated that TFTP was waiting for CPU), but also because we could dynamically adjust the throttle and simply give it more CPU without having to bounce the box and interrupt all of the TFTP booting in progress. It was a small amount of solace on what was easily the worst day we've had in a while, if not ever...
- sokoloff 12y agodtrace reminds me a lot of awk...
- bcantrill 12y agoNot an accident -- I'm a well-known awk lover, and it was very much an inspiration for DTrace. (Even down to the documentation: if you haven't read it recently, re-read "The AWK Programming Language"[1] -- it's beautifully concise.) [1] http://www.amazon.com/The-AWK-Programming-Language-Alfred/dp/020107981X http://www.amazon.com/The-AWK-Programming-Language-Alfred/dp...
- mzs 12y agoOh man Bryan I am so wary of sharing this with you on account of it being so wrong, gross, and buggy... But WTH: https://github.com/msliczniak/onetenth https://github.com/msliczniak/onetenth Sorry about the rough day you all had - cheers Mike