4 ms·

Hmm. I just have a lot of trouble believing that you run lots and lots of machines and yet have never seen a kernel crash.

by thras 17y ago

jacquesm 17y ago
Well, maybe we're just lucky but I don't think that is it. We strip the kernels down to the minimum we need on machine installation and we always try to find out what caused a problem. On top of that a buddy of mine wrote a piece of software he calls 'flame', which is used to manage 300 machines. Sure, occasionally one of those will go down. But post-mortem analysis will invariably point to some piece of hardware that broke. Either a cpu, a memory chip or a drive hanging on by a hair. I think the trick to managing large numbers of servers and not being in reboot hell all the time is to find a recipe and to stick to it, and good quality monitoring tools to give you an early warning. And if you are running on cheap non-ecc motherboards for servers that really is asking for it. I like to sleep well :) The last crash that I remember was in February this year, a faulty memory chip that had been registering complaints for a while in the mcelog (but due to a human error we missed it). We had just upgraded a machine that was already in production and we did not burn in the machine out of service. Stupid mistake... We burn in our new machines religiously and we only use top quality hardware. Still, during the burn-in phase we have a good 10% hardware failures. But once the machines pass burn in - and we really give them hell - they perform pretty good. Since we're in anecdotal territory anyway, once during the burn in testing of an ethernet switch over the weekend we came back to the office on monday morning to find the test frozen. The switch was no longer passing data in or out. A reboot would not revive it. We opened it up to find a crack running straight down the middle of one of the chips... so much for that particular brand of router :)

Hmm. I just have a lot of trouble believing that you run lots and lots of machines and yet have never seen a kernel crash.

by thras 17y ago

Hmm. I just have a lot of trouble believing that you run lots and lots of machines and yet have never seen a kernel crash.

jacquesm 17y ago

Well, maybe we're just lucky but I don't think that is it. We strip the kernels down to the minimum we need on machine installation and we always try to find out what caused a problem. On top of that a buddy of mine wrote a piece of software he calls 'flame', which is used to manage 300 machines. Sure, occasionally one of those will go down. But post-mortem analysis will invariably point to some piece of hardware that broke. Either a cpu, a memory chip or a drive hanging on by a hair. I think the trick to managing large numbers of servers and not being in reboot hell all the time is to find a recipe and to stick to it, and good quality monitoring tools to give you an early warning. And if you are running on cheap non-ecc motherboards for servers that really is asking for it. I like to sleep well :) The last crash that I remember was in February this year, a faulty memory chip that had been registering complaints for a while in the mcelog (but due to a human error we missed it). We had just upgraded a machine that was already in production and we did not burn in the machine out of service. Stupid mistake... We burn in our new machines religiously and we only use top quality hardware. Still, during the burn-in phase we have a good 10% hardware failures. But once the machines pass burn in - and we really give them hell - they perform pretty good. Since we're in anecdotal territory anyway, once during the burn in testing of an ethernet switch over the weekend we came back to the office on monday morning to find the test frozen. The switch was no longer passing data in or out. A reboot would not revive it. We opened it up to find a crack running straight down the middle of one of the chips... so much for that particular brand of router :)