4 ms·

> I mean no. If there is an error you should figure out what the problem is first. Figuring out what the problem is may take time. If a problem is impacting a

> I mean no. If there is an error you should figure out what the problem is first. Figuring out what the problem is may take time. If a problem is impacting a customer’s experience, I’d rather recover faster and then pull metadata later to try and understand the issue.

packetlost 5y ago
I think it ultimately comes down to the situation and how well you know your application. Sometimes restarting it will fuck up the data even more. It almost certainly indicates an engineering flaw, but that doesn't mean it won't happen.
papito 5y ago
Sure, but it's easy to fall into the trap of "the fix is to just restart it". That's dereliction of duty.
KaiserPro 5y ago
> Figuring out what the problem is may take time. If a problem is impacting a customer’s experience, I’d rather recover faster and then pull metadata later to try and understand the issue. How do you know it'll fix the problem, if you don't know the cause? In my experience, about 50% of the time to fix an incident is cleaning up the mess you made performing a fix based on an hypothesis that turns out to be wrong. That means longer downtime. A classic example of this is when a 60 disk raid unit crashed when performing a rebuild under load. This happened to my coworker when they were doing the daily replacing a disk task. They put in the disks, shut the draw, pressed the button that changes the activity lights to see that everything was working, and the thing froze. They rightly shat their pants. They panicked, rebooted the array and attached fileserver. A load of data was lost (the file servers had at the time the maximum amount of ram you could shove in a standard 2u two proc intel server). When it happened again, another coworker waited for the raid enclosure to reboot. Low and behold the SAS links came back and we were able to do an online fsck inside 3 minutes. no data lost. If your facing an outage, breathing space to gather thoughts is time well invested. Blind hacking, or indeed decision paralysis is damaging

packetlost 5y ago

I think it ultimately comes down to the situation and how well you know your application. Sometimes restarting it will fuck up the data even more. It almost certainly indicates an engineering flaw, but that doesn't mean it won't happen.

papito 5y ago

Sure, but it's easy to fall into the trap of "the fix is to just restart it". That's dereliction of duty.

KaiserPro 5y ago

> Figuring out what the problem is may take time. If a problem is impacting a customer’s experience, I’d rather recover faster and then pull metadata later to try and understand the issue. How do you know it'll fix the problem, if you don't know the cause? In my experience, about 50% of the time to fix an incident is cleaning up the mess you made performing a fix based on an hypothesis that turns out to be wrong. That means longer downtime. A classic example of this is when a 60 disk raid unit crashed when performing a rebuild under load. This happened to my coworker when they were doing the daily replacing a disk task. They put in the disks, shut the draw, pressed the button that changes the activity lights to see that everything was working, and the thing froze. They rightly shat their pants. They panicked, rebooted the array and attached fileserver. A load of data was lost (the file servers had at the time the maximum amount of ram you could shove in a standard 2u two proc intel server). When it happened again, another coworker waited for the raid enclosure to reboot. Low and behold the SAS links came back and we were able to do an online fsck inside 3 minutes. no data lost. If your facing an outage, breathing space to gather thoughts is time well invested. Blind hacking, or indeed decision paralysis is damaging