3 ms·
> So, many times a week we would wake up, see an instance dead (all critical processes locked in D-state), call Softlayer, and spend hours (literally) to bring
by Joakal 16y ago
> So, many times a week we would wake up, see an instance dead (all critical processes locked in D-state), call Softlayer, and spend hours (literally) to bring it back up.
Wouldn't keepalived or heartbeat help with failover? If I see an instance died, I would delete it and start a new instance with a latest backed up image. I assume it would take ~15 minutes compared to relying on the support.
- kovyrin 16y agoThe thing is, all our cloud instances were redundant so SL downtimes almost never caused real service disruptions on the site. But when one instance dies, there is a huge chance that other one (two, five, ten) will die soon as well. As for the automated kill/deploy scripts - really often even their own cloud-related portal features don't work (like when you try to restart/reset an instance and their APIs and portal just time out, etc).