3 ms·

What if reproducibility is more important than uptime? How does memory fragmentation affect performance after more than 24 hours? What happens when your serve

by fleitz 12y ago

What if reproducibility is more important than uptime? How does memory fragmentation affect performance after more than 24 hours? What happens when your server has to restart from a cold cache scenario? What happens when your server is down? What about that setting that a sys admin manually applied to the server 6 months ago to fix some issue and isn't in saved in the server config? By forcing a condition people try to avoid you get good at dealing with those situations. How often do your database servers actually fail over? How confident are you in your code and systems to actually fail over properly? How do you prime the caches when your memcached servers reboot?

laumars 12y ago
> What if reproducibility is more important than uptime? Reboots aren't something you need to reproduce multiple times a day > How does memory fragmentation affect performance after more than 24 hours? If this is a massive issue then your server daemon is piss poor. However I think you're just clutching at straws here. > What happens when your server has to restart from a cold cache scenario? That doesn't mean you have to reboot your server several times a day > What happens when your server is down? It would only be down because you're rebooting the bloody thing :p > What about that setting that a sys admin manually applied to the server 6 months ago to fix some issue and isn't in saved in the server config? You don't need to reboot a live server several times a day to apply and test server config. > By forcing a condition people try to avoid you get good at dealing with those situations. Advocating repeatedly breaking live servers as practice to know what to do when they break accidentally is the dumbest thing I've read in a while. I don't need to repeatedly walk in front of a bus to learn not to walk in front of a bus. If you need to practice test scenarios then do so on a test system - that's what they're there for! > How often do your database servers actually fail over? Through my own negligence? They haven't yet. Through dodgy code rushed live by our developers; more often that I'd like to admit. > How confident are you in your code and systems to actually fail over properly? Code: not very. But I don't manage that. Systems: very confident. But I do sane load and disaster testing on test systems; and monitoring and logging on all systems to highlight potential issues before they completely snap. > How do you prime the caches when your memcached servers reboot? We're not stupid enough to reboot all our live infrastructure (and their redundancies) when they're in use. Let alone to do it multiple times a day. -------------------------------------------- You're not convincing me that you need to reboot your servers multiple times a day; if anything, you're just convincing me that you don't have a proper dev and test infrastructure in place. And that's far more dangerous than any of the other issues you or I've raised thus far.

by fleitz 12y ago

laumars 12y ago

> What if reproducibility is more important than uptime? Reboots aren't something you need to reproduce multiple times a day > How does memory fragmentation affect performance after more than 24 hours? If this is a massive issue then your server daemon is piss poor. However I think you're just clutching at straws here. > What happens when your server has to restart from a cold cache scenario? That doesn't mean you have to reboot your server several times a day > What happens when your server is down? It would only be down because you're rebooting the bloody thing :p > What about that setting that a sys admin manually applied to the server 6 months ago to fix some issue and isn't in saved in the server config? You don't need to reboot a live server several times a day to apply and test server config. > By forcing a condition people try to avoid you get good at dealing with those situations. Advocating repeatedly breaking live servers as practice to know what to do when they break accidentally is the dumbest thing I've read in a while. I don't need to repeatedly walk in front of a bus to learn not to walk in front of a bus. If you need to practice test scenarios then do so on a test system - that's what they're there for! > How often do your database servers actually fail over? Through my own negligence? They haven't yet. Through dodgy code rushed live by our developers; more often that I'd like to admit. > How confident are you in your code and systems to actually fail over properly? Code: not very. But I don't manage that. Systems: very confident. But I do sane load and disaster testing on test systems; and monitoring and logging on all systems to highlight potential issues before they completely snap. > How do you prime the caches when your memcached servers reboot? We're not stupid enough to reboot all our live infrastructure (and their redundancies) when they're in use. Let alone to do it multiple times a day. -------------------------------------------- You're not convincing me that you need to reboot your servers multiple times a day; if anything, you're just convincing me that you don't have a proper dev and test infrastructure in place. And that's far more dangerous than any of the other issues you or I've raised thus far.