3 ms·

This is wildy uncommon. Borderline unheard of. In every job I've had, internally managed services failed with unscheduled outages at least monthly, sometimes fo

by p4wnc6 12y ago

This is wildy uncommon. Borderline unheard of. In every job I've had, internally managed services failed with unscheduled outages at least monthly, sometimes for multiple days at a time. 3 years of up time? If I was hiring for system administrators and a candidate told me they had achieved that, I would just think they were lying.

batou 12y ago
We're very good at what we do and plan carefully. No joke. That's it. 2-4s downtime for a 40Gb svn repo migration is what we aim for. To be fair we haven't patched that box's kernel since it was rebooted but we really don't have to bounce kit very often. Most updates are handled through service restarts. Only updates that are an attack vector internally are actually applied too. As for the usual problems of power, we have battery and generator backed support. Redundant NICs on two VLANs and redundant power supplies on two feeds and a RAID array obviously. We also have a large memcache deployment that hasn't been restarted in that timescale as well. Even the memcache processes are that old. Works wonderfully. Also CentOS is pretty damn stable. If this was public facing it would be a slightly different story as we'd be patching the kernel too.
stonogo 12y ago
Your systems administrators in every job you've had should be fired. There is nothing particularly astounding about three years of uptime from a correctly-administered internal system.
- emn13 12y ago
  Yeah - frankly, for most VCS's, it's pretty hard to achieve an uptime as poor as github locally(1). There's just not much that can go wrong; even doing virtually no maintenance you'd do better simply because github is hosted else and effected by all the vagaries of the internet. (1): by uptime I mean time up during office hours under the assumption that dev workstations can run. There's no point in measuring uptime during a power-outage...

by p4wnc6 12y ago

batou 12y ago

We're very good at what we do and plan carefully. No joke. That's it. 2-4s downtime for a 40Gb svn repo migration is what we aim for. To be fair we haven't patched that box's kernel since it was rebooted but we really don't have to bounce kit very often. Most updates are handled through service restarts. Only updates that are an attack vector internally are actually applied too. As for the usual problems of power, we have battery and generator backed support. Redundant NICs on two VLANs and redundant power supplies on two feeds and a RAID array obviously. We also have a large memcache deployment that hasn't been restarted in that timescale as well. Even the memcache processes are that old. Works wonderfully. Also CentOS is pretty damn stable. If this was public facing it would be a slightly different story as we'd be patching the kernel too.

stonogo 12y ago

Your systems administrators in every job you've had should be fired. There is nothing particularly astounding about three years of uptime from a correctly-administered internal system.

emn13 12y ago
Yeah - frankly, for most VCS's, it's pretty hard to achieve an uptime as poor as github locally(1). There's just not much that can go wrong; even doing virtually no maintenance you'd do better simply because github is hosted else and effected by all the vagaries of the internet. (1): by uptime I mean time up during office hours under the assumption that dev workstations can run. There's no point in measuring uptime during a power-outage...