3 ms·

> what else was a more important use of your time Keeping shit running now is always more important than keeping shit running in case of catastrophic failure.

> what else was a more important use of your time Keeping shit running now is always more important than keeping shit running in case of catastrophic failure. Because a catastrophic failure is only catastrophic if you actually have something of value in the first place. In the real world it's a balancing act in which there is no room for absolutism. Choosing which compromises to make is the hardest part of any job that comes with a level of responsibility.

mattmanser 13y ago
Indeed. Last place I worked with sysadmins, they were fire fighting like crazy, not enough of them, bad choice of suppliers, poor code bases meant things fell over and took far to long to run and a dev team begging to get off the internal network on to their own because of all the restrictions. I needed a backup of the DB of one of the three systems to test a major setting change that no-one was 100% sure would work because no-one understood the bloody awful contractor code with the even more awfully designed DB. Took them 2 weeks to get it to me. Crazy place, quit after 2 months. Really thought I'd asked the right questions too.
reitzensteinm 13y ago
I'm talking specifically about dedicated sys admins, since that's what the blog post is about; I'm no stranger to cowboy coding on a project that may or may not ever be worth anything. I've lost data once before to corrupted backups and I don't regret not investing more engineering effort. It was an MVP, and making better products gives me more leverage. But by the time you're hiring full time for the position, outside of a few edge cases where maybe you're SnapChat and you are growing 20% a week, it's probably time to settle down a bit and be sensible. At which point, if testing your backups to completion to avoid catastrophic data loss isn't #1 on the todo list, it's #2.
- vidarh 13y ago
  It may be #2 perpetually in many cases because the boss will not listen when you insist it's important. For many it then starts slipping down the list, as what is best for the company is often not best for the employee: For many it becomes a reasonable (for them personally) risk to take to bet that they'll do better from keeping the boss satisfied now rather than spend time on backups to avoid a major disaster after they've left. And yes, that means gambling that the major disaster won't hit while you're still there. I'm not saying this is how it should be, but it is how it often becomes if the sysadmin or whomever taking on those responsibilities don't report to someone who also see the data integrity as priority #1 for the sysadmin. I've worked in places where the CEO's e-mail client configuration is the #1 priority for the guy that should have been focusing on server backups, for example, and where prioritising the backups would be a bad career move for the person in question. (Yes, that is a huge warning sign that it's best to find a different job)

mattmanser 13y ago

Indeed. Last place I worked with sysadmins, they were fire fighting like crazy, not enough of them, bad choice of suppliers, poor code bases meant things fell over and took far to long to run and a dev team begging to get off the internal network on to their own because of all the restrictions. I needed a backup of the DB of one of the three systems to test a major setting change that no-one was 100% sure would work because no-one understood the bloody awful contractor code with the even more awfully designed DB. Took them 2 weeks to get it to me. Crazy place, quit after 2 months. Really thought I'd asked the right questions too.

reitzensteinm 13y ago

I'm talking specifically about dedicated sys admins, since that's what the blog post is about; I'm no stranger to cowboy coding on a project that may or may not ever be worth anything. I've lost data once before to corrupted backups and I don't regret not investing more engineering effort. It was an MVP, and making better products gives me more leverage. But by the time you're hiring full time for the position, outside of a few edge cases where maybe you're SnapChat and you are growing 20% a week, it's probably time to settle down a bit and be sensible. At which point, if testing your backups to completion to avoid catastrophic data loss isn't #1 on the todo list, it's #2.

vidarh 13y ago
It may be #2 perpetually in many cases because the boss will not listen when you insist it's important. For many it then starts slipping down the list, as what is best for the company is often not best for the employee: For many it becomes a reasonable (for them personally) risk to take to bet that they'll do better from keeping the boss satisfied now rather than spend time on backups to avoid a major disaster after they've left. And yes, that means gambling that the major disaster won't hit while you're still there. I'm not saying this is how it should be, but it is how it often becomes if the sysadmin or whomever taking on those responsibilities don't report to someone who also see the data integrity as priority #1 for the sysadmin. I've worked in places where the CEO's e-mail client configuration is the #1 priority for the guy that should have been focusing on server backups, for example, and where prioritising the backups would be a bad career move for the person in question. (Yes, that is a huge warning sign that it's best to find a different job)