5 ms·
Yes, and this is why a company really needs two technical teams (at a minimum): developers and sysadmins. My goal is stability. The dev group's goal is growth.
by dsr_ 14y ago
Yes, and this is why a company really needs two technical teams (at a minimum): developers and sysadmins.
My goal is stability. The dev group's goal is growth. In a very small startup, it's hard to hold both these ideas in your head at the same time. If you don't grow enough, you will never need stability. If you grow enough, you will eventually need stability so that your customers do not desert you. Thesis, antithesis, synthesis: supportable growth.
My ops group meets with devs for discussions of supportability and performance well before the actual handoff. We make sure that monitoring is allowed for from the beginning, that there are processes and documentation that let us fix problems at 3AM or diagnose it well enough to call the right person to fix it.
Performance is checked regularly. Is it time for new hardware? Is it time for a rethink of the architecture to support faster performance or higher scale?
The two forces will be vector-summed. It's important to get it all pointed in a direction and magnitude that leads to success.
- BryantD 14y agoAs a sysadmin, I used to think in the two goal paradigm. I actually don't any more; if your company has two groups with different goals, particularly in the early days, you'll waste time with friction. My goal as a sysadmin is to help the company succeed. I try (and sometimes fail) to assume the same positive intent of my peers in engineering. If we seem to be working in different directions, it's often because there's something one or the other of us doesn't understand. Concrete example: I have often found that my tolerance for midnight crashes is lower than that of the engineer who wrote the code. There appears to be a misalignment. But on further examination, the problem is that the engineer doesn't have the same information I do: she doesn't realize on a gut level the cost of being woken up at 2 AM. Solution: add engineers to the first level on-call rotation. Information is added; goals become more in alignment.
- mercurial 14y agoAs an engineer, this made me chuckle.
- BryantD 14y agoMission accomplished. It's kind of a trick -- I am way in favor of letting developers look at running production servers, because how better to understand a production environment? But a lot of sysadmins are wary of that, so you have to frame it in a way that benefits tech ops as well.
- donw 14y agoI don't think that these two goals are at-odds. In many companies, the problem is just that neither group, developers or sysadmins, really seems to understand the other. But both want the same thing -- to ship a working product. Developers really don't want growth at the expense of software that doesn't work. But many developers come from a computer science background, and I've never seen a computer science curriculum teach students about logging, testing, runtime instrumentation, monitoring theory, security, or deployment. In university, you write code in a very unrealistic fashion. Lots of individual coding, some teamwork, but your assignments nominally have well-defined inputs and outputs, and students effectively never have to carry projects across classes and semesters, the way that you would with a production application. Sysadmins, having been exposed to this, assume that developers are out to kill everything with a BIOS, and as a result tend to lock systems down, or implement weird hacks to get around the lack of logging or instrumentation. In companies I've consulted with, a bit of education goes a long way towards fixing this. I usually rotate a dev at a time through the sysadmin team, where they can both help in debugging problems, as well as tackling all the interrupts that would otherwise be divided evenly among all the engineers. The developers learn about why logging is important, gain a better understanding of the deployment process and environment, and the sysadmins gain a greater understanding of what the devs are trying to accomplish. It's amazingly effective in fixing the false dev/ops split, and in getting the two teams working together effectively.