3 ms·
There's two glaring problems for me: generalists and ops is a route to pain, you just may not know it yet; and every generalist solves a problem differently, an
by oogali 13y ago
There's two glaring problems for me: generalists and ops is a route to pain, you just may not know it yet; and every generalist solves a problem differently, and may not understand the short-term vs. long-term tradeoffs that they're making.
From a sysadmin troubleshooting angle...
Say, you discover a memory leak in your application which sits on top of Apache httpd and mod_php/mod_wsgi.
Do you:
a) set up a cron to restart the web server every X minutes?
b) change httpd from the default of MaxRequestsPerChild 0 to some non-zero number?
c) ask Slicehost/Rackspace/etc. to increase the amount of memory for your VM (or in EC2, migrate to a larger instance)?
d) increase the number of application instances, and add them to the load balancer?
e) write a script that frequently polls the application server, and when a failure is detected, bounce the server (or remove it from the load balancing pool, and re-add it later?)
f) do some combination of all of the above, and add this question to your list of interview questions, while posting an opening on StackOverflow Careers for a sysadmin/techops/devops candidate?
g) sit down with a staging instance, valgrind/gdb/memprof/rbtrace/etc., a method of generating traffic, and the top 20 URLs from affected production instances?
If you picked G, how many people on your team are actually proficient with those tools? How many other team members would have picked the same option as you?
While still on this option, how many people actually start working on the problem? One person, or divvied up amongst multiple people? What do you do when that person hits a wall? What happens to the rest of your deadlines?
After all of that, are you still a generalist?
- peterwwillis 13y agoFirst off, i'm the G guy. But just because i'm a halfway decent software dev doesn't mean i'm not a generalist. I'm not the SQL guy (though I can write it), and i'm not the SAN guy (though I know how to carve out new volumes). I can work on pretty much any problem, but my strengths will lead me to work on the things I know the best. So often i'll take on a bug if I know how to fix it. But if i'm busy, someone else might take care of it (perhaps the A guy). Doesn't really matter. If he needs help, he'll ask me or someone else, otherwise he'll figure it out. Generalists are great at figuring things out that they don't understand. But they also have to be willing to admit when they're stymied and ask for help.
- twistedpair 13y agoG is for generalist, right?
- jyu 13y agoG means choice g in this case, which helps you find the root problem. The other choices just treat the symptoms.
- twistedpair 13y agoObviously. Though the options appear so shockingly poor that they limn generalists as idiots (i.e. option a), hence my sardonic posit.
- miahi 13y agoI would do c) if it can be done without hassle, then e) and then g). The load balancer should do e) by default - don't forward to broken servers. g) takes time and concentration, and you don't want any interruptions while debugging.
- gohrt 13y ago> a) set up a cron to restart the web server every X minutes? Jeff Bezos was literally a billionaire, and many of his team millionaires, before Amazon.com deprecated that solution.
- stephengillie 13y agoa) is a much simpler solution to generate, design & implement than g). From an engineer's perspective, I can appreciate how g) feels like the correct solution, because it's solving the underlying technology problem. From a business perspective, pursuing g) could take a lot of time which could be used to build other parts of the service, which might make the issue disappear. Really, it's a cost-benefit question -- which takes more time, writing a) and dealing with lots of rebooting, or having someone pursue g)? Or maybe c) is cheaper than g) -- how many programmer-hours equals an extra 16GB of RAM? And if the memory leak isn't that bad (let's say 20 hours between these reboots), then a) can be said to make your service cheaper for your customers too.
- jongold 13y agoOP here - I think I was stretching it a littttle bit far to claim we're all good at sysadmin :) In reality—at least on my team—we're really good at UX -> visual design -> front-end MV* -> Rails
- contingencies 13y agoIn somewhat neat response to option 'g' and the phrase "generalists and ops is a route to pain", my view is that 'g' should be an automated part of your test suite and release process. Certainly, if not before then during the period you are resolving the issue... as a matter of principle.