4 ms·

>>> The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/

by gedrap 10y ago

>>> The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain?

cs02rm0 10y ago
Everyone was told to see if they can find cost savings. This just wasn't one though in the bigger picture and even after being told it wasn't one, even by their line manager and separately the CTO, they pursued it over and over. I don't believe anyone else was directly involved further up the food chain on their side. All it needed was for the question to be asked on the company's internal board and listen to the answer. Even trying it once or twice and I'd probably have forgotten about it in short order. This went on for a couple of years though!
onion2k 10y ago
How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain? If you're blindly following "orders" to reduce costs and doing things that push up costs elsewhere then you're not doing a good job. A good sysadmin (or the sysadmin's boss) should be able to pull up some numbers and say "Build tasks are being queued for an hour before they run. What impact is that having?", and call a wider meeting that brings together the higher-up-the-foodchain manager, the development team, and anyone else who might be affected. Ideally it'd be the higher up manager who calls that meeting of course, but they may not understand the technical issues.
- davidgerard 10y ago
  I have long thought that one of the most effective workplace sabotages a sysadmin could put into place was to implement management plans without question.
- FLUX-YOU 10y ago
  >If you're blindly following "orders" to reduce costs and doing things that push up costs elsewhere then you're not doing a good job. This is the responsibility of someone above to know whether or not the orders they give should be given. If they need to ask for information from people below them, fantastic, please help them along. Please don't fall on the sword for incompetent managers.
  - onion2k 10y ago
    This is the responsibility of someone above to know whether or not the orders they give should be given. Yes, and part of that is the people in their team(s) helping them and understanding that they're fallible and may fail to ask a pertinent question. Equally, the manager needs to be open to updates volunteered by their team without a prompt. Ultimately everyone does better if the entire group works together.
JeremyNT 10y ago
> How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain? Having worked in ops for > 10 years, this is how it usually goes. The SA's job tends to involve a lot of scepticism and caution. You look for problems and try to solve them proactively. One (often easy) way to solve many classes of problems is to throw hardware at them. Management always pushes back on this tactic. That's reasonable; they need to justify capital expenses (especially if you're self-hosted). The core issue though is that capital expenses are easy to quantify, while "lost productivity" is much harder to fully account for. If I complain that some hardware upgrade which costs $x could improve productivity, I just don't have hard numbers on my end - it's all napkin math. In many places reluctance to spend money on infrastructure is also, I think, a symptom of headcount-itis. Managers love to have more employees, and love to have more for them to do, because that makes managers seem more impressive to the org. My manager might have perverse incentives; keeping the SAs busy fighting scaling fires both makes his team look impressive because they're busier, and makes him look better because the capital expenditures are lower. Obviously, head count is expensive, so this is usually a game of appearances rather than an effective strategy to improve the bottom line. Good insight into productivity is required to catch this kind of stuff, but in the real world I've found that a lot of places just don't have an org structure capable of weighing cost / benefit properly when it comes to infrastructure.
- gedrap 10y ago
  Thanks for such a detailed response :)
_d8fd 10y ago
My favorite is being told I can't store stuff I need on some enterprise storage solution because it is running out of space, when I know in my head a couple more terabytes of storage costs way less than the amount of effort that went into discussing it by all parties involved, and there's no effort to help me find an alternative. So it goes into a different bucket, like S3, which is what we were trying to avoid (for various on-premise benefits) in the first place.

by gedrap 10y ago

cs02rm0 10y ago

Everyone was told to see if they can find cost savings. This just wasn't one though in the bigger picture and even after being told it wasn't one, even by their line manager and separately the CTO, they pursued it over and over. I don't believe anyone else was directly involved further up the food chain on their side. All it needed was for the question to be asked on the company's internal board and listen to the answer. Even trying it once or twice and I'd probably have forgotten about it in short order. This went on for a couple of years though!

onion2k 10y ago

How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain? If you're blindly following "orders" to reduce costs and doing things that push up costs elsewhere then you're not doing a good job. A good sysadmin (or the sysadmin's boss) should be able to pull up some numbers and say "Build tasks are being queued for an hour before they run. What impact is that having?", and call a wider meeting that brings together the higher-up-the-foodchain manager, the development team, and anyone else who might be affected. Ideally it'd be the higher up manager who calls that meeting of course, but they may not understand the technical issues.

davidgerard 10y ago
I have long thought that one of the most effective workplace sabotages a sysadmin could put into place was to implement management plans without question.
FLUX-YOU 10y ago
>If you're blindly following "orders" to reduce costs and doing things that push up costs elsewhere then you're not doing a good job. This is the responsibility of someone above to know whether or not the orders they give should be given. If they need to ask for information from people below them, fantastic, please help them along. Please don't fall on the sword for incompetent managers.
- onion2k 10y ago
  This is the responsibility of someone above to know whether or not the orders they give should be given. Yes, and part of that is the people in their team(s) helping them and understanding that they're fallible and may fail to ask a pertinent question. Equally, the manager needs to be open to updates volunteered by their team without a prompt. Ultimately everyone does better if the entire group works together.

JeremyNT 10y ago

> How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain? Having worked in ops for > 10 years, this is how it usually goes. The SA's job tends to involve a lot of scepticism and caution. You look for problems and try to solve them proactively. One (often easy) way to solve many classes of problems is to throw hardware at them. Management always pushes back on this tactic. That's reasonable; they need to justify capital expenses (especially if you're self-hosted). The core issue though is that capital expenses are easy to quantify, while "lost productivity" is much harder to fully account for. If I complain that some hardware upgrade which costs $x could improve productivity, I just don't have hard numbers on my end - it's all napkin math. In many places reluctance to spend money on infrastructure is also, I think, a symptom of headcount-itis. Managers love to have more employees, and love to have more for them to do, because that makes managers seem more impressive to the org. My manager might have perverse incentives; keeping the SAs busy fighting scaling fires both makes his team look impressive because they're busier, and makes him look better because the capital expenditures are lower. Obviously, head count is expensive, so this is usually a game of appearances rather than an effective strategy to improve the bottom line. Good insight into productivity is required to catch this kind of stuff, but in the real world I've found that a lot of places just don't have an org structure capable of weighing cost / benefit properly when it comes to infrastructure.

gedrap 10y ago
Thanks for such a detailed response :)

_d8fd 10y ago

My favorite is being told I can't store stuff I need on some enterprise storage solution because it is running out of space, when I know in my head a couple more terabytes of storage costs way less than the amount of effort that went into discussing it by all parties involved, and there's no effort to help me find an alternative. So it goes into a different bucket, like S3, which is what we were trying to avoid (for various on-premise benefits) in the first place.