8 ms·

This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin. Maybe my experience is unusual, but I

by cs02rm0 10y ago

This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin. Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue. And I've never thrown code over a wall without having tested it in a representative environment. The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. And then try and argue the toss about whether the CI server is cost effective and every few months keep switching it down despite the CTO saying it needs to be left alone. When a sysadmin sees an issue in "their" environment that they understand there's a tendency for some of them to just see that issue as the only thing the developer has had to deal with that month. In all likelihood, in a productive company, it's the most trivial issue the developer has had to resolve that day. Often this stuff goes more smoothly where the developers (I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going) manage production and there aren't people with separate job titles and the resulting friction between them. Sorry. There must be great sysadmins out there struggling with terrible developers, I'm sure of it. I just haven't seen it.

gedrap 10y ago
>>> The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain?
- cs02rm0 10y ago
  Everyone was told to see if they can find cost savings. This just wasn't one though in the bigger picture and even after being told it wasn't one, even by their line manager and separately the CTO, they pursued it over and over. I don't believe anyone else was directly involved further up the food chain on their side. All it needed was for the question to be asked on the company's internal board and listen to the answer. Even trying it once or twice and I'd probably have forgotten about it in short order. This went on for a couple of years though!
- onion2k 10y ago
  How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain? If you're blindly following "orders" to reduce costs and doing things that push up costs elsewhere then you're not doing a good job. A good sysadmin (or the sysadmin's boss) should be able to pull up some numbers and say "Build tasks are being queued for an hour before they run. What impact is that having?", and call a wider meeting that brings together the higher-up-the-foodchain manager, the development team, and anyone else who might be affected. Ideally it'd be the higher up manager who calls that meeting of course, but they may not understand the technical issues.
  - davidgerard 10y ago
    I have long thought that one of the most effective workplace sabotages a sysadmin could put into place was to implement management plans without question.
  - FLUX-YOU 10y ago
    >If you're blindly following "orders" to reduce costs and doing things that push up costs elsewhere then you're not doing a good job. This is the responsibility of someone above to know whether or not the orders they give should be given. If they need to ask for information from people below them, fantastic, please help them along. Please don't fall on the sword for incompetent managers.
    - onion2k 10y ago
      This is the responsibility of someone above to know whether or not the orders they give should be given. Yes, and part of that is the people in their team(s) helping them and understanding that they're fallible and may fail to ask a pertinent question. Equally, the manager needs to be open to updates volunteered by their team without a prompt. Ultimately everyone does better if the entire group works together.
- JeremyNT 10y ago
  > How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain? Having worked in ops for > 10 years, this is how it usually goes. The SA's job tends to involve a lot of scepticism and caution. You look for problems and try to solve them proactively. One (often easy) way to solve many classes of problems is to throw hardware at them. Management always pushes back on this tactic. That's reasonable; they need to justify capital expenses (especially if you're self-hosted). The core issue though is that capital expenses are easy to quantify, while "lost productivity" is much harder to fully account for. If I complain that some hardware upgrade which costs $x could improve productivity, I just don't have hard numbers on my end - it's all napkin math. In many places reluctance to spend money on infrastructure is also, I think, a symptom of headcount-itis. Managers love to have more employees, and love to have more for them to do, because that makes managers seem more impressive to the org. My manager might have perverse incentives; keeping the SAs busy fighting scaling fires both makes his team look impressive because they're busier, and makes him look better because the capital expenditures are lower. Obviously, head count is expensive, so this is usually a game of appearances rather than an effective strategy to improve the bottom line. Good insight into productivity is required to catch this kind of stuff, but in the real world I've found that a lot of places just don't have an org structure capable of weighing cost / benefit properly when it comes to infrastructure.
  - gedrap 10y ago
    Thanks for such a detailed response :)
- _d8fd 10y ago
  My favorite is being told I can't store stuff I need on some enterprise storage solution because it is running out of space, when I know in my head a couple more terabytes of storage costs way less than the amount of effort that went into discussing it by all parties involved, and there's no effort to help me find an alternative. So it goes into a different bucket, like S3, which is what we were trying to avoid (for various on-premise benefits) in the first place.
LoSboccacc 10y ago
> thrown code over a wall without having tested it this is the weirdest part of the whole devops mantra. like, I know how to evaluate the complexity and memory requirement of code way before I write it and I guess most compsci should be able to do the same. so either it's yet one more attempt in getting cheap labor into workable territory or plenty people where this myth originated are being cheated out of their money for a graduated curriculum that teaches nothing of value.
- hueving 10y ago
  >the complexity and memory requirement of code Those are only tiny slices of real production bugs. No amount of complexity analysis of your code ahead of time is going to protect you from all of the issues that arise with integrating any large system dealing with lots of requests. You run into all kinds of things like query optimization, kernel TCP tuning, load balancer problems, cache thrashing, high latency clients, out of spec clients, power failures, etc. If you think knowing the theoretical behavior of your program in an ideal environment is enough, you are exactly the type that throws code over a wall without having tested it.
  - LoSboccacc 10y ago
    funny how most of the things you list are either stuff that can be audited in code alone (query optimization, cache thrashing) or totally out of control of the developer (load balancer issues, tcp tuning) sure if you bounce them all up like that it might look like you have a point, except it falls apart when you attribute concerns properly. or please explain, how would dealing with kernel tcp tuning part-time help Joe Random developer write better code?
    - hueving 10y ago
      Query optimization can't be audited in code alone. The indexes you need depend heavily on the database system that you are using in production. Do you know at what point your DBMS stops loading the whole table in memory? Do you know what datastructure and algorithm it's using when you do a LIKE query? Cache thrashing also can't be audited in code alone without understanding the architecture that the the app is going to be deployed on. It's highly unlikely that the servers will have the same processor cache sizes, memory sizes, and numa architecture of the dev's laptop. Load balancer is something a developer should know about as well. A developer has to consider the behavior required by the application (e.g. backend session persistence, headers injected, etc). >please explain, how would dealing with kernel tcp tuning part-time help Joe Random developer write better code? Joe might learn that connections aren't as cheap as he thinks and maybe it isn't a great idea for each client to require 50 connections for the app to function. He might also learn that TCP isn't very efficient on high bandwidth, high latency, lossy networks and decide to switch to UDP with error correction. Long story short, a good developer should know everything about the environment in which the app is intended to run. "It performed ideally on my laptop" is throwing code over the wall. Civil engineers don't design bridges without understanding where the bridge will go. The same applies to software.
pjc50 10y ago
I've done the dual sysadmin/developer thing for a small company, and the problem I experienced there was completely incompatible working modes. Sysadmins must deal with interrupts (requests, crises, things driven by external schedules etc) and then in the rest of their time build systems to manage or reduce the interrupts. Developers are expected to produce work on a predictable schedule. This is disrupted by interrupts and obliterates the schedule for proactive work unless your management is very good at making it a priority. The "prevention of information services" problem is certainly real though. Perhaps it could be addressed by embedding the sysadmins in the dev teams rather than having a department of their own, but then you have to fight org hierarchy.
- s_kilk 10y ago
  I had exactly the same experience at a previous employer, almost word-for-word. Having said that, the central assertion is still correct: the absolute best developers I've ever worked with were also top-tier sysadmins (or linux experts, depending on what you want to call it).
- krylon 10y ago
  AMEN!!! In my current job, I am wearing both hats, and while I like that there is a certain variety in my work, users calling for help is highly disruptive when programming or doing some other stuff that requires deep focus. The upside that in a three-person IT department there is very little bureaucracy to fight, just the odd "organically grown" legacy system.
  - kefka 10y ago
    As an aside, did you know that the word "Amen" actually is a acronym in the Jewish language that means "El melekh neʾeman" (or AMN) which translates to "God, trustworthy King". (source: https://en.wikipedia.org/wiki/Amen https://en.wikipedia.org/wiki/Amen) I figured the etymology of that word was rather interesting. But yeah, I get the whole SysAd/Dev dual job. They're tough to balance and do effectively. SysAds are firefighters. When the nag(ios) alarm rings, we come a-callin.
sagichmal 10y ago
> I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going I'm sorry you've had such an awful experience with sysadmin colleagues that you've developed such a corrosive attitude towards them. I've worked in lots of good environments, where dev/ops was being effectively practiced, and sysadmins there were the most effective force multipliers imaginable.
- pjc50 10y ago
  > awful experience with sysadmin colleagues that you've developed such a corrosive attitude This is going to be a sensitive topic, but can we talk about ""BOFH"" culture somewhere on this thread? (Maybe I'm old and it's now dead, but I think some of it persists) I kind of understand it as a product of working in an environment where everything is urgent and nothing is appreciated, but when sysadmins come to resent the people they're supposed to be supporting then the force multiplier turns negative. Sysadmins develop strategies for reducing the number of requests at any cost, usually by making the experience as opaque and unhelpful as possible.
  - beachstartup 10y ago
    > where everything is urgent and nothing is appreciated in such an environment i wouldn't expect any other result. fix the environment, not the rational human response to it.
  - vinceguidry 10y ago
    When I was reading BOfH a few years ago I got a different gist. The BOfH is the archetype of someone who is excellent both with technology and politics. When you are in a service role, you have two competing priorities. You must deliver people the things they want, but also keep things nice and stable for yourself so you don't go crazy. An important dynamic in organizations is laziness vs. intimidation. Political savviness allows you to apply intimidation to get the lazy to do what you want. You can threaten to fire, or raise an issue that could possibly get them fired and even if it doesn't, won't make you look good. The BOfH is someone who can respond to political intimidation with adroit technical interventions to ensure that his second priority, ensuring a smoothly-running system, isn't threatened. If you read the BOfH stories carefully, you see that the operator knows where his bread is buttered and is careful to remain on good terms with the people who really have the power in the company. The whole thing is a phenomenal read on organizational dynamics.
johanneskanybal 10y ago
I believe in small cross functional teams but that you need to be a sysadmin to develop I don't agree with. Perhaps it's more your opinion of what a good developer means, most teams benefit from variety in my experience. It sounds like your biased to certain types of organisations where there's a big gap between departments.
BillyParadise 10y ago
Not particularly controversial at all, from my POV. I was a sysadmin with various ISPs in various countries for 15 years before I "turned to the dark side". I'd been using Ruby for a few years with Puppet and Chef, and after dealing with one too many "flaky coders", I picked it up. I have to say, coding is far more enjoyable, though both come in handy in my day-to-day life. It sounds like you've dealt with a few "BOFH" sysadmins. Don't worry, we're not all like that, and those that have been on both sides of the team will probably see your way. Tell your boss I'm available (remotely), by the way ;)
deleted 10y ago
[deleted]
friendzis 10y ago
I have seen this situation too many times (exaggerated a bit): D: I have noticed that task Frobnicate has not been running in Production for a month, then checked and it is not even added to scheduler! SA: There is no mention of Frobnicate in the pipeline for scheduled tasks. D: What pipeline? FancyPancyScheduler is bundled with application and tasks are defined in DB, I have done it in Staging and everything worked, Frobnicate is all the fuss in the team, you must have heard about it, why don't you check for changes in Staging? SA: We have well defined pipeline to manage scheduled tasks, currently the executing agent is Cron, not FancyPancyScheduler. --------------------- Developers and Admins have more or less the same goals (stable, maintainable and extensible), but on different pieces of the system (code vs infrastructure). In my short career I have seen problems arise where one party makes plans and changes according to current or even past (it worked like this earlier) state of the other party. This applies to both developers and admins. So I sort of agree with your sentiment, that developers need understanding of system administration. Though, depending on team size, I believe it is entirely sufficient to have someone in Developement who understands system administration and actual infrastructure, and someone in Operations who understands developement and actual stack. This is where I hope DevOps will end up at: arbitration between Developement and Operations to ensure smooth sailing forwards. Because the debate "I will do it in code" versus "this must be done on the edge" (e.g. static assets in a website. Served by application or web frontend?) will never be resolved. Edit: formatting
- devonkim 10y ago
  > Developers and Admins have more or less the same goals (stable, maintainable and extensible), but on different pieces of the system (code vs infrastructure). I disagree because this generalizes both developers and admins too much for my own comfort. I've seen sysadmins get really sloppy in the name of getting something into production quickly out of hubris without thinking about the full lifecycle of an application (common with developer-turned-sysadmin engineers - I am one and tend to be more reckless due to the reality that most of the errors I've observed would not have been caught going super slow - that adding more test code does not necessarily find the most critical of errors, just increases confidence) and especially in enterprise software most developers are sitting on features and are nearly allergic to new trends by their organizations valuing revenue loss far above losing growth opportunities. Of course the stereotype is that operations wants things stable and manageable at the behest of business while developers want to deploy new stuff faster (because the idea of development in most places is to create something new). Modern infrastructure becomes increasingly code-driven and emergent as opposed to manually formed and restrictively managed sysadmins will have more room for errors that may change this into the future. Meanwhile, developers are increasingly under greater scrutiny by society when rolling out features such that nobody can ignore the concerns and they may be eventually forced into nearly waterfall-like development patterns. We can already observe this with the infection of Agile with enterprise bureaucracy / overmanagement back into the rest of the software industry as many of the former smaller, agile tech companies become big behemoths themselves.
eternalban 10y ago
> This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin. It is an interesting exercise to generalize this statement in context of general engineering. It seems either your conclusion is held to be incorrect, or, we reach the conclusion that software development is not engineering.
- lmm 10y ago
  There's no value in arguing over the semantics of "engineering". There are huge differences between software development and e.g. civil engineering, to the point that I would be dubious about any analogy that treated them as the same thing.
  - eternalban 10y ago
    Chemical engineering, Hydroelectrical engineering, Power engineering come immediately to mind as engineering disciplines that deal with active systems that require operational management and control.
    - lmm 10y ago
      Sure, but those are still all very different from software development.
      - eternalban 10y ago
        Of course. (In my opinion, /high software/ has more in common with mathematics, music, theatre-film-dance, and architecture than it has with engineering, and /low software/ is begining to resemble boiler room operations.) But here, as example is my BSEE a.m.: http://eng.rpi.edu/academics http://eng.rpi.edu/academics And to this day, we hear about "software engineers" and "software engineering". Per my OP: "It seems either your conclusion is held to be incorrect, or, we reach the conclusion that software development is not engineering." Possibly, one reason for the prevalent problems in the pedagogical & human resource fulfillment aspects of the field is due to a miscategorization of the field.
godmodus 10y ago
True for most devs that do any backend work. Many devs out there who work with Windows or do mostly front end often have little experience in that domain. Seeing alot of work get done at uni by students - who also actually some backend (friend did a blockchain project recently) did infact do very little backend discovery - the job was delegated to another student to get the env. Up and running.
Sakes 10y ago
"This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin." Couldn't this argument be applied to any developer for any discipline/speciality? Sure, more knowledge/context is always better if reasonable to attain, but my experience suggests that your above concerns could also be addressed via team organization rather than expecting all developers to know all things.
Clubber 10y ago
I agree. I also think it should work both ways. The worst jobs I've ever had is when the sysadmins have the mindset where they own the servers and are unwilling to deviate from what they've read at the behest of the developers. "I will force AV on reads on the developer boxes." "I will install AV on the production DB servers without telling anyone in the development group, then make the developers prove AV was the cause of production slowness before removing it two weeks later." "I will force this crazy group policy on developers and when they complain, I will totally ignore them." A bad, or uncompromising sysadmin (one in the same) make development work a complete nightmare. I half believe the reason developers are embracing cloud architecture so much is to remove so many sysadmins out of the equation. On a side note, a tip for developers. Always make friends with the sysadmins. Buy them lunch or something. Right or wrong, they can make your lives much better or much more miserable.
- greedo 10y ago
  One of our web teams wanted to do a simple Wordpress deployment on LAMP. As sysadmins, that was no issue, even with clustered mariadb. But our DBA team doesn't have any experience outside of Oracle/MSSQL, and squawked about mariadb. After an hour of this BS, the manager of the web team spun up a few EC2 instances and got to work. Of course we don't have anyone familiar with EC2, so supporting that will be a learning curve for someone, but the manager is happy, he has his own sandbox without hassles from the DBAs.
- deleted 10y ago
  [deleted]
- briffle 10y ago
  what your probably not seeing, is the CIO/CSO screaming at the SA to get AV deployed on every machine in the company, to meet some audit requirement checkbox, or PCI compliance, by the end of the month.
  - Sacho 10y ago
    So the problem is effective communication? Why can't the sysadmin in this imagined scenario explain their actions this way?
technofiend 10y ago
>Often this stuff goes more smoothly where the developers (I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going) manage production and there aren't people with separate job titles and the resulting friction between them. This is not personal criticism but you know how I know you're not working in a highly regulated environment? Check out the Carnegie Mellon Capability and Maturity Model (CMM) as a counterexample of where some companies go. Development is not at one remove but two from production support. There's an "operate" team between them and production environments and in a regulated environment operate doesn't have privileged access either. That'll be a third team due to separation of duties requirements. Now imagine you're paged out to a call where your code is slow or failing and you're not even allowed to login to where the issue's happening. Fun, right? This is why I'm absolutely loving the devops changes we're seeing now - because developers can control the environment without retaining control of it. My ideal is to apply some sensible defaults (no, you can't have all my crashdump space for your app logging; ask for more disk instead, no you can't run ghost/glibc/pooodle vulnerable versions of libraries) and otherwise let the developers spec the OS as a template or dependency for their app. It's much better for me since if I'm required to troubleshoot I know my requirements are met and otherwise the developer may do as they wish. Everyone wins and my control requirements are satisfied because remember developers are never allowed production access in regulated environments. >Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue. I guess it depends on what you mean? The developer is in the best position to know what logging there is and how to enable or it increase verbosity. But they may be completely ignorant of how the operating system's tcp stack, memory management or other mechanisms work. Have you ever had to explain to someone that a java out of memory error had nothing do with the fact that linux is using otherwise idle memory to buffer i/o and that they're misreading top output? That the actual issue is their object management and just increasing the JVM's heap size is at best a bandaid? If you have a developer who insists every issue is the operating system, sometimes the SA has to know how to dig in and run stack traces, probe tools (systrace, dtrace, whatever), jmx queries, etc until they can pinpoint the offending code. As another example if you have an application that isn't draining queues quickly enough and therefore sending back tcp zero window frames upstream, what's the solution? A hypothetical lazy developer will say "it's the OS not queuing enough data, increase the OS buffers." A hypothetical lazy SA may say "it's the app not consuming packets quickly enough, rewrite the app." In reality if we've all been paged to a priority one bridge the solution will probably be the combination of the two - tactical fix of increasing buffers to create some time for development to understand why the code isn't doing what it should and fix it.
ChemicalWarfare 10y ago
DevOps approach definitely shifts some sysadmin-type roles towards the development teams. That said though there are things that belong in the realm of sysadmin responsibilities - both "old school" like let's say setting up DB replication, VPNs or a puppet master server(not that "old school" but still) and "new" - things like Kubernetes and Fleet/CoreOS for example have plenty of configuration/maintenance complexity that is better suited to be handled by a dedicated sysadmin.
falcolas 10y ago
Sysadmins are, 99% of the time, responsible for production uptime. Sadly, management includes all of the software being written in-house in this expectation. Change means instability, means that Sysadmins' feet are held to the fire - this makes them resistant to change. Software developers, on the other hand, are responsible for making changes. Adding features, pushing fixes, and so forth. These two points of view are inevitably going to cause friction. Developers are only recently starting to be held responsible for production uptime and the pages that come along with that - and it's a good thing for both sides. That "most trivial issue" for a developer is something the sysadmin was woken up for 3x in the past, and doesn't want to be woken up for again, so he pushes back. How can he not?
- lanstin 10y ago
  In 1997 when my code first started running in Live I was given a pager and told welcome to Ops. Every piece of software that company had a single dev name easily read out from the binary and used to contact when trouble occurred.
- deleted 10y ago
  [deleted]
jstimpfle 10y ago
> but I don't think you get to be a half decent developer without being a reasonable sysadmin I think you've just described me. And I don't see the argument against sysadmin. > leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. Sorry I just can't take you seriously.
- monocasa 10y ago
  Honestly, I've heard worse out of sysadmins in some places.
  - jstimpfle 10y ago
    Question: What must go wrong for a build to be 180x slower on a server than on a laptop?
    - cs02rm0 10y ago
      Using an EC2 t2.nano instance for a build server. When they're out of CPU credits it's game over.
    - contingencies 10y ago
      Maybes: Packaging latency in archive formats (compress before upload, decompress after). Network latency on the upload/download. Block IO performance on the server. Virtualization overhead. Memory or processor constraints. Assumption of equivalence is spurious (eg. server is doing multi-architecture builds and full suites of tests including eg. regression tests). Yep, something like that.
    - monocasa 10y ago
      Overcommit of resources, causing near constant thrashing to disk.
tristor 10y ago
> Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue. I've had the exact opposite experience. In most of the organizations I've worked in the "sysadmins" (mostly Systems Engineers/Operations Engineers actually) were stronger developers than the people who were actually developing the software. But that could just be a title shift, because what I've seen happen is that people who care about systems but have a development background gravitate towards operations roles and end up filling in as the "actually Senior" developer for the dev teams. In the 15 years I've been doing this, I've only occasionally met someone who has stuck hard to the development side of the house but actually is competent when it comes to systems. Most developers have zero care about any of the lower level things like networks, hardware, and even backend software/databases which are required for their application to succeed. A common scenario is that the devs choose an inappropriate backend stack because they chose the easiest things to deploy rather than what is best suited for the use case. Then when things blow up, they beg for an ops team to be created, which usually starts by hiring people who are competent enough developers they can relatively painlessly replace the entire backend with something sane (e.g. Mongo to Postgres shifts are commonplace, because Mongo is a dog in the real world). > The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. And then try and argue the toss about whether the CI server is cost effective and every few months keep switching it down despite the CTO saying it needs to be left alone. Yeah, that does sound terrible. I agree. My top 5 jobs as a systems person is the following in priority order: 1. Make sure production stays up for our customers so we keep making money. (5 9s targets) 2. Ensure the security (and compliance) of our systems so we don't get hacked and we maintain customer expectations about compliance. 3. Ensure the performance of our product/systems is up to customer expectations. 4. Make sure deployment automation is solid and streamlined so that deployments are frictionless 5. Make sure new code is actually being deployed regularly and remove impediments to deployment so customers get features faster. You'll notice a trend here I'm sure. The most important thing is the customer, then the developer. The biggest frictions I've seen between systems/development teams is when the development team believes that their desires/needs are the highest priority. The systems team is /not/ there to be at the beck and call of the development team, it's to be at the beck and call of the customer who is paying the company money. As much as possible I try to ensure the development team is having a frictionless experience, but if something will negatively impact the customer it is 100% my job to throw a roadblock in the way of the development team to prevent that. The customer of the company is my priority, and everything else is secondary.

by cs02rm0 10y ago

gedrap 10y ago

>>> The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain?

cs02rm0 10y ago
Everyone was told to see if they can find cost savings. This just wasn't one though in the bigger picture and even after being told it wasn't one, even by their line manager and separately the CTO, they pursued it over and over. I don't believe anyone else was directly involved further up the food chain on their side. All it needed was for the question to be asked on the company's internal board and listen to the answer. Even trying it once or twice and I'd probably have forgotten about it in short order. This went on for a couple of years though!
onion2k 10y ago
How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain? If you're blindly following "orders" to reduce costs and doing things that push up costs elsewhere then you're not doing a good job. A good sysadmin (or the sysadmin's boss) should be able to pull up some numbers and say "Build tasks are being queued for an hour before they run. What impact is that having?", and call a wider meeting that brings together the higher-up-the-foodchain manager, the development team, and anyone else who might be affected. Ideally it'd be the higher up manager who calls that meeting of course, but they may not understand the technical issues.
- davidgerard 10y ago
  I have long thought that one of the most effective workplace sabotages a sysadmin could put into place was to implement management plans without question.
- FLUX-YOU 10y ago
  >If you're blindly following "orders" to reduce costs and doing things that push up costs elsewhere then you're not doing a good job. This is the responsibility of someone above to know whether or not the orders they give should be given. If they need to ask for information from people below them, fantastic, please help them along. Please don't fall on the sword for incompetent managers.
  - onion2k 10y ago
    This is the responsibility of someone above to know whether or not the orders they give should be given. Yes, and part of that is the people in their team(s) helping them and understanding that they're fallible and may fail to ask a pertinent question. Equally, the manager needs to be open to updates volunteered by their team without a prompt. Ultimately everyone does better if the entire group works together.
JeremyNT 10y ago
> How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain? Having worked in ops for > 10 years, this is how it usually goes. The SA's job tends to involve a lot of scepticism and caution. You look for problems and try to solve them proactively. One (often easy) way to solve many classes of problems is to throw hardware at them. Management always pushes back on this tactic. That's reasonable; they need to justify capital expenses (especially if you're self-hosted). The core issue though is that capital expenses are easy to quantify, while "lost productivity" is much harder to fully account for. If I complain that some hardware upgrade which costs $x could improve productivity, I just don't have hard numbers on my end - it's all napkin math. In many places reluctance to spend money on infrastructure is also, I think, a symptom of headcount-itis. Managers love to have more employees, and love to have more for them to do, because that makes managers seem more impressive to the org. My manager might have perverse incentives; keeping the SAs busy fighting scaling fires both makes his team look impressive because they're busier, and makes him look better because the capital expenditures are lower. Obviously, head count is expensive, so this is usually a game of appearances rather than an effective strategy to improve the bottom line. Good insight into productivity is required to catch this kind of stuff, but in the real world I've found that a lot of places just don't have an org structure capable of weighing cost / benefit properly when it comes to infrastructure.
- gedrap 10y ago
  Thanks for such a detailed response :)
_d8fd 10y ago
My favorite is being told I can't store stuff I need on some enterprise storage solution because it is running out of space, when I know in my head a couple more terabytes of storage costs way less than the amount of effort that went into discussing it by all parties involved, and there's no effort to help me find an alternative. So it goes into a different bucket, like S3, which is what we were trying to avoid (for various on-premise benefits) in the first place.

LoSboccacc 10y ago

> thrown code over a wall without having tested it this is the weirdest part of the whole devops mantra. like, I know how to evaluate the complexity and memory requirement of code way before I write it and I guess most compsci should be able to do the same. so either it's yet one more attempt in getting cheap labor into workable territory or plenty people where this myth originated are being cheated out of their money for a graduated curriculum that teaches nothing of value.

hueving 10y ago
>the complexity and memory requirement of code Those are only tiny slices of real production bugs. No amount of complexity analysis of your code ahead of time is going to protect you from all of the issues that arise with integrating any large system dealing with lots of requests. You run into all kinds of things like query optimization, kernel TCP tuning, load balancer problems, cache thrashing, high latency clients, out of spec clients, power failures, etc. If you think knowing the theoretical behavior of your program in an ideal environment is enough, you are exactly the type that throws code over a wall without having tested it.
- LoSboccacc 10y ago
  funny how most of the things you list are either stuff that can be audited in code alone (query optimization, cache thrashing) or totally out of control of the developer (load balancer issues, tcp tuning) sure if you bounce them all up like that it might look like you have a point, except it falls apart when you attribute concerns properly. or please explain, how would dealing with kernel tcp tuning part-time help Joe Random developer write better code?
  - hueving 10y ago
    Query optimization can't be audited in code alone. The indexes you need depend heavily on the database system that you are using in production. Do you know at what point your DBMS stops loading the whole table in memory? Do you know what datastructure and algorithm it's using when you do a LIKE query? Cache thrashing also can't be audited in code alone without understanding the architecture that the the app is going to be deployed on. It's highly unlikely that the servers will have the same processor cache sizes, memory sizes, and numa architecture of the dev's laptop. Load balancer is something a developer should know about as well. A developer has to consider the behavior required by the application (e.g. backend session persistence, headers injected, etc). >please explain, how would dealing with kernel tcp tuning part-time help Joe Random developer write better code? Joe might learn that connections aren't as cheap as he thinks and maybe it isn't a great idea for each client to require 50 connections for the app to function. He might also learn that TCP isn't very efficient on high bandwidth, high latency, lossy networks and decide to switch to UDP with error correction. Long story short, a good developer should know everything about the environment in which the app is intended to run. "It performed ideally on my laptop" is throwing code over the wall. Civil engineers don't design bridges without understanding where the bridge will go. The same applies to software.

pjc50 10y ago

I've done the dual sysadmin/developer thing for a small company, and the problem I experienced there was completely incompatible working modes. Sysadmins must deal with interrupts (requests, crises, things driven by external schedules etc) and then in the rest of their time build systems to manage or reduce the interrupts. Developers are expected to produce work on a predictable schedule. This is disrupted by interrupts and obliterates the schedule for proactive work unless your management is very good at making it a priority. The "prevention of information services" problem is certainly real though. Perhaps it could be addressed by embedding the sysadmins in the dev teams rather than having a department of their own, but then you have to fight org hierarchy.

s_kilk 10y ago
I had exactly the same experience at a previous employer, almost word-for-word. Having said that, the central assertion is still correct: the absolute best developers I've ever worked with were also top-tier sysadmins (or linux experts, depending on what you want to call it).
krylon 10y ago
AMEN!!! In my current job, I am wearing both hats, and while I like that there is a certain variety in my work, users calling for help is highly disruptive when programming or doing some other stuff that requires deep focus. The upside that in a three-person IT department there is very little bureaucracy to fight, just the odd "organically grown" legacy system.
- kefka 10y ago
  As an aside, did you know that the word "Amen" actually is a acronym in the Jewish language that means "El melekh neʾeman" (or AMN) which translates to "God, trustworthy King". (source: https://en.wikipedia.org/wiki/Amen https://en.wikipedia.org/wiki/Amen) I figured the etymology of that word was rather interesting. But yeah, I get the whole SysAd/Dev dual job. They're tough to balance and do effectively. SysAds are firefighters. When the nag(ios) alarm rings, we come a-callin.

sagichmal 10y ago

> I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going I'm sorry you've had such an awful experience with sysadmin colleagues that you've developed such a corrosive attitude towards them. I've worked in lots of good environments, where dev/ops was being effectively practiced, and sysadmins there were the most effective force multipliers imaginable.

pjc50 10y ago
> awful experience with sysadmin colleagues that you've developed such a corrosive attitude This is going to be a sensitive topic, but can we talk about ""BOFH"" culture somewhere on this thread? (Maybe I'm old and it's now dead, but I think some of it persists) I kind of understand it as a product of working in an environment where everything is urgent and nothing is appreciated, but when sysadmins come to resent the people they're supposed to be supporting then the force multiplier turns negative. Sysadmins develop strategies for reducing the number of requests at any cost, usually by making the experience as opaque and unhelpful as possible.
- beachstartup 10y ago
  > where everything is urgent and nothing is appreciated in such an environment i wouldn't expect any other result. fix the environment, not the rational human response to it.
- vinceguidry 10y ago
  When I was reading BOfH a few years ago I got a different gist. The BOfH is the archetype of someone who is excellent both with technology and politics. When you are in a service role, you have two competing priorities. You must deliver people the things they want, but also keep things nice and stable for yourself so you don't go crazy. An important dynamic in organizations is laziness vs. intimidation. Political savviness allows you to apply intimidation to get the lazy to do what you want. You can threaten to fire, or raise an issue that could possibly get them fired and even if it doesn't, won't make you look good. The BOfH is someone who can respond to political intimidation with adroit technical interventions to ensure that his second priority, ensuring a smoothly-running system, isn't threatened. If you read the BOfH stories carefully, you see that the operator knows where his bread is buttered and is careful to remain on good terms with the people who really have the power in the company. The whole thing is a phenomenal read on organizational dynamics.

johanneskanybal 10y ago

I believe in small cross functional teams but that you need to be a sysadmin to develop I don't agree with. Perhaps it's more your opinion of what a good developer means, most teams benefit from variety in my experience. It sounds like your biased to certain types of organisations where there's a big gap between departments.

BillyParadise 10y ago

Not particularly controversial at all, from my POV. I was a sysadmin with various ISPs in various countries for 15 years before I "turned to the dark side". I'd been using Ruby for a few years with Puppet and Chef, and after dealing with one too many "flaky coders", I picked it up. I have to say, coding is far more enjoyable, though both come in handy in my day-to-day life. It sounds like you've dealt with a few "BOFH" sysadmins. Don't worry, we're not all like that, and those that have been on both sides of the team will probably see your way. Tell your boss I'm available (remotely), by the way ;)

deleted 10y ago

[deleted]

friendzis 10y ago

I have seen this situation too many times (exaggerated a bit): D: I have noticed that task Frobnicate has not been running in Production for a month, then checked and it is not even added to scheduler! SA: There is no mention of Frobnicate in the pipeline for scheduled tasks. D: What pipeline? FancyPancyScheduler is bundled with application and tasks are defined in DB, I have done it in Staging and everything worked, Frobnicate is all the fuss in the team, you must have heard about it, why don't you check for changes in Staging? SA: We have well defined pipeline to manage scheduled tasks, currently the executing agent is Cron, not FancyPancyScheduler. --------------------- Developers and Admins have more or less the same goals (stable, maintainable and extensible), but on different pieces of the system (code vs infrastructure). In my short career I have seen problems arise where one party makes plans and changes according to current or even past (it worked like this earlier) state of the other party. This applies to both developers and admins. So I sort of agree with your sentiment, that developers need understanding of system administration. Though, depending on team size, I believe it is entirely sufficient to have someone in Developement who understands system administration and actual infrastructure, and someone in Operations who understands developement and actual stack. This is where I hope DevOps will end up at: arbitration between Developement and Operations to ensure smooth sailing forwards. Because the debate "I will do it in code" versus "this must be done on the edge" (e.g. static assets in a website. Served by application or web frontend?) will never be resolved. Edit: formatting

devonkim 10y ago
> Developers and Admins have more or less the same goals (stable, maintainable and extensible), but on different pieces of the system (code vs infrastructure). I disagree because this generalizes both developers and admins too much for my own comfort. I've seen sysadmins get really sloppy in the name of getting something into production quickly out of hubris without thinking about the full lifecycle of an application (common with developer-turned-sysadmin engineers - I am one and tend to be more reckless due to the reality that most of the errors I've observed would not have been caught going super slow - that adding more test code does not necessarily find the most critical of errors, just increases confidence) and especially in enterprise software most developers are sitting on features and are nearly allergic to new trends by their organizations valuing revenue loss far above losing growth opportunities. Of course the stereotype is that operations wants things stable and manageable at the behest of business while developers want to deploy new stuff faster (because the idea of development in most places is to create something new). Modern infrastructure becomes increasingly code-driven and emergent as opposed to manually formed and restrictively managed sysadmins will have more room for errors that may change this into the future. Meanwhile, developers are increasingly under greater scrutiny by society when rolling out features such that nobody can ignore the concerns and they may be eventually forced into nearly waterfall-like development patterns. We can already observe this with the infection of Agile with enterprise bureaucracy / overmanagement back into the rest of the software industry as many of the former smaller, agile tech companies become big behemoths themselves.

eternalban 10y ago

> This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin. It is an interesting exercise to generalize this statement in context of general engineering. It seems either your conclusion is held to be incorrect, or, we reach the conclusion that software development is not engineering.

lmm 10y ago
There's no value in arguing over the semantics of "engineering". There are huge differences between software development and e.g. civil engineering, to the point that I would be dubious about any analogy that treated them as the same thing.
- eternalban 10y ago
  Chemical engineering, Hydroelectrical engineering, Power engineering come immediately to mind as engineering disciplines that deal with active systems that require operational management and control.
  - lmm 10y ago
    Sure, but those are still all very different from software development.
    - eternalban 10y ago
      Of course. (In my opinion, /high software/ has more in common with mathematics, music, theatre-film-dance, and architecture than it has with engineering, and /low software/ is begining to resemble boiler room operations.) But here, as example is my BSEE a.m.: http://eng.rpi.edu/academics http://eng.rpi.edu/academics And to this day, we hear about "software engineers" and "software engineering". Per my OP: "It seems either your conclusion is held to be incorrect, or, we reach the conclusion that software development is not engineering." Possibly, one reason for the prevalent problems in the pedagogical & human resource fulfillment aspects of the field is due to a miscategorization of the field.

godmodus 10y ago

True for most devs that do any backend work. Many devs out there who work with Windows or do mostly front end often have little experience in that domain. Seeing alot of work get done at uni by students - who also actually some backend (friend did a blockchain project recently) did infact do very little backend discovery - the job was delegated to another student to get the env. Up and running.

Sakes 10y ago

"This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin." Couldn't this argument be applied to any developer for any discipline/speciality? Sure, more knowledge/context is always better if reasonable to attain, but my experience suggests that your above concerns could also be addressed via team organization rather than expecting all developers to know all things.

Clubber 10y ago

I agree. I also think it should work both ways. The worst jobs I've ever had is when the sysadmins have the mindset where they own the servers and are unwilling to deviate from what they've read at the behest of the developers. "I will force AV on reads on the developer boxes." "I will install AV on the production DB servers without telling anyone in the development group, then make the developers prove AV was the cause of production slowness before removing it two weeks later." "I will force this crazy group policy on developers and when they complain, I will totally ignore them." A bad, or uncompromising sysadmin (one in the same) make development work a complete nightmare. I half believe the reason developers are embracing cloud architecture so much is to remove so many sysadmins out of the equation. On a side note, a tip for developers. Always make friends with the sysadmins. Buy them lunch or something. Right or wrong, they can make your lives much better or much more miserable.

greedo 10y ago
One of our web teams wanted to do a simple Wordpress deployment on LAMP. As sysadmins, that was no issue, even with clustered mariadb. But our DBA team doesn't have any experience outside of Oracle/MSSQL, and squawked about mariadb. After an hour of this BS, the manager of the web team spun up a few EC2 instances and got to work. Of course we don't have anyone familiar with EC2, so supporting that will be a learning curve for someone, but the manager is happy, he has his own sandbox without hassles from the DBAs.
deleted 10y ago
[deleted]
briffle 10y ago
what your probably not seeing, is the CIO/CSO screaming at the SA to get AV deployed on every machine in the company, to meet some audit requirement checkbox, or PCI compliance, by the end of the month.
- Sacho 10y ago
  So the problem is effective communication? Why can't the sysadmin in this imagined scenario explain their actions this way?

technofiend 10y ago

>Often this stuff goes more smoothly where the developers (I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going) manage production and there aren't people with separate job titles and the resulting friction between them. This is not personal criticism but you know how I know you're not working in a highly regulated environment? Check out the Carnegie Mellon Capability and Maturity Model (CMM) as a counterexample of where some companies go. Development is not at one remove but two from production support. There's an "operate" team between them and production environments and in a regulated environment operate doesn't have privileged access either. That'll be a third team due to separation of duties requirements. Now imagine you're paged out to a call where your code is slow or failing and you're not even allowed to login to where the issue's happening. Fun, right? This is why I'm absolutely loving the devops changes we're seeing now - because developers can control the environment without retaining control of it. My ideal is to apply some sensible defaults (no, you can't have all my crashdump space for your app logging; ask for more disk instead, no you can't run ghost/glibc/pooodle vulnerable versions of libraries) and otherwise let the developers spec the OS as a template or dependency for their app. It's much better for me since if I'm required to troubleshoot I know my requirements are met and otherwise the developer may do as they wish. Everyone wins and my control requirements are satisfied because remember developers are never allowed production access in regulated environments. >Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue. I guess it depends on what you mean? The developer is in the best position to know what logging there is and how to enable or it increase verbosity. But they may be completely ignorant of how the operating system's tcp stack, memory management or other mechanisms work. Have you ever had to explain to someone that a java out of memory error had nothing do with the fact that linux is using otherwise idle memory to buffer i/o and that they're misreading top output? That the actual issue is their object management and just increasing the JVM's heap size is at best a bandaid? If you have a developer who insists every issue is the operating system, sometimes the SA has to know how to dig in and run stack traces, probe tools (systrace, dtrace, whatever), jmx queries, etc until they can pinpoint the offending code. As another example if you have an application that isn't draining queues quickly enough and therefore sending back tcp zero window frames upstream, what's the solution? A hypothetical lazy developer will say "it's the OS not queuing enough data, increase the OS buffers." A hypothetical lazy SA may say "it's the app not consuming packets quickly enough, rewrite the app." In reality if we've all been paged to a priority one bridge the solution will probably be the combination of the two - tactical fix of increasing buffers to create some time for development to understand why the code isn't doing what it should and fix it.

ChemicalWarfare 10y ago

DevOps approach definitely shifts some sysadmin-type roles towards the development teams. That said though there are things that belong in the realm of sysadmin responsibilities - both "old school" like let's say setting up DB replication, VPNs or a puppet master server(not that "old school" but still) and "new" - things like Kubernetes and Fleet/CoreOS for example have plenty of configuration/maintenance complexity that is better suited to be handled by a dedicated sysadmin.

falcolas 10y ago

Sysadmins are, 99% of the time, responsible for production uptime. Sadly, management includes all of the software being written in-house in this expectation. Change means instability, means that Sysadmins' feet are held to the fire - this makes them resistant to change. Software developers, on the other hand, are responsible for making changes. Adding features, pushing fixes, and so forth. These two points of view are inevitably going to cause friction. Developers are only recently starting to be held responsible for production uptime and the pages that come along with that - and it's a good thing for both sides. That "most trivial issue" for a developer is something the sysadmin was woken up for 3x in the past, and doesn't want to be woken up for again, so he pushes back. How can he not?

lanstin 10y ago
In 1997 when my code first started running in Live I was given a pager and told welcome to Ops. Every piece of software that company had a single dev name easily read out from the binary and used to contact when trouble occurred.
deleted 10y ago
[deleted]

jstimpfle 10y ago

> but I don't think you get to be a half decent developer without being a reasonable sysadmin I think you've just described me. And I don't see the argument against sysadmin. > leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. Sorry I just can't take you seriously.

monocasa 10y ago
Honestly, I've heard worse out of sysadmins in some places.
- jstimpfle 10y ago
  Question: What must go wrong for a build to be 180x slower on a server than on a laptop?
  - cs02rm0 10y ago
    Using an EC2 t2.nano instance for a build server. When they're out of CPU credits it's game over.
  - contingencies 10y ago
    Maybes: Packaging latency in archive formats (compress before upload, decompress after). Network latency on the upload/download. Block IO performance on the server. Virtualization overhead. Memory or processor constraints. Assumption of equivalence is spurious (eg. server is doing multi-architecture builds and full suites of tests including eg. regression tests). Yep, something like that.
  - monocasa 10y ago
    Overcommit of resources, causing near constant thrashing to disk.

tristor 10y ago

> Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue. I've had the exact opposite experience. In most of the organizations I've worked in the "sysadmins" (mostly Systems Engineers/Operations Engineers actually) were stronger developers than the people who were actually developing the software. But that could just be a title shift, because what I've seen happen is that people who care about systems but have a development background gravitate towards operations roles and end up filling in as the "actually Senior" developer for the dev teams. In the 15 years I've been doing this, I've only occasionally met someone who has stuck hard to the development side of the house but actually is competent when it comes to systems. Most developers have zero care about any of the lower level things like networks, hardware, and even backend software/databases which are required for their application to succeed. A common scenario is that the devs choose an inappropriate backend stack because they chose the easiest things to deploy rather than what is best suited for the use case. Then when things blow up, they beg for an ops team to be created, which usually starts by hiring people who are competent enough developers they can relatively painlessly replace the entire backend with something sane (e.g. Mongo to Postgres shifts are commonplace, because Mongo is a dog in the real world). > The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. And then try and argue the toss about whether the CI server is cost effective and every few months keep switching it down despite the CTO saying it needs to be left alone. Yeah, that does sound terrible. I agree. My top 5 jobs as a systems person is the following in priority order: 1. Make sure production stays up for our customers so we keep making money. (5 9s targets) 2. Ensure the security (and compliance) of our systems so we don't get hacked and we maintain customer expectations about compliance. 3. Ensure the performance of our product/systems is up to customer expectations. 4. Make sure deployment automation is solid and streamlined so that deployments are frictionless 5. Make sure new code is actually being deployed regularly and remove impediments to deployment so customers get features faster. You'll notice a trend here I'm sure. The most important thing is the customer, then the developer. The biggest frictions I've seen between systems/development teams is when the development team believes that their desires/needs are the highest priority. The systems team is /not/ there to be at the beck and call of the development team, it's to be at the beck and call of the customer who is paying the company money. As much as possible I try to ensure the development team is having a frictionless experience, but if something will negatively impact the customer it is 100% my job to throw a roadblock in the way of the development team to prevent that. The customer of the company is my priority, and everything else is secondary.