27 ms·

Software Developers Should Have Sysadmin Experience

• gedrap 10y ago
  That's what I did at my tiny engineering team (3 folks) and results have been great. If you have a small organization (say <10 engineers), it's crucial that every developer writing server side applications can also do at least some sysadmin work. As the article says, it leads to deeper understanding and it really helps when thinking about scalability, fixing certain kinds of issues, etc. It also often shortens the feedback cycle and requires less throwing over the wall. As a bonus, you increase the bus factor. Automation is the key though. If everyone connects to the boxes and does random things manually over ssh, nothing good will come out of it. Still, you need to have a person or two who are responsible for the vision of the architecture/systems and who make sure that things don't go off the rails.
• arca_vorago 10y ago
  This is a short little article that just barely touches on a much deeper, often hidden issue; The state of system administration in business is abysmal. It's not the developers fault though, at least not as much as devops types would like you to believe. I think the author has a good point, in that its good to get Devs thinking about real world environments on deploy, but the real world is much more complex than concurrency of servers. All that being said though, very rarely have I as a sysadmin of 10+ years seen problems so easily attributable to devs. Of course I haven't lived in hn/sv startup land either, so take that into account, but failures in systems I have seen have almost always been a failure of management, up to and beyond C level. I could go into detail, but I'll save it for another time. Suffice it to say, what businesses need to be doing is getting better CTOs and CIOs who can bridge the gaping chasm between sysadmins and managment. Devs, you keep being Devs, and let the sysadmins be sysadmins. Cross train and communicate when you can, but don't fool yourselves, it is management that bears the responsibility and burden of you both. Management just doesn't like to admit that to themselves or anyone else, so don't play into this Devs vs sysadmins dialectic too much, lest ye find yourself the scapegoat next go round.
  • eropple 10y ago
    I agree that the dialectic shouldn't be played...but not that it's "cross-training." Rather, it's the same skillsets being applied in slightly different ways. There was a day in which "sysadmin" very often meant "shit-hot Perl slinger." It was before my day, but I know some of the graybeards who can still lay claim to it. The sysadmin who can't write good, maintainable code is going to rapidly see their positions reduced to sinecures in large and slow companies. That may be enough to finish out a career, but I wouldn't bet on it if I was under 50 (and I don't bet on it; I have always, as said elsewhere in this thread, framed myself as "a developer whose output is configured systems" rather than "a sysadmin" for this reason). Similarly, in an age where infrastructure-as-code is becoming the norm, you had better be able to work with it or, as a developer, you are very limited in what you can do without being blessed by somebody else--and the set of environments where that's gonna fly is shrinking, too. This is emphatically not to take any weight off of the shoulders of management, to be sure. But rather that I feel very strongly that what divide existed between these "disciplines" no longer exists, and both developers and traditional sysadmins need to move to catch up.
    • arca_vorago 10y ago
      > The sysadmin who can't write good, maintainable code is going to rapidly see their positions reduced to sinecures in large and slow companies. I completely disagree, but respectfully. What I think we are lacking in this discussion is a differentiation between what we mean by sysadmin as a product and as a job description. Systems Administration is a aggregate management of the technical infrastructure. (for $reasons). In super small technical infrastructure, such as webdev startups, there is very little active sysadmining to do if done properly, but as complexities grow, you need multiple people to perform all the various duties needed to maintain a system. In the performance of these duties, there are various descriptions with varying levels of requirements. What I postulate is that traditional businesses in seeing the agility of the dotcom and startup culture have been attempting to cut costs in the technical infrastructure, but for the most part because the managers of the technical infrastructure (the sysadmins) haven't had anyone to advocate on their behalf, this has resulted in poor business performance bottlenecks. I would be curious to hear what other have to say, but in the "shit-hot Perl slinger." days, usually the senior sysadmin was the slinger, and he reported directly to the CFO and CEO, if not ownership. What I argue is that systems have gained in complexity from a computing infrastructure standpoint such that this old structure no longer worked well, hence the creation of the CTO/CIO class(very similar). The problem is, they aren't doing a good job in my opinion. Therefore, in this current business climate, my argument is that the main problem is we already have good code slinging sysadmins on hand (or can train them), but what businesses need/the industry is lacking, is business/political game aware senior sysadmins to make up for that failing of the C's. Hence I disagree the days of non-code slinging sysadmins are going anywhere. Indeed, some of the best senior sysadmins I know live in meetings, but if they are performing well in fulfilling that role, and not slinging code, but instead making big picture, high-level overview decisions and then monitoring progress, I don't think there is anything wrong with that. Now, if we got CIO/CTOs back on the right track, the political/business game demand could fall and those sysadmins could get back into hard work, (which just happens to sometimes involve code-slinging to solve problems.) Devs are an entirely seperate entity, but still part of the process, in anything other than a pure web startup type businesses with little to no real infrastructure (eg, workfrom home contractors). Of course I want to qualify this in that this is ancedotal and I admit I haven't seen every environment, but I have seen many environments (as a contractor, seeing more insides than the guy going for retirement), from fortune 500s to 2 man lawyer shops. Any business that can see this issue coming and address it head on will be far ahead of the game. Those that don't, will one day have very rude wakeup calls as the complexity exponentially increases and they don't have the structures in place the handle the demand, mostly due to lack of foresight/vision.
• kqr2 10y ago
  Somewhat related, I feel that mechanical engineers who design cars should have some experience servicing vehicles. Sometimes engineers will not leave enough space, use weird fasteners, etc. that make a simple job much more complex.
  • peterwwillis 10y ago
    That is a function of design requirements, not lack of foresight. If your manager tells you the cab needs to be 20% roomier without changing the outside dimensions, or they change the engine going into the thing without changing the structural integrity of the car, there goes your maintenance space.
    • markdown 10y ago
      I feel that managers of mechanical engineers who design cars should have some experience servicing vehicles.
      • btrask 10y ago
        And if the customers are imposing these ridiculous demands, then they should have some experience servicing vehicles.
  • userbinator 10y ago
    It could be a deliberate form of obfuscation/discouraging of repair, or just the common trend of making things more complex than they really need to be.
    • slavik81 10y ago
      That's possible, but lack of insight into what's easy or hard in manufacturing or construction is a pretty common problem. I saw it a number of times when working for a few different manufacturing companies, though none of them built cars. A failure to understand construction concerns also played a role in the Hyatt Regency walkway collapse. The original design was poor, but redesign to address construction difficulties accidentally weakened the walkways further. > Havens Steel Company, the contractor responsible for manufacturing the rods, objected to the original plan, since it required the whole of the rod below the fourth floor to be screw threaded in order to screw on the nuts to hold the fourth floor walkway in place. These threads would probably have been damaged and rendered unusable as the structure for the fourth floor was hoisted into position with the rods in place. Havens therefore proposed an alternate plan in which two separate sets of tie rods would be used: one connecting the fourth floor walkway to the ceiling, and the other connecting the second floor walkway to the fourth floor walkway. > This design change proved fatal. In the original design, the beams of the fourth floor walkway had to support only the weight of the fourth floor walkway, with the weight of the second floor walkway supported completely by the rods. In the revised design, however, the fourth floor beams were required to support both the fourth floor walkway and the second floor walkway hanging from it. https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse#Investigation https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse...
• bthornbury 10y ago
  With the rise of devops (docker, puppet, ansible, etc...) , and cloud VMs, how often are sysadmins different from software devs?
  • techman9 10y ago
    They're not really and they shouldn't be (although systems knowledge here becomes increasingly important). This is the point of DevOps, that operations can be treated as a software problem and managed in an automated fashion. The role of "SysAdmin" as presented in this article is fading quickly.
  • peterwwillis 10y ago
    That question doesn't make any sense at all. That's exactly the same as saying: with the rise of do it yourself home repair, how often are plumbers different from carpenters?
    • eropple 10y ago
      I dunno, I think the question makes perfect sense. I am a software developer. I very often write Chef and a metric buttload of glue code in Ruby and Bash and occasionally PowerShell. That I can also write a Rails app and a React-Native app and a Dropwizard app (all things that I've touched this week) in addition to being able to set up Kafka in a fault-tolerant and scalable way is because it's all code. Chef etc. aren't "home repair." (Well, maybe Docker, and it shows.) They're a way of expressing subject matter knowledge. Still gotta be a developer to do that expressing in a way that isn't going to kill you, either directly or because you write code that your fellows burn you at the stake for.
      • peterwwillis 10y ago
        There's reasons we have these classes of people with certain labels. Until the 18th century, buildings were designed and built by "artistan craftsmen", people who had no formal training or education in building things (or in any capacity) but said, sure, I can make you a house. Then it would fall down and kill a family of five. All around the world today, architects and engineers are required to be licensed and prove they know what they're talking about so this doesn't happen. As a result of this process, the whole profession developed and advanced the state of how we build things. We're lucky that we work in an industry where our work is most likely not going to kill people. We don't have to be licensed to make a claim of who we are or what we can do. But it would be very disingenuous for me to call myself a professional software engineer just because i've written hundreds of thousands of lines of code, and for you to call yourself a sysadmin because you've set up some software. The mark of a professional is understanding at a deep level all of the hidden detail that goes into the work, and if we hold them up to a standard, the titles will reflect that. So, how often are software developers different from sysadmins? They're always different. They're completely different practices. As a person, you can indeed have two separate successful careers and become both, but that doesn't have anything to do with devops, which is merely the practice of dev and ops collaborating together.
• bjhdw9 10y ago
  Please fuck off. Software developers should have this, should have that.. meanwhile, you can eat burgers, get fat, blame H1B and contribute to PC culture.
• kahrkunne 10y ago
  On top of the 20 years of experience we already require of junior developers?
  • bdcravens 10y ago
    Probably preferable to calling someone a developer after a 12 week bootcamp.
    • ab1243 10y ago
      Just make sure they're 22, 23 tops. Can't have any of those 30+ year olds with their early onset dementia.
• blowski 10y ago
  Sysadmin knowledge definitely helps, but so does an MBA, knowledge of writing, public speaking, design, user experience, networking. Oh, and the domain of the problem itself. The skills I require of my developers depend on the rest of the team and the project.
  • wpietri 10y ago
    The difference with this is that server-side software has direct operational consequences. A server-side developer who never deals with ops is like a chef who never tastes the food. It's in theory possible to get right, but in practice the results tend to be poor.
    • danenania 10y ago
      That's true for backend, but frontend developers don't necessarily need a lot of ops knowledge if the team has a good separation of concerns. If frontend devs have to worry much about sys admin issues, I'd say that likely points to a flaw in the way things are being done. Of course, the more someone knows, the better. Knowledge and experience in any area of technology can improve understanding of all the others. But nobody has time to learn everything, and there's way too much to learn. Time given to learning more about sys admin issues is time lost to other potentially valuable knowledge. It's a good area to learn about, and it is essential to being a strong backend developer, but through good architecture and management practices, there are still plenty of ways to make very high value contributions for a developer who hasn't spent much time focusing on sys admin.
      • eropple 10y ago
        I think your definition of "sysadmin issues" is probably a little more limited than mine or 'wpietri's. I've had frontend developers insist that they could bake configuration settings into their webpacked artifact...which needed to be deployed into multiple environments because of course we weren't rebuilding something that had already been okayed in QA when we wanted to send it to prod. You might say that's not a "sysadmin issue," but I have seen it happen three or four times now and in each case it was the "sysadmin" (read: devops engineer) who caught the problem and explained it to the offenders in question. (Maybe it's a "build engineer issue"...but at most companies I've seen, he or she is probably the "sysadmin", too.)
• WhitneyLand 10y ago
  No. Developers do not necessarily need any experience as a sysadmin and I would advise against moving productive developers into this role for the purpose of misguided training. The argument about understanding how code scales beyond one computer is fallacious. In fact it's quite easy to learn and practice all kinds of distributed high scale architectures without getting up from a desk. I will agree it's important to understand the people and work connected to your job. It's probably a good idea to eat lunch with the sysadmin, or QA, or UX person. Would also agree its important to have respect, empathy, and collaboration others, but it doesn't mean you need to switch jobs.
  • eropple 10y ago
    I strongly disagree with this, from experience. There is no difference between a "developer" and a "sysadmin" in a healthy shop in 2016. It's all the same job. Until you actually put something into practice in a hostile environment you have no idea what rakes are going to be in that yard. I would be a bad developer if I did not understand how the systems my code runs on worked. I would be a bad sysadmin if I couldn't marshal my systems through code and fully and completely understand the software that would be run on them. (As it happens, I'm not too shabby at either, and it's additive.) The mindset that you are describing is why that "devops engineer" that gets hired so very often ends up being burnt at both ends being the savior of the team because, unlike the developers, he or she does understand both sides of the equation and is able to solve the problems that the incurious developer cannot. I would strongly urge that you reconsider. And, for your own sake, please excise the word "fallacious" from your vocabulary. It will help you, I promise.
    • greglindahl 10y ago
      Thank you for re-wording the 3rd paragraph before I could complain about it :-)
      • eropple 10y ago
        I maintain that the sentiment was not wrong. ...also, man, don't tell me you're on Hacker News while you're camping...
• rsingla 10y ago
  While I agree with the general sentiment, the examples feel fairly contrived. What I think is more practical is understanding when certain problems fall outside of your own scope or capabilities and when to engage someone else for their advice.
• pjmorris 10y ago
  Should sysadmins have software development experience (e.g. DevOps)? For what values should 'X have Y experience?' Should we go as far as... "A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects." — Robert Heinlein, Time Enough for Love
  • deleted 10y ago
    [deleted]
  • inopinatus 10y ago
    I'm with the Heinlein sentiment. Practically everyone doing one will benefit from a stint as the other. I've had a foot in both graves my entire career and it's an amazingly useful skill group. How you get there is up to you. My own path was freakishly meandering. Don't be afraid to get out of your depth, but try to have a mentor around to stop you drowning. And remember that DevOps is a mindset, not a job. If someone tries to setup a "devops team", run away.
    • Aloha 10y ago
      I'm in much the same boat, my path to get where I am, was meandering and long. I've worked with developers who fully understand the systems they are deploying on, and developers who develop for an abstraction of services rather than a real world environment - I tend to prefer the former to the later because the deployment process is much less taxing, even though the later tend to have better luck moving their software to another platform in the future. But I'll echo you, every time I've learned something hard, it's because I got way out of my depth and had to learn how to swim all over again.
    • eropple 10y ago
      > How you get there is up to you. My own path was freakishly meandering. Same here, though I was aided by being able to be really confident about stuff when people came calling with the checkbook and a quick study once I'd landed a gig. I learned to do what we would now call "sysadmin stuff" (manual administration of hardware) as a kid because I broke my Linux machines a lot. Then I went into the web development gristmill for a while. Ended up leading a multi-platform mobile team with zero mobile experience because "you're a good developer, you'll pick it up" (I did); I literally went into a devops role knowing no Ruby (to say nothing of Chef) under pretty much the same rationale. "Fake it till you make it" is real, but then you gotta make it. ;)
• xenadu02 10y ago
  I don't think "should have" is the right statement; more like having at least one person on the team with sysadmin experience is extremely helpful. I know it has been for me. Of course in HS/college I ran a website that was a frequent target of hate in the late 90s/early 2000s and it taught me about XSS and CSRF before people invented fancy terms to describe them. It also taught me about HTML/JS escaping, DoS/DDoS, SQL injection, and how all your defenses are useless if someone social-engineers their way into root and nukes everything. I have the assumption that everything is compromised and user input is toxic waste burned into my subconscious.
• icameron 10y ago
  Sysadmin or devops is responsible for uptime directly, being ultimately responsible for the services to remain performant. It's a mindset and temperament that may help developers see solutions or foresee pitfalls while designing and implementing code. Zero-downtime attitude. Making and testing deployments, upgrades and migrations to be least impactful as a core function is probably the biggest effect of putting a developer in a sysadmins shoes.
• curun1r 10y ago
  We took it one step further on my team. The best way to ensure that the pager(duty) doesn't go off at 2am in the morning is to put developers in the first on-call group. Not only did it work, but the developers got a much more nuanced view of how systems operate. When we first started, I'd see developers using ping to determine whether a nameservice entry was correct. After a few months of handling almost all of our own ops, developers knew how each piece of the puzzle worked and how it all fit together rather than the hazy, largely abstracted view they had before. But we learned that we needed it to go the opposite direction too. We had ops people who, when given a corporate-wide mandate to apply a security patch or some such task, would log into every machine and apply the patch, despite the fact that we'd been practicing immutable infrastructure with zero-downtime deployments. We hadn't given them the necessary exposure to the dev side to understand that you had to apply those fixes to a base image and trigger a redeploy. There was a lot of finger pointing a couple of weeks later after it was discovered that the fixes were overwritten by an application deploy.
  • adrianN 10y ago
    I hope you increased their salaries together with the new 2am pager duty.
• gbuk2013 10y ago
  Software developers should also have customer-facing support experience (supporting other people's code) so that they realise, the hard way, the importance of having extensive logging and debugging capabilities accessible to those unfamiliar with the codebase.
• jedberg 10y ago
  Having someone who is cross trained is always better than not, but they will be harder to find and more expensive. A frontend UI person who can write their own backends is great. A backend developer who knows javascript and can build their own frontend is great. A coder who manages their own cluster is great too, as is a sysadmin who who can write code. And a developer who can do customer service, and therefore can fix a problem for every customer at once through modification of the application, is better than just someone who can answer the phone and make the customer happy. All this is to say that someone cross trained will always add more value and will also cost a commensurate amount.
  • eropple 10y ago
    > All this is to say that someone cross trained will always add more value and will also cost a commensurate amount. Man...I wish the latter was the case. I ended up going into consulting precisely because of the way that salaried employees are valued in tech right now. I literally am all of those things, I am comfortable and have delivered at a high level mobile, web, backend, and infra projects--but the "market" loves valuing those roles individually, and not the synergistic capability of being able to do all of them. Consulting is fun, but finding a place where those talents actually are valued would be rad. (Anybody out there: have an opening for somebody who can deliver value at literally every level of your engineering organization while always being game to help bring your other developers up? Email's in my profile. ;) )
    • jedberg 10y ago
      If you claim to be an expert at more than two, then chances are you're either 1) not really that good at any of them or 2) would make a great founder! If you're good in multiple areas the only way to really get paid what you're "worth" is to start a company and use your skills to create an amazing product. Because you're right: the market doesn't value a true polygot.
      • eropple 10y ago
        I don't claim to be an expert at anything so wide as a full category of software. Hell, I don't even claim to be an expert at a language. (I learned yesterday about Ruby flip-flops, ferchrissake.) Merely (well, "merely") that I can deliver a solid product at any of those levels. =) As it happens that's the other side benefit of consulting--being able to build a product with downtime. And I am! But it's still a lottery, and risk mitigation is a thing. So I like hailing for interesting offers to come my way.
• nokya 10y ago
  I'd say there is too much effort in reasoning on the wrong problem. What worries me the most is the 'why': why do (too) many software developers don't know about sysadmin? I have been involved as a consultant in large software projects in the last two years and a vast majority of money lost in delays and bugs was caused by devs not understanding: 1) the difference between virtual memory and physical memory 2) the difference between costs of data storage per storage medium 3) the concepts of network round-trips 4) and hardware bandwidths 5) how to install and configure a web server on a workstation 6) how DNS works 7) how AD authentication works 8) what ORM frameworks do 9) how to write a raw database query (not necessarily sql) 10) the difference between navigating through database records on a database server vs. an application server vs. a client, 11) HOW TO INSTALL THEIR OWN WORKSTATION AND TROUBLESHOOT IT!!! N) etc. and those are just the topics that I can immediately remember. As I see it, it's not about "they should". For me it's about understanding how many devs deal with such a level of ignorance on the systems they interact with, on a daily basis. This situation hurt my feelings everytime it happened and I struggled to accept it. I am not a sysadmin nor a developer but my daily work is insanely improved by my (even basic) understanding of how my workstation works and how to manage it.
  • Aloha 10y ago
    I've worked with all kinds - from windows devs who can't figure out how to install visual studio - to people who understand Windows, Linux, and macOS - as well as basic system administration for each platform. The people who are most successful at rapidly developing good high quality software are more in the later group. Would you trust a RF engineer who couldn't troubleshoot his own radio designs? why would you troubleshoot a software engineer who can't troubleshoot his own software as deployed in a real world environment?
    • 0xC0DECAFE2020 10y ago
      I recently questioned someone about this very subject. They wanted to hire a "CSS expert" because the "developer" didn't have a grasp of css after having developed the project in JS/HTML. I was so confused as to how that's possible.
• cs02rm0 10y ago
  This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin. Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue. And I've never thrown code over a wall without having tested it in a representative environment. The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. And then try and argue the toss about whether the CI server is cost effective and every few months keep switching it down despite the CTO saying it needs to be left alone. When a sysadmin sees an issue in "their" environment that they understand there's a tendency for some of them to just see that issue as the only thing the developer has had to deal with that month. In all likelihood, in a productive company, it's the most trivial issue the developer has had to resolve that day. Often this stuff goes more smoothly where the developers (I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going) manage production and there aren't people with separate job titles and the resulting friction between them. Sorry. There must be great sysadmins out there struggling with terrible developers, I'm sure of it. I just haven't seen it.
  • gedrap 10y ago
    >>> The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain?
    • cs02rm0 10y ago
      Everyone was told to see if they can find cost savings. This just wasn't one though in the bigger picture and even after being told it wasn't one, even by their line manager and separately the CTO, they pursued it over and over. I don't believe anyone else was directly involved further up the food chain on their side. All it needed was for the question to be asked on the company's internal board and listen to the answer. Even trying it once or twice and I'd probably have forgotten about it in short order. This went on for a couple of years though!
• seanwilson 10y ago
  I think everyone would benefit if they learned just a small amount about what their team members have to deal with. For example, project managers should learn a little about what programming involves while managing programmers and graphics+UX designers should learn a little about coding to make their designs more practical (i.e. identifying cases where a slightly better design requires a massive amount more coding effort and probably isn't worth it).
• jordache 10y ago
  Just like how you typically distinguish front and back end developers, went wouldn't you distinguish dev from devops? Sure, it helps if an individual is not entirely myopic, and would be great if that person can wear multiple hats with skillset spanning all those disciplines. However those different labels exist primarily they align to unique mindsets and solutions to a problem in a mutually exclusive manner. Do you want a jack of all, but master of non? Seriously, how many individual out there do you think have the capacity to develop their career so they have deep capabilities for all of those disciplines?
• 0xC0DECAFE2020 10y ago
  I have been at this since the early nineties and can't comprehend that someone can be a developer and not have any experience with How to set up and administer the environment. I guess it's a normal thing now?
• PaulRobinson 10y ago
  I worked for an ISP in the late 90s/early 2000s for about a year before my final year at Uni that when I started had 2,000 customers and by the time I left we'd got to 750k customers. It was the best training I could have for writing software for the Internet. Just a couple of months ago we noticed one of our core apps was not scaling no matter how many docker containers we had spun up in Mesos. Spent a week breaking it apart using that experience and being able to make changes directly to the code base and being able to talk to devops in a language they understood, and in under 5 days we managed to identify and fix 7-8 different issues. I would value a dev with sysadmin experience _far_ higher than one without in a tech business with a headcount under 500: it's going to lead to fewer problems and issues in the short and medium term.
• teekert 10y ago
  I think many professions should have sysadmin knowledge. In my work (biology) there is a lot of unnecessary clicking, not only in Excel (where Python/R would be much more efficient) but also in web interfaces to transfer data daily. The latter is often completely automatable using cron and rsync over ssh. Many repetitive actions are a shell script or command line pipe away. But people don't know.
  • mmirate 10y ago
    > automatable using cron and rsync over ssh ... shell script or command line pipe ... Where I come from, that's called "knowing how to use a computer". Or at least it was, until Windows et al arrived. (Interesting how the user-interface technology advance called the GUI, decreases the efficiency of the human/computer interaction.) Except ... for those who still regularly use a Unix-like OS, that knowledge still is quite basic and, thus, surprisingly difficult to trumpet on a resume.
• golergka 10y ago
  Just a small nitpick with the article: it probably meant "backend software developers". Because a graphics developer who mostly writes shader code would surely benefit more from artist experience.
  • TillE 10y ago
    It's incredible how apparently 99% of HN are web developers who don't even bother considering that other things exist. It's not at all representative of the larger industry.
    • arca_vorago 10y ago
      Very much this. I don't pretend to be a dev, but I listen and work with them to accomplish business goals. Sysadmin work is very much more business management oriented in the real world, but to see so many Devs say "I know how to admin my software in prod therefore I am sysadmin too" just makes me realize just how little the Hn/sv web startup dev knows about real world business systems administration for established business. Once again, like I said in my other comment, even in web startup land, this is a failure of management, in particular CIOs and CTOs, and the management that selects them. The problem is that companies can usually manage to survive for quite a while ignoring these problems. There is little short-term feedback mechanism for duct taped systems... Until that day when shit starts dying and no one can fix it. Or a cryptovariant hits the file server. Etc.
• ultrasaurus 10y ago
  One thing I like about DevOps is never having to hear "it works on my machine!"
  • yoz-y 10y ago
    I always take the "it works on my machine" as a synonym for "you need to explain your problem better". In my experience most problems that "work for me" are caused by somebody installing an unstable build and patching it with random crap they have thought of.
• ultrasaurus 10y ago
  One thing I like about DevOps is never having to hear "it works on my machine!"
• mrmrcoleman 10y ago
  Shameless plug perhaps, but this is free and will help here: https://www.cncf.io/event/webinar-cloudnativenetworking https://www.cncf.io/event/webinar-cloudnativenetworking
• A_Crazy_Idea 10y ago
  Managers should have management experience and less nepotism experience. One step at a time, guys.
• imjustsaying 10y ago
  Maybe it's only because I've only ever worked with myself, but it's hard to imagine a software developer who isn't also a sysadmin by necessity. How do you debug things otherwise? In what city or company is the barrier to entry for software developers so low? I just can't imagine paying above median income for a software developer who can't install, for an example given in this thread, Visual Studio.
• kabdib 10y ago
  I used to work on a product where the three major teams were Client, Server and Ops. Tens of millions of customers used our stuff. Client and Server folks were on separate floors of the same building. We didn't interact much at all, except by trading bugs back and forth. The management chains met at a VP. Three or four times a year we tried to coordinate a release, and it always took at least a month. Getting a feature out the door might take a year, with all the paperwork and pipelining of release schedules. Ops was in another building. The only things that both Client and Server teams were sure of was (a) they did a bunch of customization of our stuff so that it would work, and (b) they hated us. Support was in another state. We were not allowed to talk to customers. Maybe once a year Support would fly in to talk to the teams about pain points. I think we did an okay job addressing these, but it took a long time, and customers suffered a lot. I won't talk about the disaster that ensued when Scrum was thrust upon us, or the splinter projects that spun off to try to fix things (but wound up being lots worse). You have to be close to the customer or you won't know if you're succeeding, or even on the same page. You have to know what your software is doing in production or you're just sitting in an ivory tower pontificating about angels-on-the-head-of-a-pin nonsense. You have to spend time in the trenches measuring and fixing stuff or you're hatching an unmanageable disaster. The good news is that most of this is actually kind of fun. The bad news is that, when managed badly, this can turn into a horrible and soulless grind of pager duty and making legacy code even more legacy to fix wee-hours downtime. I still get a rush when I get feedback from a real, live customer, and I think that isolating your teams from customers is one of the worst things you can do to a team and to a product. Getting teams to work on ops and support aren't bad ways to improve this.
• necklace 10y ago
  Ironically I can't see the images at the bottom of that page.
• pmontra 10y ago
  The post has a valid point even if the example is more about system architecture than system administration. In my experience a developer doing one year in an Operations department gains some insight in what is going to be important in the longest phase of the software life cycle, that is when it's exposed to customers and the company has to react quickly to problems. Proper logging, anything that can help pinpointing the cause of problems and even hot patch them at 3 AM. The usual lean startup might have little thoughts for that (is it even going to have customers?) but established businesses do. If it's 6 months to develop and it runs for 10 years (with maintenance and new features), which phase is more important to a company, development or production? I'm a developer which did a couple of years of operations and I've little doubts about it. It's where the company makes the money from.
• pantulis 10y ago
  This seems obvious to me. You'll never see your code again with the same attitude after being on call.
• b212 10y ago
  I'm a front-end developer with very little knowledge of sysadmin stuff, but I can't agree more, the best developers I've ever seen had massive sysadmin experience. Can you point some good resources about sysadmin I could use to improve my knowledge on the subject? Note I'm a noob when it comes to all this server-side magic.
  • swozey 10y ago
    When I was getting into ops 10 years ago the book "The Practice of System and Network Administration" by Thomas Limoncelli was a great overview (very high level and not specific) that we all read. The Phoenix Project by Gene Kim is kind of a more modern uptake on it, but both can be read easily by people outside of the field and I doubt the first one will ever not be relevant. Both are more about establishing proper ops methodologies. If you're looking for something more low level, like a Programmers guide to sysadmin I can't think of anything off the top of my head.
    • schnable 10y ago
      Limoncelli's newer book _The Practice of Cloud System Administration_ is awesome. It's really more about how to build distributed systems to performant and operational.
      • swozey 10y ago
        Oh wow, I need to check that out. Wasn't aware he made one cloud relative.
      • eropple 10y ago
        I started writing a book about cloud system administration before I read this one, FWIW. It's worth your time.
• AliAdams 10y ago
  The underpinning logic seems false. To extend the analogy of the house on a slope, if you don't tell the developer you need a house designed to be built on a slope, that is a problem with the spec provided to the developer. Although it can at times be more efficient for a dev to have sysadmin experience, making blanket statements requiring all developers to come with such a background is just sensationalism.
  • sah2ed 10y ago
    Wholeheartedly agree. I think the lesson the author is really trying to convey is the importance of learning about distributed computing [1] concepts and not necessarily system administration. [1] https://en.wikipedia.org/wiki/Distributed_computing https://en.wikipedia.org/wiki/Distributed_computing
• vinceguidry 10y ago
  I think the problem here is that decent admin jobs are getting harder to come by these days. Used to be any reasonable organization needed an admin just to keep systems straight, now they all rely on less-skilled help desk types and make it up with outsourcing companies, which in turn are shedding personnel and relying on big vendor contracts rather than individual skills. I predict sysadmin culture and skills will be the next Fortran.
• madihaawan 10y ago
  As software developers, this should notice and kept in mind that all these experiences are very much important. Thanks for sharing an important article.
• fnj 10y ago
  It's hard for me to "get into" developers who don't know how the system works. I don't mean they have to be full-fledged sysadmins, but it just seems natural to me that they should have a decent understanding of the electronics, how to assemble the hardware, and what's involved in tuning it and keeping it all running. It may have something to do with my education being EE, but that's not the heart of it. I never practiced EE. I taught myself (more or less in order) building circuits, programming, building PCs from pieces, and elementary system administration. I know there are lots who never "get" all three, but the best I knew did.
• kweinber 10y ago
  A great reason to have cross-experience at the management level is to understand the different motivations of the two roles: * devs are incented (paid) to introduce changes to the system...mostly in the form of new features. * sysadmins are incented to stabilize the system and make it scale. In organizations with uncoordinated or siloed management, this leads to infighting... sysadmins try to stop "troublesome" devs from making changes at all and actively slow down changes in the name of stability and developers at the other extreme try to get changes in without caring about stability and blame sysadmins for that. Great management with experience in both will find ways to incent the teams to introduce regular change while maintaining stability.
• nier 10y ago
  This needs to be mandatory for developers working on printer software.
• jcadam 10y ago
  You do not know pain until you've had to deal with government sysadmins as a contracted developer. And the govt has a strong preference for bringing sysadmins into the civil service rather than contracting that work out (like they used to) these days. Which means the ones with actual, marketable skills aren't working for the government. Spending months waiting for a sysadmin to perform a task you know takes about 3 minutes is great fun.
• eviln1 10y ago
  Hi, I'm a Sysadmin, and I've been a grumpy one through a larger part of my 15 years experience. My main issue was that Developers were acting like Users: they don't care about what you have to deal with, they want things to 'just work'. In return, I've treated them like children, in some instances yelled at them when they did dumb stuff. I've tried to educate them when possible, and was angry when the education didn't stick. At the time i was the 'King of the Hill' type of sysadmin - natural leader of a very small and tight team, kind of irreplacable, and with enough years in the company behind me to consider myself as a demi-god. When I switched companies, I came across better developers. Some had decent sysadmin skills, but the main difference was that they actually took interest in how things worked past the 'git push', and when I asked / required them to make some changes that would make my life easier, they listened, discussed and adopted when appropriate. With those same guys, I took interest in what they were doing, what their actual job was and came up with ideas that would make things easyier and run smoothly on both ends. After a while I figured out that they weren't actually better developers - they were better people. (Also, I figured out that being grumpy was not the best approach and that patience, kindness and gratitude could get people to do more than snark, humiliation and flame-throwers.) I guess my point is: you don't really NEED to have sysadmin skills to be a decent developer; what you really need is to care about what sysadmins do - be curious, talk with them and trust them when they say that your brilliant idea won't work in production.
  • joadha 10y ago
    I've worked as both a dev and sys admin, and I think this is the most reasonable response so far.
  • davidgerard 10y ago
    As a sysadmin, I love the devs who think "devops" and I make a point of saying nice things about them to the CTO.
  • pyrale 10y ago
    This, so much this. You don't need to be the sheep with 5 legs every manager wants, what you need is to be accessible to collaborate with people on problems. That's an individual skill as well as a systemic one, though.
• fauigerzigerk 10y ago
  What he's talking about is systems architecture not sysadmin experiece. Of course developers have to write code for a logical model of a machine that isn't necessarily the same as the laptop they're working on. But I don't think anyone would ever dispute that. Storing pictures on a local disk when the software is supposed to run on a cluster has absolutely nothing to do with sysadmin experience.
  • jessety 10y ago
    I came here to say this. If a developer is building an app for 100K users the same way they'd build an app for 100 people, the systems architect dropped the ball- not the developer.
• cekvenich3 10y ago
  Nope. SD should not be using DB. They should use API to make REST | Microservices, ex FireBase. SD should not be using servers. They should go serverless, deploy static HTML to CDN> etc. If you are still using docker, puppet or worried about scale/security... you are living in the past. (one ex: https://github.com/struts3/what-demos https://github.com/struts3/what-demos )
  • petra 10y ago
    You're right.But like any big change it will take time.Say 5-10 years to most new stuff being serverless.
• andrewclunn 10y ago
  Yes, yes. We get it, we should know all things, keep up with all the latest rends, work 80+ hours every week, and do our company's IT support on the side for free while we code their software. Or companies could compensate us for our time and hire more god damn people if they need it. You know, just saying.
• al2o3cr 10y ago
  In the case described by the author, it seems like most of the trouble would be resolved by the developer understanding the concept of "distributed file storage is hard" and coordinating with ops early - the dev doesn't necessarily need to know the details of setting that up, but they need enough understanding of the domain to realize there's work to be done on the issue. For that matter, I suspect most sysadmin / ops teams would be happier if developers could only strip "just" out of their vocabulary (and vice versa) - nothing more annoying than "could you JUST do {thing asker doesn't understand and thinks is trivial}"
• ovdoll 10y ago
  That’s interesting.
  • ovdoll 10y ago
    http://www.ovdoll.com/ http://www.ovdoll.com/ Silicone Sex Doll | Silicone Love Doll | Life Size Doll
• deleted 10y ago
  [deleted]
• tzakrajs 10y ago
  I have ten years of experience in systems and have written tools software in Python and Javascript for Netflix, Stanford University and Google (current). I have never been a software engineer officially, but I want to be. I am a PC gamer, anime lover, and a proud SJW. Any takers?
  • divanvisagie 10y ago
    What are you on about?
    • tzakrajs 10y ago
      I am interested in finding positions. Sorry if my post was ambiguous, I know it is a bit out of place here.
• tyingq 10y ago
  I would change it from "should have sysadmin experience" to "should have operating system knowledge, or ask the right questions". The example in the article isn't what I tend to see in real life. What I do see are things like: - Not knowing about various limits (number of open sockets, or listen queue depth for example), how to know you've hit one, how to deal with it. - Not handling various error situations correctly (can't open file due to permissions for example) - Security issues. ( socket listening on 0.0.0.0 when localhost would work, for example) - Making assumptions about things like "current working directory" or "certain environment variables will be already set for me" For many of these, including a sysadmin or system knowledgeable architect in the right discussions would suffice.
• ohstopitu 10y ago
  I've had to work at start ups where I was one of the initial developers (and the most experienced in the team) so I've been forced to learn a lot about Sysadmin and design (and a lot of surrounding dev tools for instance) but all in all - knowing more about sysadmin has made my dev experience change - I think a lot more about how it's deployed, how it works in prod, how to get data to improve my code and so on. I do feel all devs at one point should have some sysadmin experience - if not professional, then informally. That said, sysadmin is strongly divided into 2 types of tools - configuration and code. I love tools that can be coded (as an example - Gulp), over configuration (like Grunt) - as they generally have a lot less undocumented "gotchas" and getting started is generally a lot more simple. (I'm looking at you Webpack!!!)
• bandrami 10y ago
  As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us on the sysadmin side of the aisle capitulating and saying that developers can't be stopped from having the newest version of things, and I think that's a bad idea. 15 years ago, when a project would kick-off, as a sysadmin I'd be invited in and the developers and I would hash out what versions of each language and library involved the project would use. This worked well with Perl; once the stacks started gravitating to Ruby and Python it was a dismal failure. Why? Because those two ecosystems release like hummingbirds off of their ritalin. Take the release history for pip[1] (and I'm not calling pip out as particularly bad; I'm calling pip out as particularly average, which is the problem): in the year 2015, pip went from version 1.5.6 to 8.1.1 (!) through 24 version bumps, introducing thirteen (documented) backwards incompatibilities. Furthermore, there were more regression fixes from previous bumps than feature additions. You'll also notice that none of these releases are tagged "-rc1", etc., though the fact that regressions were fixed in a new bump the next day means they were release candidates rather than releases. Ruby is just as bad; the famous (and I've experienced this) example is that an in-depth tutorial can be obsoleted in the two weeks it takes you to work through it. Devs are chasing a moving target, and devs who haven't been sysadmins may have trouble seeing why that's a bad idea. [1]: https://pip.pypa.io/en/stable/news/ https://pip.pypa.io/en/stable/news/
  • berntb 10y ago
    Interesting. Is this because the Perl ecosystem is more mature or because of the philosophy of backwards compatibility? The "backwards compatibility" philosophy isn't so explicit for the ecosystem, mostly the language? Is the test-on-install-by-default making a big difference there?
    • bandrami 10y ago
      That's a good question. I think it's not a coincidence that CPAN predated the widespread use of distributed source control systems whereas pypi and gems blew up just as mercurial and git were unseating svn and cvs. It's a different release philosophy (remember, in the 1990s you often didn't even get to see pre-release CVS commits of open source projects; that was an innovation of OpenBSD). I also think the widespread use of VPSs rather than accounts on shared servers (again, containerization) was a factor. In the 90s and early 2000s, you usually (even in a corporate setting) had an unprivileged account on a server with a given version of apache and perl, your own cgi-bin directory, and possibly some latitude on a personal CPAN install directory. The lack of containerization meant you had to compromise between using newer software and breaking existing use cases. So I guess I think it's not so much about Python vs. Perl per se but about the technologies available when those languages became popular among developers.
• mrskeltal 10y ago
  As a developer with practically no sysadmin skills, how would I go about improving these skills?
  • FLUX-YOU 10y ago
    Small/micro AWS/Azure instances. Set up your own CI/CD pipeline. Read sysadmin books. Get comfortable with the environment shell (Powershell/bash/etc.) and write some scripts. Bonus points if you make an actual product that solves a problem and also practice all of these skills.
• kapilkale 10y ago
  I understand the point here; it can be useful to understand the abstractions on top of which you are working. But given most engineering projects are crud apps without scaling problems, and PaaS companies like Heroku totally abstract away the system adminstration piece for those projects... I certainly wouldn't recommend a new engineer spend any time learning system administration.
• adamdonahue 10y ago
  The general trend in software development appears to be toward generalization over specialization. I'm not sure this is a good thing.
• tete 10y ago
  I think the reverse is much more true, but maybe nobody argues about that anyway.
• krosaen 10y ago
  I think it's more accurate to say that good developers need to have some level of systems expertise which probably means OS & networking fundamentals + practical Linux experience. This is because you need to have some understanding of how your program will execute on (or across) machine(s), and some know-how to get your program up and running on a real machine available to users on a real network. Whether you need to be an expert in the latest best practices for managing a fleet of servers compliant with whatever regulations, that's more for the sys admin (though it never hurts to learn more, there's just so many things we developers could spend our time learning).
• whenwillitstop 10y ago
  Sysadmin is getting automated. There is not reason to know it.
• saosebastiao 10y ago
  I get the sentiment. Personally, I could say the same thing about control theory, systems theory, economics, operations research, and statistics. And I could also share dozens of anecdotes of bugs and suboptimal software that was produced as a result of this lack of experience. But at some point, you have to acknowledge that developers can't know everything. Go deep or go broad, but accept that both paths have their limitations and benefits. Maybe in some cases you'll run into a bug you created due to your lack of sysadmin experience, but don't feel bad about it...your experience has led you to other types of expertise. Fix it, learn from it, and move on.
• Diederich 10y ago
  I worked in the home office of WalMart Stores, the retailer, from the mid 90s until 2009 as a hybrid network engineer/developer. That is, my team worked in Network Engineering, but we wrote tons of code, because when there are millions of different IP addressable nodes on a centrally managed network, there will be code to manage it. (: In that era, I think WalMart handled this apparent developer/sysadmin confliact quite well. Believe it or not, even though we (the whole company) had exceptional uptime, we were also extremely agile. LOTS of code was written and rolled out across thousands of sites, multiple times a day. There were a number of keys to this, and I'll enumerate a few. 1. Most new hires (those that had minimal previous experience) to development positions had to spend their first six months working at one of the four or so major help desks. Note: they were paid their target salary, but in an hourly fashion. 2. The various operations teams had complete and final say so over what went out, and how incidents were handled. A couple of the big operational areas were Unix Operations, Network Operations, Windows Operations and Mainframe Operations. I called them 'teams' but each had a number of teams, and their own help desk. 3. Any time there was an impacting problem associated with a program developed by team X, a war room was called by the relevant operational area. A member of team X (a developer) had to stay in that war room (switching off between team members if it ran very long) until the production problem was not only fixed, but also deemed unlikely to recur, except in cases where that dept of a fix would take a long time. 4. Every development team had a pager rotation, with rigorous expectations about responding to such pages. This was primarily to support the previous point. 5. Because of the enormous operational scale, all of the major operational areas had dedicated teams focused on automation. My team was that team for the networking area. Furthermore, most of the rest of the operational folk read/wrote code to some extent. In short, incentives were aligned. Teams that wrote externally facing code felt pain if the stuff they wrote and released caused problems. Operational folks wrote/managed/interacted with tons of their own code in order to manage the enormous infrastructure. Also, ops folk were far more willing to let things move with velocity knowing that the people who actually wrote the code would be required to support it, globally, any time of the day or night. Another, perhaps even more important reason we were so successful during those years (and the years before) was a strong and vibrant esprit de corps. The entirety of Information Systems was, at the time, around 2000 people, and we were facilitating double digit year over year growth over a 150 billion dollar company. We had over 5000 remote sites in 15+ countries, with a diversity of software and infrastructure that was honestly pretty astounding. Each of those sites had quite a surprising amount of infrastructure. We worked hard, and we produced huge velocity with fantastic uptime. For example, the network achieved six 9s of availability a couple of quarters. In end, while things were sometimes contentious, we trusted each other, and only minority of teams were forced to work bad hours for any length of time.
• mmartinson 10y ago
  Careful with that should, or you might take someone's eye out.
• partycoder 10y ago
  As a developer the way we resolve this is by including the sysops department as a stakeholder that produces requirements for us to negotiate and implement.
• youdontknowtho 10y ago
  sysadmin really means "technical debt manager" most of the time these days. I really spend the majority of my time trying to slot something into place in a way that no one will notice that doesn't disturb existing stuff...often that means duplicating bad behavior because its become a dependency.
• timkofu 10y ago
  Agreed.
• protomyth 10y ago
  Actually, I would rather language and tool designers have a go at a true System Admin job. There is a reason PHP gets installed by default and Ruby on Rails does not. Frankly, at the end of the day, most businesses don't want something that was "working" breaking. All the conflict comes from that one desire. There is a reason System Admins have to do the "Patch Tuesday" dance and despise any software that just updates by itself. Getting yelled at because Chrome updated to a version not supported by your ISV or your local developers is an amazingly fun experience. Yep, System Admins have to be the company nanny. It sucks, but the apps need to keep working.
  • ihattendorf 10y ago
    True for the most part, however PHP is a language while Ruby on Rails is a framework.
    • protomyth 10y ago
      PHP comes with all the stuff to make a website. Ruby needs a framework, and Ruby on Rails is the most popular. I think you proved my point for me.
• mybrid 10y ago
  UC Berkeley's approach to teaching computer science I believe is the correct approach. 1. In CS150 one has to build a hardware computer. 2. In another class one has to build an OS. 3. In another class one has to build a Database. 4. In another class one has to build a Network. Education is key. Understanding the TLB inside the CPU gives one an appreciation for context switching between the only 2 users the CPU has hardwired into it: kernel and user. Being a sysadmin is insufficient experience as education. Every developer should know how to build a hardware computer from the ground up. Computer science at Berkeley doesn't have a "Java" class. The language one programs in changes depending on the Professor and topic. That way one doesn't get too attached to a language. Most developers do not understand the hardware implementation of a thread, or now-a-days a docker. As a result both get used wildly incorrectly. Threads are useful blocking on IO to keep the CPU from starving. However, switching threads simply to swap CPU bound jobs is inefficient. Just adding a thread doesn't guarantee equal service to users just like adding processes does not. Developers are using dockers today who have no clue why they are doing so...it's just that everyone is doing it and they do not want to appear stupid. Experience as a sysadmin is no substitution for education.
• nanodano 10y ago
  Funny, I always see this from the other side. I always find myself thinking "I wish sysadmins had more programming background."
• Steeeve 10y ago
  The unspoken elephant in this thread is that sysadmins and developers have broken relationships surrounded by organizational dysfunction. Personally, I feel that system administration experience of some sort is essential to development skills, and development experience of some sort is essential to system administration. Communication and leadership skills are essential to both. It _used_ to be OK if you were the jerk at the office if you were good enough at your job (sysadmin or developer). That's no longer the case because we've progressed well past the point where a single developer or sysadmin can be enough of an asset to look past their ability to work well with others. Operations needs stability. Developers need to hit moving targets. Sysadmins need to keep things manageable, compatible, and secure. These aren't incompatible goals, and usually when you get the techies talking they are sympathetic towards each other's challenges and helpful to one another. With as many project managers, engineering managers, manager managers, directors, and executives in the mix I wonder if the problems in all these environments isn't more of a historic issue with leadership failings.
• EliRivers 10y ago
  Yes, sysadmin experience will give me vital insight into my work writing embedded code for dishwashers. Snark aside, if the post was headed "Software developers need to understand the environment where their code will be running or they may not build it properly", which is the point being badly made by conflating "environment" with "some kind of commodity x86 with a standard OS", it would be a lot more applicable.
• just2n 10y ago
  Anecdotal as it is, I started my career working as a sysadmin while studying CS. When I was younger I was quite interested in security (e.g. I followed defcon and CVE lists and read a lot of manuals and source code). I built server software, broke applications, and did a lot of reverse engineering. In that time I was forced effectively to learn about how the OS worked and how to utilize it, both Linux and Windows, from C APIs to scripting, package management, and everything else needed to effectively work in those environments, primarily around reverse engineering and vulnerability research. That experience naturally lead to me becoming a sysadmin during my study at university. It was a fairly straightforward application of what I already had learned with a much larger scale of management. The primary thing I gained out of it was a drive to automate everything. When I started that job most of the sysadmin work was manual, but a few of us spent a huge amount of time focusing heavily on automation and when I left most of the work was automated and we were just doing meaningful firefighting and supporting development. As an engineer, the main benefits have been understanding how my software is going to run in an actual software/hardware stack, easily jumping into a production environment and debugging complicated issues, being able to quickly have my OS do what I want, and that drive to automate everything. A lot of that informs how I build software and in general it feels like it makes me a lot more productive.
  • AnonymousPlanet 10y ago
    I have a very similar story as a background. To add to what you said: while maintaining and debugging software installs, I learned a lot about how and when things break. Especially, how important it is to keep things simple. This turned out to be invaluable when I began working as a software developer after graduating.
• joshwcomeau 10y ago
  I think that there's good advice in this article, but a couple counter-points: - Plenty of software developers work for small- to mid-size companies where the Node or Rails server running on your machine is nearly identical to the one running in production. - Plenty of software developers work exclusively on the front-end, which means that the computer your code runs on in production is very very similar to the one you develop on (although then you have the added concern of testing on lower-end mobile devices, other browsers, etc)
• vickychijwani 10y ago
  I'm moving from dev to devops for a while, because of similar reasons. Excited to learn all these new things and peer beneath all the tools and abstractions I rely on! I have no doubt in my mind that it will make me a better developer - most abstractions have a tendency to leak, eventually.
• raarts 10y ago
  Wherever I went, I have seen the war between developers and sysadmins being fought. In enterprise the sysadmins have the upper hand, in frontend, mobile, SMB the developers. Developers don't have the burden of maintaining and upgrading multiple applications on the same server, with conflicting dependencies. 24/7. Sysadmins don't feel the pressure by end-users to deliver new features yesterday. At the moment developers are winning because Docker. But when they grab that responsibility they will be the ones called when their 50000 containers are being hacked because vulnerabilities. And they will be under fire for the system being down and that costing the company a lot of money, so everybody breathing down their necks.
• dpaluy 10y ago
  Software developers should have experience in the things they need to get their job done. This is not a revelation.