14 ms·

Why One Programmer Doesn’t Do DevOps Anymore

269 points by jjmojojjmojo 12y ago

hydrogen18 12y ago
Once again, we have the stylish new white background with barely legible text design pattern showing on HN.
- otaku888 12y ago
  It's just a common Wordpress theme: https://wordpress.com/themes/sela/ https://wordpress.com/themes/sela/
- afandian 12y ago
  Agreed, the font is very light weight. The way it renders on my screen (Macbook Air) gives me a headache.
  - agarcia-deniz 12y ago
    I'm glad it's not just me. looks like he changed it to something else
- delta1 12y ago
  What does HN have to do with the theme of this guy's blog? If you're struggling to read it, try https://www.readability.com/shorten https://www.readability.com/shorten
- nkozyra 12y ago
  The difference between this font color and #000 is actually really really minor at a larger scale. The issue tends to be exacerbated by smaller print.
- ipedrazas 12y ago
  I don't bother reading it from the source, push it to pocket, and off I go with their stylish style :)
- jjmojojjmojo 12y ago
  I switched back to the "2010" theme, hopefully people will have a better reading experience now. Thanks for letting me know (letting me know in private without the snark would have been better, but beggars can't be choosers, so I'll take what I can get).
mugsie 12y ago
DevOps as it has been implemented in tons of company is bs. Its like the "Agile" push that companies thought they could get by buying a tool, and changing nothing else (release schedules, feature based release dates etc). My favorite way of looking at it is simple - replace the word DevOps with empathy. That is the core of the idea, so lets try it - "Empathy Team", "Empathy Person", "Empathy Engineer". DevOps is about dev and ops teams communicating and having empathy for the requirements of the other team.
- smcl 12y ago
  Obligatory quote echoing your final sentence, from the seminal DevOps thought leader Jon Hendren: "DevOps is mostly just a way of reorganizing all your nerds and making them talk to each other more"
- VLM 12y ago
  "replace the word DevOps with empathy" This scheme also works with diversity, quality, best practices, and change management.
  - mugsie 12y ago
    Yup - it does. I just seem to prefer the empathy quote for some reason :)
    - mmatants 12y ago
      Empathy is the actual mechanism for how those other things (quality, etc) are achieved. Diversity helps empathy (represent the user and relate to them better).
      - VLM 12y ago
        Another way to extend the analogy is its a standard management technique when provided with something you don't want to do, set up an isolated team with no deliverables, no authority, and no direct reports. Ta Da we got us "quality" or "diversity" or "Dev Ops" and we didn't have to change anything.
chomp 12y ago
How does one reconcile DevOps with the Sarbanes-Oxley act? http://en.wikipedia.org/wiki/Separation_of_duties http://en.wikipedia.org/wiki/Separation_of_duties
- mugsie 12y ago
  With difficulty. For example - you can still have an Ops team, and they have the required access. The developers access to production is done on a case by case basis, with some sort of tracking of who logged in. and why. Also, doing full pre - merge code review helps, as it is yet another set of eyes on the code that will be pushed to production... In short - its not a "one solution fits all", and it is something that needs to be discussed with you SOX auditors, to find a working solution for your company
  - chomp 12y ago
    Thanks for the response! We keep a distinct separation of duties, but we try to keep both groups in constant contact to minimize animosity in both groups. We pick and choose some DevOps ideals to make it fit, was just curious about other people's experience.
- snorkel 12y ago
  SoD simply means your systems or processes require at least two different people to make any changes. If your entire tech team is DevOps then it simply means the dev who tests and deploys a given code change is different from the dev who wrote the code. These different roles can be per task, it doesn't have to be full-time persons assigned to full-time roles.
- emodendroket 12y ago
  It seems, from reading the article, like the need would be satisfied by having two dev-ops engineers manage something, rather than one developer and one ops guy.
- signal 12y ago
  DevOps doesn't mean people doing multiple jobs. You can collaborate with separated roles and be productive. Furthermore you can build abstraction to simply remove manual, unrecorded interaction with systems to satisfy auditing and regulation.
300bps 12y ago
It seems OP is using the word DevOps to describe what is traditionally referred to as "special projects". Essentially, the non-technical CIO hears a new buzzword and hires a team to implement it in an ivory tower without fully partnering with existing team members. From OO to SOA to Agile to DevOps and everything in between these initiatives almost always fail. That of course doesn't mean that OO is bad, or that DevOps is bad. It is just that the "special project" methodology is terrible.
KaiserPro 12y ago
So this is the thing. There are three things that really go against a true "devop" system: 1) cost. Ops are cheaper than devops, sometimes twice as much. Yes yes, flexibility. But thats a misnomer. If your dev teams are doing donkey sysadmin work, because they don't have the experience, or are up all hours fixing things, you're going to endup with a high staff turn over. 2) division of labour As a company grows, they can employ more and more specialist staff. This gives them a competative edge as they are able to make nuanced decisions based on skill, or experience. A decent storage admin will be able to build a storage system that will scale relaibly, cheaply and with less downtime than someone who is not a specialist. 3) burn out. A human is not designed to multitask 14 hours a day. Coding, testing, supporting, infrastructure procuring, for anything other than the smallest company is too much for a single role. There are great things that come from a devop "culture"; greater personal autonomy. However like all things, its not one size fits all. There is a great deal of arrogance where people assume that a sysadmin never automate anything. or that they can't program. They can and do, you know why? because they want to carry on searching ebay, scamming $someone boasting about network speed, or just plain getting shitfaced.
- RankingMember 12y ago
  I was with you until "because they want to carry on searching ebay, scamming $someone boasting about network speed, or just plain getting shitfaced.", but I'm guessing you're just being cheeky.
  - KaiserPro 12y ago
    I was bending the truth a tinybit. The idea is that a good sysadmin is an idle sysadmin. (having offloaded tech support to the juniors) unless they are playing with new stuff.
  - gknoy 12y ago
    I was thinking it was a BOFH ("Bastard Operator From Hell") reference. [0] The BOFH is a fictional sysadmin who is worse than any you have ever had, in entertaining ways. He basically does exactly the things the OP mentioned, while dealing with clueless management, undeliable vendors, and a Sith Apprentice^W^WYoung Coworker who learned well, and is constantly trying to stab him in the back. He's like a mix between Catbert and Wally. ;) 0: http://www.theregister.co.uk/2015/02/27/bofh_2015_episode_2/ http://www.theregister.co.uk/2015/02/27/bofh_2015_episode_2/
legolin 12y ago
It seems that the main problem here was not DevOps but rather the culture in which he was asked to do his work. Creating a devops team whether "by fiat or by force" doesn't sound like a recipe for a good experience. My experience is different: DevOps saves me many headaches by letting me easily deploy things for small clients. A larger company I work with uses Chef to manage their 40+ servers; the IT department doesn't touch these servers. Rather, 3 or 4 developers share in the work of writing cookbooks, adding instrumentation, deploying code and maintaining the servers. It would be handy to have a full-time guy as well (OP mentioned the amount of work needed to keep up with just your basic security patches) but we -- as developers -- love the access we have to our entire stack.
- jjmojojjmojo 12y ago
  I don't want to take away from your point, but you touched on security - I mentioned it because it was a constant pain point for both myself and for our teams. Like operations, it's something you can be good at on a superficial level, and do OK for most things, but really getting it and being good at it takes experience and time. I don't think it came through in my post, but having a security expert (or any specialist) in the mix can be a huge help in making the product better - I know this from personal experience - you just have to make sure they're integrated, challenged, and respected in the group.
dschiptsov 12y ago
It seems like a mere attempt to adapt labor-division best practices from assembly plant industry. More managers, bigger budgets, easy staff rotation - that kind of crap. Also a fresh new meme to ride on. Nothing special.
jussij 12y ago
To me DevOps is a cost cutting measure whereby an organization can save a few hundred thousand dollars by having one individual not only do the development but also handle the deployment and the testing. In theory that will eliminate the need for an additional tester and a second individual to do the change control. But those short term gains expose the organization to the massive risks that the one and only one individual will make a mistake, with no second individual available to do a double check. The reality is people make mistakes and by eliminating people from that double checking process may well save money in the short term, but in the long run it greatly adds to the of risk failure. While there are short term gains, the risks to the organization can be massive: http://www.bloomberg.com/bw/articles/2012-08-02/knight-shows-how-to-lose-440-million-in-30-minutes http://www.bloomberg.com/bw/articles/2012-08-02/knight-shows...
- mugsie 12y ago
  That is actually an example of the old style of IT. The person deploying the software had no idea how it worked on an internal level. The person who wrote the software had no idea how it was actually deployed in production. They didn't have a quick rollback in place (this is the exact reason you have quick rollback ability) The deploy was done by hand, with a user typing. That is probably the biggest risk you can have in a production system. It means that fully repeatable builds in test & pre production are impossible. I am not saying "DevOps" would fix this, but a bit of empathy / comunications between the dev team and ops team might have averted it. http://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock_trading_disruption http://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock...
- marshally 12y ago
  Well, look, I already told you. I deal with the goddamn customers so the engineers don't have to!! I have people skills!! I am good at dealing with people!!! Can't you understand that?!? WHAT THE HELL IS WRONG WITH YOU PEOPLE?!!!!!!!
SeanDav 12y ago
So many misconceptions here as to what DevOps is. I think mugsie comes closest with his use of the term "empathy". This is very very close, but I would add one more term "cooperation". DevOps, when implemented correctly can have enormous, order of magnitude increases in both productivity and quality, but it is not voodoo, despite this almost unbelievable claim. It involves empathy and cooperation and it requires this because in order to implement DevOps in a meaningful way, an organization has to vastly increase communication between "silos". It has to formalize, document and automate as many processes as possible. DevOps is NOT easy to implement and involves a cultural shift and absolute management buy-in. It is best implemented in small projects, where clear wins can be demonstrated. Use this success to bootstrap the process to bigger projects. I do agree with the OP that if you have a DevOps team, you are probably not doing DevOps. The whole point of DevOps is to get rid of silo mentality. No more us and them, but all us. Implemented correctly, everyone is DevOps.
mpdehaan2 12y ago
Having been around since the inception of this space, when things were much more tools focused, to now, when it's a lot more about "culture" (all good), I feel 95% of DevOps talks are people discussing what DevOps is. As such, it's now a metasyntactical label for a lot of things that can by good in various scenarios. I agree it is time to retire the label and instead talk about those specific things -- automation, culture, continuous improvement, etc, in nearly all instances.
bonn1 12y ago
Don't know if the concept of DevOps is bs but I know that it's unfortunate if a developer never did system administration: there are many of this "git push heroku" kind of devs who pee in their pants if they have to deal with bare metal and the production servers. It's not that they wouldn't be able too handle them, it's that they are afraid because they never did and besides, good system administration knowledge leads always to better application and system design.
- snorkel 12y ago
  I blame Java much more than heroku. Corporate drone Java developers have a much greater tendency to only write code (too much code!) and not be interested in learning anything beyond the confines of their comfy auto-completing IDE. It seems to me Java developers will howl "not my job!" the loudest whenever anyone suggests that maybe they can mange their own environments.
  - digi_owl 12y ago
    Sounds like what has been around since the mainframe days and batch computing. Or maybe one could liken it to Star Trek, with ops being the people keeping the warp core going even when faced with borgs and black holes?
  - HelloNurse 12y ago
    In the enterprise Java world, the comfy auto-completing IDE is a smart strategic choice when "managing the environment" means weeks of frustration at installing, configuring and fitting together by trial and error an application server like IBM Websphere and the application. Funny tools like Maven, Subversion and certain Eclipse plugins add extra puzzles and unpredictability for the true amateur, but the normal unforeseen complications are usually enough to forfeit any hope of good automation.
snorkel 12y ago
Unfortunately the term DevOps has been diluted in the job market. Now most DevOps job postings are actually just plain ol' sys op roles, where "development" means writing tools for the actual developers. If you want to spend at least 50% of your time writing code then avoid any job postings that mention "ops".
reitanqild 12y ago
Funny and, unfortunately, relevant, especially the last tweet: https://twitter.com/devops_borat https://twitter.com/devops_borat
passive 12y ago
It's fascinating how much this engineers experience shares with mine. We both appear to have similar approaches to working with technical systems, that has led to skillsets that are more broad than deep. We've both had interview problems because of this, and we've both ended up in DevOps. I've also had more or less the exact same experience with DevOps, though with (hopefully) a happier ending. I was on a team working for a client that desperately needed "DevOps-type" things. So I built them, and their infrastructure team ignored them. Eventually we stopped working with that client, and I was moved to a different organization, where each development team was doing it's own thing, operations-wise, and had no interest in adopting a uniform approach, so my (short) efforts there were also for naught. But then I had the opportunity to come back to the first client, as their DevOps Manager. Before I took the job, I talked to their current development organization about what their pain points were, and what actions they would be willing to take to alleviate them, and I built a strategy around that. A key part of this strategy was gradually eliminating the need for a DevOps team, as I firmly believe the best application of "DevOps" is as a joint project between Development and Operations to optimize shared processes. Once the friction is minimal, you don't need a team there anymore. So we have a developer embedded in our meetings, and have more or less full control of any infrastructure.
- Fuxy 12y ago
  I think there is a fundamental disconnect between what DevOps is and the classical business organization structure. DevOps is not a department and it will not work if it is a department DevOps is a way of doing things to achieve maximum efficiency and everyone on development and operations must embrace it for it to work. If either development or operations resist the change then either the resisting members should be eliminated or the switch over to the DevOps way aborted. DevOps doesn't work without cooperation and effort from all team members.
  - digi_owl 12y ago
    So in essence, devops treat every projects as if it was a startup? That is, to hell with departments, managers and lines of communications, one project, one team, one office?
hijinks 12y ago
As a "DevOps Engineer" (use to be called Sys Admin) I feel a lot of posters here are using the term wrong. A lot of us old sys admins are now calling ourselves devops because times are changing. To me devops isn't about taking a developer who likes ops to save some money. Its about taking ops out of their walled garden and into the development process. As someone in ops, I'd much rather be part of the development process as I'm the one supporting the code in production. If its a choice of message queues.. I'd rather have a vote to the one I know about supporting. It isn't really about cost savings as some companies I've been in I'm one of the highest paid engineers on the team. Its more about bringing ops into the process.
- leftnode 12y ago
  In addition to that, I always thought DevOps meant operations people using development practices and tools to automate and simplify their lives? Does it have to be more complex than that?
  - hijinks 12y ago
    You are correct. Before operations lived off tickets which just said what I did not how long it really took or if my estimates were off.
  - caw 12y ago
    If you take your definition as "ops people who code and automate", you get a really efficient ops team who can orchestrate several hundred or thousand servers. You still then have a friction point between "dev" and "ops" relating to hand off. Ops could go full PaaS, but more than likely you'll end up with a "cloud" infrastructure where your dev team will have to learn ops, or the traditional model where stuff gets thrown over the fence. Good luck deploying that service because there was no capacity planning and it'll take 6-8 weeks to get hardware from the vendor, regardless of whether ops can rack it and build it in under a day. It's far better to have an org overlap so that ops is involved in dev planning to help address roadblocks, assist in capacity planning and management, as well as help reduce friction in tasks.
deleted 12y ago
[deleted]
trcollinson 12y ago
I am a big believer in DevOps to the point where I often consult to "implement DevOps" at organizations. The frustrating part if that organizations seem to want to create "DevOps Teams" to augment their other teams. An organization cannot have a "Development Team", an "Operations Team", and then a "DevOps Team". DevOps is a set of good principles which help to guide the process of making, releasing, and maintaining good software products. These are by no means the only set of principles and organization should live by, but Developers and Operations should work to implement these principles because it will make their process more efficient. It is certainly not a matter of simply tossing in yet another department full of people and hoping that more cost efficient work gets completed. When I work with clients the inevitable question comes up: "What Principles should we implement then?" and the answers are usually more questions really. How long does it take developers to set up their working environments and get to coding? How is code tested and prepared for deployment? Are stakeholders feeling secure about deploying this new code to production users? Can we deploy effectively every time without deployment related issues? When something bad happens in production, what is the recovery procedure and how long does it take? The answers to these questions and many others often show that there are a few issues. First, developers and operations aren't speaking to one another enough and aren't really working together. Second, there is very little trust in the current environment and that lack of trust is made up for by adding more human intervention. Third, there is very little automation and this causes the lack of trust but few people what to change their habits to fix it. And finally, there is such a great divid between engineering and operations that the pains felt by one group and not shared by the other. DevOps has the principles needed to start to solve these problems. However, we don't need a department to implement those principles. We have enough departments.
- deleted 12y ago
  [deleted]
- sombremesa 12y ago
  Well said. I think this is what the author of the OP was also getting at - it's better to grow the culture (while sharing responsibility) where it belongs rather than purchasing it wholesale and shoving it down dev throats. As an employee of a large software focused corporation that does many of these things right, I had no idea how bad things can get for smaller, newer companies that haven't figured all of this out.
ownedthx 12y ago
Replace DevOps with QA and an equally valid article. Dedicated QA teams should be avoided unless truly necessary. Otherwise, you have the same problem of developers throwing over under-tested code to a group of separate people that are smart but treated poorly. A QA guy who loves to write automation gets the same promises... "oh you'll be programming mostly; manually test maybe once a month". But when they get into the door, it's the exact opposite. Developers should all be passionate about QA (read as: the quality of their product), as well as how their product is actually deployed (read as: executed in the most important of environments--production!).
mark_l_watson 12y ago
I have been referred to as a devops person since I have been developing distributed systems since 1998 and liked knowing and working the full stack, including hardware. My definition of devops: small teams totally own their project, iterating on design, implementation, deployment. I have no experience with one giant devops team for an entire company.
peterwwillis 12y ago
DevOps is not a job. It is marketing speak. Sure, there's this general idea that development and operations teams should work together, or of a programmer also having some experience adminning systems or vice versa. But this is as much a job description as "non-domestic car mechanic" is a job description. Generalized ideas that don't say anything but an indication of the background expected of the person. "DevOps" or not makes no difference; all corporations face the same problems with multiple segregated teams and people doing whatever they want in order to get their jobs done, as well as a sort of tunnel vision that prevents one from properly supporting your teammates or a product. To this end, there is only one thing that will make a significant difference in the work done, no matter what you call your team or position. The best engineer in the world becomes a janitor under the wrong leadership or teamwork.
lasermike026 12y ago
Sys Admin/Engineer here. We had it right 15 years ago. You couldn't become an admin unless you started as a programmer. You needed a deep understanding of your OS and good knowledge of C. At that time the devs ("programmers") were less knowledgeable than sys admins. You had to have to skills to analysis, debug and fix their code. When sys admins stopped coding the result was weaker admins. Sys admins ought to code within their domain, engineer, and maintain their systems for performance and high availability. It is long over due that sys admins return to their roots and get ahead of the pack.
- fsniper 12y ago
  I still believe, a good or the best sysadmins are the ones who can code. A sysadmin should know the system inside out, including the software running over it.
  - toomuchtodo 12y ago
    As a sysadmin for the last 15 years, I'm fine with this, as long as you're going to pay me as a senior software engineer (if I have to know the app inside and out) who is also going to be on call constantly and doing infrastructure all day long. If you as a business or employer want to continue to squeeze additional roles and knowledge into people, be prepared to pay for it.
    - fsniper 12y ago
      Of course you should get what you are worth and the position needs.
      - toomuchtodo 12y ago
        My experience has been people rarely want to pay appropriately when they're demanding you do several roles.
    - virmundi 12y ago
      Why would you want to get paid less? In my experience developers in Indy get about 80k. SysAdmins for IS and SQLServer get about 120k. They also don't have to work that long since the servers themselves are fairly robust.
unfocused 12y ago
I'm not surprised by his experience as it mirrors a bit of mine. This was previously discussed on HN: https://news.ycombinator.com/item?id=2674841 https://news.ycombinator.com/item?id=2674841 Although in his post, he says: "Another lesson learned: you can’t force cultural change. It has to start from the bottom up, and it needs breathing room to grow." I would argue that cultural change has to also start from the top. You can't change the culture if the leaders of the company don't work with you on it. The tendency of people at the bottom is to eventually start behaving like their managers as the managers project what is acceptable and not acceptable behavior. Trying to start change from the bottom up is a hard, hard thing to do, with a low probability of success if senior management isn't on board as well.
- jjmojojjmojo 12y ago
  > I would argue that cultural change has to also start from the top. This is where the "breathing room" comes in. I think that fostering an environment where people are free to organize themselves without edicts and policy produces overall better processes (and ultimately better quality). I think it's OK for management to lay a framework, provide a particular kind of training, but it's best to provide goals that forward the company, step back and let the people closest to the work decide the best way to achieve them.
diminoten 12y ago
Is it just me or are development practices (sem ver, git flow, test coverage, in-house code guidelines, even vcs and CI) the kinds of things that go out the window when a DevOps team forms, because they're "too slow"? For example, who has time to ensure 100% code coverage for your script that's going to fix the problem with your load balancer? The exact same thing I'm noticing is happening with the "automation" teams that are cropping up. It's like folks are discovering ways of getting around process, forgetting completely why the process exists in the first place.
- stunthamsterio 12y ago
  One thing I generally insist on, is that the DevOps Engineers are held to the same standard as the Dev's when it comes to version control, testing and documentation. After all - that goes hand in hand with the 'Dev' part of the title.
clavalle 12y ago
>You can’t have DevOps and still have separate operations and development teams. Period. I cannot disagree more. You can't have developers of your product knee deep in DevOps and be effective (at a certain size, smaller teams with less demanding operations needs can). DevOps should have very clear demarcation of responsibility in an organization. What the OP describes is an blending of responsibilities and that is the root of his problems. DevOps /is/ operations. It is just that simple. It is design and automation of support systems. Nothing more, nothing less.
- late2part 12y ago
  I mostly but not completely agree with your opinion. But, put it into context, there are more than one "right" definition of devops, just like there's more than one "right" definition of "Cloud" or "Cloud Computing." I think the author's story is more an admonition of blind foolishness and deceit than it is an indictment of the automation.
- efuquen 12y ago
  If 'DevOps' is operations, that what is the point of the term? Ops teams have been doing automation of support systems for a while, so what exactly does it mean to be DevOps? I think that's part of what the OP is getting at, if your fancy 'DevOps' team just ends up being separated and isolated from the dev team you're back at square one, you've accomplished nothing and the same divide exists with a new label. As someone who works on the dev side, the ops side, and the mixture of the two I feel the OPs pain acutely and have had mixed experiences. People on the dev side need to understand and willing to learn about ops, it's not rocket science and you don't have to be knee deep in it to get the basics down. When ops becomes this black box and becomes a bottleneck is when you start to see dysfunction.
  - clavalle 12y ago
    >If 'DevOps' is operations, that what is the point of the term? It is taking more of a gestalt systems approach to the entire support structure. This means more intelligent support programs and treating it like designing and delivering a software project (or, more accurately a series of subprojects) as opposed to a loose amalgam of scripts. This also requires the product development team to be keenly aware of that support architecture and how their decisions factor into it especially in terms of testing, deployment, and module interactivity. > Ops teams have been doing automation of support systems for a while, so what exactly does it mean to be DevOps? DevOps has been around for a while -- way before there was a word for it. I think DevOps has come into its own, recently, though because platforms have become hugely more flexible while at the same time a consensus built on what it means to manage those systems. The difference, I think, is whether things are built to remain somewhat static or if change is designed for. Are development ideas like encapsulation and decoupling etc designed into the system? Smart sysadmins have been doing this stuff for a long time but now organizations are taking the approach into account all the way up and down the chain. And, yes, that means that product developers need to be fully aware of operational needs but, in my opinion, I don't think that means they build the operational tools themselves. >your fancy 'DevOps' team just ends up being separated and isolated from the dev team you're back at square one Demarcation of responsibility doesn't mean demarcation of information. The product development team will need to be intimately aware of what is happening in operations and vice versa. In a DevOps implementing environment it is unavoidable. > the dev side need to understand and willing to learn about ops, it's not rocket science These days it's getting pretty close. I don't think it is fair to product developers to say "Listen, we like the idea of what Docker can do for us so you are going to learn it inside and out." Somewhere you have to say "Hey, we probably want to implement Docker. Ops, dig in. You know how our product is currently structured and how operations currently works with that. Try to figure out what needs to change and what doesn't. X and Y from the dev team will be made available for deeper discussions and testing as needed. When the overall gist of what has to change is known, we'll all get together, ops and produce dev, to get everyone on the same page." It's a bleeding of information, not a full on 'shake it all up' scenario. Neither side can be a black box anymore if the current state of software development and delivery is going to be taken advantage of. But neither can everyone know and do everything or context switch as needed. Keep in mind: I am not a DevOps theorist -- maybe I have it completely wrong according to the people that came up with it but this is my practical understanding of the movement.
hedwall 12y ago
Obligatory lusis quote and link... "Devops means giving a shit about your job enough to not pass the buck. Devops means giving a shit about your job enough to want to learn all the parts and not just your little world. Developers need to understand infrastructure. Operations people need to understand code. People need to fucking work with each other and not just occupy space next to each other." http://blog.lusis.org/blog/2013/06/04/devops-the-title-match/ http://blog.lusis.org/blog/2013/06/04/devops-the-title-match...
sbt 12y ago
DevOps may be hyped, but it's not outright bullshit. DevOps is just the realization that you would rather have a fewer people knowing everything, than many people knowing disjoint areas. The same force underpins the full stack crowd, or the notion that pizza size teams are more efficient. While this is not always possible in practice, in my experience it is better if the project is small enough that you can pull it off. The memory bandwidth in one single person's brain is simply much higher than the bandwidth between people. To maximize performance you want to run your project on as few brains as possible.
mgrennan 12y ago
Former embedded developer, former systems developer, former language developer, former application developer, former senior systems administrator and now DBA. (30 years) Simply put, the problem IS TITLES. They cause the "not my job" problem. If you've been called, help fix the problem. If you don't understand the problem, stay with it, maybe you'll learn something. Just because your a "title" doesn't mean your s--t don't stink. Unless your are designing your CPUs, writing the languages that your write your OS and the applications that run your hardware, you are riding on the back of someone else. Do what's ask of you and ask how you can make everyone's job better. Then you'll be employed and paid well for a long time.
mrjatx 12y ago
I'm devops and in my opinion the title infers this; my job is to keep the communication between developers and the operations teams in sync and copacetic. Usually, Ops doesn't know dev and dev doesn't know ops so if there's nobody to translate and explain things in each others technical language things can't be done quickly and reliably. The wrong environments will be configured, things won't have backups, it will be like the wild west and things will be stressful for both sides. My job, just like it was when I was a sysadmin, is to keep things running smoothly. To optimize both sides of the coin and to take things off their plate that I myself can deal with that might be a blocker to them finishing tasks and to us getting features out or server migrations done. Some devs know ops and some ops know dev, those guys, more often than not I just check in with and see that things are running smoothly. For the other folk I try to create systems and workflows that get blockers out of their way. I try to be a useful and reliable tool to both sides as much as I can be. Hopefully the way I've been doing this for the last two years since I moved into a more Devops related role has been helpful to everyone.
- pbhjpbhj 12y ago
  Completely OT: "copacetic", new one for me - where have you come across this word; the etymology is intriguing to me. http://en.wiktionary.org/wiki/copacetic http://en.wiktionary.org/wiki/copacetic
  - mrjatx 12y ago
    Boy that's interesting, I've never thought about it. I'm an American who grew up in Germany but have an Italian and German family who grew up in the projects of NY (the Love Canal, Niagara Falls). My grandmother (german) and grandfather (italian) married and their families constantly fought and pretty much hated one another. I remember them fighting (mostly arguing) and the men going "Ey ey, lets keep things copacetic." (They might have said capasetti) I always thought it was a pretty and a smart sounding word. I guess not.
    - pbhjpbhj 12y ago
      >pretty and a smart sounding word // Sounds good/smart to me! The wiktionary page mentions possible Italian-NY etymologies so you fit in there. Funnily enough I was going to try and be clever and guess you were a New Yorker [based on that] but noticed elsewhere you appear to be in TX. Interesting, thanks for responding.
wtbob 12y ago
From one of the comments: > It’s easier to train a developer to do sane operating procedures than to teach a sysadmin the developer mindset (testing, revision control, breaking down to logical units and similar). Having done both jobs, I don't really think that's true, unless by 'sysadmin' one means the type of low-skill position which resets passwords. A system administrator is responsible for administering a system of hosts and software; his should be a more complex job than a developers. He should be responsible for test systems as well as for production systems; his scripts should live in source control. I'll grant that functional decomposition may not be as vitally important to his work as it is to a normal developers—but then, his scripts will inevitably grow to be pretty complex, so maybe it will be after all. That doesn't mean that developers are stupid either. Sure, at BigEnterpriseCorp a developer is just a seat-warmer who implements directives like 'add a method to the Foo class which takes two arguments, adds them and returns the result,' but a real developer is a student (whether formally taught or informally self-taught) of computer science; he understands and can choose & implement both algorithms & data structures in order to solve a problem. A good DevOps hire is going to have some skill in both areas, not as a weak sysadmin and a weak dev, but as a strong sysadmin and a strong dev. A good DevOps team is going to be more skilled (and expensive) than the equivalent dev and ops teams, but it will also be more productive and less error-prone. If you're trying to build a devops team from Initech, Initron and Initrol low-skill employees, you're probably going to fail. If you're building a devops team to reduce costs, you're probably going to fail. But if you're building a devops team to increase profitability and competitiveness, you may succeed.
- acveilleux 12y ago
  In the Unix/Linux world, a lot of the Sys Admins I've met have been very strong technically. Knew at least a scripting language and basic C programming. The trend however has been for less and less coding skills and more superficial. Sometimes not even any shell scripting skills. In the Windows world, a lot of the Sys Admins do not have those skills. It's changing, things like PowerShell and various others are helping, but a lot of Windows Admin are just not that comfortable away from the GUI. I think the comment you refer to is by someone with a Windows dev background. GUIs don't version... The mainframe world folks tended to be very silo'd (and IBM engineers were never far to hold hands.)
dsr_ 12y ago
A senior systems administrator can and does write programs for automation, monitoring, configuration management, and all the little things that are not customer-facing applications. I used to be able to use "senior systems administrator" as a short-hand for those skill sets, but about 2010 that stopped working. Now I need to use "DevOps" as an imperfect substitute in the title; otherwise only junior people apply, even though the rest of the ad goes into the same depth about skills and abilities.
kokey 12y ago
I was recently looking for a new job, and from the job specs and interviews I have realised that 'DevOps' means very different things to many people. I am a system administrator who likes to automate things and write code, and also spent decades supporting developers and good Agile environments, etc. and production ops environments. Some companies think DevOps is a developer who knows how to deploy services on Ubuntu, and their interview tests the implementation of algorithms covered in recent CS courses. I removed any mention of DevOps from my CV, that made things less confusing for everyone.
serve_yay 12y ago
This piece is so confounding about what Devops is and isn't, so much that I'm ready to just flip the bozo bit on anyone who uses the term credulously. This is silly.
- jjmojojjmojo 12y ago
  Now you know how I feel.
tymski 12y ago
Whether you want to or not, it's your choice. However, the push towards DevOps as opposed to Dev is based on efficiency; it's easier to hire a developer that knows operations, instead of a developer and an operations employee. Even without DevOps, a lot of positions are Full Stack, so you need to know front and back-end development and technology. I don't think it's merely a trend, it's more or less a push towards saving money and time. If you don't want to be a full stack eng or a DevOps eng, then you're limiting yourself and you're limiting the opportunities you'll have. It sucks for those who want to limit themselves to one specific group of tasks instead of branching out, but it's just how it is.
cubano 12y ago
I think the most striking point made here was the one about culture and how without everyone buying into the DevOp paradigm, its just another unused tool. I say this because I've lived it...the last 3 (three!) companies I've contracted at were pushing dev code to live servers! Even after I setup svnserve and created the repos, no one was really interested in changing workflow. I choose svn because, lets face it, it has a much lower learning curve then git. It wasn't until I absolutely refused to work on the live server that other the devs grudgingly started using SVN...I'm sure they are all back to FTPing to the live server now that I'm gone. Without the proper culture from above, things just aren't going to change.
TranceMan 12y ago
Last month I went to the UK PHP Conference where Jeremy Quinton spoke about DevOps. The biggest take-away I got was: 'Do not set out to create a DevOps team or employ people for that role' My interpretation was the role simply doesn't exist. If your company has a team of developers and a team of people who keep the wheels turning - in an ideal world you already have a DevOps team as long as they collaborate. If you also employ smart people you will get 'developers that know how to do sysadmn' and 'sysadmins that know how to develop' - definitely this indeed should be embraced but not expected.
deleted 12y ago
[deleted]
waprin 12y ago
I think DevOps is a bit of a fad anyway, because a huge chunk of the problems people are solving will soon be solved by off-the-shelf standard cloud solutions. It makes no sense for everyone to be writing their own deployment/configuration Chef and Docker setups when they can just use increasingly powerful and flexible PaaS options like AppEngine/Heroku.
chrismarlow9 12y ago
I'm a dev who learned everything by experience. I'm in my 20's and have been doing this for ~20 years, so I have a bit of experience with both the new and the old world. My take is that the code quality of whats allowed to be in production has gotten worse, that's it. It's an effect of an abundance of RAM and virtualization. The abundance of ram lets you just trash it and do things in a horribly inefficient way. Sure the extra memory can help with time memory trade off if you have something that doesnt need to scale a great deal, but processing things inline will still be faster. Developers have forgotten how to handle streams. Usually forcing a dev to learn a highly functional language that's immutable can fix these issues (it must involve a large dataset, where you know they will tax out the memory, and then code review to help). I mention virtualization because a common pattern I've seen is to write poor code, and just put more boxes behind the load balancer when things start failing because the technical debt is so high it's an impossible mountain to climb. This obviously isn't the right approach and will leave your devops guys worn out chasing your everyday distributed systems problems that happen because it wasn't built right for distribution and the dev has never read the fallacies of distributed computing. And while you're fixing this technical debt a team of competent engineers builds a competing product and trumps you hard (second mover advantage). Maybe people paid a bit more attention when things cost 10x as much as they do now and every answer (both bad and good, and you can't tell the difference) wasn't a stackoverflow search away.....
- Varcht 12y ago
  Good point. Currently I am working on parsing a 5.5 million line STEP file. I've learned more about memory usage and high level overhead in the last week then I had in the previous 10 years of .Net dev.
- aaron_m04 12y ago
  So you're in your late 20s and started working when you were in elementary school? :-)
korzun 12y ago
I will sum this up: > We had edicts to make things happen, under the assumption that if we built it, the developers would embrace it. Followed by: > What matters is good people working together without artificial boundaries. Former will never produce latter. Always work with engineering before building anything that impacts their day to day.
stcredzero 12y ago
So that’s what DevOps really is. You take a bunch of really skilled, passionate, talented people who don’t have their heads shoved so far up their own asses that they can take the time to learn new things. What you want to look out for is the formation of "walls." People have an instinct to do this. People will form cliques -- which effectively are invisible barriers to understanding and honest collaboration. In a 5 person startup, this is easy. You have one group. Most of the critical information flows are inside this boundary, and those which aren't, the group is already intensely aware of. (Investors, customers.) However, when headcount has started to grow, and when you are forming a separate group to take on a task that requires intense collaboration, maybe you're doing the wrong thing? Maybe what you want is a separate role for people who get embedded in a variety of different groups? What if DevOps people belonged to DevOps, but also, and primarily, belonged with a particular group of developers? Then, they could have firsthand experience of intense collaboration, but also meet once a week with all the other DevOps role holders and communicate to standardize and avoid duplicated effort? A generalization of this: Make sure you're building bridges and not compartments.
TylerH 12y ago
tl;dr because waking up in the middle of the night to debug someone else's app really, really sucks.
adamconroy 12y ago
A smart, meticulous, better than other people person like myself can't get past the hubris, sorry.
devonkim 12y ago
I think the right thing to do is to firstly get rid of the term because it's become so meaningless and abused in so many ways. At least "cloud" has some semblance of at least meaning "whatever that we used to do in the 90s for IT is BAD" and implies great upheaval, but DevOps is a really, really weak movement in comparison because most companies that are dysfunctional try to contain whatever DevOps is into their existing (typically broken, so let's propagate that error 100x, right?) practices. While I agree that sysadmins should be developers first it is increasingly less likely you'll find developers willing to become computer janitors and deal with the ugly realities of operations in, for example, larger companies. I wouldn't think that John Carmack ever liked having to figure out the installer scripts and keep them working across all the supported platforms through all the BS that Microsoft and different Linux distributions could throw at you, yet that's what a more "ideal" DevOps organization described by many model would have John Carmack do. What should have happened though is that instead of hobbling the John Carmacks and dragging down their productivity, you should have fired the developers that refuse to cooperate with operations and those that keep actively putting off writing "boring" code or never documenting things that do matter for operations to figure out and learn through a failure that something broke. That kind of behavior is the developer equivalent of not being a team player at all. It's true that none of these things actually make you a better programmer in itself, but these things are what contribute to making you more PROFESSIONAL instead of a cowboy renegade programmer that doesn't care if nobody else can use his code because "clearly, if you can't figure it out quickly you don't deserve to use it" (not an exaggeration at all I've heard). Having been in both development and operations in rather large (and small!) companies now, I'd far, far, far rather stay as a hermit-crab developer than deal with the constant dysfunctions of enterprise Faustian nightmares that are required for you to deal with as an operations engineer. You can generally ignore a lot of the noise in a super-siloed off R&D organization, for example, but you cannot ignore your pager going off at 2 am because a random server crashed causing someone to get very, very angry and demand immediate resolution when it is your JOB to keep those things going. Hardly going to write anything really meaningful with that ruining your work week pretty much every week.
tomohawk 12y ago
If you want to make a devops omelet at a place that has an over the wall mentality, you have to break some eggs (and walls). I was part of a successful transition to devops from a thoroughly silo'd operation, where the boss basically gave the dev team the responsibility and authority to make it happen. He was also able to snag some of the traditional ops team and make them part of the new team. After a period of storming, the new team came up with some innovative ways to get the job done. Once the team realized that they were it in terms of getting it done, it got done. Quality and job satisfaction improved, cost was reduced. I've also seen various unsuccessful transitions. In every case, the responsibility was divided into various silos, which sandbagged everything.
DyslexicAtheist 12y ago
why devops isn't bullshit: http://blog.valbonne-consulting.com/2015/03/10/is-devops-bullshit/ http://blog.valbonne-consulting.com/2015/03/10/is-devops-bul...
parasubvert 12y ago
Devops is a professional movement to codify knowledge about how web companies operate at scale and speed. It gets misinterpreted sometimes (more lately as it becomes a fad). Doesn't mean its bullshit. Same problems with Agile, Lean, Six Sigma, or any other codification of how humans could or should work together to deliver a product. Many of the lessons learned here aren't exactly universal, for example some of the better Devops cultures (e.g Google) do have separate dev & ops teams, they just have good collaboration.

269 points by jjmojojjmojo 12y ago

hydrogen18 12y ago

Once again, we have the stylish new white background with barely legible text design pattern showing on HN.

otaku888 12y ago
It's just a common Wordpress theme: https://wordpress.com/themes/sela/ https://wordpress.com/themes/sela/
afandian 12y ago
Agreed, the font is very light weight. The way it renders on my screen (Macbook Air) gives me a headache.
- agarcia-deniz 12y ago
  I'm glad it's not just me. looks like he changed it to something else
delta1 12y ago
What does HN have to do with the theme of this guy's blog? If you're struggling to read it, try https://www.readability.com/shorten https://www.readability.com/shorten
nkozyra 12y ago
The difference between this font color and #000 is actually really really minor at a larger scale. The issue tends to be exacerbated by smaller print.
ipedrazas 12y ago
I don't bother reading it from the source, push it to pocket, and off I go with their stylish style :)
jjmojojjmojo 12y ago
I switched back to the "2010" theme, hopefully people will have a better reading experience now. Thanks for letting me know (letting me know in private without the snark would have been better, but beggars can't be choosers, so I'll take what I can get).

mugsie 12y ago

DevOps as it has been implemented in tons of company is bs. Its like the "Agile" push that companies thought they could get by buying a tool, and changing nothing else (release schedules, feature based release dates etc). My favorite way of looking at it is simple - replace the word DevOps with empathy. That is the core of the idea, so lets try it - "Empathy Team", "Empathy Person", "Empathy Engineer". DevOps is about dev and ops teams communicating and having empathy for the requirements of the other team.

smcl 12y ago
Obligatory quote echoing your final sentence, from the seminal DevOps thought leader Jon Hendren: "DevOps is mostly just a way of reorganizing all your nerds and making them talk to each other more"
VLM 12y ago
"replace the word DevOps with empathy" This scheme also works with diversity, quality, best practices, and change management.
- mugsie 12y ago
  Yup - it does. I just seem to prefer the empathy quote for some reason :)
  - mmatants 12y ago
    Empathy is the actual mechanism for how those other things (quality, etc) are achieved. Diversity helps empathy (represent the user and relate to them better).
    - VLM 12y ago
      Another way to extend the analogy is its a standard management technique when provided with something you don't want to do, set up an isolated team with no deliverables, no authority, and no direct reports. Ta Da we got us "quality" or "diversity" or "Dev Ops" and we didn't have to change anything.

chomp 12y ago

How does one reconcile DevOps with the Sarbanes-Oxley act? http://en.wikipedia.org/wiki/Separation_of_duties http://en.wikipedia.org/wiki/Separation_of_duties

mugsie 12y ago
With difficulty. For example - you can still have an Ops team, and they have the required access. The developers access to production is done on a case by case basis, with some sort of tracking of who logged in. and why. Also, doing full pre - merge code review helps, as it is yet another set of eyes on the code that will be pushed to production... In short - its not a "one solution fits all", and it is something that needs to be discussed with you SOX auditors, to find a working solution for your company
- chomp 12y ago
  Thanks for the response! We keep a distinct separation of duties, but we try to keep both groups in constant contact to minimize animosity in both groups. We pick and choose some DevOps ideals to make it fit, was just curious about other people's experience.
snorkel 12y ago
SoD simply means your systems or processes require at least two different people to make any changes. If your entire tech team is DevOps then it simply means the dev who tests and deploys a given code change is different from the dev who wrote the code. These different roles can be per task, it doesn't have to be full-time persons assigned to full-time roles.
emodendroket 12y ago
It seems, from reading the article, like the need would be satisfied by having two dev-ops engineers manage something, rather than one developer and one ops guy.
signal 12y ago
DevOps doesn't mean people doing multiple jobs. You can collaborate with separated roles and be productive. Furthermore you can build abstraction to simply remove manual, unrecorded interaction with systems to satisfy auditing and regulation.

300bps 12y ago

It seems OP is using the word DevOps to describe what is traditionally referred to as "special projects". Essentially, the non-technical CIO hears a new buzzword and hires a team to implement it in an ivory tower without fully partnering with existing team members. From OO to SOA to Agile to DevOps and everything in between these initiatives almost always fail. That of course doesn't mean that OO is bad, or that DevOps is bad. It is just that the "special project" methodology is terrible.

KaiserPro 12y ago

So this is the thing. There are three things that really go against a true "devop" system: 1) cost. Ops are cheaper than devops, sometimes twice as much. Yes yes, flexibility. But thats a misnomer. If your dev teams are doing donkey sysadmin work, because they don't have the experience, or are up all hours fixing things, you're going to endup with a high staff turn over. 2) division of labour As a company grows, they can employ more and more specialist staff. This gives them a competative edge as they are able to make nuanced decisions based on skill, or experience. A decent storage admin will be able to build a storage system that will scale relaibly, cheaply and with less downtime than someone who is not a specialist. 3) burn out. A human is not designed to multitask 14 hours a day. Coding, testing, supporting, infrastructure procuring, for anything other than the smallest company is too much for a single role. There are great things that come from a devop "culture"; greater personal autonomy. However like all things, its not one size fits all. There is a great deal of arrogance where people assume that a sysadmin never automate anything. or that they can't program. They can and do, you know why? because they want to carry on searching ebay, scamming $someone boasting about network speed, or just plain getting shitfaced.

RankingMember 12y ago
I was with you until "because they want to carry on searching ebay, scamming $someone boasting about network speed, or just plain getting shitfaced.", but I'm guessing you're just being cheeky.
- KaiserPro 12y ago
  I was bending the truth a tinybit. The idea is that a good sysadmin is an idle sysadmin. (having offloaded tech support to the juniors) unless they are playing with new stuff.
- gknoy 12y ago
  I was thinking it was a BOFH ("Bastard Operator From Hell") reference. [0] The BOFH is a fictional sysadmin who is worse than any you have ever had, in entertaining ways. He basically does exactly the things the OP mentioned, while dealing with clueless management, undeliable vendors, and a Sith Apprentice^W^WYoung Coworker who learned well, and is constantly trying to stab him in the back. He's like a mix between Catbert and Wally. ;) 0: http://www.theregister.co.uk/2015/02/27/bofh_2015_episode_2/ http://www.theregister.co.uk/2015/02/27/bofh_2015_episode_2/

legolin 12y ago

It seems that the main problem here was not DevOps but rather the culture in which he was asked to do his work. Creating a devops team whether "by fiat or by force" doesn't sound like a recipe for a good experience. My experience is different: DevOps saves me many headaches by letting me easily deploy things for small clients. A larger company I work with uses Chef to manage their 40+ servers; the IT department doesn't touch these servers. Rather, 3 or 4 developers share in the work of writing cookbooks, adding instrumentation, deploying code and maintaining the servers. It would be handy to have a full-time guy as well (OP mentioned the amount of work needed to keep up with just your basic security patches) but we -- as developers -- love the access we have to our entire stack.

jjmojojjmojo 12y ago
I don't want to take away from your point, but you touched on security - I mentioned it because it was a constant pain point for both myself and for our teams. Like operations, it's something you can be good at on a superficial level, and do OK for most things, but really getting it and being good at it takes experience and time. I don't think it came through in my post, but having a security expert (or any specialist) in the mix can be a huge help in making the product better - I know this from personal experience - you just have to make sure they're integrated, challenged, and respected in the group.

dschiptsov 12y ago

It seems like a mere attempt to adapt labor-division best practices from assembly plant industry. More managers, bigger budgets, easy staff rotation - that kind of crap. Also a fresh new meme to ride on. Nothing special.

jussij 12y ago

To me DevOps is a cost cutting measure whereby an organization can save a few hundred thousand dollars by having one individual not only do the development but also handle the deployment and the testing. In theory that will eliminate the need for an additional tester and a second individual to do the change control. But those short term gains expose the organization to the massive risks that the one and only one individual will make a mistake, with no second individual available to do a double check. The reality is people make mistakes and by eliminating people from that double checking process may well save money in the short term, but in the long run it greatly adds to the of risk failure. While there are short term gains, the risks to the organization can be massive: http://www.bloomberg.com/bw/articles/2012-08-02/knight-shows-how-to-lose-440-million-in-30-minutes http://www.bloomberg.com/bw/articles/2012-08-02/knight-shows...

mugsie 12y ago
That is actually an example of the old style of IT. The person deploying the software had no idea how it worked on an internal level. The person who wrote the software had no idea how it was actually deployed in production. They didn't have a quick rollback in place (this is the exact reason you have quick rollback ability) The deploy was done by hand, with a user typing. That is probably the biggest risk you can have in a production system. It means that fully repeatable builds in test & pre production are impossible. I am not saying "DevOps" would fix this, but a bit of empathy / comunications between the dev team and ops team might have averted it. http://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock_trading_disruption http://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock...
marshally 12y ago
Well, look, I already told you. I deal with the goddamn customers so the engineers don't have to!! I have people skills!! I am good at dealing with people!!! Can't you understand that?!? WHAT THE HELL IS WRONG WITH YOU PEOPLE?!!!!!!!

SeanDav 12y ago

So many misconceptions here as to what DevOps is. I think mugsie comes closest with his use of the term "empathy". This is very very close, but I would add one more term "cooperation". DevOps, when implemented correctly can have enormous, order of magnitude increases in both productivity and quality, but it is not voodoo, despite this almost unbelievable claim. It involves empathy and cooperation and it requires this because in order to implement DevOps in a meaningful way, an organization has to vastly increase communication between "silos". It has to formalize, document and automate as many processes as possible. DevOps is NOT easy to implement and involves a cultural shift and absolute management buy-in. It is best implemented in small projects, where clear wins can be demonstrated. Use this success to bootstrap the process to bigger projects. I do agree with the OP that if you have a DevOps team, you are probably not doing DevOps. The whole point of DevOps is to get rid of silo mentality. No more us and them, but all us. Implemented correctly, everyone is DevOps.

mpdehaan2 12y ago

Having been around since the inception of this space, when things were much more tools focused, to now, when it's a lot more about "culture" (all good), I feel 95% of DevOps talks are people discussing what DevOps is. As such, it's now a metasyntactical label for a lot of things that can by good in various scenarios. I agree it is time to retire the label and instead talk about those specific things -- automation, culture, continuous improvement, etc, in nearly all instances.

bonn1 12y ago

Don't know if the concept of DevOps is bs but I know that it's unfortunate if a developer never did system administration: there are many of this "git push heroku" kind of devs who pee in their pants if they have to deal with bare metal and the production servers. It's not that they wouldn't be able too handle them, it's that they are afraid because they never did and besides, good system administration knowledge leads always to better application and system design.

snorkel 12y ago
I blame Java much more than heroku. Corporate drone Java developers have a much greater tendency to only write code (too much code!) and not be interested in learning anything beyond the confines of their comfy auto-completing IDE. It seems to me Java developers will howl "not my job!" the loudest whenever anyone suggests that maybe they can mange their own environments.
- digi_owl 12y ago
  Sounds like what has been around since the mainframe days and batch computing. Or maybe one could liken it to Star Trek, with ops being the people keeping the warp core going even when faced with borgs and black holes?
- HelloNurse 12y ago
  In the enterprise Java world, the comfy auto-completing IDE is a smart strategic choice when "managing the environment" means weeks of frustration at installing, configuring and fitting together by trial and error an application server like IBM Websphere and the application. Funny tools like Maven, Subversion and certain Eclipse plugins add extra puzzles and unpredictability for the true amateur, but the normal unforeseen complications are usually enough to forfeit any hope of good automation.

snorkel 12y ago

Unfortunately the term DevOps has been diluted in the job market. Now most DevOps job postings are actually just plain ol' sys op roles, where "development" means writing tools for the actual developers. If you want to spend at least 50% of your time writing code then avoid any job postings that mention "ops".

reitanqild 12y ago

Funny and, unfortunately, relevant, especially the last tweet: https://twitter.com/devops_borat https://twitter.com/devops_borat

passive 12y ago

It's fascinating how much this engineers experience shares with mine. We both appear to have similar approaches to working with technical systems, that has led to skillsets that are more broad than deep. We've both had interview problems because of this, and we've both ended up in DevOps. I've also had more or less the exact same experience with DevOps, though with (hopefully) a happier ending. I was on a team working for a client that desperately needed "DevOps-type" things. So I built them, and their infrastructure team ignored them. Eventually we stopped working with that client, and I was moved to a different organization, where each development team was doing it's own thing, operations-wise, and had no interest in adopting a uniform approach, so my (short) efforts there were also for naught. But then I had the opportunity to come back to the first client, as their DevOps Manager. Before I took the job, I talked to their current development organization about what their pain points were, and what actions they would be willing to take to alleviate them, and I built a strategy around that. A key part of this strategy was gradually eliminating the need for a DevOps team, as I firmly believe the best application of "DevOps" is as a joint project between Development and Operations to optimize shared processes. Once the friction is minimal, you don't need a team there anymore. So we have a developer embedded in our meetings, and have more or less full control of any infrastructure.

Fuxy 12y ago
I think there is a fundamental disconnect between what DevOps is and the classical business organization structure. DevOps is not a department and it will not work if it is a department DevOps is a way of doing things to achieve maximum efficiency and everyone on development and operations must embrace it for it to work. If either development or operations resist the change then either the resisting members should be eliminated or the switch over to the DevOps way aborted. DevOps doesn't work without cooperation and effort from all team members.
- digi_owl 12y ago
  So in essence, devops treat every projects as if it was a startup? That is, to hell with departments, managers and lines of communications, one project, one team, one office?

hijinks 12y ago

As a "DevOps Engineer" (use to be called Sys Admin) I feel a lot of posters here are using the term wrong. A lot of us old sys admins are now calling ourselves devops because times are changing. To me devops isn't about taking a developer who likes ops to save some money. Its about taking ops out of their walled garden and into the development process. As someone in ops, I'd much rather be part of the development process as I'm the one supporting the code in production. If its a choice of message queues.. I'd rather have a vote to the one I know about supporting. It isn't really about cost savings as some companies I've been in I'm one of the highest paid engineers on the team. Its more about bringing ops into the process.

leftnode 12y ago
In addition to that, I always thought DevOps meant operations people using development practices and tools to automate and simplify their lives? Does it have to be more complex than that?
- hijinks 12y ago
  You are correct. Before operations lived off tickets which just said what I did not how long it really took or if my estimates were off.
- caw 12y ago
  If you take your definition as "ops people who code and automate", you get a really efficient ops team who can orchestrate several hundred or thousand servers. You still then have a friction point between "dev" and "ops" relating to hand off. Ops could go full PaaS, but more than likely you'll end up with a "cloud" infrastructure where your dev team will have to learn ops, or the traditional model where stuff gets thrown over the fence. Good luck deploying that service because there was no capacity planning and it'll take 6-8 weeks to get hardware from the vendor, regardless of whether ops can rack it and build it in under a day. It's far better to have an org overlap so that ops is involved in dev planning to help address roadblocks, assist in capacity planning and management, as well as help reduce friction in tasks.

deleted 12y ago

[deleted]

trcollinson 12y ago

I am a big believer in DevOps to the point where I often consult to "implement DevOps" at organizations. The frustrating part if that organizations seem to want to create "DevOps Teams" to augment their other teams. An organization cannot have a "Development Team", an "Operations Team", and then a "DevOps Team". DevOps is a set of good principles which help to guide the process of making, releasing, and maintaining good software products. These are by no means the only set of principles and organization should live by, but Developers and Operations should work to implement these principles because it will make their process more efficient. It is certainly not a matter of simply tossing in yet another department full of people and hoping that more cost efficient work gets completed. When I work with clients the inevitable question comes up: "What Principles should we implement then?" and the answers are usually more questions really. How long does it take developers to set up their working environments and get to coding? How is code tested and prepared for deployment? Are stakeholders feeling secure about deploying this new code to production users? Can we deploy effectively every time without deployment related issues? When something bad happens in production, what is the recovery procedure and how long does it take? The answers to these questions and many others often show that there are a few issues. First, developers and operations aren't speaking to one another enough and aren't really working together. Second, there is very little trust in the current environment and that lack of trust is made up for by adding more human intervention. Third, there is very little automation and this causes the lack of trust but few people what to change their habits to fix it. And finally, there is such a great divid between engineering and operations that the pains felt by one group and not shared by the other. DevOps has the principles needed to start to solve these problems. However, we don't need a department to implement those principles. We have enough departments.

deleted 12y ago
[deleted]
sombremesa 12y ago
Well said. I think this is what the author of the OP was also getting at - it's better to grow the culture (while sharing responsibility) where it belongs rather than purchasing it wholesale and shoving it down dev throats. As an employee of a large software focused corporation that does many of these things right, I had no idea how bad things can get for smaller, newer companies that haven't figured all of this out.

ownedthx 12y ago

Replace DevOps with QA and an equally valid article. Dedicated QA teams should be avoided unless truly necessary. Otherwise, you have the same problem of developers throwing over under-tested code to a group of separate people that are smart but treated poorly. A QA guy who loves to write automation gets the same promises... "oh you'll be programming mostly; manually test maybe once a month". But when they get into the door, it's the exact opposite. Developers should all be passionate about QA (read as: the quality of their product), as well as how their product is actually deployed (read as: executed in the most important of environments--production!).

mark_l_watson 12y ago

I have been referred to as a devops person since I have been developing distributed systems since 1998 and liked knowing and working the full stack, including hardware. My definition of devops: small teams totally own their project, iterating on design, implementation, deployment. I have no experience with one giant devops team for an entire company.

peterwwillis 12y ago

DevOps is not a job. It is marketing speak. Sure, there's this general idea that development and operations teams should work together, or of a programmer also having some experience adminning systems or vice versa. But this is as much a job description as "non-domestic car mechanic" is a job description. Generalized ideas that don't say anything but an indication of the background expected of the person. "DevOps" or not makes no difference; all corporations face the same problems with multiple segregated teams and people doing whatever they want in order to get their jobs done, as well as a sort of tunnel vision that prevents one from properly supporting your teammates or a product. To this end, there is only one thing that will make a significant difference in the work done, no matter what you call your team or position. The best engineer in the world becomes a janitor under the wrong leadership or teamwork.

lasermike026 12y ago

Sys Admin/Engineer here. We had it right 15 years ago. You couldn't become an admin unless you started as a programmer. You needed a deep understanding of your OS and good knowledge of C. At that time the devs ("programmers") were less knowledgeable than sys admins. You had to have to skills to analysis, debug and fix their code. When sys admins stopped coding the result was weaker admins. Sys admins ought to code within their domain, engineer, and maintain their systems for performance and high availability. It is long over due that sys admins return to their roots and get ahead of the pack.

fsniper 12y ago
I still believe, a good or the best sysadmins are the ones who can code. A sysadmin should know the system inside out, including the software running over it.
- toomuchtodo 12y ago
  As a sysadmin for the last 15 years, I'm fine with this, as long as you're going to pay me as a senior software engineer (if I have to know the app inside and out) who is also going to be on call constantly and doing infrastructure all day long. If you as a business or employer want to continue to squeeze additional roles and knowledge into people, be prepared to pay for it.
  - fsniper 12y ago
    Of course you should get what you are worth and the position needs.
    - toomuchtodo 12y ago
      My experience has been people rarely want to pay appropriately when they're demanding you do several roles.
  - virmundi 12y ago
    Why would you want to get paid less? In my experience developers in Indy get about 80k. SysAdmins for IS and SQLServer get about 120k. They also don't have to work that long since the servers themselves are fairly robust.

unfocused 12y ago

I'm not surprised by his experience as it mirrors a bit of mine. This was previously discussed on HN: https://news.ycombinator.com/item?id=2674841 https://news.ycombinator.com/item?id=2674841 Although in his post, he says: "Another lesson learned: you can’t force cultural change. It has to start from the bottom up, and it needs breathing room to grow." I would argue that cultural change has to also start from the top. You can't change the culture if the leaders of the company don't work with you on it. The tendency of people at the bottom is to eventually start behaving like their managers as the managers project what is acceptable and not acceptable behavior. Trying to start change from the bottom up is a hard, hard thing to do, with a low probability of success if senior management isn't on board as well.

jjmojojjmojo 12y ago
> I would argue that cultural change has to also start from the top. This is where the "breathing room" comes in. I think that fostering an environment where people are free to organize themselves without edicts and policy produces overall better processes (and ultimately better quality). I think it's OK for management to lay a framework, provide a particular kind of training, but it's best to provide goals that forward the company, step back and let the people closest to the work decide the best way to achieve them.

diminoten 12y ago

Is it just me or are development practices (sem ver, git flow, test coverage, in-house code guidelines, even vcs and CI) the kinds of things that go out the window when a DevOps team forms, because they're "too slow"? For example, who has time to ensure 100% code coverage for your script that's going to fix the problem with your load balancer? The exact same thing I'm noticing is happening with the "automation" teams that are cropping up. It's like folks are discovering ways of getting around process, forgetting completely why the process exists in the first place.

stunthamsterio 12y ago
One thing I generally insist on, is that the DevOps Engineers are held to the same standard as the Dev's when it comes to version control, testing and documentation. After all - that goes hand in hand with the 'Dev' part of the title.

clavalle 12y ago

>You can’t have DevOps and still have separate operations and development teams. Period. I cannot disagree more. You can't have developers of your product knee deep in DevOps and be effective (at a certain size, smaller teams with less demanding operations needs can). DevOps should have very clear demarcation of responsibility in an organization. What the OP describes is an blending of responsibilities and that is the root of his problems. DevOps /is/ operations. It is just that simple. It is design and automation of support systems. Nothing more, nothing less.

late2part 12y ago
I mostly but not completely agree with your opinion. But, put it into context, there are more than one "right" definition of devops, just like there's more than one "right" definition of "Cloud" or "Cloud Computing." I think the author's story is more an admonition of blind foolishness and deceit than it is an indictment of the automation.
efuquen 12y ago
If 'DevOps' is operations, that what is the point of the term? Ops teams have been doing automation of support systems for a while, so what exactly does it mean to be DevOps? I think that's part of what the OP is getting at, if your fancy 'DevOps' team just ends up being separated and isolated from the dev team you're back at square one, you've accomplished nothing and the same divide exists with a new label. As someone who works on the dev side, the ops side, and the mixture of the two I feel the OPs pain acutely and have had mixed experiences. People on the dev side need to understand and willing to learn about ops, it's not rocket science and you don't have to be knee deep in it to get the basics down. When ops becomes this black box and becomes a bottleneck is when you start to see dysfunction.
- clavalle 12y ago
  >If 'DevOps' is operations, that what is the point of the term? It is taking more of a gestalt systems approach to the entire support structure. This means more intelligent support programs and treating it like designing and delivering a software project (or, more accurately a series of subprojects) as opposed to a loose amalgam of scripts. This also requires the product development team to be keenly aware of that support architecture and how their decisions factor into it especially in terms of testing, deployment, and module interactivity. > Ops teams have been doing automation of support systems for a while, so what exactly does it mean to be DevOps? DevOps has been around for a while -- way before there was a word for it. I think DevOps has come into its own, recently, though because platforms have become hugely more flexible while at the same time a consensus built on what it means to manage those systems. The difference, I think, is whether things are built to remain somewhat static or if change is designed for. Are development ideas like encapsulation and decoupling etc designed into the system? Smart sysadmins have been doing this stuff for a long time but now organizations are taking the approach into account all the way up and down the chain. And, yes, that means that product developers need to be fully aware of operational needs but, in my opinion, I don't think that means they build the operational tools themselves. >your fancy 'DevOps' team just ends up being separated and isolated from the dev team you're back at square one Demarcation of responsibility doesn't mean demarcation of information. The product development team will need to be intimately aware of what is happening in operations and vice versa. In a DevOps implementing environment it is unavoidable. > the dev side need to understand and willing to learn about ops, it's not rocket science These days it's getting pretty close. I don't think it is fair to product developers to say "Listen, we like the idea of what Docker can do for us so you are going to learn it inside and out." Somewhere you have to say "Hey, we probably want to implement Docker. Ops, dig in. You know how our product is currently structured and how operations currently works with that. Try to figure out what needs to change and what doesn't. X and Y from the dev team will be made available for deeper discussions and testing as needed. When the overall gist of what has to change is known, we'll all get together, ops and produce dev, to get everyone on the same page." It's a bleeding of information, not a full on 'shake it all up' scenario. Neither side can be a black box anymore if the current state of software development and delivery is going to be taken advantage of. But neither can everyone know and do everything or context switch as needed. Keep in mind: I am not a DevOps theorist -- maybe I have it completely wrong according to the people that came up with it but this is my practical understanding of the movement.

hedwall 12y ago

Obligatory lusis quote and link... "Devops means giving a shit about your job enough to not pass the buck. Devops means giving a shit about your job enough to want to learn all the parts and not just your little world. Developers need to understand infrastructure. Operations people need to understand code. People need to fucking work with each other and not just occupy space next to each other." http://blog.lusis.org/blog/2013/06/04/devops-the-title-match/ http://blog.lusis.org/blog/2013/06/04/devops-the-title-match...

sbt 12y ago

DevOps may be hyped, but it's not outright bullshit. DevOps is just the realization that you would rather have a fewer people knowing everything, than many people knowing disjoint areas. The same force underpins the full stack crowd, or the notion that pizza size teams are more efficient. While this is not always possible in practice, in my experience it is better if the project is small enough that you can pull it off. The memory bandwidth in one single person's brain is simply much higher than the bandwidth between people. To maximize performance you want to run your project on as few brains as possible.

mgrennan 12y ago

Former embedded developer, former systems developer, former language developer, former application developer, former senior systems administrator and now DBA. (30 years) Simply put, the problem IS TITLES. They cause the "not my job" problem. If you've been called, help fix the problem. If you don't understand the problem, stay with it, maybe you'll learn something. Just because your a "title" doesn't mean your s--t don't stink. Unless your are designing your CPUs, writing the languages that your write your OS and the applications that run your hardware, you are riding on the back of someone else. Do what's ask of you and ask how you can make everyone's job better. Then you'll be employed and paid well for a long time.

mrjatx 12y ago

I'm devops and in my opinion the title infers this; my job is to keep the communication between developers and the operations teams in sync and copacetic. Usually, Ops doesn't know dev and dev doesn't know ops so if there's nobody to translate and explain things in each others technical language things can't be done quickly and reliably. The wrong environments will be configured, things won't have backups, it will be like the wild west and things will be stressful for both sides. My job, just like it was when I was a sysadmin, is to keep things running smoothly. To optimize both sides of the coin and to take things off their plate that I myself can deal with that might be a blocker to them finishing tasks and to us getting features out or server migrations done. Some devs know ops and some ops know dev, those guys, more often than not I just check in with and see that things are running smoothly. For the other folk I try to create systems and workflows that get blockers out of their way. I try to be a useful and reliable tool to both sides as much as I can be. Hopefully the way I've been doing this for the last two years since I moved into a more Devops related role has been helpful to everyone.

pbhjpbhj 12y ago
Completely OT: "copacetic", new one for me - where have you come across this word; the etymology is intriguing to me. http://en.wiktionary.org/wiki/copacetic http://en.wiktionary.org/wiki/copacetic
- mrjatx 12y ago
  Boy that's interesting, I've never thought about it. I'm an American who grew up in Germany but have an Italian and German family who grew up in the projects of NY (the Love Canal, Niagara Falls). My grandmother (german) and grandfather (italian) married and their families constantly fought and pretty much hated one another. I remember them fighting (mostly arguing) and the men going "Ey ey, lets keep things copacetic." (They might have said capasetti) I always thought it was a pretty and a smart sounding word. I guess not.
  - pbhjpbhj 12y ago
    >pretty and a smart sounding word // Sounds good/smart to me! The wiktionary page mentions possible Italian-NY etymologies so you fit in there. Funnily enough I was going to try and be clever and guess you were a New Yorker [based on that] but noticed elsewhere you appear to be in TX. Interesting, thanks for responding.

wtbob 12y ago

From one of the comments: > It’s easier to train a developer to do sane operating procedures than to teach a sysadmin the developer mindset (testing, revision control, breaking down to logical units and similar). Having done both jobs, I don't really think that's true, unless by 'sysadmin' one means the type of low-skill position which resets passwords. A system administrator is responsible for administering a system of hosts and software; his should be a more complex job than a developers. He should be responsible for test systems as well as for production systems; his scripts should live in source control. I'll grant that functional decomposition may not be as vitally important to his work as it is to a normal developers—but then, his scripts will inevitably grow to be pretty complex, so maybe it will be after all. That doesn't mean that developers are stupid either. Sure, at BigEnterpriseCorp a developer is just a seat-warmer who implements directives like 'add a method to the Foo class which takes two arguments, adds them and returns the result,' but a real developer is a student (whether formally taught or informally self-taught) of computer science; he understands and can choose & implement both algorithms & data structures in order to solve a problem. A good DevOps hire is going to have some skill in both areas, not as a weak sysadmin and a weak dev, but as a strong sysadmin and a strong dev. A good DevOps team is going to be more skilled (and expensive) than the equivalent dev and ops teams, but it will also be more productive and less error-prone. If you're trying to build a devops team from Initech, Initron and Initrol low-skill employees, you're probably going to fail. If you're building a devops team to reduce costs, you're probably going to fail. But if you're building a devops team to increase profitability and competitiveness, you may succeed.

acveilleux 12y ago
In the Unix/Linux world, a lot of the Sys Admins I've met have been very strong technically. Knew at least a scripting language and basic C programming. The trend however has been for less and less coding skills and more superficial. Sometimes not even any shell scripting skills. In the Windows world, a lot of the Sys Admins do not have those skills. It's changing, things like PowerShell and various others are helping, but a lot of Windows Admin are just not that comfortable away from the GUI. I think the comment you refer to is by someone with a Windows dev background. GUIs don't version... The mainframe world folks tended to be very silo'd (and IBM engineers were never far to hold hands.)

dsr_ 12y ago

A senior systems administrator can and does write programs for automation, monitoring, configuration management, and all the little things that are not customer-facing applications. I used to be able to use "senior systems administrator" as a short-hand for those skill sets, but about 2010 that stopped working. Now I need to use "DevOps" as an imperfect substitute in the title; otherwise only junior people apply, even though the rest of the ad goes into the same depth about skills and abilities.

kokey 12y ago

I was recently looking for a new job, and from the job specs and interviews I have realised that 'DevOps' means very different things to many people. I am a system administrator who likes to automate things and write code, and also spent decades supporting developers and good Agile environments, etc. and production ops environments. Some companies think DevOps is a developer who knows how to deploy services on Ubuntu, and their interview tests the implementation of algorithms covered in recent CS courses. I removed any mention of DevOps from my CV, that made things less confusing for everyone.

serve_yay 12y ago

This piece is so confounding about what Devops is and isn't, so much that I'm ready to just flip the bozo bit on anyone who uses the term credulously. This is silly.

jjmojojjmojo 12y ago
Now you know how I feel.

tymski 12y ago

Whether you want to or not, it's your choice. However, the push towards DevOps as opposed to Dev is based on efficiency; it's easier to hire a developer that knows operations, instead of a developer and an operations employee. Even without DevOps, a lot of positions are Full Stack, so you need to know front and back-end development and technology. I don't think it's merely a trend, it's more or less a push towards saving money and time. If you don't want to be a full stack eng or a DevOps eng, then you're limiting yourself and you're limiting the opportunities you'll have. It sucks for those who want to limit themselves to one specific group of tasks instead of branching out, but it's just how it is.

cubano 12y ago

I think the most striking point made here was the one about culture and how without everyone buying into the DevOp paradigm, its just another unused tool. I say this because I've lived it...the last 3 (three!) companies I've contracted at were pushing dev code to live servers! Even after I setup svnserve and created the repos, no one was really interested in changing workflow. I choose svn because, lets face it, it has a much lower learning curve then git. It wasn't until I absolutely refused to work on the live server that other the devs grudgingly started using SVN...I'm sure they are all back to FTPing to the live server now that I'm gone. Without the proper culture from above, things just aren't going to change.

TranceMan 12y ago

Last month I went to the UK PHP Conference where Jeremy Quinton spoke about DevOps. The biggest take-away I got was: 'Do not set out to create a DevOps team or employ people for that role' My interpretation was the role simply doesn't exist. If your company has a team of developers and a team of people who keep the wheels turning - in an ideal world you already have a DevOps team as long as they collaborate. If you also employ smart people you will get 'developers that know how to do sysadmn' and 'sysadmins that know how to develop' - definitely this indeed should be embraced but not expected.

deleted 12y ago

[deleted]

waprin 12y ago

I think DevOps is a bit of a fad anyway, because a huge chunk of the problems people are solving will soon be solved by off-the-shelf standard cloud solutions. It makes no sense for everyone to be writing their own deployment/configuration Chef and Docker setups when they can just use increasingly powerful and flexible PaaS options like AppEngine/Heroku.

chrismarlow9 12y ago

I'm a dev who learned everything by experience. I'm in my 20's and have been doing this for ~20 years, so I have a bit of experience with both the new and the old world. My take is that the code quality of whats allowed to be in production has gotten worse, that's it. It's an effect of an abundance of RAM and virtualization. The abundance of ram lets you just trash it and do things in a horribly inefficient way. Sure the extra memory can help with time memory trade off if you have something that doesnt need to scale a great deal, but processing things inline will still be faster. Developers have forgotten how to handle streams. Usually forcing a dev to learn a highly functional language that's immutable can fix these issues (it must involve a large dataset, where you know they will tax out the memory, and then code review to help). I mention virtualization because a common pattern I've seen is to write poor code, and just put more boxes behind the load balancer when things start failing because the technical debt is so high it's an impossible mountain to climb. This obviously isn't the right approach and will leave your devops guys worn out chasing your everyday distributed systems problems that happen because it wasn't built right for distribution and the dev has never read the fallacies of distributed computing. And while you're fixing this technical debt a team of competent engineers builds a competing product and trumps you hard (second mover advantage). Maybe people paid a bit more attention when things cost 10x as much as they do now and every answer (both bad and good, and you can't tell the difference) wasn't a stackoverflow search away.....

Varcht 12y ago
Good point. Currently I am working on parsing a 5.5 million line STEP file. I've learned more about memory usage and high level overhead in the last week then I had in the previous 10 years of .Net dev.
aaron_m04 12y ago
So you're in your late 20s and started working when you were in elementary school? :-)

korzun 12y ago

I will sum this up: > We had edicts to make things happen, under the assumption that if we built it, the developers would embrace it. Followed by: > What matters is good people working together without artificial boundaries. Former will never produce latter. Always work with engineering before building anything that impacts their day to day.

stcredzero 12y ago

So that’s what DevOps really is. You take a bunch of really skilled, passionate, talented people who don’t have their heads shoved so far up their own asses that they can take the time to learn new things. What you want to look out for is the formation of "walls." People have an instinct to do this. People will form cliques -- which effectively are invisible barriers to understanding and honest collaboration. In a 5 person startup, this is easy. You have one group. Most of the critical information flows are inside this boundary, and those which aren't, the group is already intensely aware of. (Investors, customers.) However, when headcount has started to grow, and when you are forming a separate group to take on a task that requires intense collaboration, maybe you're doing the wrong thing? Maybe what you want is a separate role for people who get embedded in a variety of different groups? What if DevOps people belonged to DevOps, but also, and primarily, belonged with a particular group of developers? Then, they could have firsthand experience of intense collaboration, but also meet once a week with all the other DevOps role holders and communicate to standardize and avoid duplicated effort? A generalization of this: Make sure you're building bridges and not compartments.

TylerH 12y ago

tl;dr because waking up in the middle of the night to debug someone else's app really, really sucks.

adamconroy 12y ago

A smart, meticulous, better than other people person like myself can't get past the hubris, sorry.

devonkim 12y ago

I think the right thing to do is to firstly get rid of the term because it's become so meaningless and abused in so many ways. At least "cloud" has some semblance of at least meaning "whatever that we used to do in the 90s for IT is BAD" and implies great upheaval, but DevOps is a really, really weak movement in comparison because most companies that are dysfunctional try to contain whatever DevOps is into their existing (typically broken, so let's propagate that error 100x, right?) practices. While I agree that sysadmins should be developers first it is increasingly less likely you'll find developers willing to become computer janitors and deal with the ugly realities of operations in, for example, larger companies. I wouldn't think that John Carmack ever liked having to figure out the installer scripts and keep them working across all the supported platforms through all the BS that Microsoft and different Linux distributions could throw at you, yet that's what a more "ideal" DevOps organization described by many model would have John Carmack do. What should have happened though is that instead of hobbling the John Carmacks and dragging down their productivity, you should have fired the developers that refuse to cooperate with operations and those that keep actively putting off writing "boring" code or never documenting things that do matter for operations to figure out and learn through a failure that something broke. That kind of behavior is the developer equivalent of not being a team player at all. It's true that none of these things actually make you a better programmer in itself, but these things are what contribute to making you more PROFESSIONAL instead of a cowboy renegade programmer that doesn't care if nobody else can use his code because "clearly, if you can't figure it out quickly you don't deserve to use it" (not an exaggeration at all I've heard). Having been in both development and operations in rather large (and small!) companies now, I'd far, far, far rather stay as a hermit-crab developer than deal with the constant dysfunctions of enterprise Faustian nightmares that are required for you to deal with as an operations engineer. You can generally ignore a lot of the noise in a super-siloed off R&D organization, for example, but you cannot ignore your pager going off at 2 am because a random server crashed causing someone to get very, very angry and demand immediate resolution when it is your JOB to keep those things going. Hardly going to write anything really meaningful with that ruining your work week pretty much every week.

tomohawk 12y ago

If you want to make a devops omelet at a place that has an over the wall mentality, you have to break some eggs (and walls). I was part of a successful transition to devops from a thoroughly silo'd operation, where the boss basically gave the dev team the responsibility and authority to make it happen. He was also able to snag some of the traditional ops team and make them part of the new team. After a period of storming, the new team came up with some innovative ways to get the job done. Once the team realized that they were it in terms of getting it done, it got done. Quality and job satisfaction improved, cost was reduced. I've also seen various unsuccessful transitions. In every case, the responsibility was divided into various silos, which sandbagged everything.

DyslexicAtheist 12y ago

why devops isn't bullshit: http://blog.valbonne-consulting.com/2015/03/10/is-devops-bullshit/ http://blog.valbonne-consulting.com/2015/03/10/is-devops-bul...

parasubvert 12y ago

Devops is a professional movement to codify knowledge about how web companies operate at scale and speed. It gets misinterpreted sometimes (more lately as it becomes a fad). Doesn't mean its bullshit. Same problems with Agile, Lean, Six Sigma, or any other codification of how humans could or should work together to deliver a product. Many of the lessons learned here aren't exactly universal, for example some of the better Devops cultures (e.g Google) do have separate dev & ops teams, they just have good collaboration.