12 ms·

How 'DevOps' is Killing the Developer (2014)

52 points by zippy786 11y ago

mattkrea 11y ago
One coworker posted this a while back and I just dismissed it as someone afraid of something that they can't handle. While I understand the point.. I'm also a dev and I love "DevOps". Aren't we all doing this because we love problem solving? How is this not just another problem to tackle? I love having to figure how to automate mundane tasks (again.. this is just code.. not much different than we write elsewhere) to make sure I don't have to manage any systems manually. /rant
- zippy786 11y ago
  Don't you see tight coupling between two components/roles which may be bad ? Of course, for management it's less money they have to pay so they will always prefer a one man doing everything scenario.
  - mattkrea 11y ago
    I see that I can't trust what is the traditional IT department to understand how my application works so I strongly prefer shipping Docker containers or using something like Elastic Beanstalk if that is not available.
- nstart 11y ago
  It's actually a complex debate. We as developers naturally gravitate to a role like this because it's in our nature to believe that we can solve all the tech related problems. And all other problems, we believe can be solved with tech and therefore we believe we can solve everything. Ok. That's an exaggeration but there's quite a large shred of reality in that statement. Being fullstack/devops/<insert fancy term> satisfies that craving to jump on to new problems that we developers have. And that CAN be a problem. It isn't necessarily always a problem. But think about it. How many of us prefer jumping up to try working on a brand new project. Shiny stuff everywhere. And then everything gets abandoned at the grind and polish stage. The enthusiasm is gone. We crave the next new problem that we can use as an excuse for jumping away from the grind we are in. I think what the author is trying to get across is that this new "devops" culture is creating a large number of developers who don't have deep knowledge of anything. Ask a developer (myself included) to evaluate a set of JS frameworks for a particular project which has somewhat specific requirements. Behind most answers (this is based on observation, myself included again) the primary argument will be the developer's preference; not an objective breakdown of what is most suited for the task. We might not see too much of this on HN (I think. I see a lot of intelligent debates here which I learn from all the time) but on a larger scale developers are becoming half baked. And that's basically the other side of what you pointed out. Problem solving, learning new things is a lot of fun (discovering and configuring vagrant and docker for the first time was incredibly fun). But it possible that the full stack role is churning out people with broad but just somewhat in depth knowledge. Tough call. Question is, is somewhat deep deep enough for most everything?
  - eropple 11y ago
    > I think what the author is trying to get across is that this new "devops" culture is creating a large number of developers who don't have deep knowledge of anything. I dunno about that. My experience has been more that a team may need one person who's both wide and deep (T-shaped, or maybe just "rectangular"), but the companies I run across generally seem to hire younger developers who do have one or two core competencies--I-shaped, if not just dot-shaped and hoping to train them up.
  - mattkrea 11y ago
    I can agree with that. Though on the other end of things.. I have a friend that is very into how some languages are optimized for various tasks and he'll mention to me that X language does something with their loops for speed, etc. I don't have that same low level knowledge of the languages (not as low-level at least; I have some) but at the end of the day X language is slower than those that I prefer for the type of work we do (APIs mostly) so I don't care that X language does all these things internally.. and I don't need to know because it doesn't matter to us (nor our employer) in the end. So my--hate to use a buzz phrase here but it's appropriate--'get shit done' style tends to work for me and our company because evaluating tools comes down to "does a framework or tool accomplish what we need without too much overhead". Is this the type of deep knowledge you were talking about?
    - nstart 11y ago
      Yep. The question was, is the "I can analyze and tell you if I can use this tool to get work done" knowledge deep enough for most work? Or is it essential that if you are a front end dev to at least have an in depth knowledge to analyze, explain, and predict where a JS framework will shine over another and where it's pitfalls will lie. Knowing those pitfalls is also important because someday, it will bite. I remember not knowing about the pitfalls of python and default parameters being permanently set if given or passed a mutable argument. That bit me hard. I really don't know where the line lies honestly.
- serge2k 11y ago
  The problem I have with DevOps boils down to 2 things. 1. It's really just a dumb buzzword. 2. I really don't like companies dumping more work on devs for the same pay and using a buzzword to justify it. Oh here, you get a pager now because we're DEVOPS! Hire a god damn support staff. That said, letting people form little kingdoms and toss up walls so that people just end up getting bound up in bureaucracy is also terrible.
slowmovintarget 11y ago
Yarg! Teams do DevOps. It isn't supposed to be an excuse to make developers do everything. It's supposed to combine operations and development on the same team. This is just like the perversion of agility in development by "Agile." This, too, shall pass.
- phamilton 11y ago
  I recently was part of a team that properly practiced DevOps and as soon as it started working we had a reorg and it was broken apart. If the premise of the article is "Developer time is precious, don't make them do grunt work" the I would counter with "Developer momentum is precious, don't let developers get blocked by operations/QA/release." On our team, we had Ops and QA. But I needed something done and they were already working on something then I could have a quick 15 minute chat after standup about the Puppet change I needed to make and then go do it myself. An hour later I'm done, the change is deployed and I'm moving on to the next task. Now, without an embedded Ops resource on the team, I have to file a ticket and pray someone picks it up within a few days.
  - Turbots 11y ago
    I couldn't agree more! If you've read the Phoenix Project, it clearly states DevOps solves the problem of bottlenecks, which is the single most important aspect in the Product Development lifecycle. You just can't have developers waiting anymore for Ops to do something, or vice versa... Shit needs to go fast and blocking factors need to disappear, or else you sacrifice agility
  - lowmagnet 11y ago
    The way to not get "blocked by QA" is to have automated tests and QA people who can write them as well as dream up scenarios to break your toys. They are also able to do code reviews in your stack as well.
- bro-stick 11y ago
  "Full-stack" is a skillset for founders/first employees and hack-of-all-trades SREs. Later on in a now enterprise shop, deployments and development gets formalized, so it's crucial to have people/service providers whom know the nitty-gritty of the stack components really well... but it would be a distraction to expect a senior Postgres DBA to know client-side JavaScript. The overlap on the interface between sysadmining and development is deployment, where TwelveFactors-like shops can excel and reduce dev-ops turf wars. DevOps is about working together to make the overall service better; not shifting responsibilities around, getting people to take on more hats or renaming sysadmins. Disclaimer: I've been a (in no particular order): industrial embedded developer, sysadmin, engineering manager, founder, security researcher, consultant and SRE.
bradhe 11y ago
If you're a developer working on a service your primary objective is service delivery. It's really, really difficult to do that without understanding (intimately) all the components in play from top to bottom.
- atarian 11y ago
  Understanding the entire lifecycle of a service (from development to production) is a must for a developer. But it should not be the primary objective of the developer to deliver the service; developers should be working on development. When developers are in charge of delivering a service, they will end up spending most, if not all, of their time debugging operational issues instead of improving the service. This problem gets worse over time as the service will need to be deployed to more places to address scale. An organization with good technical leadership should know this.
- onion2k 11y ago
  It's really, really difficult to do that without understanding (intimately) all the components in play from top to bottom. I disagree. If a component is essentially a black box, where I know the inputs and I can test to check I get the expected output in known conditions, then that's enough. It has to be, otherwise there wouldn't be any way to use a middleware, or a third-party API, or anything too complex for one person to understand. There are lots of software businesses where no one fully understands the internals of the product. That's fine. So long as there are people who expertly understand the part they're responsible for, and the APIs are well documented, everything will work great. The problem with devops is that it gives people the impression that everyone can be responsible for everything, with everyone on the team having a hand in the design, the code, the testing, the deployment, etc. For a sufficiently complex product that is just not possible. At some point it gets too big, and people have to start accepting that they can't do it all, and that they'll have to trust their colleagues to get things right.
  - MichaelGG 11y ago
    This works great when it works great. Black boxes everywhere. But then you run into some issue, like, I dunno, your monitoring service is telling clients the site is going down, even though you can't find any evidence of this. If you know how to go deeper then you run a packet capture and start seeing that, whatdya know, your top-3 IaaS provider's network is shit and is often dropping packets, so those timeouts are actually occurring, though not on anything you control. Now you're able to try to setup a repro, then open a ticket with enough details to get something done. Versus opening a ticket along the lines of "Hi Top-3 IaaS provider. Some monitoring service says our site doesn't always respond, but I'm sure it's not me. Please fix your network, thanks." Not knowing how things work at a deeper level leads to experienced teams working on high-profile sites to come up with stuff like: "We have 2 domains in use, but need to go SSL only. We only have a certificate for one domain. So please implement a DNS-level redirect from domain A to domain B to avoid browser SSL errors." Or an experienced dev insisting that a major vendor is shipping invalid binaries because the runtime says so. And insists they aren't metadata-only libs, but isn't comfortable opening up the binaries to verify and thus spends a day trying to figure things out another way. Being able to get to a baseline of what's working is a fundamental step in troubleshooting. Just like you would always ask a home user to make sure stuff is plugged in, verifying lower-levels of the stack are correctly operating should be a common step - it's unfortunate how common it is that something else is broken under the level you're working at.
keyanp 11y ago
Interesting read, but I wish it wasn't so inflammatory. > "...release coordinators and the like are at the bottom of the totem pole. Why is it arranged like this? Because each role can do the job of all roles below it if necessary." I find this to be belittling those who choose to work on a different part of the stack than the author. It seems to suggest there is an innate incompetence that keeps release engineers from functioning as "developers". That couldn't be further from the truth. Release Engineering @ Mozilla: https://www.youtube.com/watch?v=7j0NDGJVROI https://www.youtube.com/watch?v=7j0NDGJVROI
- falsedan 11y ago
  Release Engineer here too. FUNNY STORY: we make the developers act as release coordinators, because it distracts us from the work of improving the release process by replacing it with automated software systems. The top of his hierarchy should be PMs/executive team; betrayed by his own developer ego…
  - serge2k 11y ago
    It's a hierarchy of skills. PM and exec skills are almost entirely orthogonal to tech skills. Putting them in this hierarchy makes zero sense as a result.
    - lowmagnet 11y ago
      Because understanding the business's and customers' needs is somehow "beneath" developers in skill or status? Sounds like some people need to try the "less technical" parts of the stack like requirements gathering to appreciate their roles.
      - zaccus 11y ago
        "Orthogonal" means independent from, not inferior to.
    - falsedan 11y ago
      I see you work for a PHB
hliyan 11y ago
There are two recent trends I really hate: DevOps and the notion of the "full-stack" developer I have to question the wisdom of bunching these two together. One is about freeing the developer as much as possible from peripheral duties so he can concentrate on his primary responsibility - design and development. The other, at least to me, is more about avoiding systems composed of modules that don't fit together properly, rather than getting more work done for less money.
- krzyk 11y ago
  Not sure about the DevOps, you are talking like if operations guy would start doing some dev stuff. But our company goes the other way, they make Devs do Ops stuff and QA stuff (and bussiness analytic stuff also). I really hate this.
  - hliyan 11y ago
    Perhaps my comment wasn't clear. I'm against making developers perform operations and QA tasks too. It's the author putting devops and full-stack engineers in the same basket that I question. That said, developer/unit testing is one aspect of QA you just can't offload to a separate team.
  - lowmagnet 11y ago
    Why hate that? The work is part of what is necessary to deliver software. It's like saying you want to be a general contractor on you own house but don't want to have architected plans, framers, or a final inspection.
    - krzyk 11y ago
      I don't want to be jack of all trades, I really don't have QA mindset and I don't like software testing as a black box, I prefer unit testing. All of developers on my team hate when they have to do QA stuff. It is like being hired to do plumbing, but being forced to do gardening :) It is all on the same property but completly different tasks and skillsets.
orthoganol 11y ago
The trend OP doesn't like is primarily because 'full stack' responsibilities, DBAs, release coordinators, etc. is probably the easiest it's ever been, so you might as well have developers throw in some extra hours to do it. I've worked at a fortune 500 that used a Ruby gem called "lol_dba"... you can spin up environments and deploys trivially with PAASs, or at the least use various automation tools, scripts, and cookbooks for provisioning machines, running deploys, CI, etc. OP seems to think "full stack" developers no longer devote time to writing code... "This is why we see so many developers that can't pass FizzBuzz"... but really it just doesn't take us long to do the non developer stuff because it's just easy these days, maybe he's not aware of the tools people use.
scurvy 11y ago
I've never met a developer that could setup a MPLS circuit or OSPF over an IPSEC tunnel. The author says that developers can do these things, but I highly doubt it. Let alone be proficient enough to do it in JunOS and IOS. Network operations should not be lumped in with devops.
- uniclaude 11y ago
  > The author says that developers can do these things. I couldn't find this in the article. You sound even more outraged and polarized than OP is. I've met more than one developer able to do this (hint, I can also do it). Some people are actually paid to write networking software, and they often have a clear understanding of how JunOS and IOS work. This said, for different reasons, I agree that network operations should not generally be done by your development team, but no one is arguing this anywhere.
  - scurvy 11y ago
    No I wasn't really outraged at all. My stance is that at some point you should stop trying to hire "full stack" developers. Chances are, they're not very good at everything and will miss some tiny detail that blows up big or creates a mountain of technical debt. Outside the founders and first three engineers, you shouldn't be hiring "full stack." You should start to specialize and start fixing the first five employee's problems. As the business grows, you're going to need some specialization skills for some project or deal that comes your way. You should hire the right person for that job. Chances are, you needed them all along and this situation is a good catalyst to get them in.
- amelius 11y ago
  I think the point of the article is that if developers were required to do this, they could do it.
  - bryanrasmussen 11y ago
    yeah I think I could figure out how to do it by reading a bunch of documentation, set up some scripts to do it, whenever I have to do it take longer than necessary until I had done it enough times that I had it as one of my remembered functions, and basically perform subpar on doing this thing. Which I guess is the author's point.
jlawer 11y ago
I am an "Infrastructure Engineer" by job title, but really I am a jack of all trades. Many people working in small business need to wear many hats. You can be deep on a single area (specialist) or be spread out (generalist). Larger businesses want specialists, small business want generalists. Personally I am a generalist. I deal with security, servers, routers, switches, applications, laptops, Operating Systems, Backup, Databases, etc. I write code in python, java and starting to get better with go. Its a handful and I am always juggling things, but I also think that broad knowledge helps things. I understand TCP/IP because I wrote a TCP/IP stack. I feel I have a better understanding of the complex systems as I have a non trivial understanding of each part. Its a very different working situation, but it has its advantages and I dread ever going back to a large corporate because of it. It isn't as efficient compared to having a team of specialists, where everyone can spend 100% on their speciality, but on the other hand, you can often do things with surprisingly small teams.
- einrealist 11y ago
  I share your opinion. In agile teams, it is important to do things that work. If a team owns a stack sucessfully, there is no need to change that. But if the environment changes (e.g. growth), the team may decide to split / specialise in order to ensure that the business goal is not in danger. There is no silver bullet.
endymi0n 11y ago
I always like the band analogy: The guitarists / developers always think they can play the bass guitar, because it's basically the same, just with less strings. Yet, they don't get that it's all about groove, rhythm and dependability and are just way too unrelaxed and attention seeking for doing it right. I'll take a grumpy BOFH that wires together my AWS regions in his sleep and doesn't make costly mistakes - because he measures twice and cuts once - over some kid trying DevOps any day :)
- harryf 11y ago
  I remember once haranguing a group of Java developers to use the log4j's RollingFileAppender. No matter what arguments I used and despite many nods and promises, on live systems they just kept on using the default FileAppender like there's no tomorrow, despite it being a relatively trivial config change to switch to the RollingFileAppender. Only the sysadmins "got it" because they were the ones that got to feel the pain of cleaning up a NAS filled up by log files.
  - glenjamin 11y ago
    Out of interest, why wouldn't you configure logrotate on the machine itself, rather than making this an application concern? Presumably logrotate was already set up for the variety of other logs on the system?
    - icebraining 11y ago
      I was curious too, though it seems log4j just keeps writing to the old log (it doesn't re-open the file on SIGHUP), and copytruncate is not completely safe from data loss.
  - deleted 11y ago
    [deleted]
  - einrealist 11y ago
    Did they own the systems their applications were deployed to? Because that is the essence of full stack development. Otherwise, config management is required as boundary context between developers and operations.
tomohawk 11y ago
Devops may be a lame buzzword, but most interesting problems cross traditional boundaries. As a dev becomes more senior and wants to become more valuable, they'll take it upon themselves to learn more about the other domains they interact with. The next step is they'll start trying to take on these cross domain problems that become apparent at this point. What I've seen is that when devs stick to straight coding, the products they work on tend to be less successful and their careers tend to be less than they should be.
kriro 11y ago
The article seems all over the place. For example I think the conclusion """This is why we see so many developers that can't pass FizzBuzz: they never really had to write any code.""" Doesn't really follow from the previous parts. DevOps is not the reason for this nor is the full stack developer trend. The author also seems to start from an implied base premise that developers wouldn't do devops/fullstack if they had a choice. I disagree with that. Many developers are very interested in understanding the implications of their code better (more DBA skills help a ton for example) and quite a few gasp actually enjoy devops or understanding the entire product from start to finish at least on some level (full stack). The benefits of DevOps also seem to be completely ignored. It's precisely the fact that common development best practices are integrated with sysadmin, dba etc. knowhow which makes it useful. It's not just "let the dev be a sysadmin in a pinch". It's systematically improve things by automating them (following development best practices instead of the hacked together bash script approach). DevOps and fullstack dev are conflated throughout the entire article in a strange way which makes it kind of hard to argue.
sargun 11y ago
Personally, I love DevOps. I've played on both sides of the fence. More ops than dev, I'll admit. The way I view DevOps is that you have end-to-end guarantees about software delivery. A lot of this draws from personal experience, so YMMV. Story: I was working in an organization where engineering had built a custom solution with their own database to a problem. It was using their own DB system, with the JVM, etc.. I was oncall for said service, because their DB kept getting into a poor state. My only real recourse to a page was (a) escalate to the devs (b) restart the service. I ended up furiously rewriting the service in Erlang over a weekend, and ended up getting told that my replacement was too complicated, and the previous one worked okay, because we could just restart it. Take-aways: Without feeling the pressure, and side-effects from being woken up, and having to respond to on-call incidents for a service, it makes people put more value on the development time than operational time, although a service is very likely to be in operational time much longer than development time. I have two other stories, but it'll take some time to write, and it's getting late, so I'll type it up only if someone is interested. Just so I remember - the tale of LD_PRELOAD and HDFS, as well as the networks and root access. 1. I think that ops staff should provide shared services to services organizations (hosting, deployment, etc..) when N services need the shared service (when N>2, depending on how many resources are available in your org). 2. I also think that ops staff should be able to "consult" on projects, as members of those teams during development. This is going to mean that those engineers are probably going to need to do some more development! Or, alternatively, the project team needs to be able to backpedal on some decisions if those ops consultants come in later on the project. 3. I think dev folks need to have end-to-end responsibility, including oncall, AARs, and on-call remediation. They need to take feedback from ops folks and act upon it, just as they would if product told them to do something.
myVocatio 11y ago
To spoof J L parker, I was "once a developer". Have played many roles since (PM,strategy consultant, VP, p&l owner,...). DevOp makes sense to me for building complex solutions..but, for different reasons than discussed here. Consider this: a small team (usually 3-7 members) of "jack of all trades" is proven across industries where the team has to deliver a complex solutions under a very aggressive timeline ,requiring multiple skills. Two examples from my personal experiences: Strategy consulting - eg @ McKinsey (work includes interviewing executives, presentation, excel, data entry, hypotheses development and verification....) investment banking eg @ Goldman Sachs (work includes industry analysis, financial analysis, scenarios, presentation, excel, data entry, ...) "Jack of all trade" model in terms of DevOps works because: 1) goal for the team w/ "full-stack" skills is to have a reliably working software solution 2) communication and management overhead increase significantly as the team size grows 3) complex solutions need different "amount" of various skills at different point in solution cycle 4) adding/ removing team members comes w/ significant cost to alignment

52 points by zippy786 11y ago

One coworker posted this a while back and I just dismissed it as someone afraid of something that they can't handle. While I understand the point.. I'm also a dev and I love "DevOps". Aren't we all doing this because we love problem solving? How is this not just another problem to tackle? I love having to figure how to automate mundane tasks (again.. this is just code.. not much different than we write elsewhere) to make sure I don't have to manage any systems manually. /rant

zippy786 11y ago
Don't you see tight coupling between two components/roles which may be bad ? Of course, for management it's less money they have to pay so they will always prefer a one man doing everything scenario.
- mattkrea 11y ago
  I see that I can't trust what is the traditional IT department to understand how my application works so I strongly prefer shipping Docker containers or using something like Elastic Beanstalk if that is not available.
nstart 11y ago
It's actually a complex debate. We as developers naturally gravitate to a role like this because it's in our nature to believe that we can solve all the tech related problems. And all other problems, we believe can be solved with tech and therefore we believe we can solve everything. Ok. That's an exaggeration but there's quite a large shred of reality in that statement. Being fullstack/devops/<insert fancy term> satisfies that craving to jump on to new problems that we developers have. And that CAN be a problem. It isn't necessarily always a problem. But think about it. How many of us prefer jumping up to try working on a brand new project. Shiny stuff everywhere. And then everything gets abandoned at the grind and polish stage. The enthusiasm is gone. We crave the next new problem that we can use as an excuse for jumping away from the grind we are in. I think what the author is trying to get across is that this new "devops" culture is creating a large number of developers who don't have deep knowledge of anything. Ask a developer (myself included) to evaluate a set of JS frameworks for a particular project which has somewhat specific requirements. Behind most answers (this is based on observation, myself included again) the primary argument will be the developer's preference; not an objective breakdown of what is most suited for the task. We might not see too much of this on HN (I think. I see a lot of intelligent debates here which I learn from all the time) but on a larger scale developers are becoming half baked. And that's basically the other side of what you pointed out. Problem solving, learning new things is a lot of fun (discovering and configuring vagrant and docker for the first time was incredibly fun). But it possible that the full stack role is churning out people with broad but just somewhat in depth knowledge. Tough call. Question is, is somewhat deep deep enough for most everything?
- eropple 11y ago
  > I think what the author is trying to get across is that this new "devops" culture is creating a large number of developers who don't have deep knowledge of anything. I dunno about that. My experience has been more that a team may need one person who's both wide and deep (T-shaped, or maybe just "rectangular"), but the companies I run across generally seem to hire younger developers who do have one or two core competencies--I-shaped, if not just dot-shaped and hoping to train them up.
- mattkrea 11y ago
  I can agree with that. Though on the other end of things.. I have a friend that is very into how some languages are optimized for various tasks and he'll mention to me that X language does something with their loops for speed, etc. I don't have that same low level knowledge of the languages (not as low-level at least; I have some) but at the end of the day X language is slower than those that I prefer for the type of work we do (APIs mostly) so I don't care that X language does all these things internally.. and I don't need to know because it doesn't matter to us (nor our employer) in the end. So my--hate to use a buzz phrase here but it's appropriate--'get shit done' style tends to work for me and our company because evaluating tools comes down to "does a framework or tool accomplish what we need without too much overhead". Is this the type of deep knowledge you were talking about?
  - nstart 11y ago
    Yep. The question was, is the "I can analyze and tell you if I can use this tool to get work done" knowledge deep enough for most work? Or is it essential that if you are a front end dev to at least have an in depth knowledge to analyze, explain, and predict where a JS framework will shine over another and where it's pitfalls will lie. Knowing those pitfalls is also important because someday, it will bite. I remember not knowing about the pitfalls of python and default parameters being permanently set if given or passed a mutable argument. That bit me hard. I really don't know where the line lies honestly.
serge2k 11y ago
The problem I have with DevOps boils down to 2 things. 1. It's really just a dumb buzzword. 2. I really don't like companies dumping more work on devs for the same pay and using a buzzword to justify it. Oh here, you get a pager now because we're DEVOPS! Hire a god damn support staff. That said, letting people form little kingdoms and toss up walls so that people just end up getting bound up in bureaucracy is also terrible.

slowmovintarget 11y ago

Yarg! Teams do DevOps. It isn't supposed to be an excuse to make developers do everything. It's supposed to combine operations and development on the same team. This is just like the perversion of agility in development by "Agile." This, too, shall pass.

phamilton 11y ago
I recently was part of a team that properly practiced DevOps and as soon as it started working we had a reorg and it was broken apart. If the premise of the article is "Developer time is precious, don't make them do grunt work" the I would counter with "Developer momentum is precious, don't let developers get blocked by operations/QA/release." On our team, we had Ops and QA. But I needed something done and they were already working on something then I could have a quick 15 minute chat after standup about the Puppet change I needed to make and then go do it myself. An hour later I'm done, the change is deployed and I'm moving on to the next task. Now, without an embedded Ops resource on the team, I have to file a ticket and pray someone picks it up within a few days.
- Turbots 11y ago
  I couldn't agree more! If you've read the Phoenix Project, it clearly states DevOps solves the problem of bottlenecks, which is the single most important aspect in the Product Development lifecycle. You just can't have developers waiting anymore for Ops to do something, or vice versa... Shit needs to go fast and blocking factors need to disappear, or else you sacrifice agility
- lowmagnet 11y ago
  The way to not get "blocked by QA" is to have automated tests and QA people who can write them as well as dream up scenarios to break your toys. They are also able to do code reviews in your stack as well.
bro-stick 11y ago
"Full-stack" is a skillset for founders/first employees and hack-of-all-trades SREs. Later on in a now enterprise shop, deployments and development gets formalized, so it's crucial to have people/service providers whom know the nitty-gritty of the stack components really well... but it would be a distraction to expect a senior Postgres DBA to know client-side JavaScript. The overlap on the interface between sysadmining and development is deployment, where TwelveFactors-like shops can excel and reduce dev-ops turf wars. DevOps is about working together to make the overall service better; not shifting responsibilities around, getting people to take on more hats or renaming sysadmins. Disclaimer: I've been a (in no particular order): industrial embedded developer, sysadmin, engineering manager, founder, security researcher, consultant and SRE.

bradhe 11y ago

If you're a developer working on a service your primary objective is service delivery. It's really, really difficult to do that without understanding (intimately) all the components in play from top to bottom.

atarian 11y ago
Understanding the entire lifecycle of a service (from development to production) is a must for a developer. But it should not be the primary objective of the developer to deliver the service; developers should be working on development. When developers are in charge of delivering a service, they will end up spending most, if not all, of their time debugging operational issues instead of improving the service. This problem gets worse over time as the service will need to be deployed to more places to address scale. An organization with good technical leadership should know this.
onion2k 11y ago
It's really, really difficult to do that without understanding (intimately) all the components in play from top to bottom. I disagree. If a component is essentially a black box, where I know the inputs and I can test to check I get the expected output in known conditions, then that's enough. It has to be, otherwise there wouldn't be any way to use a middleware, or a third-party API, or anything too complex for one person to understand. There are lots of software businesses where no one fully understands the internals of the product. That's fine. So long as there are people who expertly understand the part they're responsible for, and the APIs are well documented, everything will work great. The problem with devops is that it gives people the impression that everyone can be responsible for everything, with everyone on the team having a hand in the design, the code, the testing, the deployment, etc. For a sufficiently complex product that is just not possible. At some point it gets too big, and people have to start accepting that they can't do it all, and that they'll have to trust their colleagues to get things right.
- MichaelGG 11y ago
  This works great when it works great. Black boxes everywhere. But then you run into some issue, like, I dunno, your monitoring service is telling clients the site is going down, even though you can't find any evidence of this. If you know how to go deeper then you run a packet capture and start seeing that, whatdya know, your top-3 IaaS provider's network is shit and is often dropping packets, so those timeouts are actually occurring, though not on anything you control. Now you're able to try to setup a repro, then open a ticket with enough details to get something done. Versus opening a ticket along the lines of "Hi Top-3 IaaS provider. Some monitoring service says our site doesn't always respond, but I'm sure it's not me. Please fix your network, thanks." Not knowing how things work at a deeper level leads to experienced teams working on high-profile sites to come up with stuff like: "We have 2 domains in use, but need to go SSL only. We only have a certificate for one domain. So please implement a DNS-level redirect from domain A to domain B to avoid browser SSL errors." Or an experienced dev insisting that a major vendor is shipping invalid binaries because the runtime says so. And insists they aren't metadata-only libs, but isn't comfortable opening up the binaries to verify and thus spends a day trying to figure things out another way. Being able to get to a baseline of what's working is a fundamental step in troubleshooting. Just like you would always ask a home user to make sure stuff is plugged in, verifying lower-levels of the stack are correctly operating should be a common step - it's unfortunate how common it is that something else is broken under the level you're working at.

keyanp 11y ago

Interesting read, but I wish it wasn't so inflammatory. > "...release coordinators and the like are at the bottom of the totem pole. Why is it arranged like this? Because each role can do the job of all roles below it if necessary." I find this to be belittling those who choose to work on a different part of the stack than the author. It seems to suggest there is an innate incompetence that keeps release engineers from functioning as "developers". That couldn't be further from the truth. Release Engineering @ Mozilla: https://www.youtube.com/watch?v=7j0NDGJVROI https://www.youtube.com/watch?v=7j0NDGJVROI

falsedan 11y ago
Release Engineer here too. FUNNY STORY: we make the developers act as release coordinators, because it distracts us from the work of improving the release process by replacing it with automated software systems. The top of his hierarchy should be PMs/executive team; betrayed by his own developer ego…
- serge2k 11y ago
  It's a hierarchy of skills. PM and exec skills are almost entirely orthogonal to tech skills. Putting them in this hierarchy makes zero sense as a result.
  - lowmagnet 11y ago
    Because understanding the business's and customers' needs is somehow "beneath" developers in skill or status? Sounds like some people need to try the "less technical" parts of the stack like requirements gathering to appreciate their roles.
    - zaccus 11y ago
      "Orthogonal" means independent from, not inferior to.
  - falsedan 11y ago
    I see you work for a PHB

hliyan 11y ago

There are two recent trends I really hate: DevOps and the notion of the "full-stack" developer I have to question the wisdom of bunching these two together. One is about freeing the developer as much as possible from peripheral duties so he can concentrate on his primary responsibility - design and development. The other, at least to me, is more about avoiding systems composed of modules that don't fit together properly, rather than getting more work done for less money.

krzyk 11y ago
Not sure about the DevOps, you are talking like if operations guy would start doing some dev stuff. But our company goes the other way, they make Devs do Ops stuff and QA stuff (and bussiness analytic stuff also). I really hate this.
- hliyan 11y ago
  Perhaps my comment wasn't clear. I'm against making developers perform operations and QA tasks too. It's the author putting devops and full-stack engineers in the same basket that I question. That said, developer/unit testing is one aspect of QA you just can't offload to a separate team.
- lowmagnet 11y ago
  Why hate that? The work is part of what is necessary to deliver software. It's like saying you want to be a general contractor on you own house but don't want to have architected plans, framers, or a final inspection.
  - krzyk 11y ago
    I don't want to be jack of all trades, I really don't have QA mindset and I don't like software testing as a black box, I prefer unit testing. All of developers on my team hate when they have to do QA stuff. It is like being hired to do plumbing, but being forced to do gardening :) It is all on the same property but completly different tasks and skillsets.

orthoganol 11y ago

The trend OP doesn't like is primarily because 'full stack' responsibilities, DBAs, release coordinators, etc. is probably the easiest it's ever been, so you might as well have developers throw in some extra hours to do it. I've worked at a fortune 500 that used a Ruby gem called "lol_dba"... you can spin up environments and deploys trivially with PAASs, or at the least use various automation tools, scripts, and cookbooks for provisioning machines, running deploys, CI, etc. OP seems to think "full stack" developers no longer devote time to writing code... "This is why we see so many developers that can't pass FizzBuzz"... but really it just doesn't take us long to do the non developer stuff because it's just easy these days, maybe he's not aware of the tools people use.

scurvy 11y ago

I've never met a developer that could setup a MPLS circuit or OSPF over an IPSEC tunnel. The author says that developers can do these things, but I highly doubt it. Let alone be proficient enough to do it in JunOS and IOS. Network operations should not be lumped in with devops.

uniclaude 11y ago
> The author says that developers can do these things. I couldn't find this in the article. You sound even more outraged and polarized than OP is. I've met more than one developer able to do this (hint, I can also do it). Some people are actually paid to write networking software, and they often have a clear understanding of how JunOS and IOS work. This said, for different reasons, I agree that network operations should not generally be done by your development team, but no one is arguing this anywhere.
- scurvy 11y ago
  No I wasn't really outraged at all. My stance is that at some point you should stop trying to hire "full stack" developers. Chances are, they're not very good at everything and will miss some tiny detail that blows up big or creates a mountain of technical debt. Outside the founders and first three engineers, you shouldn't be hiring "full stack." You should start to specialize and start fixing the first five employee's problems. As the business grows, you're going to need some specialization skills for some project or deal that comes your way. You should hire the right person for that job. Chances are, you needed them all along and this situation is a good catalyst to get them in.
amelius 11y ago
I think the point of the article is that if developers were required to do this, they could do it.
- bryanrasmussen 11y ago
  yeah I think I could figure out how to do it by reading a bunch of documentation, set up some scripts to do it, whenever I have to do it take longer than necessary until I had done it enough times that I had it as one of my remembered functions, and basically perform subpar on doing this thing. Which I guess is the author's point.

jlawer 11y ago

I am an "Infrastructure Engineer" by job title, but really I am a jack of all trades. Many people working in small business need to wear many hats. You can be deep on a single area (specialist) or be spread out (generalist). Larger businesses want specialists, small business want generalists. Personally I am a generalist. I deal with security, servers, routers, switches, applications, laptops, Operating Systems, Backup, Databases, etc. I write code in python, java and starting to get better with go. Its a handful and I am always juggling things, but I also think that broad knowledge helps things. I understand TCP/IP because I wrote a TCP/IP stack. I feel I have a better understanding of the complex systems as I have a non trivial understanding of each part. Its a very different working situation, but it has its advantages and I dread ever going back to a large corporate because of it. It isn't as efficient compared to having a team of specialists, where everyone can spend 100% on their speciality, but on the other hand, you can often do things with surprisingly small teams.

einrealist 11y ago
I share your opinion. In agile teams, it is important to do things that work. If a team owns a stack sucessfully, there is no need to change that. But if the environment changes (e.g. growth), the team may decide to split / specialise in order to ensure that the business goal is not in danger. There is no silver bullet.

endymi0n 11y ago

I always like the band analogy: The guitarists / developers always think they can play the bass guitar, because it's basically the same, just with less strings. Yet, they don't get that it's all about groove, rhythm and dependability and are just way too unrelaxed and attention seeking for doing it right. I'll take a grumpy BOFH that wires together my AWS regions in his sleep and doesn't make costly mistakes - because he measures twice and cuts once - over some kid trying DevOps any day :)

harryf 11y ago
I remember once haranguing a group of Java developers to use the log4j's RollingFileAppender. No matter what arguments I used and despite many nods and promises, on live systems they just kept on using the default FileAppender like there's no tomorrow, despite it being a relatively trivial config change to switch to the RollingFileAppender. Only the sysadmins "got it" because they were the ones that got to feel the pain of cleaning up a NAS filled up by log files.
- glenjamin 11y ago
  Out of interest, why wouldn't you configure logrotate on the machine itself, rather than making this an application concern? Presumably logrotate was already set up for the variety of other logs on the system?
  - icebraining 11y ago
    I was curious too, though it seems log4j just keeps writing to the old log (it doesn't re-open the file on SIGHUP), and copytruncate is not completely safe from data loss.
- deleted 11y ago
  [deleted]
- einrealist 11y ago
  Did they own the systems their applications were deployed to? Because that is the essence of full stack development. Otherwise, config management is required as boundary context between developers and operations.

tomohawk 11y ago

Devops may be a lame buzzword, but most interesting problems cross traditional boundaries. As a dev becomes more senior and wants to become more valuable, they'll take it upon themselves to learn more about the other domains they interact with. The next step is they'll start trying to take on these cross domain problems that become apparent at this point. What I've seen is that when devs stick to straight coding, the products they work on tend to be less successful and their careers tend to be less than they should be.

kriro 11y ago

The article seems all over the place. For example I think the conclusion """This is why we see so many developers that can't pass FizzBuzz: they never really had to write any code.""" Doesn't really follow from the previous parts. DevOps is not the reason for this nor is the full stack developer trend. The author also seems to start from an implied base premise that developers wouldn't do devops/fullstack if they had a choice. I disagree with that. Many developers are very interested in understanding the implications of their code better (more DBA skills help a ton for example) and quite a few gasp actually enjoy devops or understanding the entire product from start to finish at least on some level (full stack). The benefits of DevOps also seem to be completely ignored. It's precisely the fact that common development best practices are integrated with sysadmin, dba etc. knowhow which makes it useful. It's not just "let the dev be a sysadmin in a pinch". It's systematically improve things by automating them (following development best practices instead of the hacked together bash script approach). DevOps and fullstack dev are conflated throughout the entire article in a strange way which makes it kind of hard to argue.

sargun 11y ago

Personally, I love DevOps. I've played on both sides of the fence. More ops than dev, I'll admit. The way I view DevOps is that you have end-to-end guarantees about software delivery. A lot of this draws from personal experience, so YMMV. Story: I was working in an organization where engineering had built a custom solution with their own database to a problem. It was using their own DB system, with the JVM, etc.. I was oncall for said service, because their DB kept getting into a poor state. My only real recourse to a page was (a) escalate to the devs (b) restart the service. I ended up furiously rewriting the service in Erlang over a weekend, and ended up getting told that my replacement was too complicated, and the previous one worked okay, because we could just restart it. Take-aways: Without feeling the pressure, and side-effects from being woken up, and having to respond to on-call incidents for a service, it makes people put more value on the development time than operational time, although a service is very likely to be in operational time much longer than development time. I have two other stories, but it'll take some time to write, and it's getting late, so I'll type it up only if someone is interested. Just so I remember - the tale of LD_PRELOAD and HDFS, as well as the networks and root access. 1. I think that ops staff should provide shared services to services organizations (hosting, deployment, etc..) when N services need the shared service (when N>2, depending on how many resources are available in your org). 2. I also think that ops staff should be able to "consult" on projects, as members of those teams during development. This is going to mean that those engineers are probably going to need to do some more development! Or, alternatively, the project team needs to be able to backpedal on some decisions if those ops consultants come in later on the project. 3. I think dev folks need to have end-to-end responsibility, including oncall, AARs, and on-call remediation. They need to take feedback from ops folks and act upon it, just as they would if product told them to do something.

myVocatio 11y ago

To spoof J L parker, I was "once a developer". Have played many roles since (PM,strategy consultant, VP, p&l owner,...). DevOp makes sense to me for building complex solutions..but, for different reasons than discussed here. Consider this: a small team (usually 3-7 members) of "jack of all trades" is proven across industries where the team has to deliver a complex solutions under a very aggressive timeline ,requiring multiple skills. Two examples from my personal experiences: Strategy consulting - eg @ McKinsey (work includes interviewing executives, presentation, excel, data entry, hypotheses development and verification....) investment banking eg @ Goldman Sachs (work includes industry analysis, financial analysis, scenarios, presentation, excel, data entry, ...) "Jack of all trade" model in terms of DevOps works because: 1) goal for the team w/ "full-stack" skills is to have a reliably working software solution 2) communication and management overhead increase significantly as the team size grows 3) complex solutions need different "amount" of various skills at different point in solution cycle 4) adding/ removing team members comes w/ significant cost to alignment