8 ms·

I’ve long held this opinion but I consistently get drowned out. DevOps has different meaning depending on who you’re talking to, even some definitions that app

by dijit 4y ago

I’ve long held this opinion but I consistently get drowned out. DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways. All “devops” as a job title has done has muddy responsibilities and given many folks the wrong impression of what an operations discipline should be. There is also a lot of rewriting of history that gets thrown in, similar to how when people talk about cloud then the only alternative is to start making CPUs by hand and begin building your own nuclear reactors. It’s the idea of what came before, not the reality, that people seem to be defensive of. It’s honestly exhausting to discuss. So instead I became CTO so I can solve this mess properly, I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers. Roles so simple that you already have a clue what they do, which is sort of the point of job titles.

deleted 4y ago
[deleted]
duxup 4y ago
Agreed, anytime I talk with someone about DevOps ... we end up having to hash out the entire process to really know what either of us are actually talking about. Otherwise you have these situations "Yea the DevOps guy messed up the widget and nobody notic---" "Wait, what is the DevOps guy doing even touching that widget.... what is even DevOps to you?" "Bro that widget IS DevOps." -silence- Same applies to the topic of "micro-services".
- boppo1 4y ago
  >microservices I never miss an opportunity to share my favorite piece of comedy this decade: https://youtu.be/y8OnoxKotPQ https://youtu.be/y8OnoxKotPQ
  - dijit 4y ago
    it's too close to reality. There is a good talk that is also comedic as it comes from the perspective of someone who wants to fail at doing microservices. https://www.youtube.com/watch?v=GWgRw5jiYy0 https://www.youtube.com/watch?v=GWgRw5jiYy0
    - smcleod 4y ago
      Thank was pretty funny, thanks.
- soperj 4y ago
  SOA is just Spaghetti Architecture. I haven't seen an implementation that convinces me otherwise.
  - duxup 4y ago
    I feel like it has its place in large organization. But being a large organization means the odds of spaghetti are approaching 100% no matter what you do.
  - jedberg 4y ago
    What is your definition of Spaghetti Architecture? Netflix had a good SOA that enabled rapid development and had strong cut lines between services, with no way to access the data of a service without going through the service's API. I think that's where most people go wrong. They put a bunch of services in front of a shared database, which means that they don't have to go through a service's API to get to it's data, and that's what breaks everything.
    - eropple 4y ago
      This is really important, and I've gotten a lot of quizzical looks when making this assertion over the years: data is owned by one and only one service. If two pieces of code assert ownership of it by mutating that data or looking past the public encapsulation of that data, then that code is the same service. If you see a queue between two services, that is usually an indication that some ownership is being transferred (even briefly), and that is a critical point where you need introspection and observation.
      - lamontcg 4y ago
        Yeah, services should be formed around the needs of data, and you shouldn't run multiple services in the same image/container just to split the code up. It shouldn't ever be about the code. Amazon's SOA architecture at least all started out about pulling bits of data spread over hundreds of servers into services that could take advantage of caching.
      - 4y ago
- dustymcp 4y ago
  this devops is also a dumping ground for anything else not happening on a developers computer which they should control but wont because "That is devops job" mostly this happens alot with node projects for some reason..
  - duxup 4y ago
    Humans are kinda terrible when they can just "not my problem" things.
  - chrisweekly 4y ago
    "anything else not happening on a developers computer which they should control but wont" Ok, I've seen this, but IME (24y in industry, the last 6 as a consultant) in the vast majority of cases, it's more like "things devs should control but CAN'T [bc CICD etc are silo'd and owned jealously by an overburdened ops team unable or unwilling to facilitate self-service]".
    - outworlder 4y ago
      Some of the 'jealousy' may also come from bad experiences. It only takes a few instances of people royally messing stuff up in production for their deployment rights to be stripped away – and other groups catch the fallout too, as new "procedures" get implemented.
      - dijit 4y ago
        "Safety rules are written in blood" applies to parts of the organisation too. Sure, there are some overbearing procedure monkeys who really want a process on everything, but a lot of "protection" rules are there because something really bad (and expensive, financially or reputationally) has happened before.
      - kazen44 4y ago
        Also, some developers do not seem to know what kind of impact a production issue can have. Having something not work during the development cycle is annoying for one person or a team. Having something break in production usually means people get paged and (depending on what you do) the resulting issues can have major societal aswell as economic impact. In the ops world, this is especially visible if you go further down the stack. (from applications down the the network). Network architecture moves at a toad's pace compared to webdevelopment, which is a good thing considering breaking the network will usually break every other system inside a IT department/landscape.
- sbf501 4y ago
  "Hey, did the front-end new-hire upgrade the Linux distro yet?"
  - duxup 4y ago
    Side rant: There was an angry Ask HN post a while back about how terrible the "new guys" are that are out there. It then complained that the new hire wrote a horrible authentication service. I thought it was an intentionally absurd post about the expectations put on new some rando new guy to write something important / they shouldn't be working alone on ... but they were serious.
deleted 4y ago
[deleted]
calvinmorrison 4y ago
DevOps to me is all about bridging what used to be a huge gap between operations and development. Developers need to know top 10 OWASP. One cannot simply say "it works locally". Ops people need to understand how your tools and programs work at a high level at minimum.
- Izmaki 4y ago
  These two talks will change your view on DevOps forever: - https://www.youtube.com/watch?v=zwSNjVTF168 https://www.youtube.com/watch?v=zwSNjVTF168 - https://www.youtube.com/watch?v=MnyvgFDh-kw https://www.youtube.com/watch?v=MnyvgFDh-kw
  - calvinmorrison 4y ago
    not if I don't watch it.
- dijit 4y ago
  Developers and Operations have been working together for a long time. Some teams historically worked fine together, others didn't. I think the tools we have these days are much better (for ops) to ship what works on developers machines. But if your organisation had a culture of not working together then devops didn't really do much. and anyway, you've leaned into my point about it meaning different things to different people.
Izmaki 4y ago
You hire those engineers and then you have them implement and support Continuous Delivery for your business, yes? DevOps is a silly term, but one that companies seem to use for "infrastructure engineers who help us continuously deliver value to our customers, efficiently and pain-free". Give your people whatever title you feel comfortable with - but essentially they are "doing the DevOps". :P
- Izmaki 4y ago
  Did I get downvoted because I'm right and the truth hurts or is there a disagreement somebody chose to not share? Hmm...
  - burnished 4y ago
    You basically assert that the other person is categorically wrong without argument but by rereredefining words. It doesn't read as productive to me.
NikolaNovak 4y ago
>>the only alternative is to start making CPUs by hand Agreed. For some applications the cloud difference is significant; for many (most?) others though, "Cloud" is just rebrand of "Hosted". And even for more cloudy offerings, while I'm in a very specific and different part of IBM, some of the old timers/architects/powers-that-be keep trying to explain "We had that in 1969!!!" :-D Agreed also at rewriting of history when it comes to development/support/operations models. My dad has been IT director and he chuckles when I talk to him about "new and exciting paradigms" which he of course sees as turning a circle to what they had in 70's and 80's :)
- softfalcon 4y ago
  The two of you have validated my very existence as a coder. I keep jokingly telling everyone that we’re often going around in circles. No one ever believes me (lol!)
- kennend3 4y ago
  > My dad has been IT director and he chuckles when I talk to him about "new and exciting paradigms" which he of course sees as turning a circle to what they had in 70's and 80's :) As someone with 20+ years in IT, I agree - a lot of these "new and exciting paradigms" are not new at all. My personal favourite is how many large multi-nationals are now building in-house clouds? WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's?
  - willcipriano 4y ago
    > WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's? A couple million in salaries and bonuses.
  - toast0 4y ago
    > WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's? An in-house cloud sounds like the mainframe installed in the raised floor computer room at the school district office I worked at in the late 90s; of course, the 12 foot long Unisys mainframe was replaced with a Unisys 4U pentium pro box pretending to be a mainframe, and then there was a lot of extra floor space. If you're running your own 'cloud' in a (shared) colo, I dunno that that's really in-house. I guess it's still 'private cloud' though.
mannykannot 4y ago
Agreed; I have always thought of it as a rationalization of bad practices.
brightball 4y ago
The extent of devops for your developers should be: 1. Push code to an automated pipeline 2. I understand that the automated pipeline may need input from me in order to run successfully. Beyond that, you need other people building the "platform" that they're deploying to (for the most part). Probably the ideal example here is Heroku. Your devs shouldn't be required to do all of the actual ops stuff. They need access to an ops platform.
- ghaff 4y ago
  To grossly oversimplify, there are essentially two views of DevOps. 1.) What's probably the traditional view which is breaking down the walls between devs and ops, developers carrying pagers, etc. I.e. at least in an idealized world, there are no devs and ops--only DevOps. 2.) As you suggest (and which probably more closely matches how "DevOps" works especially in larger organizations), an internal (or external) operations team provides a platform that developers can use. Developers are still going to be exposed to some operational details, but a lot of them are abstracted away.
  - Izkata 4y ago
    3) The way I first heard it described years ago, bringing developer practices into ops: version control, testing, code reviews, continuous integration, etc
    - kazen44 4y ago
      > bringing developer practices into ops: version control, testing, code reviews, continuous integration, etc In my opinion, this last subset just doesn't work well for a lot of branches of operations. Sure, it might work if you run stuff in AWS and have an operations team managing that, But what if your team is responsible for things like Storage Arrays, networking equipment? Doing continious integration and code review on some of these is hard if not impossible
- dilyevsky 4y ago
  Highly dependent on the product. Anything infrastructure related you’ll have a really bad time with that model
la64710 4y ago
I think everyone can agree that for most complicated things there are many sides to a story that makes up the present state of the system. I think taking a very opinionated stand in any complicated subject by itself negates the possibility of a mature and complete analysis of any subject. So the title itself of this article is unfortunate.
- kuramitropolis 4y ago
  Shouldn't a mature and complete analysis result in a conclusion, i.e. an opinionated stand backed up by correct reasoning?
  - la64710 4y ago
    Yes it should result in a conclusion but the conclusion should not simplify it in a way that the true nature of the issue is lost.
zerkten 4y ago
>> I became CTO so I can solve this mess properly, I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers. This makes a lot of sense, but you need to realize that this is only a tiny part of a successful organization. There are many setups like that which you describe across many industries that have failed reasons beyond the role definitions. It's not enough to establish the organization. You have to keep the behaviors in check over time when you hire a build engineer with different aspirations.
Melatonic 4y ago
I agree - I think it it is much better to use titles like these and with even a short JD describing what technology is used (many of which could be "devops" tools for infra engineers) everything becomes much more clear.
indymike 4y ago
> I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers. This is a great way to do it. There seems to be a correlation between unnecessary product complexity and unnecessary corporate complexity. Being direct about roles goes a long way towards simplifying corporate complexity.
- willcipriano 4y ago
  > There seems to be a correlation between unnecessary product complexity and unnecessary corporate complexity. That's Conway's law. "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." https://en.m.wikipedia.org/wiki/Conway%27s_law https://en.m.wikipedia.org/wiki/Conway%27s_law
  - zeruch 4y ago
    Conway's Law is sorely overlooked. It really does have broad applicability in the industry, and sometimes you can use it to evaluate vendors better (read: this product is a bit haphazard...the vendor probably is also. Warning.)
  - pfarrell 4y ago
    Totally agree. The best succinct summary of Conway’s Law I've heard is “you ship your org chart.”
    - hinkley 4y ago
      Which is a good way to word it because if your org chart is screwy enough you won't ship at all.
    - writeinpencil 4y ago
      I like "the org chart is the asymptote," i.e. it's the best you could possibly do, and your reality will actually be worse. (see Casey Muratori)
  - 4y ago
dogleash 4y ago
>DevOps has different meaning depending on who you’re talking to 1000% this. I think it's an instance of how the conversation in software always talks about solutions as if they're end-all-be-all answers, rather than explain a problem in specifics and why that made the solution the right answer at the time. Then that "solution" becomes a buzzword version of itself, and popular buzzwords are tools leaders use to overcome institutional inertia. Which is a good thing, to get over that inertia. But then what comes misses the insight and understanding that eventually turned into a buzzword.
ftlio 4y ago
> I hire infra engineers, build engineers, release engineers and: backend engineers. I always thought DevOps the “function” just meant this, and being a DevOps engineer at a small company meant you did these with decreasing emphasis, where by the time you’re in the backend it’s just helping enforce logging, tracing, other observable components. Has worked for me in hiring and being hired and almost everyone I know understands this.
edmcnulty101 4y ago
I suspect most places just use the term DevOps to try to get two jobs out of one person. Then they also tend to have to do their own security and spec out their own tickets and then do Project Management. Most tech people should be called DevSecOpsBizAnalystProjectManager.
jmull 4y ago
I'm convinced the use of the term "DevOps" continues only because of its incredible polymorphic vagueness... Two intelligent, sincere, experienced tech people can discuss devops, in detail, for a good length of time, and still be talking right past each other without really noticing.
- xen2xen1 4y ago
  It exists so people who dole out money can make one person do two jobs.
vlunkr 4y ago
DevOps is part of my job description. I was fuzzy on what it meant before, after reading this article, I have no idea what it means. As a general term, it's far too vague, when in practice it seems to just mean that you'll be splitting your time between writing regular application code and writing infrastructure as code. Oh and you get to be on-call.
- chasd00 4y ago
  i'm working on an RFP response that has the word "DevSecOps" sprinkled around here and there. That's an even more subjective and ambiguous term than devops :/
  - guhidalg 4y ago
    It means you're gonna be doing dev, ops, security, compliance, UX, UI, cost optimization, build, test, release, and on and on... good luck!
- deleted 4y ago
  [deleted]
otabdeveloper4 4y ago
"Sysadmin" got rebranded as "DevOps", because "techsupport" got rebranded as "sysadmin".
- deepstack 4y ago
  glad someone said it. I mean all the CI/CD pipeline are just in the old day *nix/bsd days bash/perl/awk/sed/python/ruby (fill in the blank) scripts. One problem when IT/Software development become more main stream is that everything got another layer of obfuscation especially in corporate cultures and amongst sales and recruiters and managements. Development Operation or pipeline sounds sooo much better bash scripts.
  - ardfard 4y ago
    Just bash script isn't enough if there's no automated process that run it every time developers make changes to your main/release repository trunk. If you still run it manually to verify your build, then it's not a CI. If you're still doing manual release deployment, then it's not a CD.
- notakio 4y ago
  Yerp. As a former *nix admin, around the time that "DevOps" became a term, I read it as "system administrator, who also now has to fix developer code." Which, honestly, was already part of the job at a lot of places I've worked, owed to the age old problem of people who "test" something on their weird personal desktop environment, then hand it off and shrug off any questions with "well, it works on my desktop". That said, the part that I found offensive was that I was doing 2 jobs, but only getting paid for one. Then came DevSecOps. Around that time, I switched to just Security, and while I miss the ability to make and push changes to hundreds of thousands of machines to get things done, I don't miss any of the pressure or blame that automatically got lumped onto the sysadmin shoulders every time anything went wrong, and the complete lack of appreciation of all the times nothing went wrong, that were entirely the result of a tireless, efficient systems administration team.
  - liketochill 4y ago
tetha 4y ago
Yep, this has been the hardest fight in our "culture shift towards devops" - getting the idea and wording right. After a lot of discussion, we mostly realized that we get the most value if we define more specific operational roles. We now have the idea of infra-ops, and product-ops. infra-ops is providing a deployment platform - a container runtime and persistences. product-ops on the other hand is responsible for deploying and running the different products from development on this deployment platform. And this is giving people good ideas. Some products are very simple without harsh requirements. In these cases, one of the backend devs just takes over the product-ops role by setting up a deployment pipeline, a job and some migration handling. Other products are bigger - for example we are providing some of our systems as essentially managed systems to invididual tenants and customers. In such a case, parts of the product operations is with the product team - such as writing jobs, releasing containers and artifacts, and other parts of the product-ops role takes place in the managed services consultancy. These products are currently looking at wordings to split up the product-ops role according to their needs in their context. And that's totally fine - the infra-ops role is also divided depending on the system the engineer is working on. A postgres admin is an infra-operator, for example. And all of this results in a devops oriented culture of cooperating across team boundaries through automated processes.
biggu 4y ago
You hire build engineers and release engineers seperately? Does your product demand that level of specialization for these 2 functions? Just curious
- dijit 4y ago
  I work on AAA video games, so unfortunately yes. It's ok to have build-engineer doing release engineering if it's simple enough, but doing build engineering properly (Source control, artifact control, shared caches, dependency management of the compiler outside of the standard toolchain) is quite a large job. Release engineering itself gets complicated when you take into consideration the different target platforms, most games get published on Playstation, Xbox and one of a handful of PC platforms (notably steam), plus the online systems and of course the development environments and internal release systems which are ubiquitous. Build Engineering usually is the step between developers and QA, and release engineering is the bit after QA. Both interact a lot with QA so could be rolled into the same role.
lumost 4y ago
I don't know, every time I see a title split people start saying that's "X's job", you can't do that because it's "y's job", or I don't want to be involved in "Z". Fewer titles can avoid these discussions.
- rovr138 4y ago
  Looks like you already identified the people that should be in those roles. If they're not doing it or was overlooked for some reason, then that's an issue but not really an argument to not do it.
  - lumost 4y ago
    Since when are managers incentivized to put people where their interests align? particularly when there is a perceived role transition.
- dijit 4y ago
  Sounds like a bad culture. That won't be fixed with a new job title.
cr4nberry 4y ago
> DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways. This is always a dead giveaway that something is a buzzword Same for rest. Sometimes when people use it, it just means JSON + http requests. Other times it's supposed to be some kind of architectural style
- guhidalg 4y ago
  You can always point people to the original paper that defined REST: https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arc...
  - 331c8c71 4y ago
    Should one also point people to the original vision of OOP (by Alan Kay) when they mention the term?
    - guhidalg 4y ago
      Yes, at least when there's disagreement about what OOP means. You and I can disagree about it, but we can more easily answer "Does it match what Alan Kay wrote about?" with a yes or no.
bgro 4y ago
I don't understand how seemingly 90% of developers, can't do anything outside of their narrow scope of experience. Writing a shell script? Throw a temper tantrum, that's devops job! Have to work with legacy code? I can't believe this! Especially seniors and beyond. They force leetcode interviews they somehow pass or are grandfathered through and gatekeep "trash devs" by slamming gotchas about how the whiteboarded code discussion doesn't technically compile and how "returnAverage()" isn't a method ("did you know you have to write methods before they work? What does returnAverage() supposedly even do? Return a random character in the alphabet?") I get excited about working in different tech areas. I'm exceptional at fixing other's bugs and maintaining code. Absolutely nobody seems to be hiring for this though. It's all about college exam trivia and leetcode. When I do get into these companies, I have to work with people throwing around casual racism with HR joining in and f-slurs like it's corporate 4chan. I ask if anyone can help look at a critical bug I discovered, and nobody speaks up. Even when it turns out it was their last code change that caused it and they are the sole master expert in this area, and they were just working on organizing their desktop instead. "I can try to look at the SQL problem, since nobody else spoke up." Then I'm explaining the basics of SQL to some guy sitting around blank faced and it turns out he was hired because of his 10+ years of expertise in SQL. His whole job is to tackle problems like that. Meanwhile I'm out here looking for jobs when panic cuts happen and it takes forever because I'm drained from all the gotchas and gatekeeping in these interviews. There's no way I could live with myself if I put this experience onto others. I lose sleep when I fail to call out someone talking over a quiet person on my team. I've never seen anybody else stand up for anybody, however.
- apalumbi 4y ago
  I like to build up my teams with people that act and think like you. Sometimes I need a sniper and when I do I'll get one. But my teams need to be excited about building and running the systems that we make.
- hinkley 4y ago
  I learned SQL because my coworker was stuck on a bug she couldn't figure out and 'knew' I'd be able to solve her problem. I just kept asking questions until I found the one she missed. I also ended up becoming a bespoke VxWorks admin because the guy who volunteered was never available, and someone asserted my code wasn't working well on VxWorks, so of course I had to know enough to do benchmarks. I fixed a few problems but the real issue was the hard drive wasn't doing DMA due to the kernel not recognizing the processor revision number. Somewhere between those two events I realized that if my part of the project is great but the whole project is on fire, nobody cares about my stuff. I don't get points for being right and the team being wrong. I mean, I do for some people, but I still feel bad at the end of the project, and those are the 'points' I have to live with the most. The way that played on on the latter project is that once we got our shit together, we started work stealing from our peer orgs. Volunteering to carve off little pieces of interface between us and 'take care' of this bit of data handling here and that one over there. I came to realize that the Org subconsciously knew this and any past success they had was due to self-organization and collective work-stealing. Throw people at it, turn a blind eye to precise mission statements, and hope for the best.
usrusr 4y ago
Do those different breeds of engineers work in one team or in separate organizational branches? The problem the term 'devops' is addressing, the way I understand it, isn't that of insufficient individual jack-of-tradesness, but that of too much separation between (internal) organisations. It should be perfectly possible to build a "devops" team from deeply specialized experts and turning it into a job description seems quite a stretch to me. But you might actually want some jack-of-trades types nonetheless, because those isolated organizations weren't completely without merit: they are good at solving that problem affectionately called the bus factor. A single ops guy in a devops team is a single point of failure and the mitigation preparation of keeping some of the not-so-ops peers sufficiently in the loop will be much more dependent on organic motivation than the counterpart in a specialist company branch. There, substitutability would be much easier to ensure with formal process.
- dijit 4y ago
  > Do those different breeds of engineers work in one team or in separate organizational branches? It doesn’t matter as long as communication is open. Proper ops is all about reducing the bus factor. Procedures don’t depend on people, they depend on being good at communicating. If your ops person got hit by a car, I would expect that everything would continue running, I would also expect to find some clear documentation, this is mandatory.
xtracto 4y ago
Some time ago, i was talking to an MBA people /project manager guy was ho had a completely different definition of what "devops" meant. His meaning was some kind of project management approach. We have a way to prostitute words in this field... agile, QA, devops.
llbeansandrice 4y ago
Are you hiring? I was hired as an infrastructure engineer and I'm currently writing typescript for the frontend of an internal product.
- dijit 4y ago
  Yes, I am, the project is mostly in stealth mode but please feel free to drop me a CV: jan [at] competition [dot] company If you want more information on the project we're working on the site is https://rennsport.gg https://rennsport.gg We're building a hard-core racing simulator game with a backend which can persist car ownership in a way that feels authentic. (IE; not just tied to a game).
chrismarlow9 4y ago
This is the answer. I am interviewing right now and my previous experience includes devops roles and SRE roles, so I get contacted for both by recruiters. After hearing about the responsibilities and examples for these roles I can only come to the conclusion that titles are a waste of time. Even the "level" of a title of Staff, Principal, Lead, etc are a waste of time. Just call the job what it is and if you can't decide on what it is at least to an 80% level, maybe you're asking too much for that job?
kodah 4y ago
I get called an "Infrastructure Engineer" sometimes, I've also had the title of SRE and SWE. I don't really feel like any of these actually fit me. I do work on Cloud Infrastructure at times, but honestly it's the smallest part of what I do. It's usually addressed in the architectural designs and I have to touch it incrementally. What I do much more often is writing tools, daemons, and services for distributed systems. I end up calling myself a Distributed Systems Software Engineer to reflect the idea that what I work on is systems and software, and most of them are non-monoliths (from a systems perspective). Do you have any thoughts on Systems Engineers or the title that I prefer to call myself?
- dijit 4y ago
  In the games industry we have a role called “tools programmer” which is a person who works on tooling to accelerate development, sometimes these can be extensions to the editor for a specialisation for a game (say, a tyre builder for a racing game) and sometimes they can be more broad (a distribution tool using libtorrent so that a new build of the game can be shipped worldwide very quickly). There are people who might refer to you as a backend programmer (if the focus is services and daemons), or platform engineer (if the focus is developer velocity). Largely it depends what your primary focus is.
doctor_eval 4y ago
I agree with this but I’d add that it’s also important that the roles aren’t overly siloed. A front end engineer, for example, should be able to and even encouraged to look at backend code, and even submit PRs for it. And more importantly, the backend engineers should be encouraged to be grateful for the PRs (even if it’s not acceptable for some reason). The more eyes on the code, the better. I think developers naturally tend to specialise (at least for a time) but ultimately we all need to understand and contribute to code regardless if it’s front end, backend, build, test, …

by dijit 4y ago

deleted 4y ago

[deleted]

duxup 4y ago

Agreed, anytime I talk with someone about DevOps ... we end up having to hash out the entire process to really know what either of us are actually talking about. Otherwise you have these situations "Yea the DevOps guy messed up the widget and nobody notic---" "Wait, what is the DevOps guy doing even touching that widget.... what is even DevOps to you?" "Bro that widget IS DevOps." -silence- Same applies to the topic of "micro-services".

boppo1 4y ago
>microservices I never miss an opportunity to share my favorite piece of comedy this decade: https://youtu.be/y8OnoxKotPQ https://youtu.be/y8OnoxKotPQ
- dijit 4y ago
  it's too close to reality. There is a good talk that is also comedic as it comes from the perspective of someone who wants to fail at doing microservices. https://www.youtube.com/watch?v=GWgRw5jiYy0 https://www.youtube.com/watch?v=GWgRw5jiYy0
  - smcleod 4y ago
    Thank was pretty funny, thanks.
soperj 4y ago
SOA is just Spaghetti Architecture. I haven't seen an implementation that convinces me otherwise.
- duxup 4y ago
  I feel like it has its place in large organization. But being a large organization means the odds of spaghetti are approaching 100% no matter what you do.
- jedberg 4y ago
  What is your definition of Spaghetti Architecture? Netflix had a good SOA that enabled rapid development and had strong cut lines between services, with no way to access the data of a service without going through the service's API. I think that's where most people go wrong. They put a bunch of services in front of a shared database, which means that they don't have to go through a service's API to get to it's data, and that's what breaks everything.
  - eropple 4y ago
    This is really important, and I've gotten a lot of quizzical looks when making this assertion over the years: data is owned by one and only one service. If two pieces of code assert ownership of it by mutating that data or looking past the public encapsulation of that data, then that code is the same service. If you see a queue between two services, that is usually an indication that some ownership is being transferred (even briefly), and that is a critical point where you need introspection and observation.
    - lamontcg 4y ago
      Yeah, services should be formed around the needs of data, and you shouldn't run multiple services in the same image/container just to split the code up. It shouldn't ever be about the code. Amazon's SOA architecture at least all started out about pulling bits of data spread over hundreds of servers into services that could take advantage of caching.
    - 4y ago
dustymcp 4y ago
this devops is also a dumping ground for anything else not happening on a developers computer which they should control but wont because "That is devops job" mostly this happens alot with node projects for some reason..
- duxup 4y ago
  Humans are kinda terrible when they can just "not my problem" things.
- chrisweekly 4y ago
  "anything else not happening on a developers computer which they should control but wont" Ok, I've seen this, but IME (24y in industry, the last 6 as a consultant) in the vast majority of cases, it's more like "things devs should control but CAN'T [bc CICD etc are silo'd and owned jealously by an overburdened ops team unable or unwilling to facilitate self-service]".
  - outworlder 4y ago
    Some of the 'jealousy' may also come from bad experiences. It only takes a few instances of people royally messing stuff up in production for their deployment rights to be stripped away – and other groups catch the fallout too, as new "procedures" get implemented.
    - dijit 4y ago
      "Safety rules are written in blood" applies to parts of the organisation too. Sure, there are some overbearing procedure monkeys who really want a process on everything, but a lot of "protection" rules are there because something really bad (and expensive, financially or reputationally) has happened before.
    - kazen44 4y ago
      Also, some developers do not seem to know what kind of impact a production issue can have. Having something not work during the development cycle is annoying for one person or a team. Having something break in production usually means people get paged and (depending on what you do) the resulting issues can have major societal aswell as economic impact. In the ops world, this is especially visible if you go further down the stack. (from applications down the the network). Network architecture moves at a toad's pace compared to webdevelopment, which is a good thing considering breaking the network will usually break every other system inside a IT department/landscape.
sbf501 4y ago
"Hey, did the front-end new-hire upgrade the Linux distro yet?"
- duxup 4y ago
  Side rant: There was an angry Ask HN post a while back about how terrible the "new guys" are that are out there. It then complained that the new hire wrote a horrible authentication service. I thought it was an intentionally absurd post about the expectations put on new some rando new guy to write something important / they shouldn't be working alone on ... but they were serious.

deleted 4y ago

[deleted]

calvinmorrison 4y ago

DevOps to me is all about bridging what used to be a huge gap between operations and development. Developers need to know top 10 OWASP. One cannot simply say "it works locally". Ops people need to understand how your tools and programs work at a high level at minimum.

Izmaki 4y ago
These two talks will change your view on DevOps forever: - https://www.youtube.com/watch?v=zwSNjVTF168 https://www.youtube.com/watch?v=zwSNjVTF168 - https://www.youtube.com/watch?v=MnyvgFDh-kw https://www.youtube.com/watch?v=MnyvgFDh-kw
- calvinmorrison 4y ago
  not if I don't watch it.
dijit 4y ago
Developers and Operations have been working together for a long time. Some teams historically worked fine together, others didn't. I think the tools we have these days are much better (for ops) to ship what works on developers machines. But if your organisation had a culture of not working together then devops didn't really do much. and anyway, you've leaned into my point about it meaning different things to different people.

Izmaki 4y ago

You hire those engineers and then you have them implement and support Continuous Delivery for your business, yes? DevOps is a silly term, but one that companies seem to use for "infrastructure engineers who help us continuously deliver value to our customers, efficiently and pain-free". Give your people whatever title you feel comfortable with - but essentially they are "doing the DevOps". :P

Izmaki 4y ago
Did I get downvoted because I'm right and the truth hurts or is there a disagreement somebody chose to not share? Hmm...
- burnished 4y ago
  You basically assert that the other person is categorically wrong without argument but by rereredefining words. It doesn't read as productive to me.

NikolaNovak 4y ago

>>the only alternative is to start making CPUs by hand Agreed. For some applications the cloud difference is significant; for many (most?) others though, "Cloud" is just rebrand of "Hosted". And even for more cloudy offerings, while I'm in a very specific and different part of IBM, some of the old timers/architects/powers-that-be keep trying to explain "We had that in 1969!!!" :-D Agreed also at rewriting of history when it comes to development/support/operations models. My dad has been IT director and he chuckles when I talk to him about "new and exciting paradigms" which he of course sees as turning a circle to what they had in 70's and 80's :)

softfalcon 4y ago
The two of you have validated my very existence as a coder. I keep jokingly telling everyone that we’re often going around in circles. No one ever believes me (lol!)
kennend3 4y ago
> My dad has been IT director and he chuckles when I talk to him about "new and exciting paradigms" which he of course sees as turning a circle to what they had in 70's and 80's :) As someone with 20+ years in IT, I agree - a lot of these "new and exciting paradigms" are not new at all. My personal favourite is how many large multi-nationals are now building in-house clouds? WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's?
- willcipriano 4y ago
  > WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's? A couple million in salaries and bonuses.
- toast0 4y ago
  > WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's? An in-house cloud sounds like the mainframe installed in the raised floor computer room at the school district office I worked at in the late 90s; of course, the 12 foot long Unisys mainframe was replaced with a Unisys 4U pentium pro box pretending to be a mainframe, and then there was a lot of extra floor space. If you're running your own 'cloud' in a (shared) colo, I dunno that that's really in-house. I guess it's still 'private cloud' though.

mannykannot 4y ago

Agreed; I have always thought of it as a rationalization of bad practices.

brightball 4y ago

The extent of devops for your developers should be: 1. Push code to an automated pipeline 2. I understand that the automated pipeline may need input from me in order to run successfully. Beyond that, you need other people building the "platform" that they're deploying to (for the most part). Probably the ideal example here is Heroku. Your devs shouldn't be required to do all of the actual ops stuff. They need access to an ops platform.

ghaff 4y ago
To grossly oversimplify, there are essentially two views of DevOps. 1.) What's probably the traditional view which is breaking down the walls between devs and ops, developers carrying pagers, etc. I.e. at least in an idealized world, there are no devs and ops--only DevOps. 2.) As you suggest (and which probably more closely matches how "DevOps" works especially in larger organizations), an internal (or external) operations team provides a platform that developers can use. Developers are still going to be exposed to some operational details, but a lot of them are abstracted away.
- Izkata 4y ago
  3) The way I first heard it described years ago, bringing developer practices into ops: version control, testing, code reviews, continuous integration, etc
  - kazen44 4y ago
    > bringing developer practices into ops: version control, testing, code reviews, continuous integration, etc In my opinion, this last subset just doesn't work well for a lot of branches of operations. Sure, it might work if you run stuff in AWS and have an operations team managing that, But what if your team is responsible for things like Storage Arrays, networking equipment? Doing continious integration and code review on some of these is hard if not impossible
dilyevsky 4y ago
Highly dependent on the product. Anything infrastructure related you’ll have a really bad time with that model

la64710 4y ago

I think everyone can agree that for most complicated things there are many sides to a story that makes up the present state of the system. I think taking a very opinionated stand in any complicated subject by itself negates the possibility of a mature and complete analysis of any subject. So the title itself of this article is unfortunate.

kuramitropolis 4y ago
Shouldn't a mature and complete analysis result in a conclusion, i.e. an opinionated stand backed up by correct reasoning?
- la64710 4y ago
  Yes it should result in a conclusion but the conclusion should not simplify it in a way that the true nature of the issue is lost.

zerkten 4y ago

>> I became CTO so I can solve this mess properly, I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers. This makes a lot of sense, but you need to realize that this is only a tiny part of a successful organization. There are many setups like that which you describe across many industries that have failed reasons beyond the role definitions. It's not enough to establish the organization. You have to keep the behaviors in check over time when you hire a build engineer with different aspirations.

Melatonic 4y ago

I agree - I think it it is much better to use titles like these and with even a short JD describing what technology is used (many of which could be "devops" tools for infra engineers) everything becomes much more clear.

indymike 4y ago

> I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers. This is a great way to do it. There seems to be a correlation between unnecessary product complexity and unnecessary corporate complexity. Being direct about roles goes a long way towards simplifying corporate complexity.

willcipriano 4y ago
> There seems to be a correlation between unnecessary product complexity and unnecessary corporate complexity. That's Conway's law. "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." https://en.m.wikipedia.org/wiki/Conway%27s_law https://en.m.wikipedia.org/wiki/Conway%27s_law
- zeruch 4y ago
  Conway's Law is sorely overlooked. It really does have broad applicability in the industry, and sometimes you can use it to evaluate vendors better (read: this product is a bit haphazard...the vendor probably is also. Warning.)
- pfarrell 4y ago
  Totally agree. The best succinct summary of Conway’s Law I've heard is “you ship your org chart.”
  - hinkley 4y ago
    Which is a good way to word it because if your org chart is screwy enough you won't ship at all.
  - writeinpencil 4y ago
    I like "the org chart is the asymptote," i.e. it's the best you could possibly do, and your reality will actually be worse. (see Casey Muratori)
- 4y ago

dogleash 4y ago

>DevOps has different meaning depending on who you’re talking to 1000% this. I think it's an instance of how the conversation in software always talks about solutions as if they're end-all-be-all answers, rather than explain a problem in specifics and why that made the solution the right answer at the time. Then that "solution" becomes a buzzword version of itself, and popular buzzwords are tools leaders use to overcome institutional inertia. Which is a good thing, to get over that inertia. But then what comes misses the insight and understanding that eventually turned into a buzzword.

ftlio 4y ago

> I hire infra engineers, build engineers, release engineers and: backend engineers. I always thought DevOps the “function” just meant this, and being a DevOps engineer at a small company meant you did these with decreasing emphasis, where by the time you’re in the backend it’s just helping enforce logging, tracing, other observable components. Has worked for me in hiring and being hired and almost everyone I know understands this.

edmcnulty101 4y ago

I suspect most places just use the term DevOps to try to get two jobs out of one person. Then they also tend to have to do their own security and spec out their own tickets and then do Project Management. Most tech people should be called DevSecOpsBizAnalystProjectManager.

jmull 4y ago

I'm convinced the use of the term "DevOps" continues only because of its incredible polymorphic vagueness... Two intelligent, sincere, experienced tech people can discuss devops, in detail, for a good length of time, and still be talking right past each other without really noticing.

xen2xen1 4y ago
It exists so people who dole out money can make one person do two jobs.

vlunkr 4y ago

DevOps is part of my job description. I was fuzzy on what it meant before, after reading this article, I have no idea what it means. As a general term, it's far too vague, when in practice it seems to just mean that you'll be splitting your time between writing regular application code and writing infrastructure as code. Oh and you get to be on-call.

chasd00 4y ago
i'm working on an RFP response that has the word "DevSecOps" sprinkled around here and there. That's an even more subjective and ambiguous term than devops :/
- guhidalg 4y ago
  It means you're gonna be doing dev, ops, security, compliance, UX, UI, cost optimization, build, test, release, and on and on... good luck!
deleted 4y ago
[deleted]

otabdeveloper4 4y ago

"Sysadmin" got rebranded as "DevOps", because "techsupport" got rebranded as "sysadmin".

deepstack 4y ago
glad someone said it. I mean all the CI/CD pipeline are just in the old day *nix/bsd days bash/perl/awk/sed/python/ruby (fill in the blank) scripts. One problem when IT/Software development become more main stream is that everything got another layer of obfuscation especially in corporate cultures and amongst sales and recruiters and managements. Development Operation or pipeline sounds sooo much better bash scripts.
- ardfard 4y ago
  Just bash script isn't enough if there's no automated process that run it every time developers make changes to your main/release repository trunk. If you still run it manually to verify your build, then it's not a CI. If you're still doing manual release deployment, then it's not a CD.
notakio 4y ago
Yerp. As a former *nix admin, around the time that "DevOps" became a term, I read it as "system administrator, who also now has to fix developer code." Which, honestly, was already part of the job at a lot of places I've worked, owed to the age old problem of people who "test" something on their weird personal desktop environment, then hand it off and shrug off any questions with "well, it works on my desktop". That said, the part that I found offensive was that I was doing 2 jobs, but only getting paid for one. Then came DevSecOps. Around that time, I switched to just Security, and while I miss the ability to make and push changes to hundreds of thousands of machines to get things done, I don't miss any of the pressure or blame that automatically got lumped onto the sysadmin shoulders every time anything went wrong, and the complete lack of appreciation of all the times nothing went wrong, that were entirely the result of a tireless, efficient systems administration team.
- liketochill 4y ago

tetha 4y ago

Yep, this has been the hardest fight in our "culture shift towards devops" - getting the idea and wording right. After a lot of discussion, we mostly realized that we get the most value if we define more specific operational roles. We now have the idea of infra-ops, and product-ops. infra-ops is providing a deployment platform - a container runtime and persistences. product-ops on the other hand is responsible for deploying and running the different products from development on this deployment platform. And this is giving people good ideas. Some products are very simple without harsh requirements. In these cases, one of the backend devs just takes over the product-ops role by setting up a deployment pipeline, a job and some migration handling. Other products are bigger - for example we are providing some of our systems as essentially managed systems to invididual tenants and customers. In such a case, parts of the product operations is with the product team - such as writing jobs, releasing containers and artifacts, and other parts of the product-ops role takes place in the managed services consultancy. These products are currently looking at wordings to split up the product-ops role according to their needs in their context. And that's totally fine - the infra-ops role is also divided depending on the system the engineer is working on. A postgres admin is an infra-operator, for example. And all of this results in a devops oriented culture of cooperating across team boundaries through automated processes.

biggu 4y ago

You hire build engineers and release engineers seperately? Does your product demand that level of specialization for these 2 functions? Just curious

dijit 4y ago
I work on AAA video games, so unfortunately yes. It's ok to have build-engineer doing release engineering if it's simple enough, but doing build engineering properly (Source control, artifact control, shared caches, dependency management of the compiler outside of the standard toolchain) is quite a large job. Release engineering itself gets complicated when you take into consideration the different target platforms, most games get published on Playstation, Xbox and one of a handful of PC platforms (notably steam), plus the online systems and of course the development environments and internal release systems which are ubiquitous. Build Engineering usually is the step between developers and QA, and release engineering is the bit after QA. Both interact a lot with QA so could be rolled into the same role.

lumost 4y ago

I don't know, every time I see a title split people start saying that's "X's job", you can't do that because it's "y's job", or I don't want to be involved in "Z". Fewer titles can avoid these discussions.

rovr138 4y ago
Looks like you already identified the people that should be in those roles. If they're not doing it or was overlooked for some reason, then that's an issue but not really an argument to not do it.
- lumost 4y ago
  Since when are managers incentivized to put people where their interests align? particularly when there is a perceived role transition.
dijit 4y ago
Sounds like a bad culture. That won't be fixed with a new job title.

cr4nberry 4y ago

> DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways. This is always a dead giveaway that something is a buzzword Same for rest. Sometimes when people use it, it just means JSON + http requests. Other times it's supposed to be some kind of architectural style

guhidalg 4y ago
You can always point people to the original paper that defined REST: https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arc...
- 331c8c71 4y ago
  Should one also point people to the original vision of OOP (by Alan Kay) when they mention the term?
  - guhidalg 4y ago
    Yes, at least when there's disagreement about what OOP means. You and I can disagree about it, but we can more easily answer "Does it match what Alan Kay wrote about?" with a yes or no.

bgro 4y ago

I don't understand how seemingly 90% of developers, can't do anything outside of their narrow scope of experience. Writing a shell script? Throw a temper tantrum, that's devops job! Have to work with legacy code? I can't believe this! Especially seniors and beyond. They force leetcode interviews they somehow pass or are grandfathered through and gatekeep "trash devs" by slamming gotchas about how the whiteboarded code discussion doesn't technically compile and how "returnAverage()" isn't a method ("did you know you have to write methods before they work? What does returnAverage() supposedly even do? Return a random character in the alphabet?") I get excited about working in different tech areas. I'm exceptional at fixing other's bugs and maintaining code. Absolutely nobody seems to be hiring for this though. It's all about college exam trivia and leetcode. When I do get into these companies, I have to work with people throwing around casual racism with HR joining in and f-slurs like it's corporate 4chan. I ask if anyone can help look at a critical bug I discovered, and nobody speaks up. Even when it turns out it was their last code change that caused it and they are the sole master expert in this area, and they were just working on organizing their desktop instead. "I can try to look at the SQL problem, since nobody else spoke up." Then I'm explaining the basics of SQL to some guy sitting around blank faced and it turns out he was hired because of his 10+ years of expertise in SQL. His whole job is to tackle problems like that. Meanwhile I'm out here looking for jobs when panic cuts happen and it takes forever because I'm drained from all the gotchas and gatekeeping in these interviews. There's no way I could live with myself if I put this experience onto others. I lose sleep when I fail to call out someone talking over a quiet person on my team. I've never seen anybody else stand up for anybody, however.

apalumbi 4y ago
I like to build up my teams with people that act and think like you. Sometimes I need a sniper and when I do I'll get one. But my teams need to be excited about building and running the systems that we make.
hinkley 4y ago
I learned SQL because my coworker was stuck on a bug she couldn't figure out and 'knew' I'd be able to solve her problem. I just kept asking questions until I found the one she missed. I also ended up becoming a bespoke VxWorks admin because the guy who volunteered was never available, and someone asserted my code wasn't working well on VxWorks, so of course I had to know enough to do benchmarks. I fixed a few problems but the real issue was the hard drive wasn't doing DMA due to the kernel not recognizing the processor revision number. Somewhere between those two events I realized that if my part of the project is great but the whole project is on fire, nobody cares about my stuff. I don't get points for being right and the team being wrong. I mean, I do for some people, but I still feel bad at the end of the project, and those are the 'points' I have to live with the most. The way that played on on the latter project is that once we got our shit together, we started work stealing from our peer orgs. Volunteering to carve off little pieces of interface between us and 'take care' of this bit of data handling here and that one over there. I came to realize that the Org subconsciously knew this and any past success they had was due to self-organization and collective work-stealing. Throw people at it, turn a blind eye to precise mission statements, and hope for the best.

usrusr 4y ago

Do those different breeds of engineers work in one team or in separate organizational branches? The problem the term 'devops' is addressing, the way I understand it, isn't that of insufficient individual jack-of-tradesness, but that of too much separation between (internal) organisations. It should be perfectly possible to build a "devops" team from deeply specialized experts and turning it into a job description seems quite a stretch to me. But you might actually want some jack-of-trades types nonetheless, because those isolated organizations weren't completely without merit: they are good at solving that problem affectionately called the bus factor. A single ops guy in a devops team is a single point of failure and the mitigation preparation of keeping some of the not-so-ops peers sufficiently in the loop will be much more dependent on organic motivation than the counterpart in a specialist company branch. There, substitutability would be much easier to ensure with formal process.

dijit 4y ago
> Do those different breeds of engineers work in one team or in separate organizational branches? It doesn’t matter as long as communication is open. Proper ops is all about reducing the bus factor. Procedures don’t depend on people, they depend on being good at communicating. If your ops person got hit by a car, I would expect that everything would continue running, I would also expect to find some clear documentation, this is mandatory.

xtracto 4y ago

Some time ago, i was talking to an MBA people /project manager guy was ho had a completely different definition of what "devops" meant. His meaning was some kind of project management approach. We have a way to prostitute words in this field... agile, QA, devops.

llbeansandrice 4y ago

Are you hiring? I was hired as an infrastructure engineer and I'm currently writing typescript for the frontend of an internal product.

dijit 4y ago
Yes, I am, the project is mostly in stealth mode but please feel free to drop me a CV: jan [at] competition [dot] company If you want more information on the project we're working on the site is https://rennsport.gg https://rennsport.gg We're building a hard-core racing simulator game with a backend which can persist car ownership in a way that feels authentic. (IE; not just tied to a game).

chrismarlow9 4y ago

This is the answer. I am interviewing right now and my previous experience includes devops roles and SRE roles, so I get contacted for both by recruiters. After hearing about the responsibilities and examples for these roles I can only come to the conclusion that titles are a waste of time. Even the "level" of a title of Staff, Principal, Lead, etc are a waste of time. Just call the job what it is and if you can't decide on what it is at least to an 80% level, maybe you're asking too much for that job?

kodah 4y ago

I get called an "Infrastructure Engineer" sometimes, I've also had the title of SRE and SWE. I don't really feel like any of these actually fit me. I do work on Cloud Infrastructure at times, but honestly it's the smallest part of what I do. It's usually addressed in the architectural designs and I have to touch it incrementally. What I do much more often is writing tools, daemons, and services for distributed systems. I end up calling myself a Distributed Systems Software Engineer to reflect the idea that what I work on is systems and software, and most of them are non-monoliths (from a systems perspective). Do you have any thoughts on Systems Engineers or the title that I prefer to call myself?

dijit 4y ago
In the games industry we have a role called “tools programmer” which is a person who works on tooling to accelerate development, sometimes these can be extensions to the editor for a specialisation for a game (say, a tyre builder for a racing game) and sometimes they can be more broad (a distribution tool using libtorrent so that a new build of the game can be shipped worldwide very quickly). There are people who might refer to you as a backend programmer (if the focus is services and daemons), or platform engineer (if the focus is developer velocity). Largely it depends what your primary focus is.

doctor_eval 4y ago

I agree with this but I’d add that it’s also important that the roles aren’t overly siloed. A front end engineer, for example, should be able to and even encouraged to look at backend code, and even submit PRs for it. And more importantly, the backend engineers should be encouraged to be grateful for the PRs (even if it’s not acceptable for some reason). The more eyes on the code, the better. I think developers naturally tend to specialise (at least for a time) but ultimately we all need to understand and contribute to code regardless if it’s front end, backend, build, test, …