23 ms·

The Big DevOps Misunderstanding

• brightball 5y ago
  He’s not wrong. Over complicating DevOps does take the control away from developers but I think a lot of the Ops part of DevOps is to create an internal PaaS that your devs can deploy to themselves.
  • ltbarcly3 5y ago
    Except then it's not DevOps. If you have people working on 'DevOps' 100% of the time, that means they aren't working on Dev, and are only working on Ops. The idea behind DevOps is that Dev(elopers) do their own Op(eration)s, rather than throwing stuff over the wall for a separate team to manage. Not only has the industry done the opposite in many cases, they kept the name DevOps, meanwhile the full time operations staff have over complicated deployments and builds to the point that even they don't understand how they work anymore.
    • deleted 5y ago
      [deleted]
    • deleted 5y ago
      [deleted]
    • excuses_ 5y ago
      I agree that the term was kinda stollen and in many places Ops became DevOps while still doing Ops stuff 100%. I think there is a place for both. Going into DevOps does not mean you need to get rid of your Ops team. I like the approach when Ops are handling the backbone infrastructure of the organization while individual teams work in DevOps methodology meaning they are responsible for building necessary application infra, programs, as well as bringing it to production and monitoring.
  • jordanbeiber 5y ago
    Yes, many enterprise devs I know are completely uninterested in infrastructure. This is also a great way of scaling teams - keeping a path of least resistance that is secure and compliant while giving observability and conventions.
    • thrower123 5y ago
      Infrastructure in enterprises is usually completely uninteresting, to be fair. You need like eight different people on a call in order to stand up a VM, install a service, and configure a database connection.
      • jordanbeiber 5y ago
        Broken processes and missing self-service, as you point out, is why creating a platform team is important for these types of places. They might not know it yet, although some are trying. It doesn’t make the infra technology more or less interesting?
        • deleted 5y ago
          [deleted]
    • jbverschoor 5y ago
      Or any CS field for that matter.
• Fizzadar 5y ago
  Totally agree, I’ve always insisted myself and/or my team are called “infrastructure”[1], who provide the platform for the rest of engineering to deploy stuff to (devops). But when hiring for this team, have always used devops as the title because it brings in the right type of candidates. [1] platform is probably a better name :)
  • NAHWheatCracker 5y ago
    I always want to call my DevOps team the "tools" team, but that's not as catchy. It's helpless to argue for changing the name. You'll never convince people whose careers are tied to DevOps to change the word they use. We should come up with a buzzword that that we can use for a few years to indicate real-DevOps. It will be taken over by the same managers shortly after, but it can be used by for a little while by people who know the real deal. We can call this "buzzword maintenance". The industry of engineers agrees to regularly update lingo to keep ahead of corporate concept degradation. We can do this for agile and DevOps, at least. I suggest taborca (acrobat backwards) and DesMis (designer-mission). These were done with just a quick thesaurus search.
• hcrean 5y ago
  A sysadmin team leader walks into the room and says, "Everyone, you are now DevOps engineers, carry on", and walks out to collect a bigger paycheck...
  • ltbarcly3 5y ago
    So this, but instead of starting with Sysadmins with tens of years of unix experience, start with junior developers with 1 year of experience using Docker as a developer and have them read a blog post on how-to set up Kubernetes.
    • geodel 5y ago
      Now I feel this is how it is exactly happening at work even though I am not part of "devops" team. Any working solution at my place is now legacy ready to be dumped and replaced with next generation stack. Other day "devops" engineer couldn't figure out wrong directory permissions and whole deployment failed.
      • kazen44 5y ago
        we once had this issue at an old job. People wanted to replace a monitoring solution for something new, because snmp was "bad and outdated". SNMP is definitely not the holy grail of solutions (to be honest, its kind of a mess). But atleast SNMP is widly supported, works reasonably well, and gets you the data in a dashboard you need. Even for those old Cisco 7200's with an uptime of over a decade. Being new and shiny in operations is usually a sign of inmature technology. (with the bugs and issues associated with it).
        • aduitsis 5y ago
          Thank you for the post! At some point years ago I started realising that most problems in SNMP are stemming from badly implemented SNMP agents. Those problems can be: 1. Returning values different than what the MIB says. 2. Missing values that are not optional in the MIB. 3. Broken getnext code that returns an incorrect oid, which could even lead to loops. Net-snmp will detect it, but proper snmpwalk gets broken. 4. Broken getbulk repeater code that can cause crashes if the repeater is too large. 5. OIDs that will crash/stall/slow the agent. 6. Implementations that cannot handle simultaneous requests. Admittedly this is notoriously difficult to achieve: most MIB implementation code is not reentrant for various reasons. 7. SNMP SET is very difficult to implement, cannot be meaningfully used for almost anything beyond extremely simple stuff. 8. UDP encapsulation can cause MTU/fragmentation issues that are a nightmare to debug. Net-snmp actually implemented transports such as TCP and it was very nice, but it never caught on in general. Bonus, there is also the not related to implementation problem of: 9. The query semantics of SNMP are non-existent, and the indirection in the SMI is cumbersome, which means that as a client/manager you'll have to do all sorts of data wrangling to find what you need.
• mschuster91 5y ago
  Can we please get rid of the DevOps hype train? What gets completely ignored is that there is a reason that sysops has been a specialized role for decades. How often do we get threads here that people get their AWS accounts hijacked because of some credential leak (e.g. access keys in frontend builds), extreme expenses due to stupid autoscaling settings combined with a DoS attack or content leaks out of insecure s3 buckets? Or how much money gets wasted because clueless "devops" developers think that failsafe redundancy for their tiny PHP app is configuring a bunch of x2large instances and putting ELB in front of it instead of either going with lambda functions in the first place or at the very least use small instances and sensible autoscaling. Or ffs just take gitlab ci pipelines. Nearly every time when taking over a project or hopping in to consult I see utter fucking bullshit written by people who don't have any idea what the fuck they are doing. Yeah, your build takes 10 minutes because you don't use any cache anywhere and you don't use Nexus or any other form of repository cache. And ffs no you should not push your application image to public Dockerhub because you have no idea that Gitlab has its own registry - and no, "I read it on some 'getting started' guide for <insert JS framework here>' ain't an excuse either. People get barely trained in the technology they use, most JS frameworks (and fwiw, Gitlab and Github CI themselves!) are young enough that it's impossible to get senior people for them, and the stuff that ships for "CI templates" there is atrocious. The amount of knowledge required to securely operate any kind of hosting or development operation is immense, and acting like developers should also shoulder the role of unix sysadmin, IT security staff and cloud configurations is extremely ignorant - not to mention only half of the developers know more than "ls/cd/rm" on the command line, less if they are Windows developers. I do see value in "devops" as in "a person primarily focused on operating infrastructure, but with enough capability to troubleshoot production issues in applications", but devops as in "save money for actually qualified ops staff" is irresponsible penny pinching.
  • g051051 5y ago
    Great to see someone else saying this for a change. "Dev" and "Ops" are different skill sets with some potential overlap, but not 100%. Organizations I've been in have never done the "throw it over the wall" thing, either. Dev and Ops are partners, and treating things any other way is a recipe for failure.
  • 5y ago
• autophagian 5y ago
  I've also become a little disillusioned with the 'DevOps means developers create their infra/deploy their applications' mantra too, especially when you see several years of accumulated infrastructure spun up by developers that don't really care about its quality. Then again, having a dedicated ops team can lead to siloing and learned helplessness on the part of devs too. It's a bit cyclic.
  • Mc91 5y ago
    I agree 100%. It's cyclical, the trend goes one way for a while, then it goes the other way. "We don't need a separate devops department, the programmers can do it" "We don't need a separate QA/QE/SDET department, the programmers can do it" "We don't need DBAs designing normalized database schemas for ACID databases, we'll have the programmers use ORM/NoSQL/whatever"
    • Spivak 5y ago
      I mean this is true for the most part. It's one of those things that feels really wrong but the truth is that your code can be hot garbage for way longer than anyone cares to admit. I think a lot of shops should be popping champagne the moment they need to hire a dedicated DBA / ops person to clean house because it means they got big enough that they need to do better.
  • pas 5y ago
    If they don't care about the quality ... do they suffer from it when its problems and enjoy its fruits? It seems they either don't have the time/resource/ownership/change budget to make it better, or the team is so dysfunctional that they need to be pumped full of new blood from top to bottom. stat.
• dijit 5y ago
  I made a similar ranty blog post about devops. I was surprised by the amount of history and misconception that exists around the term. The term was originally intended as “Agile Systems Administration” but came about at the same time as a talk titled 100 deploys a day. So people conflated the two, and it didn’t help that the conference was a portmaneau of “Developers, Operations days” to encourage it being a conference for everyone. Eitherway. We’re stuck with it now, and it’s the least meaningful job title I’ve ever known. If you have to ask “what does x mean to you” then it’s a failed descriptor. http://blog.dijit.sh/devops-confusion-and-frustration http://blog.dijit.sh/devops-confusion-and-frustration
  • g051051 5y ago
    > The term was originally intended as “Agile Systems Administration” That explains why it's such a terrible idea.
    • vanusa 5y ago
      Exactly. One meaningless definition attempts to justify its existence by referencing another.
• hnthrowaway0315 5y ago
  One thing I really hate about a dedicated ops team (DevOps that is independent of devs) is that they have to use just one hammer for all kinds of nails. It's simply impossible for a small team to maintain multiple hammers. So somehow the tools and workflow are exactly same for an app that has millions of external users and an internal app that only 10 people use. You still have to go through a whole stacks of K8S/docker who-knows-wtf-they-are configurations just to publish a dashboard from an internal IP. It took more than 8 months for my previous team to try to pull it off but it never went through.
  • mrweasel 5y ago
    As an ops guy: You’re absolutely right. For better or worse I’m constantly annoyed by developers trying be “special”. Most projects involve stuffing things into a database, calling a webservice and reading stuff back. Don’t try to be clever about it, just stick to one type of database and Springboot so we can all go home and sleep with no incidents. But there are projects which really do need to be different, and we don’t deal with them well, and they don’t scale.
    • hnthrowaway0315 5y ago
      I don't blame the devops guys though, I think this is a corporate thing as teams have to follow procedures and devops are not an exception (to waive some rules for us). I just hope it's a lot easier if we can just use a VM instead, but that would need to go through security certification. I wonder how other companies deal with this kind of issues, eventually my previous team decided to host the app on one of the laptops and tell clients to visit a "192.168.0.X:port".
    • digianarchist 5y ago
      Nothing better than a team that doesn't use the internal platform. They go and write their own scripts, tools and terraform only for the developers responsible to leave the company. They found another way to skin the cat purely because they didn't like the existing approach and didn't want the hassle of getting changes integrated upstream.
• mrweasel 5y ago
  > You build it, you run it While I do like that definition, it also means that most companies/organisation cannot do DevOps, as software development is often bought as projects, and operations is a recurring cost.
  • 988747 5y ago
    Also, government regulations such as HIPAA explicitily force you to have separation of duties, so true DevOps is against them. Developer being able to push their code to production can earn you some hefty fine at the next audit.
    • sethammons 5y ago
      We pass our audits all day long and I can push to prod multiple times a day. But there are caveats. Another developer must sign off on a code review and quality check before code rolled out, and a build pipeline ensures that we run our test suites and all changes have an audit trail.
  • d0mine 5y ago
    Software is never done. It can only be abandoned. There are projects that change slowly but they do change nonetheless unless they are no longer used.
  • throwDec19 5y ago
    We get heavily clamped down on access to production by our auditors for Sarbane Oxley reason. Technically we aren't even allowed production data in our dev environment. I dont understand why any organization would allow it. Seems the tech world is still ripe to get sued.
    • throwawayboise 5y ago
      Organizations did (do?) allow it because it's easy. It's also a way to test/debug production problems with actual production data instead of spending time trying to make up data that you hope covers all the actual use cases.
• cortesoft 5y ago
  As someone who has lead a devops team, and still has a devops team reporting to me, I have also struggled with the idea of having a team called devops, since it does go against the devops philosophy of devs owning their software in production. However, in our case (and probably many others), our devops team is responsible for creating and maintaining the tools and processes that allow the dev teams to manage their systems in production. At a larger organization, it doesn’t make since for each individual dev team to have to create and maintain their own deployment and management systems… that would end up with a lot of duplicated effort and a wide variety of systems that would need to be maintained. Devops creates the tools and procedures, and dev teams use them. You might say this is just sysops, but I think the key difference is that the dev teams are the ones who are using the production management tools, not the sysops team. Devops is not the ones who are paged if a system has an issue, the dev team who owns it is paged. Devops is only paged if the tool that the dev team is using to handle the issue has issues itself.
  • sharken 5y ago
    The DevOps team can often be the team that handles Compliance, Security and Audit for the platform. Then there is the infrastructure/deployment team and finally the developers. At least in larger organizations i think that is a common way of doing things.
    • 123pie123 5y ago
      in my experience with in enterprise businesses the DevOps people are almost the worst people to handle Governance, not too great at end to end Security (I've no idea about Audit) IMHO I would guess they're too close to the dev people to care about the bigger picture
    • balp 5y ago
      If you try that you totally broken the DevOps idea, but now the whole world of ops and IT did that ages ago as anything that is configured in code, e.g. yaml, is devops. The article suggests using a higher level abstraction with better tools suited for the developers than the low level tools as docker, Kubernetes etc. You can make this into a separate internal product and still keep the devops idea. But as soon you break out any of the core responsibilities for the product from the team. If that's business, security, stability or quality you are moving away from devops. Sure devops isn't the solution to all problems. But why call it devops when one doesn't like the basic devops idea.
• recursivedoubts 5y ago
  I'm frankly astounded at the complexity around modern development due to dev ops and how much worse developer experience is because of it. I used to develop in a great IDE with debugging, right click re-run failed tests, I could follow the console right there in a nice, clean window integrated with my IDE, click on stack traces, etc. Now I'm running my app via multiple docker images, trying to get a buggy remote debugger protocol to work (no local variables, wtff?) where I have to coordinate the container and the IDE, command line wrangling / using print debugging in the container to chase down stuff, digging around through multiple log files, copying-and-pasting stack traces into my IDE to analyze, which kinda works in some cases since the file paths are different, and on and on. DevOps came and took my nice dev environment away, and I am mad.
  • stepbeek 5y ago
    Given the second principle of devops is amplifying feedback loops, I think a more charitable view could be "the common misunderstanding of DevOps has lead to a terrible dev environment". Fwiw, I'm fully onboard with you, but really love the core message of books like the phoenix project and don't really want it to be lost because of overzealous use of docker.
    • hnthrowaway0315 5y ago
      OMG, I read the sample chapters from Amazon about the Phoenix project and that was a really really good read, thank you!
      • stepbeek 5y ago
        It’s an amazingly good read. The follow up - The Unicorn Project - wasn’t as good, but definitely worth reading.
        • hnthrowaway0315 5y ago
          Thanks, really LOL a few times during the read.
• sharken 5y ago
  Developers typically care about developing the business side of the application and have little interest in the rest. Actually, it can even be split in three, where developers are done when the Pull Request is done. Deployment, tooling and IaC is then handled by developers who specialize in this area. A third group is then responsible for audit and security on the chosen platform, e.g. Azure.
  • 988747 5y ago
    >> Developers typically care about developing the business side of the application and have little interest in the rest. Correction: developers care about using cool new programming languages and frameworks. "Business side of the application" has to be forced on them by business analysts and scrum masters :P
• ownagefool 5y ago
  Counter point. How is owning a namespace on k8s with a shared services team looking after k8s health any different from using amazons aws apis and having shared responsibility with aws? I don't really understand the railing against complexity just because you can see how the sausage is made. What's the deal?
• mac-chaffee 5y ago
  > This trend is also supported by Kubernetes as this is a technology which is hard to master. Maybe this is rich coming from a Kubernetes person, but sounds like the tide is turning against this viewpoint. Running Kubernetes itself is definitely hard to master, but you can take a <10 hour online course and be able to "helm install" your application. Kubernetes can be the new PaaS (if you have someone else running the control plane and maybe a monitoring stack). But there's not a PaaS on the planet where you can just skip the step where you learn about it. Even Heroku users eventually have to learn about how connection handling and limiting works, and the specific Heroku commands to upgrade your database.
  • dijit 5y ago
    > but you can take a <10 hour online course and be able to "helm install" your application. Genuinely I believe that good QuickStarts are the cause of so many expert beginners. What I mean is: there is a person on an adjacent team who truly believes he is very competent in nearly all technologies. He is able to deploy a k3s cluster no problem. He is absolutely unable to debug that cluster, or use it in a way which is reliable (local persistent volumes means you are dependent on the node never going away), he is unable to troubleshoot networking issues that have occurred in that cluster and he has struggled many times to admit that it is unable. But heck. He can helm install no problem!
    • anticristi 5y ago
      > What I mean is: there is a person on an adjacent team who truly believes he is very competent in nearly all technologies. He is able to deploy a k3s cluster no problem. Aren't you confusing two roles, Kubernetes application developer and Kubernetes administrator? The Kubernetes application developer should focus on their application, NodeJS, Java, DotNET, whatever. Helm install (actually helm update --install) is all they should know/care when it comes to Kubernetes. K3s deploy and local storage providers? Let those be the Kubernetes administrator's headaches.
  • midrus 5y ago
    Are you comparing the cost of learning kubernetes to the cost of learning heroku? Or even AppEngine? Those are totally different leagues. Also, with heroku, AppEngine, etc you have tons of documentation (official and unofficial) and resources. It is well tested and the exact same service runs thousands of businesses. While kubernetes is like the react of the infrastructure world: you won't find two similar implementations. You need thousand tools configured around it to make it usable, and of course you might have docs about the individual pieces but you never have it for the big picture.
• Niksko 5y ago
  Kubernetes serves to allow a team within an organisation to build a PaaS that's customized for your organisation. You do this because 1. Baking in your organizational defaults and opinions reduces work and cognitive load for your engineers and 2. A PaaS is too expensive at scale (compared to IaaS), especially given what you get isn't tailored to your org.
• zthrowaway 5y ago
  My DevOps team are actually SREs, and our SRE team are just webdevs who are on call. It’s pretty ridiculous.
  • geodel 5y ago
    Could be that someone in management understood site reliability as website reliability and decided to put web devs on call.
    • dvtrn 5y ago
      As an SRE, I’m starting to think the misunderstanding of SRE tradecraft is probably more pronounced than the misunderstandings of Devops tradecraft. The risks definitely feel higher. I’m six months into a new SRE role and it really feels like instead of doing any kind of engineering I’m instead just getting “overflow pipeline work from the rest of engineering that the Devops team was too busy for” and I’m already looking for the door.
• avip 5y ago
  Personally I think it's pretty great that I can write Dockerfiles, run them locally, and call it the day because this is also what runs in production. So basically I have no clue what the dude is talking about - we have a great setup which is x10 better than what we had before. No?
  • quadrifoliate 5y ago
    > Personally I think it's pretty great that I can write Dockerfiles, run them locally, and call it the day because this is also what runs in production. It goes far, far beyond Dockerfiles. Are you on-call for production? Can you debug a problem in production? If not, then that's the root of the problem that the dude is talking about. If these are true for you (not specifically you, any reader of this): - You don't have a good idea of how to ship a change all the way to production after committing it - You would not know how to debug and fix a failure of your code in production - Your "Dev" and "DevOps" teams are in separate departments ...then congratulations, you have just reinvented the old-school sysadmin/developer dichotomy from the 90s. You just happen to be using Dockerfiles instead of Makefiles.
    • kazen44 5y ago
      Not to mention the layer that lives behind this. Can you troubleshoot network level issues that seem to occur between your system and someone else's? What about storage? Can you make sure your data is still their if your environment gets royally screwed. I know these are usually done by seperate companies nowadays, but getting an an actual computing environment up and running with redundancy in place (down to the cooling and electrical level) is no easy feat.
    • smus 5y ago
      I'm glad there's a dichotomy. I (a dev) never signed up to be on call 24/7 for production and it's ridiculous to just assume that I should be. I have a life outside of my work. If the system needs 24/7 uptime, they better be paying someone other than me to ensure that's the case, because my time is too valuable to waste my already precious freetime on fixing bugs in prod.
• pjmlp 5y ago
  Somehow it seems System Engineers got renamed into DevOps, from the kind of tasks that land on my table when I happen to wear the DevOps hat on some consulting gigs.
• KaiserPro 5y ago
  I did not entirely agree with the main assertion of the post. However having re-read it, I do now broadly agree. I would say that devops _should_ be a spectrum. I have always worked at places that never really had rigid boundaries between sysadmin and dev. So I was really quite baffled by the initial push to devops (Isn't that what you were doing before? How can you be a sysadmin if you can't code.) Having a department of devops is fine, so long as most of them are embedded into dev teams. Assuming you rotate your devops around dev teams, its a good way to remove single persons of failure, and increase standardisation. Depending on size, there might be a chance to have a core systems/platform team.
• safeerm 5y ago
  Really great post. When I was at AWS, we did everything with IaC but that luxury of time and investment is not avail to many. We're working on this problem but ensuring that devs has a) full transparency what's going on, b) ability to evolve their arch and c) building right on your AWS account. We just moved someone from Heroku to AWS handling 1B requests/day at 1/4 of the cost. have a bunch of AWS execs backing us too; they know this is a problem. Happy to chat more if you want to give it a spin: tinystacks.com
  • qaq 5y ago
    cloud is a marketing scam. It is sold to execs as a magic black box that scales and you don't have to be concerned with how it works internally. In reality it's a set of leaky abstractions and you absolutely have to understand how black box works internally plus be aware of thousands of bizarre hard/soft limits and other gotchas.
• jollybean 5y ago
  When the cloud was young, devs could indeed push their own code. They didn't need the sysops people. It was a major change. But as the cloud became more complex, invariably, SysOps just took on a whole new shape. Instead of buying hardware and keeping Ethernet boxes wired and alive, you now have K8s etc.. I suggest we have failed somewhere along the way: it shouldn't be necessary for real DevOps up to significant scale. If our tooling were good enough, it should be mostly fairly automated. 'Containers' seem to be a messy, hacky afterthought when what we really probably just needed was 'configs'. Cloud Lambdas, Queing, Caching, Logging could be feasibly made very simply configed ... on all the Cloud Providers it's fairly complicated, just beyond the ability for a regular dev to stay on top of along with the rest of dev stuff.
  • darkwater 5y ago
    Also, there should be no bugs and PMs or business owners should be able to draw their own SaaS, no need for developers as well.
• hodgesrm 5y ago
  > This trend is also supported by Kubernetes as this is a technology which is hard to master. So people start splitting up responsibilities again. One part of the team developing the application the other part of the team managing the infrastructure. Kubernetes administration is hard to master. It's analogous to running VMware in production, or any cloud technology for that matter. None of them is easy, and they all require relatively specialized knowledge. Kubernetes development on the other hand is relatively straightforward. Minikube is my goto environment whenever I have to put together an app that consists of more than a single process. As long as you pay reasonable attention it is not hard to move such applications from dev to staging to production. Edit: fixed markup
  • gravlaks 5y ago
    Yeah I don't really get the hate on Kubernetes. Run it managed with EKS, GKE or similar, set it up with the needed controller (ingress, external secret, whatmpt). The development afterward is as you say straight forward.
• Ozzie_osman 5y ago
  100% agree. The industry took the term DevOps and started applying it to old-style ops teams (only difference is the ops teams use more "modern" tools). There's still value in having an infrastructure or platform team focused on building tools to enable dev teams to more efficiently deploy, monitor, and run their code, and to step in with expertise when needed (ie help debug things like database performance, etc). More "teach a man to fish" type help, but also "give them a good boat and fishing net".
• lmilcin 5y ago
  Similar thing happened to Agile. Agile is a practice of speeding up improvement. Idea being that organisations that improve faster will, over time, overtake organisations that improve slower. If so, it makes sense to focus on the improvement loop itself. You remove management from the loop (because management takes too long and slows down improvement) and you empower and encourage individual contributors within their teams to improve the process because they can do it faster, have hands on knowledge of the process and can observe results faster. Additional benefit of Agile is that you now have a tightly knit team that tends to be more engaged with their work and more focused on thinking through what they are doing -- this tends to cause people to be more satisfied with their jobs because satisfaction of being able to affect the process positively is a great motivator. Yet, the management thinks they can come with a book and institute "Agile process" by telling everybody what they have to do, exactly. Usually, improvement within team structure is then actively discouraged with many explanations like "that's not what the book says" or "I was doing scrum at a company where it worked and so we must do the same" or "we need to standardise processes" or some other variation of it. I have observed this time and time again at various stages -- always with the same miserable outcome. At my last company they additionally decided we are introducing Scaled Agile Framework meaning, according to them, we will now plan not at team structure but at a department level. This innovation means all planning and retrospective are combined for about 400 people. Because this was too time consuming to do every two weeks, they decided we will do this every 5 sprints (10 weeks). Just to be clear -- every 5 sprints we were supposed to select and plan tickets for the next 10 weeks of development. This was presented as an efficiency improvement.
• mathfailure 5y ago
  There seems to be a quote used as the main argument: > You build it - you run it. Well, building != developing, so DevOps both build the code and run the code (written by devs), no misunderstanding.
• manish_gill 5y ago
  I had to basically give up my full time role as a Data Engineer who is supposed to solve "big data" problems (Storage, Streaming, Machine Learning pipelines) etc, and had to solely focus on DevOps because there was no other expert in the team. We (a colleague and I) basically learned Terraform, then Kubernetes, and then Helm in an extremely stressful environment. I wouldn't recommend that to most people. (Also, I can hear "you don't need k8s", but tbh, it makes the difficult things possible, and that's what we needed). I've done Ops in bare-metal environments before, and at least there, even if there were new skills to learn, the environment just wasn't so hostile for beginners, and the learning curve was simpler. The YAML (or heaven forbid, jsonnet) does *not* help. TBH, I'm convinced if the IaC bits were done in a regular Java/Go/Rust SDK style instead of mixing and matching YAML glued together via labels and tags, I would have had a much easier time. The positive side of that is that now the IaC bits are mature enough that even the Data Scientists in our team write their own Terraform code, and I've upskilled myself quite a lot. I'm slowly going back to solving Storage/Data/ML issues, which I enjoy a lot more. Maybe I'm just becoming a cranky Software Dev as I enter my 30s. I do _not_ enjoy reading YAML.
  • lukevp 5y ago
    If you haven’t heard about cdktf, check it out! It lets you do infrastructure as code but in full programming languages like Python, C# and TypeScript. It’s by hashicorp, and under the covers it runs Terraform, so you have the whole set of terraform providers at your disposal.
    • manish_gill 5y ago
      Ah, CDKTF seems to be the Pulumi equivalent for Terraform. I think I remember seeing it. We're far too invested into Terraform at this point to learn yet another tool. Oh well. Hopefully the next time I have to implement something like this, K8s ecosystem would also have similar tools (or maybe I just missed something)
      • chucky_z 5y ago
        There’s a straight up conversion tool iirc, and you can do things like include hcl as well I believe. Cdktf is just terraform at the bottom.
• emptysongglass 5y ago
  DevOps was always about de-siloing. In that, it is a culture but you need people to "be the change" and that's how I've played a DevOps Engineer for the last five years (well now SRE) and how I believe DevOps Engineers (and SREs by extension) ought to be playing their role today. Corporate management is always surprised when my first move is to go sit with the developers and ask how I can unblock their workflow, reduce pain and tighten their development loop. Most SREs and professional DevOps people I've met do not do this and insist that their function is to tell the developers how they'll build their app and in which way AKA the same shit from the 90s. The portmanteau should be a dead giveaway that the number one priority of this job is building cooperation between these divisions. And often, it's not a technical solution.
  • jchmbrln 5y ago
    I think you've hit the nail on the head. In a previous job I tried to outsource "DevOps" to Rackspace, only to learn the hard way that if you can outsource it it's not DevOps—_increasing_ siloization is the antithesis of DevOps. But this is also the point where I think SRE is most compelling. Rather than trying to build an organization around "you build it, you run it", SRE recognizes the need for specialization while deliberately striving to align incentives between the various parties via SLOs. This summer I finally hopped over to the DevOps side of the house as a manager, having always been a software engineer. My mandate from higher ups is to reform the team to start practicing SRE, but I think it's really more about reforming the whole organization. In every way possible, I'm trying to build a team of people willing to "be the change" like you say. But I also hope that SLOs, as a formal way of agreeing on priorities, will grease the wheels a bit. > And often, it's not a technical solution. Absolutely. In my experience, it's almost never technical. If you can structure the collaboration so that correct incentives apply, any technical solutions required come naturally. (I wrote about my Rackspace experience here: http://zephyri.co/2016/can-devops-be-outsourced/ http://zephyri.co/2016/can-devops-be-outsourced/)
    • t_sawyer 5y ago
      I see the function of SRE as a place to pass off an app when an app team (DevOps included) need to develop another app. SRE sets standards so a centralized team can manage the ongoing maintenance and on call break/fix of many apps in an organization while development teams move on to something else.
• anticristi 5y ago
  Kind of weird to blame Kubernetes. As long as the application team operates their application and the platform team operates their platform, I would argue that you do DevOps. It's just that one team consumes services from another team. Kubernetes just happens to be a damn good "API" between the platform and the application team. Indeed, the application team is likely to build using code, whereas the platform team is likely to use scripts and configuration. But each runs the stuff they build.
• deleted 5y ago
  [deleted]
• StreamBright 5y ago
  I think k8s is exactly the reason we are moving back to separate ops vs dev modus operandi in many cases. Projects like Terraform and Pulumi moved us towards a more vertically integrated development process, bringing the knowledge and tooling of SRE people right into the application repository. And when k8s turned out to be too complex for this setup we claim that the previous way of doing things was better. I disagree, strongly. If anything, we are going to skip k8s and just use the other solutions (AWS, Azure, even on-prem with other tools than k8s) and we are going to keep our devops tools, integrated teams and the progress we made in the last decade.
• GiorgioG 5y ago
  Kubernetes is a cancer for 99% of companies. Most companies need little more than docker compose/swarm. Kubernetes is the antithesis to KISS. If you’re Google, then you need something like K8s. Most orgs using it are pretenders or trend followers. Sadly it has become the “standard” way to deploy apps.
  • throwawayboise 5y ago
    Most people just need lxc-create or virsh create, honestly. It's the only things I have been able to get to work reliably. All the docker and k8s stuff is unnecessary complexity if you just need to run a container or VM.
    • kazen44 5y ago
      I know a couple of guys running a jails based system with carp for load balancing and redundancy. they have 4 physical servers in total and have been at it for more then a decade. the only time they had downtime was during the migration from one DC to the another one. this system runs 80% of the EDI market in my country... simple solutions are usually better.
  • NAHWheatCracker 5y ago
    I just joined the first project to use Kubernetes at my current company. I remember a conversation with a "DevOps" engineer when I started 3 years ago. They had evaluated Kubernetes and decided against it due to how complicated it is. He left a year ago, shortly before the project started. The project is a mess and no one knows what they are doing. They've been building it for almost a year and nothing works. I'm pretty sure the decision was made based on the desires of the developers with a fragile veneer about how it is necessary for the product they want to build. At least it's fun to learn, albeit frustrating to work with.
  • Toine 5y ago
    100% this, same for microservices. One of the most damaging belief a developer can have is "if my stack looks like * insert unicorn *'s stack, we will be a unicorn"
• 1270018080 5y ago
  DevOps Engineer is an oxymoron, but who cares. Prescriptive linguistics is a fools errand. DevOps engineers do important work, their job is real.
• coolgoose 5y ago
  I am so tired of devops as a term. It's a stupid bandaid solution. Infrastructure, regardless if you use a paas or not, is a tough cookie to crack and deserves a normal sysadmin role. Monitoring, resource utilization, network management, library updates, costs among a few are things you need to master, observe and proactively manage, not keep it as an afterthought since it magically just works.
  • kazen44 5y ago
    troubleshooting infrastructure is hard. You cannot usually bring down or change something in production without extensive research because well, its production. Issue usually happen either at seemingly random times, or at moments when doing something directly about it is nearly impossible because of the impact it will have on the production network/system. Also, in production (in the greater sense of the word, for instance, a datacenter with a virtualisation system running multiple workloads/applications). that most issues between systems start to creep up. Usually, systems themselves have been tested enough, its the unintended interaction between them that throws things off balance.
• jedberg 5y ago
  Devops is a philosophy not a role. The title DevOps engineer is meaningless. There are software engineers, who write software and then deploy it onto self serve infrastructure, and there are operations engineers and data engineers who build and maintain the systems that the software engineers use to deploy and operate their software. Usually when someone calls themselves a DevOps engineer they are an operations engineer, but sometimes they are a software engineer at a small company who also has to maintain their own infrastructure.
• quickthrower2 5y ago
  PaaS is easier and I prefer it for personal projects. I don’t want to use the terminal or shell at all and have minimal yaml files. For work though kubernetes means portability and not having the rug pulled from under you, you can move clouds more easily if necessary.
• gobins 5y ago
  I have been working in Infra Automation and deployment automation for a while now. As much as there are complaints about having a dedicated team called "Devops", one thing that this change has really done is that CI/CD is almost a non-negotiable of application delivery now. I am sure there are a lot of teams and orgs still starting out in this transformation but its no longer something I have to explain to application teams now.
• atmosx 5y ago
  > You build it, you run it For any set of non-trivial apps the cognitive load is simply too big. Assuming the engineer can handle it, eventually more time will be spent in keeping up with the current system(s) than writing code/promoting the business logic. A company could chose to use ready-made tools but all them get expensive pretty quickly and make their own assumptions about workflows, view-points, etc. The assumptions might fit the use case or not. Lock-in then becomes another problem. There's no easy way out, it's trade offs all the way :-) Example: The Haskell lead engineer for App1 has all the necessary knowledge to understand how the base-metal NixOs fleet is configured because there's deep familiarity with Nix. At some point the infra team will start making decisions and assumptions to automate _this_ or _that_. The engineer now will have to spent time understand what changed since last time, making sure nothing is broken while performing changes.
• lillecarl 5y ago
  I disagree, DevOps people are writing code and "code" (YAML, HCL, JSON) that's used to provision infrastructure. When developers push to master it'll build and deploy. They're still deploying, they're just getting help with running the stuff. Can we really expect everyone to understand how solid infrastructure should be built? I don't think every JS frontend dev should have this know-how, yet their code is deployed on S3 behind Cloudfront (or whatever in front of whatever, static file hosting feels like kind of a solved problem).
• streetcat1 5y ago
  I think that the op is missing a side. The article is about dev -> ops. The other side is ops->dev. I.e. since infrastructure is software, ops become dev (and not just dev become ops).
  • kazen44 5y ago
    has ops ever not been dev? Atleast in my experience (ISP/DC world) most operational people where expected to do troubleshooting which at some points, results in one having to program to get to the nitty gritty efficiently. At a certain scale, issues start to creep up that are hard if not impossible to troubleshoot without atleast some programming and CS experience. Are operational people developers? Not in the traditional sense, but the development of specific tools is required to efficiently keep the lights running. A good example of this is troubleshooting packet loss. In a large network, this can become very difficult to pinpoint because more then a few dozen hops exist, and packets drop seem to occur randomly. Without tooling to gather this data from different points in the network and doing statistics on it, it becomes nearly impossible to get a clear picture and troubleshoot efficiently.
    • cpitman 5y ago
      In my experience, mainly in the non-tech enterprise space, the classic sysadmin team is allergic to writing code that would require ongoing maintenance. The most they would do is write a throwaway script, or possibly have their own personal set of scripts. But write an actual tool for broader consumption? Never. It's one of the biggest differences I see day to day in enterprise operations and developer mindsets. Developers generally do not worry about who will support code before they write it, because their entire job is to write code. Operations teams on the other hand want to find an off the shelf tool, with enterprise support, even for what could be a fairly simple task.
• arielb1 5y ago
  The one system I've seen work is when each medium team (of say, 20-36 people) has a "devops team" of say 4-6 people that is responsible for all the infrastructure stuff, while the rest of the people can focus on product work. It's just that keeping track of infrastructure is much more efficient as a full-time job rather than a 10%-time job. It's important to do the org engineering right so that the infrastructure people and developers feel responsible for each-other and work well together - ideally both teams should be able to open PRs against each other's code if needed and get help any time. If the org engineering isn't done right, devs write code that pages the ops and the ops don't respond to devs quickly enough for things to work well ("traditional ops").
• EGreg 5y ago
  And what exactly is "Platform as a Service"? Can someone define that?
  • joedoejr 5y ago
    It is bunch of network devices, servers, oses, services and softwares running by (dev)ops
• hiptobecubic 5y ago
  There seems to be a lot of people here, and presumably a lot of companies, where no one has ever actually seen a successfully run devops team and what they provide. For one, people are arguing past each other about deployment environments vs developer tools vs being a sysadmin.
• kristianpaul 5y ago
  Could not agree more
• mongro1 5y ago
  I feel this article is missing a lot of the drivers. When automation was kicking off Ops just sucked. Infrastructure teams were slow to respond and automate, were underfunded, whatever. They sucked when Dev's were being driven to deploy faster. Dev's started do it themselves. They sucked at it too. They got some "ops" guys in the team to help them do it better. This was then called "DevOps" and the consultants built an industry around the cultural change of collaboration between Devs and Ops. DevOps smushed together so much that it became just "Devs" doing ops work. They realised that Ops is hard and 70% of their work was Ops, with 30% providing business value. Dev's don't want to Ops, they just want to Dev. Scaling dev's meant building shared deployment platforms. So ops was centralised into a "Devops" team. Usually this is jammed between Dev teams and traditional Infratructure (VMWare) teams. Kubernetes lives here. Having a devops/platform team also means you have less fingers in your cloudspace breaking things and deploying bad patterns. Good for scale. Dev's just want to write code, throw it at git, tests run, they repeat. No ops for them.
• Sosh101 5y ago
  In my experience, far too much responsibility is placed on the person generating the code nowadays. It's just got worse and worse as infrastructure has become more "software defined". Writing the code and interfacing with all the systems we have to today are two different skills, and I think not often going together.
• eranation 5y ago
  I see it as a specialization of Software Engineering. Like we have (self proclaimed as well as normalized by job posts): Frontend Developers, Backend Developers, (And Full-stack developers if one can do both, or simply, Software Engineers), And infrastructure developers. DevOps, and especially with Pulumi / CDK (where it's actually infrastructure-as-actual-code and not infrastructure-as-a-JSON/Yaml file), is simply becoming a "developer who also likes ops" type of specialization. E.g. front-end, back-end and stack-end. The nice thing is, and what I try to do in my team (both at my current startup and previously at AWS) is to break the silo and keep the "you build it you ship it" notion. Frontend - TypeScript, Backend - TypeScript, Infra - also TypeScript. When an engineer wants to build a lambda that listens to SQS or a Fargate ECS task or an S3 bucket, they can just do it. I if you know how to use the AWS SDK to let's say read/write to S3/DynamoDB/SQS, the API to create S3/DynamoDB/SQS etc is not that far off, and if you are using CDK in TypeScript for example (whether it's Terraform / CloudFormation / CDK for k8s doesn't matter) then indeed the boundary between backend code and infra code is getting smaller and smaller (both technically and also visually, the declarative code to generate an S3 bucket in CDK is pretty damn close to how it will look to generate that same S3 bucket via the SDK, it's just that CDK takes care of idempotency / change sets / diffs etc...) In summary. I agree that DevOps should mean - you build it, you ship it and that the current industry trend towards a separate DevOps department is just plain wrong, but on the other hand, it's ok for specialization within a team, some will lean toward frontend, some toward backend, some toward data engineering, some toward data science, some toward security and some toward operations. Some will do a bit of both and some will prefer to focus and specialize, but I think that the main lesson I learned is that it all should be part of the team. Like a D&D party, you don't go to an adventure with just Warriors, then send the Wizards separately to ship the dragon to production.
• siliconc0w 5y ago
  Running systems is a different set of skills than building them. You can mitigate this to an extent with a PaaS but eventually you'll run up against operational issues that require expertise to remedy properly. You can try to use generalists to handle both produce development and operational roles but you'll start accumulating inefficiencies rather quickly (i.e your entire dev team needs to understand the operational tooling so they can be on-call, you're going to be spending more for resources you aren't using optimally, and you'll likely start to accumulate toil, downtime, performance, or security problems). So specialists tend to pay for themselves pretty quickly. Even just having separate reporting chains is helpful to achieve a balance between work driving operational metrics (stability, performance, security), and product development work driving product metrics (sign ups, sales, acquisition costs, etc)
• lr4444lr 5y ago
  I think a lot of people are missing the point of the article: it's saying that devops was coined by Vogels (i.e. Amazon) as a value add because actual ops people are a necessary evil for a lot of companies that just want the features devs are cranking out to WORK without needing to be managed. Every salary that has to be paid to an ops person is a salary not spent on a dev making new features, fixing important bugs, or doing data engineering, etc. that clients are paying the real money for. Clients don't wanna hear about your deployment problems or DB backups. I'm not saying they are unimportant, but that they're painful cost centers which people looking after the bottom line would love to minimize, i.e. pay Amazon a fraction of what you'd pay full time in house people to do - at least in part. DevOps was meant to give devs more control over this work so that fewer ops people would be needed. So when there is suddenly a new generation of ops people AGAIN getting paid just to manage stuff instead of build stuff, only now in the cloud instead of on on-site bare metal, this is a BIG misunderstanding of what the intent was.
  • trabant00 5y ago
    > pay Amazon a fraction of what you'd pay full time in house people to do - at least in part. DevOps was meant to give devs more control over this work so that fewer ops people would be needed. And here is the lie in the AWS marketing promise. In reality without ops the infrastructure clueless devs will generate huge AWS bills that move the profits from AWS customers into Amazon's pockets. At the same time with production being more brittle and ci/cd becomming a complex nightmare that gradually takes larger chunks of dev time until things grind to a halt. The catch 22 being leaking abstractions. In the same way that devs cannot really empower regular users to generate apps with nocode that work well, devops cannot create iaas or paas that can be used without understanding the underlying os, networking, configuration management, package managers, etc
  • gfodor 5y ago
    Wasn't coined by Vogels: https://newrelic.com/blog/nerd-life/devops-name https://newrelic.com/blog/nerd-life/devops-name
• mgarfias 5y ago
  I thought “dev ops” was applying dev work flows to ops processes. That is, sprints, jira stories, etc. Then SRE came along, and it was all about devs doing ops work focusing on keeping things up and running. That is, a dev doing rigorous ops work with a focus on SLIs, error budgets, etc. But then, the term got trendy, leaked out of Google, and everyone wanted to be cool so they called their devops guys (which were really repurposed sysadmins) SREs but the job really didn’t change. Source: sysadmin/devops/SRE since 1995
• emptybottle 5y ago
  Is PaaS considered easier because platform operations are handled by a different group of people?
• sjtindell 5y ago
  As I learn I’m realizing that what a lot of people miss is the rigor required once you pass a certain company/infra size. The same stack and processes that worked to get you to $X million is not the one that will allow you to scale software deployments to support $X billion worth of business. Documentation, real ci/cd with multiple environments, strong access controls. It’s so easy to gloss over these when you’re small. You have someone on the team who is “the expert”, you don’t have the money to do a real blue/green setup, you don’t have the time on an incident to give someone the proper access so you just give them a shared ssh key…success at scale requires discipline that makes the work hard and not fun. But it’s the key. Once you have dozens of infra engineers and many microservice teams, this laziness (that’s what it is, I know because I’m lazy) doesn’t work.
• dorongrinstein 5y ago
  We built the https://controlplane.com https://controlplane.com platform to let developers focus on the Dev and get best practices Ops so they can deliver many times faster. Our customers get 99.999% availability and ultra low latency no matter where users are. Check it out and I'd love to hear your feedback on the Ops part in DevOps with Control Plane.
  • anotherhue 5y ago
    Control Plane is awesome and as a former SRE I recommend it whole-heartedly.
    • dorongrinstein 5y ago
      Thank you! make sure you write me an email. I'll ship you a t-shirt :)
• tuananh 5y ago
  working in a corp (10k++ employee) where everyone is specialized and outdated with the current tech scene. it's difficult to move sysops team to devops because 1. old mindset 2. outdated technologies 3. learning capabilities. devops approach is more friendly for those with dev background but not ops background. everything about devops screams developers. it's like it was built for developers only if you think about it - immutable? - reproducible? - faster loop feedback? - testability (with cdk)? writing test units? all these sounds foreign to sysops people from where i work.
• VMtest 5y ago
  The comments are really a mess Some people are suggesting to change how DevOps works by going against the definiton of DevOps In that case, please don't put DevOps in the engineering title anymore
• anotherhue 5y ago
  Are you running your application on a UNIX system? (You probably are). Do you know your octal file permissions? Your various FS choices? Your reserved ports? Your inode limits? Your uid mapping? Your SSL root trust location? Your free-space limits? Your TCP handshake? Your TLS 1.1, 1.2, 1.3 handshake? Your HTTP verbs? Firewall rules? NAT configurations? Load Balancer health checks? JVM -~XmX values? Swap size? Database connection pools? LibC implementations? GNU vs BSD command arguments? cgroups? jails? eBPF? retpolines? You're probably a developer, and these are concerns of a lower level. They are different skill sets, and pretending otherwise is marketing bullshit.
  • cakeface 5y ago
    As a developer your stack goes all the way down. An issue with a TCP 3 way handshake to your application load balancer is still an issue. An error in the filesystem that results in customer impact is still your responsibility. We can work with specialists but ultimately it is a team that owns their software running in production. Many of the things you listed are well known, monitored, and tuned by software engineers operating their software. Especially connection pools and memory settings.
• sriku 5y ago
  The "devops philosophy" is not a fundamental fact of nature (of dev teams or whatever). If deploying stuff is complicated, it is not surprising that we'll have teams that specialise in it. The naming/identification of the organization concept has to follow structures determined to be desirable for more fundamental reasons ... and not the other way around. So the right question to ask here is perhaps "why is deploying so complicated?"
• throwaway984393 5y ago
  "Things get much easier if you-" You can stop right there. There is no easy way. That is the DevOps misunderstanding. To understand why, look at your car. Your car is a complex system that appears simple. You just purchase it, put gas in it, turn a key, shift a gear, press a pedal, and it goes. And goes, and goes, and goes. But over time the tires will go bald, the brakes will wear out, you will run out of coolant and brake fluid, the engine oil will turn to sludge, and the car will break down. Past those inevitable faults, some parts will randomly fail and require troubleshooting and replacement. You have two choices: either become a mechanic, or pay for one. (Or I guess. run it into the ground and buy a new one...) Computer systems are the same. Say you just want to "buy and drive". Ok, so you buy a PaaS. You still have to learn how to drive it. And they are only easy to drive up til the point they break down and need more maintenance. And that's just driving it... Now say you're in auto racing (building your own high performance, high reliability system). Someone has to build the car, someone has to maintain it, and someone has to drive it. You can't build a race car without knowing how to change a tire; otherwise you could build a car that runs like shit on the track. But mechanical engineers don't generally race cars, so they hand it over to mechanics and drivers to deal with running it on the track. And in doing so, they teach the mechanics and drivers all the insides and outs of the car. If they didn't, the car would never leave the garage, much less win a race. Imagine a random Ford mechanic or your cousin Bobby at the helm of an F1 car. People who build systems should know how to run them. However, if people building systems don't know how to run them, they must teach whoever is running them how it was built and how it works inside and out. If anything significant changes, the mechanics need to know. And if something fails on the track, or even some unexpected brake fade, the engineers need to know so they can redesign their parts. NASCAR, Formula 1, Le Mans, etc would be completely impossible otherwise. When the cars fail, the mechanics would be constantly shouting at the engineers pulling teeth trying to figure out how to fix the car in the middle of the race. There is no time for that madness. Toyota also builds their cars by having a continuous improvement cycle between engineers, the people assembling the cars, and service technicians. Toyota has basically been doing DevOps for 70 years (if you don't know the TPS system, you don't know dick about DevOps). When I was fixing my brakes the other day, I ran into eight different operations I need to do properly. I needed to use a breaker bar and torch to unseat stuck bolts, because banging a wrench with a hammer strips bolts (guess how I know). I needed to use a custom tool to push the caliper piston back for the rear piston, because it requires twisting to reseat rear pistons. I needed anti-seize put on the rotor and wheel hub surfaces to prevent them sticking the next time I remove them. I needed to remove the caliper slide pins, clean them of dirt, and re-grease them, to prevent uneven/premature brake failure. I needed to grease the brake pad slides as well as the caliper contact points (without getting any on the pad material). I needed to torque the bolts to the right specification (the front and rear bolts take different torque). I needed to bleed the brakes with a custom appliance to remove air bubbles from the lines. And I needed to properly bed the brakes. All of those operations are necessary to prevent failure. But if I'd never learned about all the failure modes, I would have just slapped it all together and the brakes probably would have failed soon after. (The brakes will still fail eventually, I just control how soon it happens) There is no getting around the knowledge gap. There is no easy way. If you're not all on the same page, you can ship a new car, but it's likely to be a lemon.
• nimbius 5y ago
  seems to me the real point of devops was employers trying to mash together two expensive categories of labor in a desparate effort to control overhead in a nation that spent 40 years prioritizing blind consumerism, reality television and talent shows over public education and STEM. mashing together devs and ops inevitably results in schisms of devsecops and netops and ^ops because developers have enough shit to worry about on the daily without needing to learn the absolute intricacies of the OS as it applies to dev and prod. employers tried to get them around this by insisting "move fast and break things" with the push-to-prod mentality only to realize a 7 hour outage as a result kindof dims the outlook for everyone, shareholders included. so employers wound up embracing the buzz and holding the line where it made sense, with some devops as developers, other devops as mostly ops, and still more devops as network-centric roles that handle things like firewalls and traffic too. now, people are going to point to automation and say "infra as code means devops is still a departure from traditional ^ops" but heres the secret: shit like cfengine has been around since 1993 and grandstand conferences on devops didnt show up until nearly 16 years later, and hyperconvergence showed up in 2009 which basically expands on software defined network and infra, so a cogent argument can be made that automation/convergence alone isnt devops, it was always happening anyways...so theres the wizard behind the curtain: devops was just a trick to get you to do more so your boss didnt have to hire network and ops.
  • pas 5y ago
    yes but no. the problem was that in mostly big financially privilegized companies both ops, dev, net, itsec were it's own little fiefdoms. and there was no incentive to get shit done. because that would have required working with the enemy. that gets you nowhere on your way up the corporate ladder in your guild. lack of education never helps, but it's laughable to point at that as a serious causally significant factor. outsourcing, communication barriers, trust barriers, organization barriers, ossification of the business structure led to serious loss of productivity of the whole enterprise. the answer is the cross-functional team. agile. scrum. (and many names and many crazy offshoots) but the core recipe is good and sane: people above process. trust (and self-check) instead of mandatory responsibility separation to check each other. empower engineers to be able to do the right thing by default, engineer tools for everything, don't let non-tooled non-integrated handoff points develop, because that leads to fiefdoms and communication bottlenecks.
• gfodor 5y ago
  Hilariously that is not the original purpose of "DevOps", and double hilariously, it is the direct opposite: that Dev and Ops (separate groups) should work together better: https://newrelic.com/blog/nerd-life/devops-name https://newrelic.com/blog/nerd-life/devops-name
• solatic 5y ago
  Ayyy, even the author gets it wrong. DevOps is not about letting developers own services in production. This is the common misunderstanding that has lead to DevOps teams be platform teams. DevOps is about helping separate Development and Operations teams work better together so that the organization, which needs both, can benefit from their harmony. The simple fact of the matter is that the modern stack is too big. It's not possible to expect to be able to hire someone who can walk in and do their own product management + write frontend code + backend code + host the frontend on a CDN + own the backend on Kubernetes + handle all the security, monitoring, uptime requirements around the clock. Maybe over time people will expand their skillsets, but right on their first day? No way. Not a reasonable expectation. Organizations need to allow people to specialize. This is the only way anybody ever truly gets productive, they learn some part of the stack, learn in deeply, handle that part of the stack for the company, and communicate. True DevOps isn't pipeline engineering / automation engineering, it's corporate communication engineering, so that the concerns of people later in the value stream (ops, security) can be handled as early as possible (e.g. development/staging environments) to improve overall throughput. You still need dedicated ops engineers! We just call them SRE now. You still need dedicated security engineers! We still call them security engineers.
  • ghosty141 5y ago
    Exactly, this is a huge problem. A lot of developers are exactly that, developers. Sure if you have somebody who is very motivated he might know a thing or two about DevOps etc. but hiring becomes almost impossible if you expect a candidate to be a good developer + a good knowledge of devops. It's just too much to handle for one person.
  • pas 5y ago
    Letting/making devs own the services they deploy/develop is not the full picture, nor is it a real goal, but still it sums up the problem and solution space concisely. Yes, specialization is a must for productivity, but that's a given already in the old model. What the DevOps effort tries to cut through is the trust barrier between the two. Sure, it's communications engineering too, but it's easy to weasel out of any real change with that phrasing.
• User23 5y ago
  Hiring Werner’s little sweatshop from Cornel truly changed the world. It’s funny how much shit Jesse caught from recruiting for pulling off that coup.
• netfortius 5y ago
  Why are we stuck on [made up] words? Jobs shall evolve alongside technology, and shall be carried out as needed to attain objectives. I've been in the IT field (mostly infra and security) for over 25 yrs, with the last ten including the public cloud, and I have witnessed solutions and JDs changing accordingly, including birth of DevOps, NetDevOps, NetSecOps, etc., etc., but in the end all that mattered was what one needed to achieve, by using what specific tools, within which boundaries one had to work, and what was to be delivered. As my first boss in IT told me, in 1995, when he promoted me as Manager Infrastructure & Security Services: "<my_name>: You could call yourself anything you want, including Vice President of Ice Cream, and I will order business cards for you as such, but here is what I expect from you in this role, and here is how much I am going to pay you, to see it done."
• darkwater 5y ago
  To be honest, I never understood how this "DevOps vision" could scale up to an engineering department bigger than, let say, 10-15 people. At some point there are enough things to do to keep the platform working by itself to move dedicated hours from business code to platform/infrastructure code. So you can decide if you use developers time or bring in someone dedicated to this. But maybe all the developers like the OP are 10x developers that can just do everything without skipping a beat.
• tripleSex 5y ago
  I haven't chosen a software niché to focus into and am curious about Development Operations -- anybody know any internet web-sites that have ,,Leetcode''-like exercises/education? Or even books and technologies I should study would be helpful, but obviously, content recommendations which are more structured would be more useful.
  • pxeger1 5y ago
    niché?
    • tripleSex 5y ago
      Yeah, you are right pxeger1 1: niche (/NEESH/) doesn't have accent mark. (In French, the e without an accent mark is silent. According to some person on Quora . . . )
      • l9k 5y ago
        Silent at the end of a word only (and when it is part of a conjugaison or accord suffix such as -es or -ent). Also, depending on the accent of a person, the ending "e" can be pronounced, but it's rare.
• lbriner 5y ago
  Lots of comments seem to get something confused. Devops doesn't mean you won't have specialist ops teams, but it also doesn't mean that every deployment has to go via the ops team. Devops is: Moving value quickly from left to right Getting feedback quickly Dynamically changing your process to avoid past problems in the future The key point is that to achieve this to any degree, the developers must be empowered to get their code into production quickly, without needing anyone else to create a gate. You might well need some specialists to setup and maintain e.g. kubernetes, VMs whatever but the end-game is that what is left is automated and repeatable deployments that mean devs get stuff deployed. It also means that a lot more of this can (and should) be maintained by developers. A developer should be able to modify a Jenkins pipeline as much as/more easily/reliably than an ops person so they should be. Why? Get stuff quickly from left to right and don't wait for others to do it.
• arthur6667 5y ago
  After working in dev, ops & devops roles for over 10 years, I just want to add this: Expertise and specialization is not a bad thing. Sure, you can just chuck everything on a managed k8s solution, PaaS or whatever, but that does not mean you can build efficient application images; write correct and secure nginx configurations etc. Maybe you can rely on what the platform provides, but I've never seen a platform that is acctually usable in production without customization layers. Anyway, my point is, it is ok to have specialists, like for Ops and tooling. I do not & will not expect a frontend developer to be able to write a efficient docker image or how to ensure assets are properly rolled over during deploys. It simply does not scale.
  • thecleaner 5y ago
    Why would a frontend developer need to build a docker image ?
    • 2ion 5y ago
      It's "why would a frontend developer build inside a docker image". You wouldn't believe the absurd problems in npm dependency chains when you try to build natively on macos and windows and linux. Especially "native code" in npm deps is not always the best quality and tested on all platforms it is declared to support. Better to build in Linux and run in Linux all the time (via containers or VMs).
• hzlatar 5y ago
  I tend to agree with the author of the article. Somewhere along the way, term devops changed its meaning. Initially, devops symbolized a way of working and a multi-disciplinary team. Nicely captured in [1]: > Under a DevOps model, development and operations teams are no longer “siloed. Sometimes, these two teams are merged into a single team where the engineers work across the entire application lifecycle, from development and test to deployment to operations, and develop a range of skills not limited to a single function. Today, devops is often used to describe the engineering role very similar to the classic system administrator. I guess this is mostly because it sounds "more modern" and it's more attractive. But, the comments in this thread made me aware that there indeed is a role that's focused only on serving developer needs. I guess we could call an engineer a devops engineer if their job is only to build and maintain system tools for developers. [1] https://aws.amazon.com/devops/what-is-devops/ https://aws.amazon.com/devops/what-is-devops/
• therealbilly 5y ago
  I'm personally waiting for the whole DevOps thing to revert back to what we had before. I come from the mainframe world. In those days, there was real trust in the developers and we migrated our own code. We had testers and we couldn't move stuff until they verified our work. The intentions were good for implementing DevOps but I think it's gotten out of hand. It's just too complex. Just my opinion.
• AtNightWeCode 5y ago
  For as long as I can remember there have been two widely used definitions of DevOps. One is the one large private corps use where it means developers and operators work close together in the same team. The other one is the cloud thing where DevOps is a role responsible for both development and operations. “You build it, you own it, you run in”, is more or less a scam from the cloud providers cause historically it was very difficult for none-developers to troubleshoot any issues in the cloud. The mantra should be “You build it, you own it, anybody operates it”.