5 ms·

Is there trouble brewing in the land of DevOps?

30 points by EwanToo 11y ago

dkarapetyan 11y ago
Nothing new here really. Shitty architecture and practices are shitty architecture and practices. If you're not disciplined enough about how you package, deploy, and manage the lifecycle of your software then putting the blame on technology X or methodology Y will not change anything. The root cause of the problem is never the technology really. The root cause is almost always the human processes that lead to creating poorly documented and buggy software and sprinkling magic DevOps, agile, or whatever dust over things doesn't make a difference no matter how much advocates would like to believe otherwise. A concrete example is that even before containers I saw plenty of binaries that were magical black boxes that no one knew how to re-create from source anymore, well not without some human intervention anyway by poking and prodding things at certain points during the compilation process. So to say containers and microservices are making things worse or better misses the mark. Things are the same as they've ever been and the more disciplined software shops are still doing just fine. The less disciplined software shops are still finding plenty of ways to blow off their feet (referencing a quote about C++). DevOps, agile, lean, six sigma, or any other methodology has nothing to do with it.
- bbrazil 11y ago
  A phenomenon I've noticed is some believing that containers solves configuration management. It's true that containers through being immutable means you don't have to worry about what cruft has built up from previous changes (which is a big win). That still leaves you with the core problem of how you manage your configs. The core of configuration management is a) how you express the state of the world and b) how you move from state to state safely, with potentially hundreds of changes in flight at once. For the former I'm excited about projects like https://github.com/apmasell/flabbergast https://github.com/apmasell/flabbergast, and the latter is mostly a matter of breaking rollouts into individually rollbackable changes combined with good policy.
  - leg100 11y ago
    More specifically I've noticed less reliance on configuration management frameworks like puppet and chef. If you're building a container cloud, consisting of 'immutable' servers and a scheduler like kubernetes, then there's not much of a role left for one. You might use the lightweight versions (e.g. chef-solo) to bootstrap the servers, bake an AMI (or god forbid a container image). And you'll use something like cloudformation or terraform to configure the cloud. But if I'm going to toe the immutable infrastructure line I'm not going to use a config mgmt tool to update the state of my existing servers.
  - andybak 11y ago
    I've only just glanced at flabbergast but in terms of readability at least - it seems far short of something like Ansible https://github.com/apmasell/flabbergast/blob/master/flabbergast-manual.md https://github.com/apmasell/flabbergast/blob/master/flabberg... (I'm not arguing that they solve the same problem - I really have no idea - but readability matters and choosing a JSON syntax over YAML or similar when humans are meant to be parsing it seems like a mistake)
    - bbrazil 11y ago
      Flabbergast is solving a different but related problem, it's about expressing the state of your system. Lots of times you want to say the dev environment is like the prod environment, except for these 3 things. And by the way, anything in the European region needs different settings too that interact with dev/prod. I'm a fan of Ansible, but it isn't really suitable for expressing multi-dimensional and multiple-levels of exceptions, from experience it works for about 2. How that state is actually rolled out is the job of something else (which could be Ansible or the update portion of your cluster scheduler).
      - andybak 11y ago
        Fair enough but does that affect my thoughts on readability? YAML is actually a superset of the capabilities of JSON so as far as I know there's no inherent advantage to choosing JSON as the syntax.
- blackbeard 11y ago
  You're 100% right about discipline. I work for a very undisciplined and border-line incompetent company that decided to roll out a microservices architecture, continuous delivery and devops culture into a behemoth enterprise product. This has simply been a disaster because the incumbent methodology has been hack it until it mostly works. However, with respect to technology, some technologies do not lend themselves to these processes. .Net I'm looking at you here; you're a tangled build nightmare, a deployment clusterfuck, require too many resources to lift light tasks and most of your core as API is a mess of obsolete ideas,impossible to test abstractions and a bunch of dubious hand grenades (ReaderWriterLockSlim for example which has deadlocked on me numerous times and is the basis for concurrent collections).
  - rjbwork 11y ago
    We do just fine w/ DevOps on .NET. We auto build on push, publish a versioned Octopus Deploy package to our TC NuGet feed,and deploy to our dev environment many times every day. We haven't had to actually touch our build/deployment process in over 4 months, except to click the single button to move from environment to environment. It really is fantastic.
    - blackbeard 11y ago
      That works for simple deployments but when you start having to deal with signing, licensed components and stuff like ours (55 deployable packages) then it gets hairy. Also managing windows infrastructure is hell.
      - rjbwork 11y ago
        Fair, though we've got a few things now. We're all on azure and all our stuff is @ platform level, and built to throw away, so it's a bit different.
        blackbeard 11y ago
        Ok you're lucky to be on Azure PaaS!
- zwischenzug 11y ago
  I agree with you, but containerization gave us the ability to counter 15 years of technical debt. I talk a little about this here: https://www.youtube.com/watch?v=zVUPmmUU3yY https://www.youtube.com/watch?v=zVUPmmUU3yY Doing it, though, requires attention to exactly those things you outline. But the technology can make it a more tractable problem, solve-able in this case by three people working in their spare time.
  - dkarapetyan 11y ago
    ShutIt is great. The reason I say that is because I've already built 3 of these things with a similar philosophy, which is to say that bash scripts are perfectly adequate when coupled with some high-level orchestration/lifecycle mechanism that is managed with Ruby or Python. I'm definitely going to give ShutIt a try because my next gig is a Python shop and I'm not really interested in writing another custom tool. You should definitely evangelize this tool more because every single shop builds something like this but it never ends up being open sourced. That's a lot of wasted effort across the board.
- digi_owl 11y ago
  Yep. The more i read about containers and devops, the more it feels like its about making sysadmins subservient to programmer whims rather than custodians of server stability and availability. Or perhaps a "webification" of software development in general. Or perhaps one can liken it to the old story about the turtle and the hare. With programmers as the hare...
kylequest 11y ago
There's always been a gap between the devs and ops. And one of the core side effects of this gap is the knowledge transfer problem. With the devpops movement you have devs doing production deploys where the ops (or even other devs) might not know exactly what's running out there: "some IT landscapes contain components that people don't know how to recreate especially since the person that created the component has left the company" , "if we consider multiple layers of microservices then it can be quite disconcerting that some of those lower order component layers maybe in a position that we cannot recreate". The knowledge transfer problem between the devs and ops is not really a technical problem. It's a human communication problem. With the new tools (like containers and microservices) it manifests itself in a slightly different way allowing you to deal with the communication problems further down the line (because you can deploy a black box container without breaking other things running on the same machine/VM), but you still need to transfer that knowledge. I'm curious if there's a market for tools and products that automate this knowledge transfer...
stuaxo 11y ago
This article was quite hard to read beyond the pretty wanky diagram: Culture: Inertia vs Fluid / Gameable Corporate Focus: Profit / Disruption I'm sorry, someone has been reading far too many startup articles, it's like a parody.
jldugger 11y ago
Buried deep within the self-fullfilling analysis, corporate buzz words, and stale metaphors is a simple concept: the early adopters of containers implicitly embrace NoOps, and eventually the absence of explicit dependencies and documentation will drag organizations down.

30 points by EwanToo 11y ago

Nothing new here really. Shitty architecture and practices are shitty architecture and practices. If you're not disciplined enough about how you package, deploy, and manage the lifecycle of your software then putting the blame on technology X or methodology Y will not change anything. The root cause of the problem is never the technology really. The root cause is almost always the human processes that lead to creating poorly documented and buggy software and sprinkling magic DevOps, agile, or whatever dust over things doesn't make a difference no matter how much advocates would like to believe otherwise. A concrete example is that even before containers I saw plenty of binaries that were magical black boxes that no one knew how to re-create from source anymore, well not without some human intervention anyway by poking and prodding things at certain points during the compilation process. So to say containers and microservices are making things worse or better misses the mark. Things are the same as they've ever been and the more disciplined software shops are still doing just fine. The less disciplined software shops are still finding plenty of ways to blow off their feet (referencing a quote about C++). DevOps, agile, lean, six sigma, or any other methodology has nothing to do with it.

bbrazil 11y ago
A phenomenon I've noticed is some believing that containers solves configuration management. It's true that containers through being immutable means you don't have to worry about what cruft has built up from previous changes (which is a big win). That still leaves you with the core problem of how you manage your configs. The core of configuration management is a) how you express the state of the world and b) how you move from state to state safely, with potentially hundreds of changes in flight at once. For the former I'm excited about projects like https://github.com/apmasell/flabbergast https://github.com/apmasell/flabbergast, and the latter is mostly a matter of breaking rollouts into individually rollbackable changes combined with good policy.
- leg100 11y ago
  More specifically I've noticed less reliance on configuration management frameworks like puppet and chef. If you're building a container cloud, consisting of 'immutable' servers and a scheduler like kubernetes, then there's not much of a role left for one. You might use the lightweight versions (e.g. chef-solo) to bootstrap the servers, bake an AMI (or god forbid a container image). And you'll use something like cloudformation or terraform to configure the cloud. But if I'm going to toe the immutable infrastructure line I'm not going to use a config mgmt tool to update the state of my existing servers.
- andybak 11y ago
  I've only just glanced at flabbergast but in terms of readability at least - it seems far short of something like Ansible https://github.com/apmasell/flabbergast/blob/master/flabbergast-manual.md https://github.com/apmasell/flabbergast/blob/master/flabberg... (I'm not arguing that they solve the same problem - I really have no idea - but readability matters and choosing a JSON syntax over YAML or similar when humans are meant to be parsing it seems like a mistake)
  - bbrazil 11y ago
    Flabbergast is solving a different but related problem, it's about expressing the state of your system. Lots of times you want to say the dev environment is like the prod environment, except for these 3 things. And by the way, anything in the European region needs different settings too that interact with dev/prod. I'm a fan of Ansible, but it isn't really suitable for expressing multi-dimensional and multiple-levels of exceptions, from experience it works for about 2. How that state is actually rolled out is the job of something else (which could be Ansible or the update portion of your cluster scheduler).
    - andybak 11y ago
      Fair enough but does that affect my thoughts on readability? YAML is actually a superset of the capabilities of JSON so as far as I know there's no inherent advantage to choosing JSON as the syntax.
blackbeard 11y ago
You're 100% right about discipline. I work for a very undisciplined and border-line incompetent company that decided to roll out a microservices architecture, continuous delivery and devops culture into a behemoth enterprise product. This has simply been a disaster because the incumbent methodology has been hack it until it mostly works. However, with respect to technology, some technologies do not lend themselves to these processes. .Net I'm looking at you here; you're a tangled build nightmare, a deployment clusterfuck, require too many resources to lift light tasks and most of your core as API is a mess of obsolete ideas,impossible to test abstractions and a bunch of dubious hand grenades (ReaderWriterLockSlim for example which has deadlocked on me numerous times and is the basis for concurrent collections).
- rjbwork 11y ago
  We do just fine w/ DevOps on .NET. We auto build on push, publish a versioned Octopus Deploy package to our TC NuGet feed,and deploy to our dev environment many times every day. We haven't had to actually touch our build/deployment process in over 4 months, except to click the single button to move from environment to environment. It really is fantastic.
  - blackbeard 11y ago
    That works for simple deployments but when you start having to deal with signing, licensed components and stuff like ours (55 deployable packages) then it gets hairy. Also managing windows infrastructure is hell.
    - rjbwork 11y ago
      Fair, though we've got a few things now. We're all on azure and all our stuff is @ platform level, and built to throw away, so it's a bit different.
      - blackbeard 11y ago
        Ok you're lucky to be on Azure PaaS!
zwischenzug 11y ago
I agree with you, but containerization gave us the ability to counter 15 years of technical debt. I talk a little about this here: https://www.youtube.com/watch?v=zVUPmmUU3yY https://www.youtube.com/watch?v=zVUPmmUU3yY Doing it, though, requires attention to exactly those things you outline. But the technology can make it a more tractable problem, solve-able in this case by three people working in their spare time.
- dkarapetyan 11y ago
  ShutIt is great. The reason I say that is because I've already built 3 of these things with a similar philosophy, which is to say that bash scripts are perfectly adequate when coupled with some high-level orchestration/lifecycle mechanism that is managed with Ruby or Python. I'm definitely going to give ShutIt a try because my next gig is a Python shop and I'm not really interested in writing another custom tool. You should definitely evangelize this tool more because every single shop builds something like this but it never ends up being open sourced. That's a lot of wasted effort across the board.
digi_owl 11y ago
Yep. The more i read about containers and devops, the more it feels like its about making sysadmins subservient to programmer whims rather than custodians of server stability and availability. Or perhaps a "webification" of software development in general. Or perhaps one can liken it to the old story about the turtle and the hare. With programmers as the hare...

kylequest 11y ago

There's always been a gap between the devs and ops. And one of the core side effects of this gap is the knowledge transfer problem. With the devpops movement you have devs doing production deploys where the ops (or even other devs) might not know exactly what's running out there: "some IT landscapes contain components that people don't know how to recreate especially since the person that created the component has left the company" , "if we consider multiple layers of microservices then it can be quite disconcerting that some of those lower order component layers maybe in a position that we cannot recreate". The knowledge transfer problem between the devs and ops is not really a technical problem. It's a human communication problem. With the new tools (like containers and microservices) it manifests itself in a slightly different way allowing you to deal with the communication problems further down the line (because you can deploy a black box container without breaking other things running on the same machine/VM), but you still need to transfer that knowledge. I'm curious if there's a market for tools and products that automate this knowledge transfer...

stuaxo 11y ago

This article was quite hard to read beyond the pretty wanky diagram: Culture: Inertia vs Fluid / Gameable Corporate Focus: Profit / Disruption I'm sorry, someone has been reading far too many startup articles, it's like a parody.

jldugger 11y ago

Buried deep within the self-fullfilling analysis, corporate buzz words, and stale metaphors is a simple concept: the early adopters of containers implicitly embrace NoOps, and eventually the absence of explicit dependencies and documentation will drag organizations down.