5 ms·

Quit Covering Up Your Toxic Hellstew with Docker

124 points by cyberpanther 12y ago

jradd 12y ago
Very valuable concept regurgitated with [flavour–of–the–year] Docker as the focus. The article seems to portray Docker as the _cause_ of this anti–pattern without any sort of context for why Docker and not xyz or why not your implementation of xyz.
merb 12y ago
Testing of environments is needed on / with docker as well...
killertypo 12y ago
this makes sense, i work in a pretty convoluted SharePoint environment. It is completely impossible to spin up a development environment without dozens of scripts and knowing exactly what lists to manually create and what data must be present inside of them. This means that new-hires are handed a cryptic and seriously out-of-date document with instructions on how to setup a proper VM environment. They check out their code. They deploy...but wait the deploy fails because of missing data, missing document types, missing lists... Open up SharePoint, enable some doc types not turned on by default, turn on some more features, add a list. Deploy again, no wait it died again, oh now it's a different doc type and a different feature that the deployment doesn't turn on by default ...etc The end result is a mess that requires days to get up and running, not hours.
- hawkice 12y ago
  > i work in a pretty convoluted SharePoint environment For some reason I intuited that you were not going to talk about how well the organization has dealt with said complexity. I'm curious if there is any way to FFI into sharepoint and start using tools that don't actively sabotage their users.
  - bdamm 12y ago
    I'd like to know if anyone has anything nice to say about Sharepoint. As a user I find it miserable in all browsers, and have obviously avoided doing anything with it as a developer.
    - dragonwriter 12y ago
      > I'd like to know if anyone has anything nice to say about Sharepoint. Its marginally less of a total failure for issue tracking than Excel. I'm not sure that that counts as something nice.
    - hawkice 12y ago
      It's better than staring into the computer as if it were a dark abyss, screaming and longing for a way to build something. It's easy to discount how useful this is for people who really have no tools or means otherwise (not that more specialized services like, without loss of generality, Wufoo, aren't way better). The general discontent has to do (mostly) with 'Sharepoint Engineers', which is a bit like having someone build a house with legos.
      - xorcist 12y ago
        I guess a regular wiki is too much to ask for... (For the omg-cryptic-markup crowd there is always Google Docs. I seriously wonder why that hasn't crushed all trivial Sharepoint instances yet.)
    - pp19dd 12y ago
      Edward Snowden's last job was a Sharepoint administrator, IRC.
  - killertypo 12y ago
    So our answer has to just been to hire more developers, more and more and more developers. As the complexity grows, so does headcount. This is why I thank the stars that I am the opensource developer, focusing on linux / Python / Django development. I cross over into SharePoint only when we begrudgingly use it as a CMS for a windows 8 app or similar. There is an effort now to write an external service bus to manage a lot of the business logic complexity that we've baked into SharePoint. With the eventual goal of having SharePoint and other resources leverage this SB and use it instead.
dkarapetyan 12y ago
Yes, simplicity leads to understanding and I don't understand why more people don't get this simple concept. I've dealt with codebases with such a horrendous build process that it doesn't matter what kind of sugar you sprinkle on top because making any change is practically impossible. That complexity has to live somewhere and if you offload it to the docker buildfile it's still in the buildfile. The problem at the end of the day comes down to the fact that most developers either don't understand enough to build proper build pipelines or they are lazy or they don't think complexity in the build pipeline is anything to worry about. Docker does not change those things.
- bdamm 12y ago
  What if the problem actually is complex? Let's say that I'm building an embedded firmware module for a chipset that can only be compiled with a proprietary compiler under Visual Studio - and you're a Unix shop with Bamboo as the continuous integration server? Then that means the Unix bamboo server needs a Windows agent that can execute the cross-compiler inside a MS VC environment. It certainly is complex, but what piece are you going to change?
  - fragmede 12y ago
    Honestly? Port it to GCC. (Easier than LLVM, IMO) That, or switch to a different chipset, this time taking care to select something that includes unix build tools. Neither of those is trivial, nor are they cheap. But rewriting their simulator in C, as described in the article, couldn't have been cheap either.
    - new299 12y ago
      Perhaps the instruction set is undocumented? (it happens). Alternative chipsets cost x10 (also happens, and is probably correlated to documentation quality). Price is an important factor, and the messy choice is often not unreasonable.
  - plorkyeran 12y ago
    Why are you a unix shop if using Visual Studio is a hard requirement? The complexity is entirely due to you choosing tools that work poorly together, so instead pick tools that work well with the ones you can't change.
opendais 12y ago
The problem is the carpenter, not the hammer. Please stop blaming the hammers in your headlines. The tool was used incorrectly and it resulted it badness. Bringing Docker into it is pointless. This would have happened with Salt, Fabric, Chef, Puppet, etc. with the same team.
- imanaccount247 12y ago
  Nobody blamed hammers, or said they were bad. He said stop using hammers to smash other people's thumbs.
- andrewflnr 12y ago
  Even in the headline, it was obvious that Docker wasn't the problem. In the article itself it was abundantly clear that Docker was just the misused hammer closest to the front of the author's mind. It's almost as much a defense of Docker from abusers as anything else.
ilaksh 12y ago
Actually the more complex it is, the more beneficial encapsulation can be. I think maybe I know what his actual problem is. It didn't detect a change to the requirements.txt, or it is _always_ detecting a change. In the Dockerfile you want to ADD your requirements.txt first, then RUN the pip install, then ADD . http://stackoverflow.com/questions/25305788/how-to-avoid-reinstalling-packages-when-building-docker-image-for-python-project http://stackoverflow.com/questions/25305788/how-to-avoid-rei... Also check for a --no-cache
- jessaustin 12y ago
  Thanks for this comment! I read this whole thread primarily to find this information.
r0s 12y ago
This reminds me of the "Every dev should be senior" mindset that is all too common in this industry. I don't see how supposing every devOps specialist is replaceable by average engineers is a real solution.
- yummyfajitas 12y ago
  I think it's something different. "Junior devs shouldn't reinvent the wheel in production [1] until after they understand the solutions of the senior members of their field." I.e., don't use MongoDB until you fully understand SQL and don't build your own fancy planning algo until you you understand linear programming. [1] When learning, reinvent all the wheels.
  - jrockway 12y ago
    Also, don't use SQL until you fully understand MongoDB.
    - jwatte 12y ago
      And don't use MongoDB for data that matters ;-)
    - yummyfajitas 12y ago
      No. SQL is the culmination of years of research into data storage systems. It's a well established solution to a large number of problems. MongoDB simply isn't.
      - jrockway 12y ago
        So just use something you don't understand because someone said so. Got it.
        xorcist 12y ago
        No. Trust in what's battle tested in production. Use MongoDB for all your pet projects, all your development. Joining data from documents is good way to learn where SQL came from. Just don't do it in production before you master it.
scrumper 12y ago
I read the opening paragraph and thought 'ah well, another boring didactic angry developer rant.' Just as I was about to close the tab, my eye caught the start of the second paragraph: This reminds me of my days in the Space Shuttle program. Which, to put it mildly, is something of a credibility boost. So I finished the article.
- jdub 12y ago
  I enjoyed that brain reset feeling. I think I'm going to borrow the phrase... "You think a reboot is going to help? This reminds me of my days in the Space Shuttle program."
  - doug1001 12y ago
    an attention-getting phrase, but not the best one in the post as far as i am concerned--i'm going through our g/h issues just now and adding comments like "poeple let's fix this toxic hellstew!"
- TylerE 12y ago
  Is it? I mean, yes, it's literally rocket science, but the shuttle program fell well short of it's reliability goals.
  - TylerE 12y ago
    Don't see why I'm getting downvoted here, honestly. Even NASA management has admitted that the Shuttle program fell well short of the planned goals, and left us with much worse capabilities than if we had followed the "reusable thingy with wings" pipedream.
    - teraflop 12y ago
      None of the reliability or capability issues with the Space Shuttle were software-related, as far as I know, so I'm not sure how your objection is relevant. The Shuttle is generally regarded as having one of the most complex and quality-focused software design efforts in history. http://www.fastcompany.com/28121/they-write-right-stuff http://www.fastcompany.com/28121/they-write-right-stuff
contingencies 12y ago
The article is essentially discussing frustration at the hiding - by abstraction - of technical debt incurred in adopting poor software architecture and/or development process. Some potentially relevant quotes[1]: Zymurgy's First Law of Evolving Systems Dynamics: Once you open a can of worms, the only way to recan them is to use a larger can. Ducharm's Axiom: If you view your problem closely enough you will recognize yourself as part of the problem. The organization of the software and the organization of the software team will be congruent. (paraphrasing of Conway's Law) Separation of concerns ... a necessary consequence of loss of resolution due to scale ... a strategy for staying sane. (Mark Burgess, In Search of Certainty: The Science of Our Information Infrastructure, 2013) [1] Taken from my fortune clone https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup
Rapzid 12y ago
I realize a lot of people use docker for a lot of different reasons.. But the technologies seem more geared toward(and seeing as dotcloud and not docker are PaaS companies) solving deployment concerns..
gioele 12y ago
I have been working recently with a PHP application whose installation instructions are "Run this VirtualBox image inside your network, and without a proxy in front of it because we are not properly configured". This PHP application is not trivial, but also not very complex. Yet its developers do not provide this application as a PEAR package, and replied that this kind of things are superfluous these days, VM are simpler. People like these developers fail to understand that there is a lot more in a VM than just their software (from the kernel to all the exposed services) and that by distributing a VM they are also becoming the maintainers of a very complex set of dependencies. Not that they care: it took them two months to release a VM not vulnerable to heartbleed. Let's see how long before they release a VM not vulnerable to shellshock.
mrfusion 12y ago
I don't know much about docker. What is he saying I makes easier?

124 points by cyberpanther 12y ago

jradd 12y ago

Very valuable concept regurgitated with [flavour–of–the–year] Docker as the focus. The article seems to portray Docker as the _cause_ of this anti–pattern without any sort of context for why Docker and not xyz or why not your implementation of xyz.

merb 12y ago

Testing of environments is needed on / with docker as well...

killertypo 12y ago

this makes sense, i work in a pretty convoluted SharePoint environment. It is completely impossible to spin up a development environment without dozens of scripts and knowing exactly what lists to manually create and what data must be present inside of them. This means that new-hires are handed a cryptic and seriously out-of-date document with instructions on how to setup a proper VM environment. They check out their code. They deploy...but wait the deploy fails because of missing data, missing document types, missing lists... Open up SharePoint, enable some doc types not turned on by default, turn on some more features, add a list. Deploy again, no wait it died again, oh now it's a different doc type and a different feature that the deployment doesn't turn on by default ...etc The end result is a mess that requires days to get up and running, not hours.

hawkice 12y ago
> i work in a pretty convoluted SharePoint environment For some reason I intuited that you were not going to talk about how well the organization has dealt with said complexity. I'm curious if there is any way to FFI into sharepoint and start using tools that don't actively sabotage their users.
- bdamm 12y ago
  I'd like to know if anyone has anything nice to say about Sharepoint. As a user I find it miserable in all browsers, and have obviously avoided doing anything with it as a developer.
  - dragonwriter 12y ago
    > I'd like to know if anyone has anything nice to say about Sharepoint. Its marginally less of a total failure for issue tracking than Excel. I'm not sure that that counts as something nice.
  - hawkice 12y ago
    It's better than staring into the computer as if it were a dark abyss, screaming and longing for a way to build something. It's easy to discount how useful this is for people who really have no tools or means otherwise (not that more specialized services like, without loss of generality, Wufoo, aren't way better). The general discontent has to do (mostly) with 'Sharepoint Engineers', which is a bit like having someone build a house with legos.
    - xorcist 12y ago
      I guess a regular wiki is too much to ask for... (For the omg-cryptic-markup crowd there is always Google Docs. I seriously wonder why that hasn't crushed all trivial Sharepoint instances yet.)
  - pp19dd 12y ago
    Edward Snowden's last job was a Sharepoint administrator, IRC.
- killertypo 12y ago
  So our answer has to just been to hire more developers, more and more and more developers. As the complexity grows, so does headcount. This is why I thank the stars that I am the opensource developer, focusing on linux / Python / Django development. I cross over into SharePoint only when we begrudgingly use it as a CMS for a windows 8 app or similar. There is an effort now to write an external service bus to manage a lot of the business logic complexity that we've baked into SharePoint. With the eventual goal of having SharePoint and other resources leverage this SB and use it instead.

dkarapetyan 12y ago

Yes, simplicity leads to understanding and I don't understand why more people don't get this simple concept. I've dealt with codebases with such a horrendous build process that it doesn't matter what kind of sugar you sprinkle on top because making any change is practically impossible. That complexity has to live somewhere and if you offload it to the docker buildfile it's still in the buildfile. The problem at the end of the day comes down to the fact that most developers either don't understand enough to build proper build pipelines or they are lazy or they don't think complexity in the build pipeline is anything to worry about. Docker does not change those things.

bdamm 12y ago
What if the problem actually is complex? Let's say that I'm building an embedded firmware module for a chipset that can only be compiled with a proprietary compiler under Visual Studio - and you're a Unix shop with Bamboo as the continuous integration server? Then that means the Unix bamboo server needs a Windows agent that can execute the cross-compiler inside a MS VC environment. It certainly is complex, but what piece are you going to change?
- fragmede 12y ago
  Honestly? Port it to GCC. (Easier than LLVM, IMO) That, or switch to a different chipset, this time taking care to select something that includes unix build tools. Neither of those is trivial, nor are they cheap. But rewriting their simulator in C, as described in the article, couldn't have been cheap either.
  - new299 12y ago
    Perhaps the instruction set is undocumented? (it happens). Alternative chipsets cost x10 (also happens, and is probably correlated to documentation quality). Price is an important factor, and the messy choice is often not unreasonable.
- plorkyeran 12y ago
  Why are you a unix shop if using Visual Studio is a hard requirement? The complexity is entirely due to you choosing tools that work poorly together, so instead pick tools that work well with the ones you can't change.

opendais 12y ago

The problem is the carpenter, not the hammer. Please stop blaming the hammers in your headlines. The tool was used incorrectly and it resulted it badness. Bringing Docker into it is pointless. This would have happened with Salt, Fabric, Chef, Puppet, etc. with the same team.

imanaccount247 12y ago
Nobody blamed hammers, or said they were bad. He said stop using hammers to smash other people's thumbs.
andrewflnr 12y ago
Even in the headline, it was obvious that Docker wasn't the problem. In the article itself it was abundantly clear that Docker was just the misused hammer closest to the front of the author's mind. It's almost as much a defense of Docker from abusers as anything else.

ilaksh 12y ago

Actually the more complex it is, the more beneficial encapsulation can be. I think maybe I know what his actual problem is. It didn't detect a change to the requirements.txt, or it is _always_ detecting a change. In the Dockerfile you want to ADD your requirements.txt first, then RUN the pip install, then ADD . http://stackoverflow.com/questions/25305788/how-to-avoid-reinstalling-packages-when-building-docker-image-for-python-project http://stackoverflow.com/questions/25305788/how-to-avoid-rei... Also check for a --no-cache

jessaustin 12y ago
Thanks for this comment! I read this whole thread primarily to find this information.

r0s 12y ago

This reminds me of the "Every dev should be senior" mindset that is all too common in this industry. I don't see how supposing every devOps specialist is replaceable by average engineers is a real solution.

yummyfajitas 12y ago
I think it's something different. "Junior devs shouldn't reinvent the wheel in production [1] until after they understand the solutions of the senior members of their field." I.e., don't use MongoDB until you fully understand SQL and don't build your own fancy planning algo until you you understand linear programming. [1] When learning, reinvent all the wheels.
- jrockway 12y ago
  Also, don't use SQL until you fully understand MongoDB.
  - jwatte 12y ago
    And don't use MongoDB for data that matters ;-)
  - yummyfajitas 12y ago
    No. SQL is the culmination of years of research into data storage systems. It's a well established solution to a large number of problems. MongoDB simply isn't.
    - jrockway 12y ago
      So just use something you don't understand because someone said so. Got it.
      - xorcist 12y ago
        No. Trust in what's battle tested in production. Use MongoDB for all your pet projects, all your development. Joining data from documents is good way to learn where SQL came from. Just don't do it in production before you master it.

scrumper 12y ago

I read the opening paragraph and thought 'ah well, another boring didactic angry developer rant.' Just as I was about to close the tab, my eye caught the start of the second paragraph: This reminds me of my days in the Space Shuttle program. Which, to put it mildly, is something of a credibility boost. So I finished the article.

jdub 12y ago
I enjoyed that brain reset feeling. I think I'm going to borrow the phrase... "You think a reboot is going to help? This reminds me of my days in the Space Shuttle program."
- doug1001 12y ago
  an attention-getting phrase, but not the best one in the post as far as i am concerned--i'm going through our g/h issues just now and adding comments like "poeple let's fix this toxic hellstew!"
TylerE 12y ago
Is it? I mean, yes, it's literally rocket science, but the shuttle program fell well short of it's reliability goals.
- TylerE 12y ago
  Don't see why I'm getting downvoted here, honestly. Even NASA management has admitted that the Shuttle program fell well short of the planned goals, and left us with much worse capabilities than if we had followed the "reusable thingy with wings" pipedream.
  - teraflop 12y ago
    None of the reliability or capability issues with the Space Shuttle were software-related, as far as I know, so I'm not sure how your objection is relevant. The Shuttle is generally regarded as having one of the most complex and quality-focused software design efforts in history. http://www.fastcompany.com/28121/they-write-right-stuff http://www.fastcompany.com/28121/they-write-right-stuff

contingencies 12y ago

The article is essentially discussing frustration at the hiding - by abstraction - of technical debt incurred in adopting poor software architecture and/or development process. Some potentially relevant quotes[1]: Zymurgy's First Law of Evolving Systems Dynamics: Once you open a can of worms, the only way to recan them is to use a larger can. Ducharm's Axiom: If you view your problem closely enough you will recognize yourself as part of the problem. The organization of the software and the organization of the software team will be congruent. (paraphrasing of Conway's Law) Separation of concerns ... a necessary consequence of loss of resolution due to scale ... a strategy for staying sane. (Mark Burgess, In Search of Certainty: The Science of Our Information Infrastructure, 2013) [1] Taken from my fortune clone https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup

Rapzid 12y ago

I realize a lot of people use docker for a lot of different reasons.. But the technologies seem more geared toward(and seeing as dotcloud and not docker are PaaS companies) solving deployment concerns..

gioele 12y ago

I have been working recently with a PHP application whose installation instructions are "Run this VirtualBox image inside your network, and without a proxy in front of it because we are not properly configured". This PHP application is not trivial, but also not very complex. Yet its developers do not provide this application as a PEAR package, and replied that this kind of things are superfluous these days, VM are simpler. People like these developers fail to understand that there is a lot more in a VM than just their software (from the kernel to all the exposed services) and that by distributing a VM they are also becoming the maintainers of a very complex set of dependencies. Not that they care: it took them two months to release a VM not vulnerable to heartbleed. Let's see how long before they release a VM not vulnerable to shellshock.

mrfusion 12y ago

I don't know much about docker. What is he saying I makes easier?