7 ms·

I've been tasked a couple of times trying to debug issues in 3rd party docker images. Every one of those images did something different. Placed configuration, b

by lykr0n 8y ago

I've been tasked a couple of times trying to debug issues in 3rd party docker images. Every one of those images did something different. Placed configuration, binaries, libraries, or logs in different locations. A simple task on a standard CentOS 7 VM that would take 5 minutes turns into 25 minutes with most of that googling ("Where the hell is your working directory?!"). It's not a fault of docker per se, but it's the result (I think) of the docker "it just works!" mentality. What's great about my examples is that two of the applications were single binary Go applications that could be downloaded and run on almost any linux box. But they were distributed with docker ... for reasons? Docker is great when I want to run a local Redis database in one command for development. But past that- ehhhh. Each new feature seems like Docker is just digging itself deeper.

jrs95 8y ago
I think a lot of that can also come from people mis-using Docker as if it were a VM instead of trying to do the smallest amount of stuff possible in an image. But at the same time, there's stuff I wouldn't even try to manage in containers in production. Databases and message queues in particular.
- mcbutterbunz 8y ago
  We've had great success with Docker and Kubernetes for our stateless apps. As a developer, one thing I absolutely love is being able to run an image without having to install anything (other than Docker). If I want to play around with something new, its easy to get started and easy to clean up.
  - sigjuice 8y ago
    Playing with something new should be as simple as unpacking a tarball and rm -rfing the directory when done. This should be possible in the comfort of my home directory and uid. If I need a whole separate copy of a computer's files just to try out something, then something very very fundamental is broken.
    - tarr11 8y ago
      How do you install dependencies in this model? As far as can tell, you end up in situations like this https://askubuntu.com/questions/193695/installing-packages-into-local-directory https://askubuntu.com/questions/193695/installing-packages-i...
    - DrScump 8y ago
      Playing with something new should be as simple as unpacking a tarball and rm -rfing the directory when done. Needing a single install script before, and uninstall script after is acceptable.
    - white-flame 8y ago
      Yes I agree things are fundamentally broken with these situations, and they're at the common OS design level.
    - crystaln 8y ago
      What it that something depends on a different operating system and libraries that conflict with yours? Also the world you describe as should is not the world we live in. I should be able to exist in the world without danger or undue prosection.
      - sigjuice 8y ago
        Docker containers primarily mean Linux userland. It doesn’t help you with different operating systems.
  - KaiserPro 8y ago
    Hosting stateless apps is simple. If the underlying host fails? don't worry, just spin up a few new copies. Slap a load balancer infront and we can do a/b or blue green. Smashing. But where is that state held? maybe your lucky and your data can be hosted in a single RDS instance. Maybe your partition scheme is simple. However, managing distributed state is hard_.
013a 8y ago
I think this is really a problem you see in every technology that tries to make things easier. JS. MongoDB. The list goes on. When a technology democratizes access to something that was previously only accessible to those with more specialized skillsets, it will attract people who don't know enough to know how to use it right. But, well, they're still using it. Thus, left-pad. Its uncovering something that was always there; incompetence is rampant in software engineering. Ask the average engineer to properly set up an Ubuntu box with CICD for deployment and they'll likely fail, even with all the help of the internet. Ask them to get a docker image onto a server and they can make it work. Wider access to technology is... well, I don't know if its a "good thing" or not. We as an industry pretty strongly believe in "fail fast"; we write tests to make sure our code works, we structure our startups to recognize and pivot around failure, and if that failure comes we stop and say "somethings fucked, stop what we're doing". So I do get scared when a technology enters the stage which allows hard failure to be delayed as long as possible. Maybe we need more technology to be harder, have opinions, and be willing to say "you're doing this wrong, just stop and go get an adult." Because the alternative... Equifax? Yahoo? Ashley Madison?
- lykr0n 8y ago
  My personal design philosophy (from a System Admin, SRE, DevOps, Ops- whatever) is no black boxes. I'm not talking about knowing the inns and outs of every little component in my infrastructure/system, but instead of saying "This is Graylog" being able to say "This is Graylog- a Java app that uses MongoDB to index and Elasticsearch as a backend. We have to monitor JVM usage." The former is saying we depend on a system we have no idea how it works, and the latter is saying we depend on 2 java applications and a NoSQL database. It horrifies me that people run stuff in production not knowing how it's configured. But, this happens and is why there is just a booming market for DBaaS and other like productions. Docker is an amazing tool, but does make somethings a bit too easy.
- 0xfeba 8y ago
  > Its uncovering something that was always there; incompetence is rampant in software engineering. That's what I took away from the article. Production was setup and they don't know what kernel version it has, or what JVM version it's using, or which C library, etc. Or, now we want feature X, so we need to upgrade dependancy Y. I already did it on my local machine, but Bob from DevOps doesn't like upgrading things on QA/integration/prod without a change control... Granted, apps have thousands of dependencies now, so keeping track of it manually is impossible. Docker came along and promised that only whatever's in this one config file is what's installed. Easy! So it eliminated lots of the server-specific configuration management but created other problems.
jgandrews 8y ago
There it is... Docker is fantastic when "it just works" but there is a huge overhead when it doesn't.

by lykr0n 8y ago

jrs95 8y ago

I think a lot of that can also come from people mis-using Docker as if it were a VM instead of trying to do the smallest amount of stuff possible in an image. But at the same time, there's stuff I wouldn't even try to manage in containers in production. Databases and message queues in particular.

mcbutterbunz 8y ago
We've had great success with Docker and Kubernetes for our stateless apps. As a developer, one thing I absolutely love is being able to run an image without having to install anything (other than Docker). If I want to play around with something new, its easy to get started and easy to clean up.
- sigjuice 8y ago
  Playing with something new should be as simple as unpacking a tarball and rm -rfing the directory when done. This should be possible in the comfort of my home directory and uid. If I need a whole separate copy of a computer's files just to try out something, then something very very fundamental is broken.
  - tarr11 8y ago
    How do you install dependencies in this model? As far as can tell, you end up in situations like this https://askubuntu.com/questions/193695/installing-packages-into-local-directory https://askubuntu.com/questions/193695/installing-packages-i...
  - DrScump 8y ago
    Playing with something new should be as simple as unpacking a tarball and rm -rfing the directory when done. Needing a single install script before, and uninstall script after is acceptable.
  - white-flame 8y ago
    Yes I agree things are fundamentally broken with these situations, and they're at the common OS design level.
  - crystaln 8y ago
    What it that something depends on a different operating system and libraries that conflict with yours? Also the world you describe as should is not the world we live in. I should be able to exist in the world without danger or undue prosection.
    - sigjuice 8y ago
      Docker containers primarily mean Linux userland. It doesn’t help you with different operating systems.
- KaiserPro 8y ago
  Hosting stateless apps is simple. If the underlying host fails? don't worry, just spin up a few new copies. Slap a load balancer infront and we can do a/b or blue green. Smashing. But where is that state held? maybe your lucky and your data can be hosted in a single RDS instance. Maybe your partition scheme is simple. However, managing distributed state is hard_.

013a 8y ago

I think this is really a problem you see in every technology that tries to make things easier. JS. MongoDB. The list goes on. When a technology democratizes access to something that was previously only accessible to those with more specialized skillsets, it will attract people who don't know enough to know how to use it right. But, well, they're still using it. Thus, left-pad. Its uncovering something that was always there; incompetence is rampant in software engineering. Ask the average engineer to properly set up an Ubuntu box with CICD for deployment and they'll likely fail, even with all the help of the internet. Ask them to get a docker image onto a server and they can make it work. Wider access to technology is... well, I don't know if its a "good thing" or not. We as an industry pretty strongly believe in "fail fast"; we write tests to make sure our code works, we structure our startups to recognize and pivot around failure, and if that failure comes we stop and say "somethings fucked, stop what we're doing". So I do get scared when a technology enters the stage which allows hard failure to be delayed as long as possible. Maybe we need more technology to be harder, have opinions, and be willing to say "you're doing this wrong, just stop and go get an adult." Because the alternative... Equifax? Yahoo? Ashley Madison?

lykr0n 8y ago
My personal design philosophy (from a System Admin, SRE, DevOps, Ops- whatever) is no black boxes. I'm not talking about knowing the inns and outs of every little component in my infrastructure/system, but instead of saying "This is Graylog" being able to say "This is Graylog- a Java app that uses MongoDB to index and Elasticsearch as a backend. We have to monitor JVM usage." The former is saying we depend on a system we have no idea how it works, and the latter is saying we depend on 2 java applications and a NoSQL database. It horrifies me that people run stuff in production not knowing how it's configured. But, this happens and is why there is just a booming market for DBaaS and other like productions. Docker is an amazing tool, but does make somethings a bit too easy.
0xfeba 8y ago
> Its uncovering something that was always there; incompetence is rampant in software engineering. That's what I took away from the article. Production was setup and they don't know what kernel version it has, or what JVM version it's using, or which C library, etc. Or, now we want feature X, so we need to upgrade dependancy Y. I already did it on my local machine, but Bob from DevOps doesn't like upgrading things on QA/integration/prod without a change control... Granted, apps have thousands of dependencies now, so keeping track of it manually is impossible. Docker came along and promised that only whatever's in this one config file is what's installed. Easy! So it eliminated lots of the server-specific configuration management but created other problems.

jgandrews 8y ago

There it is... Docker is fantastic when "it just works" but there is a huge overhead when it doesn't.