9 ms·

This entire depending on a base container for setup and then essentially throwing away everything you get from the package manager is part of the issue. There i

This entire depending on a base container for setup and then essentially throwing away everything you get from the package manager is part of the issue. There is no real package management for Docker. Hell, there isn't even an official way to determine if your image needs an upgrade (I wrote a ruby script that gets the latest tags from a docker repo, extracts the ones with numbers and sorts them in order and compares them to what I have running). Relying on an Alpine/Debian/Ubuntu base helps to get dependencies installed quickly. Docker could have just created their own base distro and some mechanism to track package updates across images, but they did not. There are guides for making bare containers, they contain nothing .. no ip, grep, bash .. only the bare minimum libraries and requirements to run your service. They are minimal, but incredibly difficult to debug (sysdig still sucks unless you shell out money for enterprise). I feel like containers are alright, but Docker is a partial dumpster fire. cgroup isolation is good, the crazy way we deal with packages in container systems is not so good. Sure if you're just checking for base-distro packages for security vulnerabilities, you're going to find security issues that don't apply (e.g. an exploit in libpng even though your container runs nothing that even links to libpng), but it does excuse the whole issue with the way containers are constructed. I think this space is really open too, for people to find better systems that are also portable: image formats that are easy to build, easy to maintain dependencies for, and easy to run as FreeBSD jails OR Linux cgroup managed containers (Docker for FreeBSD translated images to jails, but it's been unmaintained for years).

DyslexicAtheist 8y ago
I agree the tooling is a tire fire :( > e.g. an exploit in libpng even though your container runs nothing that even links to libpng it's a problem because some services or api's could be abused giving the attacker a path to these vulnerable resources and then use that vulnerability regardless if it is used currently by your service. I like my images to only contain what's absolutely needed to do that job only. It's not so difficult to do, provided people would be willing to architect systems from the ground up, instead of pulling in a complete debian or fedora installation and then removing things (that should be outlawed imho lol). Not only do I get less attack surface but also smaller updates (which again then is incentive to do more often), less complexity, less logs, easier auditabile (now every log file or even log line might give valid clues), faster incident response, easier troubleshooting, sorry for going on and on ... It's a cultural problem too: where people working in an environment where it's normal to have every command available on a production system (really?), and where there is no barrier to install anything new that is "required" without discussion & peer review (what are we pair programming for?) or where nobody even tracks the dead weight in production or whether configs are locked down?. I sometimes think many companies lost control over this long ago. [citation needed] :(
- ghostly_s 8y ago
  I'm not familiar with Docker infrastructure but what is the alternative to "pulling in a complete debian or fedora installation and then removing things"? Compiling your own kernel and doing the whole "Linux From Scratch" thing? Isn't that incredibly time-intensive to do for every single container?
  - webdevatlurk 8y ago
    I think they mean we should prefer using stripped down base images, then installing ONLY what is needed for your application/ci+cd tasks to run.
  - barbecue_sauce 8y ago
    There are supported linux distributions like Container Linux intended for barebones deployments. Docker images layer, so you can essentially build your own base images from these primitive linux distros, and then have those available for your various applications.
  - DyslexicAtheist 8y ago
    say you're setting up a firewall, it would be a bad idea to start with a rule that allows all traffic and then think about which services should be rejected? You would start by rejecting everything and then selectively allow ports/traffic. When bootstrapping a VM or container image it's the same principle (imo), it's safer to once think about what is needed and have that locked down and under control rather than having to keep track of what else was forgotten or might be suddenly a flaw in this ever changing "threat landscape". E.g. the libpng example from above: What else is lying around that is suddenly vulnerable tomorrow? If you got time to compile your own kernel then why not. It's a process IMO and I usually get to that stage once everything else is locked down. To start a minimal set-up using alpine (or busybox) instead of ubuntu or whatever is already a huge reduction of attack surface (and complexity). Next I want restriction of all system calls on a process basis (if a syscall isn't whitelisted for this specific process it's an error). Once this is in place I might think about the kernel but in my scenarios this is rare. It really depends what your service does (what you want to protect). It's imo less work (less maintenance costs) in the long run and always knowing what you have (and why) rather than having to think "oh I didn't know there was foo too? what does foo do"? Because as code changes and as you ship updates you will anyway be attending to updates that suddenly break your setup (and run against your whitelist). But doing this review/audit during an upgrade with a system that suffers from "feature and scope creep" is impossible (at least imho). generally the less I have installed the less I need to track what might be suddenly a problem. Once my container/VM image is the way I want it I push it to my own docker registry and only deploy from there to production. EDIT: really good talk on docker security (not just microservices) https://www.youtube.com/watch?v=346WmxQ5xtk https://www.youtube.com/watch?v=346WmxQ5xtk
markbnj 8y ago
> I feel like containers are alright, but Docker is a partial dumpster fire. cgroup isolation is good, the crazy way we deal with packages in container systems is not so good. Can you expand on this? With respect to packages a container is just a mounted file system. A base image is just a way to start with a known file system state. You can do whatever you want to the file system state using whatever package manager you like. In what way should docker have attempted to exert more control over the file system to improve package management?
- Perseids 8y ago
  See my sibling comment to your post: The problem is exactly that it is just a mounted filesystem on which you can run programs. A container should not be described by a Dockerfile which is basically a glorified shell script with a caching layer on-top (and no concept of cache invalidation), but by a dependency file for a proper package manager. Docker is my go-to example for worse-is-worse, because of that. They have solved only the easy problems and gotten a phenomenally approachable UI as a result (everyone who has used the console to install dependencies can write their own Dockerfile). But in the process they have occupied the niche in which a better packaging solution could have evolved and grabbed all the mindshare with enormous marketing effort (aided by an easy-to-use product).
  - markbnj 8y ago
    I disagree. You seem to prefer a strong docker-centric idea of package management. I think that by drawing the line where they did they made it possible for anyone to use whatever package management scheme made sense for a given application. There are already so many alternatives for driving the file system to a particular state that I really fail to see how docker taking an opinionated position would have helped.
    - Perseids 8y ago
      I think we have very different ideas about the problem docker is the solution for. From the docker.com website ( https://www.docker.com/get-started https://www.docker.com/get-started ): > Building and deploying new applications is faster with containers. Docker containers wrap up software and its dependencies into a standardized unit for software development that includes everything it needs to run: code, runtime, system tools and libraries. This guarantees that your application will always run the same and makes collaboration as simple as sharing a container image. Basically I understand that as "write it on my machine, deploy it anywhere". "Everything it needs to run" are the dependencies in my lingo. So for me, all of this is dependency management. I have never asked for a way to drive a file system to a particular state, in the same way that I don't particularly care how a 'node_modules' folder is structured, as long as I can `require` whatever I want inside my programs. (My point is muddied by the other task docker fulfills for me: Software configuration by creating a directory structure with the following access rights here, writing an nginx config file there. But for me, the ideal scenario would be to reduce the accidental complexity involved in the configuration (I don't care where exactly my program is stored and how exactly it is called, I just want to run it at much reduced privileges and the way I know how to achieve that is to create a custom user and run my program under that user) and define the rest declaratively.)
Perseids 8y ago
The rise of Docker is in particular a pity, as we were getting it right before with Bundler in Ruby, pipenv etc. for Python, Maven for the JVM and NPM for Javascript (well maybe not right in the case of NPM, but better than Docker regarding this point): We declaratively describe which semantic version of a dependency we need, lock down the specific version used for testing and deployment (in a .lock file) and have an easy process to update the specific version to the newest semantic version available and test it before redeployment. With Docker and its procedural dependency declarations ("apt-get install this", "wget | bash that") we've lost all that precious version information and reproducibility. Imagine if Docker was like Bundler/Maven/… but for Debian/Ubuntu repositories. You could deterministically reproduce an image from an easily auditable package file and basically do an apt-get update on the .lock file to fix the security vulnerabilities in the selected versions.
- gitgud 8y ago
  Out of curiosity, what's not right about NPM?
  - mlthoughts2018 8y ago
    Also out of curiosity, what is right about pipenv?
  - alexanderdmitri 8y ago
    Not OP, but given the theme of this thread, npm very directly enables library/dependency bloat, if not flat out encourages it. I remember when I first starting dabbling in web development from an avionics systems background and I'd just made a little toy express app that did a couple tricks and I was kind of exploring and naive and took an innocent gander into that specially icon'd node_modules directory and it was filled with other projects and peaking into a few those I started to get this feeling of dread, the kind that starts in your stomach and slowly diffuses through the rest of you with that leaden feeling of growing hopelessness. Looking back, I should have stopped there, I really should have, but instead I popped open a terminal and, I can still hear each key as my fingers thumped it out: `tree ./node_modules` Watching the unraveling was me bearing witness to everything I had ever known and loved and had believed in and strived for get thrown naked and defenseless and pathetic into the abyss and still it kept unraveling and still I sat and watched the descent until I no longer knew where I was or even who I was and still the modules unraveled relentlessly. I spent that night looking into the mirror as if looking into the eyes of a stranger. The person staring back at me was someone I had not only ceased to know but had never known to begin with. And when the sun rose I had changed and I knew then as the sparrows began to chirp into the cold winter twilight spreading light over a now alien and unrecognizable planet that I had lost something I would never regain because to lose it is to simply realize it was never there to begin with. I steeled my nerves and pulled on my itchy long underwear and buttoned up my plaid button up and I did not put on socks and headed downstairs to the kitchen and put on Now That's What I Call Music 23 and I drank a glass of orange juice and ate my oatmeal and the mix I had hitherto so much enjoyed now struck me as chaotic and ill-thought out and then Abba came on and the song was about a Dancing Queen and I resolved myself once more and brought my dishes to the sink to rinse them and decided there was nothing to do besides async.waterfall the next set of promises only after the necessary precedents resolved (which is how we spoke back in 2016s). And I found myself back at my machine and worked like hell to reign my terror in because the tree command I had run the previous evening was still printing to stdout and I closed my eyes and blindly mapped my fingers to ctrl+c and felt just a little twinge of reassurance as the reflexive `clear` followed without me having to have consciously willed it and I took a deep breath and scratched my leg because of the itchy long underwear and then I was typing out `npm --i numbButAlive` and by the shift+b,u tap dance I began to feel hope one last time and finger resting slack on the carriage return I couldn't bring myself to push it yet for some reason and I froze and I clearly remember hearing Agnetha at that exact moment singing about that 17 year old dancing queen, "Having the time of your life, oh, see that girl" and then the hope was gone because it wasn't ever there it was really there like the hopelessness of hoping to feel hope and I felt an anger and I revolt in me rise up and the room filled with my voice "Fuck it, fuck it all!" and I meant it: fuck this world and fuck Abba and fuck Now That's What I Call Music Vol. 3 and up (with the exception of Vol. 7, maybe) and fuck all the empty dreams of all the fucking vapid nobodies polluting it with their nonsense parade of nonsense distraction after distraction after distraction from the truth that their life will end in death and so will their children's and their children's children and there's no greater significance to that, no hidden meaning or lesson, only death and emptiness and going through life knowing this and knowing how all everything you do to whittle away at the precious little time you've been alotted is a complete waste despite your protests and denials and self-deception and still you keep denying it to your dying breath because here even acceptance is denial in that acceptance is meaningless to that which is what is with or without your fucking acceptance and I found myself laughing a hollow, bitter laugh, a laugh that didn't sound like me, a sound I could never had made until that moment, this moment, a moment without joy or sadness or anything besides utter resignation and defeat and I tacked a `-g` onto the command for no other reason than fuck it, fuck everything and everyone and all their delusional basis for anything and it was easy to push the carriage return now and as I did so and I tasted something new to me then and it must have been my first taste of true freedom in all its terrifying glory, a freedom that leaves nothing to its beholder and is beholden to no one and ultimately sentenced to an eternity imprisoned within itself to as its becoming of self is just a much a defiance to what it should have been free to become.

DyslexicAtheist 8y ago

I agree the tooling is a tire fire :( > e.g. an exploit in libpng even though your container runs nothing that even links to libpng it's a problem because some services or api's could be abused giving the attacker a path to these vulnerable resources and then use that vulnerability regardless if it is used currently by your service. I like my images to only contain what's absolutely needed to do that job only. It's not so difficult to do, provided people would be willing to architect systems from the ground up, instead of pulling in a complete debian or fedora installation and then removing things (that should be outlawed imho lol). Not only do I get less attack surface but also smaller updates (which again then is incentive to do more often), less complexity, less logs, easier auditabile (now every log file or even log line might give valid clues), faster incident response, easier troubleshooting, sorry for going on and on ... It's a cultural problem too: where people working in an environment where it's normal to have every command available on a production system (really?), and where there is no barrier to install anything new that is "required" without discussion & peer review (what are we pair programming for?) or where nobody even tracks the dead weight in production or whether configs are locked down?. I sometimes think many companies lost control over this long ago. [citation needed] :(

ghostly_s 8y ago
I'm not familiar with Docker infrastructure but what is the alternative to "pulling in a complete debian or fedora installation and then removing things"? Compiling your own kernel and doing the whole "Linux From Scratch" thing? Isn't that incredibly time-intensive to do for every single container?
- webdevatlurk 8y ago
  I think they mean we should prefer using stripped down base images, then installing ONLY what is needed for your application/ci+cd tasks to run.
- barbecue_sauce 8y ago
  There are supported linux distributions like Container Linux intended for barebones deployments. Docker images layer, so you can essentially build your own base images from these primitive linux distros, and then have those available for your various applications.
- DyslexicAtheist 8y ago
  say you're setting up a firewall, it would be a bad idea to start with a rule that allows all traffic and then think about which services should be rejected? You would start by rejecting everything and then selectively allow ports/traffic. When bootstrapping a VM or container image it's the same principle (imo), it's safer to once think about what is needed and have that locked down and under control rather than having to keep track of what else was forgotten or might be suddenly a flaw in this ever changing "threat landscape". E.g. the libpng example from above: What else is lying around that is suddenly vulnerable tomorrow? If you got time to compile your own kernel then why not. It's a process IMO and I usually get to that stage once everything else is locked down. To start a minimal set-up using alpine (or busybox) instead of ubuntu or whatever is already a huge reduction of attack surface (and complexity). Next I want restriction of all system calls on a process basis (if a syscall isn't whitelisted for this specific process it's an error). Once this is in place I might think about the kernel but in my scenarios this is rare. It really depends what your service does (what you want to protect). It's imo less work (less maintenance costs) in the long run and always knowing what you have (and why) rather than having to think "oh I didn't know there was foo too? what does foo do"? Because as code changes and as you ship updates you will anyway be attending to updates that suddenly break your setup (and run against your whitelist). But doing this review/audit during an upgrade with a system that suffers from "feature and scope creep" is impossible (at least imho). generally the less I have installed the less I need to track what might be suddenly a problem. Once my container/VM image is the way I want it I push it to my own docker registry and only deploy from there to production. EDIT: really good talk on docker security (not just microservices) https://www.youtube.com/watch?v=346WmxQ5xtk https://www.youtube.com/watch?v=346WmxQ5xtk

markbnj 8y ago

> I feel like containers are alright, but Docker is a partial dumpster fire. cgroup isolation is good, the crazy way we deal with packages in container systems is not so good. Can you expand on this? With respect to packages a container is just a mounted file system. A base image is just a way to start with a known file system state. You can do whatever you want to the file system state using whatever package manager you like. In what way should docker have attempted to exert more control over the file system to improve package management?

Perseids 8y ago
See my sibling comment to your post: The problem is exactly that it is just a mounted filesystem on which you can run programs. A container should not be described by a Dockerfile which is basically a glorified shell script with a caching layer on-top (and no concept of cache invalidation), but by a dependency file for a proper package manager. Docker is my go-to example for worse-is-worse, because of that. They have solved only the easy problems and gotten a phenomenally approachable UI as a result (everyone who has used the console to install dependencies can write their own Dockerfile). But in the process they have occupied the niche in which a better packaging solution could have evolved and grabbed all the mindshare with enormous marketing effort (aided by an easy-to-use product).
- markbnj 8y ago
  I disagree. You seem to prefer a strong docker-centric idea of package management. I think that by drawing the line where they did they made it possible for anyone to use whatever package management scheme made sense for a given application. There are already so many alternatives for driving the file system to a particular state that I really fail to see how docker taking an opinionated position would have helped.
  - Perseids 8y ago
    I think we have very different ideas about the problem docker is the solution for. From the docker.com website ( https://www.docker.com/get-started https://www.docker.com/get-started ): > Building and deploying new applications is faster with containers. Docker containers wrap up software and its dependencies into a standardized unit for software development that includes everything it needs to run: code, runtime, system tools and libraries. This guarantees that your application will always run the same and makes collaboration as simple as sharing a container image. Basically I understand that as "write it on my machine, deploy it anywhere". "Everything it needs to run" are the dependencies in my lingo. So for me, all of this is dependency management. I have never asked for a way to drive a file system to a particular state, in the same way that I don't particularly care how a 'node_modules' folder is structured, as long as I can `require` whatever I want inside my programs. (My point is muddied by the other task docker fulfills for me: Software configuration by creating a directory structure with the following access rights here, writing an nginx config file there. But for me, the ideal scenario would be to reduce the accidental complexity involved in the configuration (I don't care where exactly my program is stored and how exactly it is called, I just want to run it at much reduced privileges and the way I know how to achieve that is to create a custom user and run my program under that user) and define the rest declaratively.)

Perseids 8y ago

The rise of Docker is in particular a pity, as we were getting it right before with Bundler in Ruby, pipenv etc. for Python, Maven for the JVM and NPM for Javascript (well maybe not right in the case of NPM, but better than Docker regarding this point): We declaratively describe which semantic version of a dependency we need, lock down the specific version used for testing and deployment (in a .lock file) and have an easy process to update the specific version to the newest semantic version available and test it before redeployment. With Docker and its procedural dependency declarations ("apt-get install this", "wget | bash that") we've lost all that precious version information and reproducibility. Imagine if Docker was like Bundler/Maven/… but for Debian/Ubuntu repositories. You could deterministically reproduce an image from an easily auditable package file and basically do an apt-get update on the .lock file to fix the security vulnerabilities in the selected versions.

gitgud 8y ago
Out of curiosity, what's not right about NPM?
- mlthoughts2018 8y ago
  Also out of curiosity, what is right about pipenv?
- alexanderdmitri 8y ago
  Not OP, but given the theme of this thread, npm very directly enables library/dependency bloat, if not flat out encourages it. I remember when I first starting dabbling in web development from an avionics systems background and I'd just made a little toy express app that did a couple tricks and I was kind of exploring and naive and took an innocent gander into that specially icon'd node_modules directory and it was filled with other projects and peaking into a few those I started to get this feeling of dread, the kind that starts in your stomach and slowly diffuses through the rest of you with that leaden feeling of growing hopelessness. Looking back, I should have stopped there, I really should have, but instead I popped open a terminal and, I can still hear each key as my fingers thumped it out: `tree ./node_modules` Watching the unraveling was me bearing witness to everything I had ever known and loved and had believed in and strived for get thrown naked and defenseless and pathetic into the abyss and still it kept unraveling and still I sat and watched the descent until I no longer knew where I was or even who I was and still the modules unraveled relentlessly. I spent that night looking into the mirror as if looking into the eyes of a stranger. The person staring back at me was someone I had not only ceased to know but had never known to begin with. And when the sun rose I had changed and I knew then as the sparrows began to chirp into the cold winter twilight spreading light over a now alien and unrecognizable planet that I had lost something I would never regain because to lose it is to simply realize it was never there to begin with. I steeled my nerves and pulled on my itchy long underwear and buttoned up my plaid button up and I did not put on socks and headed downstairs to the kitchen and put on Now That's What I Call Music 23 and I drank a glass of orange juice and ate my oatmeal and the mix I had hitherto so much enjoyed now struck me as chaotic and ill-thought out and then Abba came on and the song was about a Dancing Queen and I resolved myself once more and brought my dishes to the sink to rinse them and decided there was nothing to do besides async.waterfall the next set of promises only after the necessary precedents resolved (which is how we spoke back in 2016s). And I found myself back at my machine and worked like hell to reign my terror in because the tree command I had run the previous evening was still printing to stdout and I closed my eyes and blindly mapped my fingers to ctrl+c and felt just a little twinge of reassurance as the reflexive `clear` followed without me having to have consciously willed it and I took a deep breath and scratched my leg because of the itchy long underwear and then I was typing out `npm --i numbButAlive` and by the shift+b,u tap dance I began to feel hope one last time and finger resting slack on the carriage return I couldn't bring myself to push it yet for some reason and I froze and I clearly remember hearing Agnetha at that exact moment singing about that 17 year old dancing queen, "Having the time of your life, oh, see that girl" and then the hope was gone because it wasn't ever there it was really there like the hopelessness of hoping to feel hope and I felt an anger and I revolt in me rise up and the room filled with my voice "Fuck it, fuck it all!" and I meant it: fuck this world and fuck Abba and fuck Now That's What I Call Music Vol. 3 and up (with the exception of Vol. 7, maybe) and fuck all the empty dreams of all the fucking vapid nobodies polluting it with their nonsense parade of nonsense distraction after distraction after distraction from the truth that their life will end in death and so will their children's and their children's children and there's no greater significance to that, no hidden meaning or lesson, only death and emptiness and going through life knowing this and knowing how all everything you do to whittle away at the precious little time you've been alotted is a complete waste despite your protests and denials and self-deception and still you keep denying it to your dying breath because here even acceptance is denial in that acceptance is meaningless to that which is what is with or without your fucking acceptance and I found myself laughing a hollow, bitter laugh, a laugh that didn't sound like me, a sound I could never had made until that moment, this moment, a moment without joy or sadness or anything besides utter resignation and defeat and I tacked a `-g` onto the command for no other reason than fuck it, fuck everything and everyone and all their delusional basis for anything and it was easy to push the carriage return now and as I did so and I tasted something new to me then and it must have been my first taste of true freedom in all its terrifying glory, a freedom that leaves nothing to its beholder and is beholden to no one and ultimately sentenced to an eternity imprisoned within itself to as its becoming of self is just a much a defiance to what it should have been free to become.