25 ms·

Run More Stuff in Docker

199 points by psxuaw 6y ago

4778468d 6y ago
I run nothing in Docker, ever. It’s never prevented me building sophisticated systems.
anonymoushn 6y ago
> If you’re using Docker on a Mac and you’ve never tried it on Linux, you owe it to yourself to try it on Linux. So ultimately the recommendation is to install Linux and use it as my daily driver. But Linux does not run on my machine yet.
- Steltek 6y ago
  And MacOS does? Is this some Apple Silicon humble brag? Cuz that's "Apple doesn't allow developers and enthusiasts to do their thing on their own computer that they purchased" not "Linux doesn't run on my machine".
  - jolux 6y ago
    Apple definitely allows it, but as far as I know Linux doesn’t support it yet. (turn off SIP, couple other things related to the boot process I think)
  - collinvandyck76 6y ago
    OP's point stands. You assigning them with "humblebrag" intentions is on you. It's not unreasonable to think that many people have M1s now, and it will certainly increase over time.
  - viraptor 6y ago
    Not even Apple silicon. Linux is not compatible with MBPs since the touchbar models. More because of "Apple doesn't care you want this" (wifi firmware is known-broken) rather than disallowing.
  - eyelidlessness 6y ago
    Why wouldn’t Apple allow Linux to run on their ARM Macs? They’ve allowed any compatible OS to run on Intel and PPC Macs. The only thing that’s changed is that they don’t currently offer a first party multi-boot solution. But they didn’t even offer one for Windows on Intel until community solutions were relatively stable and popular, and afaik they never offered one for Linux.
sdepablos 6y ago
I'm in the same situation: until I figure (if ever) how to use NixOS and control version my OS, I tend to use Docker for any application with lots of dependencies, specially Python related ones
- physicsgraph 6y ago
  Did you try the Docker-based approach after trying Python's virtualenv? Or did you go directly to Docker?
  - sdepablos 6y ago
    No, I've using virtualenv for quite some time and I still use it when I'm the person in control of the dependencies, though I'm moving little by little to the docker way. Why? Because sometimes the problem is not only on the dependencies, but in Python itself. As an example, as of today gcloud does not support Python 3.9.
- fuzzybear3965 6y ago
  I've recently spent a few hours in NixOS land. I even built my own ISO and booted into it from an SD card. If you want a 30 minute intro to getting started then just ping me. I'm en route to something like what you want and I'm not far ahead of you.
- setheron 6y ago
  Reach out to me if you want to learn Nix I'm always happy to help people climb the steep learning curve
  - stack_underflow 6y ago
    Curious, does one need to use nixOS for an optimal experience? I've been using debian for a long time and would prefer not to have to switch OSs but the idea of using nix for having full control over my package graph is very tempting. I've been somewhat procrastinating on trying nix as I've heard GNU Guix has a similar feature set and haven't been able to decide on which one to dive into... My ideal setup would be to just be able to run a single shell script that configures a new machine to the exact state of all my other dev machines. I have a shell script that somewhat does this but it's not completely unattended and still requires a lot of manual config for certain steps. Aside from that my main use case is being able to easily share a dev/build environment with others for ensuring that they can compile a certain project exactly as I do. For now I just use docker but it's frustrating not having explicit control over the layer cache and being able to tell it what to cache and what not to.
physicsgraph 6y ago
A few years ago I set a goal to minimize the number of applications installed bare-metal on my laptop. Everything is containerized, just as the blog describes. There is some initial overhead, and some extra work associated with maintaining the infrastructure, but overall I can report improved reproducibility (due to an isolated and static environment per app). I use a bash alias for each common application. For example, "jup" launches a jupyter notebook. I have a container with the Python package "Black" which runs using a git hook to clean up my code prior to commits.
- deleted 6y ago
  [deleted]
- qmmmur 6y ago
  That just seems insane to be running black inside a container. Why dont you just configure your environment properly?
  - antoinealb 6y ago
    Serious question: Why do you believe that to be insane? The Dockerfile for a simple application such as black must be very short (probably 3-4 lines), the alias is probably quite short and the container overhead time is minimal for native docker (the story might be different for things like docker mac). On the other hand, you get some benefits from installing black through docker rather than through the system package manager: is is completely isolated from the host and the only way to break black is to update it, changing anything in the host will not break your black install. I am not sure either what you mean by "just configure your environment properly", but I am going to assume you mean installing black under a virtual env or equivalent? Then it is also annoying for different reasons: you must reinstall it once for each project, updating python to a new (major) version breaks your formatter, you cannot move the env around, to name the ones that come on top of my head.
    - viraptor 6y ago
      > you must reinstall it once for each project You can install it either in a venv outside of all projects or even "pip install --user black". > updating python to a new (major) version breaks your formatter Uninstalling the old version breaks the formatter. Installing a new one does not. Either way, with asdf, pyenv, and others you can keep all relevant version around. > you cannot move the env around Sure you can. Use "--relocatable" I agree docker may be nicer if you're not working day to day in python... but in that case black is not a great example. You probably want to have a specific version bound to the project so everyone uses it (including the ci platform).
pjmlp 6y ago
Docker is the NoSQL of 2020.
- zoover2020 6y ago
  Why so, though?
  - pjmlp 6y ago
    The usual hype curve that everyone should be using it no matter what kind of workloads.
- young_unixer 6y ago
  Not really. Containerization has clear and obvious advantages over the traditional bare-metal approach.
  - pjmlp 6y ago
    For security purposes as used since the early days in mainframes and big iron UNIX, surely. As workaround for broken dependency management, not really.
    - stickyricky 6y ago
      Whats the alternative?
      - pjmlp 6y ago
        Fix the dependencies story in what concerns Ruby and Python. Native binaries can easily do with static linking (fix glibc or replace it with proper libc like musl). Other platforms are doing quite fine without containers.
        young_unixer 6y ago
        I kind of agree with you. But in the Linux world no one has the intention to fix the package distribution problem except for that AppImage guy.
wa1987 6y ago
> On a Mac, there is a major performance hit whenever you do disk IO in a bind mount (i.e. voluming a directory of the host system into the container). Working without bind mounts is extremely limiting. [..] If you’re using Docker on a Mac and you’ve never tried it on Linux, you owe it to yourself to try it on Linux. Or use named volumes. I'm running a dockerized WordPress dev environment on my MacBook with average TTFB's of 40 ms.
- stack_underflow 6y ago
  I had to investigate why our docker-ized dev environment was so slow on macOS at my last job and this was the root cause. One project's test suite ran in about 5-10 mins in our linux CI/CD environment, and about 50+ mins in macOS with docker. There was a very in-depth thread on the docker forums where the devs explained why there was such a huge performance penalty. IIRC it was due to all the extra bookkeeping that had to be done to ensure strong consistency and correct propagation of file system events between the virtualized docker for mac environment and the host file system. The test suite would run integration tests that performed a lot of npm/yarn operations which meant lots of disk IO.
  - oauea 6y ago
    The reason is that docker on mac runs in a Linux VM and VM mounts are slow.
- oauea 6y ago
  > Or use named volumes And how do you access these from your host with high performant IO?
  - wa1987 6y ago
    There’s no need to in my situation. 99% of the files are dependencies installed through Composer (PHP) while building the Dockerfile. The few directories I need to work on directly are bind mounts and don’t have much impact on overall runtime performance.
the_duke 6y ago
For end user devices, I much prefer Nix/NixOS [1] for this kind of thing. With Flakes [2] (experimental feature), you get full reproducibility. The documentation is spotty and there is a considerable learning curve, but I've switched to NixOS on my laptop and desktop early this year and am mostly very happy with it. That doesn't cover sandboxing though. I would actually agree that sandboxing / restricting applications (Mac OS style) would be sorely needed on Linux desktop. Flatpak and Snap are trying to do this. A much saner approach than trying to find or maintain up to date, trustworthy Docker files for applications. [1] https://nixos.org/ https://nixos.org/ [2] https://nixos.wiki/wiki/Flakes https://nixos.wiki/wiki/Flakes
- timClicks 6y ago
  Snap provides a strong sandboxing layer, but IIRC Flatpak does not.
  - the_duke 6y ago
    That is incorrect. https://docs.flatpak.org/en/latest/sandbox-permissions.html https://docs.flatpak.org/en/latest/sandbox-permissions.html
- deleted 6y ago
  [deleted]
- ris 6y ago
  Absolutely. Most Dockerfiles I come across are very much not written in a robustly reproducible way. The stacking filesystems model is really not very flexible (if you want to compose two docker images with dissimilar bases to be used together you have to resort to manually specifying which files to copy across from each base). Use of OS features to achieve its ends means the docker daemon itself requires far more privileges than I'm comfortable with and makes weirdness like docker-in-docker necessary...
  - mikepurvis 6y ago
    Yeah, any Docker file that starts with `apt update; apt dist-upgrade` is an instant fail. You might be interested in this fascinating exploration of attempting to bend Docker into better caching and composition by injecting blocks of Nix packages as individual layers: https://grahamc.com/blog/nix-and-layered-docker-images https://grahamc.com/blog/nix-and-layered-docker-images
mkl95 6y ago
My personal computer doubles as my workstation. I manage my Python virtual environments with virtualenvwrapper, and I install all the other stuff -mostly OpenGL and gamedev related tools- with apt or from source. Many developers would consider this Ubuntu 20.04 install that I run on a 9 year old computer to be an accident waiting to happen. But I'm finding this Ubuntu release to be extremely stable even at 3000+ packages.
jasoneckert 6y ago
I run most of my apps in containers on my developer workstation as well. Once containers became popular and I noticed that a few of my apps were available as container images on DockerHub, it just made sense from a resource and security point of view. This is the first post I've seen advocating it, but I think it is sage advice for anyone.
- protoman3000 6y ago
  I don’t understand. How can you find a random image maintainer on DockerHub more trustworthy than the rigorous package maintainers for your average distro?
harryruhr 6y ago
Docker for every application? If you create and maintain the Docker image or Dockerfile for every applucation yourself, you must have plenty of time. If you rely on public images from Docker Hub, you must have plenty of trust in the creators of those images.
- cortesoft 6y ago
  Hmmm... what alternative does not either take time or trust?
  - harryruhr 6y ago
    It's all relative, of course. But getting a signed package from the repo of the distro I'm using for years is something different than using a random image from hub.docker.com.
    - cortesoft 6y ago
      Sure, but you are still relying on trust, and you are choosing to limit yourself to things released by your chosen distro. This is the same as if you were to pick a specific docker publisher that you trust, and only use their images.
      - t-writescode 6y ago
        It's arguable that it's not quite the same. It all comes down to consequences. If a distro messes up the trustworthiness of an application, they, the big and important company loses clout. If the application developer messes up, they also lose clout - people may stop using their software. Chances are, if you're using a third party for a third party piece of software that isn't officially dockerized by the company that developed it, nor a major distro, there's no real backlash if it doesn't work or if they get hacked, etc: "it was a third party trick, so _of course_ it wasn't trustworthy" would be the statement everyone makes. Debian messing up, or Cisco or Oracle, etc, is a much bigger deal.
  - jillesvangurp 6y ago
lrossi 6y ago
The advantage of running apps in docker is that you avoid the risk of breaking your system by adding 3rd party package repos or even worse, curl | sudo bash. Cleanup/uninstall is very simple as well. However, I don’t agree with using it for sandboxing for security. Especially if you are giving it access to the X11 socket, as the author does in the examples. It might not be obvious, but the app will have access to all your other open windows.
- gcb0 6y ago
  > It might not be obvious, but the app will have access to all your other open windows. Fun fact, most docker hosts will allow access to all your files anyway! (specially true on docker for mac, which all the cool kids(tm) here are using). Even if you restrict container host-FS access to a source repo dir, mind rogue code changing your .git hook scripts in there or you might run code outside of the container when committing ;) Another slightly relevant fun fact, USB is a bus. That means that any device can listen in on any other device. And USB access is given by default to some X-enabled docker (--tty something), and to most virtualbox machines (including the hidden one running the fake docker linux host on docker-for-mac), and more recently Google-Chrome. ;)
  - zapita 6y ago
    > and to most virtualbox machines (including the hidden one running the fake docker linux host on docker-for-mac) docker-for-mac does not use virtualbox.
    - t-writescode 6y ago
      Docker For Mac does use a virtual machine to run the Docker machine, though. It's complicated to access the automatically generated mount point on Docker's mac when you create a virtual folder but don't bind it anywhere.
      - zapita 6y ago
        But does Docker for Mac give unlimited access to the host’s usb bus like GP claims? I don’t see any evidence that it’s true, and the mistaken claim that d4m uses virtualbox increases my skepticism.
deleted 6y ago
[deleted]
Macha 6y ago
I've actually gone the opposite direction in my home lab to run less in docker - unless you're building your own images it's hard to know what's in them, which images are built on out of date OS images or dependencies, what else is running etc.
hendry 6y ago
Just some counter arguments to @jbergknoff's well put together page! Docker is the best medium for distributing - A static file is far easier to share / distribute. Cross-platform - You need an arguably complex and unstable Linux interface to run Docker images, cgroups et al Sandboxed - security claims about Docker have always been controversial. Simple Unix/BSD constructs like chroot/jails are far simpler and they are reliable Version pinning - a binary can embed a version and you can stick with it Reproducible - Everyone gets confused about Docker image checksums. `sha1sum static-binary` is far far simpler. Minimizes global state - wouldn't be a problem is people built static binaries.
- FartyMcFarter 6y ago
  Static linking is nice, but there are some licenses (notably the GPL and LGPL, which glibc uses) which don't easily allow that for closed-source software. On the technical side, as soon as some code needs to 'dlopen' something (e.g. a plugin or a system driver), you can run into trouble due to multiple instances of glibc or other dependencies running together. But if none of those are a requirement for your use case (or you have workarounds), I agree that statically linked binaries can be a nicer solution than Docker.
  - foolmeonce 6y ago
    You can statically link whatever you like, you just can't distribute it as one work. One of the crazy one executable docker containers strikes me as one work to whatever extent a static linked binary is.
    - FartyMcFarter 6y ago
      > You can statically link whatever you like, you just can't distribute it as one work. How do you distribute it then? Let's assume your statically linked binary contains both closed-source code and GPL/LGPL code. > One of the crazy one executable docker containers strikes me as one work to whatever extent a static linked binary is. I'm not a lawyer, but that's not my understanding. A docker image is a glorified collection of files with some metadata, just as a tar file is. I think it's broadly agreed that you're allowed to distribute a tar file with unmodified LGPL dynamic libraries in it without having to open-source all of your code, and I think docker images are treated the same way?
chaitanya 6y ago
If you are on a Mac or Windows, this approach only works for utilities that can run on Linux. If you need to run a native Mac program for example, docker won't be able to run that. Perhaps Nix might help here, but I've never used it, so can't say for sure.
karlding 6y ago
The blog post links to Jessie Frazelle's GitHub repo [0]. Jessie also has a blog post about this [1] from back in 2015. If you prefer video format, Jessie also has a talk at DockerCon SF 2015 [2]. [0] https://github.com/jessfraz/dockerfiles https://github.com/jessfraz/dockerfiles [1] https://blog.jessfraz.com/post/docker-containers-on-the-desktop/ https://blog.jessfraz.com/post/docker-containers-on-the-desk... [2] https://www.youtube.com/watch?v=cYsVvV1aVss https://www.youtube.com/watch?v=cYsVvV1aVss
systemvoltage 6y ago
Let's not. I don't want to install Chrome which is already 73 MB, now bloated up with a whole lotta bullshit that's 500 MB+ image. Imagine downloading every application as a docker container. WTF. Docker is for distribution of applications when deploying them to servers. As a developer, it's amazing at that and have brought peace and joy in devops. Let's leave it there, shall we?
- peteretep 6y ago
  > Imagine downloading every application as a docker container. WTF. Help me understand the WTF here, and also how that’s meaningfully different from an OS X app?
  - systemvoltage 6y ago
    Jumping from 73 MB -----> 500 MB or whatever. Have you been running Docker on your system? Have you tried running: `docker system prune -a`? It's gonna print something like Total reclaimed space: 31.2GB.
    - peteretep 6y ago
      wtf are you doing to create such large Docker containers? Are you installing each app on top of a fully-fledged Ubuntu? Are you leaving the build tools in the final container?
      - systemvoltage 6y ago
        That's what I am trying to ask myself too! I am sorry that I am a developer and I need to run a bunch of docker compose files.
      - fantod 6y ago
        I mean it could just be very many containers that add up. And in many cases you'll have loads of images close to 1GB unless you go to great lengths to try and shrink them (e.g. anything that uses Python).
johnsoft 6y ago
I appreciate the sentiment, but I wouldn't be able to put up with the slow container boot time for regular use. The author mentions this at the bottom. He claims 1sec delay. Last month I benchmarked it on new-ish hardware at 560ms, or 290ms if you disable namespaces (and so disable isolation). I wanted to also benchmark bocker[1] (docker written in bash) for a baseline comparison, but it no longer runs and I threw in the towel after ~30 mins of tinkering. Anyways, if you run something like `curl | jq | grep | less` you could be waiting a fair bit for all those containers to start. Package managers are pretty good these days. I think I can trust it to install `jq` properly. [1]: https://github.com/p8952/bocker https://github.com/p8952/bocker
- tomcam 6y ago
  > curl | jq | grep | less OK I didn’t know about this. Does each one of the piped commands cause the creation of a new container? And why?
  - djhworld 6y ago
    Assuming you've bound an alias to these commands to run them individually, each in their own container
    - tomcam 6y ago
      Thank you. Why on earth would one wish to bind these commands to their own containers? Or was that just a trivial example for illustrative purposes that I’m obsessing over?
harryruhr 6y ago
Docker has its use cases. But running everything possible in a docker container just for the sake of "dockerizing" seems a little bit excessive. One must basically maintain a (more or less) complete userland environment for every application. The idea of shared libraries is taken to absurdity this way. It would be better to build all static. Then there is the waste of resources. I'm sure with plenty GB RAM, TB of SSD space and GBit of bandwith available nowadays many people don't notice. But for what?
- cinquemb 6y ago
  > Then there is the waste of resources. I'm sure with plenty GB RAM, TB of SSD space and GBit of bandwith available nowadays many people don't notice. But for what? Yeah, you deff notice if you are trying to scale as cheaply as possible…
tracker1 6y ago
I'm not sure I'd reach for this in some of the use cases over snap/flatpak/appimage for GUI applications. For some terminal apps, maybe, but even then I usually just user my dockerfile or docker-compose file to manage these needs. What's really needed are better/more articles about how to clean off the cruft left by docker after running your builds locally. Especially when you do things like update your base image versions from node:12 to node:14 etc.
gigel82 6y ago
I wish there were real / usable Windows containers out there. I'd love to spin up a quick sandbox to run a MSVC / MSBuild, but the smallest "image" in Windows-land for that is 12Gb. Still use Docker alongside WSL2 for purely-Linux stuff like some node.js scripts or python things that don't need GPU.
- nyellin 6y ago
  I'm looking at startup ideas in this area. Would you be interested in describing your pain points and use case?
  - gigel82 6y ago
    I just want Docker containers for Windows with a reasonable size. They do have a Server Core (at about 1.5Gb) and a much slimmer NanoServer (100Mb), but the NanoServer is not capable of running the Visual C++ compiler for example. Just look at something like this which tries to compile C++ for VcPkg: https://hub.docker.com/r/hripko/vcpkg/tags?page=1&ordering=last_updated https://hub.docker.com/r/hripko/vcpkg/tags?page=1&ordering=l... (the Linux images are all < 300Mb, but the Windows one is 5Gb compressed).
- newscracker 6y ago
  It’s not clear to me what your requirements are, but take a look at Sandboxie for Windows. [1] [1]: https://github.com/sandboxie-plus/Sandboxie https://github.com/sandboxie-plus/Sandboxie
alexpapworth 6y ago
No thanks.
ridiculous_fish 6y ago
Docker is intimately tied to Linux. It it only "cross platform" in that it can use a VM (Mac, WSL2) or flaky compat layers (FreeBSD). If software embraces Docker, it effectively excludes other OSes, like Windows, BSDs, Haiku, Fuschia, etc.
- ed25519FUUU 6y ago
  Even though it doesn’t run cross platform natively, it does run cross platform acceptably. I have no issue developing, building, and shipping images on my Mac that are deployed to production.
  - chii 6y ago
    in that case, why is it that docker is considered superior when you could virtualize the windows api and run native windows apps in linux instead?
- zachrose 6y ago
  And even then trying to run Docker on Fedora will prompt you to turn off Security-Enhanced Linux features or use Podman instead. So it’s really like, what, 1/2 or 2/3 or Linux support
  - cpuguy83 6y ago
    What? Why would it prompt this? Docker supports SELinux.
    - remexre 6y ago
      I don't know the specific issue, but RH's been trying to push Podman fairly hard...
      - dajonker 6y ago
        Fedora is using cgroups v2 by default since last year. This version of cgroups was not supported by docker until a couple of weeks ago. So as a Fedora user you could either use podman or modify your installation to use cgroups v1 instead.
zaptheimpaler 6y ago
I prefer to just make a big "devbox" docker with all the utilities I need installed, and the home directory as a docker volume. The advantage is i don't have to fiddle with containers for every little program, no startup delay for every program, and my base ubuntu install stays clean and stable (which runs a VM and other services so stability is important). It does have a few warts, the main one being you can't have the entire container root volume be a docker volume, so to install new programs persistently inside the devbox i have to rebuild the image (if they don't live completely in the home volume). But logically its an okay tradeoff because the only way to make the whole environment reproducible is to specify things at build time.
- fistfucker3000 6y ago
  Is it possible to run your IDE within these docker environments? I'd love to set up a do it all docker container for the front end devs at my work.
  - amgreg 6y ago
    I use the Remote Development extension for Visual Studio Code, which, among other things, allows you to attach an editor window to a running container. https://code.visualstudio.com/docs/remote/containers https://code.visualstudio.com/docs/remote/containers
timhigins 6y ago
Just want link to Whalebrew, which achieves a lot of what this article mentions but IMO is more user-friendly. https://github.com/whalebrew/whalebrew https://github.com/whalebrew/whalebrew
mongol 6y ago
I am surprised that systemd portable services is so little known. It solves many of the same problems as Docker does but is comparatively simpler. It also requires more plumbing on top but I prefer it.
alphachloride 6y ago
I find that the time taken to get everything working means I have less time to get work done. The busywork does give me a sense of achievement, but that is not the achievement that matters. It's premature optimization to sandbox an application by default unless there is a pressing reason to do so. For e.g. zoom and hugo don't really change the filesystem or OS settings beyond the folders they output to. I don't see a reason the have them sandboxed personally.
- thefunnyman 6y ago
  Isn’t that precisely a use case for sand boxing? If I know zoom doesn’t need file system, I can deny it access and guarantee it doesn’t use the file system without me knowing about it.
  - alphachloride 6y ago
    If you don't trust an application, sandboxing is one way to go. But that goes all the way down. If you don't trust docker and the OS, run them in a VM. If you don't trust the VM, run it on a spare computer. If you don't trust that computer's hardware, then use minecraft to make a PC with your own instruction set :) Same with networking. Use HTTPS. No? Also use a VPN. No? Make your own VPN. No? Use smoke signals with one-time pad encryption.. In the end you decide at what point are you willing to delegate responsibility for things working as they say they should.
TimTheTinker 6y ago
> On a Mac, there is a major performance hit whenever you do disk IO in a bind mount (i.e. voluming a directory of the host system into the container). Working without bind mounts is extremely limiting. In my opinion, this is a big enough issue as to throw the whole idea of “Docker as cross-platform platform/target” into major question. When running more than a few containers on macOS, the performance is so bad it becomes almost unusable.
- Spivak 6y ago
  The only reason it works so well on Linux is because containers are bog standard Linux processes running with no real overhead sans the kernel structures needed to maintain the namespacing. Entries in the VFS are cheap. Once you need a hypervisor most of the benefits are gone. But if you need the hypervisor anyway which might be the case for software that doesn’t have Mac builds then it starts to look attractive.
- lloeki 6y ago
  Volumes to macOS host performance has nothing to do with bind mounts. It’s the VM folder sharing that’s crap, doubled with the attempt at Docker for Mac to translate filesystem events from fseventsd to inotify and back. Changing solutions (e.g Docker for Mac -> VirtualBox or Fusion via e.g docker-machine) makes a world of difference.
nyellin 6y ago
I've been looking at startup ideas in this space, either as a Citrix replacement (run applications locally in a secure enclave instead of running them in the cloud and paying for vms) or as a way of bringing the benefits of containers to desktop apps and traditional IT (build containers for deployments not images). Would love to hear from anyone with use cases or pain points.
gorgoiler 6y ago
If you are reading this with interest then I encourage you to give it a go. However, Docker is like a kitchen machine: good for getting something done in a particular way (stand mixer for kneading dough) but less useful for understanding what’s actually happening (when is pizza dough ready?) On Linux, picking apart LXC stuff at the command line is very much worth your time if you’re interested in how things work. (Not LXD though which is more useful as a tool than as a teaching aid.) If you have some IPv6 allocation to play with you can make some interesting infrastructure and have it do something useful on the Internet without relying on the lxc-net crutches of an automatically built bridge with NAT. It all feels very well designed as a bag of tools to let one make things rather than a complete system that guides you in only one particular direction. I use stand mixers and Docker all the time; sometimes it’s fun to get into the details too.
ChicagoDave 6y ago
This might lead to developers not knowing or understanding their environment and I have a problem with that.
ogre_codes 6y ago
> Running a program in a container is a lot like running it normally, but the user doesn’t need to jump through hoops to configure the system, build and install. Docker is itself a complex build tool which requires a bunch of install steps. If you are going to ship software to end users there is almost always a better way to bundle and ship than send someone a Docker container. Docker is not a distribution tool, if you are expecting your end users to install Docker, you've already screwed up. > Downloading a pre-compiled binary is almost like this, except with worse odds. Maybe there’s a build for your architecture. If it was statically linked, you’re golden. Otherwise, use ldd to reverse engineer the fact that you need to install libjpeg. On Mac and Windows this is almost never an issue. Even on Linux, it's pretty straight forward to statically link your binary if you aren't sure about the environment it's going to be run on. Statically linked binaries are a bit bloated... but not as bloated as a damned Docker image which contains entire dependency trees. In no case is "Making it into a Docker Image" a simpler/ better distribution mechanic.
- tedk-42 6y ago
  Sorry but you're wrong. A vast majority of software ships after it has been built as a stand alone application as well as a Docker container so that it can be run in container environments. A lot of CI systems natively support Docker because of the variety of tools and images you have available to run you CI steps in. Docker is an excellent distribution tool because it only requires Docker. You can send someone some python code and ask them to run it, only to find they're missing a bunch of C libs required to build and run the code which is a pain to help them figure out how to solve. Docker solves many of these problems but has a drawback of being more 'bloated' than other distribution mechanisms. It doesn't make it a bad one though.
  - ogre_codes 6y ago
    > You can send someone some python code and ask them to run it, only to find they're missing a bunch of C libs required to build and run the code which is a pain to help them figure out how to solve. You can send someone a Docker file only to figure out they've never heard of Docker. Let alone don't have it installed and aren't interested in setting up a container system to your 500 line Docker script. All you've done is move your problem upstream. Unless your consumer is a web developer, you are out of luck.
akhilcacharya 6y ago
I think the reasoning is perfectly sound for doing this but the performance penalty is too much to bear for most use cases - what's preventing nixpkg/guix from being used fully for this use case? Lack of available packages?
masa331 6y ago
This brings only mess and another indirection layers to your system which is hardly an advantage in my opinion. It will take more time to setup, more space on disk, it will be slower and harder use. And all the normal tools(like awk and others in the article) are easily instalable in all distros. I will rather spend time learning these tools directly instead of infinitely fiddling with docker.
siscia 6y ago
I am trying to go in the opposite direction offering a read-only, mountable file system that offer a well crafted set-up of tools and utilities. The idea is that you mount the filesystem and you got all the tools you need, and more, well installed, and that are lazily pulled from the network. You win on the space side, but you need a bit more of trust. You can find more info here: http://packages.redbeardlab.com http://packages.redbeardlab.com On the GitHub repo where you can ask for more packages to be installed: https://github.com/RedBeardLab/packages.redbeardlab.com/ https://github.com/RedBeardLab/packages.redbeardlab.com/ And in this pair or articles for specific languages Golang: https://redbeardlab.com/2020/12/21/packages-redbeardlab-com-for-golang/ https://redbeardlab.com/2020/12/21/packages-redbeardlab-com-... And for JavaScript/node: https://redbeardlab.com/2020/12/23/packages-redbeardlab-com-for-node-js-community/ https://redbeardlab.com/2020/12/23/packages-redbeardlab-com-...
eptcyka 6y ago
Instead of using docker, why not use podman? I prefer my containers in userspace with user privileges.
freebuju 6y ago
Were it not for docker, I probably would have never learned about Linux. I'll share why. I was excited to find docker. It was neat and could be setup to update your applications without manual interaction. So I had it run my media server apps on a seedbox I had rented. Fast forward years later got a form factor pc that came with the free version of Windows. Try as a might, I couldn't get docker to work on it. Required the Pro version or something like that. I remember thinking, well that sure is stupid! That was the beautiful beginning of my affair with Linux as an OS for my desktop/server needs.
blunte 6y ago
TFA could make a small paid course walking a dev through the process of setting all this up and using it effectively. I would buy it.
sweeneyrod 6y ago
Docker has on the order of 1 second startup time doesn't it? I don't understand why someone would willingly subject themselves to that for things being manually run multiple times per day without an extremely compelling reason.
0x445442 6y ago
Because of various org restrictions I attempted to create a developers image on Windows 10. I got the container running but mounting the volumes back to my host workspace was not performant. A build that took 10 seconds on the host took 90 seconds in the container.
OCISLY 6y ago
I prefer zero-dependency single file executables. Elegant and easy to comprehend. Docker only adds another layer of abstraction and its advantages are questionable.
pavlus 6y ago
TLDR: 1. You already have that on Linux. 2. Make your builds portable. If you are running mainstream Linux(Debian-based/Arch-based, probably other), most of the Docker profits can be achieved with already installed and configured systemd and your distro's package manager. Sandboxed? systemd. Simple, uniform interface? Your distro has packages, and most likely services that can and should be sandboxed already run in systemd after installation, you can tune unit-file if you want, and systemd has security checker, that shows you what application in the sandbox can and can not do, without proxying things the Docker way. Versions pinning? Pin version with your package manager. Want multiple versions? Check out DebianAlternatives system. Reproducible? Fix your build/install configs, not the environment. If it builds on your machine, but not on the other, or run flawlessly on one, but not the other, it means you have implicit dependency on the environment, or wrong dependency versions constraints, which you likely don't know about. If you don't know your dependencies, you are definitely shooting blind. Minimizes global state. Repo-based distros minimize global state by providing software that has most dependencies compatible, so you can have your minimal state and update your software too. Meanwhile with docker you need all dependencies and hope that they will match between images, so you can save on layers reusage. And if someone decides to update base image, and others dont... well, too bad, you have to have both versions of the base image.
- bayindirh 6y ago
  I want to add another point to this TL;DR: 3. Do your research before writing these blog posts. I don't want to put down anyone for writing these blog posts. The idea is nice from a distance, but the reality doesn't work like that. It's shoehorning something not suitable for this situation. There's singularity which runs on non-root environments, however it's not for desktop systems, but multi-tenant clusters. I really get frustrated when people advocate expensive abstractions for minimal gains. We can use our processing power much more efficiently while keeping almost the same properties without the costly abstractions. Piling everything on top of each other to create impenetrable and immutable abstractions is not the way to achieve this. Docker already makes debugging very hard by being immutable and impenetrable as is.
- deleted 6y ago
  [deleted]
djhworld 6y ago
I'd be interested to know if Jess Frazelle, linked in the post still maintains her Dockerize-everything setup. I remember a few years ago her write-up on that stuff. Would be good to hear her thoughts a few years on
Arch-TK 6y ago
I get that doing APIs right is hard and that when you're trying to release a proprietary product having to worry about what libraries every system comes with may be a struggle and that if you're developing a webapp you may want everyone to have a consistent test environment regardless of what flavour of archlinux they're running but I really do NOT want to have to spend 2 hours downloading 300GiB of software updates when OpenSSL wets the bed and has a critical vulnerability. The nice thing about libraries and shared objects is that when I get an email saying: "Critical CVE found in something important that may actually affect you" I want to be able to run an update command which fetches a few libraries including the vulnerable one, reboot and be back in business. I also don't really care for the "we have so much space now" argument. I certainly don't, I don't put in expensive 2TB SSDs in my laptop because I don't need them, and I don't want half of my 500GiB disk to be taken up by giant blobs of unoptimized docker images for the same reason that I don't want to run 10 copies of chrome at the same time to use a text editor, an email client, a web browser, a media player, a debugger, the thing I'm writing, three chat clients, and a partridge in a pear tree. I have extra space and extra processing power on my computer so that a: I can use it for the things I actually want to use it for and b: so that I have a snappy machine which can take an unexpected load (be it disk load or processing load) without problems.
harrisonjackson 6y ago
> slight startup delay On a mac this is a complete deal breaker for anything you need to run frequently. It is more than slight.
knur 6y ago
This reminded me of a proof of concept thing I wrote months ago, that facilitates this kind of work. https://github.com/casidiablo/macondo https://github.com/casidiablo/macondo I don't even know what it is I built, but it has been useful in some contexts. It basically allows you to easily wrap and distribute scripts (or more complex apps) that have specific dependencies that might not always be installed in the host. It does so by wrapping the script in a docker image. It also automates the annoying part of docker: mounting local paths for apps that need to interact with the host's file system. I need to write a blog post on this if anything to gather feedback. I'm still not 100% sold on the idea and there are some edge cases. Still, a fun experiment.
X6S1x6Okd1st 6y ago
Ugh at work the edit-compile-run loop only needs to be ~1 second (it's a fairly small go project), but our build system uses docker for everything, so now it's ~1 minute a 60x increase to pay for the constant costs associated with docker. Docker has tradeoffs.
sclevine 6y ago
Many comments here point out how difficult it is to manage a separate dependency stack for each container when you use Dockerfiles to build them. This problem is just as difficult, time-intensive, and security-critical for microservice apps running on K8s as it is for CLI tools and graphical apps. Worth pointing out that there is an incubating CNCF project that tries to solve this problem by forgoing Dockerfiles entirely: Cloud Native Buildpacks (https://buildpacks.io https://buildpacks.io) CNB defines safe seams between OCI image layers so that can be replaced out of order, directly on any Docker registry (only JSON requests), and en-mass. This means you can, e.g., instantly update all of your OS packages for your 1000+ containers without running any builds, as long as you use an LTS distribution with strong ABI promises (e.g., Ubuntu 20.04). Most major cloud vendors have quietly adopted it, especially for function builds: https://github.com/buildpacks/community/blob/main/ADOPTERS.md https://github.com/buildpacks/community/blob/main/ADOPTERS.m... You might recognize "buildpacks" from Heroku, and in fact the project was started several years ago in the CNCF by the folks who maintained the Heroku and Cloud Foundry buildpacks in the pre-Dockerfile era. [Disclaimer: I'm one of the founders of the project, on the VMware (formerly Cloud Foundry) side.]
- plg- 6y ago
  I hadn't heard of Buildpacks before, sounds very interesting. In particular the out of order layer replacement. I'm interested in switching to Buildpack for the images I maintain for my home cluster. Would make upgrading my base image so much simpler compared to rebuilding all the other images! I read a bunch of docs/articles since reading your comment yesterday but couldn't find any mention of this, or better yet an example. Are there some docs I missed? (I didn't look into the spec.)
  - plg- 6y ago
    Nevermind, I realized that rebase is exactly that. I had misunderstood the docs.
    - sclevine 6y ago
      Don't hesitate to reach out on Slack if you have more questions: https://slack.buildpacks.io https://slack.buildpacks.io A few tips on rebase: (1) If you want to rebase without pulling the images first (so there's no appreciable data transfer in either direction), you currently have to pass `--publish`. (2) If you need to rebase against your own copy of the runtime base image (e.g., because you relocated the upstream copy to your own registry), you can pass `--run-image <ref>`.
lenkite 6y ago
My Macbook fans rev up to the max after starting the Docker VM. And battery life falls like water running downhill. No, thanks - I shudder at the existential nightmare at being forced to do all development via Docker. Use Docker for deployment, exit the VM and get back to your work.
juancn 6y ago
Docker is ok. But please do not over use it. It's not really portable, it only runs decently on Linux, on other platforms it's a kludge.
KoenDG 6y ago
I recall someone building a docker image specifically for the purpose of setting up a development environment. A large amount of inter-connected projects for a SaaS-like service, all mounted inside the docker container, with some very specific libraries pre-compiled in the docker image, that used to take 30+ minutes to build. And now they're just in a docker image, that will work on linux and mac and I guess windows too. Nice idea in theory.

199 points by psxuaw 6y ago

I run nothing in Docker, ever. It’s never prevented me building sophisticated systems.

> If you’re using Docker on a Mac and you’ve never tried it on Linux, you owe it to yourself to try it on Linux. So ultimately the recommendation is to install Linux and use it as my daily driver. But Linux does not run on my machine yet.

Steltek 6y ago
And MacOS does? Is this some Apple Silicon humble brag? Cuz that's "Apple doesn't allow developers and enthusiasts to do their thing on their own computer that they purchased" not "Linux doesn't run on my machine".
- jolux 6y ago
  Apple definitely allows it, but as far as I know Linux doesn’t support it yet. (turn off SIP, couple other things related to the boot process I think)
- collinvandyck76 6y ago
  OP's point stands. You assigning them with "humblebrag" intentions is on you. It's not unreasonable to think that many people have M1s now, and it will certainly increase over time.
- viraptor 6y ago
  Not even Apple silicon. Linux is not compatible with MBPs since the touchbar models. More because of "Apple doesn't care you want this" (wifi firmware is known-broken) rather than disallowing.
- eyelidlessness 6y ago
  Why wouldn’t Apple allow Linux to run on their ARM Macs? They’ve allowed any compatible OS to run on Intel and PPC Macs. The only thing that’s changed is that they don’t currently offer a first party multi-boot solution. But they didn’t even offer one for Windows on Intel until community solutions were relatively stable and popular, and afaik they never offered one for Linux.

sdepablos 6y ago

I'm in the same situation: until I figure (if ever) how to use NixOS and control version my OS, I tend to use Docker for any application with lots of dependencies, specially Python related ones

physicsgraph 6y ago
Did you try the Docker-based approach after trying Python's virtualenv? Or did you go directly to Docker?
- sdepablos 6y ago
  No, I've using virtualenv for quite some time and I still use it when I'm the person in control of the dependencies, though I'm moving little by little to the docker way. Why? Because sometimes the problem is not only on the dependencies, but in Python itself. As an example, as of today gcloud does not support Python 3.9.
fuzzybear3965 6y ago
I've recently spent a few hours in NixOS land. I even built my own ISO and booted into it from an SD card. If you want a 30 minute intro to getting started then just ping me. I'm en route to something like what you want and I'm not far ahead of you.
setheron 6y ago
Reach out to me if you want to learn Nix I'm always happy to help people climb the steep learning curve
- stack_underflow 6y ago
  Curious, does one need to use nixOS for an optimal experience? I've been using debian for a long time and would prefer not to have to switch OSs but the idea of using nix for having full control over my package graph is very tempting. I've been somewhat procrastinating on trying nix as I've heard GNU Guix has a similar feature set and haven't been able to decide on which one to dive into... My ideal setup would be to just be able to run a single shell script that configures a new machine to the exact state of all my other dev machines. I have a shell script that somewhat does this but it's not completely unattended and still requires a lot of manual config for certain steps. Aside from that my main use case is being able to easily share a dev/build environment with others for ensuring that they can compile a certain project exactly as I do. For now I just use docker but it's frustrating not having explicit control over the layer cache and being able to tell it what to cache and what not to.

physicsgraph 6y ago

A few years ago I set a goal to minimize the number of applications installed bare-metal on my laptop. Everything is containerized, just as the blog describes. There is some initial overhead, and some extra work associated with maintaining the infrastructure, but overall I can report improved reproducibility (due to an isolated and static environment per app). I use a bash alias for each common application. For example, "jup" launches a jupyter notebook. I have a container with the Python package "Black" which runs using a git hook to clean up my code prior to commits.

deleted 6y ago
[deleted]
qmmmur 6y ago
That just seems insane to be running black inside a container. Why dont you just configure your environment properly?
- antoinealb 6y ago
  Serious question: Why do you believe that to be insane? The Dockerfile for a simple application such as black must be very short (probably 3-4 lines), the alias is probably quite short and the container overhead time is minimal for native docker (the story might be different for things like docker mac). On the other hand, you get some benefits from installing black through docker rather than through the system package manager: is is completely isolated from the host and the only way to break black is to update it, changing anything in the host will not break your black install. I am not sure either what you mean by "just configure your environment properly", but I am going to assume you mean installing black under a virtual env or equivalent? Then it is also annoying for different reasons: you must reinstall it once for each project, updating python to a new (major) version breaks your formatter, you cannot move the env around, to name the ones that come on top of my head.
  - viraptor 6y ago
    > you must reinstall it once for each project You can install it either in a venv outside of all projects or even "pip install --user black". > updating python to a new (major) version breaks your formatter Uninstalling the old version breaks the formatter. Installing a new one does not. Either way, with asdf, pyenv, and others you can keep all relevant version around. > you cannot move the env around Sure you can. Use "--relocatable" I agree docker may be nicer if you're not working day to day in python... but in that case black is not a great example. You probably want to have a specific version bound to the project so everyone uses it (including the ci platform).

pjmlp 6y ago

Docker is the NoSQL of 2020.

zoover2020 6y ago
Why so, though?
- pjmlp 6y ago
  The usual hype curve that everyone should be using it no matter what kind of workloads.
young_unixer 6y ago
Not really. Containerization has clear and obvious advantages over the traditional bare-metal approach.
- pjmlp 6y ago
  For security purposes as used since the early days in mainframes and big iron UNIX, surely. As workaround for broken dependency management, not really.
  - stickyricky 6y ago
    Whats the alternative?
    - pjmlp 6y ago
      Fix the dependencies story in what concerns Ruby and Python. Native binaries can easily do with static linking (fix glibc or replace it with proper libc like musl). Other platforms are doing quite fine without containers.
      - young_unixer 6y ago
        I kind of agree with you. But in the Linux world no one has the intention to fix the package distribution problem except for that AppImage guy.

wa1987 6y ago

> On a Mac, there is a major performance hit whenever you do disk IO in a bind mount (i.e. voluming a directory of the host system into the container). Working without bind mounts is extremely limiting. [..] If you’re using Docker on a Mac and you’ve never tried it on Linux, you owe it to yourself to try it on Linux. Or use named volumes. I'm running a dockerized WordPress dev environment on my MacBook with average TTFB's of 40 ms.

stack_underflow 6y ago
I had to investigate why our docker-ized dev environment was so slow on macOS at my last job and this was the root cause. One project's test suite ran in about 5-10 mins in our linux CI/CD environment, and about 50+ mins in macOS with docker. There was a very in-depth thread on the docker forums where the devs explained why there was such a huge performance penalty. IIRC it was due to all the extra bookkeeping that had to be done to ensure strong consistency and correct propagation of file system events between the virtualized docker for mac environment and the host file system. The test suite would run integration tests that performed a lot of npm/yarn operations which meant lots of disk IO.
- oauea 6y ago
  The reason is that docker on mac runs in a Linux VM and VM mounts are slow.
oauea 6y ago
> Or use named volumes And how do you access these from your host with high performant IO?
- wa1987 6y ago
  There’s no need to in my situation. 99% of the files are dependencies installed through Composer (PHP) while building the Dockerfile. The few directories I need to work on directly are bind mounts and don’t have much impact on overall runtime performance.

the_duke 6y ago

For end user devices, I much prefer Nix/NixOS [1] for this kind of thing. With Flakes [2] (experimental feature), you get full reproducibility. The documentation is spotty and there is a considerable learning curve, but I've switched to NixOS on my laptop and desktop early this year and am mostly very happy with it. That doesn't cover sandboxing though. I would actually agree that sandboxing / restricting applications (Mac OS style) would be sorely needed on Linux desktop. Flatpak and Snap are trying to do this. A much saner approach than trying to find or maintain up to date, trustworthy Docker files for applications. [1] https://nixos.org/ https://nixos.org/ [2] https://nixos.wiki/wiki/Flakes https://nixos.wiki/wiki/Flakes

timClicks 6y ago
Snap provides a strong sandboxing layer, but IIRC Flatpak does not.
- the_duke 6y ago
  That is incorrect. https://docs.flatpak.org/en/latest/sandbox-permissions.html https://docs.flatpak.org/en/latest/sandbox-permissions.html
deleted 6y ago
[deleted]
ris 6y ago
Absolutely. Most Dockerfiles I come across are very much not written in a robustly reproducible way. The stacking filesystems model is really not very flexible (if you want to compose two docker images with dissimilar bases to be used together you have to resort to manually specifying which files to copy across from each base). Use of OS features to achieve its ends means the docker daemon itself requires far more privileges than I'm comfortable with and makes weirdness like docker-in-docker necessary...
- mikepurvis 6y ago
  Yeah, any Docker file that starts with `apt update; apt dist-upgrade` is an instant fail. You might be interested in this fascinating exploration of attempting to bend Docker into better caching and composition by injecting blocks of Nix packages as individual layers: https://grahamc.com/blog/nix-and-layered-docker-images https://grahamc.com/blog/nix-and-layered-docker-images

mkl95 6y ago

My personal computer doubles as my workstation. I manage my Python virtual environments with virtualenvwrapper, and I install all the other stuff -mostly OpenGL and gamedev related tools- with apt or from source. Many developers would consider this Ubuntu 20.04 install that I run on a 9 year old computer to be an accident waiting to happen. But I'm finding this Ubuntu release to be extremely stable even at 3000+ packages.

jasoneckert 6y ago

I run most of my apps in containers on my developer workstation as well. Once containers became popular and I noticed that a few of my apps were available as container images on DockerHub, it just made sense from a resource and security point of view. This is the first post I've seen advocating it, but I think it is sage advice for anyone.

protoman3000 6y ago
I don’t understand. How can you find a random image maintainer on DockerHub more trustworthy than the rigorous package maintainers for your average distro?

harryruhr 6y ago

Docker for every application? If you create and maintain the Docker image or Dockerfile for every applucation yourself, you must have plenty of time. If you rely on public images from Docker Hub, you must have plenty of trust in the creators of those images.

cortesoft 6y ago
Hmmm... what alternative does not either take time or trust?
- harryruhr 6y ago
  It's all relative, of course. But getting a signed package from the repo of the distro I'm using for years is something different than using a random image from hub.docker.com.
  - cortesoft 6y ago
    Sure, but you are still relying on trust, and you are choosing to limit yourself to things released by your chosen distro. This is the same as if you were to pick a specific docker publisher that you trust, and only use their images.
    - t-writescode 6y ago
      It's arguable that it's not quite the same. It all comes down to consequences. If a distro messes up the trustworthiness of an application, they, the big and important company loses clout. If the application developer messes up, they also lose clout - people may stop using their software. Chances are, if you're using a third party for a third party piece of software that isn't officially dockerized by the company that developed it, nor a major distro, there's no real backlash if it doesn't work or if they get hacked, etc: "it was a third party trick, so _of course_ it wasn't trustworthy" would be the statement everyone makes. Debian messing up, or Cisco or Oracle, etc, is a much bigger deal.
- jillesvangurp 6y ago

lrossi 6y ago

The advantage of running apps in docker is that you avoid the risk of breaking your system by adding 3rd party package repos or even worse, curl | sudo bash. Cleanup/uninstall is very simple as well. However, I don’t agree with using it for sandboxing for security. Especially if you are giving it access to the X11 socket, as the author does in the examples. It might not be obvious, but the app will have access to all your other open windows.

gcb0 6y ago
> It might not be obvious, but the app will have access to all your other open windows. Fun fact, most docker hosts will allow access to all your files anyway! (specially true on docker for mac, which all the cool kids(tm) here are using). Even if you restrict container host-FS access to a source repo dir, mind rogue code changing your .git hook scripts in there or you might run code outside of the container when committing ;) Another slightly relevant fun fact, USB is a bus. That means that any device can listen in on any other device. And USB access is given by default to some X-enabled docker (--tty something), and to most virtualbox machines (including the hidden one running the fake docker linux host on docker-for-mac), and more recently Google-Chrome. ;)
- zapita 6y ago
  > and to most virtualbox machines (including the hidden one running the fake docker linux host on docker-for-mac) docker-for-mac does not use virtualbox.
  - t-writescode 6y ago
    Docker For Mac does use a virtual machine to run the Docker machine, though. It's complicated to access the automatically generated mount point on Docker's mac when you create a virtual folder but don't bind it anywhere.
    - zapita 6y ago
      But does Docker for Mac give unlimited access to the host’s usb bus like GP claims? I don’t see any evidence that it’s true, and the mistaken claim that d4m uses virtualbox increases my skepticism.

deleted 6y ago

[deleted]

Macha 6y ago

I've actually gone the opposite direction in my home lab to run less in docker - unless you're building your own images it's hard to know what's in them, which images are built on out of date OS images or dependencies, what else is running etc.

hendry 6y ago

Just some counter arguments to @jbergknoff's well put together page! Docker is the best medium for distributing - A static file is far easier to share / distribute. Cross-platform - You need an arguably complex and unstable Linux interface to run Docker images, cgroups et al Sandboxed - security claims about Docker have always been controversial. Simple Unix/BSD constructs like chroot/jails are far simpler and they are reliable Version pinning - a binary can embed a version and you can stick with it Reproducible - Everyone gets confused about Docker image checksums. `sha1sum static-binary` is far far simpler. Minimizes global state - wouldn't be a problem is people built static binaries.

FartyMcFarter 6y ago
Static linking is nice, but there are some licenses (notably the GPL and LGPL, which glibc uses) which don't easily allow that for closed-source software. On the technical side, as soon as some code needs to 'dlopen' something (e.g. a plugin or a system driver), you can run into trouble due to multiple instances of glibc or other dependencies running together. But if none of those are a requirement for your use case (or you have workarounds), I agree that statically linked binaries can be a nicer solution than Docker.
- foolmeonce 6y ago
  You can statically link whatever you like, you just can't distribute it as one work. One of the crazy one executable docker containers strikes me as one work to whatever extent a static linked binary is.
  - FartyMcFarter 6y ago
    > You can statically link whatever you like, you just can't distribute it as one work. How do you distribute it then? Let's assume your statically linked binary contains both closed-source code and GPL/LGPL code. > One of the crazy one executable docker containers strikes me as one work to whatever extent a static linked binary is. I'm not a lawyer, but that's not my understanding. A docker image is a glorified collection of files with some metadata, just as a tar file is. I think it's broadly agreed that you're allowed to distribute a tar file with unmodified LGPL dynamic libraries in it without having to open-source all of your code, and I think docker images are treated the same way?

chaitanya 6y ago

If you are on a Mac or Windows, this approach only works for utilities that can run on Linux. If you need to run a native Mac program for example, docker won't be able to run that. Perhaps Nix might help here, but I've never used it, so can't say for sure.

karlding 6y ago

The blog post links to Jessie Frazelle's GitHub repo [0]. Jessie also has a blog post about this [1] from back in 2015. If you prefer video format, Jessie also has a talk at DockerCon SF 2015 [2]. [0] https://github.com/jessfraz/dockerfiles https://github.com/jessfraz/dockerfiles [1] https://blog.jessfraz.com/post/docker-containers-on-the-desktop/ https://blog.jessfraz.com/post/docker-containers-on-the-desk... [2] https://www.youtube.com/watch?v=cYsVvV1aVss https://www.youtube.com/watch?v=cYsVvV1aVss

systemvoltage 6y ago

Let's not. I don't want to install Chrome which is already 73 MB, now bloated up with a whole lotta bullshit that's 500 MB+ image. Imagine downloading every application as a docker container. WTF. Docker is for distribution of applications when deploying them to servers. As a developer, it's amazing at that and have brought peace and joy in devops. Let's leave it there, shall we?

peteretep 6y ago
> Imagine downloading every application as a docker container. WTF. Help me understand the WTF here, and also how that’s meaningfully different from an OS X app?
- systemvoltage 6y ago
  Jumping from 73 MB -----> 500 MB or whatever. Have you been running Docker on your system? Have you tried running: `docker system prune -a`? It's gonna print something like Total reclaimed space: 31.2GB.
  - peteretep 6y ago
    wtf are you doing to create such large Docker containers? Are you installing each app on top of a fully-fledged Ubuntu? Are you leaving the build tools in the final container?
    - systemvoltage 6y ago
      That's what I am trying to ask myself too! I am sorry that I am a developer and I need to run a bunch of docker compose files.
    - fantod 6y ago
      I mean it could just be very many containers that add up. And in many cases you'll have loads of images close to 1GB unless you go to great lengths to try and shrink them (e.g. anything that uses Python).

johnsoft 6y ago

I appreciate the sentiment, but I wouldn't be able to put up with the slow container boot time for regular use. The author mentions this at the bottom. He claims 1sec delay. Last month I benchmarked it on new-ish hardware at 560ms, or 290ms if you disable namespaces (and so disable isolation). I wanted to also benchmark bocker[1] (docker written in bash) for a baseline comparison, but it no longer runs and I threw in the towel after ~30 mins of tinkering. Anyways, if you run something like `curl | jq | grep | less` you could be waiting a fair bit for all those containers to start. Package managers are pretty good these days. I think I can trust it to install `jq` properly. [1]: https://github.com/p8952/bocker https://github.com/p8952/bocker

tomcam 6y ago
> curl | jq | grep | less OK I didn’t know about this. Does each one of the piped commands cause the creation of a new container? And why?
- djhworld 6y ago
  Assuming you've bound an alias to these commands to run them individually, each in their own container
  - tomcam 6y ago
    Thank you. Why on earth would one wish to bind these commands to their own containers? Or was that just a trivial example for illustrative purposes that I’m obsessing over?

harryruhr 6y ago

Docker has its use cases. But running everything possible in a docker container just for the sake of "dockerizing" seems a little bit excessive. One must basically maintain a (more or less) complete userland environment for every application. The idea of shared libraries is taken to absurdity this way. It would be better to build all static. Then there is the waste of resources. I'm sure with plenty GB RAM, TB of SSD space and GBit of bandwith available nowadays many people don't notice. But for what?

cinquemb 6y ago
> Then there is the waste of resources. I'm sure with plenty GB RAM, TB of SSD space and GBit of bandwith available nowadays many people don't notice. But for what? Yeah, you deff notice if you are trying to scale as cheaply as possible…

tracker1 6y ago

I'm not sure I'd reach for this in some of the use cases over snap/flatpak/appimage for GUI applications. For some terminal apps, maybe, but even then I usually just user my dockerfile or docker-compose file to manage these needs. What's really needed are better/more articles about how to clean off the cruft left by docker after running your builds locally. Especially when you do things like update your base image versions from node:12 to node:14 etc.

gigel82 6y ago

I wish there were real / usable Windows containers out there. I'd love to spin up a quick sandbox to run a MSVC / MSBuild, but the smallest "image" in Windows-land for that is 12Gb. Still use Docker alongside WSL2 for purely-Linux stuff like some node.js scripts or python things that don't need GPU.

nyellin 6y ago
I'm looking at startup ideas in this area. Would you be interested in describing your pain points and use case?
- gigel82 6y ago
  I just want Docker containers for Windows with a reasonable size. They do have a Server Core (at about 1.5Gb) and a much slimmer NanoServer (100Mb), but the NanoServer is not capable of running the Visual C++ compiler for example. Just look at something like this which tries to compile C++ for VcPkg: https://hub.docker.com/r/hripko/vcpkg/tags?page=1&ordering=last_updated https://hub.docker.com/r/hripko/vcpkg/tags?page=1&ordering=l... (the Linux images are all < 300Mb, but the Windows one is 5Gb compressed).
newscracker 6y ago
It’s not clear to me what your requirements are, but take a look at Sandboxie for Windows. [1] [1]: https://github.com/sandboxie-plus/Sandboxie https://github.com/sandboxie-plus/Sandboxie

alexpapworth 6y ago

No thanks.

ridiculous_fish 6y ago

Docker is intimately tied to Linux. It it only "cross platform" in that it can use a VM (Mac, WSL2) or flaky compat layers (FreeBSD). If software embraces Docker, it effectively excludes other OSes, like Windows, BSDs, Haiku, Fuschia, etc.

ed25519FUUU 6y ago
Even though it doesn’t run cross platform natively, it does run cross platform acceptably. I have no issue developing, building, and shipping images on my Mac that are deployed to production.
- chii 6y ago
  in that case, why is it that docker is considered superior when you could virtualize the windows api and run native windows apps in linux instead?
zachrose 6y ago
And even then trying to run Docker on Fedora will prompt you to turn off Security-Enhanced Linux features or use Podman instead. So it’s really like, what, 1/2 or 2/3 or Linux support
- cpuguy83 6y ago
  What? Why would it prompt this? Docker supports SELinux.
  - remexre 6y ago
    I don't know the specific issue, but RH's been trying to push Podman fairly hard...
    - dajonker 6y ago
      Fedora is using cgroups v2 by default since last year. This version of cgroups was not supported by docker until a couple of weeks ago. So as a Fedora user you could either use podman or modify your installation to use cgroups v1 instead.

zaptheimpaler 6y ago

I prefer to just make a big "devbox" docker with all the utilities I need installed, and the home directory as a docker volume. The advantage is i don't have to fiddle with containers for every little program, no startup delay for every program, and my base ubuntu install stays clean and stable (which runs a VM and other services so stability is important). It does have a few warts, the main one being you can't have the entire container root volume be a docker volume, so to install new programs persistently inside the devbox i have to rebuild the image (if they don't live completely in the home volume). But logically its an okay tradeoff because the only way to make the whole environment reproducible is to specify things at build time.

fistfucker3000 6y ago
Is it possible to run your IDE within these docker environments? I'd love to set up a do it all docker container for the front end devs at my work.
- amgreg 6y ago
  I use the Remote Development extension for Visual Studio Code, which, among other things, allows you to attach an editor window to a running container. https://code.visualstudio.com/docs/remote/containers https://code.visualstudio.com/docs/remote/containers

timhigins 6y ago

Just want link to Whalebrew, which achieves a lot of what this article mentions but IMO is more user-friendly. https://github.com/whalebrew/whalebrew https://github.com/whalebrew/whalebrew

mongol 6y ago

I am surprised that systemd portable services is so little known. It solves many of the same problems as Docker does but is comparatively simpler. It also requires more plumbing on top but I prefer it.

alphachloride 6y ago

I find that the time taken to get everything working means I have less time to get work done. The busywork does give me a sense of achievement, but that is not the achievement that matters. It's premature optimization to sandbox an application by default unless there is a pressing reason to do so. For e.g. zoom and hugo don't really change the filesystem or OS settings beyond the folders they output to. I don't see a reason the have them sandboxed personally.

thefunnyman 6y ago
Isn’t that precisely a use case for sand boxing? If I know zoom doesn’t need file system, I can deny it access and guarantee it doesn’t use the file system without me knowing about it.
- alphachloride 6y ago
  If you don't trust an application, sandboxing is one way to go. But that goes all the way down. If you don't trust docker and the OS, run them in a VM. If you don't trust the VM, run it on a spare computer. If you don't trust that computer's hardware, then use minecraft to make a PC with your own instruction set :) Same with networking. Use HTTPS. No? Also use a VPN. No? Make your own VPN. No? Use smoke signals with one-time pad encryption.. In the end you decide at what point are you willing to delegate responsibility for things working as they say they should.

TimTheTinker 6y ago

> On a Mac, there is a major performance hit whenever you do disk IO in a bind mount (i.e. voluming a directory of the host system into the container). Working without bind mounts is extremely limiting. In my opinion, this is a big enough issue as to throw the whole idea of “Docker as cross-platform platform/target” into major question. When running more than a few containers on macOS, the performance is so bad it becomes almost unusable.

Spivak 6y ago
The only reason it works so well on Linux is because containers are bog standard Linux processes running with no real overhead sans the kernel structures needed to maintain the namespacing. Entries in the VFS are cheap. Once you need a hypervisor most of the benefits are gone. But if you need the hypervisor anyway which might be the case for software that doesn’t have Mac builds then it starts to look attractive.
lloeki 6y ago
Volumes to macOS host performance has nothing to do with bind mounts. It’s the VM folder sharing that’s crap, doubled with the attempt at Docker for Mac to translate filesystem events from fseventsd to inotify and back. Changing solutions (e.g Docker for Mac -> VirtualBox or Fusion via e.g docker-machine) makes a world of difference.

nyellin 6y ago

I've been looking at startup ideas in this space, either as a Citrix replacement (run applications locally in a secure enclave instead of running them in the cloud and paying for vms) or as a way of bringing the benefits of containers to desktop apps and traditional IT (build containers for deployments not images). Would love to hear from anyone with use cases or pain points.

gorgoiler 6y ago

If you are reading this with interest then I encourage you to give it a go. However, Docker is like a kitchen machine: good for getting something done in a particular way (stand mixer for kneading dough) but less useful for understanding what’s actually happening (when is pizza dough ready?) On Linux, picking apart LXC stuff at the command line is very much worth your time if you’re interested in how things work. (Not LXD though which is more useful as a tool than as a teaching aid.) If you have some IPv6 allocation to play with you can make some interesting infrastructure and have it do something useful on the Internet without relying on the lxc-net crutches of an automatically built bridge with NAT. It all feels very well designed as a bag of tools to let one make things rather than a complete system that guides you in only one particular direction. I use stand mixers and Docker all the time; sometimes it’s fun to get into the details too.

ChicagoDave 6y ago

This might lead to developers not knowing or understanding their environment and I have a problem with that.

ogre_codes 6y ago

> Running a program in a container is a lot like running it normally, but the user doesn’t need to jump through hoops to configure the system, build and install. Docker is itself a complex build tool which requires a bunch of install steps. If you are going to ship software to end users there is almost always a better way to bundle and ship than send someone a Docker container. Docker is not a distribution tool, if you are expecting your end users to install Docker, you've already screwed up. > Downloading a pre-compiled binary is almost like this, except with worse odds. Maybe there’s a build for your architecture. If it was statically linked, you’re golden. Otherwise, use ldd to reverse engineer the fact that you need to install libjpeg. On Mac and Windows this is almost never an issue. Even on Linux, it's pretty straight forward to statically link your binary if you aren't sure about the environment it's going to be run on. Statically linked binaries are a bit bloated... but not as bloated as a damned Docker image which contains entire dependency trees. In no case is "Making it into a Docker Image" a simpler/ better distribution mechanic.

tedk-42 6y ago
Sorry but you're wrong. A vast majority of software ships after it has been built as a stand alone application as well as a Docker container so that it can be run in container environments. A lot of CI systems natively support Docker because of the variety of tools and images you have available to run you CI steps in. Docker is an excellent distribution tool because it only requires Docker. You can send someone some python code and ask them to run it, only to find they're missing a bunch of C libs required to build and run the code which is a pain to help them figure out how to solve. Docker solves many of these problems but has a drawback of being more 'bloated' than other distribution mechanisms. It doesn't make it a bad one though.
- ogre_codes 6y ago
  > You can send someone some python code and ask them to run it, only to find they're missing a bunch of C libs required to build and run the code which is a pain to help them figure out how to solve. You can send someone a Docker file only to figure out they've never heard of Docker. Let alone don't have it installed and aren't interested in setting up a container system to your 500 line Docker script. All you've done is move your problem upstream. Unless your consumer is a web developer, you are out of luck.

akhilcacharya 6y ago

I think the reasoning is perfectly sound for doing this but the performance penalty is too much to bear for most use cases - what's preventing nixpkg/guix from being used fully for this use case? Lack of available packages?

masa331 6y ago

This brings only mess and another indirection layers to your system which is hardly an advantage in my opinion. It will take more time to setup, more space on disk, it will be slower and harder use. And all the normal tools(like awk and others in the article) are easily instalable in all distros. I will rather spend time learning these tools directly instead of infinitely fiddling with docker.

siscia 6y ago

I am trying to go in the opposite direction offering a read-only, mountable file system that offer a well crafted set-up of tools and utilities. The idea is that you mount the filesystem and you got all the tools you need, and more, well installed, and that are lazily pulled from the network. You win on the space side, but you need a bit more of trust. You can find more info here: http://packages.redbeardlab.com http://packages.redbeardlab.com On the GitHub repo where you can ask for more packages to be installed: https://github.com/RedBeardLab/packages.redbeardlab.com/ https://github.com/RedBeardLab/packages.redbeardlab.com/ And in this pair or articles for specific languages Golang: https://redbeardlab.com/2020/12/21/packages-redbeardlab-com-for-golang/ https://redbeardlab.com/2020/12/21/packages-redbeardlab-com-... And for JavaScript/node: https://redbeardlab.com/2020/12/23/packages-redbeardlab-com-for-node-js-community/ https://redbeardlab.com/2020/12/23/packages-redbeardlab-com-...

eptcyka 6y ago

Instead of using docker, why not use podman? I prefer my containers in userspace with user privileges.

freebuju 6y ago

Were it not for docker, I probably would have never learned about Linux. I'll share why. I was excited to find docker. It was neat and could be setup to update your applications without manual interaction. So I had it run my media server apps on a seedbox I had rented. Fast forward years later got a form factor pc that came with the free version of Windows. Try as a might, I couldn't get docker to work on it. Required the Pro version or something like that. I remember thinking, well that sure is stupid! That was the beautiful beginning of my affair with Linux as an OS for my desktop/server needs.

blunte 6y ago

TFA could make a small paid course walking a dev through the process of setting all this up and using it effectively. I would buy it.

sweeneyrod 6y ago

Docker has on the order of 1 second startup time doesn't it? I don't understand why someone would willingly subject themselves to that for things being manually run multiple times per day without an extremely compelling reason.

0x445442 6y ago

Because of various org restrictions I attempted to create a developers image on Windows 10. I got the container running but mounting the volumes back to my host workspace was not performant. A build that took 10 seconds on the host took 90 seconds in the container.

OCISLY 6y ago

I prefer zero-dependency single file executables. Elegant and easy to comprehend. Docker only adds another layer of abstraction and its advantages are questionable.

pavlus 6y ago

TLDR: 1. You already have that on Linux. 2. Make your builds portable. If you are running mainstream Linux(Debian-based/Arch-based, probably other), most of the Docker profits can be achieved with already installed and configured systemd and your distro's package manager. Sandboxed? systemd. Simple, uniform interface? Your distro has packages, and most likely services that can and should be sandboxed already run in systemd after installation, you can tune unit-file if you want, and systemd has security checker, that shows you what application in the sandbox can and can not do, without proxying things the Docker way. Versions pinning? Pin version with your package manager. Want multiple versions? Check out DebianAlternatives system. Reproducible? Fix your build/install configs, not the environment. If it builds on your machine, but not on the other, or run flawlessly on one, but not the other, it means you have implicit dependency on the environment, or wrong dependency versions constraints, which you likely don't know about. If you don't know your dependencies, you are definitely shooting blind. Minimizes global state. Repo-based distros minimize global state by providing software that has most dependencies compatible, so you can have your minimal state and update your software too. Meanwhile with docker you need all dependencies and hope that they will match between images, so you can save on layers reusage. And if someone decides to update base image, and others dont... well, too bad, you have to have both versions of the base image.

bayindirh 6y ago
I want to add another point to this TL;DR: 3. Do your research before writing these blog posts. I don't want to put down anyone for writing these blog posts. The idea is nice from a distance, but the reality doesn't work like that. It's shoehorning something not suitable for this situation. There's singularity which runs on non-root environments, however it's not for desktop systems, but multi-tenant clusters. I really get frustrated when people advocate expensive abstractions for minimal gains. We can use our processing power much more efficiently while keeping almost the same properties without the costly abstractions. Piling everything on top of each other to create impenetrable and immutable abstractions is not the way to achieve this. Docker already makes debugging very hard by being immutable and impenetrable as is.
deleted 6y ago
[deleted]

djhworld 6y ago

I'd be interested to know if Jess Frazelle, linked in the post still maintains her Dockerize-everything setup. I remember a few years ago her write-up on that stuff. Would be good to hear her thoughts a few years on

Arch-TK 6y ago

I get that doing APIs right is hard and that when you're trying to release a proprietary product having to worry about what libraries every system comes with may be a struggle and that if you're developing a webapp you may want everyone to have a consistent test environment regardless of what flavour of archlinux they're running but I really do NOT want to have to spend 2 hours downloading 300GiB of software updates when OpenSSL wets the bed and has a critical vulnerability. The nice thing about libraries and shared objects is that when I get an email saying: "Critical CVE found in something important that may actually affect you" I want to be able to run an update command which fetches a few libraries including the vulnerable one, reboot and be back in business. I also don't really care for the "we have so much space now" argument. I certainly don't, I don't put in expensive 2TB SSDs in my laptop because I don't need them, and I don't want half of my 500GiB disk to be taken up by giant blobs of unoptimized docker images for the same reason that I don't want to run 10 copies of chrome at the same time to use a text editor, an email client, a web browser, a media player, a debugger, the thing I'm writing, three chat clients, and a partridge in a pear tree. I have extra space and extra processing power on my computer so that a: I can use it for the things I actually want to use it for and b: so that I have a snappy machine which can take an unexpected load (be it disk load or processing load) without problems.

harrisonjackson 6y ago

> slight startup delay On a mac this is a complete deal breaker for anything you need to run frequently. It is more than slight.

knur 6y ago

This reminded me of a proof of concept thing I wrote months ago, that facilitates this kind of work. https://github.com/casidiablo/macondo https://github.com/casidiablo/macondo I don't even know what it is I built, but it has been useful in some contexts. It basically allows you to easily wrap and distribute scripts (or more complex apps) that have specific dependencies that might not always be installed in the host. It does so by wrapping the script in a docker image. It also automates the annoying part of docker: mounting local paths for apps that need to interact with the host's file system. I need to write a blog post on this if anything to gather feedback. I'm still not 100% sold on the idea and there are some edge cases. Still, a fun experiment.

X6S1x6Okd1st 6y ago

Ugh at work the edit-compile-run loop only needs to be ~1 second (it's a fairly small go project), but our build system uses docker for everything, so now it's ~1 minute a 60x increase to pay for the constant costs associated with docker. Docker has tradeoffs.

sclevine 6y ago

Many comments here point out how difficult it is to manage a separate dependency stack for each container when you use Dockerfiles to build them. This problem is just as difficult, time-intensive, and security-critical for microservice apps running on K8s as it is for CLI tools and graphical apps. Worth pointing out that there is an incubating CNCF project that tries to solve this problem by forgoing Dockerfiles entirely: Cloud Native Buildpacks (https://buildpacks.io https://buildpacks.io) CNB defines safe seams between OCI image layers so that can be replaced out of order, directly on any Docker registry (only JSON requests), and en-mass. This means you can, e.g., instantly update all of your OS packages for your 1000+ containers without running any builds, as long as you use an LTS distribution with strong ABI promises (e.g., Ubuntu 20.04). Most major cloud vendors have quietly adopted it, especially for function builds: https://github.com/buildpacks/community/blob/main/ADOPTERS.md https://github.com/buildpacks/community/blob/main/ADOPTERS.m... You might recognize "buildpacks" from Heroku, and in fact the project was started several years ago in the CNCF by the folks who maintained the Heroku and Cloud Foundry buildpacks in the pre-Dockerfile era. [Disclaimer: I'm one of the founders of the project, on the VMware (formerly Cloud Foundry) side.]

plg- 6y ago
I hadn't heard of Buildpacks before, sounds very interesting. In particular the out of order layer replacement. I'm interested in switching to Buildpack for the images I maintain for my home cluster. Would make upgrading my base image so much simpler compared to rebuilding all the other images! I read a bunch of docs/articles since reading your comment yesterday but couldn't find any mention of this, or better yet an example. Are there some docs I missed? (I didn't look into the spec.)
- plg- 6y ago
  Nevermind, I realized that rebase is exactly that. I had misunderstood the docs.
  - sclevine 6y ago
    Don't hesitate to reach out on Slack if you have more questions: https://slack.buildpacks.io https://slack.buildpacks.io A few tips on rebase: (1) If you want to rebase without pulling the images first (so there's no appreciable data transfer in either direction), you currently have to pass `--publish`. (2) If you need to rebase against your own copy of the runtime base image (e.g., because you relocated the upstream copy to your own registry), you can pass `--run-image <ref>`.

lenkite 6y ago

My Macbook fans rev up to the max after starting the Docker VM. And battery life falls like water running downhill. No, thanks - I shudder at the existential nightmare at being forced to do all development via Docker. Use Docker for deployment, exit the VM and get back to your work.

juancn 6y ago

Docker is ok. But please do not over use it. It's not really portable, it only runs decently on Linux, on other platforms it's a kludge.

KoenDG 6y ago

I recall someone building a docker image specifically for the purpose of setting up a development environment. A large amount of inter-connected projects for a SaaS-like service, all mounted inside the docker container, with some very specific libraries pre-compiled in the docker image, that used to take 30+ minutes to build. And now they're just in a docker image, that will work on linux and mac and I guess windows too. Nice idea in theory.