10 ms·

Docker Compose reads your “.env” without opt-out

160 points by otobrglez 7y ago

moomin 7y ago
Stuff like this is why docker alternatives exist.
- rumanator 7y ago
  Requiring a daemon with root privileges is another big reason to explore alternatives to docker.
  - djsumdog 7y ago
    Have you looked at podman? You can do daemon-less, user run containers.
    - rumanator 7y ago
      I've heard of podman and buildah, but honestly I haven't looked too closely into it. They do sound like Docker done right.
- djsumdog 7y ago
  This doesn't have anything to do with Docker, but specifically docker-compose. What are the alternatives you suggest to that? Minikube?
  - moomin 7y ago
    Yeah, I like Kubernetes. But I’ll admit it’s not exactly the most inviting thing to get into. It’s cool once you get the hang of it. But the attitude runs all the way through docker products: arbitrary decisions made for their benefit with no thought to the externalities.
  - flowardnut 7y ago
    garden.io is what we're replacing docker-compose with.
- imtringued 7y ago
  I don't get it. I'm not using docker just to run containers. I'm also using it to build images. Therefore having more container runtimes doesn't actually allow me to switch away from docker.
LethargicStud 7y ago
This is the exact class of problem that docker itself attempts to avoid. This is why I run docker-compose inside a docker container, so I can control exactly what it has access to and isolate it. There's a guide to do so here[1]. It has the added benefit of not making users install docker-compose itself - the only project requirement remains docker. 1: https://cloud.google.com/community/tutorials/docker-compose-on-container-optimized-os https://cloud.google.com/community/tutorials/docker-compose-...
- djsumdog 7y ago
  > docker-compose inside a docker container Do you use Docker-in-Docker or do you mount the docker socket inside your docker-compose container? Oh dear god .. it's Docker all the way down.
  - LethargicStud 7y ago
    Mount the docker socket. There's some quirks with storage volume paths. Also, security implications. Was not super hard to get working though. I'd love to go straight to containerd or even basic linux containers but I'm not willing to run kubernetes on my personal machine and haven't found any ergonomic enough ways to run containers.
    - hosh 7y ago
      I'm more than willing to run K8S on my personal machine in the form of microk8s, k3s, minikube, and similar cut-down versions of k8s.
      - arcticfox 7y ago
        The others are probably fine, but for anyone thinking about this, minikube uses 50% CPU even on powerful machines for no reason [0]. I switched to kind and it works perfectly, super lightweight. [0] https://github.com/kubernetes/minikube/issues/3207 https://github.com/kubernetes/minikube/issues/3207
rshnotsecure 7y ago
Docker Hub still does not have any form of 2FA. Even SMS 2FA would be something / great at this point. As an attacker, I would put a great deal of focus on attacking company’s registries on Docker Hub. They can’t have 2FA, so the work/reward ratio is quite high.
- icedchai 7y ago
  Why would any serious company not use a private registry?
  - sneak 7y ago
    If you follow best practices your containers don't contain secrets; distributing credentials to access a private registry is itself a bit of overhead. Not everyone makes/uses software where the software itself is secret.
    - hosh 7y ago
      We're talking about fixing the underlaying images to a fixed point, not about baking in secrets into the image. And there are plenty of uses for using proprietary images. For example, I have done workflows where the properitary code is layered on top of open-source code. (Although if I were to redo those CI/CD pipelines again, I would use the de-facto buildpack standard instead; even then, the buildpack slug itself would be properitary and requires an artifact store of some kind). Either way, using GCP or AWS registry that integrates with their IAM systems works well. The newer versions of docker contains a plugin system, so both GCP and AWS command line tooling can hook in their own authentication mechanisms in there.
    - Cpoll 7y ago
      I think the risk in this case is an attacker writing to the registry, not reading it. If I can push changes to your 'prod' tagged image, I'm going to have a very good time.
  - xemdetia 7y ago
    You would be surprised on how large a hurdle this is, even inside 'serious companies' when stable external/internal artifacts come up. A lot of developers who cut their teeth on these public repositories complain heavily over the 'friction' of ensuring that they have a reliable place to pull things from. This isn't unique to docker and also includes things like python, ruby, go, node, and other modern packaging systems. The fact that in 3 years from now when they need to update the code the vulnerable/deprecated package is gone/changed URL/whatever seems to be a learned lesson.
djsumdog 7y ago
I was going to use Docker Compose for setting up a lot of my self hosted stuff; since so many projects already had docker compose files in their git repos and it would seem to be easy to leverage that. Early on I got super frustrated with compose though (can't remember all the reasons why) and ended up just writing my own custom provisioner: https://github.com/sumdog/bee2 https://github.com/sumdog/bee2 It has unit tests, but not a lot of good errors messages and is pretty specific to the stuff I host. I'm glad I did it though; great learning experience around the Docker API and how the internals of the Docker Engine work. I still use it to maintain all my self-hosted sites and tooling. There are a lot of good libraries around the Docker API for Ruby, Python, Java/Scala, etc. If you're on a green field project setting up your local docker environments, and have the time, I'd almost suggest trying to build your own tooling from scratch rather than leveraging docker-compose at this point.
- sdan 7y ago
  I did exactly what you said you had problems with: https://sdan.xyz/sd2 https://sdan.xyz/sd2. I use Docker Compose and Traefik (for routing) and self hosted a bunch of stuff given that most self hosting programs have some type of Docker support.
- GordonS 7y ago
  I actually really like Docker Compose, and especially Swarm. Despite being a relative Docker noob, I've found it to be relatively straightforward to use, and the configuration and secret options are pretty comprehensive. Haven't hit any issues I recall. Would CNAB work with a custom provisioner, or would support for that need to be coded in?
seanhunter 7y ago
No great fan of docker and on a train with a demo laptop and no way to install it to test, but I would expect that > docker-compose --env-file /dev/null will get it to not read a .env file. Will test later/tomorrow.
- joshspankit 7y ago
  They do mention that being an option in the comments on the bug report, but still sound the alarm as that’s (seemingly) the only option. One-off fixes are possible, but that’s not a viable solution at scale.
- otobrglez 7y ago
  This doesn't work. At least not with 1.24.1.
hiccuphippo 7y ago
As a workaround you could rename your .env file to something else and mount it as .env in the docker-compose volume options.
- Ensorceled 7y ago
  How does this help all the existing applications, tools, scripts etc. that use .env?
  - otobrglez 7y ago
    If you wanna make it work; this is sadly the only way until Docker Compose is patched. Rename .env to something else and reconfigure autoenv.
    - Ensorceled 7y ago
      Right. Usually workarounds actually, you know, work :-P
  - cavemonkey 7y ago
    Those applications running inside the container would see .env as the mounted file but Docker-compose would still see the original .env file in the filesystem.
- otobrglez 7y ago
  I'm using autoenv - https://github.com/inishchith/autoenv https://github.com/inishchith/autoenv. The workaround that I've found for this problem is to set "AUTOENV_ENV_FILENAME=.env.user" and then inside the project folder, I'm now using ".env.user" instead of just ".env". I'll use this hack until Docker guys fix Docker Compose.
  - flukus 7y ago
    Does this (and the one linked to: https://direnv.net/docs/development.html https://direnv.net/docs/development.html) do anything that a simple bash entry can't: function lenv { if [ -f '.env' ]; then source .env fi } function cd { builtin cd $@; lenv; }
rubyn00bie 7y ago
Call me ol' fashioned, but if I'm not administrating hundreds or thousands of micro-services... I just use "Plain Ol' Unix Processes on ZFS," or as I lovingly call it "POUPZ." I can honestly say this exact issue is impossible for anyone who POUPZ. If you haven't tried POUPZ in production, or even POUPZ at home, my recommendation is to give it a try. I think you'll be pretty glad for once that you POUPZ where you eat.
- dgoog 7y ago
  but what if I'm trying to pretend to be an engineer and I'm really a barista that doesn't know how to linux? I have a macbook and know node.js - doesn't that help?
- yjftsjthsd-h 7y ago
  Does this address the package-like behavior that is one of docker's big selling points? I guess I could imagine building something out of zfs send+receive, but it feels like you'd need tooling to make it nice, and now you've reinvented the wheel.
lcfcjs2 7y ago
just delete your .env file, that file should never be in a code repo any way. If it is, you're doing it wrong.
gravypod 7y ago
You can run docker-compose.yml in any folder in the tree but it only reads the .env from cwd. Just CD into some place and run docker-compose
- ddtaylor 7y ago
  Thank you for the clarification.
- ohiovr 7y ago
  This works. I discovered this by accident trying to figure out why my .env files were not being read. I was in a different dir to the docker-compose.yml file and launched docker-compose with the -f filename option and could not get .env to load.
deleted 7y ago
[deleted]
abhchand 7y ago
For reasons like and to avoid confusion I always name my environment file something specific like `.env.development` or `.env.dockercompose` Any system that reads `.env` files usually allows some way to specify the exact file to be read.
- GordonS 7y ago
  Huh, I wasn't aware of this! If nothing else, I like the idea of separate files for separate concerns.

160 points by otobrglez 7y ago

moomin 7y ago

Stuff like this is why docker alternatives exist.

rumanator 7y ago
Requiring a daemon with root privileges is another big reason to explore alternatives to docker.
- djsumdog 7y ago
  Have you looked at podman? You can do daemon-less, user run containers.
  - rumanator 7y ago
    I've heard of podman and buildah, but honestly I haven't looked too closely into it. They do sound like Docker done right.
djsumdog 7y ago
This doesn't have anything to do with Docker, but specifically docker-compose. What are the alternatives you suggest to that? Minikube?
- moomin 7y ago
  Yeah, I like Kubernetes. But I’ll admit it’s not exactly the most inviting thing to get into. It’s cool once you get the hang of it. But the attitude runs all the way through docker products: arbitrary decisions made for their benefit with no thought to the externalities.
- flowardnut 7y ago
  garden.io is what we're replacing docker-compose with.
imtringued 7y ago
I don't get it. I'm not using docker just to run containers. I'm also using it to build images. Therefore having more container runtimes doesn't actually allow me to switch away from docker.

LethargicStud 7y ago

This is the exact class of problem that docker itself attempts to avoid. This is why I run docker-compose inside a docker container, so I can control exactly what it has access to and isolate it. There's a guide to do so here[1]. It has the added benefit of not making users install docker-compose itself - the only project requirement remains docker. 1: https://cloud.google.com/community/tutorials/docker-compose-on-container-optimized-os https://cloud.google.com/community/tutorials/docker-compose-...

djsumdog 7y ago
> docker-compose inside a docker container Do you use Docker-in-Docker or do you mount the docker socket inside your docker-compose container? Oh dear god .. it's Docker all the way down.
- LethargicStud 7y ago
  Mount the docker socket. There's some quirks with storage volume paths. Also, security implications. Was not super hard to get working though. I'd love to go straight to containerd or even basic linux containers but I'm not willing to run kubernetes on my personal machine and haven't found any ergonomic enough ways to run containers.
  - hosh 7y ago
    I'm more than willing to run K8S on my personal machine in the form of microk8s, k3s, minikube, and similar cut-down versions of k8s.
    - arcticfox 7y ago
      The others are probably fine, but for anyone thinking about this, minikube uses 50% CPU even on powerful machines for no reason [0]. I switched to kind and it works perfectly, super lightweight. [0] https://github.com/kubernetes/minikube/issues/3207 https://github.com/kubernetes/minikube/issues/3207

rshnotsecure 7y ago

Docker Hub still does not have any form of 2FA. Even SMS 2FA would be something / great at this point. As an attacker, I would put a great deal of focus on attacking company’s registries on Docker Hub. They can’t have 2FA, so the work/reward ratio is quite high.

icedchai 7y ago
Why would any serious company not use a private registry?
- sneak 7y ago
  If you follow best practices your containers don't contain secrets; distributing credentials to access a private registry is itself a bit of overhead. Not everyone makes/uses software where the software itself is secret.
  - hosh 7y ago
    We're talking about fixing the underlaying images to a fixed point, not about baking in secrets into the image. And there are plenty of uses for using proprietary images. For example, I have done workflows where the properitary code is layered on top of open-source code. (Although if I were to redo those CI/CD pipelines again, I would use the de-facto buildpack standard instead; even then, the buildpack slug itself would be properitary and requires an artifact store of some kind). Either way, using GCP or AWS registry that integrates with their IAM systems works well. The newer versions of docker contains a plugin system, so both GCP and AWS command line tooling can hook in their own authentication mechanisms in there.
  - Cpoll 7y ago
    I think the risk in this case is an attacker writing to the registry, not reading it. If I can push changes to your 'prod' tagged image, I'm going to have a very good time.
- xemdetia 7y ago
  You would be surprised on how large a hurdle this is, even inside 'serious companies' when stable external/internal artifacts come up. A lot of developers who cut their teeth on these public repositories complain heavily over the 'friction' of ensuring that they have a reliable place to pull things from. This isn't unique to docker and also includes things like python, ruby, go, node, and other modern packaging systems. The fact that in 3 years from now when they need to update the code the vulnerable/deprecated package is gone/changed URL/whatever seems to be a learned lesson.

djsumdog 7y ago

I was going to use Docker Compose for setting up a lot of my self hosted stuff; since so many projects already had docker compose files in their git repos and it would seem to be easy to leverage that. Early on I got super frustrated with compose though (can't remember all the reasons why) and ended up just writing my own custom provisioner: https://github.com/sumdog/bee2 https://github.com/sumdog/bee2 It has unit tests, but not a lot of good errors messages and is pretty specific to the stuff I host. I'm glad I did it though; great learning experience around the Docker API and how the internals of the Docker Engine work. I still use it to maintain all my self-hosted sites and tooling. There are a lot of good libraries around the Docker API for Ruby, Python, Java/Scala, etc. If you're on a green field project setting up your local docker environments, and have the time, I'd almost suggest trying to build your own tooling from scratch rather than leveraging docker-compose at this point.

sdan 7y ago
I did exactly what you said you had problems with: https://sdan.xyz/sd2 https://sdan.xyz/sd2. I use Docker Compose and Traefik (for routing) and self hosted a bunch of stuff given that most self hosting programs have some type of Docker support.
GordonS 7y ago
I actually really like Docker Compose, and especially Swarm. Despite being a relative Docker noob, I've found it to be relatively straightforward to use, and the configuration and secret options are pretty comprehensive. Haven't hit any issues I recall. Would CNAB work with a custom provisioner, or would support for that need to be coded in?

seanhunter 7y ago

No great fan of docker and on a train with a demo laptop and no way to install it to test, but I would expect that > docker-compose --env-file /dev/null will get it to not read a .env file. Will test later/tomorrow.

joshspankit 7y ago
They do mention that being an option in the comments on the bug report, but still sound the alarm as that’s (seemingly) the only option. One-off fixes are possible, but that’s not a viable solution at scale.
otobrglez 7y ago
This doesn't work. At least not with 1.24.1.

hiccuphippo 7y ago

As a workaround you could rename your .env file to something else and mount it as .env in the docker-compose volume options.

Ensorceled 7y ago
How does this help all the existing applications, tools, scripts etc. that use .env?
- otobrglez 7y ago
  If you wanna make it work; this is sadly the only way until Docker Compose is patched. Rename .env to something else and reconfigure autoenv.
  - Ensorceled 7y ago
    Right. Usually workarounds actually, you know, work :-P
- cavemonkey 7y ago
  Those applications running inside the container would see .env as the mounted file but Docker-compose would still see the original .env file in the filesystem.
otobrglez 7y ago
I'm using autoenv - https://github.com/inishchith/autoenv https://github.com/inishchith/autoenv. The workaround that I've found for this problem is to set "AUTOENV_ENV_FILENAME=.env.user" and then inside the project folder, I'm now using ".env.user" instead of just ".env". I'll use this hack until Docker guys fix Docker Compose.
- flukus 7y ago
  Does this (and the one linked to: https://direnv.net/docs/development.html https://direnv.net/docs/development.html) do anything that a simple bash entry can't: function lenv { if [ -f '.env' ]; then source .env fi } function cd { builtin cd $@; lenv; }

rubyn00bie 7y ago

Call me ol' fashioned, but if I'm not administrating hundreds or thousands of micro-services... I just use "Plain Ol' Unix Processes on ZFS," or as I lovingly call it "POUPZ." I can honestly say this exact issue is impossible for anyone who POUPZ. If you haven't tried POUPZ in production, or even POUPZ at home, my recommendation is to give it a try. I think you'll be pretty glad for once that you POUPZ where you eat.

dgoog 7y ago
but what if I'm trying to pretend to be an engineer and I'm really a barista that doesn't know how to linux? I have a macbook and know node.js - doesn't that help?
yjftsjthsd-h 7y ago
Does this address the package-like behavior that is one of docker's big selling points? I guess I could imagine building something out of zfs send+receive, but it feels like you'd need tooling to make it nice, and now you've reinvented the wheel.

lcfcjs2 7y ago

just delete your .env file, that file should never be in a code repo any way. If it is, you're doing it wrong.

gravypod 7y ago

You can run docker-compose.yml in any folder in the tree but it only reads the .env from cwd. Just CD into some place and run docker-compose

ddtaylor 7y ago
Thank you for the clarification.
ohiovr 7y ago
This works. I discovered this by accident trying to figure out why my .env files were not being read. I was in a different dir to the docker-compose.yml file and launched docker-compose with the -f filename option and could not get .env to load.

deleted 7y ago

[deleted]

abhchand 7y ago

For reasons like and to avoid confusion I always name my environment file something specific like `.env.development` or `.env.dockercompose` Any system that reads `.env` files usually allows some way to specify the exact file to be read.

GordonS 7y ago
Huh, I wasn't aware of this! If nothing else, I like the idea of separate files for separate concerns.