5 ms·
Using ARG in a Dockerfile – beware the gotcha
- imp0cat 3y agotldr; ARG values are only valid in the build image, not anywhere else. In a multistage build, they are only valid in the image that declared them (ie. they do not magically pass over to any other image).
- a_t48 3y agoIt's kind of funky when you use them in the names of stages, too. IIRC you have to declare them right above.
- MereInterest 3y agoI take issue with your use of “magically”. Multistage builds look like a inner scope for each stage, and an outer scope that contains them. In most programming languages, variables in an outer scope are accessible within an inner scope. I can see an argument for variables declared within one stage being inaccessible in another stage, as the FROM keyword starts a new sibling stage. However, they are still children of the outer scope, should have access to variables in the outer scope. That they do not is rather unexpected.
- imp0cat 3y agoUnderstood, sorry about that. But that's how it feels to me. It's mostly because my mental image of multistage builds is different, I see them as a way to make the final image smaller by throwing away everything from the previous image. So it does make sense to me that ARG values are thrown away after a new FROM instruction and if you need them, you need to declare them again. These two guides have led me to this view, I blame the author! ;) They're excellent BTW, I've revisited them a lot. https://vsupalov.com/docker-arg-env-variable-guide/#arg-and-env https://vsupalov.com/docker-arg-env-variable-guide/#arg-and-... https://vsupalov.com/build-docker-image-clone-private-repo-ssh-key/ https://vsupalov.com/build-docker-image-clone-private-repo-s...
- MereInterest 3y ago> Understood, sorry about that. And apologies on my side, as I worded it a bit strongly. I agree with pretty much everything in your post, except for the implication that it would be magic. To me, the expected behavior is to follow lexical scoping, just like the vast majority of languages from the past several decades. > So it does make sense to me that ARG values are thrown away after a new FROM instruction and if you need them, you need to declare them again. True, and I'd expect an ARG from a previous stage to be undefined when in the next stage. Each ARG belongs to the scope in which it is declared, and a new scope is entered with each FROM statement. However, ARG statements prior to the first FROM statement don't fit into that model. They are before any FROM statement, so they can't belong to any FROM statement. There's a design decision on whether statements before the first FROM statement are a parent scope, or whether they are unrelated scopes, to the scope used for each stage. They feel like they should be a parent scope, because these ARG definitions can be used in a FROM statement, and so it keeps surprising me when they are instead treated as unrelated scopes.
- miguelaeh 3y agoI don't see the gotcha, that's how it is supposed to work. It's just their purpose
- BonusPlay 3y agoThe gotcha is that "global" args don't propagate automatically to all stages (thin includes 1 stage builds). I want this one arg in multiple stages, so I'll declare it above everything is the chain of thought.
- johanbcn 3y agoIt's explained on the official documentation: https://docs.docker.com/reference/dockerfile/#scope https://docs.docker.com/reference/dockerfile/#scope
- hombre_fatal 3y agoTrue, though gotchas exist when user intuition doesn't match with actual behavior regardless of whether they are mentioned in docs.
- chii 3y agoPrinciple of least surprise. If you need to write in the docs about a surprise that a user otherwise wouldn't have expected, may be it's a sign that the surprise should be fixed up such that it's not surprising behaviour.
- deleted 3y ago[deleted]
- koito17 3y agoI interpret each directive in a Dockerfile as creating a new layer of an image. So this ARG-before-FROM gotcha doesn't feel like a gotcha to me, but rather, the consequence of literally interpreting "ARG" and not knowing the side-effects of a directive in a Dockerfile. (Yes, even WORKDIR, ENTRYPOINT, and related instructions create a layer, albeit a 0-byte one)
- Joker_vD 3y ago> Perhaps I should have read the entire reference document for all the Dockerfile instructions first. I really, really hate to be "that one guy" (c) Rachel Kroll, but... if you're about to use a large, old, complicated piece of software, the chances are very good that its developers have mental models about what is good software, and what is good documentation, and what is self-evident and what is not, that are very, very different from yours, especially if you're a programmer and the piece of software is aimed at the sysadmins/ops/devops — so yes, read the whole docs. The things are not what you would hope they are.
- ponector 3y agoAnd then go and read all issues logged to see how your mental model is different. Recent example I hit was a Gitlab pipeline. "When: always" setting for after_script does not mean the command will be executed always. Only if main section finished with either success or failure. If it is terminated by timeout or manually - no, script is not executed.
- anal_reactor 3y ago>so yes, read the whole docs Thanks to reading the whole docs an hour a day I avoided a bug that happens once a year and takes whole 30 minutes of debugging. "Correct practices" in a nutshell.
- MereInterest 3y agoExactly. Documentation is useful when a user knows that there are multiple reasonable options for an implementation to take. "Which byte order is used for this protocol?" is a question that can occur to a user and can be looked up in the documentation. Questions like "Does this language require explicit imports to access variables in the parent scope?" are not questions that occur to a user, because any violation of that would be so unexpected as to be a bug. Docker documents its bugs, calls them the intended behavior, and then blames users for not reading about the bug in the documentation.
- hooby 3y agoSo, each stage/each FROM has it's own scope basically. If you define an ARG in the global scope, you have to import it to the local scope before using it. Not terribly unusual, but I wasn't aware of it - so yeah, wouldn't hurt if the docs for ARG mentioned it. Maybe submitting a PR for an addition to the docs would help more people than writing a blog post about it?
- gorgoiler 3y agoPOSIX sh has “set -u” to handle this category of errors — scripts that attempt to expand an unset parameter will exit with an error message. Perl has “use warnings ‘uninitialized’” for the same reason. It sounds like this feature would be worth considering for the Dockerfile spec.
- diarrhea 3y agoEmpty/unset environment variables not, by default, being cause for immediate error and abort is the billion dollar mistake of the DevOps/infra world.
- gorgoiler 3y agoI agree but let’s hear the counter arguments so as to not sound like we aren’t listening. Environment variables can be useful. The first version of some code can be oblivious to environment variables and the second version can use them for feature flags. Because it is entirely optional to provide these, the second version of the code can be deployed to the first version’s infrastructure, and the first version will happily ignore environment variables set in the second version’s infrastructure. As a tool for incrementally moving heterogeneous parts of a system forward without needing atomic leaps, the use of environment variables as optional flags is very useful. I just wouldn’t even start from there. Instead I configure everything in code, in one repo, with all new deployments appearing as new clusters where the only thing they have in common is that they take production traffic. This kind of production infra is very useful but there are downsides for sure. It makes database schema changes, for example, something between very hard and anathema to execute.
- vbezhenar 3y agoFor this use-case you don't need anything in Dockerfile spec. ARG is not magically expanded inside RUN blocks, it's just an env available during build stage. In other words you can use this code to catch this error: ARG DEBVER="10" ARG CAPVER="7.8" FROM debian:${DEBVER} RUN <<EOF set -eu printf "DEB=${DEBVER}\nCAP=${CAPVER}\n" EOF This code fails with expected error.
- kaliszad 3y agoThat is what you get when people reinvent the wheel and lifetimes/ scopes are implicit. Docker could've used something like JSON5 [0] for their configuration format to make the lifetimes explicit. Another time when easy won over simple. [1] [0] https://json5.org/ https://json5.org/ [1] https://www.youtube.com/watch?v=SxdOUGdseq4 https://www.youtube.com/watch?v=SxdOUGdseq4
- ascar 3y agoJSON for something that essentially mirrors a shell installation process? Feels like you are trying to reinvent things with a golden hammer, not like actually making it easier.
- brainzap 3y agoARG not being availabe in ENTRYPOINT or CMD is so annoying for usability.
- bheadmaster 3y agoYou can use ENV to convert ARG into an environment variable which can be used in CMD through shell environment variable evaluation: FROM ubuntu:20.04 ARG foo ENV foo $foo CMD ["/bin/sh", "-c", "echo $foo"]
- dcormier 3y agoI remember tripping over this. It was frustrating.
- nrvn 3y agoA worse idea would be to run bash scripts adhoc without at least `set -u`. There is an unofficial bash “strict mode”, overlooked and undervalued: set -euo pipefail Just use it everywhere and use shellcheck for scripts.
- evgpbfhnr 3y ago`set -e` is evil: https://web.archive.org/web/20240511180116/https://mywiki.wooledge.org/BashFAQ/105 https://web.archive.org/web/20240511180116/https://mywiki.wo... (wooledge wiki doesn't seem to load for me for some reason so archive.org it is)
- hyperhopper 3y agoThis perfectly illustrates the problem with bash. Somebody did something the straightforward way which worked. A solution is given on a better way, but that also has a problem that somebody else didn't know about. Bash let's you do anything and have it kind of work but you never know how many footguns you're setting up for yourself.
- anal_reactor 3y agoI hate bash
- hannibalhorn 3y agoThat's an interesting link and an argument I hadn't seen before. In my "real world" usage, I've found "bash strict mode" incredibly useful, especially when scripting things that others will be modifying. It just avoids all of the "somebody forgot an error check", "there's a typo in a variable name", etc., type errors. And pouring through logs to figure out what actually went wrong in CI/CD is brutal. Looking at the examples: - I don't generally do arithmetic in bash, that's a sign of "time to change language" - Also generally use `if ...; then ...; fi` instead of `test ... && ...` If you do the advanced type of scripting that somebody who writes a Bash FAQ does, I'm sure `set -e` can be annoying. But once you learn a couple simple idioms y(covered on the original "bash strict mode" page) you don't have to know much more to have fairly robust scripts. And future maintainers will thank you for it!
- btreecat 3y agoThe CSS is whack on Firefox Mobile
- orochimaaru 3y agoI think the author is expecting ARG during runtime but it is supposed to be a compile time command. The bigger gotcha I have seen is using ENV instead of ARG. Especially for proxies. You may be using a proxy during compile time but don’t need (or it’s not configured to your network) at runtime.
- noelwelsh 3y agoThis is a perfect example of the problem in the devops world. People want to do something that has slight variations from time to time. There is a well established concept for this, called a function. Programmers use functions every day and understand their semantics. Devops tool developers do not implement functions but some other abstraction with different semantics. Programmers use this abstraction and confusion ensures. Why the people who implement these tools, who presumably are very accomplished programmers, cannot 1) see they are implementing something that is already well known and/or 2) implement it with standard semantics, is baffling to me. Kids, just say NORWAY to devops.
- sherburt3 3y agoWhat does NORWAY mean?
- steve_rambo 3y agohttps://www.bram.us/2022/01/11/yaml-the-norway-problem/ https://www.bram.us/2022/01/11/yaml-the-norway-problem/
- pdimitar 3y agoLikely a joke with the YAML parsers that parse `NO` as the boolean value `false` when the writer of the YAML file intended to put Norway's 2-character code as a key in a map.
- svieira 3y agoNorway's 2 character country code abbreviation is "NO". Which is one of the many ways to spell the boolean `false` in YAML 1.0. Which is yet-another-way-to-shoot-yourself-in-the-foot with DSLs written in YAML.
- deleted 3y ago[deleted]
- pdimitar 3y agoThat's why I plan on migrating all my shell scripts to Golang programs f.ex. using https://github.com/bitfield/script https://github.com/bitfield/script -- it already has a number of simulations of shell commands and I'd contribute others if I had the time. sh / bash / zsh scripts are just fragile and that's the inconvenient truth. People who devised the shell interpreters had good intentions but ultimately their creations grew to a scope 1000x bigger than they intended, hence all the "do X but if Y flag is set then do Z... unless flag A is also set in which case do Y and part of Z". It's horrendous and I seriously don't get what's so difficult in just coding these scripts in a programming language that provides single statically linked binaries (like Golang) and just distribute it with your images -- or run them in CI/CD and have init containers and never include them in the images in the first place. Inertia, of course. But I'll be actively working against it until I retire.
- teekert 3y agoSo many gotchas in today's world. Recently it turned out that docker-compose.yaml was helpfully exposing our DB to our local LAN (which is an entire company that has nothing to do with our dev work), despite restricting all in and out traffic to our tailnet with UFW. I know it's a known issue, it comes up here every now and then, we kept it in mind and even talked about it. But as soon as someone uses an "expose" in a docker-compose.yaml, you're f-ed. Sure, you should keep the containers talking within the docker network, but we are kind of distributed in our org (across a series of NUCs). I find it annoying that is has come to this. Now considering Podman, or VMs (to use a host firewall around the virtual stuff).
- jhardy54 3y agoAre you binding to 127.0.0.1 or 0.0.0.0? > Sure, you should keep the containers talking within the docker network, but we are kind of distributed in our org (across a series of NUCs). I don’t think I understand this, are you saying that you want the ports exposed to some devices on your network but not others?
- teekert 3y agoIn docker-compose it's just "- ports", I guess it's always 0.0.0.0, unless bound to something else (like 127.0.0.1). You can restrict it to the Tailnet IP, but then you have to hardcode the Tailscale 100.x.x.x IP address (not flexible).
- nonameiguess 3y agoThe real issue here is Linux overloading netfilter to implement both overlay networking so you can address virtual interfaces in network namespaces, and traffic blocking rules, so host-based firewalls like ufw and containerized networking implementations like Docker and Kubernetes more or less have to assume they have full control of your filter chains and it's up to you as a server admin to deconflict. The best answer is usually not to use any kind of host-based firewall if you're doing container networking and use a network firewall instead.
- mass_and_energy 3y agoI'm seeing a lot of people complaining about Dockerfile being too complicated, but if you can wrap your head around local scopes and Docker's layering it's really not rocket science. Is Dockerfile that bad or is it an overall lack of understanding? I've used Dockerfile for years and have found it to be intuitive and powerful.
- pdimitar 3y agoNot sure why you are coming here to rant about "a lot of people complaining about Dockerfile being too complicated" when the topic here is a gotcha that's not at all obvious and Docker working in a way that is not intuitive. If I specify ARG I obviously expect it to be set everywhere in the Dockerfile. Your rant is out of place because it generalizes and attempts to shift topics.
- crabbone 3y agoJavaScript is also not complicated. It's just poorly designed when it comes to matching programmer's intuition. With this out of the way: Dockerfile as a concept, i.e. a minimalist (in simple cases) way to declaratively configure a container is great. The execution of this concept is, on the other hand, awful. The behavior of gotchas like the one OP described isn't the reason for the awfulness, rather, it comes from the inconsistent-by-default builds. That is, Dockerfile is not a full description of the contents of the built image. To know what will go into the image, you also need to consider the state of the entire world (because you can download things from the Internet during build, execute code with unpredictable side effects etc.)
- deleted 3y ago[deleted]
- noop_joe 3y agoTo me this is just one of many infrastructureisms. Tools, platforms, etc are all leveraging metaphors that don't map well to the mental model of _application development_.