10 ms·

Deck-build – A powerful and tiny bash framework to build custom Docker images

61 points by d3ck 8y ago

marmaduke 8y ago
Why not just use buildah?
- warmwaffles 8y ago
  Came here to mention this. I loved rkt for the fact that I didn't need a special syntax for a file. Instead, just a shell script that built the image I needed. Honestly when it comes to deployments, it was much better for us to just get everything into one big layer and move on.
tdurden 8y ago
You can already run whatever commands you want when creating a Docker image, including your own shell scripts. It isn't clear to me why one would want to use this.
- aisofteng 8y ago
  Looks to be the type of thing where people didn’t want to learn a new technology so someone wrote a “framework” to make it “easier” for them.
  - vonseel 8y ago
    Abstractions over abstractions over abstractions. Shaking my damn head. I’ve only been in the industry 7 years or so, but I’ve already begun to prefer sticking with standard operating procedures - like using Dockerfiles for Docker, or using Swift to build iOS apps instead of react-native, because in my experience, if you don’t constantly maintain and stay on top of something that isn’t built with the officially-supported tooling, it’s going to be a lot more trouble when you have to eventually change something. Especially if that change is years down the road and when you finally need to make it, your build system was built on some esoteric unsupported third-party tool that people lost interest in two years ago and stopped developing. Not saying that’s what this is... just a warning. It’s cool to make new stuff and all that, especially when you have a real repetitive problem that isn’t solved elsewhere, but sometimes it seems like people are making problems up or creating solutions to problems that aren’t really problems in the first place. I’m tired of new stuff. I don’t want flashy. I want reliable, stable, usable things with excellent tooling around them. I don’t want something that downloads 300 packages from npm and breaks when I try to update it later and some unknown dependency that wasn’t hard-pinned gets updated in my app. Why things aren’t hard-pinned by default blows my mind. It seems like every time I have to work on a JavaScript application I run into problems like this, which is part of why I hate that juvenile ecosystem. And lack of typing information doesn’t help. At least with typed languages, if something gets upgraded that shouldn’t have been, the IDE or compiler will catch the error usually. And this is all coming from a Python guy (historically). Don’t even get me started on pipenv. /rant
  - d3ck 8y ago
    No, it's not "easier" and it doesn't replace the thinking in images and using images in the final step. The user really need to understand the docker concept. As I wrote in the other answers: It's only (another) approach to avoid complex (RUN ... if ... else ...) Dockerfiles. Hey, this is HN: We share solutions. some you like, some you don't - But the ones you don't like will give you new ideas sometimes :)
- d3ck 8y ago
  You are right, deck-build doesn't have any magic regarding the build process (this is part of the concept). But: 1.) It bundles a lot of useful functions (https://bit.ly/2N9mAEu https://bit.ly/2N9mAEu) in the "kit": Have a look at this gist: https://bit.ly/2Nby0HZ https://bit.ly/2Nby0HZ - Four lines to create a container with two users (foo and root) including their own python packages installed in their home directories (PIP_USER=1). 2.) It's very easy to expand the kit and the plan/artefact concept helps to structure and reuse code (https://bit.ly/2BEIcnH https://bit.ly/2BEIcnH) stored on your local disk or in repositories.
  - tdurden 8y ago
    I think you may have inadvertently proven my point. This kind of abstraction is completely unnecessary. You could have added users and installed requirements in a similarly small amount of Dockerfile commands.
  - dvtrn 8y ago
    Why would I use this tool for 1) instead of Make or just throw CMD useradd -m foo into my Dockerfile?
    - d3ck 8y ago
      Yes, your are both right, of course you can RUN "useradd ...". But we have often made the experience that we need to run more complex processes during docker builds (with if...else conditions etc.) resulting in complex Dockerfile RUNs. Bundling and reusing this code in plans/artefacts is very useful. In the end we use docker-build to create our images, of course it's not intended to be a substitution for the great (docker hub) image concept.
      - dvtrn 8y ago
        But we have often made the experience that we need to run more complex processes during docker builds (with if...else conditions etc.) resulting in complex Dockerfile RUNs That's fair, I've come across those types of scenarios as well, but trying to solve them during docker build seems like you're giving yourself, and inheritors of your docker image a lot of unnecessary extra work. Shouldn't those processes be solved independently of the docker daemon, and the results fed to Docker when it's ready for them?
zallarak 8y ago
I will be sad if I ever need to spend significant amounts of time configuring docker, or any deployment system.
jacques_chester 8y ago
This combines two of my least-favourite things. If anything, Dockerfiles are already too powerful[^]. It's trivial to zoom into an exciting world of asset opacity and rotting bits. Images are very useful, though. Insofar as they are liberated from the sins of Dockerfiles, there's a bright future ahead. I don't see bash as such a liberator. Swapping one mess for a different mess still leaves me in a state of resentful squalor. Disclosure: I work on Cloud Native Buildpacks (https://buildpacks.io https://buildpacks.io), so I have a horse in this race. We're getting near to our first beta release. [^] The original, pre-dang-editing title was something like "Dockerfiles not powerful enough?"
- heavenlyblue 8y ago
  >> If anything, Dockerfiles are already too powerful How? Why doesn't docker allow me to arbitrarily copy files, create any layers and execute shell commands from command-line instead, as opposed to a Dockerfile?
  - dvtrn 8y ago
    Can't you via `docker exec` or do I misunderstand the ask?
    - deathanatos 8y ago
      You can't `docker exec` unless the container is running, so you first need to `docker run` something. (Like an infinite sleep loop, like `bash -c 'while true; sleep 60; done'`.) You can create images via just `docker`, but it's a wee bit hard to do, IMO. IIRC, the docker command that captures the resulting image will by default also change the entrypoint to whatever the `run` was, which isn't helpful.
      - heavenlyblue 8y ago
        >> You can create images via just `docker`, but it's a wee bit hard to do, IMO. Also if you do that, then you can't edit the ENTRYPOINT without adding another layer because docker can't edit the respective json in the .tar file directly. As in - it has the documented functionality to do so in the "POST /commit" API, but that would never work and that parameter will be simply ignored.
tln 8y ago
I took over a Dockerfile that called out to ansible for all the heavy lifting, presumably because that's what the authors knew and liked. Translating to individual RUN lines removed 100s of lines of code, and vastly sped up the build process. In addition the build process should be more familiar to future Dockerfile devs, who can see the entire build in one file and don't need to learn an extra tool. Wrt this project, how well does it use Dockers build cache? And is it really better than idiomatic Dockerfile when you account for future developers?
- d3ck 8y ago
  Build cache is supported (https://bit.ly/2S8Z3oi https://bit.ly/2S8Z3oi), it's the job of the user to structure his code wisely (same as with docker RUN commands). And: No, it's not "better", it's a different approach. Like most/any other community solution it will not replace the standard Dockerfile process :)
  - tln 8y ago
    Thanks for the link
- jzelinskie 8y ago
  FWIW, there was an "official" project for Ansible to generate container images, but it's constantly in limbo. While I do think this is reasonable, I have an intuition that you might be trying to do too much in one image, if you feel like you need Ansible. There are good reasons to sometimes have fat-containers. Do you mind sharing what all you're installing with Ansible?
  - tln 8y ago
    It's a Django backend: the dockerfile uses apt-get and pip. Pretty simple. Using ansible from a Dockerfile in the first place was a code smell, IMO.
kissgyorgy 8y ago
Another terrible something born from not understanding a technology and coming up with something "better". You should learn the thing first! And solve problems after that.
- vonseel 8y ago
  You said it more succinctly than I did! I wish I could cuss these people out, that’s how badly it irritates me.
- deleted 8y ago
  [deleted]
- throwyyyy 8y ago
  What exactly is there to understand about dockerfiles? Everywhere I've worked the dockerfile format has consistently run into issues with versioning and build folder and file juggling. The general attitude in this thread is why HN is a shitshow.
Confiks 8y ago
There are three aspects of Docker that aren't too bad in itself (and have obvious or logical reasons), but taking them together leads to close coupling and code duplication: - There is no composition of images, but only inheritance - Shell scripts can't define any environmental variables that will survive beyond the script itself - Docker doesn't have any way to take or template values from other Dockerfiles That means that you might be able to separate RUN logic out into shell script (like the method deck-build proposes), so that you can compose multiple tools into the same image. Still any ENV variables that your tools might need (if only a simple $PATH) need to be manually copied into the composing Dockerfile and maintained when changes occur.
- d3ck 8y ago
  Thx for your constructive summary. Not sure what you mean with "tools might need" but deck-build only needs some well defined ENVs (https://bit.ly/2TSTImv https://bit.ly/2TSTImv). Regarding shell scripts used in RUN commands: Why not COPY a conf file that will be sourced by every script?
j0057 8y ago
Check out buildah, it's a tool that builds container images using real shell commands, with no big fat Docker daemon necessary. So instead of a COPY command, you can just use cp.
seifertm 8y ago
This reminds me a bit of the container image builder "Kubler" (https://github.com/edannenberg/kubler https://github.com/edannenberg/kubler) which is also configured with Bash scripts. Just creating a cross-reference for the interested here :)
- d3ck 8y ago
  Thank you, I didn't know and will have a look :)
peterwwillis 8y ago
If you find yourself using complicated custom methods with Dockerfiles, you should probably be building Linux packages, publishing to a local repo, and installing them from the Dockerfile. You gain the idempotent, immutable, versioned, system integrated, dependency-mapped, cryptographically verifiable, remotely distributed, cacheable benefits, and you don't have to adopt any new software or systems. Your organization can follow any workflow/lifecycle it wants to build and publish the packages. Once they are published, developers can just install and use them with no learning curve whatsoever.
- makapuf 8y ago
  And to create those packages, I've found fpm[1] a joy to use. [1] https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm
- d3ck 8y ago
  Got your point, and this is a nice solution. But another (not uncommon approach) is to understand the docker container itself as the application.

61 points by d3ck 8y ago

marmaduke 8y ago

Why not just use buildah?

warmwaffles 8y ago
Came here to mention this. I loved rkt for the fact that I didn't need a special syntax for a file. Instead, just a shell script that built the image I needed. Honestly when it comes to deployments, it was much better for us to just get everything into one big layer and move on.

tdurden 8y ago

You can already run whatever commands you want when creating a Docker image, including your own shell scripts. It isn't clear to me why one would want to use this.

aisofteng 8y ago
Looks to be the type of thing where people didn’t want to learn a new technology so someone wrote a “framework” to make it “easier” for them.
- vonseel 8y ago
  Abstractions over abstractions over abstractions. Shaking my damn head. I’ve only been in the industry 7 years or so, but I’ve already begun to prefer sticking with standard operating procedures - like using Dockerfiles for Docker, or using Swift to build iOS apps instead of react-native, because in my experience, if you don’t constantly maintain and stay on top of something that isn’t built with the officially-supported tooling, it’s going to be a lot more trouble when you have to eventually change something. Especially if that change is years down the road and when you finally need to make it, your build system was built on some esoteric unsupported third-party tool that people lost interest in two years ago and stopped developing. Not saying that’s what this is... just a warning. It’s cool to make new stuff and all that, especially when you have a real repetitive problem that isn’t solved elsewhere, but sometimes it seems like people are making problems up or creating solutions to problems that aren’t really problems in the first place. I’m tired of new stuff. I don’t want flashy. I want reliable, stable, usable things with excellent tooling around them. I don’t want something that downloads 300 packages from npm and breaks when I try to update it later and some unknown dependency that wasn’t hard-pinned gets updated in my app. Why things aren’t hard-pinned by default blows my mind. It seems like every time I have to work on a JavaScript application I run into problems like this, which is part of why I hate that juvenile ecosystem. And lack of typing information doesn’t help. At least with typed languages, if something gets upgraded that shouldn’t have been, the IDE or compiler will catch the error usually. And this is all coming from a Python guy (historically). Don’t even get me started on pipenv. /rant
- d3ck 8y ago
  No, it's not "easier" and it doesn't replace the thinking in images and using images in the final step. The user really need to understand the docker concept. As I wrote in the other answers: It's only (another) approach to avoid complex (RUN ... if ... else ...) Dockerfiles. Hey, this is HN: We share solutions. some you like, some you don't - But the ones you don't like will give you new ideas sometimes :)
d3ck 8y ago
You are right, deck-build doesn't have any magic regarding the build process (this is part of the concept). But: 1.) It bundles a lot of useful functions (https://bit.ly/2N9mAEu https://bit.ly/2N9mAEu) in the "kit": Have a look at this gist: https://bit.ly/2Nby0HZ https://bit.ly/2Nby0HZ - Four lines to create a container with two users (foo and root) including their own python packages installed in their home directories (PIP_USER=1). 2.) It's very easy to expand the kit and the plan/artefact concept helps to structure and reuse code (https://bit.ly/2BEIcnH https://bit.ly/2BEIcnH) stored on your local disk or in repositories.
- tdurden 8y ago
  I think you may have inadvertently proven my point. This kind of abstraction is completely unnecessary. You could have added users and installed requirements in a similarly small amount of Dockerfile commands.
- dvtrn 8y ago
  Why would I use this tool for 1) instead of Make or just throw CMD useradd -m foo into my Dockerfile?
  - d3ck 8y ago
    Yes, your are both right, of course you can RUN "useradd ...". But we have often made the experience that we need to run more complex processes during docker builds (with if...else conditions etc.) resulting in complex Dockerfile RUNs. Bundling and reusing this code in plans/artefacts is very useful. In the end we use docker-build to create our images, of course it's not intended to be a substitution for the great (docker hub) image concept.
    - dvtrn 8y ago
      But we have often made the experience that we need to run more complex processes during docker builds (with if...else conditions etc.) resulting in complex Dockerfile RUNs That's fair, I've come across those types of scenarios as well, but trying to solve them during docker build seems like you're giving yourself, and inheritors of your docker image a lot of unnecessary extra work. Shouldn't those processes be solved independently of the docker daemon, and the results fed to Docker when it's ready for them?

zallarak 8y ago

I will be sad if I ever need to spend significant amounts of time configuring docker, or any deployment system.

jacques_chester 8y ago

This combines two of my least-favourite things. If anything, Dockerfiles are already too powerful[^]. It's trivial to zoom into an exciting world of asset opacity and rotting bits. Images are very useful, though. Insofar as they are liberated from the sins of Dockerfiles, there's a bright future ahead. I don't see bash as such a liberator. Swapping one mess for a different mess still leaves me in a state of resentful squalor. Disclosure: I work on Cloud Native Buildpacks (https://buildpacks.io https://buildpacks.io), so I have a horse in this race. We're getting near to our first beta release. [^] The original, pre-dang-editing title was something like "Dockerfiles not powerful enough?"

heavenlyblue 8y ago
>> If anything, Dockerfiles are already too powerful How? Why doesn't docker allow me to arbitrarily copy files, create any layers and execute shell commands from command-line instead, as opposed to a Dockerfile?
- dvtrn 8y ago
  Can't you via `docker exec` or do I misunderstand the ask?
  - deathanatos 8y ago
    You can't `docker exec` unless the container is running, so you first need to `docker run` something. (Like an infinite sleep loop, like `bash -c 'while true; sleep 60; done'`.) You can create images via just `docker`, but it's a wee bit hard to do, IMO. IIRC, the docker command that captures the resulting image will by default also change the entrypoint to whatever the `run` was, which isn't helpful.
    - heavenlyblue 8y ago
      >> You can create images via just `docker`, but it's a wee bit hard to do, IMO. Also if you do that, then you can't edit the ENTRYPOINT without adding another layer because docker can't edit the respective json in the .tar file directly. As in - it has the documented functionality to do so in the "POST /commit" API, but that would never work and that parameter will be simply ignored.

tln 8y ago

I took over a Dockerfile that called out to ansible for all the heavy lifting, presumably because that's what the authors knew and liked. Translating to individual RUN lines removed 100s of lines of code, and vastly sped up the build process. In addition the build process should be more familiar to future Dockerfile devs, who can see the entire build in one file and don't need to learn an extra tool. Wrt this project, how well does it use Dockers build cache? And is it really better than idiomatic Dockerfile when you account for future developers?

d3ck 8y ago
Build cache is supported (https://bit.ly/2S8Z3oi https://bit.ly/2S8Z3oi), it's the job of the user to structure his code wisely (same as with docker RUN commands). And: No, it's not "better", it's a different approach. Like most/any other community solution it will not replace the standard Dockerfile process :)
- tln 8y ago
  Thanks for the link
jzelinskie 8y ago
FWIW, there was an "official" project for Ansible to generate container images, but it's constantly in limbo. While I do think this is reasonable, I have an intuition that you might be trying to do too much in one image, if you feel like you need Ansible. There are good reasons to sometimes have fat-containers. Do you mind sharing what all you're installing with Ansible?
- tln 8y ago
  It's a Django backend: the dockerfile uses apt-get and pip. Pretty simple. Using ansible from a Dockerfile in the first place was a code smell, IMO.

kissgyorgy 8y ago

Another terrible something born from not understanding a technology and coming up with something "better". You should learn the thing first! And solve problems after that.

vonseel 8y ago
You said it more succinctly than I did! I wish I could cuss these people out, that’s how badly it irritates me.
deleted 8y ago
[deleted]
throwyyyy 8y ago
What exactly is there to understand about dockerfiles? Everywhere I've worked the dockerfile format has consistently run into issues with versioning and build folder and file juggling. The general attitude in this thread is why HN is a shitshow.

Confiks 8y ago

There are three aspects of Docker that aren't too bad in itself (and have obvious or logical reasons), but taking them together leads to close coupling and code duplication: - There is no composition of images, but only inheritance - Shell scripts can't define any environmental variables that will survive beyond the script itself - Docker doesn't have any way to take or template values from other Dockerfiles That means that you might be able to separate RUN logic out into shell script (like the method deck-build proposes), so that you can compose multiple tools into the same image. Still any ENV variables that your tools might need (if only a simple $PATH) need to be manually copied into the composing Dockerfile and maintained when changes occur.

d3ck 8y ago
Thx for your constructive summary. Not sure what you mean with "tools might need" but deck-build only needs some well defined ENVs (https://bit.ly/2TSTImv https://bit.ly/2TSTImv). Regarding shell scripts used in RUN commands: Why not COPY a conf file that will be sourced by every script?

j0057 8y ago

Check out buildah, it's a tool that builds container images using real shell commands, with no big fat Docker daemon necessary. So instead of a COPY command, you can just use cp.

seifertm 8y ago

This reminds me a bit of the container image builder "Kubler" (https://github.com/edannenberg/kubler https://github.com/edannenberg/kubler) which is also configured with Bash scripts. Just creating a cross-reference for the interested here :)

d3ck 8y ago
Thank you, I didn't know and will have a look :)

peterwwillis 8y ago

If you find yourself using complicated custom methods with Dockerfiles, you should probably be building Linux packages, publishing to a local repo, and installing them from the Dockerfile. You gain the idempotent, immutable, versioned, system integrated, dependency-mapped, cryptographically verifiable, remotely distributed, cacheable benefits, and you don't have to adopt any new software or systems. Your organization can follow any workflow/lifecycle it wants to build and publish the packages. Once they are published, developers can just install and use them with no learning curve whatsoever.

makapuf 8y ago
And to create those packages, I've found fpm[1] a joy to use. [1] https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm
d3ck 8y ago
Got your point, and this is a nice solution. But another (not uncommon approach) is to understand the docker container itself as the application.