8 ms·

You Don't Need to Rebuild Your Development Docker Image on Every Code Change

119 points by one2three4 5y ago

iofiiiiiiiii 5y ago
A little known feature is that the Docker integration in Visual Studio gives you this automatically.
- robflynn 5y ago
  We use this where I work and love it. It has the added benefit of letting you package preferred vscode extensions with the devcontainer as well.
- Noumenon72 5y ago
  I think PyCharm Professional's docker-compose remote interpreter[0] does too. It was a lot of setup but it's the only way I know to have your code run inside containers and actually be able to stop at breakpoints inside them. (I don't know much about Docker.) 0: https://www.jetbrains.com/help/pycharm/using-docker-compose-as-a-remote-interpreter.html#summary https://www.jetbrains.com/help/pycharm/using-docker-compose-...
- metaltyphoon 5y ago
  Container fast run or something like that right ? If you observe the commands it does, it simply mounts everything it needs on the container and uses the first stage of the docker file (which is usually a base image) to run your stuff.
  - Vendan 5y ago
    Pretty sure they are talking about https://code.visualstudio.com/docs/remote/containers https://code.visualstudio.com/docs/remote/containers
gravypod 5y ago
While this is great for people with a fundamental understanding of containers and your prod environment this will usually lead to some issues with developers that don't need to, or want to, have context in these areas. In the past, to make a very similar workflow possible, I've built tools that automatically watch your source files and rebuild & restart only what is needed [0]. This was built for bazel + docker-compose but there isn't a reason one couldn't watch the "build:" contexts for what files are important. At a previous company one of our engineers was a huge fan of this volume mount approach and every single time something broke (which was very frequent due to some prod/dev env magic we had) I had to assist quite a few more junior devs figure out what was wrong with their machine. For those with scripting languages, was it their system's newline endings? For compiled languages, was their system SDK different then what was in the container? For prod bugs, did they forget to rebuild & test the container before opening their PR (we had no automated integration testing)? In my opinion, if you can make your build system in charge of building/packaging things you'll have a much happier time. [0] - https://github.com/CaperAi/bazel_compose https://github.com/CaperAi/bazel_compose
- Noumenon72 5y ago
  What does it mean to "make your build system in charge of building/packaging things"? Like push your code out to Jenkins and have it build it? Or have Gradle make something that runs for you?
vsupalov 5y ago
Hi HN! First time I see one of my articles on here. What a nice surprise. If the above link caught your attention, you might also enjoy the following ones: * For the quickest ROI: https://vsupalov.com/improve-your-docker-images/ https://vsupalov.com/improve-your-docker-images/ * Stuff I WISH I knew: https://vsupalov.com/12-docker-facts/ https://vsupalov.com/12-docker-facts/ * If your image builds are slow: https://vsupalov.com/5-tips-to-speed-up-docker-build/ https://vsupalov.com/5-tips-to-speed-up-docker-build/ Looking forward to join the discussion later!
SatvikBeri 5y ago
I've been doing this for about 6 months now, it saves a lot of time. Especially if you need to upload changed docker images to AWS.
gmaster1440 5y ago
https://code.visualstudio.com/docs/remote/containers https://code.visualstudio.com/docs/remote/containers
pastage 5y ago
Works remotely on kubernetes, for us the game changer was running inotify_wait on cygwin for our windows users, changes are up in about 600ms. 1. Configure a service 2. Use start command sleep infinity 3. Install inotify_wait on windows 4. Do a loop like this Look for changes, Rsync changes to pod, Pkill java, Run start.sh We do a grep on changed files and only kill java if jar/classes and other deps has changed, that makes it possible to edit html in pod and get fast updates.
- orlovs 5y ago
  I have never understood kubernetes development in local development workflow unless kubernetes functionality being developed or some yaml magic. It’s super overhead IMHO. If needed to test api’s as unit test you can always port-forward traffic of desired enpoint or use something like telepresence. Less kubernetes is always better
  - erulabs 5y ago
    It’s cpu/memory overhead yes, but between matching prod and dev, extremely simple config (redis is always located at “redis:6379”), zero setup instructions (skaffold dev), remote/local hybrid development, and fostering prod-ready skills for developers (still working on that devops koolaid), I find it -extremely- worthwhile. I’ve yet to meet someone who doesn’t hate it to start (because kubernetes) and winds up loving it and swearing by it. It’s essentially a vastly superior “vagrant up” on anabolic steroids
    - orlovs 5y ago
      I agree, for e2e/integration testing, excellent
- mehphp 5y ago
  Have you seen https://skaffold.dev/ https://skaffold.dev/ ?
  - emj 5y ago
wiredfool 5y ago
This is great, until you need to watch a tree of files in macOS. Then docker takes a core to do the fsnotify/watch. There are caching settings to make it better, but there are race conditions between the watch and when file contents change, so sometimes the JS stack will compile an inconsistent file. OTOH, on linux, it's grand.
- bdcravens 5y ago
  docker-sync fixes this by using an intermediate container that handles sync (I believe there are other options which work essentially the same way). It supports ignoring files you don't need on your main machine, like tmp folders and the like, which can improve performance further. http://docker-sync.io/ http://docker-sync.io/
snapetom 5y ago
I have to scratch my head every time I see a blog post about creating a dev environment that doesn't use volumes like this. IMO, this is the way to go and what Docker was meant to be as a dev environment. Maybe it's because I'm never confident in the changes I make, but my style is to write a few lines at a time, save, re-run or refresh. I can't imagine having to do a docker build every code change. If you do the later, it's just nominally better than developing on a remote server, which brings its own challenges.
- roofwellhams 5y ago
  Did you try typescript? Now I can write for hours without trying out the app. And when I try... It just works
  - redleather 5y ago
    This is such a weird comment.. nothing about the OC suggests they were talking about front end dev; not to mention that writing for hours before actually running it is a pretty terrible way to code. Typescript is also not a workaround for volume binding, which can accommodate any stack.
    - dehrmann 5y ago
      > nothing about the OC suggests they were talking about front end dev >> I'm never confident in the changes I make, but my style is to write a few lines at a time, save, re-run or refresh. This is a pretty typical frontend development pattern, especially seeing "refresh."
      - dividedbyzero 5y ago
        I read that as "re-run or refresh, as applicable", i.e. re-run a Go app or refresh a Node.js webapp. Besides, it absolutely can be a big issue with non-frontend development, so the point still stands.
        5y ago
iooi 5y ago
For devs on macOS, keep in mind that your filesystem is case insensitive! If you're using a linux image that expects a directory like `UpperCase` and you named it `uppercase` locally it would work.. until you bake the source for your production release and you get errors around that directory not existing. "But it works locally!"
- flas9sd 5y ago
  there can also be subtle issues with git, I recommend then to convert HFS / APFS to case-sensitive. I understand why normal people do not care, but Dev types in unix alike environments do not have a choice with decades of case-sensitive filesystems before them
tlarkworthy 5y ago
I have done this but if possible I just run my code locally and reserve docker for infra (e.g. DB). Java, nodejs, go are all very easily run locally. Maintaining a complex Dockerfile is just more crap that makes programming tedious.
- Fredej 5y ago
  Honestly my experience is basically the opposite: Maintaining a coherent development environment is what makes programming tedious. I'd much prefer to just specify in a dockerfile that I need this compiler, these tools and these env-variables, and then I never have to worry about that stuff again.
  - tlarkworthy 5y ago
    It's hotcode reload and debugging where I find docker starts to annoy me. (Also resource consumption and performance to an extent)
  - deleted 5y ago
    [deleted]
- nucleardog 5y ago
  Sure, nodejs is easy to run locally. nodejs+npm of the exact version that a specific project needs, which is different and incompatible with the version some other project needs starts to get more annoying. Basically all of our projects have some level of external dependencies in the form of libraries or binary tools. Some projects overlap dependencies, but on incompatible versions. This gets really hairy. Trying to do this without containers (from experience, this is what the devs were doing before) turns into a massive documentation project and a nightmare of trying to get all these incompatible things to exist on a single system where we don't even have a consistent target because everyone's local dev environment is different enough to make it a nightmare. And then continually hounding the devs to keep the documentation up-to-date. But even that doesn't solve the problem because "apt-get install nodejs" today is not necessarily the same as "apt-get install nodejs" in six months or a year and nobody's thinking about this so it's just "worked for me last time!". Onboarding people at one point was a _days_ long process and a group effort. Instead we just re-use the same 30-odd line Dockerfile we use for deploying out to the infra for local dev. Instead of pages upon pages of out-of-date documentation. a bunch of tribal knowledge, and continually running into new and interesting bugs, we now have a short and simple source of truth which _has_ to be up-to-date because otherwise the environment the project is deployed to is broken. Onboarding people takes 5 minutes. If you're a single person working on a single project, yeah, containerizing stuff is maybe some annoying overhead. Even then I still use it because once you've gotten past the learning curve, it's actually a really effective way to document and define your project's environment and ensure it stays consistent.
richardwhiuk 5y ago
We wrote a tool to make using development Docker images easy - https://metaswitch.github.io/floki/ https://metaswitch.github.io/floki/
- robsalasco 5y ago
  Looks awesome!
sebyx07 5y ago
For any rails dev out these volumes: - .build/.bundle-cache-dir/app:/usr/local/bundle/
gprasanth 5y ago
Another thing I've recently found helpful was docker layer caching inside github actions. Pretty easy to integrate, saves a lot of build minutes.
deleted 5y ago
[deleted]
alpb 5y ago
Skaffold is a great tool for this and employs hot reloading techniques without having to rebuild the image. https://skaffold.dev/ https://skaffold.dev/
- spiddy 5y ago
  I find it very interesting the idea of separating development image from production one and skaffold promotes it in a way ("skaffold dev" vs "skaffold run") Same as in frontend for example you don't "npm build" your way through development, instead you want hot-reload and other similar features.
aszen 5y ago
Docker truthfully told is unusable for Dev envs that need to change constantly. It was never built to handle this use case. I haven't seen a good docker compose file that can be reliably used as a development environment in multiple oses with good performance. There's all sorts of edge cases where volumes don't work. In my opinion docker is very useful for thing like dbs, queues and other processes where the underlying code doesn't change. But for everyday frontends and backends it's not worth it. Nowadays I write a shell.nix file which contains all the dependencies the project needs, it works but is definitely not as easy to learn as a docker file
- okamiueru 5y ago
  Out of curiosity, what OS are you on?
  - aszen 5y ago
    I'm on Linux, but still things aren't all rosy.
- mike_d 5y ago
  Dev should look just like production if you are doing it right, and you are correct dev was basically an afterthought in Docker. If you run this in a screen session it will auto rebuild as you make changes, which is the closest I've been able to come to containerizing my development style: https://gist.github.com/mikedamm/53dc6a78b976eeac88893427424c2527 https://gist.github.com/mikedamm/53dc6a78b976eeac88893427424...
  - aszen 5y ago
    When people say docker helps in making dev look like prod, I assume they are running everything on docker in production, but that is very rarely the case with dbs, queues and cloud specific services. Interesting I'll check this script, but I have doubts doing a rebuild is going to be quick enough even with good cache layering in the Dockerfile. Use case is that I have frontend and backend code in one repo, and make changes across them that need to be reflected in the ui.
deleted 5y ago
[deleted]
efrecon 5y ago
In the past few weeks, I have spent some time and released dew [0]. It helps encapsulating this kind of setups in configuration and minimising typing. dew is still evolving, but it has served me well. [0]: https://github.com/efrecon/dew https://github.com/efrecon/dew
wdb 5y ago
I have meant to try if VMWare's vctl is any better regarding detecting file changes
bfrog 5y ago
Nix has saner repeatable dev envs, and makes updating or adding deps a snap. Can be used to build scratch containers. Can be used to cross compile. The only major downside is it requires learning and understanding nix which I get is a hurdle, but one that's well worth it. Docker is only one container tool of many now, and it's worth exploring what else is out there.

119 points by one2three4 5y ago

iofiiiiiiiii 5y ago

A little known feature is that the Docker integration in Visual Studio gives you this automatically.

robflynn 5y ago
We use this where I work and love it. It has the added benefit of letting you package preferred vscode extensions with the devcontainer as well.
Noumenon72 5y ago
I think PyCharm Professional's docker-compose remote interpreter[0] does too. It was a lot of setup but it's the only way I know to have your code run inside containers and actually be able to stop at breakpoints inside them. (I don't know much about Docker.) 0: https://www.jetbrains.com/help/pycharm/using-docker-compose-as-a-remote-interpreter.html#summary https://www.jetbrains.com/help/pycharm/using-docker-compose-...
metaltyphoon 5y ago
Container fast run or something like that right ? If you observe the commands it does, it simply mounts everything it needs on the container and uses the first stage of the docker file (which is usually a base image) to run your stuff.
- Vendan 5y ago
  Pretty sure they are talking about https://code.visualstudio.com/docs/remote/containers https://code.visualstudio.com/docs/remote/containers

gravypod 5y ago

While this is great for people with a fundamental understanding of containers and your prod environment this will usually lead to some issues with developers that don't need to, or want to, have context in these areas. In the past, to make a very similar workflow possible, I've built tools that automatically watch your source files and rebuild & restart only what is needed [0]. This was built for bazel + docker-compose but there isn't a reason one couldn't watch the "build:" contexts for what files are important. At a previous company one of our engineers was a huge fan of this volume mount approach and every single time something broke (which was very frequent due to some prod/dev env magic we had) I had to assist quite a few more junior devs figure out what was wrong with their machine. For those with scripting languages, was it their system's newline endings? For compiled languages, was their system SDK different then what was in the container? For prod bugs, did they forget to rebuild & test the container before opening their PR (we had no automated integration testing)? In my opinion, if you can make your build system in charge of building/packaging things you'll have a much happier time. [0] - https://github.com/CaperAi/bazel_compose https://github.com/CaperAi/bazel_compose

Noumenon72 5y ago
What does it mean to "make your build system in charge of building/packaging things"? Like push your code out to Jenkins and have it build it? Or have Gradle make something that runs for you?

vsupalov 5y ago

Hi HN! First time I see one of my articles on here. What a nice surprise. If the above link caught your attention, you might also enjoy the following ones: * For the quickest ROI: https://vsupalov.com/improve-your-docker-images/ https://vsupalov.com/improve-your-docker-images/ * Stuff I WISH I knew: https://vsupalov.com/12-docker-facts/ https://vsupalov.com/12-docker-facts/ * If your image builds are slow: https://vsupalov.com/5-tips-to-speed-up-docker-build/ https://vsupalov.com/5-tips-to-speed-up-docker-build/ Looking forward to join the discussion later!

SatvikBeri 5y ago

I've been doing this for about 6 months now, it saves a lot of time. Especially if you need to upload changed docker images to AWS.

gmaster1440 5y ago

https://code.visualstudio.com/docs/remote/containers https://code.visualstudio.com/docs/remote/containers

pastage 5y ago

Works remotely on kubernetes, for us the game changer was running inotify_wait on cygwin for our windows users, changes are up in about 600ms. 1. Configure a service 2. Use start command sleep infinity 3. Install inotify_wait on windows 4. Do a loop like this Look for changes, Rsync changes to pod, Pkill java, Run start.sh We do a grep on changed files and only kill java if jar/classes and other deps has changed, that makes it possible to edit html in pod and get fast updates.

orlovs 5y ago
I have never understood kubernetes development in local development workflow unless kubernetes functionality being developed or some yaml magic. It’s super overhead IMHO. If needed to test api’s as unit test you can always port-forward traffic of desired enpoint or use something like telepresence. Less kubernetes is always better
- erulabs 5y ago
  It’s cpu/memory overhead yes, but between matching prod and dev, extremely simple config (redis is always located at “redis:6379”), zero setup instructions (skaffold dev), remote/local hybrid development, and fostering prod-ready skills for developers (still working on that devops koolaid), I find it -extremely- worthwhile. I’ve yet to meet someone who doesn’t hate it to start (because kubernetes) and winds up loving it and swearing by it. It’s essentially a vastly superior “vagrant up” on anabolic steroids
  - orlovs 5y ago
    I agree, for e2e/integration testing, excellent
mehphp 5y ago
Have you seen https://skaffold.dev/ https://skaffold.dev/ ?
- emj 5y ago

wiredfool 5y ago

This is great, until you need to watch a tree of files in macOS. Then docker takes a core to do the fsnotify/watch. There are caching settings to make it better, but there are race conditions between the watch and when file contents change, so sometimes the JS stack will compile an inconsistent file. OTOH, on linux, it's grand.

bdcravens 5y ago
docker-sync fixes this by using an intermediate container that handles sync (I believe there are other options which work essentially the same way). It supports ignoring files you don't need on your main machine, like tmp folders and the like, which can improve performance further. http://docker-sync.io/ http://docker-sync.io/

snapetom 5y ago

I have to scratch my head every time I see a blog post about creating a dev environment that doesn't use volumes like this. IMO, this is the way to go and what Docker was meant to be as a dev environment. Maybe it's because I'm never confident in the changes I make, but my style is to write a few lines at a time, save, re-run or refresh. I can't imagine having to do a docker build every code change. If you do the later, it's just nominally better than developing on a remote server, which brings its own challenges.

roofwellhams 5y ago
Did you try typescript? Now I can write for hours without trying out the app. And when I try... It just works
- redleather 5y ago
  This is such a weird comment.. nothing about the OC suggests they were talking about front end dev; not to mention that writing for hours before actually running it is a pretty terrible way to code. Typescript is also not a workaround for volume binding, which can accommodate any stack.
  - dehrmann 5y ago
    > nothing about the OC suggests they were talking about front end dev >> I'm never confident in the changes I make, but my style is to write a few lines at a time, save, re-run or refresh. This is a pretty typical frontend development pattern, especially seeing "refresh."
    - dividedbyzero 5y ago
      I read that as "re-run or refresh, as applicable", i.e. re-run a Go app or refresh a Node.js webapp. Besides, it absolutely can be a big issue with non-frontend development, so the point still stands.
      - 5y ago

iooi 5y ago

For devs on macOS, keep in mind that your filesystem is case insensitive! If you're using a linux image that expects a directory like `UpperCase` and you named it `uppercase` locally it would work.. until you bake the source for your production release and you get errors around that directory not existing. "But it works locally!"

flas9sd 5y ago
there can also be subtle issues with git, I recommend then to convert HFS / APFS to case-sensitive. I understand why normal people do not care, but Dev types in unix alike environments do not have a choice with decades of case-sensitive filesystems before them

tlarkworthy 5y ago

I have done this but if possible I just run my code locally and reserve docker for infra (e.g. DB). Java, nodejs, go are all very easily run locally. Maintaining a complex Dockerfile is just more crap that makes programming tedious.

Fredej 5y ago
Honestly my experience is basically the opposite: Maintaining a coherent development environment is what makes programming tedious. I'd much prefer to just specify in a dockerfile that I need this compiler, these tools and these env-variables, and then I never have to worry about that stuff again.
- tlarkworthy 5y ago
  It's hotcode reload and debugging where I find docker starts to annoy me. (Also resource consumption and performance to an extent)
- deleted 5y ago
  [deleted]
nucleardog 5y ago
Sure, nodejs is easy to run locally. nodejs+npm of the exact version that a specific project needs, which is different and incompatible with the version some other project needs starts to get more annoying. Basically all of our projects have some level of external dependencies in the form of libraries or binary tools. Some projects overlap dependencies, but on incompatible versions. This gets really hairy. Trying to do this without containers (from experience, this is what the devs were doing before) turns into a massive documentation project and a nightmare of trying to get all these incompatible things to exist on a single system where we don't even have a consistent target because everyone's local dev environment is different enough to make it a nightmare. And then continually hounding the devs to keep the documentation up-to-date. But even that doesn't solve the problem because "apt-get install nodejs" today is not necessarily the same as "apt-get install nodejs" in six months or a year and nobody's thinking about this so it's just "worked for me last time!". Onboarding people at one point was a _days_ long process and a group effort. Instead we just re-use the same 30-odd line Dockerfile we use for deploying out to the infra for local dev. Instead of pages upon pages of out-of-date documentation. a bunch of tribal knowledge, and continually running into new and interesting bugs, we now have a short and simple source of truth which _has_ to be up-to-date because otherwise the environment the project is deployed to is broken. Onboarding people takes 5 minutes. If you're a single person working on a single project, yeah, containerizing stuff is maybe some annoying overhead. Even then I still use it because once you've gotten past the learning curve, it's actually a really effective way to document and define your project's environment and ensure it stays consistent.

richardwhiuk 5y ago

We wrote a tool to make using development Docker images easy - https://metaswitch.github.io/floki/ https://metaswitch.github.io/floki/

robsalasco 5y ago
Looks awesome!

sebyx07 5y ago

For any rails dev out these volumes: - .build/.bundle-cache-dir/app:/usr/local/bundle/

gprasanth 5y ago

Another thing I've recently found helpful was docker layer caching inside github actions. Pretty easy to integrate, saves a lot of build minutes.

deleted 5y ago

[deleted]

alpb 5y ago

Skaffold is a great tool for this and employs hot reloading techniques without having to rebuild the image. https://skaffold.dev/ https://skaffold.dev/

spiddy 5y ago
I find it very interesting the idea of separating development image from production one and skaffold promotes it in a way ("skaffold dev" vs "skaffold run") Same as in frontend for example you don't "npm build" your way through development, instead you want hot-reload and other similar features.

aszen 5y ago

Docker truthfully told is unusable for Dev envs that need to change constantly. It was never built to handle this use case. I haven't seen a good docker compose file that can be reliably used as a development environment in multiple oses with good performance. There's all sorts of edge cases where volumes don't work. In my opinion docker is very useful for thing like dbs, queues and other processes where the underlying code doesn't change. But for everyday frontends and backends it's not worth it. Nowadays I write a shell.nix file which contains all the dependencies the project needs, it works but is definitely not as easy to learn as a docker file

okamiueru 5y ago
Out of curiosity, what OS are you on?
- aszen 5y ago
  I'm on Linux, but still things aren't all rosy.
mike_d 5y ago
Dev should look just like production if you are doing it right, and you are correct dev was basically an afterthought in Docker. If you run this in a screen session it will auto rebuild as you make changes, which is the closest I've been able to come to containerizing my development style: https://gist.github.com/mikedamm/53dc6a78b976eeac88893427424c2527 https://gist.github.com/mikedamm/53dc6a78b976eeac88893427424...
- aszen 5y ago
  When people say docker helps in making dev look like prod, I assume they are running everything on docker in production, but that is very rarely the case with dbs, queues and cloud specific services. Interesting I'll check this script, but I have doubts doing a rebuild is going to be quick enough even with good cache layering in the Dockerfile. Use case is that I have frontend and backend code in one repo, and make changes across them that need to be reflected in the ui.

deleted 5y ago

[deleted]

efrecon 5y ago

In the past few weeks, I have spent some time and released dew [0]. It helps encapsulating this kind of setups in configuration and minimising typing. dew is still evolving, but it has served me well. [0]: https://github.com/efrecon/dew https://github.com/efrecon/dew

wdb 5y ago

I have meant to try if VMWare's vctl is any better regarding detecting file changes

bfrog 5y ago

Nix has saner repeatable dev envs, and makes updating or adding deps a snap. Can be used to build scratch containers. Can be used to cross compile. The only major downside is it requires learning and understanding nix which I get is a hurdle, but one that's well worth it. Docker is only one container tool of many now, and it's worth exploring what else is out there.