3 ms·

While that's true for processes running in containers, building containers is generally noticeably slower than Makefiles as it involves several extra steps besi

by pid-1 4y ago

While that's true for processes running in containers, building containers is generally noticeably slower than Makefiles as it involves several extra steps besides executing instructions (e.g. loading build context, gziping files, calculating hashes etc...)

folkrav 4y ago
You'd typically not rebuild CI images that often.
- yjftsjthsd-h 4y ago
  If you're following the article's suggestion of using docker to replace makefiles and/or run build processes in docker and just spit out the final product using --output, the you will build the image on every single compile. And I'd try to give docker credit for caching all the layers it can, but make caches results too so it's not necessarily even winning there.
- hinkley 4y ago
  You’re building Continuous Integration images all the time. The more often the better. Hence the aspirational “continuous” in the name.
hatware 4y ago
> building containers is generally noticeably slower than Makefiles I totally agree, if the year was 1992. Compute is cheap. Storage is cheap. Let's not try and keep living in the past.
- minitech 4y ago
  No, it’s noticeably slower than Makefiles on present-year hardware. This degrades the developer experience regardless of how cheap compute and storage are. (This is mostly Docker’s fault and not inherent to containers, though.)
  - hatware 4y ago
    > This degrades the developer experience regardless of how cheap compute and storage are. I would argue that if you only have one good tool to do something, you have an opportunity to make much better tools for that something. It's not docker's job to be better than Makefile with every use-case. You do realize what you're asking for here right? It's impossible. No tool is perfect for every use-case.
    - yjftsjthsd-h 4y ago
      > It's not docker's job to be better than Makefile with every use-case. You do realize what you're asking for here right? It's impossible. No tool is perfect for every use-case. The root of this thread is > We actually use docker multistage build as mentioned here as a replacement for make file. So yes, it absolutely is reasonable to ask that docker, when used in the exact same way as make, should not give us a noticeable performance regression compared to running make.
    - pid-1 4y ago
      Did you even read the article?
      - hatware 4y ago
        345 words is barely an article. What about my comments are off topic exactly...?
  - hinkley 4y ago
    Part of CI is fast feedback. Slow builds started soon still suffer from not providing the full experience. It’s not a ratio that’s a problem, although that doesn’t help. It’s that there are psychological cutoffs where adding a minute to a build drastically changes how people interact with it. 1 vs 3 vs 7 vs >7 minutes make substantial differences. So taking a build from 8 to 6 minutes is valuable out of all proportion to the actual speed up, and slowing from 6 to 8 is a substantial loss.
toniti 4y ago
BuildKit(Docker build engine) maintainer here. It’s definitely our goal to limit the friction, so you get the benefits of containers but with similar performance as running things on the host. For the points you listed: BuildKit sends context incrementally between builds that transfers only deltas and not the whole context. Gzip is only needed when you want to push the image to a registry that doesn’t happen as often as your dev builds. For the hashes, we do need to calculate some for build cache (that of course makes your builds faster than they would be without it). For some bigger objects like layers we can defer checksum computation to push phase similarly to compression. Another thing we have done is added cache mounts that let you keep application-specific cache between builds. This is usually, where the speed benefit from running things on the host used to come if your tools wrote some cache under $HOME etc. There is still a constant container sandbox initialization time ~100ms, with some room to optimize that as well. If you have good examples of builds that can’t be easily optimized in containers would love to get your feedback/examples on the issue tracker.

by pid-1 4y ago

folkrav 4y ago

You'd typically not rebuild CI images that often.

yjftsjthsd-h 4y ago
If you're following the article's suggestion of using docker to replace makefiles and/or run build processes in docker and just spit out the final product using --output, the you will build the image on every single compile. And I'd try to give docker credit for caching all the layers it can, but make caches results too so it's not necessarily even winning there.
hinkley 4y ago
You’re building Continuous Integration images all the time. The more often the better. Hence the aspirational “continuous” in the name.

hatware 4y ago

> building containers is generally noticeably slower than Makefiles I totally agree, if the year was 1992. Compute is cheap. Storage is cheap. Let's not try and keep living in the past.

minitech 4y ago
No, it’s noticeably slower than Makefiles on present-year hardware. This degrades the developer experience regardless of how cheap compute and storage are. (This is mostly Docker’s fault and not inherent to containers, though.)
- hatware 4y ago
  > This degrades the developer experience regardless of how cheap compute and storage are. I would argue that if you only have one good tool to do something, you have an opportunity to make much better tools for that something. It's not docker's job to be better than Makefile with every use-case. You do realize what you're asking for here right? It's impossible. No tool is perfect for every use-case.
  - yjftsjthsd-h 4y ago
    > It's not docker's job to be better than Makefile with every use-case. You do realize what you're asking for here right? It's impossible. No tool is perfect for every use-case. The root of this thread is > We actually use docker multistage build as mentioned here as a replacement for make file. So yes, it absolutely is reasonable to ask that docker, when used in the exact same way as make, should not give us a noticeable performance regression compared to running make.
  - pid-1 4y ago
    Did you even read the article?
    - hatware 4y ago
      345 words is barely an article. What about my comments are off topic exactly...?
- hinkley 4y ago
  Part of CI is fast feedback. Slow builds started soon still suffer from not providing the full experience. It’s not a ratio that’s a problem, although that doesn’t help. It’s that there are psychological cutoffs where adding a minute to a build drastically changes how people interact with it. 1 vs 3 vs 7 vs >7 minutes make substantial differences. So taking a build from 8 to 6 minutes is valuable out of all proportion to the actual speed up, and slowing from 6 to 8 is a substantial loss.

toniti 4y ago

BuildKit(Docker build engine) maintainer here. It’s definitely our goal to limit the friction, so you get the benefits of containers but with similar performance as running things on the host. For the points you listed: BuildKit sends context incrementally between builds that transfers only deltas and not the whole context. Gzip is only needed when you want to push the image to a registry that doesn’t happen as often as your dev builds. For the hashes, we do need to calculate some for build cache (that of course makes your builds faster than they would be without it). For some bigger objects like layers we can defer checksum computation to push phase similarly to compression. Another thing we have done is added cache mounts that let you keep application-specific cache between builds. This is usually, where the speed benefit from running things on the host used to come if your tools wrote some cache under $HOME etc. There is still a constant container sandbox initialization time ~100ms, with some room to optimize that as well. If you have good examples of builds that can’t be easily optimized in containers would love to get your feedback/examples on the issue tracker.