3 ms·

My biggest beef with the apt/deb-based distros for container images is that apt-get updates installs take frustratingly long, whereas apks always tend to be nea

by kelvie 3y ago

My biggest beef with the apt/deb-based distros for container images is that apt-get updates installs take frustratingly long, whereas apks always tend to be near instant. I wonder what it is in the postinst and related scripts that just take so long and can't be parallelized! Most of the reason I switched from Ubuntu -> Arch is that working with alpine allowed me to realize that installing packages and their dependencies don't have to take so long.

p_l 3y ago
Have you tried building the images with eatmydata library? Essentially, dpkg is hilariously slow due to its approach in issuing fsync calls for every file - eatmydata wrapper injects a shared library which turns that into a no-op. I haven't timed it rigorously, but I experienced up to 10x speedup in container builds by just wrapping dpkg with eatmydata.
- gorgoiler 3y ago
  [Edit: modern debian images have “force-unsafe-io” turned on in /etc/dpkg/ so eatmydata shouldn’t make a difference.] I’m not the OP, but in an example I just tried it took build-essential from 48s to 39s (timed on the third step, without and then with the call to eatmydata): from debian:sid run apt update && env DEBIAN_FRONTEND=noninteractive apt-get install --yes eatmydata run apt update && eatmydata env DEBIAN_FRONTEND=noninteractive apt-get install --yes build-essential Can you replicate something faster? Was I doing something wrong?
  - TylerE 3y ago
    Yes, the second invocation will have hot caches. Basically the standard is to either test in a clean environment (e.g. immediately after a reboot) or to run the command until the times stabilize, and then take, say, the middle 3 out of 5 measurements.
    - gorgoiler 3y ago
      What is cached though? DNS? IP routes to the Debian mirrors? I ran it three more times. Plain install was 50s 48s 47s; eatmydata was 46s 46s 46s. Installing build-essential probably does enough non-dpkg IO to account for the difference (install scripts etc, for example when libc changes and the system scans for running daemons) but it isn’t much and the delta may well be statistically insignificant.
      - TylerE 3y ago
        File system. Drives cache recently read blocks. This isn’t something that’s neesesarily e en visible to the OS.
        lenova 3y ago
        I decided to benchmark these scenarios out of curiosity, because I hadn't heard of eatmydata before or of force-unsafe-io. - Docker image: `debian:stable-slim` - `docker system prune -a` ran before each test. - Packages installed: `build-essential ca-certificates libssl-dev software-properties-common wget curl git` Results: Default Debian settings: 48.32 seconds force-unsafe-io enabled: 55.86 seconds eatMyData enabled: 35.04 seconds
  - p_l 3y ago
    It's possible that the last time I dealt with that on debian force-unsafe-io wasn't set in /etc/dpkg. I haven't really built debian-based containers in last two years.

by kelvie 3y ago

p_l 3y ago

Have you tried building the images with eatmydata library? Essentially, dpkg is hilariously slow due to its approach in issuing fsync calls for every file - eatmydata wrapper injects a shared library which turns that into a no-op. I haven't timed it rigorously, but I experienced up to 10x speedup in container builds by just wrapping dpkg with eatmydata.

gorgoiler 3y ago
[Edit: modern debian images have “force-unsafe-io” turned on in /etc/dpkg/ so eatmydata shouldn’t make a difference.] I’m not the OP, but in an example I just tried it took build-essential from 48s to 39s (timed on the third step, without and then with the call to eatmydata): from debian:sid run apt update && env DEBIAN_FRONTEND=noninteractive apt-get install --yes eatmydata run apt update && eatmydata env DEBIAN_FRONTEND=noninteractive apt-get install --yes build-essential Can you replicate something faster? Was I doing something wrong?
- TylerE 3y ago
  Yes, the second invocation will have hot caches. Basically the standard is to either test in a clean environment (e.g. immediately after a reboot) or to run the command until the times stabilize, and then take, say, the middle 3 out of 5 measurements.
  - gorgoiler 3y ago
    What is cached though? DNS? IP routes to the Debian mirrors? I ran it three more times. Plain install was 50s 48s 47s; eatmydata was 46s 46s 46s. Installing build-essential probably does enough non-dpkg IO to account for the difference (install scripts etc, for example when libc changes and the system scans for running daemons) but it isn’t much and the delta may well be statistically insignificant.
    - TylerE 3y ago
      File system. Drives cache recently read blocks. This isn’t something that’s neesesarily e en visible to the OS.
      - lenova 3y ago
        I decided to benchmark these scenarios out of curiosity, because I hadn't heard of eatmydata before or of force-unsafe-io. - Docker image: `debian:stable-slim` - `docker system prune -a` ran before each test. - Packages installed: `build-essential ca-certificates libssl-dev software-properties-common wget curl git` Results: Default Debian settings: 48.32 seconds force-unsafe-io enabled: 55.86 seconds eatMyData enabled: 35.04 seconds
- p_l 3y ago
  It's possible that the last time I dealt with that on debian force-unsafe-io wasn't set in /etc/dpkg. I haven't really built debian-based containers in last two years.