16 ms·
Tiny Container Challenge: Building a 6kB Containerized HTTP Server
- paulfurtado 6y agoFWIW, if you wanted to get the static from-scratch go image a lot smaller, try running the binary through UPX. It typically improves compiled go binary sizes by absurd amounts: https://upx.github.io/ https://upx.github.io/ Also, https://busybox.net https://busybox.net is a 1.0MB static binary that provides all of the standard posix-compliant tools (sh, grep, tail, etc) and actually comes with an httpd implementation. So if you wanted a more useful container at about 1MB, you could use it. Edit: here's a previous thread on UPX that has 80 comments talking about the tradeoffs it makes. You should definitely consider these if you intend to use UPX: https://news.ycombinator.com/item?id=15456980 https://news.ycombinator.com/item?id=15456980
- daniellockard 6y agoBe sure to check out the switches, --ultra-brute is what I usually use.
- codetrotter 6y agoBeware that at least historically some antivirus software would wrongly classify any UPX-packed binary as malware. Don’t know if this is still the case but something to keep in mind.
- account42 6y ago"some antivirus software would wrongly classify X as malware" is true for any X (except malware, then would not be wrong)
- fouc 6y ago1.0MB is quite large compared to 6.34kB :)
- whizzter 6y agoCertainly, I think this is more about getting over the developer pain threshold. In terms of computer speed vs human perception I don't see either one mattering too much (compared to a 900meg base for node-debian on Docker, although I guess there is some incremental stuff for that?).
- paulfurtado 6y agoIt definitely is, although a cool thing about busybox is that you can rebuild it with any applets disabled. I'm not sure just how small it can get though. But yeah, the tradeoff of that 1MB is that you get a very functional shell environment vs just an http server. It's impressive just how small ut is, given everything that it is capable of
- shakna 6y ago> It definitely is, although a cool thing about busybox is that you can rebuild it with any applets disabled. I'm not sure just how small it can get though. They do distribute statically linked individual binaries [0]. For just HTTPD and nothing else is about 85kb. But it is a lot more than just a static file server - it comes with CGI support. [0] https://www.busybox.net/downloads/binaries/1.30.0-i686/ https://www.busybox.net/downloads/binaries/1.30.0-i686/
- js8 6y ago> But yeah, the tradeoff of that 1MB is that you get a very functional shell environment vs just an http server. It's impressive just how small ut is, given everything that it is capable of Maybe.. I suspect a well-designed Forth image could pack similar capability in less than 100kB. So it might seem low, but not a miracle.
- TonyTrapp 6y agoKeep in mind that UPX needs to decompress the whole executable to memory and thus cannot make use of OS-level memory sharing. Normally, when you load the same executable twice, its base image will only consume memory once. But it with UPX, you cannot make use of that. So in the end, it may use more resources than you thought!
- paulfurtado 6y agoDefinitely important to note, I wonder if upx could solve this via execing an memfd_create in some mode. Also wish upx came with a trivial extraction tool so you could use it for distribution but not runtime.
- prova1 6y agook.
- mikepurvis 6y agoAren’t your container layers compressed in transit anyway?
- paulfurtado 6y agoUPX does significantly more compression on binaries than gzip can on the layers. It's not just simple compression, it makes a lot of modifications to the binary to make it more compressible.
- mikepurvis 6y agoPerhaps this is what you were asking for in the GP, but it would be interesting then if UPX could apply those transformations without actually doing the final compression step itself. Perhaps the win isn't as great without the ability to smuggle specific hints to the compressor, but it would be interesting to see. On the other hand, if it could be demonstrated that modest increases to binary size correlated to significantly improved compressability, then maybe that's something that should just be considered by the compilers themselves.
- okamiueru 6y agoI did exactly this a few years back. It added 18.5 kB on top of busybox, so, not as good as the 6kB ASM server code. With busybox included, the whole image is 1.18 MiB, or 751 kB compressed. If anyone cares to take a look: https://hub.docker.com/r/swalog/tiny-hello/ https://hub.docker.com/r/swalog/tiny-hello/
- prova1 6y agoholaa
- endgame 6y agoIf you only want the sh, you can get it even smaller. This doesn't UPX the busybox bin, though: $ ls -shL `nix-build -E '(import <nixpkgs> {}).pkgsCross.musl64.busybox.override { enableStatic = true; enableMinimal = true; }'`/bin/sh 128K /nix/store/qkq9cj81mcvnfr5y04jmf6nqvnrfg2z1-busybox-1.31.1-x86_64-unknown-linux-musl/bin/sh
- davidjfelix 6y agoIf you're going for size, why not just sidecar all of the "standard posix-compliant tools" but ones that you would want normally. For example, bash instead of ash/dash. You don't need those tools all the time right?
- dolmen 6y agoWell, you don't even need a shell.
- davidjfelix 6y agoI mostly meant to sidecar the shell and treat it as a debugging utility. While I get the point of ash being smaller as a point for busybox, if size stops being a concern because the shell is in an inactive-sidecar, then you can use a shell that has less surprising edge cases for users, like bash.
- dolmen 6y agoAbout busybox: if you want a small container, you don't need the posix tools at all, but just your binary. No shell.
- spalas 6y agoHi, author of the article here! I figured there were a bunch of ways to shrink that one, and I had a C based image that was in between my GO and ASM images, but decided to cut them for the sake of brevity!
- dnautics 6y agostealing vexu's routez: https://github.com/Vexu/routez https://github.com/Vexu/routez gives you a multiprocess http server in 612K uncompressed if you compile -Drelease-small
- pjmlp 6y agoOr just ship a statically compiled binary, with resources inside, without having to mess with anyhing docker related.
- tbrock 6y agoThe golang example where he uses scratch as the base image is essentially this.
- pjmlp 6y agoExcept he packages it on a container.
- dijit 6y agoThe container, if it's `from SCRATCH` is basically just a gzip/zip archive. https://github.com/opencontainers/image-spec/blob/master/image-layout.md https://github.com/opencontainers/image-spec/blob/master/ima...
- regularfry 6y agoWhich is probably unnecessary if you've got the deployment platform to yourself and can decide that just-run-this-binary is suitable, safe, and manageable. Personally I prefer "just install this .deb" because it wraps up dependency management along with everything else, but that's not been a popular opinion for a long time.
- secondcoming 6y agoMaybe because creating custom deb packages can be a hair-pulling experience? But this is our flow too. We have a jenkins job that builds our master branch and uploads deb packages to jfrog. And then another to create a GCP image with these psckages, and a third to actually deploy these images to GCE. Containers are used for the various jenkins jobs, but not on the actual GCE instances.
- 6y ago
- xiaodai 6y agoAmazing! Thank you.
- throwaway4good 6y agoWhat is in "scratch"? Couldn't you run a C-program there or a JVM?
- onei 6y agoIf you imagine a container as a tarball, then scratch is an empty tarball. Each time you copy a file or set of files into the container during the build, you get a new tarball or layer. Some base containers provided by OS vendors, e.g. CentOS, Debian, are a single layer with the entire OS filesystem inside including tooling, libraries etc. Scratch has no tooling or libraries or anything. It's empty. You typically add a binary into it and then just run that binary, but due to the lack of libraries, it just be statically compiled. So in answer to your question, you can run statically compiled C. You can't run Java due to the lack of JVM which I believe has its on set of filesystem dependencies, but I'm not a Java dev, so I'm not entirely sure.
- throwaway4good 6y agoI meant copying the JVM unix tar ball in there and use it to run whatever. Why aren't everyone just using scratch?
- NavinF 6y agoI'm pretty sure the default build of openjdk dynamically links glibc. If you build it statically, that should work.
- throwaway4good 6y agoWhy is it not best practice to do so? The naive solution using the golang image is nearly 1GB. Why carry this extra complexity around?
- NavinF 6y agoIt is best practice and I wish it was common practice, but there's a lot of friction in the way and some people will actively work against you. Static linking glibc is painful because the former maintainer has some strong opinions: https://www.akkadia.org/drepper/no_static_linking.html https://www.akkadia.org/drepper/no_static_linking.html Most distros force you to dynamically link every dependency if you want them to package it. So the default build for most projects is dynamic You've stumbled on a holy war between distros and guys like you, me, and Linus Torvalds[1] that want to deploy a binary and just have it work everywhere. 1: https://youtu.be/5PmHRSeA2c8?t=295 https://youtu.be/5PmHRSeA2c8?t=295 (Highly recommend watching Linus's answer ~6 minutes in)
- hardwaresofton 6y agoOne thing I'm wondering -- why are people obsessed with container size? A lot of enterprise is Java shops which deal with huge dependencies, huge distributions, and generally pretty heavy software -- with internet speeds what they are today, it's not hard to throw 100-500MB around internal networks (building the code closer to the deployment area is also a solution). Even gigabyte size containers are not impossible to deploy these days with the right caching (lord help you if you invalidate a bunch of layers though). In a world of xTB thumbdrives, why are we so worried about containers being <1MB? Security is one thing, minimalism is another, but I do sometimes wonder about this. Anyway, if you want to build minimal containers and want to do it with a bit more backing than this one (pretty well written) blog post, probably also check out distroless: https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless I personally find most of the time alpine-based containers are more than small enough and good enough with nice ergonomics (sometimes you just want to shell in to the container), but maybe it makes sense to build containers with and without debugging/inspection tools (a `project:vx.x.x` and a `project:vx.x.x-debug`) and switch them out when problems arise.
- thrwaeasddsaf 6y agoI don't know about you but I work on a laptop, and I work on multiple projects, and each project might use dozens of containers for builds, testing, releases, etcetra. If every container were a gigabyte in size, majority of the laptop's internal storage would be wasted on them, nevermind the time wasted pushing and pulling updates.
- hardwaresofton 6y agoI do too (my laptop is pretty beefy though), and yes you're right this would be a problem, if containers weren't built to combat this with Copy on Write filesystems. That said, dependencies do take up a lot of space, but machines these days also come with a lot of space. People also run prune and develop caching strategies (ex. NPM) as a result. It's not like every container will be a GB in size, as it's not reasonable to have every container have that much content, but how many people are out there using 950MB dependency-filled base containers with 50MB of actual relevant application code? Most people learned the run-it-on-alpine lesson like 5+ years ago at this point.
- fouc 6y agoWonder what the smallest containerized c10k web server would look like.
- El_RIDO 6y agoTo provide a real-life use-case for such a tiny container image: With Kubernetes one may want to run the service container as a random or at least non-root user id and store data in an attached volume. That volume will usually be attached as owned by root. So often an init-container is used, which runs once, before the service container is started, to change the permissions. Here is an image with a statically linked chown binary from the busybox tools cowering this use case: https://hub.docker.com/r/privatebin/chown https://hub.docker.com/r/privatebin/chown (I am the author of that image) Benefits of using a small image are less attack surface and faster operation (less data to download, lower startup time, less memory, etc.).
- spalas 6y agoHi! Author here! My (somewhat silly) use case for the tiny container was trying to cram 10,000 pods onto a k8s cluster without breaking the bank: https://www.youtube.com/watch?v=1y2nRNexRVk https://www.youtube.com/watch?v=1y2nRNexRVk
- deleted 6y ago[deleted]
- stevefolta 6y agoI tried using Nix to build the Node version ("pkgs.dockerTools.buildImage"). That brought the image size down from 943M to 205M. Using a musl64 "crossSystem" (which required rebuilding everything) brought it down to 188M.
- prova1 6y agoeeee
- prova1 6y ago222
- friseurtermin 6y agoI love blog posts like these: they don't go after real-world problems directly but rather explore what's even possible. paulfurtado in this comment section mentioned UPX[0] as a way to get binaries even smaller. Funnily enough UPX even managed to decrease the size of the asmttpd web server for me: ### build stage ### FROM ubuntu:18.04 as builder RUN apt update RUN apt install -y make yasm as31 nasm binutils git RUN git clone https://github.com/nemasu/asmttpd.git asmttpd RUN mv asmttpd/* . RUN rm -rd asmttpd RUN make release # i have the upx binary on my machine in this case COPY ./upx-3.96-amd64_linux/upx upx RUN ./upx --brute asmttpd ### run stage ### FROM scratch COPY --from=builder /asmttpd /asmttpd COPY ./index.html /web_root/index.html CMD ["/asmttpd", "/web_root", "8080"] With a 4 byte index.html my total image size is now 5.3kB! [0]: https://github.com/upx/upx https://github.com/upx/upx
- prova1 6y agotest111
- ourmandave 6y agoWhatever it takes to stay on the AWS free tier.
- dvfjsdhgfv 6y agoUsing asm is a bit extreme and can lead to many problems. The toy project used is quite limited anyway; you could use one of the tiny C httpds and still get the size under 15k for similar functionality. Would anyone use this in production? I doubt so. You want a reliable battle-tested solution. Nginx can be compiled to 500kb without OpenSSL; if you need https then it's just a few megabytes, I find it acceptable for non-embedded solutions like containers.
- zwischenzug 6y agoReminds me of this related article: https://zwischenzugs.com/2018/05/22/a-docker-image-in-less-than-1000-bytes/ https://zwischenzugs.com/2018/05/22/a-docker-image-in-less-t... TL;DR can you get an image to run for test purposes that can fit in a screenful of base64?
- sigmonsays 6y agoI did this challenge a long time ago and found a much smaller http server in assembly. This container ends up being 436 bytes https://github.com/sigmonsays/smallest-docker-httpd https://github.com/sigmonsays/smallest-docker-httpd
- ttfkam 6y agoAbout 50KB, but supports more than just AMD64. I submitted a patch to darkhttpd so folks can build the image themselves, but you can also download the images here. https://hub.docker.com/r/mileselam/darkhttpd/tags https://hub.docker.com/r/mileselam/darkhttpd/tags
- ttfkam 6y agodarkhttpd really is awesome. https://github.com/emikulic/darkhttpd https://github.com/emikulic/darkhttpd
- deleted 6y ago[deleted]
- alpb 6y agoWhy do you need a httpd server in front of Node.js? Isn't the builtin server capable of handling http requests concurrently and routing?
- pantuza 6y agoInteresting discussion. Just a comment about the size o Golang binary. In the post example it uses standard fmt library. fmt by itself, adds several megabytes to a binary when compiled. Removing it would reduce binary size.