3 ms·
Teradata contributes the Facebook open-source project Presto. Presto uses Docker to run tests against Presto. Since the tests require Hadoop to do much of anyth
by mauvehaus 9y ago
Teradata contributes the Facebook open-source project Presto. Presto uses Docker to run tests against Presto. Since the tests require Hadoop to do much of anything useful, we install Hadoop in docker containers.
And we run tests on 3 flavors of Hadoop (HDP, CDH, and IOP), each of which is broken down into a flavor-base image with most of the packages installed, and various other images derived from that, which means we have a dependency chain that looks like:
base-image -> base-image-with-java -> flavor-base => several other images.
Enter make, to make sure that all of these get rebuilt in the correct order and that at the end, you have a consistent set of images.
https://github.com/Teradata/docker-images https://github.com/Teradata/docker-images
But wait, there's more. Docker LABEL information is contained in a layer. Our LABEL data currently includes the git hash of the repo. Which means any time you commit, the LABEL data on base-with-java changes, and invalidates everything downstream. This is terrible, because downloading the hadoop packages can take a while. So I have a WIP branch that builds the images from an unlabelled layer.
https://github.com/ebd2/docker-images/tree/from-unlabelled https://github.com/ebd2/docker-images/tree/from-unlabelled
As an added bonus, there's a graph target that automatically creates an image of the dependency graph of the images using graphviz.
Arguably, all of the above is a pretty serious misuse of both docker and make :-)
I can answer complaints about the sins I've committed with make, but the sins we've committed with Docker are (mostly) not my doing.