5 ms·

I know almost nothing about this topic. I have only really built container images with Dockerfiles. How would you introduce buildpacks to me?

jacques_chester 8y ago
I used the Onsi Haiku over on the equivalent HN thread on Heroku's blogpost: Here is my source code. Run it on the cloud for me. I do not care how. The gist is that Docker containers are awesome for the Day 1 experience. I write a Dockerfile and I'm off to the races. But then Day 2 rolls around and I have a production system with 12,000 containers[0]. 1. What the hell is in those containers, anyhow? 2. A new CVE landed and I want to upgrade all of them in a few minutes without anyone being interrupted (or even having to know). How? 3. I have a distributed system with many moving parts. I build a giant fragile hierarchy of Dockerfiles to efficiently contain the right dependencies, making development slower. Then I snap and turn it into a giant kitchen-sink Dockerfile with the union of all the dependencies in it. Now production is slow as hell. 4. Operations become upset about points 1-3. Now I can only use curated Dockerfiles, can only come through our elaborate Jenkins farm, rules rules rules. Wasn't the purpose of Dockerfiles to make this all just ... go away? Buildpacks solve all of these. I know what's in the container because buildpacks control the build. I can update CVE flaws in potentially seconds. Each container can have what it needs - no more, no less. And most important: the buildpack runs locally, or in the cluster, exactly the same. It's all the developer benefits of Dockerfiles/docker build, minus most of the suck. [0] Yahoo! Japan has 12,000 prod and 8,000 dev containers using buildpacks: https://www.slideshare.net/iranainanimosuteteshimaou/yahoo-japan20000-container-journey https://www.slideshare.net/iranainanimosuteteshimaou/yahoo-j...
- kgilpin 8y ago
  It sounds like a buildpack identifies a common case of what someone might be doing in a container / Dockerfile, and standardizes it to remove the arbitrary variation that inevitably occurs when lots of people are independently solving exactly the same problem. For example, the patch level of the OS is directly or indirectly specified in the Dockerfile, but generally doesn’t actually matter to the application. If your use case fits into a scenario which is handled by an existing buildpack, then the claim is that you’ll be better off using the buildpack because the infrastructure can make optimizations that can’t be made with arbitrary containers. If your use case isn’t covered by a buildpack, then you can either (1) make a buildpack or (2) revert to raw containers. (Although your platform may not allow (2)).
  - jacques_chester 8y ago
    I think this is a pretty solid summary. Clound Native Buildpacks are based on a very well-proved model. Heroku do this at massive scale. So does Pivotal, so do many of our various customers and customers of other buildpack-using systems like Deis.
bryanlarsen 8y ago
Step 1: replace your Dockerfile with one that consists of the single line "FROM cloud-gov-registry.app.cloud.gov/python-buildpack". The buildpack contains magic that knows how to turn a standard python program into a usable docker image. (AKA, it inspects requirements.txt, et cetera). Step 2: Now that your Dockerfiles no longer contain any real information, retool your orchestration system to use source tarballs rather than docker images.
- pm90 8y ago
  That sounds un-ideal to folks who are used to and comfortable with the Docker image/Dockerfile based workflow. What advantages does this provide over plain Dockerfiles?
  - stevehiehn 8y ago
    In a situation where there are many running containers a security patch could be applied across the board and still have confidence that running apps are not affected. This would be a case by case situation if they were all Dockerfiles.
    - jacques_chester 8y ago
      I agree with this bit. The cool part is that when the patch arrives, devs don't have to do anything to be patched. Cloud Foundry already does this with buildpacks and so does Heroku. A big part of what's new is that layer rebasing could make this really really fast.
      - pm90 8y ago
        Personally, I don't think this kind of magic is a good thing. But to each their own.
        jacques_chester 8y ago
        Could you elaborate, though? It helps us to understand what seems magical so that we can either explain it better or make the process more transparent.

jacques_chester 8y ago

I used the Onsi Haiku over on the equivalent HN thread on Heroku's blogpost: Here is my source code. Run it on the cloud for me. I do not care how. The gist is that Docker containers are awesome for the Day 1 experience. I write a Dockerfile and I'm off to the races. But then Day 2 rolls around and I have a production system with 12,000 containers[0]. 1. What the hell is in those containers, anyhow? 2. A new CVE landed and I want to upgrade all of them in a few minutes without anyone being interrupted (or even having to know). How? 3. I have a distributed system with many moving parts. I build a giant fragile hierarchy of Dockerfiles to efficiently contain the right dependencies, making development slower. Then I snap and turn it into a giant kitchen-sink Dockerfile with the union of all the dependencies in it. Now production is slow as hell. 4. Operations become upset about points 1-3. Now I can only use curated Dockerfiles, can only come through our elaborate Jenkins farm, rules rules rules. Wasn't the purpose of Dockerfiles to make this all just ... go away? Buildpacks solve all of these. I know what's in the container because buildpacks control the build. I can update CVE flaws in potentially seconds. Each container can have what it needs - no more, no less. And most important: the buildpack runs locally, or in the cluster, exactly the same. It's all the developer benefits of Dockerfiles/docker build, minus most of the suck. [0] Yahoo! Japan has 12,000 prod and 8,000 dev containers using buildpacks: https://www.slideshare.net/iranainanimosuteteshimaou/yahoo-japan20000-container-journey https://www.slideshare.net/iranainanimosuteteshimaou/yahoo-j...

kgilpin 8y ago
It sounds like a buildpack identifies a common case of what someone might be doing in a container / Dockerfile, and standardizes it to remove the arbitrary variation that inevitably occurs when lots of people are independently solving exactly the same problem. For example, the patch level of the OS is directly or indirectly specified in the Dockerfile, but generally doesn’t actually matter to the application. If your use case fits into a scenario which is handled by an existing buildpack, then the claim is that you’ll be better off using the buildpack because the infrastructure can make optimizations that can’t be made with arbitrary containers. If your use case isn’t covered by a buildpack, then you can either (1) make a buildpack or (2) revert to raw containers. (Although your platform may not allow (2)).
- jacques_chester 8y ago
  I think this is a pretty solid summary. Clound Native Buildpacks are based on a very well-proved model. Heroku do this at massive scale. So does Pivotal, so do many of our various customers and customers of other buildpack-using systems like Deis.

bryanlarsen 8y ago

Step 1: replace your Dockerfile with one that consists of the single line "FROM cloud-gov-registry.app.cloud.gov/python-buildpack". The buildpack contains magic that knows how to turn a standard python program into a usable docker image. (AKA, it inspects requirements.txt, et cetera). Step 2: Now that your Dockerfiles no longer contain any real information, retool your orchestration system to use source tarballs rather than docker images.

pm90 8y ago
That sounds un-ideal to folks who are used to and comfortable with the Docker image/Dockerfile based workflow. What advantages does this provide over plain Dockerfiles?
- stevehiehn 8y ago
  In a situation where there are many running containers a security patch could be applied across the board and still have confidence that running apps are not affected. This would be a case by case situation if they were all Dockerfiles.
  - jacques_chester 8y ago
    I agree with this bit. The cool part is that when the patch arrives, devs don't have to do anything to be patched. Cloud Foundry already does this with buildpacks and so does Heroku. A big part of what's new is that layer rebasing could make this really really fast.
    - pm90 8y ago
      Personally, I don't think this kind of magic is a good thing. But to each their own.
      - jacques_chester 8y ago
        Could you elaborate, though? It helps us to understand what seems magical so that we can either explain it better or make the process more transparent.