11 ms·
Introducing Google Cloud Container Builder
- mikebannister 10y agoReally interesting. Is it possible that you'll remove the "one build at a time" restriction at some point? I'm imagining builds across a large org some of which would need to be run in parallel. Or am I misperceiving things in one way or another? Thanks!
- skj 10y agoFor now, we ask that those orgs get in touch with us directly for the purposes of easing these restrictions.
- bsaul 10y agoWith appengine exposing instances and cloud container facilitating the deployment of software packages, i wonder how long it'll take until the two merge.
- tav 10y agoThat's basically what App Engine Flexible [0] is. It's yet to reach General Availability, but all you have to do is define a bunch of containers and a dispatch.yaml file [1], and App Engine then takes care of deploying/routing/scheduling it for you. [0] https://cloud.google.com/appengine/docs/flexible/ https://cloud.google.com/appengine/docs/flexible/ [1] https://cloud.google.com/appengine/docs/flexible/python/how-requests-are-routed#routing_with_a_dispatch_file https://cloud.google.com/appengine/docs/flexible/python/how-...
- deleted 10y ago[deleted]
- poofyleek 10y agoThis sounds useful and revived my interest in GAE. The Go example has health check code. It makes sense. Are there equivalent offerings from AWS and Azure? This looks likes a unique GAE feature.
- pm90 10y agoHah, interesting to note how much dogfooding happens at Google before they make these technologies available to outside customers. This is a really smart move on their part.
- misframer 10y agoThat's also what Amazon does, right?
- nhumrich 10y agoNot exactly. I worked at Amazon back when code piplelines/ecs came out. What happens at Amazon is that they have a system in place, but it's very specific to amazon tools, and dependent on so many things. So they take they idea and reimplement it more genericly. But they don't then re-use the new product. You end up sometimes with something great, and othertimes not so much. Pipelines internal to amazon is awesome. One of the best things about working there. I still haven't found anything like it, though gitlab is getting close. AWS pipelines on the other hand, was built by the same people, but it really isn't great at all IMO, not even the same thing. Now, core services such as ec2 and S3 are dogfooded for sure, but it is more like amazon doesn't eat their own dogfood until after the product becomes big.
- theCricketer 10y agoCould someone explain specifically what problem Container Builder solves?
- fishnchips 10y agoI'd imagine it's the same problem companies like Circle CI or Codeship solve with their Docker-focused products - testing, building and deploying containers.
- kellyjandrews 10y agoCodeship here again ;). We love when Google releases new products - it tends to improve the entire space in general. Our take is that Google Cloud Container Builder will fit in nicely with the Google ecosystem, but Codeship is trying to solve a different problem by being platform agnostic, and flexible enough to solve the holistic CI/CD workflow.
- t0mk 10y agoI think it's sort of container building as a service. It moves the burden of building Docker container from your laptop (or from your CI tool) to the GCE. It takes away the docker push, as it goes straight to the GCE registry (I guess). So if you need to do docker build too often, or maybe your container-release process feels a bit quirky, you might want to do it with the container builder. It doesn't seem too ground-breaking, people probably already have it automated somehow, or they use sth from Docker hub (https://docs.docker.com/docker-hub/builds/#remote-build-triggers https://docs.docker.com/docker-hub/builds/#remote-build-trig...), or quay.
- skj 10y agoIn addition to the "typical" kinds of builds you'll get with existing services (ie run 'docker build' and 'docker push' for me), we offer a flexible and powerful configuration language: https://cloud.google.com/container-builder/docs/api/build-steps https://cloud.google.com/container-builder/docs/api/build-st... One of the goals is to make it easy to separate the build-time environment from the run-time environment. eg, keep the JDK out of your deployable. We do this by letting you choose arbitrary container images to run with your source volume-mounted in. We provide some simple ones, like one that runs the 'docker' CLI, and we take care of authenticated docker pushes to GCR. But, you can use whatever you like as a builder.
- fishnchips 10y agoI'd love to see that target the market for CI/CD tools which cost an arm and leg while offering a shiny UI but only a minor convenience. That would require an extended tutorial section and an ecosystem of plugins.
- skj 10y agoWe agree! We maintain the official plugins (which we call "build steps" or "builders") here: https://github.com/googlecloudplatform/cloud-builders https://github.com/googlecloudplatform/cloud-builders Support is limited, more or less, to what I am really familiar with. If you have other use cases that we could support with other builders, I'd love to see a FR on that GitHub repo. Also, you can use literally any container image as a build step. Details about how to make that happen are here: https://cloud.google.com/container-builder/docs/api/custom-build-steps https://cloud.google.com/container-builder/docs/api/custom-b...
- fishnchips 10y agoI like how Codeship approaches the problem - see here for inspiration https://documentation.codeship.com/pro/getting-started/steps/ https://documentation.codeship.com/pro/getting-started/steps... Still, they're super expensive and not very performant, so I welcome competition. What I'm missing here is an easy way to link a container being tested with external containers (eg DB).
- skj 10y agore: link the container with external containers, that's a bit tricky and generally I think that's a space you should use a "dev" environment for. We like to be the connection between source and registry, with your dev environment being after the registry.
- fishnchips 10y agoThat's not how I expect CI to work. I might be wrong though.
- hollywoodlife 10y agoFifty Shades Darker full movie online free https://www.linkedin.com/pulse/topmovie-50-fifty-shades-darker-full-free-online-movie-frances-xiong https://www.linkedin.com/pulse/topmovie-50-fifty-shades-dark...
- wstrange 10y ago"First 120 build-minutes per day" : FREE Nice for Hobby projects...
- ImJasonH 10y agoWe haven't had many projects exceed their daily free tier so far, and we've been backing all App Engine Flexible deployments for a while. If you do exceed your daily free tier, you're only billed $0.0034/minute, on a per-second basis, so we expect the cost of using this product to be very low. (disclaimer: I work on this product)
- LeonM 10y agoI've ran medium sized projects (not hobby projects) on the GCCB, and they took about 5 minutes to build and test. Unless I want to build >24 times a day, this means GCCB will be free for my business in most cases. Personally, I think it's a great gesture, I didn't ask for it and yet it's free. If your builds become much more complicated, and you have an team of developers working with CI, yes, you will exceed the 120m limit. But if you are running at that scale, a couple of dollars per day for the build process is the least of your running costs.
- skj 10y agoHi all, I'm on the eng team for this service. Happy to answer any questions, and I'll do my best to monitor the threads.
- searchfaster 10y agoThanks skj.. any plans of supporting containers other than docker ?
- skj 10y agoI really like rocket, and technically you can build and push rkt images right now, though it would require a lot of rocket knowhow. Briefly, you can run any container image as a build step with this service, including one that builds rocket containers, and including one that pushes them. But, we're not currently offering any direct support for this ability.
- philips 10y agorkt supports running Docker Images[1], Open Container Initiative (OCI) Images[2], and appc images. So, any container image this system builds will run under rkt. rkt is explicitly designed as a container engine not as a container build system. And this is because build and execute are largely separate concerns. And it leaves the door open for new build systems as demonstrated here by Google. [1] https://coreos.com/rkt/docs/latest/running-docker-images.html https://coreos.com/rkt/docs/latest/running-docker-images.htm... [2] https://github.com/opencontainers/image-spec/issues/126#issuecomment-239014209 https://github.com/opencontainers/image-spec/issues/126#issu...
- deleted 10y ago[deleted]
- puzzle 10y agoIs there any caching involved? In other words, what happens if I build the same source tree twice or if one line changes between one run and the next?
- salakotolu 10y agoWas just reading about this today. Didn't know it was new.
- skj 10y agoIt's been around for a while, just under the radar. The service has backed App Engine Flexible environments for almost a year, it's been GA for months, but this is our first public spotlight.
- thejosh 10y agoDoes anybody know when they are launching in Australia? They have listed 2017 as the launch year, and I'm excited for launch :).
- vgt 10y agoWe happen to have our major annual conference, Google Next, this week. Tune in at [0] Wednesday-Friday :) (Work at Google Cloud) [0] https://cloudnext.withgoogle.com/ https://cloudnext.withgoogle.com/
- nchuhoai 10y agoI'm so surprised that both Docker's as well as Google's solution to this problem involves 0 caching. Much of our container building includes fetching dependencies from various package managers which rarely changes across builds, yet takes up the majority of the build time. Most of the time thats good enough, but for cases where we want to immediately deploy a fix, it can be quite frustrating to having to spend all this time seemingly unncessarily. Edit: Docker 1.13 has the new --cache-from option so it seems relatively straight-forward to do I guess?
- skj 10y agoOne approach to this problem that I like is storing these dependencies in an intermediate image. Then, you can do an incremental build by referencing (rather than rebuilding) this intermediate image, and pay only the cost of downloading the raw data which should be cheap compared to the normal install process.
- gkop 10y agoWhy not leverage Docker's built-in image layer caching?
- kentt 10y agoI think the idea here is that the docker level caching isn't well suited for this. Eg for us we install all relevant gems with `bundle install`. If you update one gem, that the whole layer is rebuilt. Since the whole `bundle install` layer is rebuilt, every gem is fetched again from rubygems. Yarn works the same.
- alpb 10y agoDisclaimer: I work at Google but I do not work on the product, so this is my personal experience. I recently tried out Cloud Container Builder before it went public. I host a lot of small personal Docker projects on my private GitHub repositories and build Docker images for them using CircleCI and push them to GCR (Google Container Registry). Once you figure out how to do this with $your_favorite_CI_tool, it is easy. However it took me about an hour to build this Circle CI and I had to do some tasks like set up Service Accounts, copy keys around etc. Google Container Builder eliminates this completely, you just connect your GitHub account, pick what you want to build (every commit, branch or tag) and specify the image name you want to build and forget about it forever. It runs faster compared to free CI services I've used in the past (probably because image pulls are fast b/c Google Networking, but I'm not sure how much CPU is allocated for builds). Also if you're pushing to GCR, then it'll be faster to push images as well. So overall pull/build/push cycle is just fast. To compare, build for a very simple image I have (base image docker.io/library/python) takes 3m30s on CircleCI vs 1m10s in Google Cloud Container Builder. That's 3x faster.
- ferologics 10y agoWhat size is the project? Trying to guesstimate if it's worth investigating using 3rd party vs. [google] cloud container builder - speed tradeoffs, convenience, friction etc. Cheers.
- alpb 10y agoI am simply building a Python container with about ~15 dependencies installed via pip and some ~50 MB of static assets added to the container. The base image comes from Docker Hub (hosted on S3/AWS CF). If the final destination of your Docker image is GCR, it is really fast. You're basically not waiting for your build box to start up. Based on my experience the build starts almost right away after you push to the source control system, whereas CI systems like TravisCI/CircleCI spend quite some time preparing the build env (i.e. container) and running some preliminary commands, collecting artifacts/logs etc. Google Container Builder is looks like it is designed to build containers (not to run tests) and it does that quite fast.
- 10y ago
- noway421 10y agoJust as with Amazon solution, this seems to not have caching. Baking deps into a separate image is a good solution, but oftentimes you want to rely on docker caching itself. For example docker caches `npm install` quite nicely if you split up the copying of package.json to a separate layer, which allows for a very neat flow where small app diffs result in a fast build, while adding new npm packages results in a slightly longer build. Not possible with GCCB or AWS CodeBuild.. For now, a self-managed Atlassian Bamboo instance (which can and does do caching for each worker) seems to be the best way?
- philliphaydon 10y agoPeople use bamboo? Always been told to avoid it and use Team City. I can't find any good info on bamboo tho.
- howfun 10y agoBy seeing so much confusing Docker articles lately, I have a feeling that the Docker containers are creating much more problems than they are solving.
- splitbrain 10y agoCould I use this to build images and push them to Docker Hub? The auto build feature at Docker Hub is awfully slow and often fails. Could this be a good alternative or am I missing the point of what this is?
- skj 10y agoWe don't currently have a supported way to inject Docker Hub credentials into your build, but it's a feature we have our eye on.
- hiroshi3110 10y agoBefore GCCB become GA, to workaround 120min/day restriction I wrote a command line GCCB like tool. https://github.com/hiroshi/cb https://github.com/hiroshi/cb
- 7ewis 10y agoHaven't tested out GCP yet, but am familiar with AWS and have played around with Docker a little. Am I correct in saying that all this does is effectively a: >docker build >docker push But builds it all server side, then stores it in Google's Container Registry?
- skj 10y agoNo, that's not all it does, though you can certainly use it to do that. This service runs a series of build steps on your source. It's an arbitrary execution engine with container building dressing on top (UI, CLI, etc). One of the build steps might do 'docker build'. In fact, most of the build steps that GCCB has executed do just that! But it's not limited. You can provide container images for custom build steps to do whatever you like in a repeatable, automated fashion.