4 ms·

To understand scenarios under which Docker/Kubernetes/LXC are useful, you need to understand the environment in which Linux containers originated. They were bor

by melted 11y ago

To understand scenarios under which Docker/Kubernetes/LXC are useful, you need to understand the environment in which Linux containers originated. They were born in Google data centers, where engineers may routinely need to reliably deploy tens of thousands of preemptible nodes and tie them together into a service by providing health checks, monitoring, endpoint enumeration, resource limits, isolation, and so on. Docker/Kubernetes let you do exactly that. At Google, though, Borg is run by SREs, and engineers don't have to worry about managing it, so it is quite economical to spin up even single task jobs there. Google also makes deployment much easier by using static linking, and structuring their build system outputs in such a way that they can be either easily deployed by copying over or packaging into an official, versioned deployment package (+providing command line flags through Borg config files). When tasks/jobs go away (get preempted or killed -- servers in this environment usually do not support orderly shutdown), whatever they wrote to local disk gets cleaned out, including binaries and data files. Persistent data is written to persistent, distributed storage backends, where it belongs. As you can see, most parts of this picture map nearly exactly to how Kubernetes/Docker are supposed to be used. Used in this way to manage large deployments, containers provide an unbeatable value proposition.

jacques_chester 11y ago
Put another way: Google has built several generations of internal PaaSes. This unblocks continuous deployment at the final step, so latency from idea to production falls from years/months/weeks to hours. Or minutes. Docker's had an interesting life: they built a PaaS, discarded the PaaS, now they're building a PaaS. Because that's what most developers actually need for their daily lives. It turns out that tinkering with V8s is a lot of fun for a lot of people, but most drivers just want to know how to turn on the car and have the same basic interface work for any workload: wheel, accelerator, brake. Disclaimer: I work for Pivotal, which is the majority donor of engineering effort to Cloud Foundry, a PaaS inspired in part by Google's experiences.
- melted 11y ago
  I wouldn't quite go as far as to call Borg a PaaS. It's at a somewhat lower level, closer to IaaS. You get a known quantity from it in the form of a stable, tuned, stripped down underlying Linux image, plus a relatively small set of services, but you can deploy pretty much whatever the hell you want without a lot of constraints a "true" PaaS would force you to accept.
  - jacques_chester 11y ago
    My understanding is that BOSH is the closest to this from the Cloud Foundry ecosystem.
    - deleted 11y ago
      [deleted]
- michaelbuckbee 11y ago
  This mirrors my understanding: that Docker is awesome if you want to build your own PAAS. Otherwise use CF, Heroku or whatever.
markhahn 11y ago
well, except that openvz is an example of linux containers for which Google can't take credit.
echochar 11y ago
"Google makes deployment much easier by using static linking..." Unless you only are referring to only the Go compiler, can you provide a cite for this statement? I also use static linking as much as I can. Sometimes it requires quite a bit of work due to how open source developers structure their build systems. I wonder if some folks at Google have also spent significant amounts of time unravelling idiosyncratic build systems found in open source projects to make static, portable binaries. I often see commenters on HN and elsewhere making commments against static linking. It would be useful to be able to cite to Google's internal practices; these commenters also seem to attribute superior knowhow to Google staff.
- crb 11y ago
  http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf http://static.googleusercontent.com/media/research.google.co... "Borg programs are statically linked to reduce dependencies on their runtime environment, and structured as packages of binaries and data files"
- benley 11y ago
  They actually don't completely statically link everything - there is a small set of base libraries (glibc and I forget what else), and generally all compiled dependencies other than those from the base runtime are linked into a binary. Google released a bunch of their main build system recently, so you can see various hints about their internal practices there, starting with the documentation: http://bazel.io/docs/be/c-cpp.html#cc_binary.linkstatic http://bazel.io/docs/be/c-cpp.html#cc_binary.linkstatic The OSS bazel defaults to linkstatic=0, but I think internally they use linkstatic=1 there. "mostly static" mode. That's just C/C++ of course, but a similar approach applies to Python, where there are zipped single-file executables that contain all of a python app's dependencies except for python itself and things like libc6. There's a primitive form of that in bazel at https://github.com/bazelbuild/bazel/blob/master/tools/build_rules/py_rules.bzl https://github.com/bazelbuild/bazel/blob/master/tools/build_.... Twitter's "Pants" build system is ideologically descended from Bazel, and along with it came the Pex python executable format: https://github.com/pantsbuild/pex https://github.com/pantsbuild/pex Just the fact that Go does mostly-static linking by default is a decent hint in this direction, considering where Go originated. In the end, neither approach is ideal for all scenarios. General purpose linux distributions have various very good reasons to prefer dynamic linking wherever possible. Cluster operators and anybody distributing binaries for multiple environments have lots of reasons to prefer static linking. It's not a decision to be made in a vacuum.
moron4hire 11y ago
I have zero interest in understanding how Google or Facebook or Microsoft or Apple or IBM or even Twitter does anything right now. They are big. I am not. These companies have deep-pockets of capital to spend on such luxuries as individual contributors who can make it their jobs to deep-dive into single, esoteric, poorly specified tools. I do not. Late September last year, Twitter laid off 336 people. In 15 years, I've never worked in a company that had more than even 150. A company the size of Google could probably hire and fire more people in a single year than I've ever been in professional contact. I cannot. So how they operate, from soup to nuts, probably has zero bearing on how I should operate.
digi_owl 11y ago
I really can't shake the feeling that too much of the Linux community, that has final say about the user space, is off doing some kind of architecture astronaut impression in an attempt at bootstrapping the next Google.

by melted 11y ago

jacques_chester 11y ago

Put another way: Google has built several generations of internal PaaSes. This unblocks continuous deployment at the final step, so latency from idea to production falls from years/months/weeks to hours. Or minutes. Docker's had an interesting life: they built a PaaS, discarded the PaaS, now they're building a PaaS. Because that's what most developers actually need for their daily lives. It turns out that tinkering with V8s is a lot of fun for a lot of people, but most drivers just want to know how to turn on the car and have the same basic interface work for any workload: wheel, accelerator, brake. Disclaimer: I work for Pivotal, which is the majority donor of engineering effort to Cloud Foundry, a PaaS inspired in part by Google's experiences.

melted 11y ago
I wouldn't quite go as far as to call Borg a PaaS. It's at a somewhat lower level, closer to IaaS. You get a known quantity from it in the form of a stable, tuned, stripped down underlying Linux image, plus a relatively small set of services, but you can deploy pretty much whatever the hell you want without a lot of constraints a "true" PaaS would force you to accept.
- jacques_chester 11y ago
  My understanding is that BOSH is the closest to this from the Cloud Foundry ecosystem.
  - deleted 11y ago
    [deleted]
michaelbuckbee 11y ago
This mirrors my understanding: that Docker is awesome if you want to build your own PAAS. Otherwise use CF, Heroku or whatever.

markhahn 11y ago

well, except that openvz is an example of linux containers for which Google can't take credit.

echochar 11y ago

"Google makes deployment much easier by using static linking..." Unless you only are referring to only the Go compiler, can you provide a cite for this statement? I also use static linking as much as I can. Sometimes it requires quite a bit of work due to how open source developers structure their build systems. I wonder if some folks at Google have also spent significant amounts of time unravelling idiosyncratic build systems found in open source projects to make static, portable binaries. I often see commenters on HN and elsewhere making commments against static linking. It would be useful to be able to cite to Google's internal practices; these commenters also seem to attribute superior knowhow to Google staff.

crb 11y ago
http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf http://static.googleusercontent.com/media/research.google.co... "Borg programs are statically linked to reduce dependencies on their runtime environment, and structured as packages of binaries and data files"
benley 11y ago
They actually don't completely statically link everything - there is a small set of base libraries (glibc and I forget what else), and generally all compiled dependencies other than those from the base runtime are linked into a binary. Google released a bunch of their main build system recently, so you can see various hints about their internal practices there, starting with the documentation: http://bazel.io/docs/be/c-cpp.html#cc_binary.linkstatic http://bazel.io/docs/be/c-cpp.html#cc_binary.linkstatic The OSS bazel defaults to linkstatic=0, but I think internally they use linkstatic=1 there. "mostly static" mode. That's just C/C++ of course, but a similar approach applies to Python, where there are zipped single-file executables that contain all of a python app's dependencies except for python itself and things like libc6. There's a primitive form of that in bazel at https://github.com/bazelbuild/bazel/blob/master/tools/build_rules/py_rules.bzl https://github.com/bazelbuild/bazel/blob/master/tools/build_.... Twitter's "Pants" build system is ideologically descended from Bazel, and along with it came the Pex python executable format: https://github.com/pantsbuild/pex https://github.com/pantsbuild/pex Just the fact that Go does mostly-static linking by default is a decent hint in this direction, considering where Go originated. In the end, neither approach is ideal for all scenarios. General purpose linux distributions have various very good reasons to prefer dynamic linking wherever possible. Cluster operators and anybody distributing binaries for multiple environments have lots of reasons to prefer static linking. It's not a decision to be made in a vacuum.

moron4hire 11y ago

I have zero interest in understanding how Google or Facebook or Microsoft or Apple or IBM or even Twitter does anything right now. They are big. I am not. These companies have deep-pockets of capital to spend on such luxuries as individual contributors who can make it their jobs to deep-dive into single, esoteric, poorly specified tools. I do not. Late September last year, Twitter laid off 336 people. In 15 years, I've never worked in a company that had more than even 150. A company the size of Google could probably hire and fire more people in a single year than I've ever been in professional contact. I cannot. So how they operate, from soup to nuts, probably has zero bearing on how I should operate.

digi_owl 11y ago

I really can't shake the feeling that too much of the Linux community, that has final say about the user space, is off doing some kind of architecture astronaut impression in an attempt at bootstrapping the next Google.