5 ms·

I appreciate the sentiment behind linkerd et al but I think we are really over-complicating things and tying ourselves into knots by going head-first into this

I appreciate the sentiment behind linkerd et al but I think we are really over-complicating things and tying ourselves into knots by going head-first into this world. I can't help but think we're going to regret it in a few years. The complexity introduced not only by Docker, but also k8s and all of the other things competing for a spot in the middle of your cluster, is immense and widely underestimated. Your applications need to be aware and adapted to the k8s model of automatic pod termination, no hard storage, etc. Google has had over 10 years with the best computer scientists money can buy to perfect their stack. Most companies don't. For most companies, k8s/linkerd/Docker deployments are a case of overengineering run frighteningly amok. There are many-a-snafu lying down this path. I'm very worried about how casually everyone is willing to jump aboard. One particular worry: most Dockerfiles are built on opaque images registered with an image registry, and many of the publishers are not well-known. If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance). These are frequently distributed by complete strangers. Building "from scratch" off the base image distributed by the distro itself is considered "the old way", at least among those I've worked with. My understanding is that most public image repositories are unmoderated. The moment a widely-used image gets pwned, there is going to be panic. Source: been converting entire company to Docker/k8s

kinghajj 10y ago
> Building "from scratch" off the base image distributed by the distro itself is considered "the old way" Then I don't want to be new. I started using a few of the Docker Hub images (mostly `openjdk`), but I've transitioned completely to building all runtime images off of `debian:jessie`. And even that image I may want to remove eventually, just so that our CI system isn't affected by Docker Hub outages.
rubiquity 10y ago
Yes, it's insane. I can only attribute it to that I don't work on problem of any scale (amount of apps, servers, deployments per day, etc.) and all of these people do. Who really wants to spend their time and effort running a Kubernetes master and nodes, 3+ Consul/etcd nodes, and now some fancy load balancers? The ratio of companies that would benefit from that to the number of companies working on products in the space is way out of wack.
- SEJeff 10y ago
  Well you can simply use GKE and that is all managed for you. Seemed to work quite well for Pokemon Go as they scaled out massively.
  - cookiecaper 10y ago
    "Have we made an unnecessarily contorted solution that you don't need to solve any of your real-world problems? Don't worry, you can still claim to be cool by paying us money." What a solution!
- cookiecaper 10y ago
  I would not take the claims of scale at face value. People make this claim to boost their own egos, and the reality is that many of the people who do this would be more than fine on a handful of servers. They just want to feel important. My frank take is that k8s was developed at Google by the seat of their pants (and yes, I know it's their third-gen orchestration platform), they talked about it with outsiders here and there, there was demand for it because there's demand for anything and everything with Google's name attached to it, regardless of applicability or propriety or any good sense (case in point: companies pretending to be smart by "integrating" TensorFlow), and they released k8s in response to the clamor for the mysterious Google-backed orchestration platform, much to the detriment of non-Googlers everywhere. Maybe it's a coordinated plan to harm young startups that may challenge them. Or, alternate theory: perhaps it's a coordinated plan to get people to buy far more cloud instances than they actually need, since k8s literally cannot run on a single instance; to run it locally, you have to run virtual machines. I understand that k8s has improved and that many people use it (myself among them), but it maintains that vibe of an internal tool; something so convoluted and complex and inside baseball that you'd only voluntarily undertake it if someone said your paycheck was at risk.
  - TheIronYuppie 10y ago
    Just FYI, over 50% of contributions come from !Google, and that's part of the reason that so many people (CoreOS, Redhat, Apprenda, Deis, thousands of internal folks) have rebased on it. I saw some of your other comments - I'd love to understand better what we could do to either a) make it feel like not an internal tool and b) if you didn't use Kubernetes or containers, how you'd prefer to run distributed workloads. TBC, I totally agree with you - if you're running 1-2 nodes, and don't care about downtime, you should NEVER use Kubernetes or any orchestrator. Once you get to 3+, however, I can't imagine using anything else. Disclosure: I work at Google on Kubernetes
mwpmaybe 10y ago
> If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance) I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images[0]. Similar official repositories exist for many server daemons, language runtimes, and command-line tools[1]. > My understanding is that most public image repositories are unmoderated. To the best of my understanding, the official images are maintained by Docker folks, or people associated with the upstream projects, or some mixture of both. > most Dockerfiles are built on opaque images registered with an image registry Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library. So yes, while it's true there are a lot of public images out there with questionable provenance, there's usually a solid, well-supported foundation to base your own images on. 0. https://hub.docker.com/_/ruby/ https://hub.docker.com/_/ruby/ 1. https://github.com/docker-library/official-images/tree/master/library https://github.com/docker-library/official-images/tree/maste...
- cookiecaper 10y ago
  >I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images. Ah, I didn't realize there was a blessed subset of repositories for which the Docker team has independently verified the identity. Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it? Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)? While it is nice that there are a few known-good repositories for projects popular enough to secure the approval of Docker Inc., the habit of casually importing images is still rampant, afaict. >Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library. Well, here's an example [0] of a base Dockerfile. I had to go 3 levels up from the link on the Docker Hub page to get to this one because most of the images are FROM:something. After chasing the tree up, this is a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?) and b) a good example of the "brilliance" of the Dockerfile format. That massive set of line continuations is necessary because if you implement the Dockerfile "naively", by making each modification a "RUN" command, you end up with an image that is very bloated due to the way Docker's caching layer functions. Bonus feature: if any of those commands is to return a non-0 exit code, the line will abort. Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it, so if you're not picking your image base carefully, there may not be a way to verify its contents (short of introspecting the image). [0] https://github.com/docker-library/ruby/blob/752c5f7cf44870ceae77134b346d20093053c370/2.4/Dockerfile https://github.com/docker-library/ruby/blob/752c5f7cf44870ce...
lobster_johnson 10y ago
It's perfectly normal to use base images and build your own. We do this for most things. One simple reason is control: You want to be able to follow CVEs and upgrade your images in line with patches. If every image branches off some random Docker Hub image, the only way to do that would be some kind of image vulnerability scanner, which I believe exists, but wouldn't help you actually patch anything easily. Public images are still super nice to bootstrap things. You can start with a public image, then move to a custom-built one. Lastly: I wouldn't conflate the Docker image challenge with the operational complexity of using Kubernetes and microservices. They are completely orthogonal in my mind. Docker has all sorts of issues that we need to solve sooner rather than later; Kubernetes' design, on the other hand, is sound.
- cookiecaper 10y ago
  I recognize that Docker and k8s are distinct, and I think rkt looks much smarter than Docker. Now that k8s has released a stable version capable of running non-Docker containers (as of last week), it's fair to separate them. However, I'm still struggling to see the real value proposition offered by either. The story on Docker or Docker-style containers without orchestration is that they're supposed to be easy to ensure you get an identical environment everywhere and more fully utilize your hardware resources. While that does seem to happen if you incorporate docker-compose or some other metadata about what ports to map and volumes to mount, the pains associated with using Docker have far, far outweighed the one-time cost of configuring a new application or getting a new build to work for me. And we already had pretty good tooling for this pre-container, with things like Ansible. It wasn't exact, but it was much more comfortable.
  - lobster_johnson 10y ago
    Being a Puppet user (until we've finally retired that nightmare), I couldn't be happier with this migration to containers. Docker itself is problematic for all sorts of reasons, but it at least does the most important thing, which is to run pre-packaged code in isolated containers. With Puppet, as with Ansible, you just can't guarantee an isolated environment, nor is the environment easily "replicable", in the sense that if you need more instances of the exact same thing (e.g. more data processing workers), there's nothing in the system built to treat things as N-of-anything. The only unit of provisioning is the VM. Puppet/Salt/Ansible are highly static systems. Meanwhile, Kubernetes smoothes over a lot of the rough parts of Docker: Ports, IPs, DNS/discovery, monitoring and volume management are all neatly captured by Kubernetes, where in Docker Engine they're just low-level primitives you need to manage manually. It also handles exec'ing into an existing container. The pod as the unit of orchestration is also much nicer than Docker's single-process containers. For me, the value proposition is the cloud. When I boot up N nodes, I know that Kubernetes will optimally pack them with my container, and if I boot up N+1 nodes, I know that the containers will spread to the new node, and back down again if I go down to N (either intentionally or accidentally). Orchestration, scheduling, autoscaling etc. take care of this "cloud of containers", if you will, and since every container is an isolated, identical, versioned, stateless (for the most part!), fully redundant thing, I rest assured that what is running is what I deployed. Then there are all the other nice things, like rolling deploys, cordoning, self-healing, job scheduling, secret management, being able to boot up a parallel, different version of a stack without clobbering everything else, etc. We run production and staging in the same Kubernetes cluster (different namespaces), and we can use things like resource requests/limits and node affinities to ensure that they're scheduled differently, but still using the same underlying hardware resources. Docker, to me, isn't particularly interesting without Kubernetes, although it has its uses. It's terrific to be able to "boot" a temporary Ubuntu or Elasticsearch or whatever in order to do something that I can just tear down afterwards. Build/test containers are great, both locally and in a CI environment. What pains are you referring to, specifically?

kinghajj 10y ago

> Building "from scratch" off the base image distributed by the distro itself is considered "the old way" Then I don't want to be new. I started using a few of the Docker Hub images (mostly `openjdk`), but I've transitioned completely to building all runtime images off of `debian:jessie`. And even that image I may want to remove eventually, just so that our CI system isn't affected by Docker Hub outages.

rubiquity 10y ago

Yes, it's insane. I can only attribute it to that I don't work on problem of any scale (amount of apps, servers, deployments per day, etc.) and all of these people do. Who really wants to spend their time and effort running a Kubernetes master and nodes, 3+ Consul/etcd nodes, and now some fancy load balancers? The ratio of companies that would benefit from that to the number of companies working on products in the space is way out of wack.

SEJeff 10y ago
Well you can simply use GKE and that is all managed for you. Seemed to work quite well for Pokemon Go as they scaled out massively.
- cookiecaper 10y ago
  "Have we made an unnecessarily contorted solution that you don't need to solve any of your real-world problems? Don't worry, you can still claim to be cool by paying us money." What a solution!
cookiecaper 10y ago
I would not take the claims of scale at face value. People make this claim to boost their own egos, and the reality is that many of the people who do this would be more than fine on a handful of servers. They just want to feel important. My frank take is that k8s was developed at Google by the seat of their pants (and yes, I know it's their third-gen orchestration platform), they talked about it with outsiders here and there, there was demand for it because there's demand for anything and everything with Google's name attached to it, regardless of applicability or propriety or any good sense (case in point: companies pretending to be smart by "integrating" TensorFlow), and they released k8s in response to the clamor for the mysterious Google-backed orchestration platform, much to the detriment of non-Googlers everywhere. Maybe it's a coordinated plan to harm young startups that may challenge them. Or, alternate theory: perhaps it's a coordinated plan to get people to buy far more cloud instances than they actually need, since k8s literally cannot run on a single instance; to run it locally, you have to run virtual machines. I understand that k8s has improved and that many people use it (myself among them), but it maintains that vibe of an internal tool; something so convoluted and complex and inside baseball that you'd only voluntarily undertake it if someone said your paycheck was at risk.
- TheIronYuppie 10y ago
  Just FYI, over 50% of contributions come from !Google, and that's part of the reason that so many people (CoreOS, Redhat, Apprenda, Deis, thousands of internal folks) have rebased on it. I saw some of your other comments - I'd love to understand better what we could do to either a) make it feel like not an internal tool and b) if you didn't use Kubernetes or containers, how you'd prefer to run distributed workloads. TBC, I totally agree with you - if you're running 1-2 nodes, and don't care about downtime, you should NEVER use Kubernetes or any orchestrator. Once you get to 3+, however, I can't imagine using anything else. Disclosure: I work at Google on Kubernetes

mwpmaybe 10y ago

> If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance) I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images[0]. Similar official repositories exist for many server daemons, language runtimes, and command-line tools[1]. > My understanding is that most public image repositories are unmoderated. To the best of my understanding, the official images are maintained by Docker folks, or people associated with the upstream projects, or some mixture of both. > most Dockerfiles are built on opaque images registered with an image registry Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library. So yes, while it's true there are a lot of public images out there with questionable provenance, there's usually a solid, well-supported foundation to base your own images on. 0. https://hub.docker.com/_/ruby/ https://hub.docker.com/_/ruby/ 1. https://github.com/docker-library/official-images/tree/master/library https://github.com/docker-library/official-images/tree/maste...

cookiecaper 10y ago
>I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images. Ah, I didn't realize there was a blessed subset of repositories for which the Docker team has independently verified the identity. Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it? Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)? While it is nice that there are a few known-good repositories for projects popular enough to secure the approval of Docker Inc., the habit of casually importing images is still rampant, afaict. >Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library. Well, here's an example [0] of a base Dockerfile. I had to go 3 levels up from the link on the Docker Hub page to get to this one because most of the images are FROM:something. After chasing the tree up, this is a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?) and b) a good example of the "brilliance" of the Dockerfile format. That massive set of line continuations is necessary because if you implement the Dockerfile "naively", by making each modification a "RUN" command, you end up with an image that is very bloated due to the way Docker's caching layer functions. Bonus feature: if any of those commands is to return a non-0 exit code, the line will abort. Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it, so if you're not picking your image base carefully, there may not be a way to verify its contents (short of introspecting the image). [0] https://github.com/docker-library/ruby/blob/752c5f7cf44870ceae77134b346d20093053c370/2.4/Dockerfile https://github.com/docker-library/ruby/blob/752c5f7cf44870ce...

lobster_johnson 10y ago

It's perfectly normal to use base images and build your own. We do this for most things. One simple reason is control: You want to be able to follow CVEs and upgrade your images in line with patches. If every image branches off some random Docker Hub image, the only way to do that would be some kind of image vulnerability scanner, which I believe exists, but wouldn't help you actually patch anything easily. Public images are still super nice to bootstrap things. You can start with a public image, then move to a custom-built one. Lastly: I wouldn't conflate the Docker image challenge with the operational complexity of using Kubernetes and microservices. They are completely orthogonal in my mind. Docker has all sorts of issues that we need to solve sooner rather than later; Kubernetes' design, on the other hand, is sound.

cookiecaper 10y ago
I recognize that Docker and k8s are distinct, and I think rkt looks much smarter than Docker. Now that k8s has released a stable version capable of running non-Docker containers (as of last week), it's fair to separate them. However, I'm still struggling to see the real value proposition offered by either. The story on Docker or Docker-style containers without orchestration is that they're supposed to be easy to ensure you get an identical environment everywhere and more fully utilize your hardware resources. While that does seem to happen if you incorporate docker-compose or some other metadata about what ports to map and volumes to mount, the pains associated with using Docker have far, far outweighed the one-time cost of configuring a new application or getting a new build to work for me. And we already had pretty good tooling for this pre-container, with things like Ansible. It wasn't exact, but it was much more comfortable.
- lobster_johnson 10y ago
  Being a Puppet user (until we've finally retired that nightmare), I couldn't be happier with this migration to containers. Docker itself is problematic for all sorts of reasons, but it at least does the most important thing, which is to run pre-packaged code in isolated containers. With Puppet, as with Ansible, you just can't guarantee an isolated environment, nor is the environment easily "replicable", in the sense that if you need more instances of the exact same thing (e.g. more data processing workers), there's nothing in the system built to treat things as N-of-anything. The only unit of provisioning is the VM. Puppet/Salt/Ansible are highly static systems. Meanwhile, Kubernetes smoothes over a lot of the rough parts of Docker: Ports, IPs, DNS/discovery, monitoring and volume management are all neatly captured by Kubernetes, where in Docker Engine they're just low-level primitives you need to manage manually. It also handles exec'ing into an existing container. The pod as the unit of orchestration is also much nicer than Docker's single-process containers. For me, the value proposition is the cloud. When I boot up N nodes, I know that Kubernetes will optimally pack them with my container, and if I boot up N+1 nodes, I know that the containers will spread to the new node, and back down again if I go down to N (either intentionally or accidentally). Orchestration, scheduling, autoscaling etc. take care of this "cloud of containers", if you will, and since every container is an isolated, identical, versioned, stateless (for the most part!), fully redundant thing, I rest assured that what is running is what I deployed. Then there are all the other nice things, like rolling deploys, cordoning, self-healing, job scheduling, secret management, being able to boot up a parallel, different version of a stack without clobbering everything else, etc. We run production and staging in the same Kubernetes cluster (different namespaces), and we can use things like resource requests/limits and node affinities to ensure that they're scheduled differently, but still using the same underlying hardware resources. Docker, to me, isn't particularly interesting without Kubernetes, although it has its uses. It's terrific to be able to "boot" a temporary Ubuntu or Elasticsearch or whatever in order to do something that I can just tear down afterwards. Build/test containers are great, both locally and in a CI environment. What pains are you referring to, specifically?