7 ms·

A Year of Helping Customers Build Production-Ready Kubernetes Infrastructure

82 points by migueloller 7y ago

rawoke083600 7y ago
How many of your customers actually NEEDED kubernetes ?
- k__ 7y ago
  That was my first tought. Microsoft, Amazon, Google, IBM, Alibaba and maybe the rest of the small cloud providers.
- antics 7y ago
  Ha! I co-lead these efforts, and what I will say is that we try not to "lead the witness" here. :) So customers choose Kubernetes, and then we help them, but we don't help them to choose Kubernetes, so to speak.
- MadWombat 7y ago
  shrug If you have a bunch of services that you need to reliably run in containers you are going to need an orchestration of some sort. You can attempt to reinvent the wheel by doing your own or you can use an existing solution. If you are not in the mood for wheel reinventing, your choice becomes somewhat limited. You have Kubernetes, Docker Swarm, Mesos, Nomad, a few platform specific solutions like Elastic Beanstalk and maybe some more obscure candidates. If you don't like vendor lock-in, don't like living on the cutting edge and don't want to write too much code, your choices are down to Kubernetes, Swarm and Mesos.
  - antics 7y ago
    [Pulumi kube co-lead here] I agree with most of this, but would go further. My experience (here and at Heptio) is that no one _chooses_ Kubernetes. They choose Docker, and then they (i.e., the infra team) effectively have to choose ECS or Kubernetes. In contrast, what most app developers want is a better `docker run`. I don't know how controversial that is, so I should say this is my conclusion from ~50 intensive customer qualification conversations in my 1.5 years at Pulumi and 1 year at Heptio. In each of these conversations I ask why the customer is moving to Kubernetes. In mid- or large-sized companies (i.e., 500+ employees) the answer is completely unwavering: Docker makes it easy to ship software without worrying too much about the host, and once you have that, having a way to schedule containers to run _somewhere_ is very useful. And that's effectively how the decision is made IME.
    - 7532yahoogmail 7y ago
      I am very much hunting for a cli, automatable method to do: - app installs - orchestration of same across multiple boxes in a cluster - cluster monitoring: detect when a task or box goes down running the service(s) elsewhere in the cluster It seems like kubernetes is incomplete with respect to these needs. Kubernetes depends on docker but neither does orchestration. In the context of this HN post what do I need. I am doing golang on 100pct Linux. Other than toolng I need for app installs the boxes themselves are provisioned by others. Here I am focused on app installs only. Any help much, much appreciated.
      - dankohn1 7y ago
        Take a look at Kubernetes, Helm, and the Cluster API.
        Terretta 7y ago
        Can’t tell if grandparent post wants to manage apps on a cluster (see below) or apps that are a cluster (see https://cluster-api.sigs.k8s.io/ https://cluster-api.sigs.k8s.io/) If it’s letting a dev manage apps on a cluster someone else let them put apps on, while staying in dev mode, maybe devs could appreciate the cluster provisioning SREs enabling app devs with so-called GitOps or AutoDevOps approaches, prescriptivist ways of leveraging Docker, K8s, and Helm, such as: https://www.weave.works/blog/what-is-gitops-really https://www.weave.works/blog/what-is-gitops-really https://docs.gitlab.com/ee/topics/autodevops/index.html https://docs.gitlab.com/ee/topics/autodevops/index.html
      - MadWombat 7y ago
        Kubernetes is pretty feature complete, but its focus is on containers, not naked apps. If you are willing to pack your apps into docker, kubernetes will solve the orchestration and monitoring problems. If you are not, you might want to look for something else. In particular, take a look at Nomad (https://www.nomadproject.io/ https://www.nomadproject.io/). I am not affiliated with the project in any way and at my workplace we went the kube route, but Nomad does provide an ability to orchestrate arbitrary processes across multiple hosts.
- lukehoban 7y ago
  We've had customers and users of Pulumi at many different sizes and stages looking at or migrating to Kubernetes - and I'd say there are a few consistent reasons we hear for making the choice to adopt Kubernetes. First, Kubernetes offers a rich, standards-based API for compute. For teams who want to deliver containerized applications, using Kubernetes allows targeting a standards API instead of a cloud-provider specific API. This up-levels the same benefit folks get with cloud IaaS solutions where they can write standard Linux applications. Now the standardized and "portable" part of their applications extends to some key parts of the compute infrastructure as well. Second, we see a ton of Cloud + Kubernetes - teams who are still taking advantage of a lot of managed services and operational benefits from AWS/Azure/GCP/etc., but want to combine that with the container orchestration and rich API of Kubernetes. This means leaving stateful workloads in managed services like Aurora or Cloud SQL, delegating auth to managed IAM services, etc. Pulumi happens to be a really nice fit for doing these together. And lastly, there's just a really active ecosystem that's building a lot of value around Kubernetes. Some of those pieces aren't yet fully mature, but directionally many teams are betting on this being where the next wave of mature cloud platform services will become available.
- reilly3000 7y ago
  We are going w/ Fargate for now, but the similarity in syntax may help us make the leap to k8s when we're ready.
antics 7y ago
Oh wow! I co-lead these efforts at Pulumi. I don't think we expected this to hit HN! :) Happy to answer questions about any of this. Parenthetical, one thing that didn't make it into this post that I'm particularly optimistic about is the `watch` command, which will continuously watch your code and deploy it to (say) Kubernetes. It looks like this[1]. This is very handy when you're trying to build a thing that has to integrate with a bunch of other services, but there are too many to develop locally on minikube. It basically lets you see instantly if something crashes, or whatever. In general I'm very optimistic about that sort of experience—another product in that space is Tilt[4] which I am personally a big fan of. Another thing that gets kind of a passing mention that is cooler than it looks here is our streaming `query` stuff. By exposing Kubernetes `watch`es as streaming queries, it becomes very easy to write very cool apps like this one[2] , which helps you debug services. You can actually see a bug in the Deployment controller there—we create two `Deployment`s with the same labels, and deleting one causes the controller to delete _all_ the pods. This would generate a transient 500 for any request hitting that service. I think there are probably lots of bugs in Kubernetes like this, but people don't notice them because this level of visibility previously was quite hard. :) To get an idea of how hard that sort of app is to write, you can see the code we wrote for the prototype of this SDK here[3]. [1]: https://twitter.com/funcofjoe/status/1195411343966236677?s=21 https://twitter.com/funcofjoe/status/1195411343966236677?s=2... [2]: https://drive.google.com/open?id=1Q2f8CZhswMvTliaLlHZK4vItdMVIXzOD https://drive.google.com/open?id=1Q2f8CZhswMvTliaLlHZK4vItdM... [3]: https://github.com/carbonql/carbon-ts/blob/master/examples/unix/serviceMonitor/serviceMonitor.ts https://github.com/carbonql/carbon-ts/blob/master/examples/u... [4]: https://tilt.dev/ https://tilt.dev/
- migueloller 7y ago
  I’ve been very impressed with what you guys are doing. We’re extremely excited about moving from our collection of YAML, bash, and Helm templates to Pulumi. The big question mark we had was what local development would look like. Tilt and Skaffold were the two big candidates we looked at for sane local development with Kubernetes. Looks like Pulumi’s new `watch` could give us what we need? Are there any plans to have something like rsync support to enable things like hot-reloading apps? Both Tilt and Skaffold have examples showing this capability.
bsingh4 7y ago
For AWS users AWS CDK is a far better take on infrastructure as code - strongly typed TypeScript resources and higher level constructs that hide the underlying details of load balancers, IAM, etc. Pulumi was not as useful the last time I tried it - most resources were string key-value pairs with no typing or code completion. You're still referencing documentation to do anything and defining tons of low level resources to accomplish basic tasks. Might as well be writing raw Terraform or CloudFormation configs at that point, as they have more documentation available.
- reilly3000 7y ago
  Something was wrong w/ your setup I think. All of their code has Typescript annotations, and their autocomplete w/ intellisense is remarkably useful.
  - bsingh4 7y ago
    Here are two simple database definitions: Database in Pulumi: const db = new gcp.sql.DatabaseInstance('app-db', { databaseVersion: 'POSTGRES_11', region: 'us-central1', settings: { tier: 'db-f1-micro', diskType: 'PD_HDD', }, }); Database in AWS CDK: const db = new rds.DatabaseInstance(this, 'AppDb', { engine: rds.DatabaseInstanceEngine.POSTGRES, engineVersion: '11.5', instanceClass: ec2.InstanceType.of( ec2.InstanceClass.STANDARD5, ec2.InstanceSize.XLARGE ), storageType: rds.StorageType.GP2 }) Not to mention they have higher level constructs like DatabaseCluster that will handle the underlying details of replicas and create the lower level resources for you.
    - bsingh4 7y ago
      I don't think large blocks of comments qualify as strong typing: https://github.com/pulumi/pulumi-gcp/blob/76df2e85214ad134652327fa78f2f29f50502eae/sdk/nodejs/sql/databaseInstance.ts https://github.com/pulumi/pulumi-gcp/blob/76df2e85214ad13465...
      - antics 7y ago
reilly3000 7y ago
One thing I'm murky on with Pulumi is around how to execute proper blue/green or canary deployments with a GitOps approach. Any insights?
cbushko 7y ago
Any tool that helps build infrastructure as code is great news. I hear good things about pulumi. In my case I have been moving our current infrastructure from AWS to GCP. After much evaluation and a small stint with Helm, we decided to go 100% terraform. There is more to running a product than just deploying to kubernetes and terraform providers have given us the flexibility to chain all these resources together. For example, I can create a GCP service account (google cloud provider), save its credentials as a kubernetes secret (kubernetes provider), register a name in route 53 (aws provider) and deploy our product in kubernetes (kubernetes provider). Having it all in one tool that has state is really powerful and more importantly follow the same patterns. Obviously, I am a big fan of terraform
- jen20 7y ago
  Everything you describe is also straightforward with Pulumi using `pulumi-gcp` [1], `pulumi-aws` [2] and friends - Kubernetes is one of many libraries. On top of this, you can build actual abstractions thanks to the configuration being written using a general purpose programming language - I don't think there is a first-party set of abstractions for GKE yet but there is for EKS [3]. Many of the most compelling examples deploy entire infrastructure stacks from "empty account" to multi-cloud Kubernetes. Disclaimer: I have worked extensively on Terraform (both core and providers) at and after HC, and have also contributed to Pulumi since it was open sourced. Edited for formatting. [1]: https://github.com/pulumi/pulumi-gcp https://github.com/pulumi/pulumi-gcp [2]: https://github.com/pulumi/pulumi-aws https://github.com/pulumi/pulumi-aws [3]: https://github.com/pulumi/pulumi-eks https://github.com/pulumi/pulumi-eks
houqp 7y ago
Great write up, the experience of managing k8s end to end with pulumi is so much better compared to terrafom.

82 points by migueloller 7y ago

rawoke083600 7y ago

How many of your customers actually NEEDED kubernetes ?

k__ 7y ago
That was my first tought. Microsoft, Amazon, Google, IBM, Alibaba and maybe the rest of the small cloud providers.
antics 7y ago
Ha! I co-lead these efforts, and what I will say is that we try not to "lead the witness" here. :) So customers choose Kubernetes, and then we help them, but we don't help them to choose Kubernetes, so to speak.
MadWombat 7y ago
shrug If you have a bunch of services that you need to reliably run in containers you are going to need an orchestration of some sort. You can attempt to reinvent the wheel by doing your own or you can use an existing solution. If you are not in the mood for wheel reinventing, your choice becomes somewhat limited. You have Kubernetes, Docker Swarm, Mesos, Nomad, a few platform specific solutions like Elastic Beanstalk and maybe some more obscure candidates. If you don't like vendor lock-in, don't like living on the cutting edge and don't want to write too much code, your choices are down to Kubernetes, Swarm and Mesos.
- antics 7y ago
  [Pulumi kube co-lead here] I agree with most of this, but would go further. My experience (here and at Heptio) is that no one _chooses_ Kubernetes. They choose Docker, and then they (i.e., the infra team) effectively have to choose ECS or Kubernetes. In contrast, what most app developers want is a better `docker run`. I don't know how controversial that is, so I should say this is my conclusion from ~50 intensive customer qualification conversations in my 1.5 years at Pulumi and 1 year at Heptio. In each of these conversations I ask why the customer is moving to Kubernetes. In mid- or large-sized companies (i.e., 500+ employees) the answer is completely unwavering: Docker makes it easy to ship software without worrying too much about the host, and once you have that, having a way to schedule containers to run _somewhere_ is very useful. And that's effectively how the decision is made IME.
  - 7532yahoogmail 7y ago
    I am very much hunting for a cli, automatable method to do: - app installs - orchestration of same across multiple boxes in a cluster - cluster monitoring: detect when a task or box goes down running the service(s) elsewhere in the cluster It seems like kubernetes is incomplete with respect to these needs. Kubernetes depends on docker but neither does orchestration. In the context of this HN post what do I need. I am doing golang on 100pct Linux. Other than toolng I need for app installs the boxes themselves are provisioned by others. Here I am focused on app installs only. Any help much, much appreciated.
    - dankohn1 7y ago
      Take a look at Kubernetes, Helm, and the Cluster API.
      - Terretta 7y ago
        Can’t tell if grandparent post wants to manage apps on a cluster (see below) or apps that are a cluster (see https://cluster-api.sigs.k8s.io/ https://cluster-api.sigs.k8s.io/) If it’s letting a dev manage apps on a cluster someone else let them put apps on, while staying in dev mode, maybe devs could appreciate the cluster provisioning SREs enabling app devs with so-called GitOps or AutoDevOps approaches, prescriptivist ways of leveraging Docker, K8s, and Helm, such as: https://www.weave.works/blog/what-is-gitops-really https://www.weave.works/blog/what-is-gitops-really https://docs.gitlab.com/ee/topics/autodevops/index.html https://docs.gitlab.com/ee/topics/autodevops/index.html
    - MadWombat 7y ago
      Kubernetes is pretty feature complete, but its focus is on containers, not naked apps. If you are willing to pack your apps into docker, kubernetes will solve the orchestration and monitoring problems. If you are not, you might want to look for something else. In particular, take a look at Nomad (https://www.nomadproject.io/ https://www.nomadproject.io/). I am not affiliated with the project in any way and at my workplace we went the kube route, but Nomad does provide an ability to orchestrate arbitrary processes across multiple hosts.
lukehoban 7y ago
We've had customers and users of Pulumi at many different sizes and stages looking at or migrating to Kubernetes - and I'd say there are a few consistent reasons we hear for making the choice to adopt Kubernetes. First, Kubernetes offers a rich, standards-based API for compute. For teams who want to deliver containerized applications, using Kubernetes allows targeting a standards API instead of a cloud-provider specific API. This up-levels the same benefit folks get with cloud IaaS solutions where they can write standard Linux applications. Now the standardized and "portable" part of their applications extends to some key parts of the compute infrastructure as well. Second, we see a ton of Cloud + Kubernetes - teams who are still taking advantage of a lot of managed services and operational benefits from AWS/Azure/GCP/etc., but want to combine that with the container orchestration and rich API of Kubernetes. This means leaving stateful workloads in managed services like Aurora or Cloud SQL, delegating auth to managed IAM services, etc. Pulumi happens to be a really nice fit for doing these together. And lastly, there's just a really active ecosystem that's building a lot of value around Kubernetes. Some of those pieces aren't yet fully mature, but directionally many teams are betting on this being where the next wave of mature cloud platform services will become available.
reilly3000 7y ago
We are going w/ Fargate for now, but the similarity in syntax may help us make the leap to k8s when we're ready.

antics 7y ago

Oh wow! I co-lead these efforts at Pulumi. I don't think we expected this to hit HN! :) Happy to answer questions about any of this. Parenthetical, one thing that didn't make it into this post that I'm particularly optimistic about is the `watch` command, which will continuously watch your code and deploy it to (say) Kubernetes. It looks like this[1]. This is very handy when you're trying to build a thing that has to integrate with a bunch of other services, but there are too many to develop locally on minikube. It basically lets you see instantly if something crashes, or whatever. In general I'm very optimistic about that sort of experience—another product in that space is Tilt[4] which I am personally a big fan of. Another thing that gets kind of a passing mention that is cooler than it looks here is our streaming `query` stuff. By exposing Kubernetes `watch`es as streaming queries, it becomes very easy to write very cool apps like this one[2] , which helps you debug services. You can actually see a bug in the Deployment controller there—we create two `Deployment`s with the same labels, and deleting one causes the controller to delete _all_ the pods. This would generate a transient 500 for any request hitting that service. I think there are probably lots of bugs in Kubernetes like this, but people don't notice them because this level of visibility previously was quite hard. :) To get an idea of how hard that sort of app is to write, you can see the code we wrote for the prototype of this SDK here[3]. [1]: https://twitter.com/funcofjoe/status/1195411343966236677?s=21 https://twitter.com/funcofjoe/status/1195411343966236677?s=2... [2]: https://drive.google.com/open?id=1Q2f8CZhswMvTliaLlHZK4vItdMVIXzOD https://drive.google.com/open?id=1Q2f8CZhswMvTliaLlHZK4vItdM... [3]: https://github.com/carbonql/carbon-ts/blob/master/examples/unix/serviceMonitor/serviceMonitor.ts https://github.com/carbonql/carbon-ts/blob/master/examples/u... [4]: https://tilt.dev/ https://tilt.dev/

migueloller 7y ago
I’ve been very impressed with what you guys are doing. We’re extremely excited about moving from our collection of YAML, bash, and Helm templates to Pulumi. The big question mark we had was what local development would look like. Tilt and Skaffold were the two big candidates we looked at for sane local development with Kubernetes. Looks like Pulumi’s new `watch` could give us what we need? Are there any plans to have something like rsync support to enable things like hot-reloading apps? Both Tilt and Skaffold have examples showing this capability.

bsingh4 7y ago

For AWS users AWS CDK is a far better take on infrastructure as code - strongly typed TypeScript resources and higher level constructs that hide the underlying details of load balancers, IAM, etc. Pulumi was not as useful the last time I tried it - most resources were string key-value pairs with no typing or code completion. You're still referencing documentation to do anything and defining tons of low level resources to accomplish basic tasks. Might as well be writing raw Terraform or CloudFormation configs at that point, as they have more documentation available.

reilly3000 7y ago
Something was wrong w/ your setup I think. All of their code has Typescript annotations, and their autocomplete w/ intellisense is remarkably useful.
- bsingh4 7y ago
  Here are two simple database definitions: Database in Pulumi: const db = new gcp.sql.DatabaseInstance('app-db', { databaseVersion: 'POSTGRES_11', region: 'us-central1', settings: { tier: 'db-f1-micro', diskType: 'PD_HDD', }, }); Database in AWS CDK: const db = new rds.DatabaseInstance(this, 'AppDb', { engine: rds.DatabaseInstanceEngine.POSTGRES, engineVersion: '11.5', instanceClass: ec2.InstanceType.of( ec2.InstanceClass.STANDARD5, ec2.InstanceSize.XLARGE ), storageType: rds.StorageType.GP2 }) Not to mention they have higher level constructs like DatabaseCluster that will handle the underlying details of replicas and create the lower level resources for you.
  - bsingh4 7y ago
    I don't think large blocks of comments qualify as strong typing: https://github.com/pulumi/pulumi-gcp/blob/76df2e85214ad134652327fa78f2f29f50502eae/sdk/nodejs/sql/databaseInstance.ts https://github.com/pulumi/pulumi-gcp/blob/76df2e85214ad13465...
    - antics 7y ago

reilly3000 7y ago

One thing I'm murky on with Pulumi is around how to execute proper blue/green or canary deployments with a GitOps approach. Any insights?

cbushko 7y ago

Any tool that helps build infrastructure as code is great news. I hear good things about pulumi. In my case I have been moving our current infrastructure from AWS to GCP. After much evaluation and a small stint with Helm, we decided to go 100% terraform. There is more to running a product than just deploying to kubernetes and terraform providers have given us the flexibility to chain all these resources together. For example, I can create a GCP service account (google cloud provider), save its credentials as a kubernetes secret (kubernetes provider), register a name in route 53 (aws provider) and deploy our product in kubernetes (kubernetes provider). Having it all in one tool that has state is really powerful and more importantly follow the same patterns. Obviously, I am a big fan of terraform

jen20 7y ago
Everything you describe is also straightforward with Pulumi using `pulumi-gcp` [1], `pulumi-aws` [2] and friends - Kubernetes is one of many libraries. On top of this, you can build actual abstractions thanks to the configuration being written using a general purpose programming language - I don't think there is a first-party set of abstractions for GKE yet but there is for EKS [3]. Many of the most compelling examples deploy entire infrastructure stacks from "empty account" to multi-cloud Kubernetes. Disclaimer: I have worked extensively on Terraform (both core and providers) at and after HC, and have also contributed to Pulumi since it was open sourced. Edited for formatting. [1]: https://github.com/pulumi/pulumi-gcp https://github.com/pulumi/pulumi-gcp [2]: https://github.com/pulumi/pulumi-aws https://github.com/pulumi/pulumi-aws [3]: https://github.com/pulumi/pulumi-eks https://github.com/pulumi/pulumi-eks

houqp 7y ago

Great write up, the experience of managing k8s end to end with pulumi is so much better compared to terrafom.