7 ms·

Five Months of Kubernetes

263 points by dreampeppers99 10y ago

deleted 10y ago
[deleted]
moret1979 10y ago
> In general, Elastic Beanstalk works fine and has a very gentle learning curve; it didn’t take long for all teams to start using it for their projects. This is one of the most pressing issues I tend to evaluate on infrastructures. I'm curious to see a few months down the line your opinion on how the dev teams embraced Kubernetes' setups independently or if they kept depending on a dev-ops team to do so.
- danielmartins 10y ago
  I believe the reason why people tend to restrict the cluster access to just a handful of people is because - at least for now - these tools lack proper ways to control access to specific resources by specific groups of people. I mean, there are ways to do that in Kubernetes today[1], but the setup is kept as an exercise to cluster operators. I don't know for sure, but this is less of a problem in some distributions, such as OpenShift[2]. Once this problem is solved, there's no reason to "shield" the cluster from devs. [1] http://kubernetes.io/docs/admin/authorization/ http://kubernetes.io/docs/admin/authorization/ [2] https://docs.openshift.com/enterprise/3.0/admin_guide/configuring_authentication.html https://docs.openshift.com/enterprise/3.0/admin_guide/config...
  - erchiang 10y ago
    Disclaimer - I work at CoreOS and on the Kubernetes auth sub systems Today, most of the auth-N plugins[0] (the upstream equivalent to the OpenShift doc you linked to) are relatively minimal. Most of these are aimed at providing pluggability for other apps that focus on ease of use. Google uses the webhook implementation for its GKE auth-N and we (CoreOS) continue to try to make our OpenID Connect server federate to more backends (LDAP, GitHub, etc.).[1] With these kind of tooling, it's completely possible to map auth-Z policies to, say a group of LDAP users. But yes, there's a lack of canonical documentation on how to go about this. We're always trying to negotiate how much of this should live in core Kubernetes and how much should be provided by third party services (and what the upstream docs should endorse). But today I'd still (perhaps because I'm biased) recommend giving your CEO different credentials for your prod cluster :) [0] http://kubernetes.io/docs/admin/authentication/ http://kubernetes.io/docs/admin/authentication/ [1] https://github.com/coreos/dex https://github.com/coreos/dex
    - 0xCMP 10y ago
      I like dex a lot. Very cool project. Was gonna use it + k8s for a side project until I realized for the scale I had (1 user) Auth0 + Heroku is still just fine, but dex lets you run Auth0 on your own and completely in your control which is awesome. No more setting up authentication!
    - smarterclayton 10y ago
      To add to Eric's point - the goal over the long term is to flesh this stuff out and have both flexibility and multi-tenancy in Kube. OpenShift could be opinionated about security (no compromise) while for Kube we wanted to make sure we built first a system that was flexible enough to be used in many ways. I think that was the correct choice, because most Kube deployments are still single ops team focused, and it was possible for CoreOS to build and integrate Dex without having to face a high bar. It'll get more opinionated eventually :)
    - deleted 10y ago
      [deleted]
machbio 10y ago
> First, we created one DNS record for each service (each initially pointing to the legacy deployment in Elastic Beanstalk) and made sure that all services referenced each other via this DNS. Then, it was just a matter of changing those DNS records to point the corresponding Kubernetes-managed load balancers. If anyone could explain this, does this mean - the services still are being accessed from Public IP or does Kubernetes managed load balancers are Private IPs that can the individual nodes know about ?
- danielmartins 10y ago
  By default, all ELBs created by Kubernetes are external, and they load balance traffic to every node in the cluster to the service port (each service gets its own port number which is the same in every node).
  - machbio 10y ago
    So, thats a lot of AWS traffic cost for the traffic between different services :(
    - danielmartins 10y ago
      That's true if you reference each service via its external ELB CNAME (or a Route53 DNS that points to it); however, Kubernetes comes with a built-in DNS server that you can use to discover the endpoint IP addresses in the pod CIDR (all pods are in the same subnet via the overlay network).
    - deleted 10y ago
      [deleted]
    - DigitalJack 10y ago
      You can define services to be internal. I presume this does not use an elb but I have not tried it yet.
      - swozey 10y ago
MattRogish 10y ago
We've been deploying Kubernetes for clients since, well, 1.0 (very recently) and have nothing but great things to say about it. If you want something approximating a Heroku-like experience but in your own environment (AWS, GKE, or even on-prem) K8s is a super awesome way to get there. Sure, like anything it's got some rough edges that you'll get cut on, but it improves every 3 months. :D Big kudos to the k8ts team at Goog, and all the other contributors!
- shad42 10y ago
  Did you have to upgrade clusters for your clients yet? Is there a way upgrade clusters to a newer version of k8s safely?
  - old-gregg 10y ago
    It is definitely not easy using "naked" open source Kubernetes, but it can be done. My company [1] packages Kubernetes-on-autopilot, which includes HA-configuration, single-click updates to the latest k8s version, etc. We run it for our customers on their AWS clouds and even on enterprise infrastructure. So it's just with any OSS project: DIY-vs-invest-into-tooling. A huge benefit of adopting Kubernetes for your SaaS product is this: it becomes much easier to push updates to enterprise environments. Selling SaaS to private clouds is pretty sweet: the size of a deal is larger but the incremental cost of R&D and ops (!) is now very low, because of standardized deployment models like k8s (what we do) or Mesos. [1] https://gravitational.com https://gravitational.com
    - sandGorgon 10y ago
      whats your opinion using plain vanilla Salt vs kube-up vs kube-aws vs the new kops ? the problem with k8s is that there is way too much stuff happening now and it is hard to get started quickly. For example, we started using Docker when it was alpha - for a single man startup.. getting up and started was very easy. On the other hand, using k8s on a 3 node cluster is... scary.
      - 10y ago
kozikow 10y ago
> Kubernetes does not offer a clean solution for a number of problems you might face, such as stateful applications. petset is a solution for stateful applications [1]. It's still in alpha though. I heard rumours that it will enter beta in about 3 months. 1. http://kubernetes.io/docs/user-guide/petset/ http://kubernetes.io/docs/user-guide/petset/
holografix 10y ago
Great article, very easy to read and tons of useful info. Love seeing fellow Brazilians giving it a go.
- danielmartins 10y ago
  Thanks, man! :)
hendry 10y ago
I've been using ecs-cli for my container deployments on AWS https://github.com/kaihendry/count https://github.com/kaihendry/count I wonder if I should try kubernetes. It seems a lot more complex but the tooling looks better maintained.
- danielmartins 10y ago
  There's no problem in sticking with your current solution if it's working for you.
dfsegoat 10y ago
I am very close to giving "k8s" a try. Despite a lot of work on trying to figure out how to get us up on docker for CI workloads - the ecosystem is very confusing in terms of docker-cloud, docker-compose vs. docker-machine (on linux vs. osx) etc.
asimjalis 10y ago
Did you try ECS as an alternative to Elastic Beanstalk? Would be interested in hearing your pros and cons for ECS vs Kubernetes on EC2.
- danielmartins 10y ago
  I didn't try it because, as mentioned in the footnotes, ECS isn't available for the some of the regions we use, but if you look around, you can find comparisons between the two[1]. I'll probably give ECS a go when (if?) Amazon enables it globally. [1] https://railsadventures.wordpress.com/2015/12/06/why-we-chose-kubernetes-over-ecs/ https://railsadventures.wordpress.com/2015/12/06/why-we-chos...
Perceptes 10y ago
I'd be very interested to see the code for the AWS Lambda functions mentioned—specifically the one about ephemeral development environments based on open PRs. We're building something similar at InQuicker and it'd be great to see how other people are approaching it.
- danielmartins 10y ago
  It' not open source, but I'll try to sell the idea to our CTO. :) Just to give you more details about its inner workings, this function is written in JavaScript that gets called when certain events come from GitHub ('pull_request', 'status', and 'push'), and uses kubectl to modify the corresponding deployments depending on the event. Nothing fancy there, trust me.
  - Perceptes 10y ago
    Do you only create copies of the stateless pieces of each stack, or do you also copy databases?
    - danielmartins 10y ago
      We currently only run stateless apps on Kubernetes. All databases are hosted elsewhere (RDS, MongoDB Cloud, etc)
      - Perceptes 10y ago
        Let me rephrase that: Do you spin up a new copy of any necessary data stores when you deploy a topic branch of an app, or do all versions of the app share the same view of data in their environment (e.g. staging/production)?
        danielmartins 10y ago
        No, these 'development' environments point to other services in 'staging'.

263 points by dreampeppers99 10y ago

deleted 10y ago

[deleted]

moret1979 10y ago

> In general, Elastic Beanstalk works fine and has a very gentle learning curve; it didn’t take long for all teams to start using it for their projects. This is one of the most pressing issues I tend to evaluate on infrastructures. I'm curious to see a few months down the line your opinion on how the dev teams embraced Kubernetes' setups independently or if they kept depending on a dev-ops team to do so.

danielmartins 10y ago
I believe the reason why people tend to restrict the cluster access to just a handful of people is because - at least for now - these tools lack proper ways to control access to specific resources by specific groups of people. I mean, there are ways to do that in Kubernetes today[1], but the setup is kept as an exercise to cluster operators. I don't know for sure, but this is less of a problem in some distributions, such as OpenShift[2]. Once this problem is solved, there's no reason to "shield" the cluster from devs. [1] http://kubernetes.io/docs/admin/authorization/ http://kubernetes.io/docs/admin/authorization/ [2] https://docs.openshift.com/enterprise/3.0/admin_guide/configuring_authentication.html https://docs.openshift.com/enterprise/3.0/admin_guide/config...
- erchiang 10y ago
  Disclaimer - I work at CoreOS and on the Kubernetes auth sub systems Today, most of the auth-N plugins[0] (the upstream equivalent to the OpenShift doc you linked to) are relatively minimal. Most of these are aimed at providing pluggability for other apps that focus on ease of use. Google uses the webhook implementation for its GKE auth-N and we (CoreOS) continue to try to make our OpenID Connect server federate to more backends (LDAP, GitHub, etc.).[1] With these kind of tooling, it's completely possible to map auth-Z policies to, say a group of LDAP users. But yes, there's a lack of canonical documentation on how to go about this. We're always trying to negotiate how much of this should live in core Kubernetes and how much should be provided by third party services (and what the upstream docs should endorse). But today I'd still (perhaps because I'm biased) recommend giving your CEO different credentials for your prod cluster :) [0] http://kubernetes.io/docs/admin/authentication/ http://kubernetes.io/docs/admin/authentication/ [1] https://github.com/coreos/dex https://github.com/coreos/dex
  - 0xCMP 10y ago
    I like dex a lot. Very cool project. Was gonna use it + k8s for a side project until I realized for the scale I had (1 user) Auth0 + Heroku is still just fine, but dex lets you run Auth0 on your own and completely in your control which is awesome. No more setting up authentication!
  - smarterclayton 10y ago
    To add to Eric's point - the goal over the long term is to flesh this stuff out and have both flexibility and multi-tenancy in Kube. OpenShift could be opinionated about security (no compromise) while for Kube we wanted to make sure we built first a system that was flexible enough to be used in many ways. I think that was the correct choice, because most Kube deployments are still single ops team focused, and it was possible for CoreOS to build and integrate Dex without having to face a high bar. It'll get more opinionated eventually :)
  - deleted 10y ago
    [deleted]

machbio 10y ago

> First, we created one DNS record for each service (each initially pointing to the legacy deployment in Elastic Beanstalk) and made sure that all services referenced each other via this DNS. Then, it was just a matter of changing those DNS records to point the corresponding Kubernetes-managed load balancers. If anyone could explain this, does this mean - the services still are being accessed from Public IP or does Kubernetes managed load balancers are Private IPs that can the individual nodes know about ?

danielmartins 10y ago
By default, all ELBs created by Kubernetes are external, and they load balance traffic to every node in the cluster to the service port (each service gets its own port number which is the same in every node).
- machbio 10y ago
  So, thats a lot of AWS traffic cost for the traffic between different services :(
  - danielmartins 10y ago
    That's true if you reference each service via its external ELB CNAME (or a Route53 DNS that points to it); however, Kubernetes comes with a built-in DNS server that you can use to discover the endpoint IP addresses in the pod CIDR (all pods are in the same subnet via the overlay network).
  - deleted 10y ago
    [deleted]
  - DigitalJack 10y ago
    You can define services to be internal. I presume this does not use an elb but I have not tried it yet.
    - swozey 10y ago

MattRogish 10y ago

We've been deploying Kubernetes for clients since, well, 1.0 (very recently) and have nothing but great things to say about it. If you want something approximating a Heroku-like experience but in your own environment (AWS, GKE, or even on-prem) K8s is a super awesome way to get there. Sure, like anything it's got some rough edges that you'll get cut on, but it improves every 3 months. :D Big kudos to the k8ts team at Goog, and all the other contributors!

shad42 10y ago
Did you have to upgrade clusters for your clients yet? Is there a way upgrade clusters to a newer version of k8s safely?
- old-gregg 10y ago
  It is definitely not easy using "naked" open source Kubernetes, but it can be done. My company [1] packages Kubernetes-on-autopilot, which includes HA-configuration, single-click updates to the latest k8s version, etc. We run it for our customers on their AWS clouds and even on enterprise infrastructure. So it's just with any OSS project: DIY-vs-invest-into-tooling. A huge benefit of adopting Kubernetes for your SaaS product is this: it becomes much easier to push updates to enterprise environments. Selling SaaS to private clouds is pretty sweet: the size of a deal is larger but the incremental cost of R&D and ops (!) is now very low, because of standardized deployment models like k8s (what we do) or Mesos. [1] https://gravitational.com https://gravitational.com
  - sandGorgon 10y ago
    whats your opinion using plain vanilla Salt vs kube-up vs kube-aws vs the new kops ? the problem with k8s is that there is way too much stuff happening now and it is hard to get started quickly. For example, we started using Docker when it was alpha - for a single man startup.. getting up and started was very easy. On the other hand, using k8s on a 3 node cluster is... scary.
    - 10y ago

kozikow 10y ago

> Kubernetes does not offer a clean solution for a number of problems you might face, such as stateful applications. petset is a solution for stateful applications [1]. It's still in alpha though. I heard rumours that it will enter beta in about 3 months. 1. http://kubernetes.io/docs/user-guide/petset/ http://kubernetes.io/docs/user-guide/petset/

holografix 10y ago

Great article, very easy to read and tons of useful info. Love seeing fellow Brazilians giving it a go.

danielmartins 10y ago
Thanks, man! :)

hendry 10y ago

I've been using ecs-cli for my container deployments on AWS https://github.com/kaihendry/count https://github.com/kaihendry/count I wonder if I should try kubernetes. It seems a lot more complex but the tooling looks better maintained.

danielmartins 10y ago
There's no problem in sticking with your current solution if it's working for you.

dfsegoat 10y ago

I am very close to giving "k8s" a try. Despite a lot of work on trying to figure out how to get us up on docker for CI workloads - the ecosystem is very confusing in terms of docker-cloud, docker-compose vs. docker-machine (on linux vs. osx) etc.

asimjalis 10y ago

Did you try ECS as an alternative to Elastic Beanstalk? Would be interested in hearing your pros and cons for ECS vs Kubernetes on EC2.

danielmartins 10y ago
I didn't try it because, as mentioned in the footnotes, ECS isn't available for the some of the regions we use, but if you look around, you can find comparisons between the two[1]. I'll probably give ECS a go when (if?) Amazon enables it globally. [1] https://railsadventures.wordpress.com/2015/12/06/why-we-chose-kubernetes-over-ecs/ https://railsadventures.wordpress.com/2015/12/06/why-we-chos...

Perceptes 10y ago

I'd be very interested to see the code for the AWS Lambda functions mentioned—specifically the one about ephemeral development environments based on open PRs. We're building something similar at InQuicker and it'd be great to see how other people are approaching it.

danielmartins 10y ago
It' not open source, but I'll try to sell the idea to our CTO. :) Just to give you more details about its inner workings, this function is written in JavaScript that gets called when certain events come from GitHub ('pull_request', 'status', and 'push'), and uses kubectl to modify the corresponding deployments depending on the event. Nothing fancy there, trust me.
- Perceptes 10y ago
  Do you only create copies of the stateless pieces of each stack, or do you also copy databases?
  - danielmartins 10y ago
    We currently only run stateless apps on Kubernetes. All databases are hosted elsewhere (RDS, MongoDB Cloud, etc)
    - Perceptes 10y ago
      Let me rephrase that: Do you spin up a new copy of any necessary data stores when you deploy a topic branch of an app, or do all versions of the app share the same view of data in their environment (e.g. staging/production)?
      - danielmartins 10y ago
        No, these 'development' environments point to other services in 'staging'.