7 ms·

AWS Service Operator for Kubernetes Now Available

119 points by SoapSeller 8y ago

simonebrunozzi 8y ago
The cloud fight of 2019-2020: AWS vs GCP in the Kubernetes arena. Curious to see who's going to win, of if it's going to be a tie. Jokes apart: GCP got a head start in containers thanks to Kubernetes; AWS realized it and tried to catch up. Dominating the space will have huge consequences down the road. My humble view is that whoever starts a RedHat-like service (with support, and SLAs, and enterprise services) on top of Kubernetes, might get the upper hand. Having built Kubernetes might not be enough for GCP to maintain the lead.
- Artemis2 8y ago
  Do you mean OpenShift? https://www.openshift.com https://www.openshift.com
- k__ 8y ago
  As far as I know the container fights are done Kubernetes won even AWS admits it, now it's all about serverless technologies.
  - hn_throwaway_99 8y ago
    For serverless technologies to "win" they have to solve the "cold start" problem. AWS likes to pitch Lambdas as an easy mobile backed, but if you need to talk to a DB (which most mobile backends do) then you'll want to put your Lambda in a VPC, which makes cold starts on the order of 5-10 seconds, which is a deal breaker for most synchronous APIs. I don't understand why AWS or GCP haven't added "pre-warming" requests to their cloud functions, similar to App Engine.
    - stevehawk 8y ago
      why the need to put lambdas in a VPC in order to hit a DB?
      - jflans 8y ago
        I think to do otherwise you'd have to have the DB open to the public internet.
      - buzzdenver 8y ago
        Because the DB doesn't have a public IP, which is very good.
      - merlincorey 8y ago
        A VPC in AWS is essentially a virtual datacenter. For many years now, essentially all AWS services are tied to a VPC. Each account gets 5 VPCs per region, by default. Whether you use RDS or EC2 to setup a database server, it will be tied to a VPC for networking isolation purposes. As such you then would need the Lambda in the VPC, or to allow public internet access to the database. The point is pretty moot though, because you can schedule Cloudwatch Events every 4 minutes to keep a lambda warm, if necessary. Frameworks like Zappa even do this for you automatically.
        skluck 8y ago
        It is somewhat arbitrary though, isn't it? If AWS adds the ability to use security groups without a VPC, a lot of these issues go away wrt vpc or siloed infrastructure limits (cold starts). There are still reasons to be in a private network - Being "one typo away" from exposing your services/db to the world is scary. But that seems like a solveable problem as well...
        hn_throwaway_99 8y ago
        > The point is pretty moot though, because you can schedule Cloudwatch Events every 4 minutes to keep a lambda warm, if necessary. I encourage you to read this article, https://theburningmonk.com/2018/01/im-afraid-youre-thinking-about-aws-lambda-cold-starts-all-wrong/ https://theburningmonk.com/2018/01/im-afraid-youre-thinking-... , because if you're running a web API with Lambdas, keeping one instance warm with the "cloudwatch event every 4 minutes" trick will most definitely not solve your cold start issues.
    - k__ 8y ago
      Hasn't AppSync already solved that problem?
    - 013a 8y ago
      It kind of makes sense, but mostly from a marketing and planning standpoint. If you need pre-warming, that probably means you know enough about lambda (and AWS) to know that you need pre-warming. Setting it up via a Cloudwatch event is easy as pie. Think about the implications if they added a button to the lambda console to "pre-warm". There are two options: (1) set up the cloudwatch event for you (which is a similar pattern we've seen AWS use for things like DynamoDB table autoscaling), or (2) have some other internal system which can keep them warm. Its easy to say "just do (1), it'd be so easy", but the issue is that it introduces a very weird cost pattern to lambda. Lambda isn't just billed per invocation, its billed essentially with time live. So if they auto-configure a cloudwatch event, lets say it sends an empty `{}` argument, they have no idea how long your function is designed to run given that input. Moreover, they don't even know that your function won't error with that input. So they've got this new feature and even they can't predict what it will do to your bill or system stability, given the fact that we're dealing with arbitrary code blobs. The only option is (2). Now think about allocating engineering effort to this problem: as a manager, would you rather allocate a team to work on an extra complex scheduling parameter, or continue to improve the fundamental warm-up time for any function? Maybe both. But now you've got this extra parameter there which increases customer expectations and makes future scheduling work much more difficult.
      - jacques_chester 8y ago
        The keep-alive behaviour of Lambda is essentially a subsidy. I suspect that AWS have paid a lot of attention to ways to cut it down, or are at least hoping the they can drive down the idling cost without anyone noticing too much of a performance hit.
    - whopa 8y ago
      I think this is why AWS promotes DynamoDB so much, because it's one of the few datastores that work sanely with non-VPC lambdas.
    - jacques_chester 8y ago
      > For serverless technologies to "win" they have to solve the "cold start" problem. I've said here and elsewhere before that autoscaling is easy to say and hard to do. We keep looking to autoscalers to divine our economic preferences, which they cannot do for us. What's been missing is the ability to explicitly trade off latency for expense. The best you can do is to a) attack startup time any how, any way possible, b) react sanely to unexpected traffic changes, c) make reasonable forecasts and d) explicitly tune cost of idleness vs cost of delay vs probability of delay. These help, but the problem will never fully go away. (Unless you've discovered an escape hatch from either of causality or integral calculus. If you have, please share it with the class.)
    - tango12 8y ago
      Or you can use serverless asynchronously only. Have a low latency container based API with min replicas and auto-scale, almost like an atomic CRUD API. Move as much to async serverless which is triggered on events.
- mahmoudimus 8y ago
  I think that is heptio
- jacques_chester 8y ago
  > My humble view is that whoever starts a RedHat-like service (with support, and SLAs, and enterprise services) on top of Kubernetes, might get the upper hand. Does Red Hat count as Red Hat-like? Because they've had OpenShift Origin for several years now.
extraterra 8y ago
It's great to see Kubernetes being integrated more tightly with the AWS ecosystem. If now all cloud providers open sourced their MySQL/PostgreSQL forks...
- jjeaff 8y ago
  Are there any good docker compose or helm files out there right now that would work for a nice production ready MySQL or PostgreSQL dB? It seems like k8s has everything you would need to have the redundant data sources, failover, and point in time recovery options that cloudsql or auroradb have.
  - jacques_chester 8y ago
    I am aware of Crunchy Data doing PostgreSQL-on-Kubernetes: https://www.crunchydata.com/products/crunchy-postgresql-for-kubernetes/ https://www.crunchydata.com/products/crunchy-postgresql-for-...
  - wskinner 8y ago
    We run the official Postgres docker image as a StatfeulSet. Very easy to setup, and we haven’t had any issues with it.
    - tango12 8y ago
      Do you run a single instance Postgres or a cluster? What kind of failure modes have you tested if you’re running a cluster?
      - wskinner 8y ago
        Single instance. And we have small scale.
  - dkhenry 8y ago
    The open source database vitess is pretty much designed to be a cloud native production mysql and there are helm charts for it (https://github.com/vitessio/vitess/tree/master/helm https://github.com/vitessio/vitess/tree/master/helm) an operator based on the the metacontrollers (https://github.com/vitessio/vitess-operator https://github.com/vitessio/vitess-operator) and a company called planetscale thats making an operator that has even more features then those two (I work for planetscale and have been building our operator )
iddqd 8y ago
For me, this is the most exciting thing AWS has launched in a while!
bearforcenine 8y ago
>we need to set up a way to manage AWS IAM credentials to Kubernetes pods... In a production system, this should be done using a tool such as kube2iam or kiam... I am curious if AWS has any plans to build an IAM integration for K8s that provides IAM credentials/roles directly to pods. An integration through EKS or K8s directly would make interacting with AWS resources very easy. Being able to authenticate to the K8s cluster using https://github.com/kubernetes-sigs/aws-iam-authenticator https://github.com/kubernetes-sigs/aws-iam-authenticator is nice, but it doesn't help give pods IAM roles.
- tomnipotent 8y ago
  AWS always launches services half-baked - I actually kinda love them for it. It's not always polished, but it's usually good enough to help me get stuff done. I'm sure we'll see much tighter integration over time.
- twalla 8y ago
  https://github.com/jtblin/kube2iam https://github.com/jtblin/kube2iam is probably what you're looking for, it uses iptables to allow/disallow pods requests to the ec2 metadata service based on kubernetes annotations in fact, if you check out the source (located here: https://github.com/awslabs/aws-service-operator https://github.com/awslabs/aws-service-operator) it's recommended to use kube2iam edit: haven't fully read the article yet but if the operator supports managing IAM roles thru a CRD you could potentially create the role and attach it via annotation in one go. double edit: looks like IAM roles aren't directly supported yet, the following is what appears to be supported: - cloudformation templates - dynamodb - s3 - sns subscriptions and topics - sqs queues - ecr repos
  - bearforcenine 8y ago
    I understand that both kube2iam and kiam exist and they're both fine. I am interested in Amazon/EKS directly supporting something. Hopefully we will see something soon via CRD or otherwise.
  - Rapzid 8y ago
    If I still need to create an IAM role externally for the pod granting it access to the different resources.. Might as well create the resources themselves as well?
juancampa 8y ago
Is there something equivalent for GCP? AFAIK you control load balancers via Ingress objects but that's about it in terms of integration, right?
- wstrange 8y ago
  GCP has service catalog[1] that can interface to various GCP services (Spanner, cloud sql, pub sub, etc.). Service catalog is based on the open service broker spec. [1] https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/use-service-catalog https://cloud.google.com/kubernetes-engine/docs/how-to/add-o...
  - joseph 8y ago
    Amazon has had that for a while too, but I just learned about it recently, see [1]. I wonder if this is doing the same thing under the hood, or if it is a competing project within Amazon. Regardless, something like this is sorely needed for making infrastructure changes deployable along with application changes. 1. https://aws.amazon.com/blogs/opensource/provision-aws-services-kubernetes-aws-service-broker/ https://aws.amazon.com/blogs/opensource/provision-aws-servic...
    - jacques_chester 8y ago
      The difference between "meet customers at a Kubernetes CRD" and "meet customers at an OSBAPI broker" is small enough that it's not worth paying strategy tax to crimp the latter, especially since these customer groups will be overlapping but not equivalent sets (this is how enterprise software grows so vast). Rephrasing: AWS are smart to have a bob each way.
InTheArena 8y ago
This is absolutely awesome. I've thought about doing this a couple of times, as a abstraction layer on top of different clouds, but this is really cool... It's also the first time that I think we have really seen AWS really contribute something to the K8s ecosystem they do lots of good work at the CNCF that is interesting and innovative. (EKS is not as capable as GKS or AKS, and even things like HPA only recently are enabled).
- patrickg_zill 8y ago
  Does this work benefit those who are not using AWS?
mooreds 8y ago
Why didn't they launch with RDS support? Seems like a no brainer.
- christopherhein 8y ago
  Great question, it's a little more complicated than one might think at first. In trying to build a "batteries included" experience I'd need to have per-view into your cluster and what VPC, Subnet and AZ you are running in I don't want to make this a configuration option so I need to build out a way to collect this information dynamically so that I can make sure we create DB subnets for the RDS to provision into. Then I need to configure depending on the engine (pg, mysql etc) the port and security group configuration. All in all the CFT and is more complicated and with the way the resources are code generated it requires heavy customization. All that being said it is well up on the top of my list to implement. Also always interested in letting other folks come and contribute if they feel inclined. :)
  - mooreds 8y ago
    Ah, maybe I meant "it's a no brainer from a user perspective". Thanks for the explanation.
    - christopherhein 8y ago
      :) No worries, if you have other services that you find would be really useful please file issues https://github.com/awslabs/aws-service-operator/issues https://github.com/awslabs/aws-service-operator/issues
neurobashing 8y ago
What terminal theme was that in the screenshot? I saw command icons, so it’s a Mac, but how do you make iterm look like that?
- Eldt 8y ago
  Take a look at the zsh/fish themes
- sbr464 8y ago
  Looks like oh-my-zsh default
blazespin 8y ago
Just an attempt to de-comiditize cloud infra .. what we need is a cloud agnostic solution built by someone other than aws/azure/goog
- iamgopal 8y ago
  What's wrong with them ? Three is enough competition.
  - rorykoehler 8y ago
    Cost is what is wrong. They will form a cartel if there are no competing solutions that reign them in.

119 points by SoapSeller 8y ago

The cloud fight of 2019-2020: AWS vs GCP in the Kubernetes arena. Curious to see who's going to win, of if it's going to be a tie. Jokes apart: GCP got a head start in containers thanks to Kubernetes; AWS realized it and tried to catch up. Dominating the space will have huge consequences down the road. My humble view is that whoever starts a RedHat-like service (with support, and SLAs, and enterprise services) on top of Kubernetes, might get the upper hand. Having built Kubernetes might not be enough for GCP to maintain the lead.

Artemis2 8y ago
Do you mean OpenShift? https://www.openshift.com https://www.openshift.com
k__ 8y ago
As far as I know the container fights are done Kubernetes won even AWS admits it, now it's all about serverless technologies.
- hn_throwaway_99 8y ago
  For serverless technologies to "win" they have to solve the "cold start" problem. AWS likes to pitch Lambdas as an easy mobile backed, but if you need to talk to a DB (which most mobile backends do) then you'll want to put your Lambda in a VPC, which makes cold starts on the order of 5-10 seconds, which is a deal breaker for most synchronous APIs. I don't understand why AWS or GCP haven't added "pre-warming" requests to their cloud functions, similar to App Engine.
  - stevehawk 8y ago
    why the need to put lambdas in a VPC in order to hit a DB?
    - jflans 8y ago
      I think to do otherwise you'd have to have the DB open to the public internet.
    - buzzdenver 8y ago
      Because the DB doesn't have a public IP, which is very good.
    - merlincorey 8y ago
      A VPC in AWS is essentially a virtual datacenter. For many years now, essentially all AWS services are tied to a VPC. Each account gets 5 VPCs per region, by default. Whether you use RDS or EC2 to setup a database server, it will be tied to a VPC for networking isolation purposes. As such you then would need the Lambda in the VPC, or to allow public internet access to the database. The point is pretty moot though, because you can schedule Cloudwatch Events every 4 minutes to keep a lambda warm, if necessary. Frameworks like Zappa even do this for you automatically.
      - skluck 8y ago
        It is somewhat arbitrary though, isn't it? If AWS adds the ability to use security groups without a VPC, a lot of these issues go away wrt vpc or siloed infrastructure limits (cold starts). There are still reasons to be in a private network - Being "one typo away" from exposing your services/db to the world is scary. But that seems like a solveable problem as well...
      - hn_throwaway_99 8y ago
        > The point is pretty moot though, because you can schedule Cloudwatch Events every 4 minutes to keep a lambda warm, if necessary. I encourage you to read this article, https://theburningmonk.com/2018/01/im-afraid-youre-thinking-about-aws-lambda-cold-starts-all-wrong/ https://theburningmonk.com/2018/01/im-afraid-youre-thinking-... , because if you're running a web API with Lambdas, keeping one instance warm with the "cloudwatch event every 4 minutes" trick will most definitely not solve your cold start issues.
  - k__ 8y ago
    Hasn't AppSync already solved that problem?
  - 013a 8y ago
    It kind of makes sense, but mostly from a marketing and planning standpoint. If you need pre-warming, that probably means you know enough about lambda (and AWS) to know that you need pre-warming. Setting it up via a Cloudwatch event is easy as pie. Think about the implications if they added a button to the lambda console to "pre-warm". There are two options: (1) set up the cloudwatch event for you (which is a similar pattern we've seen AWS use for things like DynamoDB table autoscaling), or (2) have some other internal system which can keep them warm. Its easy to say "just do (1), it'd be so easy", but the issue is that it introduces a very weird cost pattern to lambda. Lambda isn't just billed per invocation, its billed essentially with time live. So if they auto-configure a cloudwatch event, lets say it sends an empty `{}` argument, they have no idea how long your function is designed to run given that input. Moreover, they don't even know that your function won't error with that input. So they've got this new feature and even they can't predict what it will do to your bill or system stability, given the fact that we're dealing with arbitrary code blobs. The only option is (2). Now think about allocating engineering effort to this problem: as a manager, would you rather allocate a team to work on an extra complex scheduling parameter, or continue to improve the fundamental warm-up time for any function? Maybe both. But now you've got this extra parameter there which increases customer expectations and makes future scheduling work much more difficult.
    - jacques_chester 8y ago
      The keep-alive behaviour of Lambda is essentially a subsidy. I suspect that AWS have paid a lot of attention to ways to cut it down, or are at least hoping the they can drive down the idling cost without anyone noticing too much of a performance hit.
  - whopa 8y ago
    I think this is why AWS promotes DynamoDB so much, because it's one of the few datastores that work sanely with non-VPC lambdas.
  - jacques_chester 8y ago
    > For serverless technologies to "win" they have to solve the "cold start" problem. I've said here and elsewhere before that autoscaling is easy to say and hard to do. We keep looking to autoscalers to divine our economic preferences, which they cannot do for us. What's been missing is the ability to explicitly trade off latency for expense. The best you can do is to a) attack startup time any how, any way possible, b) react sanely to unexpected traffic changes, c) make reasonable forecasts and d) explicitly tune cost of idleness vs cost of delay vs probability of delay. These help, but the problem will never fully go away. (Unless you've discovered an escape hatch from either of causality or integral calculus. If you have, please share it with the class.)
  - tango12 8y ago
    Or you can use serverless asynchronously only. Have a low latency container based API with min replicas and auto-scale, almost like an atomic CRUD API. Move as much to async serverless which is triggered on events.
mahmoudimus 8y ago
I think that is heptio
jacques_chester 8y ago
> My humble view is that whoever starts a RedHat-like service (with support, and SLAs, and enterprise services) on top of Kubernetes, might get the upper hand. Does Red Hat count as Red Hat-like? Because they've had OpenShift Origin for several years now.

extraterra 8y ago

It's great to see Kubernetes being integrated more tightly with the AWS ecosystem. If now all cloud providers open sourced their MySQL/PostgreSQL forks...

jjeaff 8y ago
Are there any good docker compose or helm files out there right now that would work for a nice production ready MySQL or PostgreSQL dB? It seems like k8s has everything you would need to have the redundant data sources, failover, and point in time recovery options that cloudsql or auroradb have.
- jacques_chester 8y ago
  I am aware of Crunchy Data doing PostgreSQL-on-Kubernetes: https://www.crunchydata.com/products/crunchy-postgresql-for-kubernetes/ https://www.crunchydata.com/products/crunchy-postgresql-for-...
- wskinner 8y ago
  We run the official Postgres docker image as a StatfeulSet. Very easy to setup, and we haven’t had any issues with it.
  - tango12 8y ago
    Do you run a single instance Postgres or a cluster? What kind of failure modes have you tested if you’re running a cluster?
    - wskinner 8y ago
      Single instance. And we have small scale.
- dkhenry 8y ago
  The open source database vitess is pretty much designed to be a cloud native production mysql and there are helm charts for it (https://github.com/vitessio/vitess/tree/master/helm https://github.com/vitessio/vitess/tree/master/helm) an operator based on the the metacontrollers (https://github.com/vitessio/vitess-operator https://github.com/vitessio/vitess-operator) and a company called planetscale thats making an operator that has even more features then those two (I work for planetscale and have been building our operator )

iddqd 8y ago

For me, this is the most exciting thing AWS has launched in a while!

bearforcenine 8y ago

>we need to set up a way to manage AWS IAM credentials to Kubernetes pods... In a production system, this should be done using a tool such as kube2iam or kiam... I am curious if AWS has any plans to build an IAM integration for K8s that provides IAM credentials/roles directly to pods. An integration through EKS or K8s directly would make interacting with AWS resources very easy. Being able to authenticate to the K8s cluster using https://github.com/kubernetes-sigs/aws-iam-authenticator https://github.com/kubernetes-sigs/aws-iam-authenticator is nice, but it doesn't help give pods IAM roles.

tomnipotent 8y ago
AWS always launches services half-baked - I actually kinda love them for it. It's not always polished, but it's usually good enough to help me get stuff done. I'm sure we'll see much tighter integration over time.
twalla 8y ago
https://github.com/jtblin/kube2iam https://github.com/jtblin/kube2iam is probably what you're looking for, it uses iptables to allow/disallow pods requests to the ec2 metadata service based on kubernetes annotations in fact, if you check out the source (located here: https://github.com/awslabs/aws-service-operator https://github.com/awslabs/aws-service-operator) it's recommended to use kube2iam edit: haven't fully read the article yet but if the operator supports managing IAM roles thru a CRD you could potentially create the role and attach it via annotation in one go. double edit: looks like IAM roles aren't directly supported yet, the following is what appears to be supported: - cloudformation templates - dynamodb - s3 - sns subscriptions and topics - sqs queues - ecr repos
- bearforcenine 8y ago
  I understand that both kube2iam and kiam exist and they're both fine. I am interested in Amazon/EKS directly supporting something. Hopefully we will see something soon via CRD or otherwise.
- Rapzid 8y ago
  If I still need to create an IAM role externally for the pod granting it access to the different resources.. Might as well create the resources themselves as well?

juancampa 8y ago

Is there something equivalent for GCP? AFAIK you control load balancers via Ingress objects but that's about it in terms of integration, right?

wstrange 8y ago
GCP has service catalog[1] that can interface to various GCP services (Spanner, cloud sql, pub sub, etc.). Service catalog is based on the open service broker spec. [1] https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/use-service-catalog https://cloud.google.com/kubernetes-engine/docs/how-to/add-o...
- joseph 8y ago
  Amazon has had that for a while too, but I just learned about it recently, see [1]. I wonder if this is doing the same thing under the hood, or if it is a competing project within Amazon. Regardless, something like this is sorely needed for making infrastructure changes deployable along with application changes. 1. https://aws.amazon.com/blogs/opensource/provision-aws-services-kubernetes-aws-service-broker/ https://aws.amazon.com/blogs/opensource/provision-aws-servic...
  - jacques_chester 8y ago
    The difference between "meet customers at a Kubernetes CRD" and "meet customers at an OSBAPI broker" is small enough that it's not worth paying strategy tax to crimp the latter, especially since these customer groups will be overlapping but not equivalent sets (this is how enterprise software grows so vast). Rephrasing: AWS are smart to have a bob each way.

InTheArena 8y ago

This is absolutely awesome. I've thought about doing this a couple of times, as a abstraction layer on top of different clouds, but this is really cool... It's also the first time that I think we have really seen AWS really contribute something to the K8s ecosystem they do lots of good work at the CNCF that is interesting and innovative. (EKS is not as capable as GKS or AKS, and even things like HPA only recently are enabled).

patrickg_zill 8y ago
Does this work benefit those who are not using AWS?

mooreds 8y ago

Why didn't they launch with RDS support? Seems like a no brainer.

christopherhein 8y ago
Great question, it's a little more complicated than one might think at first. In trying to build a "batteries included" experience I'd need to have per-view into your cluster and what VPC, Subnet and AZ you are running in I don't want to make this a configuration option so I need to build out a way to collect this information dynamically so that I can make sure we create DB subnets for the RDS to provision into. Then I need to configure depending on the engine (pg, mysql etc) the port and security group configuration. All in all the CFT and is more complicated and with the way the resources are code generated it requires heavy customization. All that being said it is well up on the top of my list to implement. Also always interested in letting other folks come and contribute if they feel inclined. :)
- mooreds 8y ago
  Ah, maybe I meant "it's a no brainer from a user perspective". Thanks for the explanation.
  - christopherhein 8y ago
    :) No worries, if you have other services that you find would be really useful please file issues https://github.com/awslabs/aws-service-operator/issues https://github.com/awslabs/aws-service-operator/issues

neurobashing 8y ago

What terminal theme was that in the screenshot? I saw command icons, so it’s a Mac, but how do you make iterm look like that?

Eldt 8y ago
Take a look at the zsh/fish themes
sbr464 8y ago
Looks like oh-my-zsh default

blazespin 8y ago

Just an attempt to de-comiditize cloud infra .. what we need is a cloud agnostic solution built by someone other than aws/azure/goog

iamgopal 8y ago
What's wrong with them ? Three is enough competition.
- rorykoehler 8y ago
  Cost is what is wrong. They will form a cartel if there are no competing solutions that reign them in.