13 ms·

The AWS Controllers for Kubernetes

189 points by bdcravens 6y ago

moondev 6y ago
Seems odd there is no controller for EC2 or even planned on the roadmap https://github.com/aws/aws-controllers-k8s/projects/1 https://github.com/aws/aws-controllers-k8s/projects/1
- alexeldeib 6y ago
  It's not weird at all. A prime use case for this is to use Kubernetes itself for the compute layer and orchestrating peripheral AWS components using Kubernetes as the common control plane. You can orchestrate entire application stacks (pods, persistent storage, cloud resources as CRDs) using this approach.
  - harpratap 6y ago
    There is a fairly decent demand for orchestrating VMs using kubernetes (kubervirt), many legacy apps are too expensive to be rewritten in a cloud native way
- paxys 6y ago
  There is no point in including services that are already directly mapped to a Kubernetes resource. EC2 instances are nodes, EBS/EFS can be mounted as volumes etc. This project adds Custom Resource Definitions for the rest. What I really want is RDS/Aurora though.
  - moondev 6y ago
    Not all ec2 instances are k8s nodes though. My thought is using this to manage all ec2 instances via k8s. Instead of the aws console
sytringy05 6y ago
I can't decide if I think this is a good idea or not. Conceptually I like that I can get a s3 bucket/rds db/sqs queue by using kubectl but I'm not sure if that's the best way to manage the lifecycle, especially on something like a container registry, that likely outlives any given k8s cluster.
- throwayws 6y ago
  Maybe it makes sense for development, short-lived resources or cattle. Don't start about cross-cluster use though, having to think about where a resource is defined/used is a mess. The same issue is with Service Catalog and the open service broker model. AWS ACK is way better in its expressiveness, in comparison. (one crd per resource) It is more difficult to generate them though, compared to simply adding existing brokers.
- dmlittle 6y ago
  S3 buckets and RDS instances probably also outlive your cluster if you're treating them as cattle. I want to make sure I don't delete my RDS instance while I'm doing a blue/green cluster upgrade. I can see defining SQS next to your application being handy (they're somewhat stateful while there are messages in the queue but I'm sure you can do something to ensure they're drained before destroying them).
- closeparen 6y ago
  Why are these clusters going away?
  - harpratap 6y ago
    Most of the times it is because of fear of breaking something during a rolling update. So people usually create a new cluster from scratch and slowly migrate one by one.
  - sytringy05 6y ago
    We rebuild ours all the time. New config, k8s version upgrade, node OS patching.
wavesquid 6y ago
This is great! Are other companies doing similar things? e.g. I would love to be able to set up Cloudflare Access for services in k8s
- levi_b 6y ago
  We just released an operator with similar goals, but it works across clouds. More details here: https://www.pulumi.com/blog/pulumi-kubernetes-operator/ https://www.pulumi.com/blog/pulumi-kubernetes-operator/
- sytringy05 6y ago
  GCP (Config Connector) and Azure (Service something) both have similar things. I've not heard of it happening outside a managed k8s env.
  - harpratap 6y ago
    https://crossplane.io https://crossplane.io is doing a multi-cloud one
    - zxienin 6y ago
      I like their work, but their OAM centricity is too heavy an opiniation.
      - bassamtabbara 6y ago
        disclaimer: I'm a maintainer on Crossplane. OAM is an optional feature of crossplane - you don't have to use it if you don't want to
        zxienin 6y ago
        Good to know, at least that warm me up to crossplane further. The messaging might need update, including within the docs. I mean - crossplane is the OAM implement - coupled with OAM sprinkled all over docs, gave me very different impression. This aside, I think crossplane work is interesting.
harpratap 6y ago
Glad to see AWS finally embracing Kubernetes too. Google did a similar thing a while back - https://cloud.google.com/config-connector/ https://cloud.google.com/config-connector/ So I guess this solidifies Kubernetes as the defacto standard of Cloud Platforms.
- zxienin 6y ago
  Azure as well https://github.com/Azure/azure-service-operator https://github.com/Azure/azure-service-operator
  - FridgeSeal 6y ago
    Bold of you to assume that: * the magic permissions ghost that runs in Azure whose job it is to inexplicably deny you resources to things won’t interfere * Said Azure service will stay up long enough to be useful * you finish writing the insane amount of config Azure services seemingly require before the heat death of the universe. * Azure decides that it likes you and won’t arbitrarily block you from attaching things to your cluster/nodes because it’s the wrong moon cycle/day of the week/device type/etc * you can somehow navigate Azures kafka-esque documentation got figure out which services you’re actually allowed to do this with. It is only a slight exaggeration to say that Azure is the most painful and frustrating software/cloud product I’ve used in a long time, probably ever, and I earnestly look forward to having literally any excuse to migrate off it.
    - zxienin 6y ago
      > and I earnestly look forward to having literally any excuse to migrate off it Azure’s enterprise sales machine will make sure your wait is long. They sell bulk block deals to CIO, not to devs directly.
    - ithkuil 6y ago
      Give it time, it will get better. When I was forced to use it in 2010 it was horrible; now ten years later is definitely better. In 2030 it will rock!
ponderingfish 6y ago
Orchestration tools are the way forward especially when it comes to on-demand video compression - it's helpful to have the tools to be able to spin up 100s of servers to handle peak loads and then go down to nothing. Kubernetes is so helpful in this.
- jtsiskin 6y ago
  Would AWS spot instances be useful here?
  - big-malloc 6y ago
    Currently the cluster autoscaler supports using a pool of spot instances based on pricing, which is super helpful for test clusters, and there are some other tools available to ensure that you can evict your spot nodes when amazon needs them back
- Scaevolus 6y ago
  Are you talking about on-demand compression for viewers, or on-demand for uploads? The economics of on-demand transcoding are usually pretty bad in the cloud.
hardwaresofton 6y ago
At the risk of being early, RIP CloudFormation. I posited that this was the benefit in knowing Kubernetes all along, and possibly the ace up GCP's sleeve -- soon no cloud provider will have to offer their own interface, they'll all just offer the one invented by Kubernetes.
- delhanty 6y ago
  GCP successfully executing a strategy described by Joel Spolsky in 2002 [0]: >Smart companies try to commoditize their products’ complements. [0] Strategy Letter V https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ (edited)
- EwanToo 6y ago
  You're very early (though maybe not wrong eventually), quoting from the article "ACK is available as a developer preview, supporting the following AWS services: Amazon API Gateway V2 Amazon DynamoDB Amazon ECR Amazon S3 Amazon SNS Amazon SQS" And: "Specifically, in the coming months we plan to focus on: Amazon Relational Database Service (RDS), track via the RDS label. Amazon ElastiCache offers fully managed Redis and Memcached, track via the Elasticache label." In 5 years maybe this will be comparable to Cloudformation, but AWS will likely launch services faster than this adds them.
- StreamBright 6y ago
  You are assuming a lot here. Kubernetes is not nearly as popular and used as much to justify that we only need that interface. CloudFormation is replaced with Terraform, Ansible and Pulumi much more than it is replaced with Kubernetes.
  - hardwaresofton 6y ago
    I am assuming a lot, very possibly super early but I do want to point out that it's terraform/pulumi, and separately ansible (unless you're talking about ansible for provisioning/cloud modules[0] which no one ever seems to know about). But what I'm trying to get at is that the kubernetes interface basically is the terraform/pulumi interface, just with a different configuration language (and less built-in templating macros, which is a GOOD thing, see CloudFormation), but better, because it's continuous. You write a configuration (no matter how dynamic you make things, there's one version that's sent out to AWS/GCP/whatever) with terraform/pulumi, and terraform/pulumi make it true once. You write a relatively similar configuration with k8s and it is made to be true once, but also continuously. As far as popularity goes, kubernetes is insanely popular -- kubernetes was adopted by just about every cloud provider with a speed I've never seen before. Terraform/Pulumi basically took on the work of adapting to cloud provider APIs, not the other way around. [0]: https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html https://docs.ansible.com/ansible/latest/modules/list_of_clou...
zxienin 6y ago
There is now secular push towards use of custom operators instead of OSB. I wonder what finally caused this.
- jacques_chester 6y ago
  A mix of factors, I think. 1. OSBAPI is not widely known outside of the Cloud Foundry community it came from. In turn that's because Cloud Foundry is not widely known either. Its backers never bothered to market Cloud Foundry or OSBAPI to a wider audience. 2. It imposes a relatively high barrier to entry for implementers. You need to fill in a lot of capabilities before your service can appear in a conformant marketplace. With CRDs you can have a prototype by lunchtime. It might be crappy and you will reinvent a whole bunch of wheels, but the first attempt is easy. 3. Fashion trends. The first appearance of OSBAPI in Kubernetes-land used API aggregation, which was supplanted by CRDs. Later implementations switched to CRDs but by then the ship was already sailing. 4. RDD. You get more points for writing your own freeform controller than for implementing a standard that isn't the latest, coolest, hippest thing. It's very frustrating as an observer without any way to influence events. OSBAPI was an important attempt to save a great deal of pain. It provided a uniform model, so that improvements could be shared amongst all implementations, so that tools could work against standard interfaces in predictable ways, so that end-users had one and only one set of concepts, terms and tools to learn. It also made a crisp division between marketplaces, provisioning and binding. What we have instead is a mess. Everyone writing their own things, their own way. No standards, no uniformity, different terms, different assumptions, different levels of functionality. No meaningful marketplace concept. Provisioning conflated with binding and vice versa. It is a medium-sized disaster, diffuse but very real. And thanks to the marketing genius of enterprise vendors who never saw a short-term buck in broad spectrum developer awareness, it is basically an invisible disaster. What we're heading towards now is seen as normal and ordinary. And it drives me bonkers.
  - zxienin 6y ago
    I’d agree on the mess. I also find it on over-engineered side. Do I really need service discovery of services that I already know of, from AWS GCP...?
Niksko 6y ago
The approach of generating the code from the existing Golang API bindings means that hopefully this project will get support for lots of resources pretty quickly. Excited about this, though you do wonder whether it'll suffer the same fate as Cloudformation: the Cloudformation team finds out about new feature launches the same time that the general public does. If the Kubernetes operator lags behind, you're going to have to fall back to something else if you need cutting edge features.
solatic 6y ago
Currently no support for provisioning IAM permissions. ACK will be happy to construct an S3 bucket for you, that is then inaccessible, unless you use dangerous IAM wildcard permissions. Team is concerned about the security ramifications of setting up IAM permissions from ACK: https://github.com/aws/aws-controllers-k8s/issues/22#issuecomment-595816197 https://github.com/aws/aws-controllers-k8s/issues/22#issueco... Look, it'll be great when it matures... but this is very much in the developer preview stage. Caveat emptor.
Legogris 6y ago
Am I the only one who's pessimistic about this? One of the big upsides of Kubernetes is having portable workloads and provisioning cloud-provider specific resources (whose lifecycles very likely outlive clusters!) in Kubernetes just seems wrong to me. Kubernetes is great for managing, orchestrating and directing traffic for containerized workloads but it really shouldn't be The One Tool For Everything. Coupling everything together like this just seems to make things less manageable. IMO infrastructure including managed services are better provisioned through tools like Terraform and Pulumi.
- solatic 6y ago
  The issue (or benefit, depending on your perspective) with Terraform is that it's a one-shot CLI binary. If you're not running the binary, then it's not doing anything. If you want a long-running daemon that responds to non-human-initiated events, then Terraform isn't a good tool. Any time you try to declaratively define state, if you don't have a daemon enforcing the declarative state, then you will suffer from drift. One approach is the one Terraform has - assume that drift is possible, so ask the user to run a separate plan stage, and manually reconcile drift if needed. Another approach is the controller approach, where the controller tries to do the right thing and errors/alerts if it doesn't know what to do.
  - redwood 6y ago
    This is why Hashicorp needs to accelerate their cloud offering. Frankly I get the sense they got a little bit too addicted to central ops driven on-prem style deals for Vault but in the public cloud they need to be front and center with SaaS which is a long road. They have a rudimentary Terraform SaaS I believe but none for Vault as far as I'm aware. I see a lot of folks going straight to cloud provider services because of this. You sum it up well... In these times you don't want to run a daemon
    - t3rabytes 6y ago
      They used to have a managed Vault offering! But then it disappeared one day never to return.
thinkersilver 6y ago
Kubernetes is becoming the lingua-franca of building infrastructure. Through CRDs and the kube api spec I can - start an single application - deploy a set of interconnected apps - define network topologies - define traffic flows - define vertical and horizontal scaling of resources And now I can define AWS resources. This creates an interesting scenario where infrastructure can be defined by the k8s API resources and not necessarily have k8s build it. For example podman starting containers off a K8S deployment spec. It's an API first approach and its great for interoperability. The only downside is managing the yaml and keeping it consistent across the interdependencies.
- vtkacenko 6y ago
  I’d love an option for the yaml to be changed to something akin to gRPC .proto configs.
- brown9-2 6y ago
  The Kubernetes resource model is the real lingua franca
- soulnothing 6y ago
  I really wish fabric8, and more specifically kotlin k8 dsl[2] was getting more traction. It removes the down side of yaml all over the place. It's missing the package management features of helm. But I have several jars acting as baseline deployments, and provisioning. It works really well, and I have an entire language. So I can map over a list of services, instead of do templating. The other big down side is a java run takes a minute or two to kick off I was resilient to k8 for a long time. Complexity was secondary to cost, but Digital ocean has a relatively cheap implementation now. This commonality and perseverance of tooling is great. I want metrics, a simple annotation. I want a secret injected from vault, just add an annotation. It's also cloud agnostic, so this logic can be deployed any where some one provides a k8 offering. EKS was very powerful. As running service accounts via non managed clusters. Removed the need to pass an access key pair to the application. That service account just ran with a corresponding iam role. [1] https://github.com/fabric8io/kubernetes-client https://github.com/fabric8io/kubernetes-client [2] https://github.com/fkorotkov/k8s-kotlin-dsl https://github.com/fkorotkov/k8s-kotlin-dsl
ransom1538 6y ago
Here is my container: run it. Where is my url? The end. No, I don't want Terraforms, puppets, yaml files, load balancers, nodes, pods, k8s, chaos monkeys, Pulumies, pumas, unicorns, trees, portobilities, or shards. I love cloudrun and fargate. Cloudrun has like 5 settings, I wish it had like 2.
- throwayws 6y ago
  What do you use to provision managed services like rds? Or are you describing your dream UI :)
  - ransom1538 6y ago
    Literally that. The actual RDS ui (aws or gcp). I can setup a new db instance in 3 clicks. Slap in the dbs ip into the container.
    - throwayws 6y ago
      Your point makes totally sense. Though I also like to be able to bring up (and destroy) my servers, with a single command, when needed. Going through the same ui every time is a little annoying.
    - marcinzm 6y ago
      That works when you have one container. When you have 200 and fifty databases and four identical environments (prod, qa, staging, dev) and so on it becomes a lot more error prone. These systems exist for those use cases. Granted even with one container and a single DB I found terraform useful as I don't have to fiddle with whatever DB, container, etc. settings there are every time. I also prefer to not have to memorize or figure out a UI I use once every other month. Too many UIs across too many things.
- bdcravens 6y ago
  So basically, Heroku :-)
1-KB-OK 6y ago
Interesting how they enforce the namespaced scope for ACK custom resources. This is a logical design choice but makes it trickier for operators to use. Say I have an operator watching all namespaces on the cluster. Since operator CRDs are global in scope it makes sense for some operators to be installed as singletons. A CR for this operator gets created in some namespace, and it wants to talk to s3 -- it has to bring along its own s3 credentials and only that CR is allowed to use the s3 bucket? You can imagine a scenario where multiple CRs across namespaces want access to the same s3 bucket.
etxm 6y ago
This is nice from the app manifest perspective because you can declare your database right along side your deployment. The provisioning time of a deployment and an RDS instance is very different though. This is probably most useful when you’re starting a service up for the first time. This is also when it’s not going to work as expected due to that latency of RDS starting up while your app crashed repeatedly waiting for that connection string. This would be really nice for buckets and near instant provisioned resources, but also kinda scary that someone could nuke a whole data store because they got a trigger finger with a weird deployment and deleted and reapplied it. My feelings, they are mixed. :D
toumorokoshi 6y ago
This has been posted a couple times, but GCP has an equivalent that's been around for a while: https://cloud.google.com/config-connector/docs/overview https://cloud.google.com/config-connector/docs/overview disclaimer: I work at GCP.
deleted 6y ago
[deleted]
sytse 6y ago
This feels like Crossplane.io but limited to only AWS. Kelsey seems to think the same https://twitter.com/kelseyhightower/status/1296321377134231552 https://twitter.com/kelseyhightower/status/12963213771342315...
la6471 6y ago
Everything from DNS to AWS SDKs gets reinvented in Kubernetes. It is the most anal approach of Infrastructure design I have ever seen in the last three decades. A good design builds on the things that are already there and does not goes around trying to change every well established protocol in the world.KISS.
sunilkumarc 6y ago
Wow. Now we can directly manage AWS services from Kubernetes. Github: https://aws.github.io/aws-controllers-k8s/ https://aws.github.io/aws-controllers-k8s/ On a different note, recently I was looking to learn AWS concepts through online courses. After so much of research I finally found this e-book on Gumroad which is written by Daniel Vassallo who has worked in AWS team for 10+ years. I found this e-book very helpful as a beginner. This book covers most of the topics that you need to learn to get started: If someone is interested, here is the link :) https://gumroad.com/a/238777459/MsVlG https://gumroad.com/a/238777459/MsVlG
MichaelMoser123 6y ago
Kubernetes is supposed to be cloud vendor agnostic; the cloud vendors counter that by having extension operators to create some tie in to the kubernetes deployment of their making. I guess the 'kubernetes' way would be to create a generalized object for 'object store', that would be implemented by means of s3 on aws and on azure it would be done as blob storage. Now with this approach you can only do the common features between all platforms, you would have a problem with features exclusive to aws for instance, or you would need some mapping from a generalized CRD object to specific implementation of each platform.

189 points by bdcravens 6y ago

Seems odd there is no controller for EC2 or even planned on the roadmap https://github.com/aws/aws-controllers-k8s/projects/1 https://github.com/aws/aws-controllers-k8s/projects/1

alexeldeib 6y ago
It's not weird at all. A prime use case for this is to use Kubernetes itself for the compute layer and orchestrating peripheral AWS components using Kubernetes as the common control plane. You can orchestrate entire application stacks (pods, persistent storage, cloud resources as CRDs) using this approach.
- harpratap 6y ago
  There is a fairly decent demand for orchestrating VMs using kubernetes (kubervirt), many legacy apps are too expensive to be rewritten in a cloud native way
paxys 6y ago
There is no point in including services that are already directly mapped to a Kubernetes resource. EC2 instances are nodes, EBS/EFS can be mounted as volumes etc. This project adds Custom Resource Definitions for the rest. What I really want is RDS/Aurora though.
- moondev 6y ago
  Not all ec2 instances are k8s nodes though. My thought is using this to manage all ec2 instances via k8s. Instead of the aws console

sytringy05 6y ago

I can't decide if I think this is a good idea or not. Conceptually I like that I can get a s3 bucket/rds db/sqs queue by using kubectl but I'm not sure if that's the best way to manage the lifecycle, especially on something like a container registry, that likely outlives any given k8s cluster.

throwayws 6y ago
Maybe it makes sense for development, short-lived resources or cattle. Don't start about cross-cluster use though, having to think about where a resource is defined/used is a mess. The same issue is with Service Catalog and the open service broker model. AWS ACK is way better in its expressiveness, in comparison. (one crd per resource) It is more difficult to generate them though, compared to simply adding existing brokers.
dmlittle 6y ago
S3 buckets and RDS instances probably also outlive your cluster if you're treating them as cattle. I want to make sure I don't delete my RDS instance while I'm doing a blue/green cluster upgrade. I can see defining SQS next to your application being handy (they're somewhat stateful while there are messages in the queue but I'm sure you can do something to ensure they're drained before destroying them).
closeparen 6y ago
Why are these clusters going away?
- harpratap 6y ago
  Most of the times it is because of fear of breaking something during a rolling update. So people usually create a new cluster from scratch and slowly migrate one by one.
- sytringy05 6y ago
  We rebuild ours all the time. New config, k8s version upgrade, node OS patching.

wavesquid 6y ago

This is great! Are other companies doing similar things? e.g. I would love to be able to set up Cloudflare Access for services in k8s

levi_b 6y ago
We just released an operator with similar goals, but it works across clouds. More details here: https://www.pulumi.com/blog/pulumi-kubernetes-operator/ https://www.pulumi.com/blog/pulumi-kubernetes-operator/
sytringy05 6y ago
GCP (Config Connector) and Azure (Service something) both have similar things. I've not heard of it happening outside a managed k8s env.
- harpratap 6y ago
  https://crossplane.io https://crossplane.io is doing a multi-cloud one
  - zxienin 6y ago
    I like their work, but their OAM centricity is too heavy an opiniation.
    - bassamtabbara 6y ago
      disclaimer: I'm a maintainer on Crossplane. OAM is an optional feature of crossplane - you don't have to use it if you don't want to
      - zxienin 6y ago
        Good to know, at least that warm me up to crossplane further. The messaging might need update, including within the docs. I mean - crossplane is the OAM implement - coupled with OAM sprinkled all over docs, gave me very different impression. This aside, I think crossplane work is interesting.

harpratap 6y ago

Glad to see AWS finally embracing Kubernetes too. Google did a similar thing a while back - https://cloud.google.com/config-connector/ https://cloud.google.com/config-connector/ So I guess this solidifies Kubernetes as the defacto standard of Cloud Platforms.

zxienin 6y ago
Azure as well https://github.com/Azure/azure-service-operator https://github.com/Azure/azure-service-operator
- FridgeSeal 6y ago
  Bold of you to assume that: * the magic permissions ghost that runs in Azure whose job it is to inexplicably deny you resources to things won’t interfere * Said Azure service will stay up long enough to be useful * you finish writing the insane amount of config Azure services seemingly require before the heat death of the universe. * Azure decides that it likes you and won’t arbitrarily block you from attaching things to your cluster/nodes because it’s the wrong moon cycle/day of the week/device type/etc * you can somehow navigate Azures kafka-esque documentation got figure out which services you’re actually allowed to do this with. It is only a slight exaggeration to say that Azure is the most painful and frustrating software/cloud product I’ve used in a long time, probably ever, and I earnestly look forward to having literally any excuse to migrate off it.
  - zxienin 6y ago
    > and I earnestly look forward to having literally any excuse to migrate off it Azure’s enterprise sales machine will make sure your wait is long. They sell bulk block deals to CIO, not to devs directly.
  - ithkuil 6y ago
    Give it time, it will get better. When I was forced to use it in 2010 it was horrible; now ten years later is definitely better. In 2030 it will rock!

ponderingfish 6y ago

Orchestration tools are the way forward especially when it comes to on-demand video compression - it's helpful to have the tools to be able to spin up 100s of servers to handle peak loads and then go down to nothing. Kubernetes is so helpful in this.

jtsiskin 6y ago
Would AWS spot instances be useful here?
- big-malloc 6y ago
  Currently the cluster autoscaler supports using a pool of spot instances based on pricing, which is super helpful for test clusters, and there are some other tools available to ensure that you can evict your spot nodes when amazon needs them back
Scaevolus 6y ago
Are you talking about on-demand compression for viewers, or on-demand for uploads? The economics of on-demand transcoding are usually pretty bad in the cloud.

hardwaresofton 6y ago

At the risk of being early, RIP CloudFormation. I posited that this was the benefit in knowing Kubernetes all along, and possibly the ace up GCP's sleeve -- soon no cloud provider will have to offer their own interface, they'll all just offer the one invented by Kubernetes.

delhanty 6y ago
GCP successfully executing a strategy described by Joel Spolsky in 2002 [0]: >Smart companies try to commoditize their products’ complements. [0] Strategy Letter V https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ (edited)
EwanToo 6y ago
You're very early (though maybe not wrong eventually), quoting from the article "ACK is available as a developer preview, supporting the following AWS services: Amazon API Gateway V2 Amazon DynamoDB Amazon ECR Amazon S3 Amazon SNS Amazon SQS" And: "Specifically, in the coming months we plan to focus on: Amazon Relational Database Service (RDS), track via the RDS label. Amazon ElastiCache offers fully managed Redis and Memcached, track via the Elasticache label." In 5 years maybe this will be comparable to Cloudformation, but AWS will likely launch services faster than this adds them.
StreamBright 6y ago
You are assuming a lot here. Kubernetes is not nearly as popular and used as much to justify that we only need that interface. CloudFormation is replaced with Terraform, Ansible and Pulumi much more than it is replaced with Kubernetes.
- hardwaresofton 6y ago
  I am assuming a lot, very possibly super early but I do want to point out that it's terraform/pulumi, and separately ansible (unless you're talking about ansible for provisioning/cloud modules[0] which no one ever seems to know about). But what I'm trying to get at is that the kubernetes interface basically is the terraform/pulumi interface, just with a different configuration language (and less built-in templating macros, which is a GOOD thing, see CloudFormation), but better, because it's continuous. You write a configuration (no matter how dynamic you make things, there's one version that's sent out to AWS/GCP/whatever) with terraform/pulumi, and terraform/pulumi make it true once. You write a relatively similar configuration with k8s and it is made to be true once, but also continuously. As far as popularity goes, kubernetes is insanely popular -- kubernetes was adopted by just about every cloud provider with a speed I've never seen before. Terraform/Pulumi basically took on the work of adapting to cloud provider APIs, not the other way around. [0]: https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html https://docs.ansible.com/ansible/latest/modules/list_of_clou...

zxienin 6y ago

There is now secular push towards use of custom operators instead of OSB. I wonder what finally caused this.

jacques_chester 6y ago
A mix of factors, I think. 1. OSBAPI is not widely known outside of the Cloud Foundry community it came from. In turn that's because Cloud Foundry is not widely known either. Its backers never bothered to market Cloud Foundry or OSBAPI to a wider audience. 2. It imposes a relatively high barrier to entry for implementers. You need to fill in a lot of capabilities before your service can appear in a conformant marketplace. With CRDs you can have a prototype by lunchtime. It might be crappy and you will reinvent a whole bunch of wheels, but the first attempt is easy. 3. Fashion trends. The first appearance of OSBAPI in Kubernetes-land used API aggregation, which was supplanted by CRDs. Later implementations switched to CRDs but by then the ship was already sailing. 4. RDD. You get more points for writing your own freeform controller than for implementing a standard that isn't the latest, coolest, hippest thing. It's very frustrating as an observer without any way to influence events. OSBAPI was an important attempt to save a great deal of pain. It provided a uniform model, so that improvements could be shared amongst all implementations, so that tools could work against standard interfaces in predictable ways, so that end-users had one and only one set of concepts, terms and tools to learn. It also made a crisp division between marketplaces, provisioning and binding. What we have instead is a mess. Everyone writing their own things, their own way. No standards, no uniformity, different terms, different assumptions, different levels of functionality. No meaningful marketplace concept. Provisioning conflated with binding and vice versa. It is a medium-sized disaster, diffuse but very real. And thanks to the marketing genius of enterprise vendors who never saw a short-term buck in broad spectrum developer awareness, it is basically an invisible disaster. What we're heading towards now is seen as normal and ordinary. And it drives me bonkers.
- zxienin 6y ago
  I’d agree on the mess. I also find it on over-engineered side. Do I really need service discovery of services that I already know of, from AWS GCP...?

Niksko 6y ago

The approach of generating the code from the existing Golang API bindings means that hopefully this project will get support for lots of resources pretty quickly. Excited about this, though you do wonder whether it'll suffer the same fate as Cloudformation: the Cloudformation team finds out about new feature launches the same time that the general public does. If the Kubernetes operator lags behind, you're going to have to fall back to something else if you need cutting edge features.

solatic 6y ago

Currently no support for provisioning IAM permissions. ACK will be happy to construct an S3 bucket for you, that is then inaccessible, unless you use dangerous IAM wildcard permissions. Team is concerned about the security ramifications of setting up IAM permissions from ACK: https://github.com/aws/aws-controllers-k8s/issues/22#issuecomment-595816197 https://github.com/aws/aws-controllers-k8s/issues/22#issueco... Look, it'll be great when it matures... but this is very much in the developer preview stage. Caveat emptor.

Legogris 6y ago

Am I the only one who's pessimistic about this? One of the big upsides of Kubernetes is having portable workloads and provisioning cloud-provider specific resources (whose lifecycles very likely outlive clusters!) in Kubernetes just seems wrong to me. Kubernetes is great for managing, orchestrating and directing traffic for containerized workloads but it really shouldn't be The One Tool For Everything. Coupling everything together like this just seems to make things less manageable. IMO infrastructure including managed services are better provisioned through tools like Terraform and Pulumi.

solatic 6y ago
The issue (or benefit, depending on your perspective) with Terraform is that it's a one-shot CLI binary. If you're not running the binary, then it's not doing anything. If you want a long-running daemon that responds to non-human-initiated events, then Terraform isn't a good tool. Any time you try to declaratively define state, if you don't have a daemon enforcing the declarative state, then you will suffer from drift. One approach is the one Terraform has - assume that drift is possible, so ask the user to run a separate plan stage, and manually reconcile drift if needed. Another approach is the controller approach, where the controller tries to do the right thing and errors/alerts if it doesn't know what to do.
- redwood 6y ago
  This is why Hashicorp needs to accelerate their cloud offering. Frankly I get the sense they got a little bit too addicted to central ops driven on-prem style deals for Vault but in the public cloud they need to be front and center with SaaS which is a long road. They have a rudimentary Terraform SaaS I believe but none for Vault as far as I'm aware. I see a lot of folks going straight to cloud provider services because of this. You sum it up well... In these times you don't want to run a daemon
  - t3rabytes 6y ago
    They used to have a managed Vault offering! But then it disappeared one day never to return.

thinkersilver 6y ago

Kubernetes is becoming the lingua-franca of building infrastructure. Through CRDs and the kube api spec I can - start an single application - deploy a set of interconnected apps - define network topologies - define traffic flows - define vertical and horizontal scaling of resources And now I can define AWS resources. This creates an interesting scenario where infrastructure can be defined by the k8s API resources and not necessarily have k8s build it. For example podman starting containers off a K8S deployment spec. It's an API first approach and its great for interoperability. The only downside is managing the yaml and keeping it consistent across the interdependencies.

vtkacenko 6y ago
I’d love an option for the yaml to be changed to something akin to gRPC .proto configs.
brown9-2 6y ago
The Kubernetes resource model is the real lingua franca
soulnothing 6y ago
I really wish fabric8, and more specifically kotlin k8 dsl[2] was getting more traction. It removes the down side of yaml all over the place. It's missing the package management features of helm. But I have several jars acting as baseline deployments, and provisioning. It works really well, and I have an entire language. So I can map over a list of services, instead of do templating. The other big down side is a java run takes a minute or two to kick off I was resilient to k8 for a long time. Complexity was secondary to cost, but Digital ocean has a relatively cheap implementation now. This commonality and perseverance of tooling is great. I want metrics, a simple annotation. I want a secret injected from vault, just add an annotation. It's also cloud agnostic, so this logic can be deployed any where some one provides a k8 offering. EKS was very powerful. As running service accounts via non managed clusters. Removed the need to pass an access key pair to the application. That service account just ran with a corresponding iam role. [1] https://github.com/fabric8io/kubernetes-client https://github.com/fabric8io/kubernetes-client [2] https://github.com/fkorotkov/k8s-kotlin-dsl https://github.com/fkorotkov/k8s-kotlin-dsl

ransom1538 6y ago

Here is my container: run it. Where is my url? The end. No, I don't want Terraforms, puppets, yaml files, load balancers, nodes, pods, k8s, chaos monkeys, Pulumies, pumas, unicorns, trees, portobilities, or shards. I love cloudrun and fargate. Cloudrun has like 5 settings, I wish it had like 2.

throwayws 6y ago
What do you use to provision managed services like rds? Or are you describing your dream UI :)
- ransom1538 6y ago
  Literally that. The actual RDS ui (aws or gcp). I can setup a new db instance in 3 clicks. Slap in the dbs ip into the container.
  - throwayws 6y ago
    Your point makes totally sense. Though I also like to be able to bring up (and destroy) my servers, with a single command, when needed. Going through the same ui every time is a little annoying.
  - marcinzm 6y ago
    That works when you have one container. When you have 200 and fifty databases and four identical environments (prod, qa, staging, dev) and so on it becomes a lot more error prone. These systems exist for those use cases. Granted even with one container and a single DB I found terraform useful as I don't have to fiddle with whatever DB, container, etc. settings there are every time. I also prefer to not have to memorize or figure out a UI I use once every other month. Too many UIs across too many things.
bdcravens 6y ago
So basically, Heroku :-)

1-KB-OK 6y ago

Interesting how they enforce the namespaced scope for ACK custom resources. This is a logical design choice but makes it trickier for operators to use. Say I have an operator watching all namespaces on the cluster. Since operator CRDs are global in scope it makes sense for some operators to be installed as singletons. A CR for this operator gets created in some namespace, and it wants to talk to s3 -- it has to bring along its own s3 credentials and only that CR is allowed to use the s3 bucket? You can imagine a scenario where multiple CRs across namespaces want access to the same s3 bucket.

etxm 6y ago

This is nice from the app manifest perspective because you can declare your database right along side your deployment. The provisioning time of a deployment and an RDS instance is very different though. This is probably most useful when you’re starting a service up for the first time. This is also when it’s not going to work as expected due to that latency of RDS starting up while your app crashed repeatedly waiting for that connection string. This would be really nice for buckets and near instant provisioned resources, but also kinda scary that someone could nuke a whole data store because they got a trigger finger with a weird deployment and deleted and reapplied it. My feelings, they are mixed. :D

toumorokoshi 6y ago

This has been posted a couple times, but GCP has an equivalent that's been around for a while: https://cloud.google.com/config-connector/docs/overview https://cloud.google.com/config-connector/docs/overview disclaimer: I work at GCP.

deleted 6y ago

[deleted]

sytse 6y ago

This feels like Crossplane.io but limited to only AWS. Kelsey seems to think the same https://twitter.com/kelseyhightower/status/1296321377134231552 https://twitter.com/kelseyhightower/status/12963213771342315...

la6471 6y ago

Everything from DNS to AWS SDKs gets reinvented in Kubernetes. It is the most anal approach of Infrastructure design I have ever seen in the last three decades. A good design builds on the things that are already there and does not goes around trying to change every well established protocol in the world.KISS.

sunilkumarc 6y ago

Wow. Now we can directly manage AWS services from Kubernetes. Github: https://aws.github.io/aws-controllers-k8s/ https://aws.github.io/aws-controllers-k8s/ On a different note, recently I was looking to learn AWS concepts through online courses. After so much of research I finally found this e-book on Gumroad which is written by Daniel Vassallo who has worked in AWS team for 10+ years. I found this e-book very helpful as a beginner. This book covers most of the topics that you need to learn to get started: If someone is interested, here is the link :) https://gumroad.com/a/238777459/MsVlG https://gumroad.com/a/238777459/MsVlG

MichaelMoser123 6y ago

Kubernetes is supposed to be cloud vendor agnostic; the cloud vendors counter that by having extension operators to create some tie in to the kubernetes deployment of their making. I guess the 'kubernetes' way would be to create a generalized object for 'object store', that would be implemented by means of s3 on aws and on azure it would be done as blob storage. Now with this approach you can only do the common features between all platforms, you would have a problem with features exclusive to aws for instance, or you would need some mapping from a generalized CRD object to specific implementation of each platform.