7 ms·

Kubernetes Config Connector: Provision GCP Infrastructure Using Kubernetes

55 points by migueloller 7y ago

ksajadi 7y ago
I wonder if the k8s style infrastructure definition becomes the norm, what's going to be the future like for tools like Terraform
- zxcmx 7y ago
  Provisioning cloud stuff ends up being about the relationships between all the things. Plus a bunch of dirty domain knowledge about retries and resource schemas. It's very difficult to delete infra in AWS without a resource graph once you have enough things that refer to each other. You can only get so far with a bunch of isolated yaml manifests for single resources. So the MVP of anything that can solve the general case basically ends up doing the exact same things as terraform... and might as well actually be terraform.
  - ec109685 7y ago
    Can’t the operator that expresses the state of cloud resources take into account ordering and dirty tricks?
ec109685 7y ago
AWS started building something similar but momentum seems to have stopped once the developer left: https://aws.amazon.com/blogs/opensource/aws-service-operator-kubernetes-available/ https://aws.amazon.com/blogs/opensource/aws-service-operator... It is too bad because declarative specifications and control loops to have state converge to what it should be is a nice pattern. Keeping up with all of AWS’s surface area though is hard if their heart isn’t in it.
- zxcmx 7y ago
  We thought about this pattern for a while and ultimately decided we didn't want app compute to directly provision supporting infrastructure. You end up in this weird place where your compute nodes can, at any time, assume super powerful roles that can change nacls, security groups, completely destroy database clusters or what have you. It feels like running your app as root and database admin, but more. I guess if you completely trust your kube control plane? Like everything I'm sure it has a use somewhere.
  - captn3m0 7y ago
    We're trying it out in a dev-cluster/dev-account setup to let teams provision their own buckets/SQS and some supporting infrastructure easily without AWS access.
    - deleted 7y ago
      [deleted]
  - migueloller 7y ago
    With properly configured RBAC this shouldn’t be an issue. The problem is making sure those policies are properly enforced. The tools are there, though.
    - deleted 7y ago
      [deleted]
  - jwhitlark 7y ago
    I think these kind of tools are more about provisioning a "normal" cluster from an "admin" cluster. That way the power flows downhill, as it were.
    - zxcmx 7y ago
      Yeah I think this could probably work well in gcp. In AWS you end up with a manually built/managed root cluster and if you've automated deploying / upgrading (which you probably should) that then you might as well just use those pipelines and code for all your other clusters too instead of having 1 special thing. If the infrastructure gets complex there will start to be dependencies between stuff, there is definitely a tipping point where you need a real iac pipeline (tf/cf) instead in any case. I think captn3mo proposed one of the more reasonable use cases which is provisioning stuff like queues and dbs for test envs or ci, we ended up stubbing out aws for ci and using the same pipeline code we use to deploy to prod instead for "full fat" testing but I can see where it might make sense to use kube manifests. Probably some creative uses for "deployment per customer" SaaSs too.
ec109685 7y ago
Why doesn’t GCP use AWS’s feature announcement style and have a little blog written that walks the developer through how to use it? So much better to get an immediate feel for a service.
- spew 7y ago
  The getting started guide may be what you are looking for: https://cloud.google.com/config-connector/docs/how-to/getting-started https://cloud.google.com/config-connector/docs/how-to/gettin...
- jvolkman 7y ago
  There may have been one, but this isn't new. Here's a video covering it at GCP Next in April: https://www.youtube.com/watch?v=s_hiFuRDJSE https://www.youtube.com/watch?v=s_hiFuRDJSE
- alpb 7y ago
  Mostly because the service is still beta.
jbigelow76 7y ago
Spin up a k8s cluster to manage your other resources that will be used by your... other k8s clusters. Is kubernetes just one big snake eating its own tail now?
- tyingq 7y ago
  The movie "Inception" popped into my head after seeing the headline. Managing k8s version upgrades across both sides is probably fun.
- srmatto 7y ago
  Kubernetes Gardener uses a main cluster to provision and maintain child clusters. https://gardener.cloud/ https://gardener.cloud/
tracker1 7y ago
Definitely cool... although AWS is really entrenched and Azure's getting there too for a lot of orgs. Especially historically MS oriented shops and those with some apps tethered to windows where others can be containerized.
leg100 7y ago
K8s is in one sense a convergence engine, bringing container resources across a cluster into line with a given desired state. So why not cloud resources too? Having to run a cluster to do so doesn't feel quite right, however. But then if you're using Terraform - which is the most popular "cloud convergence engine" - you've probably provisioned compute resources with which to run Terraform. So not that strange.
robszumski 7y ago
I am really interested in using this GCP Operator and the corresponding ones from AWS and Azure to replace functionality from Open Service Brokers with a more Kubernetes-native experience. Even better, this model uses the RBAC, namespaces, quotas and tools that developers are already using.
Yeroc 7y ago
Sounds similar in concept to Crossplane (https://crossplane.io/ https://crossplane.io/) except that Crossplane is trying to be cloud-agnostic?
- techthumb 7y ago
  I wonder if Crossplane will use this behind the scenes.
farisjarrah 7y ago
Just gotta say, this looks much friendlier then Google CFT or Deployment Manager tools that they were pushing before.

55 points by migueloller 7y ago

I wonder if the k8s style infrastructure definition becomes the norm, what's going to be the future like for tools like Terraform

zxcmx 7y ago
Provisioning cloud stuff ends up being about the relationships between all the things. Plus a bunch of dirty domain knowledge about retries and resource schemas. It's very difficult to delete infra in AWS without a resource graph once you have enough things that refer to each other. You can only get so far with a bunch of isolated yaml manifests for single resources. So the MVP of anything that can solve the general case basically ends up doing the exact same things as terraform... and might as well actually be terraform.
- ec109685 7y ago
  Can’t the operator that expresses the state of cloud resources take into account ordering and dirty tricks?

ec109685 7y ago

AWS started building something similar but momentum seems to have stopped once the developer left: https://aws.amazon.com/blogs/opensource/aws-service-operator-kubernetes-available/ https://aws.amazon.com/blogs/opensource/aws-service-operator... It is too bad because declarative specifications and control loops to have state converge to what it should be is a nice pattern. Keeping up with all of AWS’s surface area though is hard if their heart isn’t in it.

zxcmx 7y ago
We thought about this pattern for a while and ultimately decided we didn't want app compute to directly provision supporting infrastructure. You end up in this weird place where your compute nodes can, at any time, assume super powerful roles that can change nacls, security groups, completely destroy database clusters or what have you. It feels like running your app as root and database admin, but more. I guess if you completely trust your kube control plane? Like everything I'm sure it has a use somewhere.
- captn3m0 7y ago
  We're trying it out in a dev-cluster/dev-account setup to let teams provision their own buckets/SQS and some supporting infrastructure easily without AWS access.
  - deleted 7y ago
    [deleted]
- migueloller 7y ago
  With properly configured RBAC this shouldn’t be an issue. The problem is making sure those policies are properly enforced. The tools are there, though.
  - deleted 7y ago
    [deleted]
- jwhitlark 7y ago
  I think these kind of tools are more about provisioning a "normal" cluster from an "admin" cluster. That way the power flows downhill, as it were.
  - zxcmx 7y ago
    Yeah I think this could probably work well in gcp. In AWS you end up with a manually built/managed root cluster and if you've automated deploying / upgrading (which you probably should) that then you might as well just use those pipelines and code for all your other clusters too instead of having 1 special thing. If the infrastructure gets complex there will start to be dependencies between stuff, there is definitely a tipping point where you need a real iac pipeline (tf/cf) instead in any case. I think captn3mo proposed one of the more reasonable use cases which is provisioning stuff like queues and dbs for test envs or ci, we ended up stubbing out aws for ci and using the same pipeline code we use to deploy to prod instead for "full fat" testing but I can see where it might make sense to use kube manifests. Probably some creative uses for "deployment per customer" SaaSs too.

ec109685 7y ago

Why doesn’t GCP use AWS’s feature announcement style and have a little blog written that walks the developer through how to use it? So much better to get an immediate feel for a service.

spew 7y ago
The getting started guide may be what you are looking for: https://cloud.google.com/config-connector/docs/how-to/getting-started https://cloud.google.com/config-connector/docs/how-to/gettin...
jvolkman 7y ago
There may have been one, but this isn't new. Here's a video covering it at GCP Next in April: https://www.youtube.com/watch?v=s_hiFuRDJSE https://www.youtube.com/watch?v=s_hiFuRDJSE
alpb 7y ago
Mostly because the service is still beta.

jbigelow76 7y ago

Spin up a k8s cluster to manage your other resources that will be used by your... other k8s clusters. Is kubernetes just one big snake eating its own tail now?

tyingq 7y ago
The movie "Inception" popped into my head after seeing the headline. Managing k8s version upgrades across both sides is probably fun.
srmatto 7y ago
Kubernetes Gardener uses a main cluster to provision and maintain child clusters. https://gardener.cloud/ https://gardener.cloud/

tracker1 7y ago

Definitely cool... although AWS is really entrenched and Azure's getting there too for a lot of orgs. Especially historically MS oriented shops and those with some apps tethered to windows where others can be containerized.

leg100 7y ago

K8s is in one sense a convergence engine, bringing container resources across a cluster into line with a given desired state. So why not cloud resources too? Having to run a cluster to do so doesn't feel quite right, however. But then if you're using Terraform - which is the most popular "cloud convergence engine" - you've probably provisioned compute resources with which to run Terraform. So not that strange.

robszumski 7y ago

I am really interested in using this GCP Operator and the corresponding ones from AWS and Azure to replace functionality from Open Service Brokers with a more Kubernetes-native experience. Even better, this model uses the RBAC, namespaces, quotas and tools that developers are already using.

Yeroc 7y ago

Sounds similar in concept to Crossplane (https://crossplane.io/ https://crossplane.io/) except that Crossplane is trying to be cloud-agnostic?

techthumb 7y ago
I wonder if Crossplane will use this behind the scenes.

farisjarrah 7y ago

Just gotta say, this looks much friendlier then Google CFT or Deployment Manager tools that they were pushing before.