4 ms·

We thought about this pattern for a while and ultimately decided we didn't want app compute to directly provision supporting infrastructure. You end up in this

by zxcmx 7y ago

We thought about this pattern for a while and ultimately decided we didn't want app compute to directly provision supporting infrastructure. You end up in this weird place where your compute nodes can, at any time, assume super powerful roles that can change nacls, security groups, completely destroy database clusters or what have you. It feels like running your app as root and database admin, but more. I guess if you completely trust your kube control plane? Like everything I'm sure it has a use somewhere.

captn3m0 7y ago
We're trying it out in a dev-cluster/dev-account setup to let teams provision their own buckets/SQS and some supporting infrastructure easily without AWS access.
- deleted 7y ago
  [deleted]
migueloller 7y ago
With properly configured RBAC this shouldn’t be an issue. The problem is making sure those policies are properly enforced. The tools are there, though.
- deleted 7y ago
  [deleted]
jwhitlark 7y ago
I think these kind of tools are more about provisioning a "normal" cluster from an "admin" cluster. That way the power flows downhill, as it were.
- zxcmx 7y ago
  Yeah I think this could probably work well in gcp. In AWS you end up with a manually built/managed root cluster and if you've automated deploying / upgrading (which you probably should) that then you might as well just use those pipelines and code for all your other clusters too instead of having 1 special thing. If the infrastructure gets complex there will start to be dependencies between stuff, there is definitely a tipping point where you need a real iac pipeline (tf/cf) instead in any case. I think captn3mo proposed one of the more reasonable use cases which is provisioning stuff like queues and dbs for test envs or ci, we ended up stubbing out aws for ci and using the same pipeline code we use to deploy to prod instead for "full fat" testing but I can see where it might make sense to use kube manifests. Probably some creative uses for "deployment per customer" SaaSs too.
joseph 7y ago
The app wouldn't build the infrastructure though, a controller on the cluster would. If you're using e.g. kube2iam or kiam, then only the controller would receive permissions. If you're in AWS and the controller uses CloudFormation to provision the resources, you can define a service role that CloudFormation will assume. Then your controller mainly just needs PassRole permissions for the service role, which means you don't need anything close to "root" privileges on your cluster.

by zxcmx 7y ago

captn3m0 7y ago

We're trying it out in a dev-cluster/dev-account setup to let teams provision their own buckets/SQS and some supporting infrastructure easily without AWS access.

deleted 7y ago
[deleted]

migueloller 7y ago

With properly configured RBAC this shouldn’t be an issue. The problem is making sure those policies are properly enforced. The tools are there, though.

deleted 7y ago
[deleted]

jwhitlark 7y ago

I think these kind of tools are more about provisioning a "normal" cluster from an "admin" cluster. That way the power flows downhill, as it were.

zxcmx 7y ago
Yeah I think this could probably work well in gcp. In AWS you end up with a manually built/managed root cluster and if you've automated deploying / upgrading (which you probably should) that then you might as well just use those pipelines and code for all your other clusters too instead of having 1 special thing. If the infrastructure gets complex there will start to be dependencies between stuff, there is definitely a tipping point where you need a real iac pipeline (tf/cf) instead in any case. I think captn3mo proposed one of the more reasonable use cases which is provisioning stuff like queues and dbs for test envs or ci, we ended up stubbing out aws for ci and using the same pipeline code we use to deploy to prod instead for "full fat" testing but I can see where it might make sense to use kube manifests. Probably some creative uses for "deployment per customer" SaaSs too.

joseph 7y ago

The app wouldn't build the infrastructure though, a controller on the cluster would. If you're using e.g. kube2iam or kiam, then only the controller would receive permissions. If you're in AWS and the controller uses CloudFormation to provision the resources, you can define a service role that CloudFormation will assume. Then your controller mainly just needs PassRole permissions for the service role, which means you don't need anything close to "root" privileges on your cluster.