4 ms·

I hope this will allow us to contribute toward fixing some of the longstanding issues. One that I'm still surprised hasn't been addressed is the way that it gra

I hope this will allow us to contribute toward fixing some of the longstanding issues. One that I'm still surprised hasn't been addressed is the way that it grants irrevocable admin permissions for the IAM security principal that created the cluster. And it does it invisibily–there's no way to see what IAM role or user has system:masters since the permission doesn't appear in the configmap or any other user-visible configuration. Last I checked you can't even see it in the dashboard and have to make note of it yourself for future reference.

inyorgroove 5y ago
Its even worse than you describe, its by name. It doesn't have to be the same IAM user, that user just have to have the same username as the original creator. Support told me this when we got locked out of our cluster and the creator had moved on to other things.
- raffraffraff 5y ago
  Oh. Wow. That is honestly insane.
- brazzledazzle 5y ago
  Yeah, unfortunately for security minded folks the unique ID of the IAM security principal isn't used instead of (or in conjunction with) the ARN. Or perhaps fortunately if you are blindsided by a security setting that makes you wonder how thorny it is on the backend if this shipped to production in the first place. I experienced something similar. We tried recreating it on a whim thinking it wouldn't work but needless to say everyone on that call was surprised. I know they did (do?) some weird stuff exposing the control plane with VPC Endoints that seem like some kind of weird hybrid of AWS-managed endpoints and privatelink. I always wondered how scrappy the team had to be at the beginning when they were still pushing ECS hard.
  - bogomipz 5y ago
    Agreed. This is extra bizarre considering their own documentation on the purpose of IAM unique IDS states: >"However, every IAM user has a unique ID, even if you create a new IAM user that reuses a friendly name you deleted before. In the example, the old IAM user David and the new IAM user David have different unique IDs. You can create resource-based policies that grant access by unique ID and not just by user name. Doing so reduces the chance that you could inadvertently grant access to information that an employee should not have"[1] [1] https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-unique-ids https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_i...
gizdan 5y ago
My searches on this are coming up empty. Do have more info or perhaps a link that further explains this issue?
- arianvanp 5y ago
  https://aws.github.io/aws-eks-best-practices/security/docs/iam/#use-iam-roles-when-multiple-users-need-identical-access-to-the-cluster https://aws.github.io/aws-eks-best-practices/security/docs/i...
  - Jenk 5y ago
    To save a click/scroll/combination-there-of: https://aws.github.io/aws-eks-best-practices/security/docs/iam/#create-the-cluster-with-a-dedicated-iam-role https://aws.github.io/aws-eks-best-practices/security/docs/i... > When you create an Amazon EKS cluster, the IAM entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster's RBAC configuration. This access cannot be removed and is not managed through the aws-auth ConfigMap. Therefore it is a good idea to create the cluster with a dedicated IAM role and regularly audit who can assume this role. This role should not be used to perform routine actions on the cluster, and instead additional users should be granted access to the cluster through the aws-auth ConfigMap for this purpose. After the aws-auth ConfigMap is configured, the role can be deleted and only recreated in an emergency / break glass scenario where the aws-auth ConfigMap is corrupted and the cluster is otherwise inaccessible. This can be particularly useful in production clusters which do not usually have direct user access configured. Emphasis mine.
    - gizdan 5y ago
      Thank you!
micah_chatt 5y ago
EKS Engineer here. This is something we have lots of thoughts about and want to make better in EKS on AWS. For EKS Anywhere, you can configure your cluster with OIDC auth today, and IAM auth is coming very soon. https://github.com/aws/eks-anywhere/issues/90 https://github.com/aws/eks-anywhere/issues/90
- bogomipz 5y ago
  Could you shed some light on why this was done this way? Was there a technical reason for not uasing ARNs or unique IDs?

inyorgroove 5y ago

Its even worse than you describe, its by name. It doesn't have to be the same IAM user, that user just have to have the same username as the original creator. Support told me this when we got locked out of our cluster and the creator had moved on to other things.

raffraffraff 5y ago
Oh. Wow. That is honestly insane.
brazzledazzle 5y ago
Yeah, unfortunately for security minded folks the unique ID of the IAM security principal isn't used instead of (or in conjunction with) the ARN. Or perhaps fortunately if you are blindsided by a security setting that makes you wonder how thorny it is on the backend if this shipped to production in the first place. I experienced something similar. We tried recreating it on a whim thinking it wouldn't work but needless to say everyone on that call was surprised. I know they did (do?) some weird stuff exposing the control plane with VPC Endoints that seem like some kind of weird hybrid of AWS-managed endpoints and privatelink. I always wondered how scrappy the team had to be at the beginning when they were still pushing ECS hard.
- bogomipz 5y ago
  Agreed. This is extra bizarre considering their own documentation on the purpose of IAM unique IDS states: >"However, every IAM user has a unique ID, even if you create a new IAM user that reuses a friendly name you deleted before. In the example, the old IAM user David and the new IAM user David have different unique IDs. You can create resource-based policies that grant access by unique ID and not just by user name. Doing so reduces the chance that you could inadvertently grant access to information that an employee should not have"[1] [1] https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-unique-ids https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_i...

gizdan 5y ago

My searches on this are coming up empty. Do have more info or perhaps a link that further explains this issue?

arianvanp 5y ago
https://aws.github.io/aws-eks-best-practices/security/docs/iam/#use-iam-roles-when-multiple-users-need-identical-access-to-the-cluster https://aws.github.io/aws-eks-best-practices/security/docs/i...
- Jenk 5y ago
  To save a click/scroll/combination-there-of: https://aws.github.io/aws-eks-best-practices/security/docs/iam/#create-the-cluster-with-a-dedicated-iam-role https://aws.github.io/aws-eks-best-practices/security/docs/i... > When you create an Amazon EKS cluster, the IAM entity user or role, such as a federated user that creates the cluster, is automatically granted system:masters permissions in the cluster's RBAC configuration. This access cannot be removed and is not managed through the aws-auth ConfigMap. Therefore it is a good idea to create the cluster with a dedicated IAM role and regularly audit who can assume this role. This role should not be used to perform routine actions on the cluster, and instead additional users should be granted access to the cluster through the aws-auth ConfigMap for this purpose. After the aws-auth ConfigMap is configured, the role can be deleted and only recreated in an emergency / break glass scenario where the aws-auth ConfigMap is corrupted and the cluster is otherwise inaccessible. This can be particularly useful in production clusters which do not usually have direct user access configured. Emphasis mine.
  - gizdan 5y ago
    Thank you!

micah_chatt 5y ago

EKS Engineer here. This is something we have lots of thoughts about and want to make better in EKS on AWS. For EKS Anywhere, you can configure your cluster with OIDC auth today, and IAM auth is coming very soon. https://github.com/aws/eks-anywhere/issues/90 https://github.com/aws/eks-anywhere/issues/90

bogomipz 5y ago
Could you shed some light on why this was done this way? Was there a technical reason for not uasing ARNs or unique IDs?