4 ms·

No this is exactly how it works. There’s technically ideal policy and a bunch of exceptions which are behind a “break glass in case of emergency” account. Reali

by gw98 4y ago

No this is exactly how it works. There’s technically ideal policy and a bunch of exceptions which are behind a “break glass in case of emergency” account. Realistically nothing works however many hours you spend in IAM unless you use the emergency account so that’s what everyone does.

ownagefool 4y ago
Right. In practice, you manually bootstrap your CI/CD, and if you really want to be secret squirrel secure, you can audit that. Everything going forward gets to be commited via code, peer review, least privledged access, and is automatically deployed by the robot. At that point, you might hand out some break glass credentials for a few items, or when your CI/CD goes down, but usage of that should alert and be manually reviewed. You now have a pretty decent setup, as everything is checked.
- gw98 4y ago
  That's what we say we do. Reality is that most of the time the entire team is logged in via delegated admin accounts working out what the automation fucked up or deleting resources which don't want to go away when the automation craps out or futzing with things that are broken or missing in Terraform. Fully driving a cloud via code is a complete lie.
  - ownagefool 4y ago
    I don't think it's a lie; just a pain in the arse. I've worked in, and defined, secure environments where your only option is you need to raise another PR. For me this often degenerates into the PR spinning up a copy in another account, and the PR history being "test, test, test, ffs, test". Like this: https://github.com/secureweb/homelab/commits/parca https://github.com/secureweb/homelab/commits/parca The more practical way around the problem is to have non-production accounts where you can delegate admin, without the risk of the production and tooling accounts, and use that to make sure your code works before you actually raise the PR and merge. It's prudent to remember that such security should be risk based, so whilst there's risk that you leak non-prod credentials to a crypto farmer, the risk of your teams being non-productive is greater. Largely agree with you though. I usually just run k8s environments and make concession to use the cloud stuff when it makes sense. Not being able to run locally is akin to punching yourself in the nuts. Creating a namespace in k8s and deploying ephemeral tests either locally or in a dev cluster is better experience in my opinion. But you do have a lot of folks who treat their cloud provider like their favourite sports team.
  - oneplane 4y ago
    We have a 20:1 ratio of developers and platform engineers where developers cannot do anything in AWS except via Infrastructure as Code. Works very well. Yes, you do need to make sure you only platform things you actually need, and not try to abstract the entire service catalog. But if you just pick 4 RDBMS options, 2 Document-oriented options, 2 queues, 2 PubSubs, 1 object store, 1 traffic management option, and one catch-all observability option, you really don't need to do all that much over time. Granted, this doesn't work for small-scale stuff or immature organisations.
icedchai 4y ago
IAM is a big pain. When deploying "something new" (say a lambda executing another lambda that accesses dynamo...) I spend more timing screwing around with terraform to configure IAM roles, permissions, and other garbage than I do actually debugging the code. Many junior developers don't understand it at all. "It works with my credentials!" (Of course it does: you're running it with admin access in your own dev account.)

by gw98 4y ago

ownagefool 4y ago

Right. In practice, you manually bootstrap your CI/CD, and if you really want to be secret squirrel secure, you can audit that. Everything going forward gets to be commited via code, peer review, least privledged access, and is automatically deployed by the robot. At that point, you might hand out some break glass credentials for a few items, or when your CI/CD goes down, but usage of that should alert and be manually reviewed. You now have a pretty decent setup, as everything is checked.

gw98 4y ago
That's what we say we do. Reality is that most of the time the entire team is logged in via delegated admin accounts working out what the automation fucked up or deleting resources which don't want to go away when the automation craps out or futzing with things that are broken or missing in Terraform. Fully driving a cloud via code is a complete lie.
- ownagefool 4y ago
  I don't think it's a lie; just a pain in the arse. I've worked in, and defined, secure environments where your only option is you need to raise another PR. For me this often degenerates into the PR spinning up a copy in another account, and the PR history being "test, test, test, ffs, test". Like this: https://github.com/secureweb/homelab/commits/parca https://github.com/secureweb/homelab/commits/parca The more practical way around the problem is to have non-production accounts where you can delegate admin, without the risk of the production and tooling accounts, and use that to make sure your code works before you actually raise the PR and merge. It's prudent to remember that such security should be risk based, so whilst there's risk that you leak non-prod credentials to a crypto farmer, the risk of your teams being non-productive is greater. Largely agree with you though. I usually just run k8s environments and make concession to use the cloud stuff when it makes sense. Not being able to run locally is akin to punching yourself in the nuts. Creating a namespace in k8s and deploying ephemeral tests either locally or in a dev cluster is better experience in my opinion. But you do have a lot of folks who treat their cloud provider like their favourite sports team.
- oneplane 4y ago
  We have a 20:1 ratio of developers and platform engineers where developers cannot do anything in AWS except via Infrastructure as Code. Works very well. Yes, you do need to make sure you only platform things you actually need, and not try to abstract the entire service catalog. But if you just pick 4 RDBMS options, 2 Document-oriented options, 2 queues, 2 PubSubs, 1 object store, 1 traffic management option, and one catch-all observability option, you really don't need to do all that much over time. Granted, this doesn't work for small-scale stuff or immature organisations.

icedchai 4y ago

IAM is a big pain. When deploying "something new" (say a lambda executing another lambda that accesses dynamo...) I spend more timing screwing around with terraform to configure IAM roles, permissions, and other garbage than I do actually debugging the code. Many junior developers don't understand it at all. "It works with my credentials!" (Of course it does: you're running it with admin access in your own dev account.)