5 ms·

I always smile at posts like this. They're right and wrong at the same time. Systems should be "as simple as possible, but no simpler". And thinking that you ca

I always smile at posts like this. They're right and wrong at the same time. Systems should be "as simple as possible, but no simpler". And thinking that you can gloss over the detail is just going to create more hassle later on. IAM is just complex. I can't think of any implementation of "users, groups, roles, policies, identity providers, oidc" that is truly simple. I'm reminded of a guy I worked with, who fought against Kubernetes adoption because it was "too complex", only to slowly reinvent Kubernetes badly, adhoc, out of vault, consul, systemd, nomad, iscsi, ansible, jenkins, puppet, bash, spit, glue... making lots of mistakes along the way. You think you don't need to implement some feature until you do. Another thing I'll say about AWS (having been the sole infra guy at a few startups) is that it's well within most people's abilities to learn it. And you can usually avoid the shitty stuff. You think lambdas stuck? Don't use them! You could use EKS, ECS or bare EC2.

cyberpunk 5mo ago
Another point is that while it can be more expensive than self hosting, the savings are dwarfed by the engineering costs. A decent infrastructure engineer working for 2 man months on your “money saving” ovh setup costs you more than you can possibly save by not just using fargate or rds whatever.
- noprocrasted 5mo ago
  But unless you're on a PaaS, you have "infrastructure engineers" already. So why not at least let them make back their salary by making them built a cost-efficient infrastructure?
- izacus 5mo ago
  This is rarely actually true but it's a common falsehood told by people who have financial interest of keeping everyone on AWS. And that includes engkneers that only know how to use AWS and are terrified at having to learn something else.
- amluto 5mo ago
  How much would you pay for 2 months of infrastructure engineer time? And how many millions / tens of millions / hundreds of millions are you imagining being spent on overpriced AWS services? (Also, those AWS services are not engineering-free. I tried to migrate a system to RDS once and gave up after quite a few hours when I got to the part of the documentation that suggested that I edit my sql dump using sed to get it into a form that RDS would accept. No, thanks.)
le-mark 5mo ago
> fought against X adoption because it was "too complex", only to slowly reinvent X badly This is a surprisingly common pattern in technology and software. Some things are definitively the “standard” at this point yet so many people simply refuse to spend the time to properly learn them.
- ipsento606 5mo ago
  > This is a surprisingly common pattern in technology and software. Some things are definitively the “standard” It is also a surprisingly common pattern to adopt very complicated solutions for applications that are never going to need them ultimately it is not possible to come up with a "standard" that is an acceptable replacement for good judgement
Aperocky 5mo ago
Some internal perspective - IAM has maybe thousands of options but fundamentally it is "what does this role have access to doing (action + resource)" + "who has access to this role". That is really it from a 10k foot level. IAM is great because it applies internally just like it does externally. The internal AWS team don't get more access than you do, and if we get access to do certain thing on your account to perform specific service that's because you have a service principle in your IAM trust relationship that allowed us access, that you can see, and audit. For instance, lambdas have lambda role because you don't want lambda service just reading your S3 buckets because "we're AWS we automatically get access", you can absolutely see and control access, even if it is internal to AWS.
- paulddraper 5mo ago
  > that's because you have a service principle in your IAM trust relationship that allowed us access That’s why it’s so complicated!!! I don’t understand how I should evaluate trust for your internal EBS org versus your internal ALB org. I kinda just expect it to be all “AWS” trust. And it’s all garbage anyway. There’s no way I can prevent the hypothetically untrustworthy EBS team from surreptitiously adding charges to my account if they want to. Right? This would maybe make some sense if I could top level turn off/on services, but that isn’t how it works. — I have no doubt this makes some sense from someone inside the machine, but from the outside it’s not helpful nor useful.
  - jimbobimbo 5mo ago
    >I kinda just expect it to be all “AWS” trust. This would be very unwise from security standpoint. Internal access to customer stuff is granular and made hard for internal staff to gain, to minimize chances of screw up intentional or not.
    - NewJazz 5mo ago
      I agree. Adding a service principal always raises an eyebrow for me, just a blanket "hey we're aws trust me bro" is a little bonkers.
hedora 5mo ago
IAM is unnecessarily bad. I recently had to set a trivial policy, and was doing it correctly. The console kept warning me that I was giving root AWS access to my external application because they want people to use the locked in AWS path, and I was running off cloud. On top of that, they break copy paste on the web console, so you can’t just ctrl-c ctrl-v and then ask Claude to explain their WTF-ery. Instead, you have to OCR or send a PNG. I honestly did not think they could make IAM worse, yet here we are. Bastards.
- hedora 5mo ago
  I guess I should also point out that I’ve used AWS at extremely large scale in the past, which is why I’m running this subproject on another cloud. As for simple permissions, go read the UNIX paper. It spends a page or two on their approach and is all you need. Then, read the paper on mapping between NTFS SMB ACLs and NFS. It’s either impossible or undecidable, depending on the deployment. IAM is from the windows acl lineage which is known pessimal from a usability and security perspective.
  - cyberax 5mo ago
    IAM is NOT from any lineage. It has grown organically and is complicated, just as any other policy language. AWS even uses an automatic proof assistant to verify IAM policies. However, the secret to IAM in AWS is to NOT use IAM. Just create separate AWS accounts for separate services and only share whatever resources are needed. Then you can have dead simple IAM policies because you won't need to do granular permissions ("AWS role X can access database Y").
    - dotwaffle 5mo ago
      > Just create separate AWS accounts for separate services My understanding is that different AWS accounts have different mappings of availability zones, so it's very easy to suddenly find yourself with an unexpected bandwidth bill due to all the cross-az traffic. I've been irritated at AWS (and the other large cloud providers) that they charge $0.01/GB for cross-az traffic. That's $3.24/Mbps -- about the same I was paying for internet transit (as in: from London to anywhere in the world) 20 years ago, and this is just between two datacenters in the same city controlled by the same organisation, markup must be 10,000x or more considering these places are cross-connected with massive bundles of fiber!
Aeolun 5mo ago
I think the big problems with amazon IAM is not that it’s inherently complex, it’s that every team in AWS came up with their own way to define permissions and the calls these allow you to make. So the API Gateway set of permissions uses a completely different method for no discernable reason.
hunterpayne 5mo ago
"who fought against Kubernetes adoption because it was "too complex", only to slowly reinvent Kubernetes badly," If you are dynamically scaling a set of web services sure. The problem is that people use k8s for running batch pipelines and streaming analytic services and a bunch of other things too. And k8s is terrible at doing those things and entirely too complex. And if you don't have to scale your web services very often, then k8s is a waste in that case too. Its a right tool for the job and k8s's job isn't deploying to the cloud, its dynamically scaling a website.

cyberpunk 5mo ago

Another point is that while it can be more expensive than self hosting, the savings are dwarfed by the engineering costs. A decent infrastructure engineer working for 2 man months on your “money saving” ovh setup costs you more than you can possibly save by not just using fargate or rds whatever.

noprocrasted 5mo ago
But unless you're on a PaaS, you have "infrastructure engineers" already. So why not at least let them make back their salary by making them built a cost-efficient infrastructure?
izacus 5mo ago
This is rarely actually true but it's a common falsehood told by people who have financial interest of keeping everyone on AWS. And that includes engkneers that only know how to use AWS and are terrified at having to learn something else.
amluto 5mo ago
How much would you pay for 2 months of infrastructure engineer time? And how many millions / tens of millions / hundreds of millions are you imagining being spent on overpriced AWS services? (Also, those AWS services are not engineering-free. I tried to migrate a system to RDS once and gave up after quite a few hours when I got to the part of the documentation that suggested that I edit my sql dump using sed to get it into a form that RDS would accept. No, thanks.)

le-mark 5mo ago

> fought against X adoption because it was "too complex", only to slowly reinvent X badly This is a surprisingly common pattern in technology and software. Some things are definitively the “standard” at this point yet so many people simply refuse to spend the time to properly learn them.

ipsento606 5mo ago
> This is a surprisingly common pattern in technology and software. Some things are definitively the “standard” It is also a surprisingly common pattern to adopt very complicated solutions for applications that are never going to need them ultimately it is not possible to come up with a "standard" that is an acceptable replacement for good judgement

Aperocky 5mo ago

Some internal perspective - IAM has maybe thousands of options but fundamentally it is "what does this role have access to doing (action + resource)" + "who has access to this role". That is really it from a 10k foot level. IAM is great because it applies internally just like it does externally. The internal AWS team don't get more access than you do, and if we get access to do certain thing on your account to perform specific service that's because you have a service principle in your IAM trust relationship that allowed us access, that you can see, and audit. For instance, lambdas have lambda role because you don't want lambda service just reading your S3 buckets because "we're AWS we automatically get access", you can absolutely see and control access, even if it is internal to AWS.

paulddraper 5mo ago
> that's because you have a service principle in your IAM trust relationship that allowed us access That’s why it’s so complicated!!! I don’t understand how I should evaluate trust for your internal EBS org versus your internal ALB org. I kinda just expect it to be all “AWS” trust. And it’s all garbage anyway. There’s no way I can prevent the hypothetically untrustworthy EBS team from surreptitiously adding charges to my account if they want to. Right? This would maybe make some sense if I could top level turn off/on services, but that isn’t how it works. — I have no doubt this makes some sense from someone inside the machine, but from the outside it’s not helpful nor useful.
- jimbobimbo 5mo ago
  >I kinda just expect it to be all “AWS” trust. This would be very unwise from security standpoint. Internal access to customer stuff is granular and made hard for internal staff to gain, to minimize chances of screw up intentional or not.
  - NewJazz 5mo ago
    I agree. Adding a service principal always raises an eyebrow for me, just a blanket "hey we're aws trust me bro" is a little bonkers.

hedora 5mo ago

IAM is unnecessarily bad. I recently had to set a trivial policy, and was doing it correctly. The console kept warning me that I was giving root AWS access to my external application because they want people to use the locked in AWS path, and I was running off cloud. On top of that, they break copy paste on the web console, so you can’t just ctrl-c ctrl-v and then ask Claude to explain their WTF-ery. Instead, you have to OCR or send a PNG. I honestly did not think they could make IAM worse, yet here we are. Bastards.

hedora 5mo ago
I guess I should also point out that I’ve used AWS at extremely large scale in the past, which is why I’m running this subproject on another cloud. As for simple permissions, go read the UNIX paper. It spends a page or two on their approach and is all you need. Then, read the paper on mapping between NTFS SMB ACLs and NFS. It’s either impossible or undecidable, depending on the deployment. IAM is from the windows acl lineage which is known pessimal from a usability and security perspective.
- cyberax 5mo ago
  IAM is NOT from any lineage. It has grown organically and is complicated, just as any other policy language. AWS even uses an automatic proof assistant to verify IAM policies. However, the secret to IAM in AWS is to NOT use IAM. Just create separate AWS accounts for separate services and only share whatever resources are needed. Then you can have dead simple IAM policies because you won't need to do granular permissions ("AWS role X can access database Y").
  - dotwaffle 5mo ago
    > Just create separate AWS accounts for separate services My understanding is that different AWS accounts have different mappings of availability zones, so it's very easy to suddenly find yourself with an unexpected bandwidth bill due to all the cross-az traffic. I've been irritated at AWS (and the other large cloud providers) that they charge $0.01/GB for cross-az traffic. That's $3.24/Mbps -- about the same I was paying for internet transit (as in: from London to anywhere in the world) 20 years ago, and this is just between two datacenters in the same city controlled by the same organisation, markup must be 10,000x or more considering these places are cross-connected with massive bundles of fiber!

Aeolun 5mo ago

I think the big problems with amazon IAM is not that it’s inherently complex, it’s that every team in AWS came up with their own way to define permissions and the calls these allow you to make. So the API Gateway set of permissions uses a completely different method for no discernable reason.

hunterpayne 5mo ago

"who fought against Kubernetes adoption because it was "too complex", only to slowly reinvent Kubernetes badly," If you are dynamically scaling a set of web services sure. The problem is that people use k8s for running batch pipelines and streaming analytic services and a bunch of other things too. And k8s is terrible at doing those things and entirely too complex. And if you don't have to scale your web services very often, then k8s is a waste in that case too. Its a right tool for the job and k8s's job isn't deploying to the cloud, its dynamically scaling a website.