6 ms·

Why is debugging IAM policies and auth issues such complete dogshit on AWS? Seriously, if you want to be a competitor, focus on this. I've wasted hours trying

Why is debugging IAM policies and auth issues such complete dogshit on AWS? Seriously, if you want to be a competitor, focus on this. I've wasted hours trying to work out why I get an unauthorised error, and the official docs say to manually pick your way through the 8 or so different policies that might apply (SCPs, IAM, resource policies, etc). Yeah right. When you eventually find the page showing how to use Athena with Cloudtrail, you discover half the requests are inexplicably missing. And even though error messages include a request ID, you can't easily query for them, if it's even possible. Maybe it is and I just haven't discovered the right athena incantation. They should make it easy to just query by request ID and get clear messages telling you what policy denied the request. It's a total train wreck from start to finish. I guess by keeping such a shitshow they sell more support contracts though.

trevorsullivan 3y ago
Pretty much everything in AWS APIs is a trainwreck. They mindless wrap objects other objects, for seemingly no reason. The customer bears the burden of learning and implementing the extra logic for complex data structures. It's incredibly annoying. They need to pair up UX designers with engineers, but they don't.
benterix 3y ago
A cynic in me is happy about this state of affairs - the companies who happily joined the AWS bandwagon have to pay the price (including my salary). For my projects I almost always choose something else that offers better value, except in very rare cases when it makes sense. A decade ago or so AWS was marketed as - to simplify - "fire all your sysadmins because we manage the systems for you" to the current state where good AWS DevOps engineers are expensive and not so easy to find and you pay through the nose if you really follow AWS recommendations, especially for larger setups.
- nprateem 3y ago
  Pretty tedious work though. I'd far rather feel I was actually doing useful work instead of dealing with this stupid crap.
- steveBK123 3y ago
  Yeah this has been my observation. CTOs bought the "fire all your sysadmins" literally, and now we have a 20% overhead on all app devs who are supposed to learn all this stuff. It's crazy to me. It's literally the opposite of every management training / book about focussing on highest value work / where you add value / where you provide expertise.
  - renegade-otter 3y ago
    20%? Seems like at least half of all the work has to do with infrastructure plumbing, now that people who are supposed to write product code are expected to be Cloud prodigies.
    - hooverd 3y ago
      I hate CloudFormation I hate CloudFormation. Unless you're using AWS as yet another VPS provider, creating a "cloudy" product requires way too much knowledge about AWS services as a developer.
    - ecshafer 3y ago
      When I was at a company that went all in on AWS. I went from 100% of time working on product, to easy 75% of time working on AWS. Even though we had teams who just set up infra on aws for us to use and it was supposed to be "so easy". There was far more time setting up AWS infra than writing code.
      - steveBK123 3y ago
        Yes, and management discounts that .. if you make me, a product expert, working on application software.. deal with this stuff, I am going to do the worst, most rapid, half-assed, security and everything else be damned job. I have clients, product, scrum masters, tech leads, and others breathing down my neck on feature delivery, and you're asking me to figure this alphabet soup out, lol.. don't expect happy results dudes. I've been a domain expert in a field for 20 years, what value do I bring to the client in dealing with this alphabet soup of AWS networking/security? Either it works and client gets what they want, or it doesn't and they don't. This stuff is a full time job and should be respected as such. If you think you can push it all down on app dev, you are going to have unhappy results.
- anyoneamous 3y ago
  I never saw it marketed that way - but even so, anyone stupid enough to fall for it deserves every NAT gateway charge they've had since.
  - 2-718-281-828 3y ago
    would you mind elaborating on the nat gateway part? are you insinuating that nat gateway as mean to provide oneway internet access for instances in private subnets are misguided or suboptimal?
    - hooverd 3y ago
      Managed NAT Gateways are one of the more expensive services in the known universe.
      - 2-718-281-828 3y ago
        what would you use instead?
        benterix 3y ago
        AWS NAT Gateway pricing is so outrageous that smart people came up with a peculiar hybrid: https://www.lastweekinaws.com/blog/an-alternat-future-we-now-have-a-nat-gateway-replacement/ https://www.lastweekinaws.com/blog/an-alternat-future-we-now... I wouldn't use it in production (some companies do, for internal things) but the very fact it exists speaks volumes.
        berkle4455 3y ago
        a tiny ec2 box that provides NAT for about $3/mo.
devonkim 3y ago
FWIW, more debugging information in an IAM failure is another round of information available to me as an attacker. If such a service existed available to the customer this also becomes potentially available to an attacker. In fact, it may wind up violating a few companies’ security and compliance policies. However, I think that customers should have the option once they sign some stuff acknowledging that they may be really, really screwed similar to Bad Things done as a root user in AWS. Given the distributed nature of so many requests in AWS I’d imagine that a GraphQL-like setup would be conceptually compatible with such a system. It’s not terribly far from tracing systems like OTEL
- nprateem 3y ago
  For some APIs they return encrypted payloads with actually useful debugging info in, but there seem to be plenty where you just end up in a black hole of "computer says no"
- JoBrad 3y ago
  They could put better information in Cloudtrail, at least.
  - booi 3y ago
    They do! Most all IAM errors will list the actor, the resource, and the missing permission
bob1029 3y ago
> Why is debugging IAM policies and auth issues such complete dogshit on AWS? Seriously, if you want to be a competitor, focus on this. System managed identities + AAD is an absolute joy in my experience on Azure. I can grant a function app access to a SQL database by simply using the literal name of the function app (qualified by the object's identifier if aliased). No passwords, certificates, etc. involved anymore. It literally just works. Combine their auth package with things like application insights, and you are not really ever stuck thinking "WTF happened", assuming you have enough awareness to look at those logs. > I guess by keeping such a shitshow they sell more support contracts though. This is also a problem with Azure. You really have to know where all of the mines are located before you start walking through their field or you will wind up in a similar state of contorted frustration. I'd argue that Microsoft leaves substantially better hints regarding where mines might be, but they still conveniently leave out enough hints such that only a wizard with 2 decades of experience can pierce the final veil of bullshit and avoid a 5-10x multiple on cost & complexity.
- deleted 3y ago
  [deleted]
- cameronh90 3y ago
  Unfortunately with Azure, while authentication may be more obvious, 0x80003007 ERR_UNKNOWN type things aren't. Also, what's up with the way almost every any change in Azure takes some random period, sometimes hours, to take effect and there's not necessarily any indication that anything is happening.
- scubbo 3y ago
  > I can grant a function app access to a SQL database by simply using the literal name of the function app (qualified by the object's identifier if aliased). No passwords, certificates, etc. involved anymore. It literally just works I can't comment on the general correctness of your comparison (never having used Azure), but this particular one is flawed, since the same is true in CDK. That said, there's enough arcane weirdness in AWS' configuration requirements that I suspect your point is _broadly_ correct.
- jiggawatts 3y ago
evulhotdog 3y ago
Those request IDs can be used if you escalate to AWS support, they can point you in the right direction. It’s generally not a fast process, but it’s something.
- nprateem 3y ago
  Exactly. The tools exist, they just don't make them available to customers.
awinter-py 3y ago
ugh access + least privilege on AWS + GCP are really hard not made easier by the fact that debugging service accounts can sometimes require rebooting boxes or clusters 'cloud traceroute' to discover where exactly the problem is would be amazing (in fairness there are 'least privilege' tools I haven't tried -- listed below and I recall reading about others) 1. iam access analyzer https://aws.amazon.com/blogs/security/iam-access-analyzer-makes-it-easier-to-implement-least-privilege-permissions-by-generating-iam-policies-based-on-access-activity/ https://aws.amazon.com/blogs/security/iam-access-analyzer-ma... 2. airiam https://github.com/bridgecrewio/AirIAM https://github.com/bridgecrewio/AirIAM 3. policy simulator https://cloud.google.com/policy-intelligence/docs/iam-simulator-overview https://cloud.google.com/policy-intelligence/docs/iam-simula...
adduc 3y ago
Cloudtrail writes to s3 at intervals, which means it make take a few minutes for the log entry related to your request ID to show. I've found Athena doesn't appear to search through all log entries (possible user error on my part); I've been downloading the cloudtrail logs directly from S3 and grepping through the logs directly to find the relevant entries to figure out the permissions needed. If you need to decode an encrypted error message, use `aws sts decode-authorization-message --encoded-message $MESSAGE`. It'll return a JSON string. I typically use `aws --profile limbledevadmin sts decode-authorization-message --output text --encoded-message $MSG | jq .context.action` to extract the needed permission.

trevorsullivan 3y ago

Pretty much everything in AWS APIs is a trainwreck. They mindless wrap objects other objects, for seemingly no reason. The customer bears the burden of learning and implementing the extra logic for complex data structures. It's incredibly annoying. They need to pair up UX designers with engineers, but they don't.

benterix 3y ago

A cynic in me is happy about this state of affairs - the companies who happily joined the AWS bandwagon have to pay the price (including my salary). For my projects I almost always choose something else that offers better value, except in very rare cases when it makes sense. A decade ago or so AWS was marketed as - to simplify - "fire all your sysadmins because we manage the systems for you" to the current state where good AWS DevOps engineers are expensive and not so easy to find and you pay through the nose if you really follow AWS recommendations, especially for larger setups.

nprateem 3y ago
Pretty tedious work though. I'd far rather feel I was actually doing useful work instead of dealing with this stupid crap.
steveBK123 3y ago
Yeah this has been my observation. CTOs bought the "fire all your sysadmins" literally, and now we have a 20% overhead on all app devs who are supposed to learn all this stuff. It's crazy to me. It's literally the opposite of every management training / book about focussing on highest value work / where you add value / where you provide expertise.
- renegade-otter 3y ago
  20%? Seems like at least half of all the work has to do with infrastructure plumbing, now that people who are supposed to write product code are expected to be Cloud prodigies.
  - hooverd 3y ago
    I hate CloudFormation I hate CloudFormation. Unless you're using AWS as yet another VPS provider, creating a "cloudy" product requires way too much knowledge about AWS services as a developer.
  - ecshafer 3y ago
    When I was at a company that went all in on AWS. I went from 100% of time working on product, to easy 75% of time working on AWS. Even though we had teams who just set up infra on aws for us to use and it was supposed to be "so easy". There was far more time setting up AWS infra than writing code.
    - steveBK123 3y ago
      Yes, and management discounts that .. if you make me, a product expert, working on application software.. deal with this stuff, I am going to do the worst, most rapid, half-assed, security and everything else be damned job. I have clients, product, scrum masters, tech leads, and others breathing down my neck on feature delivery, and you're asking me to figure this alphabet soup out, lol.. don't expect happy results dudes. I've been a domain expert in a field for 20 years, what value do I bring to the client in dealing with this alphabet soup of AWS networking/security? Either it works and client gets what they want, or it doesn't and they don't. This stuff is a full time job and should be respected as such. If you think you can push it all down on app dev, you are going to have unhappy results.
anyoneamous 3y ago
I never saw it marketed that way - but even so, anyone stupid enough to fall for it deserves every NAT gateway charge they've had since.
- 2-718-281-828 3y ago
  would you mind elaborating on the nat gateway part? are you insinuating that nat gateway as mean to provide oneway internet access for instances in private subnets are misguided or suboptimal?
  - hooverd 3y ago
    Managed NAT Gateways are one of the more expensive services in the known universe.
    - 2-718-281-828 3y ago
      what would you use instead?
      - benterix 3y ago
        AWS NAT Gateway pricing is so outrageous that smart people came up with a peculiar hybrid: https://www.lastweekinaws.com/blog/an-alternat-future-we-now-have-a-nat-gateway-replacement/ https://www.lastweekinaws.com/blog/an-alternat-future-we-now... I wouldn't use it in production (some companies do, for internal things) but the very fact it exists speaks volumes.
      - berkle4455 3y ago
        a tiny ec2 box that provides NAT for about $3/mo.

devonkim 3y ago

FWIW, more debugging information in an IAM failure is another round of information available to me as an attacker. If such a service existed available to the customer this also becomes potentially available to an attacker. In fact, it may wind up violating a few companies’ security and compliance policies. However, I think that customers should have the option once they sign some stuff acknowledging that they may be really, really screwed similar to Bad Things done as a root user in AWS. Given the distributed nature of so many requests in AWS I’d imagine that a GraphQL-like setup would be conceptually compatible with such a system. It’s not terribly far from tracing systems like OTEL

nprateem 3y ago
For some APIs they return encrypted payloads with actually useful debugging info in, but there seem to be plenty where you just end up in a black hole of "computer says no"
JoBrad 3y ago
They could put better information in Cloudtrail, at least.
- booi 3y ago
  They do! Most all IAM errors will list the actor, the resource, and the missing permission

bob1029 3y ago

> Why is debugging IAM policies and auth issues such complete dogshit on AWS? Seriously, if you want to be a competitor, focus on this. System managed identities + AAD is an absolute joy in my experience on Azure. I can grant a function app access to a SQL database by simply using the literal name of the function app (qualified by the object's identifier if aliased). No passwords, certificates, etc. involved anymore. It literally just works. Combine their auth package with things like application insights, and you are not really ever stuck thinking "WTF happened", assuming you have enough awareness to look at those logs. > I guess by keeping such a shitshow they sell more support contracts though. This is also a problem with Azure. You really have to know where all of the mines are located before you start walking through their field or you will wind up in a similar state of contorted frustration. I'd argue that Microsoft leaves substantially better hints regarding where mines might be, but they still conveniently leave out enough hints such that only a wizard with 2 decades of experience can pierce the final veil of bullshit and avoid a 5-10x multiple on cost & complexity.

deleted 3y ago
[deleted]
cameronh90 3y ago
Unfortunately with Azure, while authentication may be more obvious, 0x80003007 ERR_UNKNOWN type things aren't. Also, what's up with the way almost every any change in Azure takes some random period, sometimes hours, to take effect and there's not necessarily any indication that anything is happening.
scubbo 3y ago
> I can grant a function app access to a SQL database by simply using the literal name of the function app (qualified by the object's identifier if aliased). No passwords, certificates, etc. involved anymore. It literally just works I can't comment on the general correctness of your comparison (never having used Azure), but this particular one is flawed, since the same is true in CDK. That said, there's enough arcane weirdness in AWS' configuration requirements that I suspect your point is _broadly_ correct.
jiggawatts 3y ago

evulhotdog 3y ago

Those request IDs can be used if you escalate to AWS support, they can point you in the right direction. It’s generally not a fast process, but it’s something.

nprateem 3y ago
Exactly. The tools exist, they just don't make them available to customers.

awinter-py 3y ago

ugh access + least privilege on AWS + GCP are really hard not made easier by the fact that debugging service accounts can sometimes require rebooting boxes or clusters 'cloud traceroute' to discover where exactly the problem is would be amazing (in fairness there are 'least privilege' tools I haven't tried -- listed below and I recall reading about others) 1. iam access analyzer https://aws.amazon.com/blogs/security/iam-access-analyzer-makes-it-easier-to-implement-least-privilege-permissions-by-generating-iam-policies-based-on-access-activity/ https://aws.amazon.com/blogs/security/iam-access-analyzer-ma... 2. airiam https://github.com/bridgecrewio/AirIAM https://github.com/bridgecrewio/AirIAM 3. policy simulator https://cloud.google.com/policy-intelligence/docs/iam-simulator-overview https://cloud.google.com/policy-intelligence/docs/iam-simula...

adduc 3y ago

Cloudtrail writes to s3 at intervals, which means it make take a few minutes for the log entry related to your request ID to show. I've found Athena doesn't appear to search through all log entries (possible user error on my part); I've been downloading the cloudtrail logs directly from S3 and grepping through the logs directly to find the relevant entries to figure out the permissions needed. If you need to decode an encrypted error message, use `aws sts decode-authorization-message --encoded-message $MESSAGE`. It'll return a JSON string. I typically use `aws --profile limbledevadmin sts decode-authorization-message --output text --encoded-message $MSG | jq .context.action` to extract the needed permission.