4 ms·

I get that some alerts should be sent, that AWS should scan for common scams, react faster to support tickets etc. But how does one get enough access to someone

by atraac 5y ago

I get that some alerts should be sent, that AWS should scan for common scams, react faster to support tickets etc. But how does one get enough access to someone's AWS account to create a lambda that mines crypto? Something has an access to your credit card and bills you monthly, yet you do not have 2FA? I would understand if this was some dependency chain attack, but this is just a new lambda with an .sh script. Also what AWS income has to do to with his issue?

rwmj 5y ago
From a later tweet: "I don't know how my key leaked. It's a 9yr old account, under utilized + running old projects. AWS do have alerting systems you can use to catch overspend. You should. They didn't exist when I signed up. And with 200 links in their main menu, they're not easy to find." Don't blame the user for what is essentially Amazon's problem. They don't have a way to set a hard limit on spend. They don't even do basic velocity tracking apparently.
- atraac 5y ago
  I'm all up for hating corporations but how is this Amazon's problem/fault? You leaked a key, what do you expect Amazon to do? This is like software development 101, protect any kind of secret/key, don't store/send it in plain text, don't make it public. He's using arguments like 'but as a solo founder...', what kind of excuse is this? I fully agree that AWS could improve detection of these kinds of things and that the guy needs actual support but this whole twitter thread sounds like he is blaming AWS for his own mistake. Just ask for help, don't blame them. No one is forcing anyone to use AWS, you know what you get into when you make an account.
  - wccrawford 5y ago
    They expect Amazon to allow people to set maximize spending amounts on their accounts. They have steadfastly refused to implement this.
  - KronisLV 5y ago
    In my eyes, the situation is simple: human beings are fallible. If there's a chance of your credentials leaking and being exploited that way, that will most certainly happen to someone sooner or later. I recall a VPS that i had years ago being hijacked due to Docker socket being exposed publicly (when i set one up for testing and was just learning about Docker), which was mining crypto throughout the night before i noticed it. Imagine what would happen if there had been some sort of auto scaling in place and i had used a managed Kubernetes solution (many of which didn't yet exist back then)... Is it Amazon's problem? Not really, apart from the bad PR that charging private individuals exorbitant sums (for them) might cause, so sometimes these bills are waived after people mess up. For the corporation at the size of Amazon, that's probably not too much in the grand scheme of things, but it's also understandable why people might be appropriately upset about not being able to set hard spending limits, merely alerts (which aren't enabled by default). That's just the reality that we live in for now, however one can also think about why a corporation that cares a lot about uptime might not necessarily want to implement functionality that'd let resources be easily cut off when spend limits are hit or something similar, since most businesses that utilize their services might just swallow the occasional expensive bill like that anyways.
  - deleted 5y ago
    [deleted]
  - robinwarren 5y ago
    It would also be software dev 101 for an organisation of AWS's size and clout to do some pretty basic things - If someone's costs increase drastically (ie spending in 24hrs what they spent in the last year or etc.) then maybe flag it up to them - If the above is combined with costs on services/regions they've not used, definitely maybe contact them? - If the above is combined with a well known exploit used on compromised accounts, again maybe contact them? Not saying the OP has no responsibility here. Just that I don't think it is that black and white. There is plenty AWS could do to avoid this sort of thing happening. Totally fair to call them out for that. Especially when the cost of phoning support is $2-3k specifically because of the new costs on your account!
  - jffry 5y ago
    Blaming the victim for AWS having inadequate spending controls is kinda crummy. Absolute spending caps, or spending growth rate limits, would both be useful even outside of situations where a user's credential is compromised. One example: When teaching colleagues how to use AWS tooling, I gave them full access to a sandbox account where they can learn without disrupting anybody else's infra. It would be nice to be able to place a spending cap on that account. Another example: With things like autoscaling, it's very easy to accidentally misconfigure it to where it will happily start launching tons and tons of VMs. You might do this and not notice. If I had the ability to voluntarily opt into limits on how fast I can increase the aggregate $/second spend rate, then AWS could alert me in general when I'm doing something that exceeds that soft limit.
    - tssva 5y ago
      Trying to turn this into him somehow being victimized by AWS and the original comment as being victim blaming in this regard is kind of crummy. The OP was a victim of the hacker and not AWS.
      - jffry 5y ago
        I did not say that this person was victimized by AWS, you are putting words in my mouth. We agree that this person has been victimized by a criminal. My point is that AWS failing to implement spending controls makes this type of attack significantly more damaging, and the slow turnaround on CloudWatch spending alarms means five or six-figure sums of money can be spent before customers are alerted. AWS can and should do more to give its customers additional control over spending as a defense-in-depth measure. This would be valuable both to limit unauthorized spending (this scenario, or a "rogue employee" scenario) and also unanticipated spending ("autoscaling gone wild" type scenarios). AWS is a big target, and the fact that credentials can be used to spend an unlimited quantity of money means they are a huge target for criminals. Defense in depth would be wise in this situation!
  - Tuna-Fish 5y ago
    I expect them to allow me to set a limit on monthly spend, and cut my access to limit further costs when that is exceeded. Security needs to have depth. The one thing we have learned in these few decades of learning how to do online security is that any single measure can fail, and given enough scale, will fail. For anything to work, you need to be able to layer your security measures so that one failure, no matter how grave, will be fatal. AWS could add a very simple security measure here, making their service dramatically safer for their users, but they choose not to.
  - msh 5y ago
    If I leave my credit card somewhere or the details gets stolen some how, so a thief gets it I do not have unlimited liability for the abuse of the card.
    - PixelOfDeath 5y ago
      I'm very sorry Ms. or Mrs. msh. But the 3.4 trillion dollars have to be paid back by Monday! And take better care of your information next time. It is the duty of every customer to check cash dispenser for skimmers before using them! This is clearly written out in out general terms and conditions!
- bogwog 5y ago
  I wonder if this is something Amazon just doesn’t want to solve? Maybe they did a cost-benefit analysis between making AWS extremely quick to get started vs adding a bunch of security and intelligent lockout features, etc and found that the problem is not big enough to justify degrading the user experience? Every time I see a story like this (and there have been many), Amazon always ends up dropping the bill. Idk what this one hack on Lambda actually cost Amazon, but I’m guessing it’s nowhere near $45k.
  - unclebucknasty 5y ago
    >Maybe they did a cost-benefit analysis between making AWS extremely quick to get started Yeah, one key issue is that it's deceptively easy to set something up in AWS, but there's actually a ton of customizability, complexity and potential gotchas lurking in the background. They've essentially SaaS-ified sysadmin, network admin, DBA, etc. roles. So relatively casual users are pointing, clicking, copying and pasting like they're just navigating any ol' app, but beneath the surface they're commanding potentially massive/unlimited resources with the attendant costs those resources impute. Makes you wonder whether it's just too much power to casually have on offer.

by atraac 5y ago

rwmj 5y ago

From a later tweet: "I don't know how my key leaked. It's a 9yr old account, under utilized + running old projects. AWS do have alerting systems you can use to catch overspend. You should. They didn't exist when I signed up. And with 200 links in their main menu, they're not easy to find." Don't blame the user for what is essentially Amazon's problem. They don't have a way to set a hard limit on spend. They don't even do basic velocity tracking apparently.

atraac 5y ago
I'm all up for hating corporations but how is this Amazon's problem/fault? You leaked a key, what do you expect Amazon to do? This is like software development 101, protect any kind of secret/key, don't store/send it in plain text, don't make it public. He's using arguments like 'but as a solo founder...', what kind of excuse is this? I fully agree that AWS could improve detection of these kinds of things and that the guy needs actual support but this whole twitter thread sounds like he is blaming AWS for his own mistake. Just ask for help, don't blame them. No one is forcing anyone to use AWS, you know what you get into when you make an account.
- wccrawford 5y ago
  They expect Amazon to allow people to set maximize spending amounts on their accounts. They have steadfastly refused to implement this.
- KronisLV 5y ago
  In my eyes, the situation is simple: human beings are fallible. If there's a chance of your credentials leaking and being exploited that way, that will most certainly happen to someone sooner or later. I recall a VPS that i had years ago being hijacked due to Docker socket being exposed publicly (when i set one up for testing and was just learning about Docker), which was mining crypto throughout the night before i noticed it. Imagine what would happen if there had been some sort of auto scaling in place and i had used a managed Kubernetes solution (many of which didn't yet exist back then)... Is it Amazon's problem? Not really, apart from the bad PR that charging private individuals exorbitant sums (for them) might cause, so sometimes these bills are waived after people mess up. For the corporation at the size of Amazon, that's probably not too much in the grand scheme of things, but it's also understandable why people might be appropriately upset about not being able to set hard spending limits, merely alerts (which aren't enabled by default). That's just the reality that we live in for now, however one can also think about why a corporation that cares a lot about uptime might not necessarily want to implement functionality that'd let resources be easily cut off when spend limits are hit or something similar, since most businesses that utilize their services might just swallow the occasional expensive bill like that anyways.
- deleted 5y ago
  [deleted]
- robinwarren 5y ago
  It would also be software dev 101 for an organisation of AWS's size and clout to do some pretty basic things - If someone's costs increase drastically (ie spending in 24hrs what they spent in the last year or etc.) then maybe flag it up to them - If the above is combined with costs on services/regions they've not used, definitely maybe contact them? - If the above is combined with a well known exploit used on compromised accounts, again maybe contact them? Not saying the OP has no responsibility here. Just that I don't think it is that black and white. There is plenty AWS could do to avoid this sort of thing happening. Totally fair to call them out for that. Especially when the cost of phoning support is $2-3k specifically because of the new costs on your account!
- jffry 5y ago
  Blaming the victim for AWS having inadequate spending controls is kinda crummy. Absolute spending caps, or spending growth rate limits, would both be useful even outside of situations where a user's credential is compromised. One example: When teaching colleagues how to use AWS tooling, I gave them full access to a sandbox account where they can learn without disrupting anybody else's infra. It would be nice to be able to place a spending cap on that account. Another example: With things like autoscaling, it's very easy to accidentally misconfigure it to where it will happily start launching tons and tons of VMs. You might do this and not notice. If I had the ability to voluntarily opt into limits on how fast I can increase the aggregate $/second spend rate, then AWS could alert me in general when I'm doing something that exceeds that soft limit.
  - tssva 5y ago
    Trying to turn this into him somehow being victimized by AWS and the original comment as being victim blaming in this regard is kind of crummy. The OP was a victim of the hacker and not AWS.
    - jffry 5y ago
      I did not say that this person was victimized by AWS, you are putting words in my mouth. We agree that this person has been victimized by a criminal. My point is that AWS failing to implement spending controls makes this type of attack significantly more damaging, and the slow turnaround on CloudWatch spending alarms means five or six-figure sums of money can be spent before customers are alerted. AWS can and should do more to give its customers additional control over spending as a defense-in-depth measure. This would be valuable both to limit unauthorized spending (this scenario, or a "rogue employee" scenario) and also unanticipated spending ("autoscaling gone wild" type scenarios). AWS is a big target, and the fact that credentials can be used to spend an unlimited quantity of money means they are a huge target for criminals. Defense in depth would be wise in this situation!
- Tuna-Fish 5y ago
  I expect them to allow me to set a limit on monthly spend, and cut my access to limit further costs when that is exceeded. Security needs to have depth. The one thing we have learned in these few decades of learning how to do online security is that any single measure can fail, and given enough scale, will fail. For anything to work, you need to be able to layer your security measures so that one failure, no matter how grave, will be fatal. AWS could add a very simple security measure here, making their service dramatically safer for their users, but they choose not to.
- msh 5y ago
  If I leave my credit card somewhere or the details gets stolen some how, so a thief gets it I do not have unlimited liability for the abuse of the card.
  - PixelOfDeath 5y ago
    I'm very sorry Ms. or Mrs. msh. But the 3.4 trillion dollars have to be paid back by Monday! And take better care of your information next time. It is the duty of every customer to check cash dispenser for skimmers before using them! This is clearly written out in out general terms and conditions!
bogwog 5y ago
I wonder if this is something Amazon just doesn’t want to solve? Maybe they did a cost-benefit analysis between making AWS extremely quick to get started vs adding a bunch of security and intelligent lockout features, etc and found that the problem is not big enough to justify degrading the user experience? Every time I see a story like this (and there have been many), Amazon always ends up dropping the bill. Idk what this one hack on Lambda actually cost Amazon, but I’m guessing it’s nowhere near $45k.
- unclebucknasty 5y ago
  >Maybe they did a cost-benefit analysis between making AWS extremely quick to get started Yeah, one key issue is that it's deceptively easy to set something up in AWS, but there's actually a ton of customizability, complexity and potential gotchas lurking in the background. They've essentially SaaS-ified sysadmin, network admin, DBA, etc. roles. So relatively casual users are pointing, clicking, copying and pasting like they're just navigating any ol' app, but beneath the surface they're commanding potentially massive/unlimited resources with the attendant costs those resources impute. Makes you wonder whether it's just too much power to casually have on offer.