10 ms·

$45k in AWS charges due to account being hacked to mine cryptocurrency

146 points by tomduncalf 5y ago

openfuture 5y ago
Honestly AWS is just a casino; people are blowing ridiculous amounts on RNG to try and find nice hash tables that could potentially make money by manipulating cattle.
- astrea 5y ago
  Surely you could think of at least one more use case for AWS.
  - openfuture 5y ago
    Not one that requires me to be on the hook for potentially infinite money.
rwmj 5y ago
Doesn't AWS let you set a monthly limit? I only see a way to set spending alerts, not a hard "do not spend more than X" limit. There's no way on earth I would use AWS on my personal card unless I could set a limit.
- beardyw 5y ago
  I wouldn't sleep.
- hdjjhhvvhga 5y ago
  No, they explicitly refuse implementing hard limits for obvious reasons in spite of many users asking for it for years.
- brodouevencode 5y ago
  To the author's point - if you have a strange charge on your credit card your credit card company will reach out to you and let you know. AWS should have similar controls - if your spend gets out of control within a X amount of time period they should reach out to you. Many of the CFM platforms have this as part of their suite of tools (usually called anomaly spend or something similar) and AWS should offer some native tooling.
  - zokier 5y ago
    AWS allows configuring cost alerts via AWS Budgets service. https://aws.amazon.com/aws-cost-management/aws-budgets/ https://aws.amazon.com/aws-cost-management/aws-budgets/
    - brodouevencode 5y ago
      Yes - that's an opt-in service. IMO it should be on by default.
- s_dev 5y ago
  If a hacker can access your AWS account? Does it matter -- what AWS limits were imposed? Those 'real' limits would have to be set by the card issuer. Not AWS. Otherwise trivial for hacker to change the limits.
  - xdennis 5y ago
    Amazon should notify you if the limit is changed. You should also be able to set non-editable limits. There's no way I, as a private individual, would spend 45000$ on AWS.
    - s_dev 5y ago
      >Amazon should notify you if the limit is changed. Again if a hacker was to access your AWS account with root this feature would be disabled and you would not be notified. Someone accessing your AWS is fundamentally the problem.
    - xeromal 5y ago
      lol @ non-editable limits
  - deleted 5y ago
    [deleted]
  - bg24 5y ago
    Card limits come much later. Usage comes first, then the bill (end of the month).
- inoop 5y ago
  Even if AWS had a hard spending limit, if an attacker has your root credentials, couldn't they still just increase the limits?
atraac 5y ago
I get that some alerts should be sent, that AWS should scan for common scams, react faster to support tickets etc. But how does one get enough access to someone's AWS account to create a lambda that mines crypto? Something has an access to your credit card and bills you monthly, yet you do not have 2FA? I would understand if this was some dependency chain attack, but this is just a new lambda with an .sh script. Also what AWS income has to do to with his issue?
- rwmj 5y ago
  From a later tweet: "I don't know how my key leaked. It's a 9yr old account, under utilized + running old projects. AWS do have alerting systems you can use to catch overspend. You should. They didn't exist when I signed up. And with 200 links in their main menu, they're not easy to find." Don't blame the user for what is essentially Amazon's problem. They don't have a way to set a hard limit on spend. They don't even do basic velocity tracking apparently.
  - atraac 5y ago
    I'm all up for hating corporations but how is this Amazon's problem/fault? You leaked a key, what do you expect Amazon to do? This is like software development 101, protect any kind of secret/key, don't store/send it in plain text, don't make it public. He's using arguments like 'but as a solo founder...', what kind of excuse is this? I fully agree that AWS could improve detection of these kinds of things and that the guy needs actual support but this whole twitter thread sounds like he is blaming AWS for his own mistake. Just ask for help, don't blame them. No one is forcing anyone to use AWS, you know what you get into when you make an account.
    - wccrawford 5y ago
      They expect Amazon to allow people to set maximize spending amounts on their accounts. They have steadfastly refused to implement this.
    - KronisLV 5y ago
      In my eyes, the situation is simple: human beings are fallible. If there's a chance of your credentials leaking and being exploited that way, that will most certainly happen to someone sooner or later. I recall a VPS that i had years ago being hijacked due to Docker socket being exposed publicly (when i set one up for testing and was just learning about Docker), which was mining crypto throughout the night before i noticed it. Imagine what would happen if there had been some sort of auto scaling in place and i had used a managed Kubernetes solution (many of which didn't yet exist back then)... Is it Amazon's problem? Not really, apart from the bad PR that charging private individuals exorbitant sums (for them) might cause, so sometimes these bills are waived after people mess up. For the corporation at the size of Amazon, that's probably not too much in the grand scheme of things, but it's also understandable why people might be appropriately upset about not being able to set hard spending limits, merely alerts (which aren't enabled by default). That's just the reality that we live in for now, however one can also think about why a corporation that cares a lot about uptime might not necessarily want to implement functionality that'd let resources be easily cut off when spend limits are hit or something similar, since most businesses that utilize their services might just swallow the occasional expensive bill like that anyways.
KronisLV 5y ago
Actually discussed the problems around dynamic billing on HN a while ago, here's my comment on the topic, alongside some other platform recommendations, at least as far as VPS hosting is concerned (which is suitable for me since i run a lot of software in containers, but perhaps not to everyone else): https://news.ycombinator.com/item?id=29339477 https://news.ycombinator.com/item?id=29339477 From there, a discussion continued, but the problem of what to do when some sort of resource allotment is exceeded, is still not solved. Personally, i think that static allotments (e.g. you get a VPS with X bandwidth per a month, where the port speed will be decreased 10x until end of billing period in case it's exceeded) are the way to go for people working on non-critical projects, where downtime or degraded performance is preferable to not being able to buy food or afford rent. Though with some platforms, you still need to protect yourself, either with virtual cards or other means, since there's no guarantee that cases like those in this post will be noticed and the fees even waived by the corporations. Of course, the legal aspects of all of that are somewhat unclear. All the more reason to look for solutions with simpler billing, or at least billing alerts in lieu of that.
- ocdtrekkie 5y ago
  I'm still absolutely blown away we tolerate an industry model with uncapped cost. The fact that anyone will even do business with a cloud provider built like this absolutely blows my mind, like every CFO all collectively lost their marbles. Fortune 500 companies have literally just decided handing blank checks to Amazon and Microsoft every month to make their technology needs disappear was good business sense.
  - egwor 5y ago
    I wonder if they have different contracts from most of us?
  - treeman79 5y ago
    How health care works. Show up anywhere. They can more or less charge you anything they want, destroy your credit if you can’t pay. Of course the conversation is usually about how to get someone else to pay the bill instead of get costs under control.
xuki 5y ago
To me, the most outrage thing is AWS is charging $45k to mine $800 worth of XMR. Using dedicated servers, it would cost maybe $2,000-$3,000.
- sharklazer 5y ago
  Doing it yourself makes things, faster, better and cheaper than Amazon.
  - sofixa 5y ago
    Faster? I really want to know how you can have servers, with all the required networking, and all the associated and easily accessible redundancy ( multi-DC or multi region) faster than a few clicks and max 5 minutes wait time. Not to mention the managed services, how long does it take even for an experienced SRE, to setup a production-ready (with backups, monitoring, HA, etc.) RabbitMQ/database/whatever cluster? AWS is much easier to use than a DIY infrastructure, unless significant resources are invested in that DIY infrastructure.
    - sharklazer 5y ago
      Sorry, didn’t mean you get your setup faster, but that you get faster setups. I had an inkling that would be misunderstood. Yes you can spin up a crap fast on amazon. But it’s crap and you will pay an arm and leg for it. I don’t care about “easy” as much I care about sustainable and cheap.
      - mschuster91 5y ago
        > Yes you can spin up a crap fast on amazon. But it’s crap and you will pay an arm and leg for it. Amazon RDS is everything but not crap. The great value-add that RDS provides is that they take care of all the tasks you'd normally need many days of setting up, evaluating, regularly checking they still work: replication, automated failover, backup/restore, version upgrades.
superasn 5y ago
AWS really needs a hard limit of $50 by default. Any user that needs more needs to explicitly set the new limit only after some type of 2FA preferably an OTP verification. Most of the times these hacks are often due to leaked API keys. These soft limits and email alerts are easy to miss and a lot happens by the time you can stop it.
- sixothree 5y ago
  Agreed. Things like static IP addresses and storage incur costs over time regardless of whether they are in use. But a limit would at least prevent new resources like this from being provisioned. Cost management at AWS is an absolute mess. And I believe it is done on purpose. I was tasked with finding out how much a virtual machine was costing us. It didn't have tags. The task was very literally impossible.
spicyusername 5y ago
This happened to me back in 2014, albeit with a smaller bill of around ~$10k. I was pretty inexperienced and I accidentally committed my Amazon credentials to git and went to bed. When I woke up I had a handful of emails from Amazon alerting me to the situation. Back then $10k was almost a third of my yearly income, so it was a pretty traumatic morning say the least hah. When I spoke with an Amazon representative they were very nice about the situation and dropped all the charges, since it was obvious what had happened. I'm sure that's what will happen in this situation, especially given the social media attention.
onion2k 5y ago
This is exactly the scenario that people post in every thread about AWS failing to implement budgets and hard limits, and every time people always say "but AWS will just forgive the fee!" as an excuse for AWS being so user hostile. So far, this time, they haven't. I hope they do for this guy's sake, but to not have that as an official policy, or to implement a tech solution, is beyond terrible.
- dan1234 5y ago
  Yeah, happened to me with an account that: * Never used anything above the free tier * Never had a credit card attached * Was only ever logged in from the UK * Hadn't been used at all in 3 years Naively my root credentials were the same as my Linkedin details (which were all leaked), and one random Sunday, someone from Brazil logged in and spun up 50+ GPU instances across multiple zones. Amazon let that go for around 12 hours before they sent me an email asking me to check that my deployments were working as expected. £2,000 in just those few hours. I did get the fee waived (though they took two weeks to finally decide this), and I accepted that it was my fault for reusing passwords and not enabling MFA (which IIRC may not have been an option when the account was opened) but I was very critical of Amazon for not preventing expensive nodes from being spun up when there wasn't even a card attached to the account.
  - chrisandchris 5y ago
    I really have issues understanding the "I did re-use my password for a service that might charge me an indefinit amount. But i still think, it's the services error". I mean, it could "never" happened if you just thought of a better password for your AWS account. Why blame AWS for that? All the preventive things AWS could implement wimm help the few users who didn't get that they should good passwords and hinder all the users who "know" what they are doing and just need some machines up-and-running. IMHO.
- vasili111 5y ago
  Maybe it is more profitable for them to not have hard limits for legit cloud uses and deal with hackings in a way to just forgive. I am pretty sure they have some calculations done showing what way is more profitable for them.
scrose 5y ago
I feel like whenever this topic comes up, people go back and forth over whether or not AWS should allow you to set a hard limit and what that entails on both sides / why it’s impossible. I feel like a better approach would be to specify what services you actually want to run and that requires a separate secret key that you will never use for other development tasks, and you have the ability set hard limits for those services. AWS already has hard limits on things like the number of S3 buckets you can create, that require manually requesting an increase if you want more. Why can’t people say ‘I only want a maximum of 5 EC2 servers, and I don’t want the ability to spin up anything above a t3.large(translated to maybe ~20cents/hour per server)’. A similar approach can be taken for other resources. This gets around the issue of ‘what should AWS do if you hit your spending cap’ by allowing you to set an upper bound in the amount of hardware you can actually spin up. It also solves the issue of a compromised dev key or root user account that would likely trivially allow someone to remove a spending limit anyway.
- zokier 5y ago
  > I feel like a better approach would be to specify what services you actually want to run and that requires a separate secret key that you will never use for other development tasks, You can get pretty far along by never using the root user, and having very limited iam policy for your day-to-day user. That is also what AWS documents as best-practice: > We strongly recommend that you do not use the root user for your everyday tasks, even the administrative ones. https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practi... You can not set quotas that way, but you can at least limit which services are accessible. Furthermore e.g. ec2 has fairly rich set of conditions you can use in policies to lock down the access even more: https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonec2.html#amazonec2-policy-keys https://docs.aws.amazon.com/service-authorization/latest/ref...
  - caymanjim 5y ago
    This is the right solution. Secure your root account, don't share the credentials, use 2FA, etc. Make role-based accounts with limited permissions and never put anything in the root account. AWS recommends it, and their UI encourages it, but doesn't require it. Third-party auditing services like Vanta are great for identifying issues like this that should be addressed. That said, AWS should make it easier or even require it. AWS is full of footguns and weak security defaults and small players (like the person who tweeted about the bill) lack the knowledge and experience to adequately defend themselves.
hn_throwaway_69 5y ago
I haven't used my old AWS account in a while, it's just sitting dormant. Since it's old it's linked to my Amazon account (root credentials == Amazon.com credentials). I don't want to monitor it and these stories give me some anxiety. I guess I'm supposed to close my AWS account? I wonder if this is even possible if it's the same credentials as your Amazon account.
politician 5y ago
I've had two experiences with this. Once due to a credential leak through a public GitHub Gist (PSA: Gists are public). Once through the failure to delete a resource with premium IOPS for a couple months (PSA: It's easy to leak resources when developing Terraform scripts on AWS - remember to clean up afterwards.) In both cases, we received a partial credit. Amazon expects you to setup cost explorer and budget controls before doing anything else. These features must be manually activated on every billing account. It's kind of maddening actually that even turning on Cost Explorer takes 24 hours. PSA: Turn on Cost Explorer immediately after creating a new AWS account and don't give out API keys until you've setup budget limits. If you join a new company in a role where you have access to the AWS billing account, make sure the cost and budget features are enabled.
libertine 5y ago
How are these companies allowed to provide services without direct customer support. We all understand they don't want to pay for the labor, but that can't be an excuse.
nitred 5y ago
Just curious, is there any other way to monetize compute/memory/space power apart from cryptocurrency?
- randomhodler84 5y ago
  Make software products or services that people want to buy, and sell them.
jerf 5y ago
So, I have a personal AWS account, and setting up cost controls seems like a good idea. Let's try it out. "Welcome to AWS Cost Management Since this is your first visit, it will take some time to prepare your cost and usage data. Please check back in 24 hours." My "cost and usage data" is that I have an S3 account costing ~$10 a month containing my cloud backups of my family's high-priority data, for maybe the last 18 months. That's it. I just reloaded in case 24 hours was some sort of pessimistic estimate and they actually had it done in a couple of minutes, but the message persists. WTF. I can set up any of dozens of services in thirty seconds and spend thousands, but it's going to take 24 hours to understand ~$100 in costs over the course of the last year? Someone could read aloud the entire history of the account, every S3 transaction I did, in that amount of time. I can really feel the love here.
- StratusBen 5y ago
  Not to mention AWS only updates billing data 2-4 times per day. So even if you do have this enabled, on an accrued cost basis you're going to have to wait 6-8 hours to know about it from Cost Explorer.
- ineedasername 5y ago
  AWS doesn't benefit by providing excellent cost controls and visibility into spending. I'm not saying that neglecting this is a conscious, cynical choice. It may just be that there's simple no incentive to build it in the first place. A self-motivated engineer or team manager is probably not going to say, "I know! Instead of working on the next bit of shiny exciting tech, I'm going to work on cost controls!" Though sure, it could also be deliberate.
  - pradn 5y ago
    That's not true at all. Businesses want to reduce risk. Providing cost controls assuages a client's fears. If you're picking between two clouds, cost is a concern, and cost risk is a big factor in that. Would you put your career on the line for a riskier cloud? That's not only cost, but also security, privacy, legal risk, compliance, etc. Clouds spend a great deal of resources to improve all these things, especially for highly-regulated industries which constitute a large part of the potential customer-base: health care, banking, government, polluters.
codegeek 5y ago
I believe this is more by design than anything. AWS makes real money from the big guns where they are spending 100s and 1000s of dollars per month (may be even a Million I am sure). If they put limits, it could hurt their business with these big guns who usually have stricter controls on their end and are not leaking keys or don't have 2 factor Auth enabled etc. The challenge is for the smaller guys who are just experimenting with AWS. I see a lot of these stories where they haven't even used the account much or at all and then get hacked. Of course AWS can impose limits but they clearly don't want to. They would rather refund these cases whenever they feel like.
- iso1631 5y ago
  I'm certain our monthly corporate bill for amazon is high 6 figures if not 7
- dendriti 5y ago
  I have contracted with several midsize B2B outfits who hit low-to-mid 6 figure USD billing per month for AWS. Surely the biggest players are spending millions per month. At that point, you'd expect Amazon to be much more accommodating if a billing error or exploit were to occur.
yosito 5y ago
This happened to me. Luckily it was caught before the bill went over $500, and Amazon refunded all the charges. I have no idea how my account was compromised, and Amazon wasn't able to help me investigate. And they made me go through and delete resources across several different regions in a painstaking manual process that took me an entire work day.
ChicagoBoy11 5y ago
I remember being very new to all of this (still am) and being absolutely petrified that as I was exploring an Amazon stack that I'd unintentionally rack up thousands of dollars of charges for an honest mistake. There was part of me that felt so much better about using App Engine at the time; in that case, I had full confidence that the worst I could do is have the app I was building just break if I got to a certain quota limit. That put my mind SO MUCH more at ease. The risk of an accidental cost overrun has kept me from learnings/using AWS to certainly a detrimental degree for me, but the fear of something like this happening is very, very real.
- StratusBen 5y ago
  Disclosure - I'm Co-Founder and CEO of https://vantage.sh/ https://vantage.sh/ Hopefully the downvote brigade doesn't come after me as I'm truly just trying to share this as I think it'd be helpful in the context of this post and I imagine the majority of folks on HN would fit into our free tier...but this is a huge reason why people use Vantage (linked above) We profile all resources by hitting List/Describe service APIs in your AWS account and automatically apply rates based off of that. So essentially you get a full inventory of everything in your AWS account and see exactly how much its costing you per hour and also if it were extrapolated out for the full month. People tend to use Vantage as a companion to their IaC that's provisioning things to gut-check from a cost perspective.
  - mook 5y ago
    Your product doesn't help this situation at all. The problem is people want capped spending for unimportant projects (just cut off service rather than accruing charges), which can only be done by AWS. You're just providing reporting assuming no additional charges.
xwdv 5y ago
This guy is toast, without evidence to prove he has no relation to this hacker, the charges will not reverse.
ransom1538 5y ago
Can you use a Visa gift card with a $200 amount? Then you could just use a fake name, fake email, done. Need more money? - just add a new gift card.
f00zz 5y ago
Worst part is that the cryptominer probably made $1.32...
laegooose 5y ago
Annoyingly, AWS won't allow you to lower the quotas aka service limits. They can only go up. Even after you completed the project and don't need them anymore, they remain forever and create an increased risk.
bdcravens 5y ago
Happened to me, just a bit over $50k. Took a couple hours of work on my part (they asked me to enable CloudTrail and do some additional securing), and they removed the charges. Also, it's less of a "hack" than a likely publishing of AWS keys (if you publish them in a public Github repo, they're compromised in seconds)
Syzygies 5y ago
I just went to set up a hard limit on several accounts. There is email notification. The only actions stop individual services on individual Amazon server locations. There is no global "DROP DEAD" action freezing my account and my liability to Amazon. My ideas for fake notification addresses and their purported actions are likely against HN rules. I'm just going to leave Amazon AWS.
wilhil 5y ago
We are a very large AWS reseller and this drives me bonkers with EVERY Cloud, and I've written about this countless times. There is almost no way what so ever to limit risk and it's crazy. Cloud operators state that it is because they don't want to shut down popular services or similar - and to be honest, I get this argument - but, it is just awful in practice. Nearly every cloud has cost report delays of anything up to a month for some services and it's just a nightmare. I kind of agree that if you have a service costing £5,000 a month - you shouldn't limit spend to £5,001 - but, you should be able to limit to £7,500 - enough room to burst/grow, but not have risk! All clouds could learn a thing or two from telecoms and rate limits... e.g. if you have services that when not in use have a fixed monthly limit - e.g. storage - then take that out of monthly limit and then use the excess for variable costs. e.g. storage costs £500 a month, compute costs £200 a month... if a client sets a budget of £1,000 - then deduct £700, maybe ~£100 extra for idle transactions (if in a cloud/model that charges), then have prepay of £200 for "extras". I just see "the clouds" do AMAZING new features - all this crazy and cool AI bits, yet they can't get basic costs correct - it's crazy in my mind.
mindslight 5y ago
The large cloud companies seem dead set against creating any type of hard billing limit. Just let me set a magnitude of charges I can ever expect from the relationship, rather than unconstrained open-ended liability. We all know it would be straightforward to implement. Even if the time to shut down systems creates some overage that the provider is unwilling to eat (although I can't believe it would cost more than continually refunding these bills), then it would be perfectly fine for "$200 hard limit" to mean start shutting down systems at $100. Just let me limit the damn magnitude! But since they've all decided it's good business to be sticking users with bullshit bills, we're left to create our own. I've got LLCs on the brain, so I'm wondering if running one's cloud expenditures through an LLC would work. Besides the paperwork overhead, the biggest worry would be piercing of the corporate veil for being undercapitalized. But I hope it would be hard to argue that a company was undercapitalized because it lacked the ability to pay for unexpected expenses that it did not initiate. Does anybody know the legal and practical issues with signing up for these cloud providers through an LLC? Even if you're already operating as a company (eg startup), it would still be prudent to sign up through a separate subsidiary. At the very least an extra layer would make for some red tape they'd have to cut through to press the debt.
errcorrectcode 5y ago
First, the complainant needs to take ownership of poor secrets generation and management. Their account wouldn't have been hacked if they used strong password policies. There is also the remote possibility they're playing the victim to get out of charges after they realized it wasn't economical to mine on AWS.
SavantIdiot 5y ago
They stole is ROOT credentials. This is not about sneaky AWS, this is about a hack. There is not much you can do if someone gets root access to your system. Setting up a billing alert to notify you your aggregate cost exceeds a certain limit takes about 90 seconds. I don't understand the overwhelming friction to the AWS billing tools in this thread.
tacon 5y ago
I took his advice and went to the Cost Anomaly Detection console and started to set up an alert for my $0.56/mo S3 charges. I want individual alerts, so I have to use SNS. I want a text message for an alert, so I am pushed into creating a 10DLC origin (even while in the SMS sandbox). A 10DLC registration is $50 for a brand or company, and $50 for each "campaign", plus $15/mo for each campaign. Am I missing something?
xrd 5y ago
I've had two machines on Google Cloud that racked up $500 in charges from cryptocurrency thieves. They send me alerts but it is so hard to get through the billing dashboards and understand what is legitimate.
mihaylov 5y ago
After seeing this I logged into my existing but not really used AWS account and enabled MFA. I did not get any email alert or notification about to change which seems very weird

146 points by tomduncalf 5y ago

Honestly AWS is just a casino; people are blowing ridiculous amounts on RNG to try and find nice hash tables that could potentially make money by manipulating cattle.

astrea 5y ago
Surely you could think of at least one more use case for AWS.
- openfuture 5y ago
  Not one that requires me to be on the hook for potentially infinite money.

rwmj 5y ago

Doesn't AWS let you set a monthly limit? I only see a way to set spending alerts, not a hard "do not spend more than X" limit. There's no way on earth I would use AWS on my personal card unless I could set a limit.

beardyw 5y ago
I wouldn't sleep.
hdjjhhvvhga 5y ago
No, they explicitly refuse implementing hard limits for obvious reasons in spite of many users asking for it for years.
brodouevencode 5y ago
To the author's point - if you have a strange charge on your credit card your credit card company will reach out to you and let you know. AWS should have similar controls - if your spend gets out of control within a X amount of time period they should reach out to you. Many of the CFM platforms have this as part of their suite of tools (usually called anomaly spend or something similar) and AWS should offer some native tooling.
- zokier 5y ago
  AWS allows configuring cost alerts via AWS Budgets service. https://aws.amazon.com/aws-cost-management/aws-budgets/ https://aws.amazon.com/aws-cost-management/aws-budgets/
  - brodouevencode 5y ago
    Yes - that's an opt-in service. IMO it should be on by default.
s_dev 5y ago
If a hacker can access your AWS account? Does it matter -- what AWS limits were imposed? Those 'real' limits would have to be set by the card issuer. Not AWS. Otherwise trivial for hacker to change the limits.
- xdennis 5y ago
  Amazon should notify you if the limit is changed. You should also be able to set non-editable limits. There's no way I, as a private individual, would spend 45000$ on AWS.
  - s_dev 5y ago
    >Amazon should notify you if the limit is changed. Again if a hacker was to access your AWS account with root this feature would be disabled and you would not be notified. Someone accessing your AWS is fundamentally the problem.
  - xeromal 5y ago
    lol @ non-editable limits
- deleted 5y ago
  [deleted]
- bg24 5y ago
  Card limits come much later. Usage comes first, then the bill (end of the month).
inoop 5y ago
Even if AWS had a hard spending limit, if an attacker has your root credentials, couldn't they still just increase the limits?

atraac 5y ago

I get that some alerts should be sent, that AWS should scan for common scams, react faster to support tickets etc. But how does one get enough access to someone's AWS account to create a lambda that mines crypto? Something has an access to your credit card and bills you monthly, yet you do not have 2FA? I would understand if this was some dependency chain attack, but this is just a new lambda with an .sh script. Also what AWS income has to do to with his issue?

rwmj 5y ago
From a later tweet: "I don't know how my key leaked. It's a 9yr old account, under utilized + running old projects. AWS do have alerting systems you can use to catch overspend. You should. They didn't exist when I signed up. And with 200 links in their main menu, they're not easy to find." Don't blame the user for what is essentially Amazon's problem. They don't have a way to set a hard limit on spend. They don't even do basic velocity tracking apparently.
- atraac 5y ago
  I'm all up for hating corporations but how is this Amazon's problem/fault? You leaked a key, what do you expect Amazon to do? This is like software development 101, protect any kind of secret/key, don't store/send it in plain text, don't make it public. He's using arguments like 'but as a solo founder...', what kind of excuse is this? I fully agree that AWS could improve detection of these kinds of things and that the guy needs actual support but this whole twitter thread sounds like he is blaming AWS for his own mistake. Just ask for help, don't blame them. No one is forcing anyone to use AWS, you know what you get into when you make an account.
  - wccrawford 5y ago
    They expect Amazon to allow people to set maximize spending amounts on their accounts. They have steadfastly refused to implement this.
  - KronisLV 5y ago
    In my eyes, the situation is simple: human beings are fallible. If there's a chance of your credentials leaking and being exploited that way, that will most certainly happen to someone sooner or later. I recall a VPS that i had years ago being hijacked due to Docker socket being exposed publicly (when i set one up for testing and was just learning about Docker), which was mining crypto throughout the night before i noticed it. Imagine what would happen if there had been some sort of auto scaling in place and i had used a managed Kubernetes solution (many of which didn't yet exist back then)... Is it Amazon's problem? Not really, apart from the bad PR that charging private individuals exorbitant sums (for them) might cause, so sometimes these bills are waived after people mess up. For the corporation at the size of Amazon, that's probably not too much in the grand scheme of things, but it's also understandable why people might be appropriately upset about not being able to set hard spending limits, merely alerts (which aren't enabled by default). That's just the reality that we live in for now, however one can also think about why a corporation that cares a lot about uptime might not necessarily want to implement functionality that'd let resources be easily cut off when spend limits are hit or something similar, since most businesses that utilize their services might just swallow the occasional expensive bill like that anyways.

KronisLV 5y ago

Actually discussed the problems around dynamic billing on HN a while ago, here's my comment on the topic, alongside some other platform recommendations, at least as far as VPS hosting is concerned (which is suitable for me since i run a lot of software in containers, but perhaps not to everyone else): https://news.ycombinator.com/item?id=29339477 https://news.ycombinator.com/item?id=29339477 From there, a discussion continued, but the problem of what to do when some sort of resource allotment is exceeded, is still not solved. Personally, i think that static allotments (e.g. you get a VPS with X bandwidth per a month, where the port speed will be decreased 10x until end of billing period in case it's exceeded) are the way to go for people working on non-critical projects, where downtime or degraded performance is preferable to not being able to buy food or afford rent. Though with some platforms, you still need to protect yourself, either with virtual cards or other means, since there's no guarantee that cases like those in this post will be noticed and the fees even waived by the corporations. Of course, the legal aspects of all of that are somewhat unclear. All the more reason to look for solutions with simpler billing, or at least billing alerts in lieu of that.

ocdtrekkie 5y ago
I'm still absolutely blown away we tolerate an industry model with uncapped cost. The fact that anyone will even do business with a cloud provider built like this absolutely blows my mind, like every CFO all collectively lost their marbles. Fortune 500 companies have literally just decided handing blank checks to Amazon and Microsoft every month to make their technology needs disappear was good business sense.
- egwor 5y ago
  I wonder if they have different contracts from most of us?
- treeman79 5y ago
  How health care works. Show up anywhere. They can more or less charge you anything they want, destroy your credit if you can’t pay. Of course the conversation is usually about how to get someone else to pay the bill instead of get costs under control.

xuki 5y ago

To me, the most outrage thing is AWS is charging $45k to mine $800 worth of XMR. Using dedicated servers, it would cost maybe $2,000-$3,000.

sharklazer 5y ago
Doing it yourself makes things, faster, better and cheaper than Amazon.
- sofixa 5y ago
  Faster? I really want to know how you can have servers, with all the required networking, and all the associated and easily accessible redundancy ( multi-DC or multi region) faster than a few clicks and max 5 minutes wait time. Not to mention the managed services, how long does it take even for an experienced SRE, to setup a production-ready (with backups, monitoring, HA, etc.) RabbitMQ/database/whatever cluster? AWS is much easier to use than a DIY infrastructure, unless significant resources are invested in that DIY infrastructure.
  - sharklazer 5y ago
    Sorry, didn’t mean you get your setup faster, but that you get faster setups. I had an inkling that would be misunderstood. Yes you can spin up a crap fast on amazon. But it’s crap and you will pay an arm and leg for it. I don’t care about “easy” as much I care about sustainable and cheap.
    - mschuster91 5y ago
      > Yes you can spin up a crap fast on amazon. But it’s crap and you will pay an arm and leg for it. Amazon RDS is everything but not crap. The great value-add that RDS provides is that they take care of all the tasks you'd normally need many days of setting up, evaluating, regularly checking they still work: replication, automated failover, backup/restore, version upgrades.

superasn 5y ago

AWS really needs a hard limit of $50 by default. Any user that needs more needs to explicitly set the new limit only after some type of 2FA preferably an OTP verification. Most of the times these hacks are often due to leaked API keys. These soft limits and email alerts are easy to miss and a lot happens by the time you can stop it.

sixothree 5y ago
Agreed. Things like static IP addresses and storage incur costs over time regardless of whether they are in use. But a limit would at least prevent new resources like this from being provisioned. Cost management at AWS is an absolute mess. And I believe it is done on purpose. I was tasked with finding out how much a virtual machine was costing us. It didn't have tags. The task was very literally impossible.

spicyusername 5y ago

This happened to me back in 2014, albeit with a smaller bill of around ~$10k. I was pretty inexperienced and I accidentally committed my Amazon credentials to git and went to bed. When I woke up I had a handful of emails from Amazon alerting me to the situation. Back then $10k was almost a third of my yearly income, so it was a pretty traumatic morning say the least hah. When I spoke with an Amazon representative they were very nice about the situation and dropped all the charges, since it was obvious what had happened. I'm sure that's what will happen in this situation, especially given the social media attention.

onion2k 5y ago

This is exactly the scenario that people post in every thread about AWS failing to implement budgets and hard limits, and every time people always say "but AWS will just forgive the fee!" as an excuse for AWS being so user hostile. So far, this time, they haven't. I hope they do for this guy's sake, but to not have that as an official policy, or to implement a tech solution, is beyond terrible.

dan1234 5y ago
Yeah, happened to me with an account that: * Never used anything above the free tier * Never had a credit card attached * Was only ever logged in from the UK * Hadn't been used at all in 3 years Naively my root credentials were the same as my Linkedin details (which were all leaked), and one random Sunday, someone from Brazil logged in and spun up 50+ GPU instances across multiple zones. Amazon let that go for around 12 hours before they sent me an email asking me to check that my deployments were working as expected. £2,000 in just those few hours. I did get the fee waived (though they took two weeks to finally decide this), and I accepted that it was my fault for reusing passwords and not enabling MFA (which IIRC may not have been an option when the account was opened) but I was very critical of Amazon for not preventing expensive nodes from being spun up when there wasn't even a card attached to the account.
- chrisandchris 5y ago
  I really have issues understanding the "I did re-use my password for a service that might charge me an indefinit amount. But i still think, it's the services error". I mean, it could "never" happened if you just thought of a better password for your AWS account. Why blame AWS for that? All the preventive things AWS could implement wimm help the few users who didn't get that they should good passwords and hinder all the users who "know" what they are doing and just need some machines up-and-running. IMHO.
vasili111 5y ago
Maybe it is more profitable for them to not have hard limits for legit cloud uses and deal with hackings in a way to just forgive. I am pretty sure they have some calculations done showing what way is more profitable for them.

scrose 5y ago

I feel like whenever this topic comes up, people go back and forth over whether or not AWS should allow you to set a hard limit and what that entails on both sides / why it’s impossible. I feel like a better approach would be to specify what services you actually want to run and that requires a separate secret key that you will never use for other development tasks, and you have the ability set hard limits for those services. AWS already has hard limits on things like the number of S3 buckets you can create, that require manually requesting an increase if you want more. Why can’t people say ‘I only want a maximum of 5 EC2 servers, and I don’t want the ability to spin up anything above a t3.large(translated to maybe ~20cents/hour per server)’. A similar approach can be taken for other resources. This gets around the issue of ‘what should AWS do if you hit your spending cap’ by allowing you to set an upper bound in the amount of hardware you can actually spin up. It also solves the issue of a compromised dev key or root user account that would likely trivially allow someone to remove a spending limit anyway.

zokier 5y ago
> I feel like a better approach would be to specify what services you actually want to run and that requires a separate secret key that you will never use for other development tasks, You can get pretty far along by never using the root user, and having very limited iam policy for your day-to-day user. That is also what AWS documents as best-practice: > We strongly recommend that you do not use the root user for your everyday tasks, even the administrative ones. https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practi... You can not set quotas that way, but you can at least limit which services are accessible. Furthermore e.g. ec2 has fairly rich set of conditions you can use in policies to lock down the access even more: https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonec2.html#amazonec2-policy-keys https://docs.aws.amazon.com/service-authorization/latest/ref...
- caymanjim 5y ago
  This is the right solution. Secure your root account, don't share the credentials, use 2FA, etc. Make role-based accounts with limited permissions and never put anything in the root account. AWS recommends it, and their UI encourages it, but doesn't require it. Third-party auditing services like Vanta are great for identifying issues like this that should be addressed. That said, AWS should make it easier or even require it. AWS is full of footguns and weak security defaults and small players (like the person who tweeted about the bill) lack the knowledge and experience to adequately defend themselves.

hn_throwaway_69 5y ago

I haven't used my old AWS account in a while, it's just sitting dormant. Since it's old it's linked to my Amazon account (root credentials == Amazon.com credentials). I don't want to monitor it and these stories give me some anxiety. I guess I'm supposed to close my AWS account? I wonder if this is even possible if it's the same credentials as your Amazon account.

politician 5y ago

I've had two experiences with this. Once due to a credential leak through a public GitHub Gist (PSA: Gists are public). Once through the failure to delete a resource with premium IOPS for a couple months (PSA: It's easy to leak resources when developing Terraform scripts on AWS - remember to clean up afterwards.) In both cases, we received a partial credit. Amazon expects you to setup cost explorer and budget controls before doing anything else. These features must be manually activated on every billing account. It's kind of maddening actually that even turning on Cost Explorer takes 24 hours. PSA: Turn on Cost Explorer immediately after creating a new AWS account and don't give out API keys until you've setup budget limits. If you join a new company in a role where you have access to the AWS billing account, make sure the cost and budget features are enabled.

libertine 5y ago

How are these companies allowed to provide services without direct customer support. We all understand they don't want to pay for the labor, but that can't be an excuse.

nitred 5y ago

Just curious, is there any other way to monetize compute/memory/space power apart from cryptocurrency?

randomhodler84 5y ago
Make software products or services that people want to buy, and sell them.

jerf 5y ago

So, I have a personal AWS account, and setting up cost controls seems like a good idea. Let's try it out. "Welcome to AWS Cost Management Since this is your first visit, it will take some time to prepare your cost and usage data. Please check back in 24 hours." My "cost and usage data" is that I have an S3 account costing ~$10 a month containing my cloud backups of my family's high-priority data, for maybe the last 18 months. That's it. I just reloaded in case 24 hours was some sort of pessimistic estimate and they actually had it done in a couple of minutes, but the message persists. WTF. I can set up any of dozens of services in thirty seconds and spend thousands, but it's going to take 24 hours to understand ~$100 in costs over the course of the last year? Someone could read aloud the entire history of the account, every S3 transaction I did, in that amount of time. I can really feel the love here.

StratusBen 5y ago
Not to mention AWS only updates billing data 2-4 times per day. So even if you do have this enabled, on an accrued cost basis you're going to have to wait 6-8 hours to know about it from Cost Explorer.
ineedasername 5y ago
AWS doesn't benefit by providing excellent cost controls and visibility into spending. I'm not saying that neglecting this is a conscious, cynical choice. It may just be that there's simple no incentive to build it in the first place. A self-motivated engineer or team manager is probably not going to say, "I know! Instead of working on the next bit of shiny exciting tech, I'm going to work on cost controls!" Though sure, it could also be deliberate.
- pradn 5y ago
  That's not true at all. Businesses want to reduce risk. Providing cost controls assuages a client's fears. If you're picking between two clouds, cost is a concern, and cost risk is a big factor in that. Would you put your career on the line for a riskier cloud? That's not only cost, but also security, privacy, legal risk, compliance, etc. Clouds spend a great deal of resources to improve all these things, especially for highly-regulated industries which constitute a large part of the potential customer-base: health care, banking, government, polluters.

codegeek 5y ago

I believe this is more by design than anything. AWS makes real money from the big guns where they are spending 100s and 1000s of dollars per month (may be even a Million I am sure). If they put limits, it could hurt their business with these big guns who usually have stricter controls on their end and are not leaking keys or don't have 2 factor Auth enabled etc. The challenge is for the smaller guys who are just experimenting with AWS. I see a lot of these stories where they haven't even used the account much or at all and then get hacked. Of course AWS can impose limits but they clearly don't want to. They would rather refund these cases whenever they feel like.

iso1631 5y ago
I'm certain our monthly corporate bill for amazon is high 6 figures if not 7
dendriti 5y ago
I have contracted with several midsize B2B outfits who hit low-to-mid 6 figure USD billing per month for AWS. Surely the biggest players are spending millions per month. At that point, you'd expect Amazon to be much more accommodating if a billing error or exploit were to occur.

yosito 5y ago

This happened to me. Luckily it was caught before the bill went over $500, and Amazon refunded all the charges. I have no idea how my account was compromised, and Amazon wasn't able to help me investigate. And they made me go through and delete resources across several different regions in a painstaking manual process that took me an entire work day.

ChicagoBoy11 5y ago

I remember being very new to all of this (still am) and being absolutely petrified that as I was exploring an Amazon stack that I'd unintentionally rack up thousands of dollars of charges for an honest mistake. There was part of me that felt so much better about using App Engine at the time; in that case, I had full confidence that the worst I could do is have the app I was building just break if I got to a certain quota limit. That put my mind SO MUCH more at ease. The risk of an accidental cost overrun has kept me from learnings/using AWS to certainly a detrimental degree for me, but the fear of something like this happening is very, very real.

StratusBen 5y ago
Disclosure - I'm Co-Founder and CEO of https://vantage.sh/ https://vantage.sh/ Hopefully the downvote brigade doesn't come after me as I'm truly just trying to share this as I think it'd be helpful in the context of this post and I imagine the majority of folks on HN would fit into our free tier...but this is a huge reason why people use Vantage (linked above) We profile all resources by hitting List/Describe service APIs in your AWS account and automatically apply rates based off of that. So essentially you get a full inventory of everything in your AWS account and see exactly how much its costing you per hour and also if it were extrapolated out for the full month. People tend to use Vantage as a companion to their IaC that's provisioning things to gut-check from a cost perspective.
- mook 5y ago
  Your product doesn't help this situation at all. The problem is people want capped spending for unimportant projects (just cut off service rather than accruing charges), which can only be done by AWS. You're just providing reporting assuming no additional charges.

xwdv 5y ago

This guy is toast, without evidence to prove he has no relation to this hacker, the charges will not reverse.

ransom1538 5y ago

Can you use a Visa gift card with a $200 amount? Then you could just use a fake name, fake email, done. Need more money? - just add a new gift card.

f00zz 5y ago

Worst part is that the cryptominer probably made $1.32...

laegooose 5y ago

Annoyingly, AWS won't allow you to lower the quotas aka service limits. They can only go up. Even after you completed the project and don't need them anymore, they remain forever and create an increased risk.

bdcravens 5y ago

Happened to me, just a bit over $50k. Took a couple hours of work on my part (they asked me to enable CloudTrail and do some additional securing), and they removed the charges. Also, it's less of a "hack" than a likely publishing of AWS keys (if you publish them in a public Github repo, they're compromised in seconds)

Syzygies 5y ago

I just went to set up a hard limit on several accounts. There is email notification. The only actions stop individual services on individual Amazon server locations. There is no global "DROP DEAD" action freezing my account and my liability to Amazon. My ideas for fake notification addresses and their purported actions are likely against HN rules. I'm just going to leave Amazon AWS.

wilhil 5y ago

We are a very large AWS reseller and this drives me bonkers with EVERY Cloud, and I've written about this countless times. There is almost no way what so ever to limit risk and it's crazy. Cloud operators state that it is because they don't want to shut down popular services or similar - and to be honest, I get this argument - but, it is just awful in practice. Nearly every cloud has cost report delays of anything up to a month for some services and it's just a nightmare. I kind of agree that if you have a service costing £5,000 a month - you shouldn't limit spend to £5,001 - but, you should be able to limit to £7,500 - enough room to burst/grow, but not have risk! All clouds could learn a thing or two from telecoms and rate limits... e.g. if you have services that when not in use have a fixed monthly limit - e.g. storage - then take that out of monthly limit and then use the excess for variable costs. e.g. storage costs £500 a month, compute costs £200 a month... if a client sets a budget of £1,000 - then deduct £700, maybe ~£100 extra for idle transactions (if in a cloud/model that charges), then have prepay of £200 for "extras". I just see "the clouds" do AMAZING new features - all this crazy and cool AI bits, yet they can't get basic costs correct - it's crazy in my mind.

mindslight 5y ago

The large cloud companies seem dead set against creating any type of hard billing limit. Just let me set a magnitude of charges I can ever expect from the relationship, rather than unconstrained open-ended liability. We all know it would be straightforward to implement. Even if the time to shut down systems creates some overage that the provider is unwilling to eat (although I can't believe it would cost more than continually refunding these bills), then it would be perfectly fine for "$200 hard limit" to mean start shutting down systems at $100. Just let me limit the damn magnitude! But since they've all decided it's good business to be sticking users with bullshit bills, we're left to create our own. I've got LLCs on the brain, so I'm wondering if running one's cloud expenditures through an LLC would work. Besides the paperwork overhead, the biggest worry would be piercing of the corporate veil for being undercapitalized. But I hope it would be hard to argue that a company was undercapitalized because it lacked the ability to pay for unexpected expenses that it did not initiate. Does anybody know the legal and practical issues with signing up for these cloud providers through an LLC? Even if you're already operating as a company (eg startup), it would still be prudent to sign up through a separate subsidiary. At the very least an extra layer would make for some red tape they'd have to cut through to press the debt.

errcorrectcode 5y ago

First, the complainant needs to take ownership of poor secrets generation and management. Their account wouldn't have been hacked if they used strong password policies. There is also the remote possibility they're playing the victim to get out of charges after they realized it wasn't economical to mine on AWS.

SavantIdiot 5y ago

They stole is ROOT credentials. This is not about sneaky AWS, this is about a hack. There is not much you can do if someone gets root access to your system. Setting up a billing alert to notify you your aggregate cost exceeds a certain limit takes about 90 seconds. I don't understand the overwhelming friction to the AWS billing tools in this thread.

tacon 5y ago

I took his advice and went to the Cost Anomaly Detection console and started to set up an alert for my $0.56/mo S3 charges. I want individual alerts, so I have to use SNS. I want a text message for an alert, so I am pushed into creating a 10DLC origin (even while in the SMS sandbox). A 10DLC registration is $50 for a brand or company, and $50 for each "campaign", plus $15/mo for each campaign. Am I missing something?

xrd 5y ago

I've had two machines on Google Cloud that racked up $500 in charges from cryptocurrency thieves. They send me alerts but it is so hard to get through the billing dashboards and understand what is legitimate.

mihaylov 5y ago

After seeing this I logged into my existing but not really used AWS account and enabled MFA. I did not get any email alert or notification about to change which seems very weird