10 ms·

Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

• shyn3 4y ago
• calvinmorrison 4y ago
  I'm not sure why you would be liable for fraud
  • bloppe 4y ago
    Assuming the perpetrators are never caught and there is no insurance, then this isn't about liability. It's just theft.
  • gigel82 4y ago
    To play the devil's advocate, why would Azure be liable for OP's security lapse? This would've been much easier if someone stole your credit card and bought things with it (the CC company would help with the chargeback).
    • causi 4y ago
      Because it wasn't OP who was hacked? The victim of the crime is Azure, not OP. If I have an Xbox account and someone hacks it and buys a bunch of games, it is the criminal who is deceiving Microsoft into thinking they are someone else. Microsoft trying to charge me for something someone else did would just be a second incidence of fraud. I could leave my Amazon account open on my desk and, assuming I could prove it with security camera footage, someone could walk up and order something and it still would be them defrauding Amazon, not them defrauding me.
      • cmeacham98 4y ago
        I'm not a lawyer, so this is not a legal prescription (I do not know who is legally liable in this scenario, I suspect it depends a lot on the details). That said, it seems like for society to work as it does we need people to take some level of responsibility over their action and inaction related to account security. If I live in the world you describe all online services will be forced to make you upload a photo ID for each purchase to confirm it is you.
        • kelnos 4y ago
          The problem with this stance is that the corporation naturally has much more power in the economic relationship than the customer does. If you give the vendor too much leeway to say "the customer should have been more careful with their credentials!" then they will always say that -- and usually prevail in that opinion -- even when the customer couldn't reasonably have done better.
          • cmeacham98 4y ago
            You seem to believe I said something like "we should always believe the company no matter what the evidence says", but if you reread my comment you'll find that I didn't.
      • calvinmorrison 4y ago
        Azure does bills on credit, IE: you spend and pay later. That's up to them, but it's far riskier than prepurchased credits. I'd find a jury unwilling to believe that a similar real life scenario would raise no flags. It's only a flag raiser because tech companies have automated away all human interaction with billing. Imagine someone claiming to be bob, who regularly shops at the grocery store for 100 dollars a week, now wants to come in and spend, say, 10,000 dollars, on credit. This would be a red flag to any proprietor. Now imagine that proprietor going after bob, who was not there, and claim he is responsible.
        • gigel82 4y ago
• nwiswell 4y ago
  Step 1: Read your cloud services contract with Microsoft very carefully. What does it say about your liability for fraud? Step 2: Read your business insurance policy very carefully. What does it say about fraud coverage? What are the limits and exclusions? Step 3: Unless 1 or 2 makes it real clear the business is not liable, get a lawyer.
  • akerl_ 4y ago
    Business insurance? That doesn’t sound like part of a minimum viable product.
    • deleted 4y ago
      [deleted]
    • Krisjohn 4y ago
  • dhx 4y ago
    Relevant agreements are perhaps one of: 1. OP <--> Partner (confidential agreement) <--> Microsoft (MPA)[1] 2. OP <--> Microsoft (MCA)[2] 3. OP <--> Microsoft (MOSA)[3] The different types of agreements have different limitations of liability clauses. What OP wrote indicates a "partner" is involved and if this is the case, Microsoft have essentially shifted liability for fraud and billing non-payments onto the "partner"[4], who would then either wear the cost or try to shift this liability to the OP. It's not that straightforward though as any of the three parties could have a share of liability, and the "partner" would be very unlikely to want to get in a dispute with Microsoft as this would impact their other business. Liabilities are possibly also impacted by default spending limits and caps that are imposed by Microsoft on different services[5]. Allowing a $200,000 bill for one month (a 200x increase) has the appearance of being very poor financial management from the "partner" as they're potentially going to be stuck with unsecured $200,000+ liabilities from their customers if the customers became insolvent. I suppose it is possible the OP and "partner" have a bank guarantee in place to cover at least $200,000 but I'd hazard a guess they may just try to rely on an insurance policy instead to cover these rare events. [1] Microsoft Partner Agreement (MPA): https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE3wgW1 https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE... [2] Microsoft Customer Agreement (MCA): https://www.microsoft.com/licensing/docs/customeragreement https://www.microsoft.com/licensing/docs/customeragreement [3] Microsoft Online Subscription Agreement (MOSA): https://azure.microsoft.com/en-au/support/legal/subscription-agreement/?country=us&language=en https://azure.microsoft.com/en-au/support/legal/subscription... [4] https://learn.microsoft.com/en-us/partner-center/non-payment-fraud-misuse https://learn.microsoft.com/en-us/partner-center/non-payment... [5] https://azure.microsoft.com/en-us/support/legal/offer-details/ https://azure.microsoft.com/en-us/support/legal/offer-detail...
• bearjaws 4y ago
  AWS would reimburse this if it was the first time. Maybe some hope for MS to do the same?
  • benjaminwootton 4y ago
    I had an incident where a SaaS product went haywire and ran up a $10k bill. I was quite shocked when AWS didn’t write it off and pursued me for the money. You can’t necessarily assume a cloud provider will have your back in these situations.
    • schnebbau 4y ago
      I accidentally ordered a $200 bottle of wine instead of a $20 bottle at a restaurant once, but didn't realise until after the whole bottle was gone. I was not shocked when the restaurant expected me to pay for it, because obviously I was totally responsible. Why would AWS be any different?
      • BartjeD 4y ago
        ... Because building and running a piece of software is a vastly more complicated process than ordering a bottle of wine and drinking it. The way to hell is paved with well intended analogies.
        • schnebbau 4y ago
          The complexity of the process of causing loss to another does not absolve you from it.
          • benjaminwootton 4y ago
            It certainly mitigates it. I have many years of experience with AWS and installed Databricks, an off the shelf product through their SaaS portal. I certainly feel less responsible and more aggrieved when that situation costs me a lot of money than in the $200 wine example.
• waste_monk 4y ago
  >but I think it's more like a stolen credit card situation How did the account get compromised? What was the nature of the attack (e.g. cryptocurrency mining, expensive egress traffic for file hosting, etc.)? Every (consumer) credit card I've seen requires you to take reasonable steps to keep the cards secure to be eligible for fraud protection (e.g. changing the PIN if compromised, not lending it to people, alerting the issuer ASAP in case of suspected fraud, etc.). I do not use Azure but I would imagine that it works the same way - that is, if you fail to follow basic security precautions (enabling MFA, not using shared accounts or passwords that have been known to be compromised in a leak, etc.) you'll probably end up stuck with the bill. Hopefully you had things reasonably well secured.
  • deleted 4y ago
    [deleted]
• kureikain 4y ago
  Try to contact Microsoft support immediately. Don't rely on the distributor/vendor, they act very slowly. You're a customer of Azure, you can contact them by any mean, the fact you pay through a distributor doesn't change that relationship. So I would open a Azure support, and also will try to find Azure team on Twitter/Hacker News etc and contact them politely for help. There is no way you would have to pay this bill. They will sort out something or even waived it if it's the first time.
• NorwegianDude 4y ago
  That really sucks and is the risk of using cloud solutions with no spending limit and a lack of monitoring. You should still have someone to keep an eye on it when using cloud solutions. And when you already have someone to keep an eye on it there's a good chance you might be better off managing the infrastructure yourself.
  • highwaylights 4y ago
    Not really sure how this exact scenario works but if their account was legitimately hacked couldnt the hacker just remove the caps?
    • imadethis 4y ago
      Removing the caps should trigger an out-of-band notification such as an email to all stakeholders, ensuring the legit admins will be notified. This should also be the case for admin removal or disabling billing alerts. I have no idea if Azure actually does this though.
    • ownagefool 4y ago
      So it depends how and what gets hacked. The type of company posting about getting hacked like this is probably using the root / admin accounts to do most things. Their lowest hanging fruit and biggest wins would probably MFA, then SSO. However, IAM is generally powerful enough to allow you to configure what an account can do. So best practice, you also want to think about how you're going to lose credentials. - Sharing passwords across services - leak of your .dotfiles, either by having your laptop pwned, or uploading your .dotfiles to a public repo as a backup or something. - Accidently pasting into the wrong window or something. SSO & MFA defeats all of these with exception that your sts token will be signed for 1h in those .dotfiles when you auth yourself. I'm not sure what happens if you remove it from the token from the device, but the device itself being compromised would allow someone to piggy back your session. Ergo, you move to least privilege access, and then if your laptop, server, or ci/cd runner gets hijacked, then it's only able to do whatever it was allowed to do in the first place. The last part is you need to detect the misuse. When you have least privileged access, and a pretty locked down account, the hope is when a session is hijacked, the attacker will attempt to use the credentials and get an access denied. This should allow you to detect and remediate the reason for leak. Obviously this turns your cloud install into a lot more work, and you still also need to look at maintaining and patching the actual services so they're not compromised in the first place.
• SoftTalker 4y ago
  Don't pay it. Send them notice, by registered letter, that the charges are fradulent. If a credit card was charged, try to initiate a chargeback/fraud claim. Once you pay it, you lose all leverage. You're much less likely to ever get any money back. Probably consult with a lawyer. Cloud hosting charges are basically all profit for the hosting company. They didn't really lose anything except a bit of electricity. In my experience, companies are pretty willing to forgive fraudulent charges if you don't have an unusual history of them.
  • Salgat 4y ago
    If your business is dependent on Azure, what happens when they shut down your services for lack of payment? This seems extremely risky.
    • Rastonbury 4y ago
      They business might not be able to pay 100x their monthly cost either
  • kelnos 4y ago
    If choosing to go this route, back up any data stored on Azure, and start looking into how to migrate everything to AWS or GCP or something else, without incurring too much downtime. Refusing to pay the charge, issuing a chargeback, or even getting a lawyer involved could get OP's account terminated, or at least suspended.
    • amerkhalid 4y ago
      Or even go back to good old VPS/metal servers. Maybe a little more work upfront but less likely to have surprises like this. Also more competition in that space and no worries about vendor lock-ins.
  • dusted 4y ago
    Playing the devils advocate here (though, to be honest, I am very much a hosted-on-premise kind of angry old man here).. Allowing that is a slippery slope for a cloud host. If people can simply say "oh, someone used our credentials to do that thing that cost a lot of money" as a get-out-of-bill card.. If they were legitimately hacked, as in, the intruder did NOT simply obtain their access credentials, but actually bypassed the security system itself (hacking into the actual azure host, or exploiting a technical glitch in the azure login system) then, of course they should forgive the bill (and apologize to their customers)..
• xwolfi 4y ago
  When that happened to us, we found an article showing Tesla got hacked the same week as us (was aws) and they got the money back, so why not us? We got the money back and fired the guy who had a jenkins opened without password, granting terminal access to anyone.
  • ayewo 4y ago
    You say: "We got the money back and fired the guy who had a jenkins opened without password, granting terminal access to anyone." Why did you fire the employee (?) so quickly? Did he have a history of negligence and/or incompetence on the job and this was the last straw that broke the camel's back?
  • oasisbob 4y ago
    Why was that guy running Jenkins without a password?
    • onion2k 4y ago
      Years ago when I actually did any devops running services without a password was common. They would configured to only be accessible using SSH from a locked down IP range. It's far more secure than password based access (if you get it right).
      • 988747 4y ago
        Yes, but that was a design/security choice. What OP described seems like a pure negligence ("setting up authentication is hard, so let's skip it").
  • onion2k 4y ago
    It's a shame you failed to learn a lesson about how security is something the entire team is responsible for, and how a mistake like deploying something without a password is a failure of your processes rather than a failure of any one person. By firing the guy all you've done is made everyone paranoid which will slow you down; similar mistakes are still just as likely to happen again.
• mkl95 4y ago
  I can't help you with the legal side of things, but moving forward I advise hiring some security-aware infra guy. The root cause of most of these incidents is some human being incompetent (leading to things like poor security and relying on manual processes) or reckless.
• gigel82 4y ago
  This is a conundrum. On one hand, I understand how frustrating something like this can be. But on the other hand, your cloud provider did provide those services that you're being billed for. So they did incur costs, why would they just eat those costs? Unless they're somehow at fault by exposing your credentials or making it easier for hackers to log in without 2FA or something of that nature. If you're using a credit card to pay (though can't see a credit card having a 200k limit, even business) you might want to see if they can help (though it's not the credit card itself that was stolen, so it's unlikely they'd cover you). Otherwise, I'd imagine you're SOL unless you have some other insurance you can rely on.
  • sam0x17 4y ago
    > why would they just eat those costs? Beacuse the public indignation directed at cloud companies who don't always eat the costs in these situations vastly outweighs the cost of simply eating these costs, at least for cloud companies at the top tier of economies of scale (AWS, GCP, Azure, etc) If AWS didn't always eat costs like this, startups might think twice before using AWS, etc, etc.
    • kelnos 4y ago
      Exactly this. Cloud providers have to acknowledge that building software on their platforms is necessarily complex, and inevitably bugs can cause extremely undesirable behavior. These providers are already charging a premium for instant, on-demand provisioning and nearly limitless pay-as-you-go resources, and know that there are too few guardrails in place to prevent accidental runaway situations. "Goodwill" has value to a corporation. Taking a hard line against legitimate mistakes that anyone (yes, anyone) can make costs them goodwill, and costs them customers. And beyond that, while accidental/fraudulent usage doesn't cost them $0, the services are marked up to the point that they probably doesn't really lose that much by forgiving the charges.
• sarlalian 4y ago
  Please setup billing alerts, know what your daily spend should be, add a little for if things grow a little unexpectedly. But you should absolutely be getting alerts if your spend is out of the ordinary for > 2 hours.
  • justsomehnguy 4y ago
    Please, demand what you can have a hard limit on spending for the service/account or just a banal pre-pay. It's amazing what I can have a pre-paid account for a VPS hosting in Nicaragua, yet Amazon doesn't have this as an option. /rant
    • janosdebugs 4y ago
      What is AWS/Azure supposed to do if you run out of funds? Delete your resources? Storage costs money too, simply shutting down the services won't quite cut it. VPS consumption is relatively easy to predict, complex cloud services are not.
      • popotamonga 4y ago
        same thing it does when you dont pay. all goes down. databases readonly mode. Eventually it all gets deleted if you dont pay for a another month or so.
      • matkoniecz 4y ago
        It can be configurable. For my use I would be fine with terminating all activities and use reserve funds to pay for data storage.
      • igorkraw 4y ago
        Ask the customer to pay collateral when enabling this for a service, then use that to pay for the data storage in the shutdown grace period, reimburse it (or count it against future use) if it was never used. You can also easily add this on all the elastic services,e.g. your price per GB simply goes up a bit for S3 in the first month, until the amount of data you have is fully in your "insurance". Or, you know, price it in and do your own risk estimates to make this a seemless experience, but AWS got away with not offering anything, so I'm already assuming you want to be passing all the cost to the customer (heck, amazon, if you are reading this, you can even skim a little bit of the "insurance" money! )
• matkoniecz 4y ago
  If not done already, prepare off-cloud backup and consider migration plans. There is some risk that they will terminate your account.
• just-tom 4y ago
  I'm quite surprised that there isn't some kind of monthly budget control. For every new project I set the budget to be 4-5x my expected expense.
• madaxe_again 4y ago
  I’m sorry to hear this, this is a tough situation. Microsoft might, but are unlikely, to help you out. Similar situation with your bank. Neither face a legal obligation to help you, just potential bad PR if they don’t. Your best bet may be bankruptcy. It sounds terrible, but assuming you have an LLC/Ltd company, you can clear out your coffers, wind up, pay them pennies on the dollar, if anything, and start a new business. You may need to go through an lawyer or administrator depending on bankruptcy laws where you are. I’ve taken a client through this, after a similar situation - they ended up with a vast bill to a supplier brought about by someone else using their credentials, and the supplier not being willing to budge. It cost about a week of time and about $2k in legal fees. I’ve also been on the receiving end, where I presented a legitimate invoice and rather than pay the client reincorporated and kept the IP - which sucks, but Microsoft will be insured against insolvencies, so I wouldn’t feel bad about it. You’re just allowing their insurer to help everyone out.
  • kelnos 4y ago
    > Microsoft might, but are unlikely, to help you out. Is that true? I have no experience with Microsoft, but I've heard quite a few stories of Amazon crediting AWS accounts when customers write in to say their account was compromised. Or even cases when the customer themselves screwed up some permissions in a way that ended up costing an arm and a leg. Hard to believe this practice would be unique to AWS.
    • madaxe_again 4y ago
      You hear the stories with the happy outcomes - I would wager this is selection bias at work, as I’ve seen several instances firsthand where the outcome has been that the bill stands. It’s still worth trying, as a first resort, of course, but it isn’t something I’d count on.
• nurettin 4y ago
  I asked microsoft support to vaive the past two months of billing because I left open a database cluster which I created for testing purposes. They promptly replied, took me through the steps and vaived the bills. So maybe just file a support ticket, or have your distributor file a ticket for you?
• m4jor 4y ago
  You had no 2FA enabled?
• ivanchaz 4y ago
  That's unfortunate situation. It happened to me once before (though, we was using AWS that time. And, I believe the cost was smaller than the one you have right now). What we did to recover the cost was to contact the account manager for our region at the time. So, maybe you could have better luck trying to find the particular person in linkedin. Or, have you tried opened a ticket from Azure console? Nonetheless, I hope after everything has been settled down, you won't fire anyone (and treat it as learning opportunity)
• teeray 4y ago
  I wonder if you can take out an insurance policy against this. Many of them have cyber-fraud coverage… perhaps this would qualify.
• cheri9 4y ago
  [dead]