28 ms·
Microsoft warns thousands of cloud customers of exposed databases
- deleted 5y ago[deleted]
- abledon 5y ago>Microsoft agreed to pay Wiz $40,000 for finding the flaw and reporting it, A vulnerability on this scale.. and they pay out only 40K ? I don't understand, this is peanuts compared to the damages this could cause no? Why do they not pay out respectable amounts?
- cheschire 5y agoseriously if they started paying out something comparable to small ransoms, some folks may be swayed back from the dark side.
- inter_netuser 5y agomulti-millions?
- SilverRed 5y agoThey pay out the amount that researchers find worthwhile for the time they spent. The vast majority of people do not want to commit a serious crime in order to make a little bit more money so Microsoft does not have to pay the same rates.
- ClumsyPilot 5y ago"They pay out the amount that researchers find worthwhile for the time they spent." How have you confirmed that this amount is accurate? Have there been bo zero-days on the black market, instead all hackers have gone to microsoft? Would the number of hack on black market change is amount paid out increased 10x? It seems you have no basis for claiming that the amount paid is optimal
- SilverRed 5y agoI actually didn't claim it was optimal. I simply disputed the claim that they should pay equal to the damages that could have been caused.
- beermonster 5y agoThat was my first thoughts..
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- wil421 5y agoWasn’t Wiz founded by an ex Microsoft Cloud security engineer?
- beermonster 5y agoYes, second paragraph in the article “ Wiz Chief Technology Officer Ami Luttwak is a former chief technology officer at Microsoft's Cloud Security Group.”
- mensetmanusman 5y agoYes, very interesting strat. Start another company to warn your prior employer who wasn’t listening… and get paid more.
- onionisafruit 5y agoYou just reminded me of something I heard on a recent Darknet Diaries episode: > Some of the best hackers within the NSA turned into independent contractors so they could work faster and make more money, but were on the outside? This is one of those things that someone like Microsoft is afraid of, too. If they pay too much for bugs, then some of their internal bug hunters might decide to quit but keep doing the same thing; just make more money on the outside. He even used Microsoft as an example. I wonder if they paid less because the researcher was a turncoat in their mind.
- eljimmy 5y agoThey don’t need to pay out anything.
- belter 5y agoYou have a point there and the researcher has also no obligation of disclosing it to them.
- latch 5y agoBut they do have obligations that make it otherwise worthless to the them.
- ClumsyPilot 5y agoThey could contact a three letter agency, that's totally legal. Or they could release information publically, without informing M$
- Galanwe 5y agoThis logic could apply to any profession. You just happen to be on the expert side for this one. Should a surgeon ask you for millions? After all he saved your life. What about the mechanic that finds an issue with your breaks? If you paid someone not for the work done, but for the resulting loss if criminals exploited something, a lot of professions would ask for crazy prices.
- athorax 5y agoOr maybe a biomedical researcher discovers you have a special mutation that makes your kidney worth millions on the black market and informs you instead of the guys with a tub full of ice ready to go
- Gimpei 5y agoYou're thinking about this in moralistic terms of deservedness. The shock I encountered at the 40K was that it wasn't enough to incentive future security researchers to find bugs. This just seems like a bad business decision.
- asah 5y agoI think people underestimate how much work it is to find and report security issues... Including the unpaid labor of training... Anyway, seems like an NFT with the instructions would've sold for far more than 40k...
- onionisafruit 5y agoAren’t the contents of NFTs public before the sale? I agree with your general point that there were buyers willing to pay more, but NFT might not be the right vehicle for it.
- cwilkes 5y agoIt could be an NFT of the digitally signed exploit. Or something, NFTs confuse me
- mikem170 5y agoMaybe it's an imperfect market. Is there an auction site for zero days where Microsoft could bid on their own exploits? I'd be in favor of something like this. Why should the criminal justice system and every tax payer be on the hook for protecting these big companies from the consequences of their bug-ridden software?
- jonas21 5y agoBecause the rule of law?
- mikem170 5y agoDo you think laws are sacrosanct and can't be examined or changed? Do you think they don't need to be justified? There's many examples where the law doesn't stop companies from ripping off individuals, obfuscating terms and conditions to their advantage, hiding prices, etc. Recent example that bothers me is the paperwork at a doctors office, expecting the patient to be responsible for all bills insurance doesn't pay, without letting the patient know the costs involved, basically signing a blank check, even on a signature pad where the contract can't be seen, being told "this is just consent for the exam". The justification that these are standard forms is not a justification. Too often big companies get subsidized at public expense. That money goes right from taxpayers to shareholders. Microsoft should deal with the fallout from it's less secure software, not taxpayers. Instead these big companies race ahead to make features and sales, playing games with juristictions to avoid paying taxes, etc. Doesn't seem fair, and just saying "because rule of law" doesn't seem like a good explanation of why we should defend this kind of thing. Maybe a bug exploit should be considered protected free speech. How about that law?
- 0x426577617265 5y agozerodium does payout for zero days -- but I don't think this would qualify. Scroll down to see list of prices. https://zerodium.com/program.html https://zerodium.com/program.html
- 1vuio0pswjnm7 5y ago"... this is peanuts compared to the damages this could cause no?" But if no one can succesfully sue for those damages, then what is it really worth. The idea that Microsoft with its gargantuan resources cannot discover these mistakes and has to wait for "security researchers" to discover them and then pay them a paltry sum for their "valuable work" makes little sense; a more sensible interpretation is that the company has little incentive to find such mistakes because it has little exposure to potential liability arising from them. It needs to stage "security theater" to avoid reputational damage but does not need to achieve real results. Mistake after mistake, the stock price is not affected. Regarding an ongoing lapse in quality control that spans four decades, it has no skin in the game.
- kerng 5y agoI think payouts are defined in bug bounty programs upfront, so there are naturally upper bounds.
- chovybizzass 5y agono firewalls and dbs w/ no authentication...happened to mongodb users a few years ago.
- dantiberian 5y agohttps://www.wiz.io/blog/chaosdb-how-we-hacked-thousands-of-azure-customers-databases https://www.wiz.io/blog/chaosdb-how-we-hacked-thousands-of-a... is from the researchers and has more details.
- redwood 5y agoCosmos DB can't be used for anything real, can it?
- jiggawatts 5y agoIn principle it sounds good, but from what I've heard it is over-priced and far too slow compared to typical IaaS-hosted solutions. Notably, some of the customers mentioned in the security researcher's blog are the type that have exploding wallets. As in: "My wallet is bursting open because there's too much money in it! Mr Cloud Sales Guy, can you help me with this problem?"
- to11mtm 5y agoHonestly? I don't know for sure. On one hand, my understanding is that they used TLA+ to validate the model[0]. One would assume that if they went to that trouble there is at least a specification for how it should work. OTOH, Specifications can be flawed. This security issue appears unrelated to the design of Cosmos DB itself tho. [0] - https://en.wikipedia.org/wiki/TLA%2B#Industry_use https://en.wikipedia.org/wiki/TLA%2B#Industry_use
- buitreVirtual 5y agoFor formal verification to catch the bug, the bug would have to be at the design level and the model would have to have enough detail to include the design bug.
- manigandham 5y agoTLA+ is used to design the database itself, including the storage layer, distributed operations, and consensus/consistency models. This bug is a security issues in an entirely separate component. No formal logic is going to warn you about exposed read/write keys.
- EMM_386 5y agoInteresting what the weak point was here. > The flaw was in a visualization tool called Jupyter Notebook, which has been available for years but was enabled by default in Cosmos beginning in February.
- theshadowknows 5y ago“ A federally contracted research lab tracks all known security flaws in software and rates them by severity.” …where can one see this list?
- er4hn 5y agoProbably the MITRE CVE database
- thanksforfish 5y agohttps://cve.mitre.org/cve/ https://cve.mitre.org/cve/
- jiggawatts 5y agoSpeaking of CosmosDB -- other than this security issue -- has anyone here ever used it for anything practical? To me it seemed vaguely interesting but a little bit overpriced for real-world usage outside of fortune 500s or big government, where budgets aren't always a concern.
- thrixton 5y agoSure, as an entity store for run of the mill CRUD apps. It has a MongoDB API Layer and a Cassandra API Layer (as well as SQL which is preferred). Works just fine, apparently cost can blow out though with high usage, I haven't got there yet. Does have a free tier.
- Bellyache5 5y agoI tried Cosmos via the Mongo compatibility layer and performance was abysmal, even when cranking the provisioned performance way up. I ended up getting significantly better performance from a reasonably sized MongoDB container running on AKS.
- tiew9Vii 5y agoWas this taking a MongoDB database app and pointing to CosmosDB or was this developing a app around CosmosDB's partitioning etc but using the MongoDB API for read/write? The former, lift and shift scenario I can understand as you need to structure your data to work best with CosmosDB/DynamoDB etc. If it's the latter that is useful to know as I'm currently building a POC on Azure so I should watch out for it.
- Bellyache5 5y agoYes, so not the ideal use case. The compatibility Cosmos offered was nice and didn't present any issues, it's just the performance that suffered. This was all part of a benchmarking exercise to figure out how to scale an app with minimal changes so we expected some performance penalty, just not as steep as what we observed.
- haolez 5y agoHow can CosmosDB's architecture and design make it possible to have a read-write key that works on all customers' databases? Yikes!
- buitreVirtual 5y agoI think each database has its own key, but all keys were accessible via Jupiter Notebook.
- zeusk 5y agobecause there isn't. There's a vulnerability in some jupyter notebook implementation allowing them access to unrelated notebooks which hold read/write keys to database they're connected to.
- wokwokwok 5y agoIt looks more like they had all the jupyter notebooks for all customers sitting side by side, and each notebook had a master key for the related DB. ...but due to a poor deployment strategy, the instances were not sandboxed from each other. (and yes, you would be correct in thinking that was a catastrophic failure; it's like having a folder with a directory per customer and a master access key in each folder, and WOOOPS, didn't realize people might look at ../Amazon/master_key.txt It's pretty hard to brush this one off; it's a colossal screw up)
- shirt 5y agoYou got it :)
- onionisafruit 5y agoSeveral people had to have thought it was a good idea to add the jupyter notebook service with the primary keys to every cosmosdb instance even though a tiny portion of customers would ever even look at it. Yet it’s still fine to add that extra attack surface. I’ve seen some azure documentation that stresses you should never ever use your primary keys in a deployed service. Yet they do it willy nilly.
- dlor 5y agoFrom the researchers: > So you can imagine our surprise when we were able to gain complete unrestricted access to the accounts and databases of several thousand Microsoft Azure customers, including many Fortune 500 companies. This is basically worst-case scenario for a database provider. https://www.wiz.io/blog/chaosdb-how-we-hacked-thousands-of-azure-customers-databases https://www.wiz.io/blog/chaosdb-how-we-hacked-thousands-of-a...
- onionisafruit 5y agoIt’s worst-case for their customers. They’re the ones who need to figure out what data to consider compromised. Microsoft gets a temporary mark on their reputation until their pr and marketing departments make us forget about it. That’s it. Nobody is even going to lose their bonus over this.
- DaiPlusPlus 5y agoAlso, it's CosmosDB - it has a certain reputation internally in the Azure org...
- geoduck14 5y agoPlease elaborate
- outside1234 5y agoIt sucks, is expensive, and was designed by someone who was fired.
- lloydatkinson 5y agoDo you have some sources? What was they fired for? I thought they got Leslie Lanport to design it
- onionisafruit 5y agoLet me take a wild guess. I bet they work for GitHub now.
- deleted 5y ago[deleted]
- skee_0x4459 5y agois this why i got a weird notification on my iphone lastnight that a bunch of my passwords were exposed in a data breach? amipwned didnt have anything new and my iphone didnt appear to provide any more details about where and when the breach occurred.
- YLYvYkHeB2NRNT 5y agoDefaults matter.
- macintux 5y agoIt’s quite amazing what poor practices arise in shared environments. About 20 years ago I was working with a large company that had dedicated servers in a huge east coast data center. The servers were firewalled, but they also were attached to a secondary network for scheduled backups. I decided I should explore that further, and sure enough I was able to connect via SSH to other customers’ servers. I alerted the hosting company, but they didn’t seem to take it very seriously.
- onionisafruit 5y agoWe should never believe a hosting or cloud provider when they say we have a segregated instance. They may even believe it themselves, but there is always some service connecting them all.
- h2odragon 5y agoVisited a BIG datacenter; top grade security and visitor's passes and all the trimmings. Buddy giving me the tour shows me a couple of the things they show to VIPs, its early Sunday morning and they still caught me trying to take a couple pictures, very high grade and attentive folks. Then we go back to his office, and load up the 1,000lb of computer gear he's donating and I'm transporting. We wheel 3 cartloads of stuff down the back elevator, through the secured DC floor we saw through glass earlier that i couldn't take a picture of, and out an open back door right near where i parked my truck. no further interaction with security, no badge waving, no questions asked.
- macintux 5y agoReminds me of Michael Keaton’s line in The Paper: > A clipboard and a confident wave will get you into any building in the world!
- blibble 5y ago> We rarely see security teams move so fast! They disabled the vulnerable notebook feature within 48 hours after we reported it. It’s still turned off for all customers pending a security redesign. if this is amazing I'd hate to see bad
- onionisafruit 5y agoThat’s what I was thinking. I was once responsible for a RCE vulnerability that was reported to hackerone on Christmas afternoon. Even then I took less than 1/48th the time to disable the feature and had a patch ready in half the time it took them to just disable jupyter notebooks. I’m not claiming to be better at my job than their response team. I’m almost certainly not. The difference is I didn’t have to have meetings with 20 different internal stakeholders before acting.
- grawprog 5y agoWhen I worked for a shop running windows powered CNC machines, an update broke a bunch of more updated machines one of our partner shops used, dozens of shops were affected by this. It took a few hours for Microsoft to fix the problem, they responded within a half hour or so of the first complaint. Two days seems like a long time for Microsoft to respond when commercial customer's businesses are at stake. Say what you want about Microsoft, i don't really like them, but their enterprise/commercial support is honestly pretty top notch, I'm really surprised it took them that long to get on top of that considering the calibre of customers involved.
- kerng 5y agoYeah, 48 hours doesn't seem too fast - but better then multiple months as with Exchange
- mrtweetyhack 5y agothe words Microsoft and security don't go together
- ctvo 5y agoThis is bonkers. - Microsoft introduced Jupyter notebook support for Cosmos (who even asked for this?) and it was turned on for all customers automatically February 2021 - Security researchers find a way to break out of the container running the notebook and get long lived credentials to other people’s databases - The credentials can be used off network. I assume with the public Azure API. Don’t let your intern projects go into production without thorough security review.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- ra120271 5y agoAgree and that is why building a secure Cloud Landing Zone for enterprises goes hand in hand enabling developers to have the autonomy with Guardrails This is my early and potentially slightly erroneous understanding of the situation: - Jupiter Notebook support was enabled if you used the SQL API and not the Gremlin or MongoDB API. What is unclear is whether breaking out of the Notbook container gave you access to keys for just instances using SQL API or any API. - CosmoDB is very weak in enforcing identity perimeters so keys are a weak point. Enforcing something like hourly key rotations is left up to the customer to build. - Azure is pretty much an "it's public IP" unless you explicitly make it private. And even when they add controls like Private end points they have weird routing mechanisms that can result in traffic bypassing controls like hub firewalls. - Using things like CosmoDB firewalls, private endpoints, automatic regular key rotations, and only enabling features you need should have helped if this was a hostile breach. Take everything I said with a pinch of salt :)
- manigandham 5y ago"notebook support for Cosmos (who even asked for this?) " Lots of users. The functionality itself is great. CosmosDB is a nice OLTP datastore with a flexible schema, and the ability to run OLAP queries over that data with SQL makes it a powerful tool.
- cm2187 5y agoMy (SQL) database in Azure is protected by a firewall with an IP white list by default. Is it different with Cosmos DB? Or does the Jupyter notebook thing goes around the firewall?
- plasma 5y agoYou should be OK, CosmosDB is not the same as MySQL offering.
- cm2187 5y agoI am not worried about my DB but just curious of whether they were exposing CosmosDB to anyone on the WAN, or whether the Jupyter notebook circumvented the firewall.
- lmeyerov 5y agodefault cosmos db is public ip, and Jupyter is preloaded with your db ip & key, so breaking notebook isolation escalates to db comprimise the blessing/curse of Jupyter being a university oss project and private companies largely not contributing back their multitenant notebook stuff (as hosting is the $ maker) is this kind of scenario. the project provides some multitenant stuff, but guessing not what MS uses. Data science envs generally need wide read access across many data sources, yet mostly only used by a few trusted-yet-security-agnostic power users in an org, which leads to relatively low sec eng infra investments. so my guess is Jupyter security flaws (ex: extension vulnerabilities) are increasingly ripe targets for big escalations.
- auggierose 5y agoI tried using Azure once and couldn’t get it to work for me. No, it’s not me, it’s Microsoft.
- darkcha0s 5y agoOdd, I've used it many times and everything works.
- auggierose 5y agoGood for you!
- scoopertrooper 5y agoSo you couldn't work out how to use something that thousands of other people can use just fine and you conclude it is the maker of that thing, which is the problem. I'm starting to understand why you found using Azure such a challenge.
- auggierose 5y agoI know I didn't make a mistake, and it didn't work. If I remember correctly, something in the UI of allocating or starting VMs crashed. I am not some naive computer user assuming the mistake is with me. I know it's with the software. And I don't have time to work with buggy software. There is better software out there, for example GCP, which has sane APIs and actually works. To be honest, I don't think you understand anything.
- scoopertrooper 5y agoEven accepting all your statements of fact, you are still exhibiting some poor reasoning skills. From your story, it sounds like you briefly tried Azure and something failed and from that you inferred that Azure must fail constantly. In reality, all cloud providers have hiccups now and then and it seems like you may have caught Azure on a bad day. Imagine, if the first day you tried GCP you stumbled into one of its many incidents[1] and gave up on it entirely. You'd have been deprived of a lifetime of joy with its mentally balanced API. [1] https://status.cloud.google.com/summary https://status.cloud.google.com/summary
- CyanLite4 5y agoWhile this is bad, it only affects those who intentionally left their firewall open to the outside world. Any half-decent software engineer knows to firewall the database server, no matter the vendor.
- cutemonster 5y agoI think that's a misunderstanding -- this affects all Cosmo DB users who had gotten Juniper enabled. The way to avoid this, wouldn't have been firewalls, but using another database
- miken123 5y ago> Luttwak's team found the problem, dubbed ChaosDB, on Aug. 9 and notified Microsoft Aug. 12, Luttwak said. Then mailing customers on the 26th. Depending on whether or not this can be classified as a 'personal data breach' (IMO you should treat it as such, since it has such widespread implications) that may be a GDPR violation. The GDPR requires 72h notice to the data protection authorities and notice without 'undue delay' to the controller.
- kerng 5y agoIt doesnt sound like a breach (in the legal sense) occured, at least with the knowledge at hand. Microsoft has bug bounty programs, so such hacking is legally covered and processes seem to have been followed (e.g. researcher didn't unnecessarily access data they didnt own etc). Will be interesting to learn more info down the road
- raffraffraff 5y ago> Wiz Chief Technology Officer Ami Luttwak is a former chief technology officer at Microsoft's Cloud Security Group Now, I bet you anything I could leave my current employer, get a job with a security firm and then find a major hole in my old company's service. Perhaps one that my team warned the CosmoDB team about internally. Or am I just being cynical?
- detaro 5y ago> Perhaps one that my team warned the CosmoDB team about internally. If they still have that hole 18 months later, they kind of deserve it...
- danoise19 5y agoIf you're a CTO of cloud security, you don't warn, you just command.This seems a kind of revenge because the flaw already existed when he was a CTO there. He won 40.000 to "fix" the flaw now, and he already found more other problems on AWS and GCP, this guy will make a million soon. That's why Nadella is bringing more and more hindis to MS, they are at least more trustable!
- siculars 5y agoI think they should add a few zeros onto that reward. These bounty programs should be dropping 10x dollars on these bugs.
- halothymyname 5y agook
- fowl2 5y agoAzure services... Like a Jupyter notebook? :0
- bob1029 5y agoI'll just print this off and add it to my pile labeled "reasons we operate exclusively on-prem"... Its been a long time since someone forced me to break out these articles. Very comforting to see the justification continue with such frequency. Don't get me wrong - I absolutely love parts of what Microsoft does. Just not this whole cloud thing. I get, on average, 3 Microsoft major change notifications every day. Can Microsoft even keep up with their own mess internally? I totally stopped trying from the outside. I just keep an eye on .NET and visual studio stack these days. Everything else winds up in my spam folder.
- ComodoHacker 5y agoFrom the title, I thought Microsoft was scanning its cloud for unsecured public facing DBs and alerting customers. So nice of them. Then I read the article...
- tsjq 5y agowhy are the default config / settings so lax ?
- overwatch02 5y agoexactly my thoughts on this
- scns 5y agoConvenience über alles, user friendlyness gone wrong would be my guess.
- eevahr 5y ago20 billion over 5 years seems a bit much, no?
- sershe 5y agoI used to work on a massively distributed system called Cosmos in Microsoft (there are some public slides and blog posts about it), so I was surprised to hear this, but apparently "Cosmos DB" has nothing to do with Cosmos, it's a completely different system. Azure actually, in my opinion, made the original Cosmos worse via "synergy"; but apparently they ALSO used the name for something unrelated :) WTF? Now I cannot even tell people anymore because the name is forever associated with this debacle.