6 ms·

Microsoft leaks TLS private key for cloud ERP product

210 points by mgliwka 9y ago

lbtuda 9y ago
If we have an global scale attacker which can sniff the internet traffic to specific hosts, or an attacker capable of BGP hijacking, this attacker would be able to attack all companies who use Microsoft Dynamics (industrial espionage?). He would just need to sniff the credentials and log in. After the Snowden leaks we all know that this is possible.
- rjzzleep 9y ago
  > After the Snowden leaks we all know that this is possible. This bothers me. A lot of people might remember Narus and Narusinsight[1] > Narus is noted for having created NarusInsight, a supercomputer system, whose installation in AT&T's San Francisco Internet backbone gave rise to a 2006 class action lawsuit by the Electronic Frontier Foundation against AT&T, Hepting v. AT&T. But sure, I'm certain there are people that actually worked for these companies and can tell you how their stuff doesn't really work as advertised. Anyway, what exactly new did Snowden bring to the table in this particular context? [1]: https://en.wikipedia.org/wiki/Narus_(company)#NarusInsight https://en.wikipedia.org/wiki/Narus_(company)#NarusInsight
  - jgrahamc 9y ago
    > Anyway, what exactly new did Snowden bring to the table in this particular context? A "heroic" character that people could identify with to frame the rest of the story.
  - lbtuda 9y ago
    xkeyscore --> https://en.wikipedia.org/wiki/XKeyscore https://en.wikipedia.org/wiki/XKeyscore
  - jrochkind1 9y ago
    > Anyway, what exactly new did Snowden bring to the table in this particular context? Scale and scope. And more attention to some things that were already known by those paying attention, true.
  - sp332 9y ago
    PRISM https://en.wikipedia.org/wiki/PRISM_(surveillance_program) https://en.wikipedia.org/wiki/PRISM_(surveillance_program) And also that GCHQ tapped Google's datacenter links. https://arstechnica.com/tech-policy/2013/10/new-docs-show-nsa-taps-google-yahoo-data-center-links/ https://arstechnica.com/tech-policy/2013/10/new-docs-show-ns... After that information came out, Google started encrypting their internal networks.
- Donald 9y ago
  Also, if such an actor was storing encrypted traffic as well, they could now easily decrypt this traffic en mass.
  - moondev 9y ago
    Yikes.. So compromise a public wifi and MITM + store any traffic pointed at the affected domain(s), then simply sign up for their own ERP account, download the key and decrypt.
    - willstrafach 9y ago
      Only if PFS is not in use though.
    - kbirkeland 9y ago
      Most public wifi hotspots I've seen are unencrypted, so there'd be no need to do a MitM - just be within range to decode the client and AP transmissions.
      - moondev 9y ago
        Even on an unsecured network would the transmissions not be encrypted via tls from your computer to the server?
  - lbtuda 9y ago
    https://nsa.gov1.info/utah-data-center/ https://nsa.gov1.info/utah-data-center/
  - tinus_hn 9y ago
    If you use ephemeral keys, as you should, the contents of past encryption is still secure if the private key gets leaked. The encryption keys are newly generated each time, the certified private/public key pair is only used to validate these encryption keys belong to server that has the certified keys. This is called Forward Secrecy.
    - 9y ago
baybal2 9y ago
This is rather common security hole on different app and web hosting services that provide "We will spare you the pain of getting ssl cert by yourself, and get you one for free," but which instead give you their wildcars cert. Wildcard certs in a shared VPS-like environment is a red flag.
- hannob 9y ago
  If you are aware of such issues specifically feel free to contact me. I'll make sure they get handled properly.
rootsudo 9y ago
Darn, I saw this, it's also a bit similar on the onmicrosoft.com domain for 365 clients. Interesting, Interesting.
moondev 9y ago
Does the cloud ERP product use only the private key for rdp authentication?
- mgliwka 9y ago
  No, the RDP authentication works using a Username/Password-pair displayed in the Life Cycle Services control panel. The shared private key was used to encrypt and authenticate the web traffic for all customers.
  - moondev 9y ago
    I see, I agree that is still pretty bad. Wonder how many other RDP+IIS products use wildcard certs and share this same vulnerability to leak keys?
  - paulirwin 9y ago
    It makes sense that their "quick" solution is to give each customer their own cert/key, but I'm surprised that they didn't go with a reverse proxy approach from the beginning, where the wildcard key would be kept securely at their reverse proxy layer that no customers would have access to. Having to maintain TLS certs in IIS on each of these VMs seems like it would cause them more work than it's worth. Having experience with the platform, do you have any insight into why they would do TLS at the VM rather than via a reverse proxy?
    - mgliwka 9y ago
      As far as I remember, there was already a Powershell script as part of their deployment mechanism in place to deploy the certificates in the first place. Just the certificate provided to this script was a wildcard one. I agree on the reverse proxy part, but am not familiar enough with the product and Microsofts history to comment on that.
jrochkind1 9y ago
Inability to get a response upon reporting the vulnerability without heroic measures is more disturbing to me than the vulnerability. Vulnerabilities _will_ happen (even if this one is particularly... basic), but if the company isn't paying attention to responsible private disclosures, that's even more disastrous. MS's eventual changing of their key practices suggests that they do agree this was a vulnerability, so discussions of how bad this vulnerability is seem irrelevant to me. The disturbing thing is not about how bad the vulnerability is, but about inability, as far as we can tell, to get MS to even analyze and evaluate the vulnerabilty without heroic measures including journalists threatening exposure.
- da_chicken 9y ago
  Exactly. It's very easy to fix a poor security configuration. It's very difficult to fix a poor security culture.
  - mgliwka 9y ago
    There is an article from 2007 [1], quote: "Working in the Microsoft Security Response Centre (MSRC) has been voted number six out of the ten worst jobs in science in 2007 [...]" But I believe (I hope?) a lot has changed since then. [1] https://www.computerworld.com.au/article/205327/microsoft_staffers_debate_worst_jobs_science_nomination/ https://www.computerworld.com.au/article/205327/microsoft_st...
sulam 9y ago
For those, like me, who were mightily confused by the timeline posted at the end — he switches to euro style dates in September and October but used US style dates in August and December.
- mgliwka 9y ago
  Thank you for noticing, somebody is reading until the end ;-) I will fix this right away! Update: fixed.
  - sulam 9y ago
    Future readers thank you. :)
  - jwilk 9y ago
    Please consider using a sane date format. https://xkcd.com/1179/ https://xkcd.com/1179/
    - jazoom 9y ago
      What's not "sane" about the format most of the world uses? dd/mm/yy Maybe you meant "unambiguous"?
      - nerdponx 9y ago
        "Ambiguous" and "insane" are best friends.
        jazoom 9y ago
        In this case it's only ambiguous because a populated country uses an insane version and plasters those insanely-formatted dates all over the world wide web.
        tatersolid 9y ago
        The middle-endian format makes perfect sense as it is the same order of elements used in spoken English: “April 22nd 2017” = “4/22/2017”. I’ve never heard anyone say “22nd of April 2017” in normal conversation. Maybe on a wedding invite.
hannob 9y ago
The Golem.de text that is mentioned in the article is now also available in an English translation: https://www.golem.de/news/microsoft-dynamics-365-wildcard-certificate-with-a-private-key-for-everyone-1712-131544.html https://www.golem.de/news/microsoft-dynamics-365-wildcard-ce...
mycall 9y ago
> “[c]ontrols exist in production environments that render the described technique ineffective [..]" That's a pretty naive look at penetration testing. Look, we have controls, so that CAN'T happen. Right.
deleted 9y ago
[deleted]
tialaramex 9y ago
Hanno points out two big problems here that are not so obvious. Firstly there is supposed to be a Problem Reporting mechanism for the certificates themselves. If you can't get the application developer to pull their finger out but you have acquired a Private Key you shouldn't have, you should be able to file such a report and get satisfaction in hours not weeks. Sophisticated users can prove they have the key without revealing it, but that's polite rather than obligatory. This mechanism either wasn't apparent to the problem's discoverer or didn't work. Neither is OK. Secondly, even without being able to get a copy of the shared private key, sharing is a risk. If we confuse a remote client into talking to our service rather than the one they expected, they'll never know the difference because the keys check out fine. The latter is why wildcards, while convenient, are not always a wise choice for security.

210 points by mgliwka 9y ago

If we have an global scale attacker which can sniff the internet traffic to specific hosts, or an attacker capable of BGP hijacking, this attacker would be able to attack all companies who use Microsoft Dynamics (industrial espionage?). He would just need to sniff the credentials and log in. After the Snowden leaks we all know that this is possible.

rjzzleep 9y ago
> After the Snowden leaks we all know that this is possible. This bothers me. A lot of people might remember Narus and Narusinsight[1] > Narus is noted for having created NarusInsight, a supercomputer system, whose installation in AT&T's San Francisco Internet backbone gave rise to a 2006 class action lawsuit by the Electronic Frontier Foundation against AT&T, Hepting v. AT&T. But sure, I'm certain there are people that actually worked for these companies and can tell you how their stuff doesn't really work as advertised. Anyway, what exactly new did Snowden bring to the table in this particular context? [1]: https://en.wikipedia.org/wiki/Narus_(company)#NarusInsight https://en.wikipedia.org/wiki/Narus_(company)#NarusInsight
- jgrahamc 9y ago
  > Anyway, what exactly new did Snowden bring to the table in this particular context? A "heroic" character that people could identify with to frame the rest of the story.
- lbtuda 9y ago
  xkeyscore --> https://en.wikipedia.org/wiki/XKeyscore https://en.wikipedia.org/wiki/XKeyscore
- jrochkind1 9y ago
  > Anyway, what exactly new did Snowden bring to the table in this particular context? Scale and scope. And more attention to some things that were already known by those paying attention, true.
- sp332 9y ago
  PRISM https://en.wikipedia.org/wiki/PRISM_(surveillance_program) https://en.wikipedia.org/wiki/PRISM_(surveillance_program) And also that GCHQ tapped Google's datacenter links. https://arstechnica.com/tech-policy/2013/10/new-docs-show-nsa-taps-google-yahoo-data-center-links/ https://arstechnica.com/tech-policy/2013/10/new-docs-show-ns... After that information came out, Google started encrypting their internal networks.
Donald 9y ago
Also, if such an actor was storing encrypted traffic as well, they could now easily decrypt this traffic en mass.
- moondev 9y ago
  Yikes.. So compromise a public wifi and MITM + store any traffic pointed at the affected domain(s), then simply sign up for their own ERP account, download the key and decrypt.
  - willstrafach 9y ago
    Only if PFS is not in use though.
  - kbirkeland 9y ago
    Most public wifi hotspots I've seen are unencrypted, so there'd be no need to do a MitM - just be within range to decode the client and AP transmissions.
    - moondev 9y ago
      Even on an unsecured network would the transmissions not be encrypted via tls from your computer to the server?
- lbtuda 9y ago
  https://nsa.gov1.info/utah-data-center/ https://nsa.gov1.info/utah-data-center/
- tinus_hn 9y ago
  If you use ephemeral keys, as you should, the contents of past encryption is still secure if the private key gets leaked. The encryption keys are newly generated each time, the certified private/public key pair is only used to validate these encryption keys belong to server that has the certified keys. This is called Forward Secrecy.
  - 9y ago

baybal2 9y ago

This is rather common security hole on different app and web hosting services that provide "We will spare you the pain of getting ssl cert by yourself, and get you one for free," but which instead give you their wildcars cert. Wildcard certs in a shared VPS-like environment is a red flag.

hannob 9y ago
If you are aware of such issues specifically feel free to contact me. I'll make sure they get handled properly.

rootsudo 9y ago

Darn, I saw this, it's also a bit similar on the onmicrosoft.com domain for 365 clients. Interesting, Interesting.

moondev 9y ago

Does the cloud ERP product use only the private key for rdp authentication?

mgliwka 9y ago
No, the RDP authentication works using a Username/Password-pair displayed in the Life Cycle Services control panel. The shared private key was used to encrypt and authenticate the web traffic for all customers.
- moondev 9y ago
  I see, I agree that is still pretty bad. Wonder how many other RDP+IIS products use wildcard certs and share this same vulnerability to leak keys?
- paulirwin 9y ago
  It makes sense that their "quick" solution is to give each customer their own cert/key, but I'm surprised that they didn't go with a reverse proxy approach from the beginning, where the wildcard key would be kept securely at their reverse proxy layer that no customers would have access to. Having to maintain TLS certs in IIS on each of these VMs seems like it would cause them more work than it's worth. Having experience with the platform, do you have any insight into why they would do TLS at the VM rather than via a reverse proxy?
  - mgliwka 9y ago
    As far as I remember, there was already a Powershell script as part of their deployment mechanism in place to deploy the certificates in the first place. Just the certificate provided to this script was a wildcard one. I agree on the reverse proxy part, but am not familiar enough with the product and Microsofts history to comment on that.

jrochkind1 9y ago

Inability to get a response upon reporting the vulnerability without heroic measures is more disturbing to me than the vulnerability. Vulnerabilities _will_ happen (even if this one is particularly... basic), but if the company isn't paying attention to responsible private disclosures, that's even more disastrous. MS's eventual changing of their key practices suggests that they do agree this was a vulnerability, so discussions of how bad this vulnerability is seem irrelevant to me. The disturbing thing is not about how bad the vulnerability is, but about inability, as far as we can tell, to get MS to even analyze and evaluate the vulnerabilty without heroic measures including journalists threatening exposure.

da_chicken 9y ago
Exactly. It's very easy to fix a poor security configuration. It's very difficult to fix a poor security culture.
- mgliwka 9y ago
  There is an article from 2007 [1], quote: "Working in the Microsoft Security Response Centre (MSRC) has been voted number six out of the ten worst jobs in science in 2007 [...]" But I believe (I hope?) a lot has changed since then. [1] https://www.computerworld.com.au/article/205327/microsoft_staffers_debate_worst_jobs_science_nomination/ https://www.computerworld.com.au/article/205327/microsoft_st...

sulam 9y ago

For those, like me, who were mightily confused by the timeline posted at the end — he switches to euro style dates in September and October but used US style dates in August and December.

mgliwka 9y ago
Thank you for noticing, somebody is reading until the end ;-) I will fix this right away! Update: fixed.
- sulam 9y ago
  Future readers thank you. :)
- jwilk 9y ago
  Please consider using a sane date format. https://xkcd.com/1179/ https://xkcd.com/1179/
  - jazoom 9y ago
    What's not "sane" about the format most of the world uses? dd/mm/yy Maybe you meant "unambiguous"?
    - nerdponx 9y ago
      "Ambiguous" and "insane" are best friends.
      - jazoom 9y ago
        In this case it's only ambiguous because a populated country uses an insane version and plasters those insanely-formatted dates all over the world wide web.
        tatersolid 9y ago
        The middle-endian format makes perfect sense as it is the same order of elements used in spoken English: “April 22nd 2017” = “4/22/2017”. I’ve never heard anyone say “22nd of April 2017” in normal conversation. Maybe on a wedding invite.

hannob 9y ago

The Golem.de text that is mentioned in the article is now also available in an English translation: https://www.golem.de/news/microsoft-dynamics-365-wildcard-certificate-with-a-private-key-for-everyone-1712-131544.html https://www.golem.de/news/microsoft-dynamics-365-wildcard-ce...

mycall 9y ago

> “[c]ontrols exist in production environments that render the described technique ineffective [..]" That's a pretty naive look at penetration testing. Look, we have controls, so that CAN'T happen. Right.

deleted 9y ago

[deleted]

tialaramex 9y ago

Hanno points out two big problems here that are not so obvious. Firstly there is supposed to be a Problem Reporting mechanism for the certificates themselves. If you can't get the application developer to pull their finger out but you have acquired a Private Key you shouldn't have, you should be able to file such a report and get satisfaction in hours not weeks. Sophisticated users can prove they have the key without revealing it, but that's polite rather than obligatory. This mechanism either wasn't apparent to the problem's discoverer or didn't work. Neither is OK. Secondly, even without being able to get a copy of the shared private key, sharing is a risk. If we confuse a remote client into talking to our service rather than the one they expected, they'll never know the difference because the keys check out fine. The latter is why wildcards, while convenient, are not always a wise choice for security.