9 ms·
23,000 HTTPS certificates axed after CEO e-mails private keys
- Bartweiss 9y ago> As the holder of the root certificate used to sign the TLS certificates Trustico was reselling, Symantec was ultimately responsible for ensuring this requirement was being followed, although in fairness, there was probably no way for Symantec to detect a violation. This is in some sense the scariest part in the whole mess. Root authorities might have responsibility, but the basic logic of cert resellers makes an error like this easy and undetectable.
- kerng 9y agoOne part that is missing is why did the CEO say those certs were already compromised before they he mailed them? He mailed them sort of as proof they are compromised?? Im confused. Any thoughts? From the article: "When Rowley asked for proof the certificates were compromised, the Trustico CEO emailed the private keys of 23,000 certificates"
- pfg 9y agoHe's arguing that the various issues that lead to Symantec being gradually distrusted mean that they believed all their existing non-expired, non-revoked certificate were compromised (in the sense that they should not be trusted) and should be revoked. Maybe it's just me, but a company who stores their customer's private keys, sends them around in a ZIP file via email and has a trivial code injection vulnerability on their site, with a process running as root, is probably not the one I want to make that call.
- BuildTheRobots 9y ago> Maybe it's just me, but a company who stores their customer's private keys, sends them around in a ZIP file via email and has a trivial code injection vulnerability on their site, with a process running as root, is probably not the one I want to make that call. On the flip side, if a company with such little regard for security thinks there's a problem with the Symantec, you've got to wonder. They're basically saying "we're dangerously incompetent, however even _we_ can see that Symantec are broken" - and might actually have a point.
- pfg 9y agoSymantec is already getting distrusted and the Web PKI community has decided on a plan that minimizes user impact while doing that. It's very doubtful that a company as incompetent as Trustico has the expertise to come up with a more well-informed decision or a better approach. Certainly attempting to get 50,000 certificates revoked immediately without doing any work to reissue those certificates prior to that is a terrible approach. Considering the fact that they recently switched to a new CA partner, a more likely explanation is that this was a misguided attempt to somehow keep their existing business rather than let DigiCert pick them up somehow. Obviously it failed in a spectacular fashion.
- amluto 9y agoBut this is all nonsense. Symantec never had the private keys, so there was nothing at all unsafe about the certificates.
- dragonwriter 9y ago> Maybe it's just me, but a company who stores their customer's private keys, sends them around in a ZIP file via email and has a trivial code injection vulnerability on their site, with a process running as root, is probably not the one I want to make that call. I dunno. If a company knows that it does all those things, I think that it is 100% correct in making the call that all of its active certificates should be treated as compromised.
- pfg 9y agoThey're not claiming the certificates were compromised because of their own practices, in fact they repeatedly emphasise that there was no issue on their end. I'm not questioning that the certificates became compromised the moment they sent them via email, or really the moment they generated and stored them given the level of security competence they've shown thus far.
- dragonwriter 9y ago> They're not claiming the certificates were compromised because of their own practices I didn't say their reasoning was correct, only the conclusion.
- INTPenis 9y agoGod I hope my co-workers in the PKI group would say "hell no" if our CEO came and asked for the private keys. Show some damn backbone. Though on 2nd thought, maybe having a society with a good social safety net and collective bargaining for workers would be a good foundation for a positive outcome in that situation. If workers are afraid of losing their jobs because they'd lose health insurance and other vital benefits then they'd be more likely to cave. Sorry but being from a former socialist country in the top 5 world ranking of living standards I can't help but make it a social discussion.
- ben_jones 9y agoThe way I see it, most undergrads would write malware, spyware, adware, and bloatware, for a low six figure salary. Likewise I think most developers outside of SV, SEA, NY, etc., would do the same for that amount. I have little faith in software developers and administrators to show a backbone when their salary is on the line.
- cityroasted 9y agoLooks like Trustico is a UK company.
- Piskvorrr 9y agoThe issue wasn't "CEO gets what CEO wants," but that the private keys were even retrievable by someone else than their holder.
- corobo 9y agoWhen these things come up they often have this sort of comment attached. The comments always make it look like the conversation went "Hey can I get them private keys?" and our sysadmin hero either says "no" or our oppressed sysadmin afraid to lose their job says "sure, here you go boss!" I guess where I'm going with this is - do people really do that? Either say yes or no? Where's the part where you explain the reason behind no? "If I do that it compromises xyz putting us at risk of exploits a,b,c and lawsuits 1,2,3" and then a more informed CEO either asks again or drops whatever they were trying to do
- rasz 9y ago
- tialaramex 9y agoA question I still have is what Trustico _hoped_ was going to happen here. Was there some arrangement where if the certs have to be revoked they get a refund? Trustico also don't seem very clear on what exactly they're accusing Symantec of, the Ars article quotes them "We believe Symantec to have operated our account in a manner whereby it had been compromised" and "We believe the orders placed via our Symantec account were at risk and were poorly managed" In the relationship we're talking about here, a reseller to a CA, basically the only valuable information that can be compromised is payment instructions. All the important seeming cryptographic stuff? All of that is allowed to be public by design, the CSR, the Certificates themselves, any validation messages used - all public thanks to asymmetric cryptography. Symantec got in trouble last year for inadequate (in fact basically non-existent) oversight of their RAs, particularly the Korean CrossCert. Unlike a reseller the RAs were trusted to do the domain validation themselves, so if CrossCert said "We checked and it's OK to issue a Thawte cert for www.example.com with this key" then Symantec just did it and didn't ask any questions (though they do seem to have checked they got paid...). Resellers aren't trusted with validation, they're just glorified sales people who just take a cut and help with price discrimination, what's there to "compromise" months after you cease the relationship?
- Ajedi32 9y agoI suspect the logic was something along the lines of "Symantec has acted in an untrustworthy manner (enough to get them distrusted by several major browser vendors) therefore we consider all our customer's certificates that were issued by them to be untrustworthy ("compromised") and want them revoked." Still doesn't explain why they thought that revoking 50k of their customers certificates with less than 24-hours notice would be a good idea though. That really just seems like shooting themselves in the foot.
- rando444 9y agoCan we take a step back even further? I can't figure out the logic that got this started: "These Private Keys are stored in cold storage, for the purpose of revocation." Like what? Why? Since when do you need a private key to revoke a certificate? I feel like I understand certificates really well, but .. am I missing something?
- dsfyu404ed 9y ago"We haven't finished getting to the bottom of this and don't have the post-mortem yet but this is very bad and you need to revoke all of these ASAP" "We're not revoking 50k certs without a damn good reason" "Attached is your reason"
- yjftsjthsd-h 9y agoYou gotta admit, it's a terribly compelling reason. And I do mean terribly.
- linkmotif 9y ago> When Rowley asked for proof the certificates were compromised, the Trustico CEO emailed the private keys of 23,000 certificates Voilà, compromised certs! Couldn’t tell if that was supposed to be the idea. But amusing nonetheless.
- Bartweiss 9y agoI can't tell if the logic was "we shouldn't have these keys so the certs are blown" or "watch, I just compromised the certs!" Either way, it's very much a case of "my neighborhood isn't safe - because I keep committing crimes there." edit: Oh, reading the Mozilla page it's also "we allowed code injection on the cert generation page". That's even worse.
- pilif 9y agoAlso, that script was running as root https://mobile.twitter.com/taviso/status/969230138956136448 https://mobile.twitter.com/taviso/status/969230138956136448
- Bartweiss 9y agoHah, seriously? ...I wonder whether the keys would have been more or less safe if, instead of all that, they just uploaded them unsecured to Pastebin. At least then they might have gone unnoticed, instead of being handed straight to possible bad actors.
- lima 9y agoThe mozilla.dev.security.policy thread linked in the article is a much better read: https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/wxX4Yv0E3Mk https://groups.google.com/forum/#!topic/mozilla.dev.security... tl;dr: Trustico told Digicert that all 50k of their certificates should be revoked, without specifying a reason as to why they were compromised. 50k is a LOT of certificates, so Digicert asked for details. Trustico then sent Digicert the private keys, and Digicert said "Welp, guess now they're compromised".
- lima 9y ago> Finally -- the CSR/key generation form page incorporates JavaScript from at least 5 or 6 different companies (including ad servers), which would allow any of those third parties (intentionally or through compromise of their own) to capture generated keys. This is a reckless amount of exposure, to the point that even if the keys were generated completely inside the browser and never exposed to the server (which does not appear to be the case), I would consider them compromised at the time of generation. Also, wow. That's extremely reckless.
- Bartweiss 9y agoJesus. That means it's not just about "offering revocation" and storing them on the wrong side; even people who didn't use that option are completely compromised.
- tialaramex 9y agoIf you generated your own Private Key and used that to make one or more CSRs, you can show any fool the CSR and no harm will come to you, at least on the cryptographic side. Upload it to 4chan, print it out on a T-shirt, doesn't matter. Any certificates produced are worthless to everybody except you, because you have the Private Key, and a _competent_ Certificate Authority (in this story that's Symantec or DigiCert, not Trustico) should only issue certificates for the names requested in the CSR, not any other names. A customer who did things sensibly (their own Private Key, make a CSR etcetera) and used Trustico say in 2016, needn't be concerned that these development will end with their certificates unilaterally revoked on Trustico's say-so. They might not want to do any more business with Trustico, but there's no particular reason this incident should affect their feelings about DigiCert or Comodo or any of the Symantec brands (Verisign / Thawte / RapidSSL / etcetera) now controlled by DigiCert.
- dancek 9y agoThis was very irresponsible from the CEO, but should not have been possible at all. The private key should be generated where it's used, a certificate signing request created and sent to a certificate authority. After the CA makes sure the CSR is legit, they can sign the certificate without seeing the private key. Copying the private key outside the server is an unnecessary risk. Giving it to a third party such as a CA or allowing them to generate it, moreso.
- sp332 9y agoSpeaking of things that shouldn't be possible: arbitrary code execution as root https://twitter.com/svblxyz/status/969220402768736258 https://twitter.com/svblxyz/status/969220402768736258 Looks like the whole site is now offline.
- SEJeff 9y agoHaving the private key on a server for a certificate authority in 2018 is an unnecessary risk. They should have had a hardware security module where the private key can not be extracted out. When you can buy one relatively cheaply even from yubico, there is simply no excuse to not using one as a CA. https://www.yubico.com/products/yubihsm/ https://www.yubico.com/products/yubihsm/ All of their certs should be revoked and they should go out of business for technical incompetence plain and simple.
- verytrivial 9y agoForgive my ignorance, but how does a device capable of storing something like 127 keys help a CA whom should have been storing zero client keys? Do you mean for their own keys?
- SEJeff 9y agoTL;DNR: I'm an idiot and misread the story thinking the executive sent the ca private key, not the client keys This is even more egregious. Thanks for the cluebat hit to the face!
- cjalmeida 9y ago
- DyslexicAtheist 9y agoOops Trustico server allows arbitrary RCE incl root shell: https://twitter.com/svblxyz/status/969220402768736258 https://twitter.com/svblxyz/status/969220402768736258 see also https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/BLvabFwcJqo https://groups.google.com/forum/#!topic/mozilla.dev.security...
- DyslexicAtheist 9y agodid they really generate the certificate pair server-side? pkiJS / WebCrypto² API are a keygen¹ replacement so that you can let the browser generate the certificate pair client-side. If the user isn't informed that key generation happens locally then maybe it looks as if the certificate is "downloaded". But with these methods when implemented correctly, certificates never goes to the server (and is saved locally). I'm not a fan of the above approach even though Mozilla sells it as safe: e.g. internal direct access to the OS RNG (instead of doing it in JavaScript Math-random() which led to a lot of critic throughout history of browser based crypto) , since the browser is still a sandbox, and IMO unsuitable for private key storage (even just for a few seconds). Would be interesting to know how trustico implemented this. It's somehow so crazy and negligent a thing to imagine. There is no explanation to why they didn't just implement it using standard software stacks for PKI (primekey has a jboss thing they could have used). Literally copy/pasting things from SO or some example code found in the manuals from WebCrypto would have been better than what they did. These guys have "trust" literally in their name. EDIT: Tippfehler __ ¹ https://github.com/w3ctag/wiki/wiki/keygen https://github.com/w3ctag/wiki/wiki/keygen ² https://developer.mozilla.org/en-US/docs/Web/API/Web_Crypto_API https://developer.mozilla.org/en-US/docs/Web/API/Web_Crypto_...
- nolok 9y ago> These guys have "trust" literally in their name. Generally speaking, this is more a red flag than an endorsement.
- SilasX 9y agoStupid question: I thought it was accepted practice [1] to generate the private key server-side so long as you don't store after the user's downloaded it? (Not that Trustico followed that part either.) [1] though not something I would endorse!
- emilfihlman 9y agoSeems to me that Digicert is tactically forgetting to say what they said to Trustico. Additionally there's not an issue with Trustico having access to the private keys. It simply means you are trusting Trustico and if you accept that it's no different from trusting Google or Facebook or your bank with your private data.
- bllguo 9y agoSo what's your response to this? > Finally -- the CSR/key generation form page incorporates JavaScript from at least 5 or 6 different companies (including ad servers), which would allow any of those third parties (intentionally or through compromise of their own) to capture generated keys. This is a reckless amount of exposure, to the point that even if the keys were generated completely inside the browser and never exposed to the server (which does not appear to be the case), I would consider them compromised at the time of generation.