8 ms·

WoSign Incidents Report Update [pdf]

113 points by xnyhps 10y ago

cperciva 10y ago
Is the CEO stepping down because fraudulent certificates were signed, or is he stepping down because they got caught?
- mcbits 10y ago
  He is being "relieved of his duties as CEO" according to the report. Saying he's "stepping down" is IMHO like saying a fugitive "turned himself in" when police arrived at his house with an arrest warrant. (Although I don't think he is being accused of anything criminal here.) Most of the incidents appear to stem from technical errors and poorly implemented processes, but the whole list points to his inability to lead this kind of company effectively. I'm always skeptical of whether replacing a few executives can actually fix the cultural problems that were fostered/ignored in a company over time. The new leadership has to overcome a lot of inertia, and that's assuming they're any better than the old. They're also still answering to the same investors and pressures.
- hueving 10y ago
  The latter. Otherwise he would have stepped down weeks ago.
zamber 10y ago
Cross linking the earlier WoSign scandal reports: https://news.ycombinator.com/item?id=12389573 https://news.ycombinator.com/item?id=12389573 https://news.ycombinator.com/item?id=12582534 https://news.ycombinator.com/item?id=12582534 https://news.ycombinator.com/item?id=12617659 https://news.ycombinator.com/item?id=12617659 So if I get it right, WoSign will cease to exist as a CA given the 1y probation proposed by Mozilla and the general distrust that will follow?
- inimino 10y ago
  That was Mozilla's action plan, this is Wosign's response, now it is back in Mozilla's court.
gogopuppygogo 10y ago
Mirror of the PDF version of the incident report if they take down the original or it goes down: http://clicky.strapr.com/3g1s1y1k0Y2J http://clicky.strapr.com/3g1s1y1k0Y2J
dredmorbius 10y ago
This story is a bit of a mess to make sense of coming in cold and reading a Google Groups summary. Here's my read, which may help clarify the story for others. Mozilla have an excellent explanation document covering the backdated certs in detail here: https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBGv6szuSB4sMYUcVrR8vQ/preview https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBG... (Thanks to @xnyhps for the link in a reply to this comment.) WoSign, described elsewhere as China's largest certificates authority, are a CA who have been found to have backdated SHA1 ceritificates to work around browser restrictions on SHA1 cert issueances. SHA1 is no longer considered secure. Resolution of that issue is discussed in new mozilla.dev.security.policy Usenet group peered by Google Groups: https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/1XI3Y7PJ1Uc/qc9PvezXFwAJ https://groups.google.com/forum/#!msg/mozilla.dev.security.p... A better source for WoSign's update to the story is in the PDF posted to the newsgroup, here: https://www.wosign.com/report/WoSign_Incident_Report_Update_07102016.pdf https://www.wosign.com/report/WoSign_Incident_Report_Update_... Titled "WoSign Incidents Report Update". Which is even less descriptive than the title presently given on this HN post, though perhaps what HN posting guidelines prefer. I'll let @dang wrestle his conscience on that one. In that document are several issues listed, the one relevant to this HN post appears to be: "9. Issue S: Backdated SHA-1 Certs (January 2016) "WoSign has issued certificates after January 1st 2016 but backdated the notBefore date to be in December 2015. This has the effect of avoiding the blocks in browsers regarding SHA-1 certs issued after January 1st 2016. The number of certs affected is probably 67, but may be a few more or less." Following down from there, several corporate restructuring steps are mentioned, including: 360’s Corporate Development team has been notified to execute the process to legally separate Wosign and Startcom and to begin executing personnel reassignments. StartCom’s chairman will be Xiaosheng Tan (Chief Security Officer of Qihoo 360). StartCom’s CEO will be Inigo Barreira (formerly GM of StartCom Europe). Richard Wang will be relieved of his duties as CEO of WoSign. There is background on the story from: "WoSign Mis-Issued SHA-1 SSL Certificates [Updated]" (August 24, 2016) https://www.thesslstore.com/blog/wosign-mis-issued-sha-1-ssl/ https://www.thesslstore.com/blog/wosign-mis-issued-sha-1-ssl... "Mozilla Ready to Ban WoSign Certificates for One Year After Shady Behavior" (September 26, 2016) The second article details Mozilla's issues with WoSign, including purchase of an Israeli CA, StartCom http://news.softpedia.com/news/mozilla-ready-to-ban-wosign-certificates-for-one-year-after-shady-behavior-508674.shtml http://news.softpedia.com/news/mozilla-ready-to-ban-wosign-c... I'm not claiming anything other than a 15 minute familiarity with the situation here. I may have heard earlier rumblings but really haven't followed this at all and wasn't consciously aware of particulars.
ctz 10y ago
> "Many customers in China find it important to use a domestic CA for purposes of security." That's not how the CA system works. Your security is unaffected by what CA you choose; it is invariably the minimum of all trusted CAs.
- hueving 10y ago
  It certainly is if you strip down the certs trusted.
  - ctz 10y ago
    Well, true. But that would mainly affect availability :)
- deleted 10y ago
  [deleted]
- angry_octet 10y ago
  Which is a good reason to trust fewer CAs, especially those from countries which have poor transparency.
  - pfg 10y ago
    I honestly don't think the location should be a factor. How many countries are there that don't have things like national security letters? I'd rather have a system where the same criteria apply to all CAs independent of their jurisdiction, coupled with mechanisms that guarantee transparency (like Certificate Transparency) and stuff like key pinning.
    - hedora 10y ago
      I don't think the general public realizes how important this is. I only recently found out that anyone that can reliably man in the middle your server can get a valid HTTPS cert from CA's like Let's Encrypt. That includes cloud providers, backbone providers and possibly the local government where the server is located. [edit: It also includes anyone that can temporarily modify or spoof your DNS records.]
ajdlinux 10y ago
Direct link to PDF report: https://www.wosign.com/report/WoSign_Incident_Report_Update_07102016.pdf https://www.wosign.com/report/WoSign_Incident_Report_Update_...
tarancato 10y ago
So much money they are making and they can't hire anybody who can write English properly or write a PDF that is not composed of a handful of font faces and sizes. I also find it funny they have fired the CEO (the PDF does not say he stepped down voluntarily) but he's the one sending that link to the mailing list. I call bs.
- johnnyfaehell 10y ago
  > So much money they are making and they can't hire anybody who can write English properly If you're going to make fun of someone's English then you should probably write better English yourself. At the end of the day English isn't their first language and they'll probably very rarely have to use English. I think some slack should be given on this front.
  - tarancato 10y ago
    I don't speak English natively, and any corrections are welcome. On the other hand I don't run a multi-million business based on writing comments on Hacker News so currently hiring a PR person to proofread these comments is not within the boundaries of my budget. WoSign runs a business that is basically based on trust, and Nigerian scammers have sent me PDFs that looked far more convincing and trustable than that one WoSign posted.
  - Normal_gaussian 10y ago
    tarancato isn't making any money from these comments, so it seems reasonable that they are not using some form of proofreader.
    - johnnyfaehell 10y ago
      I am not bothered by their English or using a proof reader. I just think it's hypocritical to throw stones at people for not having good English when they don't operate in an English speaking market when yours isn't perfect either. Also do you honestly think that was written by the business team, do you think the business team would have been able to properly edit a technical document in another language?
nikcub 10y ago
Interesting aside that came out of this case is the issue of cross-signing intermediary and root certs and it not being disclosed. The WoSign roots were cross-signed by[0] Comodo and StartCom (owned by WoSign, but we didn't know that), so even with WoSign roots being revoked, there would still be a verification path. Nice to see that now there is an effort to disclose all of these[2][3], and[1]: > Mozilla now requires the disclosue of all intermedidate certificates, including those cross-certificates. [0] https://wiki.mozilla.org/CA:WoSign_Issues#Cross_Signing https://wiki.mozilla.org/CA:WoSign_Issues#Cross_Signing [1] https://groups.google.com/d/msg/mozilla.dev.security.policy/k9PBmyLCi8I/5c1c6L7JFwAJ https://groups.google.com/d/msg/mozilla.dev.security.policy/... [2] https://crt.sh/mozilla-disclosures https://crt.sh/mozilla-disclosures [3] https://secure.comodo.com/products/publiclyDisclosedSubCACerts https://secure.comodo.com/products/publiclyDisclosedSubCACer...
terom 10y ago
Awesome use of load balancing for request retry across multiple backend servers: > This was caused by the CMS (Certificate Management System), when it sent the signing request of the certificate to the signing server A, which had no response, then the CMS sent it to the other newly added signing server B. After a while the signing server A signed the certificate and sent to the CMS and also to the subscriber, then the subscriber installed the cert in its website and hat's why Censys recorded this certificate; in the meantime, the signing server B also signed this certificate some time later (in seconds) and sent it to the CMS, the CMS accepted it and rewrote it in the DB. > This issue happened after adding another signing server on Jan 5th 2015, and found it on April 9th 2015. When had the two signing servers added a load balancer, but the configuration was not properly done because it didn't lock the request. Mind you, that's a perfectly legit technical bug. Maybe they were using nginx for load balancing POST requests? :) https://news.ycombinator.com/item?id=11217477 https://news.ycombinator.com/item?id=11217477
- therein 10y ago
  Yup, retrying PUT/POST is a bad idea unless you're really careful.
lifeisstillgood 10y ago
So I am wondering how this mess got started Is it just incredible incompetence on WoSign's part or is there a stronger reason why China's largest CA is trying to keep issuing weak certificates? Is is tinfoil-hattery to assume that the Chinese government is not ready for SSL to start getting unreadable again?
- im3w1l 10y ago
  Compatibility with legacy software that only understands the older certs is my guess.
- nikic 10y ago
  > The charged 42 backdated certificates are an intentional activity that we try to help the desperate customers since there are more than 3M users still using Windows XP sp2 in China. We like to make things simple that don’t realize how serious this solution was. I think this is all there really is to it. Some customers wanted to have backdated SHA1 certificates so they could continue supporting platforms that do not accept SHA256 certificates. WoSign decided to oblige them.
koolba 10y ago
When you have a business that effectively prints money, why do something this stupid? I also wonder how much more effort they thought it was to write the code to backdate the certs (rather than use "now") v.s. code for upgrading to SHA-256.
- 0x0 10y ago
  It's not about a lack of support for SHA-256 on the CA side. But backdated SHA-1 certs are in demand because they allow other systems to keep working, systems like embedded/PoS computers that lack support for SHA-256, while simultaneously dodging errors on modern browsers that only allow SHA-1 certs if they were issued before a certain date. Wosign was certainly capable of issuing SHA-256 certificates. But the customers needed SHA-1 certificates with a backdated issue timestamp. And Wosign was willing to fake the issue timestamp on new certificates, probably a lucrative market because no other reputable CA would be willing to do so.
  - koolba 10y ago
    > Wosign was certainly capable of issuing SHA-256 certificates. But the customers needed SHA-1 certificates with a backdated issue timestamp. So they had a special side deal to get backdated certs? There's no way they were doing this for everybody or for "regular price" right?
    - dogma1138 10y ago
      Doesn't matter they violated the chain of trust, the fact that they took extra money to do this doesn't make it any more or less of an offense. EDIT: Was reading out of date info :) ~~However like some one else pointed out out of the discussion in the group looks like it was a technical bug, which is still bad, but at least not maliciously bad.~~ Gah HN why you still no do markdown....
      - 0x0 10y ago
        It was definitively done on purpose based on customer requests. Look at the linked PDF, section 9.2.
nly 10y ago
A shame. WoSign were super generous with their free certificate offering long before LetsEncrypt was a thing. They were a a handy alternative. We should be thanking them for their free certs, and thanking them again now for giving us another example of how the PKI is a farce. The chances are there are a bunch other 'reputable' CAs out there playing these games.
- icebraining 10y ago
  But that's why browsers are increasingly requiring CAs to implement https://en.wikipedia.org/wiki/Certificate_Transparency https://en.wikipedia.org/wiki/Certificate_Transparency
- bigiain 10y ago
  Interestingly - StartCom's free certs used to only have 1yr validity - one I got last week is good for 3 years. Cynical-me suspects they're doing that to improve their chances of sitting out a limited 1yr timeout. It'd be almost indistinguishable from a death penalty if all their existing client certs expired while they were unable to renew them - I wonder how long they've been signing certs for 3yrs instead of 1?

113 points by xnyhps 10y ago

cperciva 10y ago

Is the CEO stepping down because fraudulent certificates were signed, or is he stepping down because they got caught?

mcbits 10y ago
He is being "relieved of his duties as CEO" according to the report. Saying he's "stepping down" is IMHO like saying a fugitive "turned himself in" when police arrived at his house with an arrest warrant. (Although I don't think he is being accused of anything criminal here.) Most of the incidents appear to stem from technical errors and poorly implemented processes, but the whole list points to his inability to lead this kind of company effectively. I'm always skeptical of whether replacing a few executives can actually fix the cultural problems that were fostered/ignored in a company over time. The new leadership has to overcome a lot of inertia, and that's assuming they're any better than the old. They're also still answering to the same investors and pressures.
hueving 10y ago
The latter. Otherwise he would have stepped down weeks ago.

zamber 10y ago

Cross linking the earlier WoSign scandal reports: https://news.ycombinator.com/item?id=12389573 https://news.ycombinator.com/item?id=12389573 https://news.ycombinator.com/item?id=12582534 https://news.ycombinator.com/item?id=12582534 https://news.ycombinator.com/item?id=12617659 https://news.ycombinator.com/item?id=12617659 So if I get it right, WoSign will cease to exist as a CA given the 1y probation proposed by Mozilla and the general distrust that will follow?

inimino 10y ago
That was Mozilla's action plan, this is Wosign's response, now it is back in Mozilla's court.

gogopuppygogo 10y ago

Mirror of the PDF version of the incident report if they take down the original or it goes down: http://clicky.strapr.com/3g1s1y1k0Y2J http://clicky.strapr.com/3g1s1y1k0Y2J

dredmorbius 10y ago

This story is a bit of a mess to make sense of coming in cold and reading a Google Groups summary. Here's my read, which may help clarify the story for others. Mozilla have an excellent explanation document covering the backdated certs in detail here: https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBGv6szuSB4sMYUcVrR8vQ/preview https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBG... (Thanks to @xnyhps for the link in a reply to this comment.) WoSign, described elsewhere as China's largest certificates authority, are a CA who have been found to have backdated SHA1 ceritificates to work around browser restrictions on SHA1 cert issueances. SHA1 is no longer considered secure. Resolution of that issue is discussed in new mozilla.dev.security.policy Usenet group peered by Google Groups: https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/1XI3Y7PJ1Uc/qc9PvezXFwAJ https://groups.google.com/forum/#!msg/mozilla.dev.security.p... A better source for WoSign's update to the story is in the PDF posted to the newsgroup, here: https://www.wosign.com/report/WoSign_Incident_Report_Update_07102016.pdf https://www.wosign.com/report/WoSign_Incident_Report_Update_... Titled "WoSign Incidents Report Update". Which is even less descriptive than the title presently given on this HN post, though perhaps what HN posting guidelines prefer. I'll let @dang wrestle his conscience on that one. In that document are several issues listed, the one relevant to this HN post appears to be: "9. Issue S: Backdated SHA-1 Certs (January 2016) "WoSign has issued certificates after January 1st 2016 but backdated the notBefore date to be in December 2015. This has the effect of avoiding the blocks in browsers regarding SHA-1 certs issued after January 1st 2016. The number of certs affected is probably 67, but may be a few more or less." Following down from there, several corporate restructuring steps are mentioned, including: 360’s Corporate Development team has been notified to execute the process to legally separate Wosign and Startcom and to begin executing personnel reassignments. StartCom’s chairman will be Xiaosheng Tan (Chief Security Officer of Qihoo 360). StartCom’s CEO will be Inigo Barreira (formerly GM of StartCom Europe). Richard Wang will be relieved of his duties as CEO of WoSign. There is background on the story from: "WoSign Mis-Issued SHA-1 SSL Certificates [Updated]" (August 24, 2016) https://www.thesslstore.com/blog/wosign-mis-issued-sha-1-ssl/ https://www.thesslstore.com/blog/wosign-mis-issued-sha-1-ssl... "Mozilla Ready to Ban WoSign Certificates for One Year After Shady Behavior" (September 26, 2016) The second article details Mozilla's issues with WoSign, including purchase of an Israeli CA, StartCom http://news.softpedia.com/news/mozilla-ready-to-ban-wosign-certificates-for-one-year-after-shady-behavior-508674.shtml http://news.softpedia.com/news/mozilla-ready-to-ban-wosign-c... I'm not claiming anything other than a 15 minute familiarity with the situation here. I may have heard earlier rumblings but really haven't followed this at all and wasn't consciously aware of particulars.

ctz 10y ago

> "Many customers in China find it important to use a domestic CA for purposes of security." That's not how the CA system works. Your security is unaffected by what CA you choose; it is invariably the minimum of all trusted CAs.

hueving 10y ago
It certainly is if you strip down the certs trusted.
- ctz 10y ago
  Well, true. But that would mainly affect availability :)
deleted 10y ago
[deleted]
angry_octet 10y ago
Which is a good reason to trust fewer CAs, especially those from countries which have poor transparency.
- pfg 10y ago
  I honestly don't think the location should be a factor. How many countries are there that don't have things like national security letters? I'd rather have a system where the same criteria apply to all CAs independent of their jurisdiction, coupled with mechanisms that guarantee transparency (like Certificate Transparency) and stuff like key pinning.
  - hedora 10y ago
    I don't think the general public realizes how important this is. I only recently found out that anyone that can reliably man in the middle your server can get a valid HTTPS cert from CA's like Let's Encrypt. That includes cloud providers, backbone providers and possibly the local government where the server is located. [edit: It also includes anyone that can temporarily modify or spoof your DNS records.]

ajdlinux 10y ago

Direct link to PDF report: https://www.wosign.com/report/WoSign_Incident_Report_Update_07102016.pdf https://www.wosign.com/report/WoSign_Incident_Report_Update_...

tarancato 10y ago

So much money they are making and they can't hire anybody who can write English properly or write a PDF that is not composed of a handful of font faces and sizes. I also find it funny they have fired the CEO (the PDF does not say he stepped down voluntarily) but he's the one sending that link to the mailing list. I call bs.

johnnyfaehell 10y ago
> So much money they are making and they can't hire anybody who can write English properly If you're going to make fun of someone's English then you should probably write better English yourself. At the end of the day English isn't their first language and they'll probably very rarely have to use English. I think some slack should be given on this front.
- tarancato 10y ago
  I don't speak English natively, and any corrections are welcome. On the other hand I don't run a multi-million business based on writing comments on Hacker News so currently hiring a PR person to proofread these comments is not within the boundaries of my budget. WoSign runs a business that is basically based on trust, and Nigerian scammers have sent me PDFs that looked far more convincing and trustable than that one WoSign posted.
- Normal_gaussian 10y ago
  tarancato isn't making any money from these comments, so it seems reasonable that they are not using some form of proofreader.
  - johnnyfaehell 10y ago
    I am not bothered by their English or using a proof reader. I just think it's hypocritical to throw stones at people for not having good English when they don't operate in an English speaking market when yours isn't perfect either. Also do you honestly think that was written by the business team, do you think the business team would have been able to properly edit a technical document in another language?

nikcub 10y ago

Interesting aside that came out of this case is the issue of cross-signing intermediary and root certs and it not being disclosed. The WoSign roots were cross-signed by[0] Comodo and StartCom (owned by WoSign, but we didn't know that), so even with WoSign roots being revoked, there would still be a verification path. Nice to see that now there is an effort to disclose all of these[2][3], and[1]: > Mozilla now requires the disclosue of all intermedidate certificates, including those cross-certificates. [0] https://wiki.mozilla.org/CA:WoSign_Issues#Cross_Signing https://wiki.mozilla.org/CA:WoSign_Issues#Cross_Signing [1] https://groups.google.com/d/msg/mozilla.dev.security.policy/k9PBmyLCi8I/5c1c6L7JFwAJ https://groups.google.com/d/msg/mozilla.dev.security.policy/... [2] https://crt.sh/mozilla-disclosures https://crt.sh/mozilla-disclosures [3] https://secure.comodo.com/products/publiclyDisclosedSubCACerts https://secure.comodo.com/products/publiclyDisclosedSubCACer...

terom 10y ago

Awesome use of load balancing for request retry across multiple backend servers: > This was caused by the CMS (Certificate Management System), when it sent the signing request of the certificate to the signing server A, which had no response, then the CMS sent it to the other newly added signing server B. After a while the signing server A signed the certificate and sent to the CMS and also to the subscriber, then the subscriber installed the cert in its website and hat's why Censys recorded this certificate; in the meantime, the signing server B also signed this certificate some time later (in seconds) and sent it to the CMS, the CMS accepted it and rewrote it in the DB. > This issue happened after adding another signing server on Jan 5th 2015, and found it on April 9th 2015. When had the two signing servers added a load balancer, but the configuration was not properly done because it didn't lock the request. Mind you, that's a perfectly legit technical bug. Maybe they were using nginx for load balancing POST requests? :) https://news.ycombinator.com/item?id=11217477 https://news.ycombinator.com/item?id=11217477

therein 10y ago
Yup, retrying PUT/POST is a bad idea unless you're really careful.

lifeisstillgood 10y ago

So I am wondering how this mess got started Is it just incredible incompetence on WoSign's part or is there a stronger reason why China's largest CA is trying to keep issuing weak certificates? Is is tinfoil-hattery to assume that the Chinese government is not ready for SSL to start getting unreadable again?

im3w1l 10y ago
Compatibility with legacy software that only understands the older certs is my guess.
nikic 10y ago
> The charged 42 backdated certificates are an intentional activity that we try to help the desperate customers since there are more than 3M users still using Windows XP sp2 in China. We like to make things simple that don’t realize how serious this solution was. I think this is all there really is to it. Some customers wanted to have backdated SHA1 certificates so they could continue supporting platforms that do not accept SHA256 certificates. WoSign decided to oblige them.

koolba 10y ago

When you have a business that effectively prints money, why do something this stupid? I also wonder how much more effort they thought it was to write the code to backdate the certs (rather than use "now") v.s. code for upgrading to SHA-256.

0x0 10y ago
It's not about a lack of support for SHA-256 on the CA side. But backdated SHA-1 certs are in demand because they allow other systems to keep working, systems like embedded/PoS computers that lack support for SHA-256, while simultaneously dodging errors on modern browsers that only allow SHA-1 certs if they were issued before a certain date. Wosign was certainly capable of issuing SHA-256 certificates. But the customers needed SHA-1 certificates with a backdated issue timestamp. And Wosign was willing to fake the issue timestamp on new certificates, probably a lucrative market because no other reputable CA would be willing to do so.
- koolba 10y ago
  > Wosign was certainly capable of issuing SHA-256 certificates. But the customers needed SHA-1 certificates with a backdated issue timestamp. So they had a special side deal to get backdated certs? There's no way they were doing this for everybody or for "regular price" right?
  - dogma1138 10y ago
    Doesn't matter they violated the chain of trust, the fact that they took extra money to do this doesn't make it any more or less of an offense. EDIT: Was reading out of date info :) ~~However like some one else pointed out out of the discussion in the group looks like it was a technical bug, which is still bad, but at least not maliciously bad.~~ Gah HN why you still no do markdown....
    - 0x0 10y ago
      It was definitively done on purpose based on customer requests. Look at the linked PDF, section 9.2.

nly 10y ago

A shame. WoSign were super generous with their free certificate offering long before LetsEncrypt was a thing. They were a a handy alternative. We should be thanking them for their free certs, and thanking them again now for giving us another example of how the PKI is a farce. The chances are there are a bunch other 'reputable' CAs out there playing these games.

icebraining 10y ago
But that's why browsers are increasingly requiring CAs to implement https://en.wikipedia.org/wiki/Certificate_Transparency https://en.wikipedia.org/wiki/Certificate_Transparency
bigiain 10y ago
Interestingly - StartCom's free certs used to only have 1yr validity - one I got last week is good for 3 years. Cynical-me suspects they're doing that to improve their chances of sitting out a limited 1yr timeout. It'd be almost indistinguishable from a death penalty if all their existing client certs expired while they were unable to renew them - I wonder how long they've been signing certs for 3yrs instead of 1?