47 ms·

Remediation Plan for WoSign and StartCom

81 points by asayler 10y ago

0x0 10y ago
So they are actually kicking out StartCom as well. Is this new? Apple was quick to move to kick out WoSign but they seemed to keep StartCom around. https://support.apple.com/en-us/HT204132 https://support.apple.com/en-us/HT204132
- Sanddancer 10y ago
  Mozilla's discussion started with startcom as well as wosign, as they share ownership and shared significant amounts of infrastructure.
- rawfan 10y ago
  I read about that before. StartCom is owned by WoSign now and there's evidency they completely moved to WoSigns infrastructure.
- codyro 10y ago
  I believe so - they're owned by the same company and it wasn't disclosed properly leading to some trust issues. Additionally there seems to be a lot of co-mingling between the companies in regards to code bases and signing practices. I'd check out https://wiki.mozilla.org/CA:WoSign_Issues https://wiki.mozilla.org/CA:WoSign_Issues and look for "StartCom" for examples.
  - 0x0 10y ago
    I remember the secret StartCom change of ownership came up very early in these discussions (I even saw random forum posts, on HN and elsewhere, almost a year earlier, when people noticed the StartCom servers mysteriously switched to Chinese IP addresses, and switched all my certs away as a precaution before there was any talk about CA mismanagement). But until now I've only seen talk of actually kicking out WoSign. Good riddance either way. Wonder what happened to the StartCom people, they seemed to be clued in back in the days. Shame.
    - pfg 10y ago
      The original plan[1] was to distrust both WoSign and StartCom after a certain date. Shortly after that, Mozilla met with representatives from Qihoo, WoSign and StartCom, and considered the possibility of treating StartCom separately under certain conditions[2]. The latest remediation plan seems to discard that notion (except that only WoSign will have to wait a year to re-apply). [1]: https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBGv6szuSB4sMYUcVrR8vQ/preview https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBG... [2]: https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/Mi9s9JWVdk4 https://groups.google.com/forum/#!topic/mozilla.dev.security...
- asayler 10y ago
  Yes, the Startcom roots are included in the set to be distrusted. Mozilla is allowing Startcom to re-apply sooner than Wosign, but both will have to go through the entire CA vetting process again, and Startcom will also have to prove it's no longer controlled by Wosign.
- Crosseye_Jack 10y ago
  WoSign brought StartCom but didn't tell anyone (against Mozilla's root cert policy) and insisted they were separate businesses when called out on it. Mozilla looked into it, found evidence that StartCom was now owned by WoSign and WoSign finally came clean they owned StartCom. The reasons StartCom is being distrusted too is because the WoSign code base (that a couple of parts are shared with StartCom including the issuance tech) has been found to be buggy so until qihoo 360 (WoSigns parent company) can prove that WoSign and StartCom are now 2 complete separate businesses as part of qihoo 360's plan to remove WoSigns CEO and separate the companies the loss of trust has to be applied to both. Oh and that loss of trust... WoSign's CEO (someone who has been in on CA/B forum meetings discussing the sun setting of SHA1 certs) authorised a backdated SHA1 cert to be issued for an AU payment processor and bypassing the legit method of applying for one (which he was also at the meetings that set up the SHA1 exception process) using StartCom's root while insisting the two were CA's were not linked. So Mozilla have said if qihoo 360 break up WoSign and StartCom (as qihoo 360 proposed), StartCom doesn't share WoSign's infrastructure after the break up and can prove this to the Mozilla community that StartCom and regain the trust of the Mozilla community they won't have to wait the min year to reapply. Apple were quick to kick WoSign but qihoo 360/StartCom had requested a meeting with Mozilla to discuss a mitigation plan (Relieve WoSign's CEO oh his duties, separate the two CA's, put in respected security people as CEO's in the two broken up CA's) to get back on the road to solving this fucking mess. Guess that looking at the evidence Mozilla released Apple's root team decided that WoSign had already lost their trust but wanted to hear out qihoo 360/StartCom before making a decision on StartCom too.
neom 10y ago
It's really great to see E&Y HK being held to account on this also. :thumbs_up:
- dredmorbius 10y ago
  My jaw hit the floor on that item. Trust matters.
- kchoudhu 10y ago
  Why just HK, though?
  - neom 10y ago
    From my reading: because it was only E&Y HK that was found to be delinquent in their obligations, and there isn't evidence that it's a systemic issue. IMHO: there should be a full audit of E&Y practises globally in order to continue to perform services pertaining to the certificate process.
AdmiralAsshat 10y ago
I'm glad that Mozilla is moving forward on this. There should be a zero tolerance policy for intentional deception on something as critical to the internet backbone as certificate authorities.
jasonjei 10y ago
Just curious about the root certificate distrust--are users capable of re-adding trust to distrusted certificates? Or is this hard coded into the browser? I'm assuming Mozilla stores certificates outside OS stores like Keychain and Windows?
- asayler 10y ago
  In general, locally added roots are trusted above all else -- and will even override cert pinning on most systems. Thus, if a user were to manually re-add the Wosign or Startcom roots to the local Mozilla trust store, they would continue to be trusted.
  - pfg 10y ago
    Sounds about right, but one thing to keep in mind is that "Removal of root" is only one possible route Mozilla can go for. They could also revoke (root or intermediate) certificate(s) through OneCRL, and while I haven't tried this, my guess would be that OneCRL trumps locally-added roots. That being said, the current plan is not to remove any of the roots (at least until all active certificates chaining up to those roots have expired), but rather not to trust certificates chaining to those roots with a notBefore date > October 21, 2016.
- 0x0 10y ago
  Yes but why would you? If you have control of all your clients to push such a change, why not just set up a private CA instead of opening yourself up to the whims of a proven cheating CA?
  - Sanddancer 10y ago
    Less difficult. Setting up a private CA means you have to be the CA, and vet and/or create a cert for every wosign/startcom site that people visit. One could trust them just enough depending on how heavily they depend on affected sites. I personally would rather whitelist sites as-needed, but can see why some admins would go the easier route.
scrollaway 10y ago
What did I miss? Last I heard about this, the plan was to ban them from issuing new certificates for a year. Did something change?
- InclinedPlane 10y ago
  They admitted they screwed up and the CEO stepped down.
  - xnyhps 10y ago
    "Will step down", according to his own post on https://groups.google.com/d/msg/mozilla.dev.security.policy/Mi9s9JWVdk4/zxERs26nGQAJ https://groups.google.com/d/msg/mozilla.dev.security.policy/....
- tptacek 10y ago
  That remains the plan.
  - avian 10y ago
    > 1) Distrust certificates chaining up to Affected Roots with a notBefore date after October 21, 2016. > ... > 4) Remove the Affected Roots from NSS after the SSL certificates issued before October 1, 2016, have expired or have been replaced. This sounds more serious than that. It says they can re-apply for inclusion of new roots next June though. So in practice it might really be just a one-year ban, if they will apply and pass the inclusion process.
    - SysArchitect 10y ago
      A 1 year ban is a long time for a company that sells certs. It might be the end of Wosign.
      - bigiain 10y ago
        Interestingly - this is a ban on their roots. How much do you want to bet they're already working out how to supply new and renewing customers with certs provided by some other CA? I notice the most recent StartSSL cert I got has a 3 year validity instead of their previous standard of 1 year - presumably in the hope that when my cert needs renewing they'll be able to provide that service. (I do have a handful of their certs which will expire during this 1 year ban. I'll certainly be needing to go elsewhere to renew them (finally time to learn how to auto-deploy LetEncrypt certs to Amazon ELB I guess, or maybe move all those domains to Route53 - I probably should have made time for that already...
jpablo 10y ago
I dont like this route. The only value for the WoSign keys for a whole year would be issuing certificates with a doctored notBefore date, and they can't do that publically.
- mintplant 10y ago
  It's an attempt to avoid instantly breaking all the sites across the web using WoSign/StartCom certificates. A year should give customers enough time to learn about the issue and switch providers. Meanwhile, WoSign can't sign up new customers or renew existing ones (at least, not if they want those new certs to work in Firefox).
- deleted 10y ago
  [deleted]
- michaelt 10y ago
  They could start reselling certs from a CA. That way their customers wouldn't have to sign up with a different CA. Then a year later, they could get everything back to normal.
guelo 10y ago
I wish I could replace my OS's certificate store with Mozilla's.
- mintplant 10y ago
  The full cert store data is here, if you'd like to try: https://hg.mozilla.org/mozilla-central/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt https://hg.mozilla.org/mozilla-central/raw-file/tip/security...
  - guelo 10y ago
    Assuming the notBefore date logic is going to be implemented in Mozilla's Network Security Services library the task would be to replace the OS's security library, which is probably not possible.
Diti 10y ago
I am glad this is happening. I have lost all trust in StartCom when they blatantly ignored the issues surrounding Heartbleed, refusing to renew certificates, despite every other CA doing so. I hope their learn their lesson, and try to be more honest in the future!
- theGimp 10y ago
  Small nitpick: StartCom refused to revoke certificates at the time, not renew them.
  - SysArchitect 10y ago
    They wouldn't let you renew them either unless you revoked first... Revocation cost $59 at the time. Was painful.
    - tajen 10y ago
      Certificates are very expensive with most providers, $59 is a bargain depending on your needs. The sole reason I've been staying with StartSSL is I've SSL'd all my subdomains (it's awesome for Postgres, for example), and a wildcard certificate costs $300 to $500 at all other shops. By the way, anyone knows a cheaper wildcard certificate provider?
      - anodari 10y ago
        Give a look at https://cheapsslsecurity.com/ https://cheapsslsecurity.com/
      - techdragon 10y ago
        AWS offers free certificates, including free wildcard ones. I've only used them with other AWS services so I'm not sure how easy it would be to use one of them outside the AWS ecosystem.
      - mydigitalself 10y ago
        I've used Gandi just about my entire life for DNS & Certs. It was probably their tagline that sold me. Anyhoo, they do wildcard for starting at 120,00 € excl. VAT/year. https://www.gandi.net/ https://www.gandi.net/
ldng 10y ago
Any advice if I just want to go ahead and kickout WoSign and StartCom ? Or do I have to wait on Mozilla ?
- gizmo686 10y ago
  In firefox: Edit > Preferences > Advanced > View Certificates Then navigate to the WoSign and StartCom certificates and distrust them.
merb 10y ago
i will probably get back to firefox for their nice cert policy.
ComodoHacker 10y ago
Now I'm waiting for Google's and Microsoft's reaction to the issue.
ComodoHacker 10y ago
The sibling thread[1], discussing StartCom and Qihoo, is also interesting, featuring people from both. 1. https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/TbDYE69YP8E https://groups.google.com/forum/#!topic/mozilla.dev.security...
rkapsoro 10y ago
Hey, where can I find more details on the EY Hong Kong audit of WoSign and Startcom? How did they fail?
SysArchitect 10y ago
What does it take to build a Certificate Transparency log? Is this something that we could allow someone like the EFF/Let's Encrypt run?
- pfg 10y ago
  Google open sourced their server implementation[1]. The problem that a number of CAs and other log operators have run into is that Google has rather strict requirements[2] for uptime and such (which makes sense!) if SCTs from that log server are to be accepted by Chrome, so you'd probably need a dedicated team capable of operating a HA cluster. [1]: https://github.com/google/certificate-transparency https://github.com/google/certificate-transparency [2]: https://www.chromium.org/Home/chromium-security/certificate-transparency/log-policy https://www.chromium.org/Home/chromium-security/certificate-...

81 points by asayler 10y ago

0x0 10y ago

So they are actually kicking out StartCom as well. Is this new? Apple was quick to move to kick out WoSign but they seemed to keep StartCom around. https://support.apple.com/en-us/HT204132 https://support.apple.com/en-us/HT204132

Sanddancer 10y ago
Mozilla's discussion started with startcom as well as wosign, as they share ownership and shared significant amounts of infrastructure.
rawfan 10y ago
I read about that before. StartCom is owned by WoSign now and there's evidency they completely moved to WoSigns infrastructure.
codyro 10y ago
I believe so - they're owned by the same company and it wasn't disclosed properly leading to some trust issues. Additionally there seems to be a lot of co-mingling between the companies in regards to code bases and signing practices. I'd check out https://wiki.mozilla.org/CA:WoSign_Issues https://wiki.mozilla.org/CA:WoSign_Issues and look for "StartCom" for examples.
- 0x0 10y ago
  I remember the secret StartCom change of ownership came up very early in these discussions (I even saw random forum posts, on HN and elsewhere, almost a year earlier, when people noticed the StartCom servers mysteriously switched to Chinese IP addresses, and switched all my certs away as a precaution before there was any talk about CA mismanagement). But until now I've only seen talk of actually kicking out WoSign. Good riddance either way. Wonder what happened to the StartCom people, they seemed to be clued in back in the days. Shame.
  - pfg 10y ago
    The original plan[1] was to distrust both WoSign and StartCom after a certain date. Shortly after that, Mozilla met with representatives from Qihoo, WoSign and StartCom, and considered the possibility of treating StartCom separately under certain conditions[2]. The latest remediation plan seems to discard that notion (except that only WoSign will have to wait a year to re-apply). [1]: https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBGv6szuSB4sMYUcVrR8vQ/preview https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBG... [2]: https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/Mi9s9JWVdk4 https://groups.google.com/forum/#!topic/mozilla.dev.security...
asayler 10y ago
Yes, the Startcom roots are included in the set to be distrusted. Mozilla is allowing Startcom to re-apply sooner than Wosign, but both will have to go through the entire CA vetting process again, and Startcom will also have to prove it's no longer controlled by Wosign.
Crosseye_Jack 10y ago
WoSign brought StartCom but didn't tell anyone (against Mozilla's root cert policy) and insisted they were separate businesses when called out on it. Mozilla looked into it, found evidence that StartCom was now owned by WoSign and WoSign finally came clean they owned StartCom. The reasons StartCom is being distrusted too is because the WoSign code base (that a couple of parts are shared with StartCom including the issuance tech) has been found to be buggy so until qihoo 360 (WoSigns parent company) can prove that WoSign and StartCom are now 2 complete separate businesses as part of qihoo 360's plan to remove WoSigns CEO and separate the companies the loss of trust has to be applied to both. Oh and that loss of trust... WoSign's CEO (someone who has been in on CA/B forum meetings discussing the sun setting of SHA1 certs) authorised a backdated SHA1 cert to be issued for an AU payment processor and bypassing the legit method of applying for one (which he was also at the meetings that set up the SHA1 exception process) using StartCom's root while insisting the two were CA's were not linked. So Mozilla have said if qihoo 360 break up WoSign and StartCom (as qihoo 360 proposed), StartCom doesn't share WoSign's infrastructure after the break up and can prove this to the Mozilla community that StartCom and regain the trust of the Mozilla community they won't have to wait the min year to reapply. Apple were quick to kick WoSign but qihoo 360/StartCom had requested a meeting with Mozilla to discuss a mitigation plan (Relieve WoSign's CEO oh his duties, separate the two CA's, put in respected security people as CEO's in the two broken up CA's) to get back on the road to solving this fucking mess. Guess that looking at the evidence Mozilla released Apple's root team decided that WoSign had already lost their trust but wanted to hear out qihoo 360/StartCom before making a decision on StartCom too.

neom 10y ago

It's really great to see E&Y HK being held to account on this also. :thumbs_up:

dredmorbius 10y ago
My jaw hit the floor on that item. Trust matters.
kchoudhu 10y ago
Why just HK, though?
- neom 10y ago
  From my reading: because it was only E&Y HK that was found to be delinquent in their obligations, and there isn't evidence that it's a systemic issue. IMHO: there should be a full audit of E&Y practises globally in order to continue to perform services pertaining to the certificate process.

AdmiralAsshat 10y ago

I'm glad that Mozilla is moving forward on this. There should be a zero tolerance policy for intentional deception on something as critical to the internet backbone as certificate authorities.

jasonjei 10y ago

Just curious about the root certificate distrust--are users capable of re-adding trust to distrusted certificates? Or is this hard coded into the browser? I'm assuming Mozilla stores certificates outside OS stores like Keychain and Windows?

asayler 10y ago
In general, locally added roots are trusted above all else -- and will even override cert pinning on most systems. Thus, if a user were to manually re-add the Wosign or Startcom roots to the local Mozilla trust store, they would continue to be trusted.
- pfg 10y ago
  Sounds about right, but one thing to keep in mind is that "Removal of root" is only one possible route Mozilla can go for. They could also revoke (root or intermediate) certificate(s) through OneCRL, and while I haven't tried this, my guess would be that OneCRL trumps locally-added roots. That being said, the current plan is not to remove any of the roots (at least until all active certificates chaining up to those roots have expired), but rather not to trust certificates chaining to those roots with a notBefore date > October 21, 2016.
0x0 10y ago
Yes but why would you? If you have control of all your clients to push such a change, why not just set up a private CA instead of opening yourself up to the whims of a proven cheating CA?
- Sanddancer 10y ago
  Less difficult. Setting up a private CA means you have to be the CA, and vet and/or create a cert for every wosign/startcom site that people visit. One could trust them just enough depending on how heavily they depend on affected sites. I personally would rather whitelist sites as-needed, but can see why some admins would go the easier route.

scrollaway 10y ago

What did I miss? Last I heard about this, the plan was to ban them from issuing new certificates for a year. Did something change?

InclinedPlane 10y ago
They admitted they screwed up and the CEO stepped down.
- xnyhps 10y ago
  "Will step down", according to his own post on https://groups.google.com/d/msg/mozilla.dev.security.policy/Mi9s9JWVdk4/zxERs26nGQAJ https://groups.google.com/d/msg/mozilla.dev.security.policy/....
tptacek 10y ago
That remains the plan.
- avian 10y ago
  > 1) Distrust certificates chaining up to Affected Roots with a notBefore date after October 21, 2016. > ... > 4) Remove the Affected Roots from NSS after the SSL certificates issued before October 1, 2016, have expired or have been replaced. This sounds more serious than that. It says they can re-apply for inclusion of new roots next June though. So in practice it might really be just a one-year ban, if they will apply and pass the inclusion process.
  - SysArchitect 10y ago
    A 1 year ban is a long time for a company that sells certs. It might be the end of Wosign.
    - bigiain 10y ago
      Interestingly - this is a ban on their roots. How much do you want to bet they're already working out how to supply new and renewing customers with certs provided by some other CA? I notice the most recent StartSSL cert I got has a 3 year validity instead of their previous standard of 1 year - presumably in the hope that when my cert needs renewing they'll be able to provide that service. (I do have a handful of their certs which will expire during this 1 year ban. I'll certainly be needing to go elsewhere to renew them (finally time to learn how to auto-deploy LetEncrypt certs to Amazon ELB I guess, or maybe move all those domains to Route53 - I probably should have made time for that already...

jpablo 10y ago

I dont like this route. The only value for the WoSign keys for a whole year would be issuing certificates with a doctored notBefore date, and they can't do that publically.

mintplant 10y ago
It's an attempt to avoid instantly breaking all the sites across the web using WoSign/StartCom certificates. A year should give customers enough time to learn about the issue and switch providers. Meanwhile, WoSign can't sign up new customers or renew existing ones (at least, not if they want those new certs to work in Firefox).
deleted 10y ago
[deleted]
michaelt 10y ago
They could start reselling certs from a CA. That way their customers wouldn't have to sign up with a different CA. Then a year later, they could get everything back to normal.

guelo 10y ago

I wish I could replace my OS's certificate store with Mozilla's.

mintplant 10y ago
The full cert store data is here, if you'd like to try: https://hg.mozilla.org/mozilla-central/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt https://hg.mozilla.org/mozilla-central/raw-file/tip/security...
- guelo 10y ago
  Assuming the notBefore date logic is going to be implemented in Mozilla's Network Security Services library the task would be to replace the OS's security library, which is probably not possible.

Diti 10y ago

I am glad this is happening. I have lost all trust in StartCom when they blatantly ignored the issues surrounding Heartbleed, refusing to renew certificates, despite every other CA doing so. I hope their learn their lesson, and try to be more honest in the future!

theGimp 10y ago
Small nitpick: StartCom refused to revoke certificates at the time, not renew them.
- SysArchitect 10y ago
  They wouldn't let you renew them either unless you revoked first... Revocation cost $59 at the time. Was painful.
  - tajen 10y ago
    Certificates are very expensive with most providers, $59 is a bargain depending on your needs. The sole reason I've been staying with StartSSL is I've SSL'd all my subdomains (it's awesome for Postgres, for example), and a wildcard certificate costs $300 to $500 at all other shops. By the way, anyone knows a cheaper wildcard certificate provider?
    - anodari 10y ago
      Give a look at https://cheapsslsecurity.com/ https://cheapsslsecurity.com/
    - techdragon 10y ago
      AWS offers free certificates, including free wildcard ones. I've only used them with other AWS services so I'm not sure how easy it would be to use one of them outside the AWS ecosystem.
    - mydigitalself 10y ago
      I've used Gandi just about my entire life for DNS & Certs. It was probably their tagline that sold me. Anyhoo, they do wildcard for starting at 120,00 € excl. VAT/year. https://www.gandi.net/ https://www.gandi.net/

ldng 10y ago

Any advice if I just want to go ahead and kickout WoSign and StartCom ? Or do I have to wait on Mozilla ?

gizmo686 10y ago
In firefox: Edit > Preferences > Advanced > View Certificates Then navigate to the WoSign and StartCom certificates and distrust them.

merb 10y ago

i will probably get back to firefox for their nice cert policy.

ComodoHacker 10y ago

Now I'm waiting for Google's and Microsoft's reaction to the issue.

ComodoHacker 10y ago

The sibling thread[1], discussing StartCom and Qihoo, is also interesting, featuring people from both. 1. https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/TbDYE69YP8E https://groups.google.com/forum/#!topic/mozilla.dev.security...

rkapsoro 10y ago

Hey, where can I find more details on the EY Hong Kong audit of WoSign and Startcom? How did they fail?

SysArchitect 10y ago

What does it take to build a Certificate Transparency log? Is this something that we could allow someone like the EFF/Let's Encrypt run?

pfg 10y ago
Google open sourced their server implementation[1]. The problem that a number of CAs and other log operators have run into is that Google has rather strict requirements[2] for uptime and such (which makes sense!) if SCTs from that log server are to be accepted by Chrome, so you'd probably need a dedicated team capable of operating a HA cluster. [1]: https://github.com/google/certificate-transparency https://github.com/google/certificate-transparency [2]: https://www.chromium.org/Home/chromium-security/certificate-transparency/log-policy https://www.chromium.org/Home/chromium-security/certificate-...