11 ms·

Symantec CA Response to Google Proposal and Community Feedback

125 points by mentat 9y ago

Aaron1011 9y ago
> This cohort is an important constituency that we believe has been under-represented to date in the public commentary that has been posted to the Google and Mozilla boards since large organizations rarely authorize employees to engage in such public discussions, particularly in an area related to security. Are these large organizations somehow incapable of putting out official statements regarding CAs? If they're being 'under-represented', it's their own fault for not speaking up.
- mdip 9y ago
  To me it was really bold of them to take the tone and stance that they did with regard to these large organizations. They're making the case as to why Google needed to do what it did ... all of these large organizations that will be severely impacted by having to replace all of these certificates are relying on a CA that has shown itself to not be worthy of the trust that CAs are relied upon to provide. I'm at a loss for a good analogy so I'll pick a lousy one: it'd be like if Samsung's response to the exploding Note 7 recall mandate were "Don't do this -- you're going to keep some of the biggest customers -- users of our phones -- from being able to make emergency calls/check their e-mail/respond to texts" ...yeah, but I'm still not OK with the small possibility that carrying my phone in my front pocket might result in third-degree genital burns.
  - skuhn 9y ago
    As it happens, your hypothetical situation was exactly what did transpire: Samsung wanted to remotely deactivate the phones, but Verizon refused on those grounds. http://bgr.com/2016/12/09/galaxy-note-7-recall-verizon-samsung/ http://bgr.com/2016/12/09/galaxy-note-7-recall-verizon-samsu...
    - mdip 9y ago
      ... and here I thought comparing security to burning genitals might have been too much of a farce, but that takes the cake, there. :o)
tyingq 9y ago
I don't think Google was soliciting for a counter proposal from Symantec. Will be interesting to see their reply, and whether it's a literal reply or just a version push of chrome with their original plan.[1] [1] https://groups.google.com/a/chromium.org/forum/m/#!topic/blink-dev/eUAKwjihhBs https://groups.google.com/a/chromium.org/forum/m/#!topic/bli... Edit: They did ask for community feedback, comments on risk, etc. But they do already have a timeline. See link above.
- toomuchtodo 9y ago
  "This site does not support Chrome. Please use a browser that does not take unilateral CA authority action." might very well be the response of orgs married to Symantec. As a user, you need your bank (or other large org) more than you need your preference of browser.
  - tyingq 9y ago
    Bold move though, when the fix is just buying a new cert. And the browser you're blocking has majority market share.
    - delinka 9y ago
      "Buying" as if Let's Encrypt isn't a thing. ;-)
      - walrus01 9y ago
        well, for any organization that is doing serious ecommerce these days, you would want an EV SSL cert. Which is about $95/year from namecheap or a competitor. The GUI advantages for ordinary totally ignorant end users of seeing the happy green bar at the top are enough to justify the less than $10 per month ongoing cost.
      - tyingq 9y ago
        In this context, I assume we're talking about EV certs. Though LE might be an emergency temp option for the big customers that forgot to plan around this.
revelation 9y ago
I love how none of the example "dependencies" they give should be using public CA in the first place.
- ahmeni 9y ago
  Given some of the internal CA systems I've dealt within the past, I'd almost prefer a public CA in some cases. Sometimes your internal CA is just the group with manual access to the certificate provisioning and signing systems with either no API or some awful re-implemented API.
  - revelation 9y ago
    What API do you need? The signing system should be airgapped or you end up with the same shit that is the public CA system such as roots sitting on public FTP servers. It's a bunch of command line scripts because if you are using it any different way you are probably doing it wrong.
    - Spooky23 9y ago
      Seriously. The fancy CA systems are wacky expensive.
      - revelation 9y ago
        I'm sure people make plenty of money wrapping OpenSSL in ever more elaborate Enterprise schemes. But if the net effect is that one bug in a frontend can cause the server to tell the cert server to tell the secure enclave to tell the wire mesh protected crypto coprocessor to sign a bogus certificate then what are we really doing here.
        Spooky23 9y ago
        They sure do! We have an offline root on an HSM for various reasons. We looked at various systems to manage intermediate ca issuance and reminders for renewal and the cost was incredible... most were >$200k! Using scripts to automatically generate certificates and using the ugly but effective Windows CA web interface for "vanity" stuff does 90% of what those solutions do. I think we had an intern write a monitoring plugin to open tickets for soon to expire certificates and a few other things that covered the remaining 10%.
MichaelGG 9y ago
>require these applications to be recoded, recompiled and redistributed. Aka "updated". The entire post is basically "ok how about we be really good from now on and suffer no consequences, cause it'd be really shitty for us if we had to be penalised". They also posture a lot talking about how big their customers are, almost boasting about how inflexible and slow these big companies are, as if that's somehow Google's or general Internet users' problem.
- deleted 9y ago
  [deleted]
- dude01 9y ago
  Yeah, they're using the argument: we're too big to fail.
  - walrus01 9y ago
    there are a number of similarities between symantec and some wall street dinosaurs.
- snomad 9y ago
  Isn't that largely Google's stance with regard to Android? They blame the carriers and point to how slow they are, meanwhile tons of smart phones go without patching, which is no good for the general internet or users'
  - AceJohnny2 9y ago
    No, there's a huge difference. The difference here is that the Manufacturers and the Carriers are gatekeepers to the customer's phones. The manufacturer made customizations to Android for their phone. Android isn't a cleanly-separated stack where the manufacturer modifies only a certain part of the stack for their phone but Google could go and update another part without risk of breaking anything (is any software project?) Then, the manufacturers sell the phones to carriers, who usually request extra customizations to (at best) add their logo, or replace default apps with the carrier's own version. This means that any updates to Android likely require updates to this customization. But even if that customization was free (from a development perspective), Carriers still have maintenance/service contracts with the manufacturers such that any manufacturer's updates must go through the carrier's "quality control" procedure. You wouldn't want an update to make the, say, LTE radio misbehave and break the carrier's network would you? Oh no... (and you certainly wouldn't want to "forget" to replace the good default app with the Carrier's crappy one) So there you go, a quick glimpse into the complex economic/contractual obligations that delays or blocks Android updates. FWIW, Google has been working both technically and contractually to untangle this mess, and is one reason for the existence of their "own" line of Nexus and now Pixel phones.
ahmeni 9y ago
There is a significant amount of "we will" language in here, rather than "we have started" or "have already begun". The choice of language speaks volumes that they only appear to be willing to take necessary audit action if Google agrees to it, rather than something that needs to be done regardless.
geofft 9y ago
> These customers include many of the largest financial services, critical infrastructure, retail and healthcare organizations in the world, as well as many government agencies. This cohort is an important constituency that we believe has been under-represented to date in the public commentary that has been posted to the Google and Mozilla boards since large organizations rarely authorize employees to engage in such public discussions, particularly in an area related to security. ... well that's their problem, right? You can't simultaneously say "These are some of the most important organizations in the world and you'll cause worldwide chaos" and "Won't someone listen to these poor companies, I am the Symantorax, I speak for the cohort, for the cohort has no tongues."
- mdip 9y ago
  > These are some of the most important organizations in the world and you'll cause worldwide chaos Thanks for that -- reading this comment connected a few thoughts I was struggling with. Their response felt like "if you don't follow through with your plan to punish us we'll do all of these nice things for you." It reminds me of when my kids get caught doing something wrong and I get "but I'll do all of my chores and treat my siblings nice" along with a handful of other promises of future deeds which I know will be partially or completely not-done. How different the result would have been if they came to me with the chores done and some/all of the promises fulfilled already. It would have been much better for Symantec to have responded with these items in the "completed" category with a smaller number of "we will's". But to the quote I highlighted: the issue at hand is the past failures indicate that "some of the most important organizations in the world" need to not be trusting Symantec in the first place. They've had a big enough history of failures as a CA to get the attention of Google, who I'm sure isn't terribly interested in degrading the browsing experience of their customers. The choice becomes: (1) Banhammer Symantec, take the resulting pain all at once, visibly and obviously, and get the problem fixed painfully (but it's fixed) or (2) Trust that things will be different this time and if they're not, have another security failure of some kind that results in these critically important company's customers' data being exposed, which is a circumstance that will almost certainly go unnoticed for a period of time, will be covered up if the exposure is minimal and covering up the exposure is possible or may not be discovered at all. The CA system is centered around trusting a handful of rather powerful bodies with an enormous responsibility around the overall security of the Internet. It's not a great model to begin with, but it's what we have and I haven't read about any alternatives that are viable. The only way to keep a model like this secure is to ensure that the CAs themselves are actually trustworthy and one of the more likely ways that will happen is to enforce violations of that trust in the manner Google is suggesting.
bitmapbrother 9y ago
Symantec is just going through the 5 stages of CA grief. First they were oblivious to it, then they were angry about it and called Google irresponsible. They're now at the proposal stage. Next will be depression and finally acceptance for their incompetence.
durkie 9y ago
> Embedded devices that are pinned to certificates issued by a Symantec public root to communicate to resources over the Internet or Intranet. Replacing these certificates would result in immediate failures and the need to recode and reimage the firmware for these devices. > Mobile applications that have pinned certificates. Replacing server certificates would require these applications to be recoded, recompiled and redistributed. Are either of these relevant? Isn't google's proposal to eventually have Chrome stop trusting the Symantec CA system? That change seems like it would have no effect on 1) embedded devices that aren't running Chrome 2) things that already are in place that trust the CA.
- viraptor 9y ago
  It was either a bizarre misunderstanding on what will happen on Symantec's side, or "pinned" was used not in the technical meaning. I could see someone translating "these devices use Symantec's server certs and serve web traffic" to "these devices are pinned to certs issued by Symantec". But if that's the case, they should pay more attention in announcements like this one...
- ademarre 9y ago
  I think the point there is that if Chrome doesn't trust that CA, then the servers will need to be updated to use a different CA, which will break clients that pinned the old CA.
  - Manishearth 9y ago
    Iirc you can have certs signed by multiple roots though, right? In which case these sites can just get their cert signed by the pinned symantec root and a real still-valid root. Nobody's stopping symantec from signing new certs, they just won't work in Chrome (and likely other browsers) Or maybe I've misunderstood the situation.
    - wnoise 9y ago
      Unfortunately, you can't keep multiple chains in a single X.509 certificate. While you can have a given key signed by multiple CAs, AIUI, this just lets you switch easily, not tell a client about both, and just work if they trust either CA.
kaishiro 9y ago
Does anyone else get a blocking "Symantec Connect" and "Loading your community experience" for a good 10 seconds before it loads? Could just be my mobile connection but man is that lame.
- joncampbelldev 9y ago
  I've heard elsewhere in this thread that blocking ads/tracking breaks their site.
jupp0r 9y ago
Notably missing from their post: This is how we make sure issuing certificates for gmail.com, etc will never happen again. An external audit every 3 months is not going to fix anything.
lsh123 9y ago
CA infrastructure business is built on trust. Symantec lost the trust and got thrown out by Chrome. I hope that as the result Symantec will lose significant amount of CA business thus making a show case for other CAs to demonstrate that good processes and trust are key for staying in this business.
dtemp 9y ago
I'm still trying to figure out if my 36 month wildcard carts from RapidSSL are going to be distrusted. Their intermediate is signed by GeoTrust which is owned by Symantec, and a blog post says with Chrome 59, certs with a period over 33 months will be distrusted. Chrome Canary on v60 shows them still working.
auditplan 9y ago
Read the statement carefully. Symantec still does not offer a verifiable audit trail like Google's Certificate Transparency requires. They just offer to have 3rd parties review the certificates sold to customers. Without a COMPLETE audit trail Symantec can still continue to secretly issue additional certificates not part of any audit trail. The whole announcement rather decreases my trust in Symantec even more.
chaz6 9y ago
The irony is, for an industry based on trust, if this abuse is not penalized, then the industry as a whole is pointless.
unixhero 9y ago
"Community" - as if we are downstream, something easily managed. I see it as a polite way of saying "You kids".
varunvkrishnan 9y ago
Incompetence needs to be penalized. Period. Symantec's reasoning is pathetic and infact a veiled blackmail attempt.
jwilk 9y ago
Archived copy, which can be read with JS disabled: https://archive.fo/FMemh https://archive.fo/FMemh

125 points by mentat 9y ago

> This cohort is an important constituency that we believe has been under-represented to date in the public commentary that has been posted to the Google and Mozilla boards since large organizations rarely authorize employees to engage in such public discussions, particularly in an area related to security. Are these large organizations somehow incapable of putting out official statements regarding CAs? If they're being 'under-represented', it's their own fault for not speaking up.

mdip 9y ago
To me it was really bold of them to take the tone and stance that they did with regard to these large organizations. They're making the case as to why Google needed to do what it did ... all of these large organizations that will be severely impacted by having to replace all of these certificates are relying on a CA that has shown itself to not be worthy of the trust that CAs are relied upon to provide. I'm at a loss for a good analogy so I'll pick a lousy one: it'd be like if Samsung's response to the exploding Note 7 recall mandate were "Don't do this -- you're going to keep some of the biggest customers -- users of our phones -- from being able to make emergency calls/check their e-mail/respond to texts" ...yeah, but I'm still not OK with the small possibility that carrying my phone in my front pocket might result in third-degree genital burns.
- skuhn 9y ago
  As it happens, your hypothetical situation was exactly what did transpire: Samsung wanted to remotely deactivate the phones, but Verizon refused on those grounds. http://bgr.com/2016/12/09/galaxy-note-7-recall-verizon-samsung/ http://bgr.com/2016/12/09/galaxy-note-7-recall-verizon-samsu...
  - mdip 9y ago
    ... and here I thought comparing security to burning genitals might have been too much of a farce, but that takes the cake, there. :o)

tyingq 9y ago

I don't think Google was soliciting for a counter proposal from Symantec. Will be interesting to see their reply, and whether it's a literal reply or just a version push of chrome with their original plan.[1] [1] https://groups.google.com/a/chromium.org/forum/m/#!topic/blink-dev/eUAKwjihhBs https://groups.google.com/a/chromium.org/forum/m/#!topic/bli... Edit: They did ask for community feedback, comments on risk, etc. But they do already have a timeline. See link above.

toomuchtodo 9y ago
"This site does not support Chrome. Please use a browser that does not take unilateral CA authority action." might very well be the response of orgs married to Symantec. As a user, you need your bank (or other large org) more than you need your preference of browser.
- tyingq 9y ago
  Bold move though, when the fix is just buying a new cert. And the browser you're blocking has majority market share.
  - delinka 9y ago
    "Buying" as if Let's Encrypt isn't a thing. ;-)
    - walrus01 9y ago
      well, for any organization that is doing serious ecommerce these days, you would want an EV SSL cert. Which is about $95/year from namecheap or a competitor. The GUI advantages for ordinary totally ignorant end users of seeing the happy green bar at the top are enough to justify the less than $10 per month ongoing cost.
    - tyingq 9y ago
      In this context, I assume we're talking about EV certs. Though LE might be an emergency temp option for the big customers that forgot to plan around this.

revelation 9y ago

I love how none of the example "dependencies" they give should be using public CA in the first place.

ahmeni 9y ago
Given some of the internal CA systems I've dealt within the past, I'd almost prefer a public CA in some cases. Sometimes your internal CA is just the group with manual access to the certificate provisioning and signing systems with either no API or some awful re-implemented API.
- revelation 9y ago
  What API do you need? The signing system should be airgapped or you end up with the same shit that is the public CA system such as roots sitting on public FTP servers. It's a bunch of command line scripts because if you are using it any different way you are probably doing it wrong.
  - Spooky23 9y ago
    Seriously. The fancy CA systems are wacky expensive.
    - revelation 9y ago
      I'm sure people make plenty of money wrapping OpenSSL in ever more elaborate Enterprise schemes. But if the net effect is that one bug in a frontend can cause the server to tell the cert server to tell the secure enclave to tell the wire mesh protected crypto coprocessor to sign a bogus certificate then what are we really doing here.
      - Spooky23 9y ago
        They sure do! We have an offline root on an HSM for various reasons. We looked at various systems to manage intermediate ca issuance and reminders for renewal and the cost was incredible... most were >$200k! Using scripts to automatically generate certificates and using the ugly but effective Windows CA web interface for "vanity" stuff does 90% of what those solutions do. I think we had an intern write a monitoring plugin to open tickets for soon to expire certificates and a few other things that covered the remaining 10%.

MichaelGG 9y ago

>require these applications to be recoded, recompiled and redistributed. Aka "updated". The entire post is basically "ok how about we be really good from now on and suffer no consequences, cause it'd be really shitty for us if we had to be penalised". They also posture a lot talking about how big their customers are, almost boasting about how inflexible and slow these big companies are, as if that's somehow Google's or general Internet users' problem.

deleted 9y ago
[deleted]
dude01 9y ago
Yeah, they're using the argument: we're too big to fail.
- walrus01 9y ago
  there are a number of similarities between symantec and some wall street dinosaurs.
snomad 9y ago
Isn't that largely Google's stance with regard to Android? They blame the carriers and point to how slow they are, meanwhile tons of smart phones go without patching, which is no good for the general internet or users'
- AceJohnny2 9y ago
  No, there's a huge difference. The difference here is that the Manufacturers and the Carriers are gatekeepers to the customer's phones. The manufacturer made customizations to Android for their phone. Android isn't a cleanly-separated stack where the manufacturer modifies only a certain part of the stack for their phone but Google could go and update another part without risk of breaking anything (is any software project?) Then, the manufacturers sell the phones to carriers, who usually request extra customizations to (at best) add their logo, or replace default apps with the carrier's own version. This means that any updates to Android likely require updates to this customization. But even if that customization was free (from a development perspective), Carriers still have maintenance/service contracts with the manufacturers such that any manufacturer's updates must go through the carrier's "quality control" procedure. You wouldn't want an update to make the, say, LTE radio misbehave and break the carrier's network would you? Oh no... (and you certainly wouldn't want to "forget" to replace the good default app with the Carrier's crappy one) So there you go, a quick glimpse into the complex economic/contractual obligations that delays or blocks Android updates. FWIW, Google has been working both technically and contractually to untangle this mess, and is one reason for the existence of their "own" line of Nexus and now Pixel phones.

ahmeni 9y ago

There is a significant amount of "we will" language in here, rather than "we have started" or "have already begun". The choice of language speaks volumes that they only appear to be willing to take necessary audit action if Google agrees to it, rather than something that needs to be done regardless.

geofft 9y ago

> These customers include many of the largest financial services, critical infrastructure, retail and healthcare organizations in the world, as well as many government agencies. This cohort is an important constituency that we believe has been under-represented to date in the public commentary that has been posted to the Google and Mozilla boards since large organizations rarely authorize employees to engage in such public discussions, particularly in an area related to security. ... well that's their problem, right? You can't simultaneously say "These are some of the most important organizations in the world and you'll cause worldwide chaos" and "Won't someone listen to these poor companies, I am the Symantorax, I speak for the cohort, for the cohort has no tongues."

mdip 9y ago
> These are some of the most important organizations in the world and you'll cause worldwide chaos Thanks for that -- reading this comment connected a few thoughts I was struggling with. Their response felt like "if you don't follow through with your plan to punish us we'll do all of these nice things for you." It reminds me of when my kids get caught doing something wrong and I get "but I'll do all of my chores and treat my siblings nice" along with a handful of other promises of future deeds which I know will be partially or completely not-done. How different the result would have been if they came to me with the chores done and some/all of the promises fulfilled already. It would have been much better for Symantec to have responded with these items in the "completed" category with a smaller number of "we will's". But to the quote I highlighted: the issue at hand is the past failures indicate that "some of the most important organizations in the world" need to not be trusting Symantec in the first place. They've had a big enough history of failures as a CA to get the attention of Google, who I'm sure isn't terribly interested in degrading the browsing experience of their customers. The choice becomes: (1) Banhammer Symantec, take the resulting pain all at once, visibly and obviously, and get the problem fixed painfully (but it's fixed) or (2) Trust that things will be different this time and if they're not, have another security failure of some kind that results in these critically important company's customers' data being exposed, which is a circumstance that will almost certainly go unnoticed for a period of time, will be covered up if the exposure is minimal and covering up the exposure is possible or may not be discovered at all. The CA system is centered around trusting a handful of rather powerful bodies with an enormous responsibility around the overall security of the Internet. It's not a great model to begin with, but it's what we have and I haven't read about any alternatives that are viable. The only way to keep a model like this secure is to ensure that the CAs themselves are actually trustworthy and one of the more likely ways that will happen is to enforce violations of that trust in the manner Google is suggesting.

bitmapbrother 9y ago

Symantec is just going through the 5 stages of CA grief. First they were oblivious to it, then they were angry about it and called Google irresponsible. They're now at the proposal stage. Next will be depression and finally acceptance for their incompetence.

durkie 9y ago

> Embedded devices that are pinned to certificates issued by a Symantec public root to communicate to resources over the Internet or Intranet. Replacing these certificates would result in immediate failures and the need to recode and reimage the firmware for these devices. > Mobile applications that have pinned certificates. Replacing server certificates would require these applications to be recoded, recompiled and redistributed. Are either of these relevant? Isn't google's proposal to eventually have Chrome stop trusting the Symantec CA system? That change seems like it would have no effect on 1) embedded devices that aren't running Chrome 2) things that already are in place that trust the CA.

viraptor 9y ago
It was either a bizarre misunderstanding on what will happen on Symantec's side, or "pinned" was used not in the technical meaning. I could see someone translating "these devices use Symantec's server certs and serve web traffic" to "these devices are pinned to certs issued by Symantec". But if that's the case, they should pay more attention in announcements like this one...
ademarre 9y ago
I think the point there is that if Chrome doesn't trust that CA, then the servers will need to be updated to use a different CA, which will break clients that pinned the old CA.
- Manishearth 9y ago
  Iirc you can have certs signed by multiple roots though, right? In which case these sites can just get their cert signed by the pinned symantec root and a real still-valid root. Nobody's stopping symantec from signing new certs, they just won't work in Chrome (and likely other browsers) Or maybe I've misunderstood the situation.
  - wnoise 9y ago
    Unfortunately, you can't keep multiple chains in a single X.509 certificate. While you can have a given key signed by multiple CAs, AIUI, this just lets you switch easily, not tell a client about both, and just work if they trust either CA.

kaishiro 9y ago

Does anyone else get a blocking "Symantec Connect" and "Loading your community experience" for a good 10 seconds before it loads? Could just be my mobile connection but man is that lame.

joncampbelldev 9y ago
I've heard elsewhere in this thread that blocking ads/tracking breaks their site.

jupp0r 9y ago

Notably missing from their post: This is how we make sure issuing certificates for gmail.com, etc will never happen again. An external audit every 3 months is not going to fix anything.

lsh123 9y ago

CA infrastructure business is built on trust. Symantec lost the trust and got thrown out by Chrome. I hope that as the result Symantec will lose significant amount of CA business thus making a show case for other CAs to demonstrate that good processes and trust are key for staying in this business.

dtemp 9y ago

I'm still trying to figure out if my 36 month wildcard carts from RapidSSL are going to be distrusted. Their intermediate is signed by GeoTrust which is owned by Symantec, and a blog post says with Chrome 59, certs with a period over 33 months will be distrusted. Chrome Canary on v60 shows them still working.

auditplan 9y ago

Read the statement carefully. Symantec still does not offer a verifiable audit trail like Google's Certificate Transparency requires. They just offer to have 3rd parties review the certificates sold to customers. Without a COMPLETE audit trail Symantec can still continue to secretly issue additional certificates not part of any audit trail. The whole announcement rather decreases my trust in Symantec even more.

chaz6 9y ago

The irony is, for an industry based on trust, if this abuse is not penalized, then the industry as a whole is pointless.

unixhero 9y ago

"Community" - as if we are downstream, something easily managed. I see it as a polite way of saying "You kids".

varunvkrishnan 9y ago

Incompetence needs to be penalized. Period. Symantec's reasoning is pathetic and infact a veiled blackmail attempt.

jwilk 9y ago

Archived copy, which can be read with JS disabled: https://archive.fo/FMemh https://archive.fo/FMemh