45 ms·

Author here. It has gotten kind of hard to follow what has happened, so here's a chronology: 1. In September, Chrome 53 was released, which enabled mandatory

by agwa 10y ago

Author here. It has gotten kind of hard to follow what has happened, so here's a chronology: 1. In September, Chrome 53 was released, which enabled mandatory Certificate Transparency for Symantec certificates due to Symantec's history of incompetence. Some website operators, such as Chase, asked Symantec to submit their certificates to Certificate Transparency logs in such a way that the certificate wouldn't be trusted by Chrome, triggering the ERR_CERTIFICATE_TRANSPARENCY_REQUIRED error. This is when I wrote this blog post. 2. Last week, an internal timebomb expired in older versions of Chrome causing this error message for any website using a Symantec certificate issued since June. Basically, Chrome contains a list of Certificate Transparency logs that it trusts, and this list has a 10 week expiration date. So if Chrome was built more than 10 weeks ago, there would be no trusted Certificate Transparency logs, and therefore any certificate that was supposed to be logged (such as new Symantec certs) would be untrusted and display this error message. The Chrome team was able to fix this within 24 hours by remotely disabling CT enforcement in Chrome. (When Chrome starts up, it fetches a list of feature flags from a Chrome server using a system called Finch which is independent of the normal upgrade system.) 3. Today, the Chromium packages in several Linux distros, including Ubuntu, became 10 weeks old. For some reason, they have not picked up the Finch update, and so they are displaying this error message for all Symantec certificates issued since June. This is not confirmed yet, but the current hypothesis is that the distros have disabled Finch for privacy reasons. It will probably require a distro package upgrade to fix.

swalsh 10y ago
Is there any reason, in 2016, to use Symantec over LetsEncrypt?
- xyzzy123 10y ago
  extended validation (mandatory for bigcos) and support contracts (important for corporates, not sure how useful "in real life"). Also, inventory management, which is helpful when you have hundreds or thousands of certs.
  - walrus01 10y ago
    There are lots of EV issuers who haven't been caught doing shady shit like Symantec...
    - nailer 10y ago
      cough Hey there! CertSimple only does EV, and we do it completely differently from every other company: we check as much of your company's details before you pay, matching your order to a registered/active entity, flagging up things before asking for your credit card number, and helping you resolve any missing identification steps based on your company, order and the domain names involved. I've been on HN for a decade and was at YC in Mountain View last week for the 10-min final interview last week (we didn't make it, which I blame on me being a jetlagged mess). OTOH the AirBnB we stayed in used a customer as their ISP. We're used by a bunch of companies HN folks might know, including Travis, Tito, and Monzo. https://certsimple.com/about https://certsimple.com/about
- deleted 10y ago
  [deleted]
- jlgaddis 10y ago
  I use LetsEncrypt on a (personal) site or two and on a couple for work. We've got customers that mostly run Windows, however, and it's a helluva lot easier to, for example, just get a three-year certificate from <vendor>, install it, and then forget about it for the next three years. Fortunately, I don't (usually) have to deal with the Windows boxes (i.e. actually installing and/or configuring the certificates) but I'm often the one acquiring the certificates.
- JoshTriplett 10y ago
  Wildcards.
  - pilom 10y ago
    We use wildcard certs for a number of domains to support affiliate subdomains. So affiliate.example.com and affiliate2.example.com are all served by the same servers and thus all need to validate with one cert.
    - duaneb 10y ago
      You can use SAN certificates to avoid wildcards while still validating for a number of domains. They don't even need to be subdomains.
      - Johnny555 10y ago
        There's a limit to how many SAN's you can fit in one cert. There is apparently no defined upper bound, but dependent on the client's implementation. 25 - 100 names seems to be the common limit supported by most registrars. In the parent poster's case, it sounds like a good use case for a wildcard cert. They may have thousands of affiliates, and may not know all of the affiliates ahead of time, so with a SAN cert, would need to reissue the certificate each time a new affiliate signs up. https://community.letsencrypt.org/t/subjectaltname-certificates/100/16 https://community.letsencrypt.org/t/subjectaltname-certifica...
        duaneb 10y ago
        Oh, I completely agree—I maintain a SAN and it's a royal pain in the ass. But there are options, and a SAN has lower risk if your key is leaked (only the enumerated domains will validate).
- NeutronBoy 10y ago
  For small startups, probably not. Large organisations have far more stringent processes around certificates. Auditing, validation, testing, regulatory requirements, EV, support contracts and SLAs, etc.
  - gcb0 10y ago
    yet all that don't add absolute any value to the company nor security to their data. since they're still paying a vendor full of problems. incompetence promoting incompetence. the staple of big corporations.
- discreditable 10y ago
  When it comes to Certificate Transparency, LE doesn't bed SCTs in the issued certificates, which is mildly annoying.
  - pfg 10y ago
    Practically speaking, this has no impact until browsers start requiring CT (which is in about a year for Chrome).
- agwa 10y ago
  I can't speak for Symantec, but folks use SSLMate (which resells Comodo) because of the customer support, the wildcard certificates, the lack of rate limits, and the central management of certificates (which integrates with Cert Spotter, our Certificate Transparency monitor). Also, some of our customers have special requirements and unfortunately can't automate certificate issuance. These customers want year long certificates with email validation, but still prefer using SSLMate's command line workflow over the clunky web interfaces of most CAs.
- mootothemax 10y ago
  The Symantec "this website is ssl protected by"-badge has high customer recognition and will raise conversion rates a tenth of a percent or two. If I hadn't seen it repeatedly with my own eyes - most trust images will get you a small bump, for some reason Symantec performs the best. It's understandable and yet very frustrating!
  - asclepi 10y ago
    It's not "Symantec", it's not "Norton", it's the "VeriSign checkmark". That's why they also bought the checkmark from VeriSign (which was also their corporate logo) when they acquired the CA business, and VeriSign went on and adopted a new corporate logo. The checkmark been around since at least 1997, and in it existence it has been associated with the most trustworthy institutions. It has become a synonym of trust. That checkmark is one of Symantec's most valuable assets.
- asclepi 10y ago
  Apart from the reasons posted earlier, "legacy" also comes into play. Symantec acquired the CA business from the old VeriSign in 2010, which back then was the largest CA and had a large corporate client portfolio accustomed to paying $$$$ per certificate per year. Corporations can become very reluctant to change, even if it benefits them, and there are some who just pay up every year without researching alternatives or even being aware of any. You will find that there are a lot of similarities with the domain name industry where there is also a company operating with a VeriSign heritage: Network Solutions. Network Solutions also charge premium prices, because their legacy customers expect those. And although market share is slowly eroding over the years and cheap (and even free) competitors rise, more than enough legacy customers stay on board to remain very profitable for many years to come. What definitely helps is that most of the processes in the domain registration and CA industries can be automated and don't need a lot of maintenance.
  - yuhong 10y ago
    And they tends to need legacy stuff such as SHA-1 certificates more often.
bradleyjg 10y ago
The Chrome team was able to fix this within 24 hours by remotely disabling CT enforcement in Chrome. (When Chrome starts up, it fetches a list of feature flags from a Chrome server using a system called Finch which is independent of the normal upgrade system.) After reading the blogpost, it seems like it was working as intended. Why "fix" it?
- agwa 10y ago
  The expiration was designed back when Certificate Transparency was only required to make Extended Validation certificates display a green bar. The intention was never to make certificates fail entirely when a Chrome build was more than 10 weeks old. Now that CT is being used for more than just a green bar, the log list expiration is being revisited.
duaneb 10y ago
> by remotely disabling CT enforcement in Chrome A) why is this possible, and B) isn't the whole point of the expiring list to provide a stick? Why remove the stick at the last moment?
darklajid 10y ago
» When Chrome starts up, it fetches a list of feature flags from a Chrome server using a system called Finch which is independent of the normal upgrade system. I'm not a Chrome user. But that sounds awful at first. What is the idea behind this service? Is there any documentation about the 'features' these flags can enable/disable? I understand that I'm paranoid at times AND I really dislike Google, but why would you have an extra channel to influence deployments, other than offering a global update file?
- sbierwagen 10y ago
  If you "really" dislike Google, then why are you using Chrome?
  - shawkinaw 10y ago
    > I'm not a Chrome user.
    - sbierwagen 10y ago
      Oops, dang it.
  - vuanotino 10y ago
    I'm not the GP, and even though I personally really dislike Google too, I'm forced to use Chrome because Firefox fucking sucks.
- kodablah 10y ago
  I fail to see this as any more harmful than the auto updating feature. I understand the concern of multiple avenues to phone home as being worse than one, but it's negligible considering it's the same company. Coupled with all of their other services for security incidents, prediction, auto correct, spelling, usage stats, dangerous page warnings, etc, I think it's just another log on the fire and not worth being concerned about specifically.
  - 10y ago
agwa 10y ago
Update: Finch is always disabled in Chromium builds. Disabling it was not a decision by distro maintainers. Supposedly, the following command line options will enable it, but it seems like just upgrading is a better option: --variations-server-url=https://clients4.google.com/chrome-variations/seed https://clients4.google.com/chrome-variations/seed --fake-variations-channel=stable

by agwa 10y ago

swalsh 10y ago

Is there any reason, in 2016, to use Symantec over LetsEncrypt?

xyzzy123 10y ago
extended validation (mandatory for bigcos) and support contracts (important for corporates, not sure how useful "in real life"). Also, inventory management, which is helpful when you have hundreds or thousands of certs.
- walrus01 10y ago
  There are lots of EV issuers who haven't been caught doing shady shit like Symantec...
  - nailer 10y ago
    cough Hey there! CertSimple only does EV, and we do it completely differently from every other company: we check as much of your company's details before you pay, matching your order to a registered/active entity, flagging up things before asking for your credit card number, and helping you resolve any missing identification steps based on your company, order and the domain names involved. I've been on HN for a decade and was at YC in Mountain View last week for the 10-min final interview last week (we didn't make it, which I blame on me being a jetlagged mess). OTOH the AirBnB we stayed in used a customer as their ISP. We're used by a bunch of companies HN folks might know, including Travis, Tito, and Monzo. https://certsimple.com/about https://certsimple.com/about
deleted 10y ago
[deleted]
jlgaddis 10y ago
I use LetsEncrypt on a (personal) site or two and on a couple for work. We've got customers that mostly run Windows, however, and it's a helluva lot easier to, for example, just get a three-year certificate from <vendor>, install it, and then forget about it for the next three years. Fortunately, I don't (usually) have to deal with the Windows boxes (i.e. actually installing and/or configuring the certificates) but I'm often the one acquiring the certificates.
JoshTriplett 10y ago
Wildcards.
- pilom 10y ago
  We use wildcard certs for a number of domains to support affiliate subdomains. So affiliate.example.com and affiliate2.example.com are all served by the same servers and thus all need to validate with one cert.
  - duaneb 10y ago
    You can use SAN certificates to avoid wildcards while still validating for a number of domains. They don't even need to be subdomains.
    - Johnny555 10y ago
      There's a limit to how many SAN's you can fit in one cert. There is apparently no defined upper bound, but dependent on the client's implementation. 25 - 100 names seems to be the common limit supported by most registrars. In the parent poster's case, it sounds like a good use case for a wildcard cert. They may have thousands of affiliates, and may not know all of the affiliates ahead of time, so with a SAN cert, would need to reissue the certificate each time a new affiliate signs up. https://community.letsencrypt.org/t/subjectaltname-certificates/100/16 https://community.letsencrypt.org/t/subjectaltname-certifica...
      - duaneb 10y ago
        Oh, I completely agree—I maintain a SAN and it's a royal pain in the ass. But there are options, and a SAN has lower risk if your key is leaked (only the enumerated domains will validate).
NeutronBoy 10y ago
For small startups, probably not. Large organisations have far more stringent processes around certificates. Auditing, validation, testing, regulatory requirements, EV, support contracts and SLAs, etc.
- gcb0 10y ago
  yet all that don't add absolute any value to the company nor security to their data. since they're still paying a vendor full of problems. incompetence promoting incompetence. the staple of big corporations.
discreditable 10y ago
When it comes to Certificate Transparency, LE doesn't bed SCTs in the issued certificates, which is mildly annoying.
- pfg 10y ago
  Practically speaking, this has no impact until browsers start requiring CT (which is in about a year for Chrome).
agwa 10y ago
I can't speak for Symantec, but folks use SSLMate (which resells Comodo) because of the customer support, the wildcard certificates, the lack of rate limits, and the central management of certificates (which integrates with Cert Spotter, our Certificate Transparency monitor). Also, some of our customers have special requirements and unfortunately can't automate certificate issuance. These customers want year long certificates with email validation, but still prefer using SSLMate's command line workflow over the clunky web interfaces of most CAs.
mootothemax 10y ago
The Symantec "this website is ssl protected by"-badge has high customer recognition and will raise conversion rates a tenth of a percent or two. If I hadn't seen it repeatedly with my own eyes - most trust images will get you a small bump, for some reason Symantec performs the best. It's understandable and yet very frustrating!
- asclepi 10y ago
  It's not "Symantec", it's not "Norton", it's the "VeriSign checkmark". That's why they also bought the checkmark from VeriSign (which was also their corporate logo) when they acquired the CA business, and VeriSign went on and adopted a new corporate logo. The checkmark been around since at least 1997, and in it existence it has been associated with the most trustworthy institutions. It has become a synonym of trust. That checkmark is one of Symantec's most valuable assets.
asclepi 10y ago
Apart from the reasons posted earlier, "legacy" also comes into play. Symantec acquired the CA business from the old VeriSign in 2010, which back then was the largest CA and had a large corporate client portfolio accustomed to paying $$$$ per certificate per year. Corporations can become very reluctant to change, even if it benefits them, and there are some who just pay up every year without researching alternatives or even being aware of any. You will find that there are a lot of similarities with the domain name industry where there is also a company operating with a VeriSign heritage: Network Solutions. Network Solutions also charge premium prices, because their legacy customers expect those. And although market share is slowly eroding over the years and cheap (and even free) competitors rise, more than enough legacy customers stay on board to remain very profitable for many years to come. What definitely helps is that most of the processes in the domain registration and CA industries can be automated and don't need a lot of maintenance.
- yuhong 10y ago
  And they tends to need legacy stuff such as SHA-1 certificates more often.

bradleyjg 10y ago

The Chrome team was able to fix this within 24 hours by remotely disabling CT enforcement in Chrome. (When Chrome starts up, it fetches a list of feature flags from a Chrome server using a system called Finch which is independent of the normal upgrade system.) After reading the blogpost, it seems like it was working as intended. Why "fix" it?

agwa 10y ago
The expiration was designed back when Certificate Transparency was only required to make Extended Validation certificates display a green bar. The intention was never to make certificates fail entirely when a Chrome build was more than 10 weeks old. Now that CT is being used for more than just a green bar, the log list expiration is being revisited.

duaneb 10y ago

> by remotely disabling CT enforcement in Chrome A) why is this possible, and B) isn't the whole point of the expiring list to provide a stick? Why remove the stick at the last moment?

darklajid 10y ago

» When Chrome starts up, it fetches a list of feature flags from a Chrome server using a system called Finch which is independent of the normal upgrade system. I'm not a Chrome user. But that sounds awful at first. What is the idea behind this service? Is there any documentation about the 'features' these flags can enable/disable? I understand that I'm paranoid at times AND I really dislike Google, but why would you have an extra channel to influence deployments, other than offering a global update file?

sbierwagen 10y ago
If you "really" dislike Google, then why are you using Chrome?
- shawkinaw 10y ago
  > I'm not a Chrome user.
  - sbierwagen 10y ago
    Oops, dang it.
- vuanotino 10y ago
  I'm not the GP, and even though I personally really dislike Google too, I'm forced to use Chrome because Firefox fucking sucks.
kodablah 10y ago
I fail to see this as any more harmful than the auto updating feature. I understand the concern of multiple avenues to phone home as being worse than one, but it's negligible considering it's the same company. Coupled with all of their other services for security incidents, prediction, auto correct, spelling, usage stats, dangerous page warnings, etc, I think it's just another log on the fire and not worth being concerned about specifically.
- 10y ago

agwa 10y ago

Update: Finch is always disabled in Chromium builds. Disabling it was not a decision by distro maintainers. Supposedly, the following command line options will enable it, but it seems like just upgrading is a better option: --variations-server-url=https://clients4.google.com/chrome-variations/seed https://clients4.google.com/chrome-variations/seed --fake-variations-channel=stable