6 ms·

GlobalSign: RSA-1024 leaf certificate issued after 2013-12-31

119 points by tech234a 6y ago

SheinhardtWigCo 6y ago
tl;dr: the certificate was requested in 2010 and an email challenge was sent at that time, but the validation URL wasn't clicked until last week, and the certificate was allowed to be issued because of a misconfigured linter.
- felixhandte 6y ago
  Do I also understand correctly that a certificate was issued recently based on proving ownership of a domain ten years ago? That also sounds like a serious issue.
  - tristor 6y ago
    Technically, it appears that a validation link to prove domain ownership was sent 10 (or 11) years prior, but wasn't clicked until recently. The issue isn't that ownership was proven ten years ago, the issue is that the link to do so was still valid for such a long period of time rather than being expired.
    - kevincox 6y ago
      Well yes. That was an obvious issue. However the point is that the email was received 10 years ago, so you are only proving that you had control of the domain at that time. Of course validation links should expire in a relatively short timeframe.
  - tialaramex 6y ago
    Right. GlobalSign asserts in their incident report that: > All other orders in the GCC platform that are older than 1 year and have not been fully processed (940909 orders) have been moved into a cancelled state on 9/02/2021. So (in this platform at least, and I believe this is either their oldest or one of their oldest legacy platforms) until this fix there were actually almost one million such certificates that could in principle be created based on information over a year old. They've attached a list of 112 other certificates that really were issued - and I think a reasonable person would judge should not have been issued, all have expired (and so there's no point in revoking them now). This at least shows they were thinking about the potential wider problem and not just focused on damage control for this one certificate, so that's something. The reference to "Ballot 169" is to the Ten Blessed Methods, Ballot 169 was a vote by the CA/Browser Forum to modify the Baseline Requirements to require these specific methods (today there aren't actually ten of them, but the general idea of spelling out how to validate control remains) rather than allowing CAs to each make up whatever methods of domain validation they thought would be good enough in their opinion. Ballot 169 actually got caught in a big bureaucratic mess but Mozilla forced the issue by making the Ten Blessed Methods part of its own rules anyway while the mess was sorted out so the CAs had to obey. Under the Ten Blessed Methods there's an explicit deadline, the random codes they're sending out by email are only good for thirty days. But I think we can judge from this that GlobalSign's prior method never expired codes at all, and you can imagine that when a deadline was introduced nobody wanted to retroactively extend that deadline to previous codes. At the moment that decision happens it seems superficially OK. Last year it was OK for Sue in accounting to take six weeks to finally click the link, next year the Ten Blessed Methods mean we'll have to show a "I'm sorry this request has expired, please order again" message but why jump ahead, we should let Sue's old order go through whenever she gets around to clicking it. It takes a particular type of person to ask "What if Sue takes six years instead of six weeks?" and those sorts of people don't tend to work for a CA. For example, the Chrome browser has code to actually enforce the requirements limiting leaf certificate lifetimes. In the modern era the browser code matches exactly what the rules say. Slightly before that the rules are rather woolly and so the browser approximates them, how long is "39 months" in terms a machine can understand? Chrome's programmers worked out a rationale for 1188 days as the answer. But at the outset there just weren't any rules. You could react to that by saying OK, if the certificate claims to have been issued in 1995 then I guess we believe them and accept any validity period whatsoever. 40 year leaf certificate? No problem. This opens an enormous security hole. So, Chrome's engineers just picked 10 years. And just like that, the hole closes, because sure enough it has been more than 10 years since the last date when such "No rules" certificates could be issued, and so there are no certificates issued recently enough to be unexpired which are allowed a ten year expiry.
    - schoen 6y ago
      Tialaramex, your PKI history analysis is a treasure of Hacker News!
- mariusmarais 6y ago
  Seems like someone finally got to Inbox Zero. But terrifying at the same time. I wonder how many valuable targets might have had outstanding validations from 10 years ago, or even 3 years ago.
  - FDSGSG 6y ago
    Not very terrifying. Odds are someone with access to ancient validation links would also be able to perform new validations.
    - mariusmarais 6y ago
      I was thinking about domains that may have transferred recently. We took over a very "nice" domain last year, but obviously not the historic emails.
fomine3 6y ago
Manual verification for papaya is funny part.
leereeves 6y ago
Also, the domain was flagged as high risk because celticpapaya.com contains the keyword "pay". And, ironically, it did turn out to be high risk.
tristor 6y ago
This is a deeply concerning issue to have happen, but I do want to commend GlobalSign for their detailed response in the ticket. It looks like there's other issues they have not yet addressed per the last comment from Ryan Sleevi, but their response so far is very helpful in understanding what occurred.
eqvinox 6y ago
I don't think I've ever laughed so hard at a security critical bug :'( But yeah, as Ryan Sleevi asks on the bugreport, I wonder how many other certificates may have been issued based on "historical" verification tokens...
- 35fbe7d3d5b9 6y ago
  Good news, you don't have to wonder how many GlobalSign issued, just look at the Excel file attached to the ticket and you can see all 113 of their "oopsies".
  - tialaramex 6y ago
    Note that (opened in LibreOffice but I assume this isn't a document compat issue) the URLs all start https// but should begin https:// https:// instead. They're links into crt.sh which is a public view into the contents of the Certificate Transparency Logs, although they've also baked the certificate serial numbers into these links so if crt.sh goes away you can rewrite them as queries for any other system. It looks like an expected long tail, most of them were probably issued only days or weeks after the codes used should have expired (in 2017). What made this one jump out was the use of a 1024-bit RSA key. When it was really ordered that would be completely normal, but even in 2017 it would have been obviously prohibited, I have looked at a few of the other 112 and they're unextraordinary, typically 2048-bit RSA keys, likely originally ordered earlier in 2017. It would be useful (perhaps I will ask on m.d.s.policy) to know what the actual original order dates were for the 112 certificates in this list.
    - bombcar 6y ago
      I assume the https// is intentional to prevent autolinking or similar (excel loves to make hyperlinks and not everyone knows how to beat it into submission).
    - 35fbe7d3d5b9 6y ago
      > It would be useful (perhaps I will ask on m.d.s.policy) to know what the actual original order dates were for the 112 certificates in this list. It'd be really fascinating to compare that with WHOIS data for the domains. That might show other misissuances - certificates issued for non-registered domains.
tptacek 6y ago
What you really want here is the other side of the story, from the CelticPapaya people. Does anyone know anyone there? From Archive.org, it appears the site has never worked properly. I found a couple of email addresses, though, in the Google cache.
- tristor 6y ago
  Yeah, I'd be interested to understand the circumstances that lead to them opening and clicking a link in an email this old.
  - londons_explore 6y ago
    During a migration from one email system to another, it wouldn't be unusual for some virus scanner product to click every link in every email... Or alternatively, when searching their inbox for "SSL certificate" they found the verification email and clicked the link as a quick way to get to the homepage of whichever company they used last time for their certificate.
    - comex 6y ago
      Or they searched for SSL certificate, found the email, and didn’t notice the date. Perhaps it was marked unread, given that the email verification was never completed in 2010. They thought it was something they needed to do, and did it.
    - tialaramex 6y ago
      It's definitely plausible that the historic URLs which caused this are vulnerable to just being opened automatically by a machine (such as a "virus scanner"). The right fix here is that the URL leads to a simple POST form where you confirm you want the certificate issued. Tools like virus scanners don't fill out forms. We know that by the time the Ten Blessed Methods were coming into existence CAs were discovering that they had numerous cases where an order would be successfully "validated" by contact with a human, but without any human actually doing anything. Email links "clicked" by automated systems, a system that checks you've uploaded a file to your web server treats the 404 error it gets as an acceptable response showing the file was uploaded, several hilarious goofs. [ The irony, for those playing along at home, is that we don't know of any incidents where this was used for an attack. It's like coming home to discover the reason you didn't have your front door key is because you left it on the kitchen counter, with the door unlocked, all day. But no harm done... ] The current authorised method closest in spirit to what GlobalSign were doing back in 2010 is 3.2.2.4.4 (Constructed Email to Domain Contact). This method actually doesn't tell them to prevent this particular mishap (machines clicking links), so perhaps that should be corrected.
cmroanirgo 6y ago
The main take away I see is a reorg/consolidation of their processes: > GlobalSign has concluded there are too many procedural and technical gaps in the current process to prevent incorrect CA configuration occurring in the future and has decided to halt the provisioning of new CA in the GCC platform until 31/03/2021, this in order to fully rebuild the CA request, creation and provisioning process. Whereas the previous process could be considered fragmented (with different tickets and information depending on the team that had to execute a specific sub-action) we will be building a new flow that consolidates all required information and actions into a single process, with additional steps in terms of compliance oversight and a hard compliance approval step prior to putting a new CA into production / issuing state. We will share the details of the new process 16/02/2021 and will complete implementation before 31/03/2021. CA provisioning on the GCC platform will only resume after the implementation of the new process has been successfully completed.
RandomBK 6y ago
> Certificate ordered with ID [...] for www.celticpapaya.com. > Domain flagged as high-risk (keyword "pay") Somehow, I found this line particularly funny. Simple systems and heuristics like keyword searching continue to power and protect systems around the world, triggering false positives (and negatives) that can often be opaque to outside observers. Will ML and NLP eventually overtake these systems, bringing along their own pros/cons? Or will these systems continue to proliferate into the far future, such that in the distant year 2077, some poor traveler gets dragged out for special screening because they listed "burgundy" in their customs manifest?
- kaszanka 6y ago
  This is as transparent to outside observers as it gets compared to ML.
- kevincox 6y ago
  I also find it funny. But the key redeeming feature is that it was paused, and automatically sent to human review. So sure, your cert takes a little while to issue but no significant harm. The obvious contrast is Google closing legitimate paying accounts with no option for human review.
  - FDSGSG 6y ago
    What's the point of having a CA perform phishing checks anyway? Especially since the phishers can just use letsencrypt.

119 points by tech234a 6y ago

tl;dr: the certificate was requested in 2010 and an email challenge was sent at that time, but the validation URL wasn't clicked until last week, and the certificate was allowed to be issued because of a misconfigured linter.

felixhandte 6y ago
Do I also understand correctly that a certificate was issued recently based on proving ownership of a domain ten years ago? That also sounds like a serious issue.
- tristor 6y ago
  Technically, it appears that a validation link to prove domain ownership was sent 10 (or 11) years prior, but wasn't clicked until recently. The issue isn't that ownership was proven ten years ago, the issue is that the link to do so was still valid for such a long period of time rather than being expired.
  - kevincox 6y ago
    Well yes. That was an obvious issue. However the point is that the email was received 10 years ago, so you are only proving that you had control of the domain at that time. Of course validation links should expire in a relatively short timeframe.
- tialaramex 6y ago
  Right. GlobalSign asserts in their incident report that: > All other orders in the GCC platform that are older than 1 year and have not been fully processed (940909 orders) have been moved into a cancelled state on 9/02/2021. So (in this platform at least, and I believe this is either their oldest or one of their oldest legacy platforms) until this fix there were actually almost one million such certificates that could in principle be created based on information over a year old. They've attached a list of 112 other certificates that really were issued - and I think a reasonable person would judge should not have been issued, all have expired (and so there's no point in revoking them now). This at least shows they were thinking about the potential wider problem and not just focused on damage control for this one certificate, so that's something. The reference to "Ballot 169" is to the Ten Blessed Methods, Ballot 169 was a vote by the CA/Browser Forum to modify the Baseline Requirements to require these specific methods (today there aren't actually ten of them, but the general idea of spelling out how to validate control remains) rather than allowing CAs to each make up whatever methods of domain validation they thought would be good enough in their opinion. Ballot 169 actually got caught in a big bureaucratic mess but Mozilla forced the issue by making the Ten Blessed Methods part of its own rules anyway while the mess was sorted out so the CAs had to obey. Under the Ten Blessed Methods there's an explicit deadline, the random codes they're sending out by email are only good for thirty days. But I think we can judge from this that GlobalSign's prior method never expired codes at all, and you can imagine that when a deadline was introduced nobody wanted to retroactively extend that deadline to previous codes. At the moment that decision happens it seems superficially OK. Last year it was OK for Sue in accounting to take six weeks to finally click the link, next year the Ten Blessed Methods mean we'll have to show a "I'm sorry this request has expired, please order again" message but why jump ahead, we should let Sue's old order go through whenever she gets around to clicking it. It takes a particular type of person to ask "What if Sue takes six years instead of six weeks?" and those sorts of people don't tend to work for a CA. For example, the Chrome browser has code to actually enforce the requirements limiting leaf certificate lifetimes. In the modern era the browser code matches exactly what the rules say. Slightly before that the rules are rather woolly and so the browser approximates them, how long is "39 months" in terms a machine can understand? Chrome's programmers worked out a rationale for 1188 days as the answer. But at the outset there just weren't any rules. You could react to that by saying OK, if the certificate claims to have been issued in 1995 then I guess we believe them and accept any validity period whatsoever. 40 year leaf certificate? No problem. This opens an enormous security hole. So, Chrome's engineers just picked 10 years. And just like that, the hole closes, because sure enough it has been more than 10 years since the last date when such "No rules" certificates could be issued, and so there are no certificates issued recently enough to be unexpired which are allowed a ten year expiry.
  - schoen 6y ago
    Tialaramex, your PKI history analysis is a treasure of Hacker News!
mariusmarais 6y ago
Seems like someone finally got to Inbox Zero. But terrifying at the same time. I wonder how many valuable targets might have had outstanding validations from 10 years ago, or even 3 years ago.
- FDSGSG 6y ago
  Not very terrifying. Odds are someone with access to ancient validation links would also be able to perform new validations.
  - mariusmarais 6y ago
    I was thinking about domains that may have transferred recently. We took over a very "nice" domain last year, but obviously not the historic emails.

fomine3 6y ago

Manual verification for papaya is funny part.

leereeves 6y ago

Also, the domain was flagged as high risk because celticpapaya.com contains the keyword "pay". And, ironically, it did turn out to be high risk.

tristor 6y ago

This is a deeply concerning issue to have happen, but I do want to commend GlobalSign for their detailed response in the ticket. It looks like there's other issues they have not yet addressed per the last comment from Ryan Sleevi, but their response so far is very helpful in understanding what occurred.

eqvinox 6y ago

I don't think I've ever laughed so hard at a security critical bug :'( But yeah, as Ryan Sleevi asks on the bugreport, I wonder how many other certificates may have been issued based on "historical" verification tokens...

35fbe7d3d5b9 6y ago
Good news, you don't have to wonder how many GlobalSign issued, just look at the Excel file attached to the ticket and you can see all 113 of their "oopsies".
- tialaramex 6y ago
  Note that (opened in LibreOffice but I assume this isn't a document compat issue) the URLs all start https// but should begin https:// https:// instead. They're links into crt.sh which is a public view into the contents of the Certificate Transparency Logs, although they've also baked the certificate serial numbers into these links so if crt.sh goes away you can rewrite them as queries for any other system. It looks like an expected long tail, most of them were probably issued only days or weeks after the codes used should have expired (in 2017). What made this one jump out was the use of a 1024-bit RSA key. When it was really ordered that would be completely normal, but even in 2017 it would have been obviously prohibited, I have looked at a few of the other 112 and they're unextraordinary, typically 2048-bit RSA keys, likely originally ordered earlier in 2017. It would be useful (perhaps I will ask on m.d.s.policy) to know what the actual original order dates were for the 112 certificates in this list.
  - bombcar 6y ago
    I assume the https// is intentional to prevent autolinking or similar (excel loves to make hyperlinks and not everyone knows how to beat it into submission).
  - 35fbe7d3d5b9 6y ago
    > It would be useful (perhaps I will ask on m.d.s.policy) to know what the actual original order dates were for the 112 certificates in this list. It'd be really fascinating to compare that with WHOIS data for the domains. That might show other misissuances - certificates issued for non-registered domains.

tptacek 6y ago

What you really want here is the other side of the story, from the CelticPapaya people. Does anyone know anyone there? From Archive.org, it appears the site has never worked properly. I found a couple of email addresses, though, in the Google cache.

tristor 6y ago
Yeah, I'd be interested to understand the circumstances that lead to them opening and clicking a link in an email this old.
- londons_explore 6y ago
  During a migration from one email system to another, it wouldn't be unusual for some virus scanner product to click every link in every email... Or alternatively, when searching their inbox for "SSL certificate" they found the verification email and clicked the link as a quick way to get to the homepage of whichever company they used last time for their certificate.
  - comex 6y ago
    Or they searched for SSL certificate, found the email, and didn’t notice the date. Perhaps it was marked unread, given that the email verification was never completed in 2010. They thought it was something they needed to do, and did it.
  - tialaramex 6y ago
    It's definitely plausible that the historic URLs which caused this are vulnerable to just being opened automatically by a machine (such as a "virus scanner"). The right fix here is that the URL leads to a simple POST form where you confirm you want the certificate issued. Tools like virus scanners don't fill out forms. We know that by the time the Ten Blessed Methods were coming into existence CAs were discovering that they had numerous cases where an order would be successfully "validated" by contact with a human, but without any human actually doing anything. Email links "clicked" by automated systems, a system that checks you've uploaded a file to your web server treats the 404 error it gets as an acceptable response showing the file was uploaded, several hilarious goofs. [ The irony, for those playing along at home, is that we don't know of any incidents where this was used for an attack. It's like coming home to discover the reason you didn't have your front door key is because you left it on the kitchen counter, with the door unlocked, all day. But no harm done... ] The current authorised method closest in spirit to what GlobalSign were doing back in 2010 is 3.2.2.4.4 (Constructed Email to Domain Contact). This method actually doesn't tell them to prevent this particular mishap (machines clicking links), so perhaps that should be corrected.

cmroanirgo 6y ago

The main take away I see is a reorg/consolidation of their processes: > GlobalSign has concluded there are too many procedural and technical gaps in the current process to prevent incorrect CA configuration occurring in the future and has decided to halt the provisioning of new CA in the GCC platform until 31/03/2021, this in order to fully rebuild the CA request, creation and provisioning process. Whereas the previous process could be considered fragmented (with different tickets and information depending on the team that had to execute a specific sub-action) we will be building a new flow that consolidates all required information and actions into a single process, with additional steps in terms of compliance oversight and a hard compliance approval step prior to putting a new CA into production / issuing state. We will share the details of the new process 16/02/2021 and will complete implementation before 31/03/2021. CA provisioning on the GCC platform will only resume after the implementation of the new process has been successfully completed.

RandomBK 6y ago

> Certificate ordered with ID [...] for www.celticpapaya.com. > Domain flagged as high-risk (keyword "pay") Somehow, I found this line particularly funny. Simple systems and heuristics like keyword searching continue to power and protect systems around the world, triggering false positives (and negatives) that can often be opaque to outside observers. Will ML and NLP eventually overtake these systems, bringing along their own pros/cons? Or will these systems continue to proliferate into the far future, such that in the distant year 2077, some poor traveler gets dragged out for special screening because they listed "burgundy" in their customs manifest?

kaszanka 6y ago
This is as transparent to outside observers as it gets compared to ML.
kevincox 6y ago
I also find it funny. But the key redeeming feature is that it was paused, and automatically sent to human review. So sure, your cert takes a little while to issue but no significant harm. The obvious contrast is Google closing legitimate paying accounts with no option for human review.
- FDSGSG 6y ago
  What's the point of having a CA perform phishing checks anyway? Especially since the phishers can just use letsencrypt.