7 ms·

Sunlight, a Certificate Transparency log implementation

217 points by ben_s 3y ago

Dowwie 3y ago
https://github.com/FiloSottile/sunlight https://github.com/FiloSottile/sunlight
mcpherrinm 3y ago
I’m excited for others to see the work we’ve been doing to make CT easier to run. Also see the discussion on the ct-policy mailing list: https://groups.google.com/a/chromium.org/g/ct-policy/c/v9JzlbphYBs https://groups.google.com/a/chromium.org/g/ct-policy/c/v9Jzl... (I am the author of the blog post)
- FiloSottile 3y ago
  This is one of the projects I've been most excited about in the last few years. It let me backport to Certificate Transparency some of the modern designs ideas that came after it. Beyond the Let's Encrypt announcement and the ct-policy thread (which includes a technical and advantages summary), here are a few resources that might be interesting. - Design document, including architecture and tradeoffs: https://filippo.io/a-different-CT-log https://filippo.io/a-different-CT-log - Implementation: https://github.com/FiloSottile/sunlight https://github.com/FiloSottile/sunlight - API specification: https://c2sp.org/sunlight https://c2sp.org/sunlight - Website, including test logs and feedback channels: https://sunlight.dev/ https://sunlight.dev/ If you’re thinking “oh we could use something similar” please reach out! Sunlight is retrofitting some of the modern tlog designs on a legacy system. With a greenfield deployment you can do even better! I’m working with the Sigsum project on specs, tooling, and a support ecosystem to make deploying tlogs easier and safer.
  - cpach 3y ago
    Cool! tlog = transparency log, but not neccessarily for X509 certificates?
    - FiloSottile 3y ago
      Exactly! It's a growing ecosystem including things like https://transparency.dev https://transparency.dev, the Go Checksum Database, https://www.sigsum.org https://www.sigsum.org, SigStore, and even key transparency solutions like WhatsApp's. One thing you end up needing to deploy tlogs is a way to reassure clients the tree is not forked, and for that you mostly need witness cosigning, where a quorum of third parties attest that a signed tree head is consistent with all the other ones they've seen. I've worked with the Sigsum project and the Google TrustFabric team on an interoperable specification for witnessing (which Sunlight interoperates with), and I am now working to develop a public, reliable ecosystem of witnesses. Once you have witnessing, running a log can be as easy as hosting a few files in a GitHub repo or S3 bucket, updated with a batch script. I am very excited to make it possible for any project to get better-than-CT accountability for ~free. (You might want to catch my RWC 2024 talk about this once it comes out!)
simonw 3y ago
A lot of the benefits I see people proposing for blockchain systems - things like "an immutable log of real estate ownership that no-one can ever tamper with" - look to me like they would be better handled by the certificate transparency log mechanism rather than by a proof-of-work/proof-of-stake blockchain.
- deleted 3y ago
  [deleted]
- ulrikrasmussen 3y ago
  I agree. I have been toying with the idea of a system where you have a set of participants who each have responsibilities for maintaining some state that users and other participants rely on. For example, a network of banks including a central bank, but the network could also include merchants who rely on the banks for payments and trusted third parties who keep track of agreements such as loan contracts. Instead of putting everything on a blockchain, make each participant be the sole writer for one or more ledgers (e.g. one ledger per account they maintain) and use a non-repudiable two-phase commit protocol to coordinate multiple transactions performed atomically across ledgers. Using the techniques from certificate transparency, as soon as a participant has published that they have performed a transaction on their ledger, then they cannot take that back again without an auditor finding out. They also cannot branch their ledgers (i.e. double spend) as long as all participants periodically gossip received transactions to the auditors. Or rather, they CAN, but the auditor will detect it eventually and have proof of it. What you get is a sort of multi-party SWIFT-like messaging system which doesn't require banks to change their basic accounting infrastructure and tokenize everything.
- kfreds 3y ago
  Absolutely. The vast majority of such use cases are way better served by a transparency log than a blockchain. Imagine an embedded system that only accepts firmware updates that are digitally signed and discoverable in a public append-only log. Such a proof would have to be attached to the firmware update just like the digital signature, with a size constraint on the order of kilobytes. To be fair I haven't kept up-to-date with blockchains, but I find it hard to believe that any existing blockchain could solve this. Transparency logs can. I wonder when businesses that currently use blockchains as immutable logs will realise that there is a more cost-effective solution. One of the reasons it hasn't happened yet is because blockchains have magnitudes more mindshare than transparency logging. In fact, there doesn't seem to be that many people who deeply understand the design parameter space of transparency logs. Shameless plug: My colleagues and I have spent a few years designing one - Sigsum. As Filippo mentions he has contributed to its design as well.
kardianos 3y ago
Great call out to the Go checksum database. Go walks a great line between distributed system and central verification.
JanisErdmanis 3y ago
Is there a specification one can follow to implement a verifier for certificate transparency logs? I looked up RFC 6962, but it does not contain test vectors. I am particularly interested in history tree inclusion and consistency proofs and don’t care how tree root is signed.
- FiloSottile 3y ago
  We're working on it! RFC 6962 specifies inclusion and consistency proofs, but indeed it's missing test vectors. Keep an eye on https://c2sp.org https://c2sp.org and https://c2sp.org/CCTV https://c2sp.org/CCTV.
phasmantistes 3y ago
I'm super excited about Sunlight. The CT ecosystem is really fragile right now, with current log implemetations being expensive to operate and very difficult to operate correctly, as evidenced by the recent failures of multiple logs[1][2]. And if too many logs fall over, it becomes infeasible to include the requisite number of SCTs in certificates, or worse, already-issued certificates can become effectively untrusted. With Sunlight reducing costs by a couple orders or magnitude and significantly easing deployment complexity, it will be a huge boon to the whole ecosystem. I really hope log monitors begin crawling sunlight logs and browsers accept them as trusted in the near future. [1]: https://groups.google.com/a/chromium.org/g/ct-policy/c/6mvSoETm9Qk https://groups.google.com/a/chromium.org/g/ct-policy/c/6mvSo... [2]: https://groups.google.com/a/chromium.org/g/ct-policy/c/_dhkSzwoZuE https://groups.google.com/a/chromium.org/g/ct-policy/c/_dhkS...
- chtitux 3y ago
  [1]: a delete command was wrongly issued to all Cassandra servers during a planned maintenance [2]: a database server had its disk full that lead to a corrupted database SRE is tough.
- deleted 3y ago
  [deleted]
hedora 3y ago
This is probably a step forward for HTTPS, but it still bugs me that most of the internet is running on a cryptography system that doesn't provide confidentiality or integrity. Instead, it makes it possible for you to detect that someone's been using a rogue certificate after the fact. It bugs me even more that many security certification folks insist on using this junk to manage SSH keys (out of the box, SSH provides stronger security guarantees than X.509 certificates). The whole situation reminds me of Crypto AG: https://en.wikipedia.org/wiki/Crypto_AG https://en.wikipedia.org/wiki/Crypto_AG
- mcpherrinm 3y ago
  I'm biased, of course, because I work for Let's Encrypt, and spend a lot of time thinking about the trade-offs and how to make this ecosystem better. I think claiming that "SSH provides stronger security guarantees than X.509 certificates" is pretty outlandish. SSH is trust-on-first-use. How many people actually verify the key fingerprints? Maybe you do when you're sshing to your single corporate bastion, or to Github.com, but could you imagine if you had to do that for every website? It simply would not work. It only barely does for SSH because of the (relatively) low amount of use SSH gets. When at a previous company I set up an SSH CA, we found an SSH MITM within 3 days (though benign; an ISP-provided modem/router was resolving all hostnames to itself when the WAN was down), and the user was planning on just accepting the new SSH host key. Certificate Transparency is an auditing system, so of course it's after-the-fact. It has helped to improve the CA system immensely. While there's still work to do, the TLS ecosystem is unrivalled in terms of actual security benefit delivered to the most people.
  - hedora 3y ago
    With SSH, if you care, you just need to check the fingerprints. With HTTPS, you have to implement certificate pinning (or set up a private CA and configure everything to ignore the system roots of trust), which is much more onerous. Thanks for helping to improve https, since it helps in practice. edit: Another way to put it is that if you want to reduce your attack surface to just the client and server machine, it's easy enough to scale SSH fingerprint verification to a few dozen machines. It's much, much harder to get SSL to provide that property for even two machines. As for scaling to 100's of machines, it's not obvious to me that bootstrapping and rsyncing a known_hosts file and /etc/ssh_config that turns off trust on first use is harder than setting up a CA. For millions of machines, SSH falls over (because known_hosts would be too big), but at that point, you're talking about what percentage of machines are compromised in steady state, not how to keep all attackers out. That's where HTTPS works well.
harikb 3y ago
One of the folks behind this effort, Filippo (https://filippo.io/ https://filippo.io/) is now, as he calls it, a full time professional open source maintainer https://words.filippo.io/full-time-maintainer/ https://words.filippo.io/full-time-maintainer/ > That means I spend most of my time on maintenance, and I offer retainers to companies that benefit from my work and from access to my planning and my expertise. He is trying a new way of life for highly-skilled OSS maintainers. Hopefully this works out well for him.
- RIMR 3y ago
  I knew I had heard of this guy before! I enjoyed this project of his: https://filippo.io/behindthesofa/ https://filippo.io/behindthesofa/
- deleted 3y ago
  [deleted]
kfreds 3y ago
Transparency logging is a fantastic complement to public-key cryptography. A prioritised list of the most important cryptographic constructions may very well look like this: symmetric ciphers, hash functions, MACs, public-key cryptography, transparency logging. If you're interested in security and/or cryptography and you don't know what transparency logging is, you're missing out.
- deleted 3y ago
  [deleted]
gnfargbl 3y ago
The "standard" list of all known CT logs is maintained by Google at https://www.gstatic.com/ct/log_list/v3/all_logs_list.json https://www.gstatic.com/ct/log_list/v3/all_logs_list.json and follows the schema at https://www.gstatic.com/ct/log_list/v3/log_list_schema.json https://www.gstatic.com/ct/log_list/v3/log_list_schema.json. As far as I can tell, that schema currently only supports RFC6962 logs. Are there plans in train to support enumeration of Sunlight-format logs? Or am I just far too impatient?
- mcpherrinm 3y ago
  Apple also has a list at https://valid.apple.com/ct/log_list/current_log_list.json https://valid.apple.com/ct/log_list/current_log_list.json Getting them accepted as trusted lists is the goal, but how exactly that happens is not yet determined. Both the Apple and Chrome CT programs have responded to our announcement email at https://groups.google.com/a/chromium.org/g/ct-policy/c/v9JzlbphYBs https://groups.google.com/a/chromium.org/g/ct-policy/c/v9Jzl... > As the specification, implementations, and ecosystem support evolve, we look forward to being able to include Sunlight logs alongside RFC6962 logs in Chrome. While we're not quite there yet, we encourage the CT community to experiment with the prototype logs and specification as much as possible. > That said, we’re not planning on accepting new Sunlight logs quite yet in the Apple CT Program (though we hope that is the end result). Before that jump, we’d especially like to hear additional input (even if it’s just “LGTM”) from CT log Monitors and Auditors, as well as CAs and other relying parties submitting certificates to CT logs. Along with that, we invite input from additional SMEs able to perform (informal) security and risk assessments of the proposal — especially if they’re accompanied by filed issues :) The log list schema is a relatively small problem and should be easily enough to solve.
  - gnfargbl 3y ago
    Thanks for replying to what was a rather half-baked question, typed out in some excitement. I guess a more thought-through version is as follows: Currently, the CT logs with state == usable are operated by only six entities (Cloudflare, DigiCert, Google, LE, Sectigo, TrustAsia). In the earlier days of CT logs, there were a larger number of log operators. I think it's a reasonable assumption that we've ended up in the current situation at least partly because of the cost of log operation? Since Sunlight aims to drastically reduce the cost of log operation, might we expect the number of log operators to rise, and if so would there be likely changes to the ways that log operators are enumerated as "trusted", or would things likely be as they are now?
poweredger640 3y ago
I was pretty sure CT was gonna die when Nimbus went down last year. Like, if Digicert or LE's logs went down while Nimbus was down, ha ha
- mcpherrinm 3y ago
  There are six trusted operators, many with multiple logs. So there is some degree of redundancy, but this project is definitely an outcome of those same concerns.
savoyard 3y ago
Curiously, “Sunlight” was the codename for the original CT spec: https://datatracker.ietf.org/doc/draft-laurie-pki-sunlight/00/ https://datatracker.ietf.org/doc/draft-laurie-pki-sunlight/0...
- FiloSottile 3y ago
  Purely coincidental! ;-)

217 points by ben_s 3y ago

https://github.com/FiloSottile/sunlight https://github.com/FiloSottile/sunlight

I’m excited for others to see the work we’ve been doing to make CT easier to run. Also see the discussion on the ct-policy mailing list: https://groups.google.com/a/chromium.org/g/ct-policy/c/v9JzlbphYBs https://groups.google.com/a/chromium.org/g/ct-policy/c/v9Jzl... (I am the author of the blog post)

FiloSottile 3y ago
This is one of the projects I've been most excited about in the last few years. It let me backport to Certificate Transparency some of the modern designs ideas that came after it. Beyond the Let's Encrypt announcement and the ct-policy thread (which includes a technical and advantages summary), here are a few resources that might be interesting. - Design document, including architecture and tradeoffs: https://filippo.io/a-different-CT-log https://filippo.io/a-different-CT-log - Implementation: https://github.com/FiloSottile/sunlight https://github.com/FiloSottile/sunlight - API specification: https://c2sp.org/sunlight https://c2sp.org/sunlight - Website, including test logs and feedback channels: https://sunlight.dev/ https://sunlight.dev/ If you’re thinking “oh we could use something similar” please reach out! Sunlight is retrofitting some of the modern tlog designs on a legacy system. With a greenfield deployment you can do even better! I’m working with the Sigsum project on specs, tooling, and a support ecosystem to make deploying tlogs easier and safer.
- cpach 3y ago
  Cool! tlog = transparency log, but not neccessarily for X509 certificates?
  - FiloSottile 3y ago
    Exactly! It's a growing ecosystem including things like https://transparency.dev https://transparency.dev, the Go Checksum Database, https://www.sigsum.org https://www.sigsum.org, SigStore, and even key transparency solutions like WhatsApp's. One thing you end up needing to deploy tlogs is a way to reassure clients the tree is not forked, and for that you mostly need witness cosigning, where a quorum of third parties attest that a signed tree head is consistent with all the other ones they've seen. I've worked with the Sigsum project and the Google TrustFabric team on an interoperable specification for witnessing (which Sunlight interoperates with), and I am now working to develop a public, reliable ecosystem of witnesses. Once you have witnessing, running a log can be as easy as hosting a few files in a GitHub repo or S3 bucket, updated with a batch script. I am very excited to make it possible for any project to get better-than-CT accountability for ~free. (You might want to catch my RWC 2024 talk about this once it comes out!)

simonw 3y ago

A lot of the benefits I see people proposing for blockchain systems - things like "an immutable log of real estate ownership that no-one can ever tamper with" - look to me like they would be better handled by the certificate transparency log mechanism rather than by a proof-of-work/proof-of-stake blockchain.

deleted 3y ago
[deleted]
ulrikrasmussen 3y ago
I agree. I have been toying with the idea of a system where you have a set of participants who each have responsibilities for maintaining some state that users and other participants rely on. For example, a network of banks including a central bank, but the network could also include merchants who rely on the banks for payments and trusted third parties who keep track of agreements such as loan contracts. Instead of putting everything on a blockchain, make each participant be the sole writer for one or more ledgers (e.g. one ledger per account they maintain) and use a non-repudiable two-phase commit protocol to coordinate multiple transactions performed atomically across ledgers. Using the techniques from certificate transparency, as soon as a participant has published that they have performed a transaction on their ledger, then they cannot take that back again without an auditor finding out. They also cannot branch their ledgers (i.e. double spend) as long as all participants periodically gossip received transactions to the auditors. Or rather, they CAN, but the auditor will detect it eventually and have proof of it. What you get is a sort of multi-party SWIFT-like messaging system which doesn't require banks to change their basic accounting infrastructure and tokenize everything.
kfreds 3y ago
Absolutely. The vast majority of such use cases are way better served by a transparency log than a blockchain. Imagine an embedded system that only accepts firmware updates that are digitally signed and discoverable in a public append-only log. Such a proof would have to be attached to the firmware update just like the digital signature, with a size constraint on the order of kilobytes. To be fair I haven't kept up-to-date with blockchains, but I find it hard to believe that any existing blockchain could solve this. Transparency logs can. I wonder when businesses that currently use blockchains as immutable logs will realise that there is a more cost-effective solution. One of the reasons it hasn't happened yet is because blockchains have magnitudes more mindshare than transparency logging. In fact, there doesn't seem to be that many people who deeply understand the design parameter space of transparency logs. Shameless plug: My colleagues and I have spent a few years designing one - Sigsum. As Filippo mentions he has contributed to its design as well.

kardianos 3y ago

Great call out to the Go checksum database. Go walks a great line between distributed system and central verification.

JanisErdmanis 3y ago

Is there a specification one can follow to implement a verifier for certificate transparency logs? I looked up RFC 6962, but it does not contain test vectors. I am particularly interested in history tree inclusion and consistency proofs and don’t care how tree root is signed.

FiloSottile 3y ago
We're working on it! RFC 6962 specifies inclusion and consistency proofs, but indeed it's missing test vectors. Keep an eye on https://c2sp.org https://c2sp.org and https://c2sp.org/CCTV https://c2sp.org/CCTV.

phasmantistes 3y ago

I'm super excited about Sunlight. The CT ecosystem is really fragile right now, with current log implemetations being expensive to operate and very difficult to operate correctly, as evidenced by the recent failures of multiple logs[1][2]. And if too many logs fall over, it becomes infeasible to include the requisite number of SCTs in certificates, or worse, already-issued certificates can become effectively untrusted. With Sunlight reducing costs by a couple orders or magnitude and significantly easing deployment complexity, it will be a huge boon to the whole ecosystem. I really hope log monitors begin crawling sunlight logs and browsers accept them as trusted in the near future. [1]: https://groups.google.com/a/chromium.org/g/ct-policy/c/6mvSoETm9Qk https://groups.google.com/a/chromium.org/g/ct-policy/c/6mvSo... [2]: https://groups.google.com/a/chromium.org/g/ct-policy/c/_dhkSzwoZuE https://groups.google.com/a/chromium.org/g/ct-policy/c/_dhkS...

chtitux 3y ago
[1]: a delete command was wrongly issued to all Cassandra servers during a planned maintenance [2]: a database server had its disk full that lead to a corrupted database SRE is tough.
deleted 3y ago
[deleted]

hedora 3y ago

This is probably a step forward for HTTPS, but it still bugs me that most of the internet is running on a cryptography system that doesn't provide confidentiality or integrity. Instead, it makes it possible for you to detect that someone's been using a rogue certificate after the fact. It bugs me even more that many security certification folks insist on using this junk to manage SSH keys (out of the box, SSH provides stronger security guarantees than X.509 certificates). The whole situation reminds me of Crypto AG: https://en.wikipedia.org/wiki/Crypto_AG https://en.wikipedia.org/wiki/Crypto_AG

mcpherrinm 3y ago
I'm biased, of course, because I work for Let's Encrypt, and spend a lot of time thinking about the trade-offs and how to make this ecosystem better. I think claiming that "SSH provides stronger security guarantees than X.509 certificates" is pretty outlandish. SSH is trust-on-first-use. How many people actually verify the key fingerprints? Maybe you do when you're sshing to your single corporate bastion, or to Github.com, but could you imagine if you had to do that for every website? It simply would not work. It only barely does for SSH because of the (relatively) low amount of use SSH gets. When at a previous company I set up an SSH CA, we found an SSH MITM within 3 days (though benign; an ISP-provided modem/router was resolving all hostnames to itself when the WAN was down), and the user was planning on just accepting the new SSH host key. Certificate Transparency is an auditing system, so of course it's after-the-fact. It has helped to improve the CA system immensely. While there's still work to do, the TLS ecosystem is unrivalled in terms of actual security benefit delivered to the most people.
- hedora 3y ago
  With SSH, if you care, you just need to check the fingerprints. With HTTPS, you have to implement certificate pinning (or set up a private CA and configure everything to ignore the system roots of trust), which is much more onerous. Thanks for helping to improve https, since it helps in practice. edit: Another way to put it is that if you want to reduce your attack surface to just the client and server machine, it's easy enough to scale SSH fingerprint verification to a few dozen machines. It's much, much harder to get SSL to provide that property for even two machines. As for scaling to 100's of machines, it's not obvious to me that bootstrapping and rsyncing a known_hosts file and /etc/ssh_config that turns off trust on first use is harder than setting up a CA. For millions of machines, SSH falls over (because known_hosts would be too big), but at that point, you're talking about what percentage of machines are compromised in steady state, not how to keep all attackers out. That's where HTTPS works well.

harikb 3y ago

One of the folks behind this effort, Filippo (https://filippo.io/ https://filippo.io/) is now, as he calls it, a full time professional open source maintainer https://words.filippo.io/full-time-maintainer/ https://words.filippo.io/full-time-maintainer/ > That means I spend most of my time on maintenance, and I offer retainers to companies that benefit from my work and from access to my planning and my expertise. He is trying a new way of life for highly-skilled OSS maintainers. Hopefully this works out well for him.

RIMR 3y ago
I knew I had heard of this guy before! I enjoyed this project of his: https://filippo.io/behindthesofa/ https://filippo.io/behindthesofa/
deleted 3y ago
[deleted]

kfreds 3y ago

Transparency logging is a fantastic complement to public-key cryptography. A prioritised list of the most important cryptographic constructions may very well look like this: symmetric ciphers, hash functions, MACs, public-key cryptography, transparency logging. If you're interested in security and/or cryptography and you don't know what transparency logging is, you're missing out.

deleted 3y ago
[deleted]

gnfargbl 3y ago

The "standard" list of all known CT logs is maintained by Google at https://www.gstatic.com/ct/log_list/v3/all_logs_list.json https://www.gstatic.com/ct/log_list/v3/all_logs_list.json and follows the schema at https://www.gstatic.com/ct/log_list/v3/log_list_schema.json https://www.gstatic.com/ct/log_list/v3/log_list_schema.json. As far as I can tell, that schema currently only supports RFC6962 logs. Are there plans in train to support enumeration of Sunlight-format logs? Or am I just far too impatient?

mcpherrinm 3y ago
Apple also has a list at https://valid.apple.com/ct/log_list/current_log_list.json https://valid.apple.com/ct/log_list/current_log_list.json Getting them accepted as trusted lists is the goal, but how exactly that happens is not yet determined. Both the Apple and Chrome CT programs have responded to our announcement email at https://groups.google.com/a/chromium.org/g/ct-policy/c/v9JzlbphYBs https://groups.google.com/a/chromium.org/g/ct-policy/c/v9Jzl... > As the specification, implementations, and ecosystem support evolve, we look forward to being able to include Sunlight logs alongside RFC6962 logs in Chrome. While we're not quite there yet, we encourage the CT community to experiment with the prototype logs and specification as much as possible. > That said, we’re not planning on accepting new Sunlight logs quite yet in the Apple CT Program (though we hope that is the end result). Before that jump, we’d especially like to hear additional input (even if it’s just “LGTM”) from CT log Monitors and Auditors, as well as CAs and other relying parties submitting certificates to CT logs. Along with that, we invite input from additional SMEs able to perform (informal) security and risk assessments of the proposal — especially if they’re accompanied by filed issues :) The log list schema is a relatively small problem and should be easily enough to solve.
- gnfargbl 3y ago
  Thanks for replying to what was a rather half-baked question, typed out in some excitement. I guess a more thought-through version is as follows: Currently, the CT logs with state == usable are operated by only six entities (Cloudflare, DigiCert, Google, LE, Sectigo, TrustAsia). In the earlier days of CT logs, there were a larger number of log operators. I think it's a reasonable assumption that we've ended up in the current situation at least partly because of the cost of log operation? Since Sunlight aims to drastically reduce the cost of log operation, might we expect the number of log operators to rise, and if so would there be likely changes to the ways that log operators are enumerated as "trusted", or would things likely be as they are now?

poweredger640 3y ago

I was pretty sure CT was gonna die when Nimbus went down last year. Like, if Digicert or LE's logs went down while Nimbus was down, ha ha

mcpherrinm 3y ago
There are six trusted operators, many with multiple logs. So there is some degree of redundancy, but this project is definitely an outcome of those same concerns.

savoyard 3y ago

Curiously, “Sunlight” was the codename for the original CT spec: https://datatracker.ietf.org/doc/draft-laurie-pki-sunlight/00/ https://datatracker.ietf.org/doc/draft-laurie-pki-sunlight/0...

FiloSottile 3y ago
Purely coincidental! ;-)