6 ms·

Show HN: CertStream – See SSL certs as they're issued in real time

142 points by zer01 9y ago

sheltowt 9y ago
Looks great Ryan!
zer01 9y ago
Hey folks, developer of CertStream here. You can read more about the motivations and implementation behind this project by visiting the announcement page (https://medium.com/cali-dog-security/introducing-certstream-3fc13bb98067 https://medium.com/cali-dog-security/introducing-certstream-...) on my company's blog. I'm also happy to field any questions anyone may have!
- kbody 9y ago
  Interesting project. I was wondering if you think they are any privacy issues around Certificate Transparency, like grouping ownership of domains through the timings.
  - zer01 9y ago
    Hmm, that's an interesting thing I haven't given much thought to. I think that it would be somewhat difficult to pull off a correlation attack/leakage as the CTLs tend to dump in batches vs every poll returning new results, but I think once you remove a lot of the noise (cloudflare SNI certs, testing domains, etc) it'd potentially show some interesting patterns. https://github.com/CaliDog/certstream-python/blob/master/examples/stat_windows.py https://github.com/CaliDog/certstream-python/blob/master/exa... This demo would be a decent starting point to that analysis if you'd be interested in toying with it!
- tty7 9y ago
  Nice work! Congrats How are you dealing with googles 60min-ish flushes of their log?
waibelp 9y ago
Sweet landingpage!
- zer01 9y ago
  Thanks! It's a few iterations in, and my designer (http://www.jweiller.com/ http://www.jweiller.com/) was definitely responsible for the best looking parts!
  - finnn 9y ago
    heads up the <title> isn't getting set, it just shows the URL.
    - zer01 9y ago
      Thanks for the heads up! It also isn't displaying the favicon properly it seems :-/. https://media1.giphy.com/media/R54jhpzpARmVy/giphy.gif https://media1.giphy.com/media/R54jhpzpARmVy/giphy.gif TL;DR - Curse you Webpack! ::shakes fist::
mastax 9y ago
There seems to be a big list of domains streaming through in alphabetical order. First there were dozens all starting with J, now there are dozens starting with K. Looks like gemalto is going through the day's domains in alphabetical order and adding them one at a time?
- zer01 9y ago
  Yeah, it's interesting to see how certs are issued from the larger lists, since they tend to come in a deluge. This goes doubly for the cloudflare SNI certificates for their edge nodes!
y0ghur7_xxx 9y ago
A bit OT, but there is a thing about CT that I keep asking me and I can't seem to find an answer: Say I use a letsencrypt certificate for my nextcloud installation on my home server. Now say that the german police is investigating on me. They call up the BSI and tell them "hey, can you please issue me a signed cert for nextcloud.y0ghur7.xxx so we can mitm this guy? And please, don't log it to the CT, so nobody will ever know" I will probably never know, because BSI is a trusted CA in all browsers and other http clients so they don't complain (and I am not looking at what certificate the server sent me every time I open my nc page), and nobody else will ever know that that certificate was released. Am I right? So what does CT buy me?
- bouk 9y ago
  At some point browsers will stop allowing certificates that are not logged through CT
  - zer01 9y ago
    This is (imo) the most correct response to this query. The idea behind the CT lists is that browsers will flip into a "if it's no publicly identifiable, it's not valid" mode. The Coinbase Engineering blog actually just put out an article on this very topic today - https://engineering.coinbase.com/moving-to-expect-ct-d0d26a07046d https://engineering.coinbase.com/moving-to-expect-ct-d0d26a0....
  - y0ghur7_xxx 9y ago
    > At some point browsers will stop allowing certificates that are not logged through CT Makes sense. So to be sure nobody issued a cert for one of my properties I would have to check regularly on CT logs to be sure that only certs requested by me are issued. But in that case, if someone requests a cert for one of my properties, and that cert was not requested by me, what do I do? Do I tell mozilla and google that "someone issued cert id 4d8effdd25 for my nextcloud installation (or my forum where some rebellious users meet up sometimes) to mitm me, but it was not me". Will they belive me? And it will be probably to late anyway, because propagation to a CT log can take up to one day, so they got data on all the traffic for a whole day.
andrewstuart2 9y ago
Interesting timing, considering the talk [0] on this very topic just uploaded to YouTube yesterday morning from DefCon 25. Basically, this is offering his observation (CTL can be used to get a real-time list of new domain names, which can be exploited), as a service. Seems like Hanno Bõck could at least use a shout out if it was related to his work. Either way, the talk is worth a watch. [0] https://www.youtube.com/watch?v=TMNeSnjZfCI&list=PL9fPq3eQfaaDEQqRyr-0cmF1HTuidm97j&index=11 https://www.youtube.com/watch?v=TMNeSnjZfCI&list=PL9fPq3eQfa...
- zer01 9y ago
  Interesting! I haven't been able to attend Defcon in the past few years so I haven't seen that talk (or heard of his research), but it's something I've thought about for a while now - using CTLs as a means to jump into the early process of setting up webapps and whatnot. Thanks for the video!
- tty7 9y ago
  Shout out isnt needed, i did the same thing as Hanno over a weekend early this year. Been kicking myself since defcon that i didn't submit a talk! Anyone who reads the certificate transparency log rfc can quickly realize whats possible. I've also been following calidog since his first medium post, ive got my own similar cert scanner/tracker.
mschuster91 9y ago
One thing I totally dislike with CT is that literally everybody can see all the subdomains that my certificate is valid for (esp. LetsEncrypt), but also for cases where your "normal" wildcard-cert does not work - e.g. .foo.de is covered, but because wildcards dont go beyond 1 level, .bar.foo.de is not covered, and so everyone can see that there is one (or more) subdomains at bar.foo.de. Let's assume an attacker finds a RCE in JIRA, Confluence or Gitlab... now everything the attacker has to do to find a list of candidates is to run a simple grep -i gitlab|jira|confluence|whatever on the CT logs, while he'd have to go the brute-force route before CT.
- quesera 9y ago
  > grep -i gitlab|jira|confluence|whatever Don't name hosts after products! But more generally, you should never expose Confluence etc to the greater internet. That's just asking for trouble. It will find you, greppable hostname or not.
- Operyl 9y ago
  Soooo, in most circumstances these products are set to be allowed to be crawled by robots for search engines. I think it’d be a hell of a lot more accurate and powerful to use Google with keywords (like powered by lines) than say have to get into possibly vague hostnames (like docs, bugs, issues, git etc).
- tscs37 9y ago
  You should not expect a service which has been exposed to the open internet to be hidden. If your last defense is that nobody knows the domainname, then you've lost. Not knowing the domainname shouldn't be any defense at all.
  - mschuster91 9y ago
    > You should not expect a service which has been exposed to the open internet to be hidden. Of course not, but CT dramatically lowers the bar for attackers. That's what I mean.
wybiral 9y ago
Very cool! I love the simple JSON-over-WebSockets API and would love to see more streaming APIs like this available in the wild.
tscs37 9y ago
This is quite neat. I'll probably look at the go-library to build something out of that. In the meanwhile, crt.sh also offers a Atom feed of issued certificates. Definitely not realtime but also works.

142 points by zer01 9y ago

Looks great Ryan!

Hey folks, developer of CertStream here. You can read more about the motivations and implementation behind this project by visiting the announcement page (https://medium.com/cali-dog-security/introducing-certstream-3fc13bb98067 https://medium.com/cali-dog-security/introducing-certstream-...) on my company's blog. I'm also happy to field any questions anyone may have!

kbody 9y ago
Interesting project. I was wondering if you think they are any privacy issues around Certificate Transparency, like grouping ownership of domains through the timings.
- zer01 9y ago
  Hmm, that's an interesting thing I haven't given much thought to. I think that it would be somewhat difficult to pull off a correlation attack/leakage as the CTLs tend to dump in batches vs every poll returning new results, but I think once you remove a lot of the noise (cloudflare SNI certs, testing domains, etc) it'd potentially show some interesting patterns. https://github.com/CaliDog/certstream-python/blob/master/examples/stat_windows.py https://github.com/CaliDog/certstream-python/blob/master/exa... This demo would be a decent starting point to that analysis if you'd be interested in toying with it!
tty7 9y ago
Nice work! Congrats How are you dealing with googles 60min-ish flushes of their log?

waibelp 9y ago

Sweet landingpage!

zer01 9y ago
Thanks! It's a few iterations in, and my designer (http://www.jweiller.com/ http://www.jweiller.com/) was definitely responsible for the best looking parts!
- finnn 9y ago
  heads up the <title> isn't getting set, it just shows the URL.
  - zer01 9y ago
    Thanks for the heads up! It also isn't displaying the favicon properly it seems :-/. https://media1.giphy.com/media/R54jhpzpARmVy/giphy.gif https://media1.giphy.com/media/R54jhpzpARmVy/giphy.gif TL;DR - Curse you Webpack! ::shakes fist::

mastax 9y ago

There seems to be a big list of domains streaming through in alphabetical order. First there were dozens all starting with J, now there are dozens starting with K. Looks like gemalto is going through the day's domains in alphabetical order and adding them one at a time?

zer01 9y ago
Yeah, it's interesting to see how certs are issued from the larger lists, since they tend to come in a deluge. This goes doubly for the cloudflare SNI certificates for their edge nodes!

y0ghur7_xxx 9y ago

A bit OT, but there is a thing about CT that I keep asking me and I can't seem to find an answer: Say I use a letsencrypt certificate for my nextcloud installation on my home server. Now say that the german police is investigating on me. They call up the BSI and tell them "hey, can you please issue me a signed cert for nextcloud.y0ghur7.xxx so we can mitm this guy? And please, don't log it to the CT, so nobody will ever know" I will probably never know, because BSI is a trusted CA in all browsers and other http clients so they don't complain (and I am not looking at what certificate the server sent me every time I open my nc page), and nobody else will ever know that that certificate was released. Am I right? So what does CT buy me?

bouk 9y ago
At some point browsers will stop allowing certificates that are not logged through CT
- zer01 9y ago
  This is (imo) the most correct response to this query. The idea behind the CT lists is that browsers will flip into a "if it's no publicly identifiable, it's not valid" mode. The Coinbase Engineering blog actually just put out an article on this very topic today - https://engineering.coinbase.com/moving-to-expect-ct-d0d26a07046d https://engineering.coinbase.com/moving-to-expect-ct-d0d26a0....
- y0ghur7_xxx 9y ago
  > At some point browsers will stop allowing certificates that are not logged through CT Makes sense. So to be sure nobody issued a cert for one of my properties I would have to check regularly on CT logs to be sure that only certs requested by me are issued. But in that case, if someone requests a cert for one of my properties, and that cert was not requested by me, what do I do? Do I tell mozilla and google that "someone issued cert id 4d8effdd25 for my nextcloud installation (or my forum where some rebellious users meet up sometimes) to mitm me, but it was not me". Will they belive me? And it will be probably to late anyway, because propagation to a CT log can take up to one day, so they got data on all the traffic for a whole day.

andrewstuart2 9y ago

Interesting timing, considering the talk [0] on this very topic just uploaded to YouTube yesterday morning from DefCon 25. Basically, this is offering his observation (CTL can be used to get a real-time list of new domain names, which can be exploited), as a service. Seems like Hanno Bõck could at least use a shout out if it was related to his work. Either way, the talk is worth a watch. [0] https://www.youtube.com/watch?v=TMNeSnjZfCI&list=PL9fPq3eQfaaDEQqRyr-0cmF1HTuidm97j&index=11 https://www.youtube.com/watch?v=TMNeSnjZfCI&list=PL9fPq3eQfa...

zer01 9y ago
Interesting! I haven't been able to attend Defcon in the past few years so I haven't seen that talk (or heard of his research), but it's something I've thought about for a while now - using CTLs as a means to jump into the early process of setting up webapps and whatnot. Thanks for the video!
tty7 9y ago
Shout out isnt needed, i did the same thing as Hanno over a weekend early this year. Been kicking myself since defcon that i didn't submit a talk! Anyone who reads the certificate transparency log rfc can quickly realize whats possible. I've also been following calidog since his first medium post, ive got my own similar cert scanner/tracker.

mschuster91 9y ago

One thing I totally dislike with CT is that literally everybody can see all the subdomains that my certificate is valid for (esp. LetsEncrypt), but also for cases where your "normal" wildcard-cert does not work - e.g. .foo.de is covered, but because wildcards dont go beyond 1 level, .bar.foo.de is not covered, and so everyone can see that there is one (or more) subdomains at bar.foo.de. Let's assume an attacker finds a RCE in JIRA, Confluence or Gitlab... now everything the attacker has to do to find a list of candidates is to run a simple grep -i gitlab|jira|confluence|whatever on the CT logs, while he'd have to go the brute-force route before CT.

quesera 9y ago
> grep -i gitlab|jira|confluence|whatever Don't name hosts after products! But more generally, you should never expose Confluence etc to the greater internet. That's just asking for trouble. It will find you, greppable hostname or not.
Operyl 9y ago
Soooo, in most circumstances these products are set to be allowed to be crawled by robots for search engines. I think it’d be a hell of a lot more accurate and powerful to use Google with keywords (like powered by lines) than say have to get into possibly vague hostnames (like docs, bugs, issues, git etc).
tscs37 9y ago
You should not expect a service which has been exposed to the open internet to be hidden. If your last defense is that nobody knows the domainname, then you've lost. Not knowing the domainname shouldn't be any defense at all.
- mschuster91 9y ago
  > You should not expect a service which has been exposed to the open internet to be hidden. Of course not, but CT dramatically lowers the bar for attackers. That's what I mean.

wybiral 9y ago

Very cool! I love the simple JSON-over-WebSockets API and would love to see more streaming APIs like this available in the wild.

tscs37 9y ago

This is quite neat. I'll probably look at the go-library to build something out of that. In the meanwhile, crt.sh also offers a Atom feed of issued certificates. Definitely not realtime but also works.