4 ms·

Shameless plug: Check out my Certificate Transparency monitor at https://www.merklemap.com https://www.merklemap.com The scale is massive, I just crossed 100B

by Eikon 2y ago

Shameless plug: Check out my Certificate Transparency monitor at https://www.merklemap.com https://www.merklemap.com The scale is massive, I just crossed 100B rows in the main database! :)

AznHisoka 2y ago
Why do it only show a few subdomains for .statuspage.io? I would have expected at least 10K or so. https://www.merklemap.com/search?query=*.statuspage.io&page=0 https://www.merklemap.com/search?query=*.statuspage.io&page=... Is my query wrong or are you just showing less results intentionally if you’re not paying?
- Eikon 2y ago
  > Why do it only show a few subdomains for .statuspage.io? I would have expected at least 10K or so. https://www.merklemap.com/search?query=*.statuspage.io&page= https://www.merklemap.com/search?query=*.statuspage.io&page=... Because they have a wildcard for *.statuspage.io, which they are probably hosting their pages on. > Is my query wrong or are you just showing less results intentionally if you’re not paying? No, results are the same but not sorted.
gessha 2y ago
This is really cool! It discovered even subdomains that lived for a few days on my site. If it’s not a secret, how do you discover those? Is it by listening to DNS record changes?
- Eikon 2y ago
  By using... certificate transparency logs. https://www.merklemap.com/documentation/how-it-works https://www.merklemap.com/documentation/how-it-works
tgsovlerkhgsel 2y ago
Are you continuously monitoring consistency proofs? Or in other words, would someone (you or someone else) actually notice if a log changed its contents retroactively?
- Eikon 2y ago
  Not yet, but that’s definitely the short term plan!
Etheryte 2y ago
I'm clearly not the target audience for this, so excuse me if this is a dumb question: what is this tool used for? Who would usually use it and for what purpose?
- Eikon 2y ago
  Anyone setting up infrastructure, security researchers, security teams and IT teams. It’s also actually very useful too in the brand management field, especially to detect phishing websites.
immibis 2y ago
I tried to do something like this one time and had a problem just finding the logs. All information on the internet points to the fact that certain logs exist, but not how to access them. Are they not public access? Do you have a B2B relationship with the companies like Cloudflare that run logs?
- tialaramex 2y ago
  They're required to be public services. https://crt.sh/monitored-logs https://crt.sh/monitored-logs is the list of logs monitored by crt.sh (a public log monitor operated by Sectigo, a commercial CA) if that helps. Each of the major browsers also publishes which logs they trust and provides information about e.g. distrust of logs. Is the problem that you couldn't figure out how to use a log? It doesn't just have a web site where you can type in searches you need to be able to use their web API as defined in the protocol documentation.
antonios 2y ago
That's interesting, can you share more information about your tech stack?
- Eikon 2y ago
  Merklemap is running PostgreSQL as the primary database, currently scaling at ~18TB on NVMe storage, and around 30TB of actual certificates that are stored on s3. The backend is implemented in Rust (handling web services, search functionality, and data ingestion pipelines). The frontend is built with Next.js.
ambigious7777 2y ago
Why not use something like crt.sh?
- Eikon 2y ago
  Why not use something like Merklemap?

by Eikon 2y ago

Shameless plug: Check out my Certificate Transparency monitor at https://www.merklemap.com https://www.merklemap.com The scale is massive, I just crossed 100B rows in the main database! :)

AznHisoka 2y ago

Why do it only show a few subdomains for .statuspage.io? I would have expected at least 10K or so. https://www.merklemap.com/search?query=*.statuspage.io&page=0 https://www.merklemap.com/search?query=*.statuspage.io&page=... Is my query wrong or are you just showing less results intentionally if you’re not paying?

Eikon 2y ago
> Why do it only show a few subdomains for .statuspage.io? I would have expected at least 10K or so. https://www.merklemap.com/search?query=*.statuspage.io&page= https://www.merklemap.com/search?query=*.statuspage.io&page=... Because they have a wildcard for *.statuspage.io, which they are probably hosting their pages on. > Is my query wrong or are you just showing less results intentionally if you’re not paying? No, results are the same but not sorted.

gessha 2y ago

This is really cool! It discovered even subdomains that lived for a few days on my site. If it’s not a secret, how do you discover those? Is it by listening to DNS record changes?

Eikon 2y ago
By using... certificate transparency logs. https://www.merklemap.com/documentation/how-it-works https://www.merklemap.com/documentation/how-it-works

tgsovlerkhgsel 2y ago

Are you continuously monitoring consistency proofs? Or in other words, would someone (you or someone else) actually notice if a log changed its contents retroactively?

Eikon 2y ago
Not yet, but that’s definitely the short term plan!

Etheryte 2y ago

I'm clearly not the target audience for this, so excuse me if this is a dumb question: what is this tool used for? Who would usually use it and for what purpose?

Eikon 2y ago
Anyone setting up infrastructure, security researchers, security teams and IT teams. It’s also actually very useful too in the brand management field, especially to detect phishing websites.

immibis 2y ago

I tried to do something like this one time and had a problem just finding the logs. All information on the internet points to the fact that certain logs exist, but not how to access them. Are they not public access? Do you have a B2B relationship with the companies like Cloudflare that run logs?

tialaramex 2y ago
They're required to be public services. https://crt.sh/monitored-logs https://crt.sh/monitored-logs is the list of logs monitored by crt.sh (a public log monitor operated by Sectigo, a commercial CA) if that helps. Each of the major browsers also publishes which logs they trust and provides information about e.g. distrust of logs. Is the problem that you couldn't figure out how to use a log? It doesn't just have a web site where you can type in searches you need to be able to use their web API as defined in the protocol documentation.

antonios 2y ago

That's interesting, can you share more information about your tech stack?

Eikon 2y ago
Merklemap is running PostgreSQL as the primary database, currently scaling at ~18TB on NVMe storage, and around 30TB of actual certificates that are stored on s3. The backend is implemented in Rust (handling web services, search functionality, and data ingestion pipelines). The frontend is built with Next.js.

ambigious7777 2y ago

Why not use something like crt.sh?

Eikon 2y ago
Why not use something like Merklemap?