5 ms·
Thousands of systems, from Google to script kiddies to OpenAI to nigerian call scammers to cybersecurity firms, actively watch the certificate transparency logs
by 827a 10mo ago
Thousands of systems, from Google to script kiddies to OpenAI to nigerian call scammers to cybersecurity firms, actively watch the certificate transparency logs for exactly this reason. Yawn.
- H8crilA 10mo agoFor those that never looked at the CT logs: https://crt.sh/?q=ycombinator.com https://crt.sh/?q=ycombinator.com (the site may occasionally fail to load)
- Eikon 10mo agoShameless plug :) https://www.merklemap.com/search?query=ycombinator.com&page=0 https://www.merklemap.com/search?query=ycombinator.com&page=... Entries are indexed by subdomain instead of by certificate (click an entry to see all certificates for that subdomain). Also, you can search for any substring (that was quite the journey to implement so it's fast enough across almost 5B entries): https://www.merklemap.com/search?query=ycombi&page=0 https://www.merklemap.com/search?query=ycombi&page=0
- chuckadams 10mo agoAny insights you can share on how you made search so fast? What kind of resources does it take to implement it?
- deleted 10mo ago[deleted]
- Eikon 10mo agoMost of merklemap is stored on ZeroFS [0] and thus allows to scale IO ressources quite crazily :) [0] https://github.com/Barre/ZeroFS https://github.com/Barre/ZeroFS
- jddj 10mo ago> Watch Ubuntu boot from ZeroFS Love it
- rendaw 10mo agoHow does ZeroFS handle consistency with writes?
- Eikon 10mo agoIf you use 9P or NBD it handles fsync as expected. With NFS, it's time based. https://github.com/Barre/ZeroFS#9p-recommended-for-better-performance-and-fsync-posix-semantics https://github.com/Barre/ZeroFS#9p-recommended-for-better-pe...
- rendaw 10mo agoOh awesome! I was searching for consistency, but I guess durability is the word used for filesystems. Thanks!
- nerdsniper 10mo agoThank you!!! Needed exactly this at work.
- Eikon 10mo agoGlad it was helpful!
- TacticalCoder 10mo agoNot 100% related but not 100% not-related either: I've got a script that generates variations of the domain names I use the most... All the most common typos/mispelling, all the "1337" variations, all the Levenhstein edit distance of 1, quite some of the 2, etc. For example for "lillybank.com", I'll generate: llllybank.com liliybank.com ... and countless others. Hundreds of thousands of entries. They then are null-routed from my unbound DNS resolver. My browsers are forced into "corporate" settings where they cannot use DoH/DoT: it's all, between my browsers and my unbound resolver, in the clear. All DNS UDP traffic that contains any Unicode domain name is blocked by the firewall. No DNS over TCP is allowed (and, no, I don't care). I also block entire countries' TLD as well as entire countries' IP blocks. Been running a setup like that (and many killfiles, and DNS resolvers known to block all known porn and know malware sites etc.) since years now already. The Internet keeps working fine.
- rendaw 10mo agoThe first page of results doesn't include ycombinator.com. I get `app.baby-ycombinator.com`, `ycombinator.comchat.com`, everything in between. Substring doesn't seem like what I'd want in a subdomain search.
- Eikon 10mo ago> Substring doesn't seem like what I'd want in a subdomain search. Well, if you want only subdomains search for *.ycombinator.com. https://www.merklemap.com/search?query=*.ycombinator.com&page=0 https://www.merklemap.com/search?query=*.ycombinator.com&pag...
- 1vuio0pswjnm7 10mo agoConsidering how it must be getting hammered what with the "AI" nonsense, it's interesting how crt.sh continues to remain usable, particularly the (limited) direct PostgresSQL db access To me, this is evidence that SQL databases with high traffic can be made directly accessible on the public internet crt.sh seems to be more accessible at certain times of the day. I can remember when it had no such accessibility issues
- miki123211 10mo agoIt is not usable. It's the only website I know of where queries can just randomly fail for no reason, and they don't even have an automatic retry mechanism. Even the worst enterprise nightmares I've seen weren't this user unfriendly.
- pavel_lishin 10mo agoWhat's the yawn for?
- xpe 10mo agoPresumably this is well-known among people that already know about this. P.S. In the hopes of making this more than just a sarcastic comment, the question of "How do people bootstrap knowledge?" is kind of interesting. [1] > To tackle a hard problem, it is often wise to reuse and recombine existing knowledge. Such an ability to bootstrap enables us to grow rich mental concepts despite limited cognitive resources. Here we present a computational model of conceptual bootstrapping. This model uses a dynamic conceptual repertoire that can cache and later reuse elements of earlier insights in principled ways, modelling learning as a series of compositional generalizations. This model predicts systematically different learned concepts when the same evidence is processed in different orders, without any extra assumptions about previous beliefs or background knowledge. Across four behavioural experiments (total n = 570), we demonstrate strong curriculum-order and conceptual garden-pathing effects that closely resemble our model predictions and differ from those of alternative accounts. Taken together, this work offers a computational account of how past experiences shape future conceptual discoveries and showcases the importance of curriculum design in human inductive concept inferences. [1]: https://www.nature.com/articles/s41562-023-01719-1 https://www.nature.com/articles/s41562-023-01719-1
- jfindper 10mo agoIt implies that this is boring and not article/post-worthy (which I agree with). Certificate transparency logs are intended to be consumed by others. That is indeed what is happening. Not interesting.
- pavel_lishin 10mo ago> It implies that this is boring and not article/post-worthy (which I agree with). It's certainly news to me, and presumably some others, that this exists.
- 10mo ago
- irishcoffee 10mo ago[flagged]
- edvinbesic 10mo agoYou are implying that a law is being broken, but isn't this the equivalent of going to city hall to pull public land records?
- formerly_proven 10mo agoThe whole point of CT logs is to make issuance of certificates in the public WebPKI… public.
- tsimionescu 10mo agoThe whole point of the CT logs is to be a public list of all domains which have TLS certs issued by the Web PKI. People are reading this list. I really don't see what is either surprising or in any way problematic in doing so.
- jfindper 10mo agoThe intended purpose of certificate transparency logs is to be viewed by others! Perhaps you should save your "gross" judgement for when you better understand what's happening?
- ekr____ 10mo agoWith that said, given that (1) pre-certificates in the log are big and (2) lifetimes are shortening and so there will be a lot of duplicates, it seems like it would be good for someone to make a feed that was just new domain names.
- 827a 10mo agoThese exist for apex domains; the real use-case is subdomains.
- ekr____ 10mo agoSure, but the subdomains will be duplicated for the same reasons.
- Eikon 10mo agoMerklemap offers that: https://www.merklemap.com/documentation/live-tail https://www.merklemap.com/documentation/live-tail
- agwa 10mo agoThere's an extension to static-ct-api, currently implemented by Sunlight logs, that provides a feed of just SANs and CNs: https://github.com/FiloSottile/sunlight/blob/main/names-tiles.md https://github.com/FiloSottile/sunlight/blob/main/names-tile... For example: curl https://tuscolo2026h1.skylight.geomys.org/tile/names/000 | gunzip (It doesn't deduplicate if the same domain name appears in multiple certificates, but it's still a substantial reduction in bandwidth compared to serving the entire (pre)certificate.)
- raldi 10mo agoWhat reason?
- electroly 10mo agoThe CT log tells you about new websites as soon as they come online. Good if you're intending to scrape the web.
- 1vuio0pswjnm7 10mo ago"... for exacty this reason." Needs clarification. What reason
- TeMPOraL 10mo agoKnowing what DNS names are actually used. EDIT: that's the flip side of supporting HTTPS that's not well-known among developers - by acquiring a legitimate certificate for your service to enable HTTPS, you also announce to the entire world, through a public log, that your service exists.
- thesuitonym 10mo agoI don't really see how this is a flip-side. If you're putting something on the web, presumably you want it to be accessed by others, so this is actually a benefit. If you didn't want others to access your service, maybe consider putting it in a private space.
- aziaziazi 10mo agoThere’s usages of https that don’t overlap with "the (public) web".
- tbrownaw 10mo agoAll of the internal stuff at $employer uses a private CA. I suspect this is fairly universal at places that aren't super tiny.
- TeMPOraL 10mo agoProblem is a lack of solutions that work at places that are tiny, such as a small company, or a household. This is yet another area of the computing ecosystem that forgets there are other use cases for computers than commerce.
- 1vuio0pswjnm7 10mo ago
- kccqzy 10mo agoCertificate transparency log is a Google project. They don’t need to scrape it. They host all the data. It’s one of those projects where Google hosts it because it thinks it genuinely improves the internet, by reducing certificate authority abuse.
- tech234a 10mo agoThe Web Archive also uses the Certificate Transparency logs, some websites that aren't linked anywhere end up in the Wayback Machine this way: https://archive.org/details/certificate-transparency?tab=about https://archive.org/details/certificate-transparency?tab=abo...