4 ms·

Shameless plug :) https://www.merklemap.com/search?query=ycombinator.com&page=0 https://www.merklemap.com/search?query=ycombinator.com&page=... Entries are in

by Eikon 10mo ago

Shameless plug :) https://www.merklemap.com/search?query=ycombinator.com&page=0 https://www.merklemap.com/search?query=ycombinator.com&page=... Entries are indexed by subdomain instead of by certificate (click an entry to see all certificates for that subdomain). Also, you can search for any substring (that was quite the journey to implement so it's fast enough across almost 5B entries): https://www.merklemap.com/search?query=ycombi&page=0 https://www.merklemap.com/search?query=ycombi&page=0

chuckadams 10mo ago
Any insights you can share on how you made search so fast? What kind of resources does it take to implement it?
- deleted 10mo ago
  [deleted]
- Eikon 10mo ago
  Most of merklemap is stored on ZeroFS [0] and thus allows to scale IO ressources quite crazily :) [0] https://github.com/Barre/ZeroFS https://github.com/Barre/ZeroFS
  - jddj 10mo ago
    > Watch Ubuntu boot from ZeroFS Love it
  - rendaw 10mo ago
    How does ZeroFS handle consistency with writes?
    - Eikon 10mo ago
      If you use 9P or NBD it handles fsync as expected. With NFS, it's time based. https://github.com/Barre/ZeroFS#9p-recommended-for-better-performance-and-fsync-posix-semantics https://github.com/Barre/ZeroFS#9p-recommended-for-better-pe...
      - rendaw 10mo ago
        Oh awesome! I was searching for consistency, but I guess durability is the word used for filesystems. Thanks!
nerdsniper 10mo ago
Thank you!!! Needed exactly this at work.
- Eikon 10mo ago
  Glad it was helpful!
TacticalCoder 10mo ago
Not 100% related but not 100% not-related either: I've got a script that generates variations of the domain names I use the most... All the most common typos/mispelling, all the "1337" variations, all the Levenhstein edit distance of 1, quite some of the 2, etc. For example for "lillybank.com", I'll generate: llllybank.com liliybank.com ... and countless others. Hundreds of thousands of entries. They then are null-routed from my unbound DNS resolver. My browsers are forced into "corporate" settings where they cannot use DoH/DoT: it's all, between my browsers and my unbound resolver, in the clear. All DNS UDP traffic that contains any Unicode domain name is blocked by the firewall. No DNS over TCP is allowed (and, no, I don't care). I also block entire countries' TLD as well as entire countries' IP blocks. Been running a setup like that (and many killfiles, and DNS resolvers known to block all known porn and know malware sites etc.) since years now already. The Internet keeps working fine.
rendaw 10mo ago
The first page of results doesn't include ycombinator.com. I get `app.baby-ycombinator.com`, `ycombinator.comchat.com`, everything in between. Substring doesn't seem like what I'd want in a subdomain search.
- Eikon 10mo ago
  > Substring doesn't seem like what I'd want in a subdomain search. Well, if you want only subdomains search for *.ycombinator.com. https://www.merklemap.com/search?query=*.ycombinator.com&page=0 https://www.merklemap.com/search?query=*.ycombinator.com&pag...

by Eikon 10mo ago

chuckadams 10mo ago

Any insights you can share on how you made search so fast? What kind of resources does it take to implement it?

deleted 10mo ago
[deleted]
Eikon 10mo ago
Most of merklemap is stored on ZeroFS [0] and thus allows to scale IO ressources quite crazily :) [0] https://github.com/Barre/ZeroFS https://github.com/Barre/ZeroFS
- jddj 10mo ago
  > Watch Ubuntu boot from ZeroFS Love it
- rendaw 10mo ago
  How does ZeroFS handle consistency with writes?
  - Eikon 10mo ago
    If you use 9P or NBD it handles fsync as expected. With NFS, it's time based. https://github.com/Barre/ZeroFS#9p-recommended-for-better-performance-and-fsync-posix-semantics https://github.com/Barre/ZeroFS#9p-recommended-for-better-pe...
    - rendaw 10mo ago
      Oh awesome! I was searching for consistency, but I guess durability is the word used for filesystems. Thanks!

nerdsniper 10mo ago

Thank you!!! Needed exactly this at work.

Eikon 10mo ago
Glad it was helpful!

TacticalCoder 10mo ago

Not 100% related but not 100% not-related either: I've got a script that generates variations of the domain names I use the most... All the most common typos/mispelling, all the "1337" variations, all the Levenhstein edit distance of 1, quite some of the 2, etc. For example for "lillybank.com", I'll generate: llllybank.com liliybank.com ... and countless others. Hundreds of thousands of entries. They then are null-routed from my unbound DNS resolver. My browsers are forced into "corporate" settings where they cannot use DoH/DoT: it's all, between my browsers and my unbound resolver, in the clear. All DNS UDP traffic that contains any Unicode domain name is blocked by the firewall. No DNS over TCP is allowed (and, no, I don't care). I also block entire countries' TLD as well as entire countries' IP blocks. Been running a setup like that (and many killfiles, and DNS resolvers known to block all known porn and know malware sites etc.) since years now already. The Internet keeps working fine.

rendaw 10mo ago

The first page of results doesn't include ycombinator.com. I get `app.baby-ycombinator.com`, `ycombinator.comchat.com`, everything in between. Substring doesn't seem like what I'd want in a subdomain search.

Eikon 10mo ago
> Substring doesn't seem like what I'd want in a subdomain search. Well, if you want only subdomains search for *.ycombinator.com. https://www.merklemap.com/search?query=*.ycombinator.com&page=0 https://www.merklemap.com/search?query=*.ycombinator.com&pag...