9 ms·
Show HN: NameMonk – fast domain search using web sockets and Bloom filters
- tectonic 13y agoHey HN, I wanted to show you a site that I've been working on. It provides super fast domain lookup using WebSockets to a custom domain lookup server in EventMachine and a set of overlapping Bloom filters. Would love your feedback!
- pudquick 13y agoNifty! A few questions if you don't mind? - What is the source for your list of names to generate the Bloom filters? - How many names do you estimate are in the list(s)? - How long does it take to generate the Bloom filters? - I would assume your filters are based on registered domains, so when you get a positive match do you verify it with a Whois lookup to account for the false positives Bloom filters generate? Also - any thoughts on open sourcing this? My primary concern with third-party domain search engines like this is them lying / stealing my domain ideas. Running a search like this locally would be very interesting.
- tectonic 13y agoGlad you like it! The domain name list is provided in an 8GB dump from Verisign. You sign up to get access, but the process isn't too bad. There are around 100 million registered .com domains right now. Building the bloom filter takes about 20 minutes. It would be smart to double check with Whois data, but that would slow the process down. Right now I link to the registrars instead since they will do the Whois verification. I'd definitely consider open sourcing or collaborating on the code. Send me a note.
- pudquick 13y agoThanks for responding! For anyone looking for that Verisign program, this page appears to have the info: http://www.verisigninc.com/zone http://www.verisigninc.com/zone
- danshapiro 13y agoIt would be neat if someone could post it publicly. Sounds like a fun dataset to experiment with.
- stevewilhelm 13y agohttp://domai.nr/ http://domai.nr/ has some cool features that you might want to incorporate.
- jffry 13y agoCool! It seems like it assumes the TLD is .com for now - do you plan on expanding it? Maybe do input.<tld> for .com, .net, .org, .io, etc? Also it seems like your data source is out of date - for example, your site tells me that namemonk.com is totally available
- tectonic 13y agoThanks. Yea, I'm probably going to follow up with other TLDs in the future.
- jffry 13y agoAlso I just view-sourced... holy hell, what are you doing with 130KB of minified JavaScript? I'm guessing its a huge blob of libraries globbed together? Also your session cookie's name was amusing.
- tectonic 13y agoThat's what you get when you include jQuery UI for no reason... good catch, thanks!
- thex86 13y agoVery nice. If you could add support for other TLDs, I would definitely use this and recommend to my friends.
- antichaos 13y agoAwesome work! How did you use Bloom filters to achieve low latency lookup?
- tectonic 13y agoI periodically download the 8GB com.zone root name file from Verisign and compress it into a set Bloom filters that allow for very fast lookup while using less memory than a textual hash. For fast responses, the filters are served over a UNIX domain socket to the WebSocket server on the backend.
- recuter 13y agoThat's a great approach I've thought of before -- so how big does the bloomified version end up being?
- tectonic 13y agoAbout 1GB to get very low error rates.
- hncommenter13 13y agoWould you mind sharing a bit about how the process to gain access to the root zone file works? I've seen the application form, but I'm curious how the whole thing goes from application to access. Any insights you can share? I've been kicking around a few ideas, but didn't want to go through an involved process if it would only end in heartbreak. Does it require a specific use case as part of the application?
- tectonic 13y agoThe process wasn't bad when I did it, actually a year or two ago. It took me this long to get around to doing anything with the data. :) If you'd like to collaborate, I'm totally game. Send me a note.
- tectonic 13y agoAlso, jffry appears to have figured out the name of the other version of this site. http://findafuckingdomain.com http://findafuckingdomain.com is a humor version using the same backend. ;)
- jffry 13y agoReminds me of http://www.whatthefuckshouldimakefordinner.com/ http://www.whatthefuckshouldimakefordinner.com/
- ngoel36 13y agoVery fast, but to be perfectly honest, still a bit useless for actually finding a business domain name without support for other TLD's. My base point of comparison is domai.nr which does a pretty speedy job across numerous TLDs.
- seldo 13y agoYeah, I'm not sure speed was the metric I needed improved when it comes to domain search, though it's a nice demo project.
- tectonic 13y agoI've been playing with other search methods. Once you have all domains in memory, it's easier to do things like lookup every English word or pronounceable combinations of letters very quickly. I also highly recommend http://bustaname.com http://bustaname.com
- guigui 13y agoNice idea, I could really use this sometimes. However, when I search for namemonk.com it tells me that it's totally available. How reliable is this?
- jffry 13y agoHis FAQ page says the data is refreshed every 24 hours. > whois namemonk.com ... Creation date: 13 Jul 2013 03:26:00 Expiration date: 13 Jul 2014 03:26:00
- tectonic 13y agoIt's pretty accurate, with about a 24 hour delay. Namemonk was registered yesterday and hasn't been updated in the root data dump from Verisign yet.
- mariocesar 13y agoAlso says domicilio.com is available. I think it's using simple dns lookups, if you made a whois call to the domain you will see that the domain is registered but is not in use.
- jtokoph 13y agoIf you could add more TLDs, you could win me away from https://domize.com/ https://domize.com/
- ideaoverload 13y agoTry http://domaintyper.com http://domaintyper.com Very fast , multiple TLDs and gives you suggestions for names based on partial input.
- frozenport 13y agoOh No namemonk.com is totally available
- deleted 13y ago[deleted]
- jasonlingx 13y agohttp://domfinder.com http://domfinder.com is agonisingly slow in comparison but does give you some smart suggestions
- gruseom 13y agoIt's great how fast this is. Suggestion: make the underlined text "not available" be a link to whois information when a domain is taken. The way you have it now, it looks clickable, and that's what I expected it to tell me. Or, come to think of it, don't do that. Just display the whois info automatically when a domain is taken while keeping everything as fast as it is now. That would be awesome!
- tectonic 13y agoThanks for the suggestion! It's hard to do Whois as quickly, but I'll think on it.
- furtivefelon 13y agoSame speed as instantdomainsearch.com, but that service does not track what you search.
- tectonic 13y agoI don't track what you search beyond looking it up in the Bloom filter. How could any site not transmit the domain for lookup? Frankly, I don't understand this comment.
- SenorWilson 13y agoProbably the fact you're using google, facebook, and twitter. All 3 will store some information... Nice name for your session cookie btw :)
- chemcoder 13y agodoes it work for top level domains other than .com? I own a .in domain and the search did not yield a correct result.
- nostromo 13y agohttp://i.imgur.com/d0oGNIe.png http://i.imgur.com/d0oGNIe.png It was the first domain that came to mind. :)
- tectonic 13y agoNameMonk was registered less than a day ago, so it's not in the root data dump yet.
- mjackson 13y agoMy favorite: http://instantdomainsearch.com http://instantdomainsearch.com Includes the ability to save and encrypt searches, links to whois data, as well as alternative names based on similar spelling.
- workhere-io 13y agoAnd then there's http://instantname.me/ http://instantname.me/.
- jwheeler79 13y agoI realize I can't police the internet, but I just want to say that when someone works hard on something and wants to show it off, and your gut reaction is to show off something you didn't work on at all for karma points, it's the ultimate form of troll-dom in my opinion.
- danshapiro 13y agoYour comment makes perfect sense if HN is a resource for submitters to get attention for their new projects. In that case, it's very frustrating and distracting to be shown alternatives. Trolling, as you say. If HN is a resource for readers, then this sort of reply is immensely helpful, because it lets people who are interested in the project and/or its results explore other interesting, related projects. Quite the opposite of trolling. Turns out HN is both.
- jwheeler79 13y agoI guess I can't argue with that.
- TeMPOraL 13y agoWait, what? Why do you suppose GP is "showing off for karma points"? I found this link very relevant, and in general showing similar/related solutions is very informative for both the author and people who read comment threads.
- mmahemoff 13y ago
- justhw 13y agoYou should incorporate colors (green avail, red not avail) so we can grasp quickly.
- tectonic 13y agoFor the humor version with the same backend: http://findafuckingdomain.com http://findafuckingdomain.com
- syassami 13y agoI'm a fan of http://namevine.com/ http://namevine.com/, it includes twitter/facebook/etc registrations.
- tomphoolery 13y agoCool! I really would like to see how this was accomplished, wanna open-source it?
- cmer 13y agoSo let me give you a hint. I am/was the guy who made AjaxWhois back in the days (a similar site, one of the first). It was hugely popular and I was making thousands a month (totally passive) with it until it got acquired and went to shit. I tested a bunch of things, ads in the sidebar, different registrars, etc, but the only thing that ever worked was GoDaddy, right in the search results. Ads in the sidebars are a waste of your time so don't even bother. I know how technical people feel about GD (I feel the same) but the truth is, their crazy upselling techniques work and you get paid commission on those, not so much on the domain name itself. I was often making north of $100 per customer I referred to them. Not sure if they still pay the same but needless to say, that was pretty awesome I also had coupon codes that gave my users a discount AND made me a lot of money. If you can get that, it's the holy grail. It just made no sense for my users not to click on the ad. Good luck!
- tectonic 13y agoThanks for the tips, cmer! I was always a big fan of AjaxWhois and am sad to see that it's gone. Why did you decide to sell it?
- cmer 13y agoAt the time I was working on my startup Defensio and just didn't have time to maintain AjaxWhois. Traffic was high, I was getting love letters weekly but I felt I was I wasn't worthy since I couldn't give it the attention it deserved. Sellin seemed like the right move at the time. I wish I kept it since even less work was put into it post acquisiiton and the new owner eventually forgot to renew the domain name and it just died. Pretty sad ending for a site that so many people loved. Oh well!
- cenhyperion 13y agoHow about some better registrars than GoDaddy as the default list? Hover.com, gandi.net are both much much much better registrars, have much better support, aren't scummy, etc.
- beau 13y agoIn my experience building http://instantdomainsearch.com/ http://instantdomainsearch.com/ I found that the "slow" part was actually the network round-trip to the server. WebSockets don't make packets move across the network any faster. We came up with a way to select the closest/fastest server when you first visit the site: http://instantdomainsearch.com/articles/faster_domain_name_search/ http://instantdomainsearch.com/articles/faster_domain_name_s... This works well, but I think I'll switch to Amazon's Route 53 (DNS) implementation now that it can do failover.
- jackdwyer 13y agoI wrote a real dirty domain name scanner for this, too bad the results have been incorrect so far. https://gist.github.com/jackdwyer/5994001 https://gist.github.com/jackdwyer/5994001 /edit: note it is just getting all 4 letter combinations at the moment.