4 ms·

Show HN: A Search Engine for Developers, Marketers, Web Designers, and Hackers

63 points by nerdydata 13y ago

Watup Hackernews, We have created a new type of search engine that searches on the full source code of webpages, not just the on-page text. It's called NerdyData (nerdydata.com) and we're launching today! Our search engine is different from search engines you've used before. Traditional search engines are geared towards providing answers, whereas our goal is to give you the best list of results for a query. Our crawler has visited over 140 million homepages and collected terabytes of HTML, Javascript, and CSS code. We've also designed several search interfaces that allow anybody to query against the source code of webpages, or download a list of sites containing a specific term. We even offer a search interface specifically for SEO's and marketers that allow you to search within specific HTML tags like meta descriptions and meta keywords. About us: We're two 23 year olds who recently graduated from Stony Brook University in New York, and we are ridiculously passionate about making search better for our fellow nerds out there. If you'd like to know more, you can email me at dave@nerdydata.com Thank you so much for you time!

nerdydata 13y ago
Clickable link: http://nerdydata.com http://nerdydata.com
aram 13y ago
Very interesting service. At first it didn't occur to me why would I want to search the source code, but your examples were good. What is the logic behind the search criteria? I entered `<div id="main">`[1] and the first result is LinkedIn with `<div id="main-wrapper">` highlighted, which shouldn't match my query directly as I entered a closed tag. Some of the other results also didn't contain exactly what I was searching for - e.g. `<div id="main" class="main">`, `<div id="main" class="clear">` etc. A technical explanation on this would be great. Also, one minor note - David's photo in the front page footer is broken [1] [1] https://search.nerdydata.com/search/#!/searchTerm=<div https://search.nerdydata.com/search/#!/searchTerm=<div id="main">/searchPage=1/sort=pop [2] http://www.arc.losrios.edu/Images/Images-arc/Foreign_Lang_Dept/Faculty%20pictures/no%20image%20avatar%20man-1.jpg http://www.arc.losrios.edu/Images/Images-arc/Foreign_Lang_De...
jayzalowitz 13y ago
First off Stony Brook! Second, open crawl? Third, what's your stack?
- nerdydata 13y ago
  1. Go Seawolves! 2. We built our own distributed crawler with a couple of dope servers. 3. Not ready to share our stack setup yet (work in progress) but we will have a write up shortly!
ErikAugust 13y ago
Do results sort based on mozRank, Alexa, PageRank? Or something proprietary?
ScottWhigham 13y ago
Love it - great idea and great presentation. I only tried a few cursory searches but it was definitely bookmark-worthy. Kudos :) I'd love to hear you guys talk about the risks of searching for "dicey"/risky characters/terms and how you protect your data/server from malicious searches.
- nerdydata 13y ago
  When the launch craziness winds down, we plan to start a blog and nerd the fuck out. We had so much fun making this and want to help others get past the insane obstacles we faced while building a search engine.
013 13y ago
Is there any content filtering? When I went on to Image Locator, then "See an example" it will search for imgur.com. The first page of results shows 'thebiglistofporn.com', 'entensity.com'(Not sure if it's NSFW, but it looks it from the thumbnail.) I know it's not too much of a big deal, but porn thumbnails wasn't something I was expecting to see on a websites example page.
- nerdydata 13y ago
  We do not filter content, but thanks for letting us know about that thumbnail on the example page.
michaelmior 13y ago
Looks pretty cool. I really dig the design. I was a little annoyed that I seem to have 10 credits without noticing and used them up just messing around. (I can't really tell because clicking on the credits doesn't do anything). It would be nice to make this more obvious up front. Anyway, I can see this being a pretty useful tool in some scenarios. Also note that "Locater" is spelled "Locator" (sidebar on the left)
- deleted 13y ago
  [deleted]
ilikehunny 13y ago
your site has a really bad user rating? http://www.mywot.com/en/scorecard/nerdydata.com?utm_source=addon&utm_content=warn-viewsc http://www.mywot.com/en/scorecard/nerdydata.com?utm_source=a...
- suhastech 13y ago
  This might give a hint http://web.archive.org/web/*/nerdydata.com http://web.archive.org/web/*/nerdydata.com
- sadkingbilly 13y ago
  I've never trusted wot. They are abused by groups of "wot cops" that organize in the forums and target web sites to bash (deserved or not). As far as the site goes, it looks like they used to be a more generic seo search engine, as recently as a few months ago.
- skinnymuch 13y ago
  That site is awful. Only thing worse is using BBB for credibility.
kevrone 13y ago
This is awesome. Especially for finding use cases and implementation examples of front-end libraries.
- nirvanatikku 13y ago
  That was my first thought, too. I checked out where my various projects are being used on the web, something I've been wanting to see for some time, and was pleased to learn about various properties using it.
- tehaaron 13y ago
  Exactly what I would use it for but then I realized it wasn't free and to me its not worth the money.
snake_plissken 13y ago
Good god, my attention span just reached terminal velocity. I like. Very cool and original.
mkessy 13y ago
This is really cool. As a wanna be web dev I would love to hear about your guys' design process and the technologies you're using. I'm curious about how credits will work. It seems that with traditional search engines like google, people often make many queries and subsequently refine their searches based on the results, so I could search for "best new car" look at the results then decide I might get better results with "highest rated new car" etc. If I'm paying by credits that doesn't seem like the best option since I'll waste credits, so it seems very important to know exactly what I'm looking for ahead of time. It looks like you guys understand this because there are some tools to refine searches. I'm also wondering if you guys plan on adding more tools to analyze the search results, or increase details of queries. I could imagine searching by css selector syntax or regexp, or you could even design your own basic language to query by, especially if your userbase is pretty technical. Looks great so far I look forward to following you alls progress.
jwarkentin 13y ago
I thought it would be really cool until I realized the whole credit thing. It often takes many permutations of a search to find the results you're looking for. If each attempt costs money and the whole thing might be a waste in the end then it's hardly worthwhile. I've always been annoyed that Google won't search for symbols. As a programmer that makes searching for certain things anywhere from difficult to impossible. I was hopeful that this would be a search engine I could use to search for code snippets or queries that involve symbols. Let's say, for example, that I was browsing through some PHP code and saw a double dollar sign ($$) and wanted to know what it was used for (yes, I already know, it's just an example). I might try searching for something like 'PHP $$'. This search engine is useless for that. That's the real problem I need a search engine to solve.
- brodney 13y ago
  Check out http://www.symbolhound.com/ http://www.symbolhound.com/ for a programmer friendly search engine.
  - jwarkentin 13y ago
    You just made my day :)
- gtani 13y ago
  You can search stackoverflow, but not github for "symbolic operators" http://stackoverflow.com/search?q=[php]+%22%24%24%22 http://stackoverflow.com/search?q=[php]+%22%24%24%22 also, languages that need it have their own type signature based search (haskell, scala) http://scalex.org/ http://scalex.org/ http://www.haskell.org/hoogle/ http://www.haskell.org/hoogle/
- ultimatedelman 13y ago
  agree about the credits. i know you're trying to monetize, but forcing me to pay for searches (and view more results) is a bit on the ridiculous side. there are better ways to monetize than that. from a UX perspective, any time you _prevent_ a user from doing something, ie, you present them with the option to view more things and when they click on it, it says you need to pay to view the things, that is BAD UX. better UX would be NOT presenting the user with the option to view more, but where they would look for this feature, have a link that says something like, "want to view more results? upgrade to our premium plan!" Much better UX because as a user I'm not tricked into thinking I'm getting something and not having it be delivered. but yeah, unless you're more explicit about the credits thing, it's going to be your downfall. i like the design and the results that i got from my one search, though :)
ollerac 13y ago
I don't get how to use the seo search. How do I search for all websites that have a certain name in their title for example? <title>name</title> Just returns the exact match. And how do I search meta descriptions and meta keywords?
- nerdydata 13y ago
  SEO Search is a premium feature which lets you search inside of pre-parsed html tags. You can learn more about it here https://search.nerdydata.com/documentation/#seoSearch https://search.nerdydata.com/documentation/#seoSearch
kyle_martin1 13y ago
This would be a very good tool to find XSS vulnerable websites.
- cwings 13y ago
  Or find all sites with a vulnerable installation of wordpress.
brbcoding 13y ago
Whoa... Searches must be based on IP. Shared with a co-worker and they can see all of my searches, and I can see theirs.
- deleted 13y ago
  [deleted]
ScottWhigham 13y ago
Anyone tried comparing to http://searchcode.com/ http://searchcode.com/ ? I haven't - just ran across it right now doing an unrelated search.
jwcrux 13y ago
Just a small request - it would be way cool to have a "site:" filter like in Google searches. A lot of the results found are irrelevant to me as it is. Cool site - great work!
- nerdydata 13y ago
  We have a Refine Tool that allows for AND,OR,NOT filters. It's only available for paid users, but really allows you to push the limits and get the exact query you want. Thanks for the feedback!
bapi 13y ago
Search terms like consumer_secret ou api_secret and you'll find lot of things... :( https://search.nerdydata.com/search/#!/searchTerm=consumer_secret/searchPage=1/sort=pop https://search.nerdydata.com/search/#!/searchTerm=consumer_s... With great power, comes great responsibility.
nerdydata 13y ago
Did we mention our "Basic" account is free and gives you 200 credits to use? https://search.nerdydata.com/pricing?hackerNews https://search.nerdydata.com/pricing?hackerNews
- bapi 13y ago
  No , because yesterday it was 49$ or 99$ Idon't remember exactly :). Good move I think to let people like me use it and find value in it as the promise is great.
adamnemecek 13y ago
Honestly, I'm not a fan of the name.
sachin0235 13y ago
while it is good, i am wondering what it can do which google can not? For example i searched for android accountmanager and results i got were far less productive than what google returns me back. If you want to build it, build it the way it perceives user perspective well
dgunn 13y ago
This could be great. You need to monetize in a better way. Make searching free for everyone (cause no one is going to pay for it). Then, package up some pre-built searches (backlink search, competitor stuff, etc), build a special interface, and sell that to SEO's as a tool. This is just my first thought. You have data from over 140 MM pages. You can almost certainly use that data to make a product worth money to someone. I just can't imagine selling searches will work. The big players in the space have made search into a human right. I know you have to make money somehow but charging for searches almost has to be the wrong answer.

63 points by nerdydata 13y ago

nerdydata 13y ago

Clickable link: http://nerdydata.com http://nerdydata.com

aram 13y ago

Very interesting service. At first it didn't occur to me why would I want to search the source code, but your examples were good. What is the logic behind the search criteria? I entered `<div id="main">`[1] and the first result is LinkedIn with `<div id="main-wrapper">` highlighted, which shouldn't match my query directly as I entered a closed tag. Some of the other results also didn't contain exactly what I was searching for - e.g. `<div id="main" class="main">`, `<div id="main" class="clear">` etc. A technical explanation on this would be great. Also, one minor note - David's photo in the front page footer is broken [1] [1] https://search.nerdydata.com/search/#!/searchTerm=<div https://search.nerdydata.com/search/#!/searchTerm=<div id="main">/searchPage=1/sort=pop [2] http://www.arc.losrios.edu/Images/Images-arc/Foreign_Lang_Dept/Faculty%20pictures/no%20image%20avatar%20man-1.jpg http://www.arc.losrios.edu/Images/Images-arc/Foreign_Lang_De...

jayzalowitz 13y ago

First off Stony Brook! Second, open crawl? Third, what's your stack?

nerdydata 13y ago
1. Go Seawolves! 2. We built our own distributed crawler with a couple of dope servers. 3. Not ready to share our stack setup yet (work in progress) but we will have a write up shortly!

ErikAugust 13y ago

Do results sort based on mozRank, Alexa, PageRank? Or something proprietary?

ScottWhigham 13y ago

Love it - great idea and great presentation. I only tried a few cursory searches but it was definitely bookmark-worthy. Kudos :) I'd love to hear you guys talk about the risks of searching for "dicey"/risky characters/terms and how you protect your data/server from malicious searches.

nerdydata 13y ago
When the launch craziness winds down, we plan to start a blog and nerd the fuck out. We had so much fun making this and want to help others get past the insane obstacles we faced while building a search engine.

013 13y ago

Is there any content filtering? When I went on to Image Locator, then "See an example" it will search for imgur.com. The first page of results shows 'thebiglistofporn.com', 'entensity.com'(Not sure if it's NSFW, but it looks it from the thumbnail.) I know it's not too much of a big deal, but porn thumbnails wasn't something I was expecting to see on a websites example page.

nerdydata 13y ago
We do not filter content, but thanks for letting us know about that thumbnail on the example page.

michaelmior 13y ago

Looks pretty cool. I really dig the design. I was a little annoyed that I seem to have 10 credits without noticing and used them up just messing around. (I can't really tell because clicking on the credits doesn't do anything). It would be nice to make this more obvious up front. Anyway, I can see this being a pretty useful tool in some scenarios. Also note that "Locater" is spelled "Locator" (sidebar on the left)

deleted 13y ago
[deleted]

ilikehunny 13y ago

your site has a really bad user rating? http://www.mywot.com/en/scorecard/nerdydata.com?utm_source=addon&utm_content=warn-viewsc http://www.mywot.com/en/scorecard/nerdydata.com?utm_source=a...

suhastech 13y ago
This might give a hint http://web.archive.org/web/*/nerdydata.com http://web.archive.org/web/*/nerdydata.com
sadkingbilly 13y ago
I've never trusted wot. They are abused by groups of "wot cops" that organize in the forums and target web sites to bash (deserved or not). As far as the site goes, it looks like they used to be a more generic seo search engine, as recently as a few months ago.
skinnymuch 13y ago
That site is awful. Only thing worse is using BBB for credibility.

kevrone 13y ago

This is awesome. Especially for finding use cases and implementation examples of front-end libraries.

nirvanatikku 13y ago
That was my first thought, too. I checked out where my various projects are being used on the web, something I've been wanting to see for some time, and was pleased to learn about various properties using it.
tehaaron 13y ago
Exactly what I would use it for but then I realized it wasn't free and to me its not worth the money.

snake_plissken 13y ago

Good god, my attention span just reached terminal velocity. I like. Very cool and original.

mkessy 13y ago

This is really cool. As a wanna be web dev I would love to hear about your guys' design process and the technologies you're using. I'm curious about how credits will work. It seems that with traditional search engines like google, people often make many queries and subsequently refine their searches based on the results, so I could search for "best new car" look at the results then decide I might get better results with "highest rated new car" etc. If I'm paying by credits that doesn't seem like the best option since I'll waste credits, so it seems very important to know exactly what I'm looking for ahead of time. It looks like you guys understand this because there are some tools to refine searches. I'm also wondering if you guys plan on adding more tools to analyze the search results, or increase details of queries. I could imagine searching by css selector syntax or regexp, or you could even design your own basic language to query by, especially if your userbase is pretty technical. Looks great so far I look forward to following you alls progress.

jwarkentin 13y ago

I thought it would be really cool until I realized the whole credit thing. It often takes many permutations of a search to find the results you're looking for. If each attempt costs money and the whole thing might be a waste in the end then it's hardly worthwhile. I've always been annoyed that Google won't search for symbols. As a programmer that makes searching for certain things anywhere from difficult to impossible. I was hopeful that this would be a search engine I could use to search for code snippets or queries that involve symbols. Let's say, for example, that I was browsing through some PHP code and saw a double dollar sign ($$) and wanted to know what it was used for (yes, I already know, it's just an example). I might try searching for something like 'PHP $$'. This search engine is useless for that. That's the real problem I need a search engine to solve.

brodney 13y ago
Check out http://www.symbolhound.com/ http://www.symbolhound.com/ for a programmer friendly search engine.
- jwarkentin 13y ago
  You just made my day :)
gtani 13y ago
You can search stackoverflow, but not github for "symbolic operators" http://stackoverflow.com/search?q=[php]+%22%24%24%22 http://stackoverflow.com/search?q=[php]+%22%24%24%22 also, languages that need it have their own type signature based search (haskell, scala) http://scalex.org/ http://scalex.org/ http://www.haskell.org/hoogle/ http://www.haskell.org/hoogle/
ultimatedelman 13y ago
agree about the credits. i know you're trying to monetize, but forcing me to pay for searches (and view more results) is a bit on the ridiculous side. there are better ways to monetize than that. from a UX perspective, any time you _prevent_ a user from doing something, ie, you present them with the option to view more things and when they click on it, it says you need to pay to view the things, that is BAD UX. better UX would be NOT presenting the user with the option to view more, but where they would look for this feature, have a link that says something like, "want to view more results? upgrade to our premium plan!" Much better UX because as a user I'm not tricked into thinking I'm getting something and not having it be delivered. but yeah, unless you're more explicit about the credits thing, it's going to be your downfall. i like the design and the results that i got from my one search, though :)

ollerac 13y ago

I don't get how to use the seo search. How do I search for all websites that have a certain name in their title for example? <title>name</title> Just returns the exact match. And how do I search meta descriptions and meta keywords?

nerdydata 13y ago
SEO Search is a premium feature which lets you search inside of pre-parsed html tags. You can learn more about it here https://search.nerdydata.com/documentation/#seoSearch https://search.nerdydata.com/documentation/#seoSearch

kyle_martin1 13y ago

This would be a very good tool to find XSS vulnerable websites.

cwings 13y ago
Or find all sites with a vulnerable installation of wordpress.

brbcoding 13y ago

Whoa... Searches must be based on IP. Shared with a co-worker and they can see all of my searches, and I can see theirs.

deleted 13y ago
[deleted]

ScottWhigham 13y ago

Anyone tried comparing to http://searchcode.com/ http://searchcode.com/ ? I haven't - just ran across it right now doing an unrelated search.

jwcrux 13y ago

Just a small request - it would be way cool to have a "site:" filter like in Google searches. A lot of the results found are irrelevant to me as it is. Cool site - great work!

nerdydata 13y ago
We have a Refine Tool that allows for AND,OR,NOT filters. It's only available for paid users, but really allows you to push the limits and get the exact query you want. Thanks for the feedback!

bapi 13y ago

Search terms like consumer_secret ou api_secret and you'll find lot of things... :( https://search.nerdydata.com/search/#!/searchTerm=consumer_secret/searchPage=1/sort=pop https://search.nerdydata.com/search/#!/searchTerm=consumer_s... With great power, comes great responsibility.

nerdydata 13y ago

Did we mention our "Basic" account is free and gives you 200 credits to use? https://search.nerdydata.com/pricing?hackerNews https://search.nerdydata.com/pricing?hackerNews

bapi 13y ago
No , because yesterday it was 49$ or 99$ Idon't remember exactly :). Good move I think to let people like me use it and find value in it as the promise is great.

adamnemecek 13y ago

Honestly, I'm not a fan of the name.

sachin0235 13y ago

while it is good, i am wondering what it can do which google can not? For example i searched for android accountmanager and results i got were far less productive than what google returns me back. If you want to build it, build it the way it perceives user perspective well

dgunn 13y ago

This could be great. You need to monetize in a better way. Make searching free for everyone (cause no one is going to pay for it). Then, package up some pre-built searches (backlink search, competitor stuff, etc), build a special interface, and sell that to SEO's as a tool. This is just my first thought. You have data from over 140 MM pages. You can almost certainly use that data to make a product worth money to someone. I just can't imagine selling searches will work. The big players in the space have made search into a human right. I know you have to make money somehow but charging for searches almost has to be the wrong answer.