4 ms·

Ask HN: If you were to build a new search engine. What would be your plan?

8 points by mlejva 6y ago

codegladiator 6y ago
I would start with certain category of the web, by topic and start with a topic niche search engine, and start to cover more categories one by one if the search model is successful.
- strongbond 6y ago
  +1
danskeren 6y ago
I've been working on and off on a search engine for the past few months. The way I see it you have 2 choices if you want to build a general-purpose search engine: 1. Scrape results from other search engines, similar to Searx 2. Build a Bing wrapper, similar to DuckDuckGo, Ecosia, et al. With Ask.Moe I've tried both. The problem with scraping other search engines is that they (especially Google) are very quick to CAPTCHA your requests, so you'd need to setup and maintain tons of proxies and you'd likely struggle with search engines returning results based on the proxy location since they like to ignore URL parameters/user settings. The problem with building a Bing wrapper is that it's incredibly expensive ($3/1000 transactions for just Web, $7/1000 if you want Web/News/Image/Video/etc. as well). Let's say reaching the frontpage of Reddit/HN netted you 100.000 visitors submitting an average of 10 queries - that would end up costing you $3000 or $7000 depending on which plan you were using. And you most likely wouldn't have any ads/affiliate links to offset the costs since Microsoft Advertising (I believe that's what DuckDuckGo/Ecosia use) is invite-only and won't even entertain talking to you about your new search engine. The reason you see a new search engine pop-up every month and disappear again the next is because it's easy to build a Bing wrapper but incredibly difficult to make it profitable. My plan is to build various categories (so far I've built https://ask.moe/math https://ask.moe/math and https://ask.moe/currency https://ask.moe/currency - podcast or news will likely be the next) that are inexpensive to operate and can hopefully generate some revenue. I'm also strongly considering supporting accounts where users can buy credits to use the general-purpose search engine, but I think it will be very difficult to convince people to pay to use a search engine.
bwb 6y ago
Who needs to search for something the most? What topic doesn't have a destination website? I would figure out a niche that needs search and is an industry that is highly fragmented and likely to stay that way. Unfortunately, i am not sure if there are any :), at this point it might make more sense to build a destination site to claim that niche. I.e. you go to amazon for books/shopping, you go to facebook groups or reddit for info from people, travel stays at booking/airbnb... etc etc. what niche really needs external search? /phone ramble
- dslkjfgsdgvjv 6y ago
  > I would figure out a niche that needs search and is an industry that is highly fragmented and likely to stay that way. Unfortunately, i am not sure if there are any :), The wedding industry is one of them.
  - bwb 6y ago
    Ya but hyper local...
cblconfederate 6y ago
Im actually planning to build an ad-only search. People will only see ads and the highest bidder goes on top. It will be 80% as good as google
- tudelo 6y ago
  Any idea of user interest here?
  - cblconfederate 6y ago
    billions of people search google everyday to see ads
- dslkjfgsdgvjv 6y ago
  I actually had the same thought not so long ago. When you really think about it you don't need to scrape and sort the entire web to have a shopping search engine. Instead you could focus on the design and have visually appealing ads instead of Google like ads.
VoodooJuJu 6y ago
First, I would gather some key buzzwords such as machine-learning, privacy-first, de-google, and whatever else the kids/investors like these days. Then, I'd do my best to create a brand associated with some such trendy buzzwords. Ideally I'd have some friends, or friends who have friends; friends who feel like investing and/or shilling hard to other investors, and who can also shill hard to potential users. If all goes well, after a couple years or so of heavy shilling and growth hacking and whatnot, me and my friends get our accountant and lawyer friends to do some sort of equity/legal sleight of hand, we the founders make a nice exit just before or just as our grassroots, cool, hip new product degenerates into some spurious incarnation of what it claimed it wouldn't become, and we the founders live happily ever after. Somewhere in that plan we build a search engine or whatever, but that's not really important.

8 points by mlejva 6y ago

I would start with certain category of the web, by topic and start with a topic niche search engine, and start to cover more categories one by one if the search model is successful.

strongbond 6y ago
+1

danskeren 6y ago

I've been working on and off on a search engine for the past few months. The way I see it you have 2 choices if you want to build a general-purpose search engine: 1. Scrape results from other search engines, similar to Searx 2. Build a Bing wrapper, similar to DuckDuckGo, Ecosia, et al. With Ask.Moe I've tried both. The problem with scraping other search engines is that they (especially Google) are very quick to CAPTCHA your requests, so you'd need to setup and maintain tons of proxies and you'd likely struggle with search engines returning results based on the proxy location since they like to ignore URL parameters/user settings. The problem with building a Bing wrapper is that it's incredibly expensive ($3/1000 transactions for just Web, $7/1000 if you want Web/News/Image/Video/etc. as well). Let's say reaching the frontpage of Reddit/HN netted you 100.000 visitors submitting an average of 10 queries - that would end up costing you $3000 or $7000 depending on which plan you were using. And you most likely wouldn't have any ads/affiliate links to offset the costs since Microsoft Advertising (I believe that's what DuckDuckGo/Ecosia use) is invite-only and won't even entertain talking to you about your new search engine. The reason you see a new search engine pop-up every month and disappear again the next is because it's easy to build a Bing wrapper but incredibly difficult to make it profitable. My plan is to build various categories (so far I've built https://ask.moe/math https://ask.moe/math and https://ask.moe/currency https://ask.moe/currency - podcast or news will likely be the next) that are inexpensive to operate and can hopefully generate some revenue. I'm also strongly considering supporting accounts where users can buy credits to use the general-purpose search engine, but I think it will be very difficult to convince people to pay to use a search engine.

bwb 6y ago

Who needs to search for something the most? What topic doesn't have a destination website? I would figure out a niche that needs search and is an industry that is highly fragmented and likely to stay that way. Unfortunately, i am not sure if there are any :), at this point it might make more sense to build a destination site to claim that niche. I.e. you go to amazon for books/shopping, you go to facebook groups or reddit for info from people, travel stays at booking/airbnb... etc etc. what niche really needs external search? /phone ramble

dslkjfgsdgvjv 6y ago
> I would figure out a niche that needs search and is an industry that is highly fragmented and likely to stay that way. Unfortunately, i am not sure if there are any :), The wedding industry is one of them.
- bwb 6y ago
  Ya but hyper local...

cblconfederate 6y ago

Im actually planning to build an ad-only search. People will only see ads and the highest bidder goes on top. It will be 80% as good as google

tudelo 6y ago
Any idea of user interest here?
- cblconfederate 6y ago
  billions of people search google everyday to see ads
dslkjfgsdgvjv 6y ago
I actually had the same thought not so long ago. When you really think about it you don't need to scrape and sort the entire web to have a shopping search engine. Instead you could focus on the design and have visually appealing ads instead of Google like ads.

VoodooJuJu 6y ago

First, I would gather some key buzzwords such as machine-learning, privacy-first, de-google, and whatever else the kids/investors like these days. Then, I'd do my best to create a brand associated with some such trendy buzzwords. Ideally I'd have some friends, or friends who have friends; friends who feel like investing and/or shilling hard to other investors, and who can also shill hard to potential users. If all goes well, after a couple years or so of heavy shilling and growth hacking and whatnot, me and my friends get our accountant and lawyer friends to do some sort of equity/legal sleight of hand, we the founders make a nice exit just before or just as our grassroots, cool, hip new product degenerates into some spurious incarnation of what it claimed it wouldn't become, and we the founders live happily ever after. Somewhere in that plan we build a search engine or whatever, but that's not really important.