8 ms·
Ask HN: Has Google/Bing won the search engine war?
Recently I've been working on a project that required me to search a limited set of domains.
1)I tried Google Custom Search but they restrict usage to 10000 API calls per day at $50/day. I cannot go higher than this even if I wanted to pay.
2)So I decided to write a crawler that will go to these domains and index their pages so I can put them in ElasticSearch and create a basic search engine.
3)When I created the crawler I came across domains that do not allow crawling their content e.g. Yelp, Craigslist, etc. have locked down their sites for crawling. However I see results from these on Google.
4)It seems the only option is to use their APIs (if they provide one) to get data from these domains. This can be a nightmare to maintain as the number of domains increases.
5)I want to respect the domains' policies and not use shady tactics to crawl their pages.
So essentially these domains allow google and bing to crawl their sites because they are big and established and not having their pages show up on Google or Bing would drastically affect their web traffic but smaller startups would be left out in the cold.
So my question is: What is the possibility of a new search engine emerging if the web is locked down for crawlers?
- int64 13y agoIf the service is exposed publicly to the web, It can be crawled regardless of whatever guards are in place by the service provider. Browser emulation will be a good start.
- palakchokshi 13y agoI know we can crawl them. There are no technical issues with crawling them. The issue is I want to respect their TOS. There are multiple ways to circumvent their anti-crawling code but that means any new search engine will have its roots in "shady" tactics of crawling. IMHO crawling should be allowed if the purpose of the crawler is to show results that drive traffic back to the sites and not mashup the content and deliver it as "original content" on the site that crawled these domains. However e.g. Yelp's TOS do not allow for these types of crawlers that essentially drive traffic back to Yelp.
- argumentum 13y ago> The issue is I want to respect their TOS. Permitting certain search engines to crawl but not others is anticompetitive and violates the principle of an open web. > any new search engine will have its roots in "shady" tactics of crawling. Who cares? If you are successful enough, then you'll negotiate with them later. Don't worry about it now. Crawl away!
- palakchokshi 13y agohttp://yelp.com/robots.txt http://yelp.com/robots.txt It seems you can contact Yelp and tell them how you plan to use their data and maybe they'll let you crawl their site. I really want to explore any and all alternative options before I decide to crawl away :)
- argumentum 13y agoAre you hacker, or aren't you? Dumb rules exist to be broken.
- palakchokshi 13y agoAgreed but no harm in accessing the vast knowledge of the HN community to exhaust alternatives before tightening my hacker cap and plunging in head first.
- AznHisoka 13y agoAgreed. 99% of the time these big companies don't care. Also, use proxies. if they come knocking on your door to take down the content, then worry about it.
- palakchokshi 13y agoYelp is going after startups that crawl their site[1]. I don't want to make Yelp the poster site for this because other big sites do this too. [1]http://www.courthousenews.com/2012/01/27/43403.htm http://www.courthousenews.com/2012/01/27/43403.htm
- dm2 13y agoIt's all about innovating and offering something that Google doesn't offer, such as privacy or smart widgets. See: DuckDuckGo.com You're not going to crawl / index better than Google. Maybe you could combine the DMOZ directory and Google results somehow. Rather than replacing Google, try to supplement their services or just somehow provide more value to users.
- palakchokshi 13y agoyou are right. I don't want to make a better search engine than Google's. I do want to provide more personalized results than Google's. To do that I need to give my users a basic result set and then take input from their behavior to provide a more personalized experience. I do want to be able to search over a limited set of domains without hitting a 10000 req/day limit. I typed "sushi" on duckduckgo.com and saw a result from Yelp. If duckduckgo.com used Yelp's API to get that result then you can imagine the complexity of maintaining API connections to hundreds of services like Yelp that do not allow crawling. However if duckduckgo.com crawled/scraped Yelp to get that result then that's against Yelp's TOS which would cause Yelp to send a cease and desist order to duckduckgo.com. IMHO any crawler that drives traffic back to the original site (like Google, Bing, duckduckgo, etc) should be allowed to crawl the site but Yelp's TOS prevents that.
- dm2 13y agoDuckDuckGo lets developers build the extensions for them, which is very clever. http://duckduckhack.com/ http://duckduckhack.com/ Plus, the privacy promotion from them helped drive developers to their sites. Restrictions on what people can do on the internet is a slippery slope.
- timr 13y agoDDG uses Bing's search API. That's why you see Yelp on DDG. I used to work at Yelp, and I can tell you that if there were no restrictions on who crawled the site, we'd have been regularly DOSed by robots. I don't (and didn't) like the precedent, but I understand some of the motivation.
- rip747 13y agoif the cap on google custom search is the only roadblock, then why not use bing's: http://datamarket.azure.com/dataset/8818F55E-2FE5-4CE3-A617-0B8BA8419F65 http://datamarket.azure.com/dataset/8818F55E-2FE5-4CE3-A617-...
- palakchokshi 13y agoThe data returned by Bing is limited. e.g. Bing does not return an image with it's results while Google does. I want to show 1 image from the page along with Title, Description, preview text. Bing has separate searches for web and image however the results are not from the same domains so I can't correlate results and associate an image to a web result. Secondly Bing does not allow search across a limited set of domains. Yahoo BOSS API allows search across limited set of domains however it's done by passing a comma separated list in the URL. This means I can't put 1000 domains in the URL. Moreover it's search results are in a similar format as Bing's. Google Custom Search is the best option in terms of how it allows you to specify the domains you want to search in and save that as a custom search engine and then use the custom search engine's ID in the API request. Furthermore Google Custom Search's result JSON data is more comprehensive than either Bing or Yahoo BOSS.
- rip747 13y agofair enough. have you checked out google enterprise search then? https://www.google.com/enterprise/search/products/gss.html#pricing_content https://www.google.com/enterprise/search/products/gss.html#p...
- palakchokshi 13y agoEnterprise Search is for searching within your own site (or bunch of domains you own). I actually use Google Search Appliance on my day job so I have a good idea of what it does and it's not what I need.
- xpose2000 13y agoThis is slightly off-topic, but still very much related. There will never be an all encompassing search engine to rival Google. I would argue, one could not even rival Bing. The future of search, as with most internet companies moving forward, is all about mastering a niche. You may not be able to beat Google as an overall search engine, but if you dedicate your time and money towards one specific niche, say sports, then you have a fighting chance to beat them at that.
- palakchokshi 13y agoThe era in which Google and Bing became what they are is one in which the web was more open. With walled gardens cropping up everywhere it is becoming tougher to create a search engine. Another point against creating an all encompassing search engine is the sheer size of the web today which makes creating a search engine for the entire web a daunting task. Google and Bing started when the web was much smaller and hence they were able to grow with the web. IMHO there are multiple ways to create a search engine: 1)Mastering search within a vertical e.g. sports, music, etc. and then expanding to other verticals. 2)Go after a limited popular area of the web and provide a more personalized search experience(this limits discoverability of less popular sites but there are ways to overcome this) 3)Enhance the results of existing search engines by providing context, personalization, privacy features,etc. (this relies heavily on existing engines which makes it subject to their TOS) 4)Come up with an algorithm that blows Google's or Bing's out of the water and VCs line up to give you money so you can build the infrastructure to make it a reality (I wish I had this too... sadly I don't)
- krishna2 13y agoHave you tried blekko.com ? The slashtags feature specifically does what you are describing here. Of course, using this method will not get you data from sites that explicitly ban crawling (like yelp.com). Disclaimer: I used to work at blekko and I implemented the slashtags feature.
- greglindahl 13y agoblekko's "ScoutJet" crawler is allowed to crawl yelp. We sent them an email before we launched, and they were nice enough to allow it.
- palakchokshi 13y agoI see ScoutJet in Yelp's Robots.txt file. I guess I'll have to ask them nicely to allow my crawler too. But Yelp is one of many sites that don't allow crawlers so I'll have to ask each one nicely to allow mine. I had even set up a temp google site explaining what my crawler uses the data for (essentially showing results with correct attribution and driving traffic back to original site).
- palakchokshi 13y agoDo you know if Blekko asked all the sites it wanted to crawl, to allow its ScoutJet crawler to crawl their pages?