8 ms·
One thing I hope this project does that Google fails to do is give developers a good API to access search. Google closed down their first web search API and now
by onion2k 11y ago
One thing I hope this project does that Google fails to do is give developers a good API to access search. Google closed down their first web search API and now only give developers access to a limited Custom Search API that's rate limited to 100 queries a day for free with a hard limit of 10k searches - that makes it either very hard to develop anything against or relatively expensive. There are other options (Bing, Faroo, raw access to CommonCrawl) but they're either low quality or hard to work with. A good quality, straightforward, open web search API would be awesome.
- faizshah 11y agoFrom their FAQ: "We will particularly stand out with features that are not in the best interest of commercial search engines: showing less (if no) ads, having an open API, and generally not trying to maximize the amount of time users spend on our service." Seems like right now they are focusing on getting contributors though.
- visarga 11y agoI second this. The search engine could be an API to the web. There are lots of resources that could be pulled out that way.
- mtrn 11y agoI would pay for an API, that gives me access to partial quality crawl content. It would be even better, if the web would be treated, at least for parts, as a digital library and non-profit organizations would recognize the value of access to such a resource and provide it (just as maintenance for roads or public schools).
- sylvinus 11y agoThanks for the feedback! What would be your usecase for that API?
- jclos 11y agoI'm not the original commenter but there could be a big huge case in research. Lots of researchers work on UIs for search, interactive search systems, and query refinement algorithms that are really just abstract layers over an existing search engine. It used to be that we could just overlay stuff over Google, but most search engines nowadays are a pain to work with.
- mtrn 11y agoI see scientific agents on the horizon and with them a new wave of research. But access to raw data, e.g. publications, is very limited and restricted. It's a pity.
- struct 11y agoSome of my primary interests are building websites that can analyse other websites and for that, I need a keyword index and access to the underlying crawl data (as in, a response that can point me to the exact file offsets that contain the pages). Think [1] but with keywords instead of URLs. [1] http://index.commoncrawl.org/ http://index.commoncrawl.org/
- sylvinus 11y agoThanks! We probably won't expose the underlying crawl data ourselves but being able to reference it just like [1] does is indeed important.
- mtrn 11y agoI would love to have a list of URL of all .edu domains, which contain the word "publications". It's a bit silly, but I'd love to build a open version something like Google Scholar. I can think of many other use cases as well, where a product needs to built from larger, but very selected set of raw input or pages. BTW: Thanks for Hackday Paris 2011! Loved that event and venue :)
- sylvinus 11y ago
- partiallypro 11y agoCan I ask what's wrong with Bing's API (outside of it being Bing)? 5000 monthly transactions before you begin paying, and it doesn't appear to be hard to work with?
- rgovind 11y agoI am currently evaluating http://www.faroo.com/hp/api/api.html http://www.faroo.com/hp/api/api.html Anybody here has any thoughts/experience with it?