5 ms·

What are the alternatives for a full web search API? Yahoo BOSS closed down a few weeks ago. Bing seems the last man standing. And, unfortunately, none of the o

What are the alternatives for a full web search API? Yahoo BOSS closed down a few weeks ago. Bing seems the last man standing. And, unfortunately, none of the ones I've tried are very good; one project I'm working on needs to prioritize newer results over older ones, and only Google had a sort by date option (which I think was also removed in the past).

bake 11y ago
Google Custom Search is one option, but it has a limit of 10,000 queries per day and is very expensive compared to Bing ($5 per 1,000 queries). I've migrated to Bing, myself -- no volume limits, generous free tier, and much cheaper than Google.
- SwellJoe 11y ago
  I'm not seeing how to make GCS perform a full web search. It won't allow creation of a search without at least one schema defined, and I need schemaless data (or rather, I need the whole web, including a variety of schemas). Am I missing something?
- jrochkind1 11y ago
  Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous URL: https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57-A49D98D29A44 https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57... . I think that's what you're talking about? Note the link on that page to "Bing API FAQ" -- is broken. If this wasn't MS, the poor docs would make me think it was surely a terrible product.
  - woodruffw 11y ago
    >Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. You can, refer to this SO answer: http://stackoverflow.com/a/11206266/3354209 http://stackoverflow.com/a/11206266/3354209
    - jrochkind1 11y ago
      Thanks. One of the comments on that SO answer from Aug 2015 says: > WARNING: we did development using the free version, but to upgrade to the paid version (to do more than 100 searches), google forces you to turn off the "search the entire web but emphasize included sites" I remain not very confident that you can really search the whole web with google custom search, or if you can that it's not some kind of a loophole that google might close without warning. But if someone has actually done this succesfully, with a paid account for more than 100/queries a day, I would definitely be interested in hearing about it!
      - hantusk 11y ago
        I am doing this successfully with 1,000 queries in a day, and the limit it 10k queries per day (pay per 1,000 queries) The billing system is confusing. You just have to sign up for billing, and you can raise the limit. The upgrade in the google system, is only if you want site search with no ads.
      - ellyagg 11y ago
        Yes, you can search the whole web. We have been using a pool of credit cards to make up to a million requests per month.
  - phatrice 11y ago
    You can check out the new home of Bing Search API here: https://www.microsoft.com/cognitive-services/en-us/bing-web-search-api https://www.microsoft.com/cognitive-services/en-us/bing-web-... You can get key right from there and an API console.
    - jrochkind1 11y ago
      Thanks! Should I chalk up to conspiracy that this page is nearly impossible to find on, um, Google, searching terms that you would reasonably use to find it? But if you have an in with MS, it would probably help an awful lot if they removed the old pages with broken links, unhelpful content, and possibly out of date inaccurate content too. The page you link to has "coming soon" for pricing for Search (which was kind of confusing to find) -- does that mean that the pricing info on the page I found is not accurate, or soon won't be, with no indication on the page I found that that will be the case? MS is really really bad at documenting this service. Only because it's MS am I willing to entertain that the quality of docs may not reflect the quality of service, generally I'd give up on something that markets and explains itself so poorly and figure that if they can't get that right, they probably didn't get the service right either.
    - bake 11y ago
      Here's the old home, with pricing laid out (missing from the new one): https://datamarket.azure.com/dataset/bing/searchweb#terms https://datamarket.azure.com/dataset/bing/searchweb#terms
  - bake 11y ago
    Here's the older link to the Bing Web Search API -- while the new page doesn't list actual prices, this one does in nice detail: https://datamarket.azure.com/dataset/bing/searchweb#terms https://datamarket.azure.com/dataset/bing/searchweb#terms
  - jpsaroudakis 11y ago
    Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services https://www.microsoft.com/cognitive-services. All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pricing plan if you want to extend the monthly quota. Some folks mentioned this page: https://datamarket.azure.com/dataset/bing/search https://datamarket.azure.com/dataset/bing/search. I want to clarify that the page includes the old Bing Search APIs that are still in use, but will be deprecated in the future. You can send feedback or questions here: https://cognitive.uservoice.com/ https://cognitive.uservoice.com/ [Edit] Here is also a blog post with useful resources: https://blogs.msdn.microsoft.com/bingdevcenter/2016/04/13/recap-bing-at-build-2016/ https://blogs.msdn.microsoft.com/bingdevcenter/2016/04/13/re...
    - SwellJoe 11y ago
      I keep checking out the Bing API every couple months, hoping for sort-by options (I need last changed date, most recent first). Any hope of ever seeing sort-by-date from Bing? Google has it, I don't think anyone else does.
    - tokensimian 11y ago
      Hi, just a quick bit of feedback -- 1k API calls / month is so low as to be a nonstarter. Of the half dozen hobby (non-commercial) projects I am currently working on, this amount is tragically low. Per day would give me pause. I don't mean to rain on your efforts, but I would personally never consider using an API with such limits, even for trivial hobby projects.
      - trakout 11y ago
        Agreed -- 1k/mo is not worth the initial time investment.
      - deleted 11y ago
        [deleted]
woodruffw 11y ago
Google Custom Search[1] is still available, albeit limited to 100 free queries a day (and 10k max, even when paying). That being said, I'm still looking for alternatives as well. This broke my IRC bot and a few scripts rather unexpectedly. [1]: http://www.google.com/cse/ http://www.google.com/cse/
- nerdy 11y ago
  Doesn't this only allow you to search your own page, as mentioned at the URL you linked? "With Google Custom Search, add a search box to your homepage to help people find what they need on your website." (emphasis mine) Their video also seems to indicate the same thing: https://youtu.be/Qd9z48Bo8ZA https://youtu.be/Qd9z48Bo8ZA They say you can search "one website or even a specific topic across multiple websites". This is specifying multiple domains, not a full-index search.
skykooler 11y ago
There was someone on HN a few weeks back who was making a new search engine - I'm curious as to whether they offer a search API.
pajop 11y ago
the first thing that came to mind was DuckDuckGo :) https://duckduckgo.com/api https://duckduckgo.com/api
- SwellJoe 11y ago
  DDG is not a web search API.
- boyter 11y ago
  Its not a web index. They are unable to offer that as a lot of their results come from external services such as Yandex and Bing. The results are their zeroclick info snippets and it is very useful for pulling an answer from wikipedia since they have already parsed it.
  - chris_wot 11y ago
    But do they consider spidering context problematic? What would stop someone building an API on top of their simple websearch?
- jrochkind1 11y ago
  It has no web search api.
  - skrowl 11y ago
    Their search result HTML is pretty easy to scrape though
boyter 11y ago
Yandex Bing Gigablast Those are the only web scale indexes that have an API I am aware of these days.
- DonHopkins 11y ago
  Hoolie's search API hasn't been returning any articles critical of Gavin Belson any more, but it sure is faster now that they're using that new ranking system!
dwynings 11y ago
http://gigablast.com/searchfeed.html http://gigablast.com/searchfeed.html

bake 11y ago

Google Custom Search is one option, but it has a limit of 10,000 queries per day and is very expensive compared to Bing ($5 per 1,000 queries). I've migrated to Bing, myself -- no volume limits, generous free tier, and much cheaper than Google.

SwellJoe 11y ago
I'm not seeing how to make GCS perform a full web search. It won't allow creation of a search without at least one schema defined, and I need schemaless data (or rather, I need the whole web, including a variety of schemas). Am I missing something?
jrochkind1 11y ago
Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous URL: https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57-A49D98D29A44 https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57... . I think that's what you're talking about? Note the link on that page to "Bing API FAQ" -- is broken. If this wasn't MS, the poor docs would make me think it was surely a terrible product.
- woodruffw 11y ago
  >Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. You can, refer to this SO answer: http://stackoverflow.com/a/11206266/3354209 http://stackoverflow.com/a/11206266/3354209
  - jrochkind1 11y ago
    Thanks. One of the comments on that SO answer from Aug 2015 says: > WARNING: we did development using the free version, but to upgrade to the paid version (to do more than 100 searches), google forces you to turn off the "search the entire web but emphasize included sites" I remain not very confident that you can really search the whole web with google custom search, or if you can that it's not some kind of a loophole that google might close without warning. But if someone has actually done this succesfully, with a paid account for more than 100/queries a day, I would definitely be interested in hearing about it!
    - hantusk 11y ago
      I am doing this successfully with 1,000 queries in a day, and the limit it 10k queries per day (pay per 1,000 queries) The billing system is confusing. You just have to sign up for billing, and you can raise the limit. The upgrade in the google system, is only if you want site search with no ads.
    - ellyagg 11y ago
      Yes, you can search the whole web. We have been using a pool of credit cards to make up to a million requests per month.
- phatrice 11y ago
  You can check out the new home of Bing Search API here: https://www.microsoft.com/cognitive-services/en-us/bing-web-search-api https://www.microsoft.com/cognitive-services/en-us/bing-web-... You can get key right from there and an API console.
  - jrochkind1 11y ago
    Thanks! Should I chalk up to conspiracy that this page is nearly impossible to find on, um, Google, searching terms that you would reasonably use to find it? But if you have an in with MS, it would probably help an awful lot if they removed the old pages with broken links, unhelpful content, and possibly out of date inaccurate content too. The page you link to has "coming soon" for pricing for Search (which was kind of confusing to find) -- does that mean that the pricing info on the page I found is not accurate, or soon won't be, with no indication on the page I found that that will be the case? MS is really really bad at documenting this service. Only because it's MS am I willing to entertain that the quality of docs may not reflect the quality of service, generally I'd give up on something that markets and explains itself so poorly and figure that if they can't get that right, they probably didn't get the service right either.
  - bake 11y ago
    Here's the old home, with pricing laid out (missing from the new one): https://datamarket.azure.com/dataset/bing/searchweb#terms https://datamarket.azure.com/dataset/bing/searchweb#terms
- bake 11y ago
  Here's the older link to the Bing Web Search API -- while the new page doesn't list actual prices, this one does in nice detail: https://datamarket.azure.com/dataset/bing/searchweb#terms https://datamarket.azure.com/dataset/bing/searchweb#terms
- jpsaroudakis 11y ago
  Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services https://www.microsoft.com/cognitive-services. All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pricing plan if you want to extend the monthly quota. Some folks mentioned this page: https://datamarket.azure.com/dataset/bing/search https://datamarket.azure.com/dataset/bing/search. I want to clarify that the page includes the old Bing Search APIs that are still in use, but will be deprecated in the future. You can send feedback or questions here: https://cognitive.uservoice.com/ https://cognitive.uservoice.com/ [Edit] Here is also a blog post with useful resources: https://blogs.msdn.microsoft.com/bingdevcenter/2016/04/13/recap-bing-at-build-2016/ https://blogs.msdn.microsoft.com/bingdevcenter/2016/04/13/re...
  - SwellJoe 11y ago
    I keep checking out the Bing API every couple months, hoping for sort-by options (I need last changed date, most recent first). Any hope of ever seeing sort-by-date from Bing? Google has it, I don't think anyone else does.
  - tokensimian 11y ago
    Hi, just a quick bit of feedback -- 1k API calls / month is so low as to be a nonstarter. Of the half dozen hobby (non-commercial) projects I am currently working on, this amount is tragically low. Per day would give me pause. I don't mean to rain on your efforts, but I would personally never consider using an API with such limits, even for trivial hobby projects.
    - trakout 11y ago
      Agreed -- 1k/mo is not worth the initial time investment.
    - deleted 11y ago
      [deleted]

woodruffw 11y ago

Google Custom Search[1] is still available, albeit limited to 100 free queries a day (and 10k max, even when paying). That being said, I'm still looking for alternatives as well. This broke my IRC bot and a few scripts rather unexpectedly. [1]: http://www.google.com/cse/ http://www.google.com/cse/

nerdy 11y ago
Doesn't this only allow you to search your own page, as mentioned at the URL you linked? "With Google Custom Search, add a search box to your homepage to help people find what they need on your website." (emphasis mine) Their video also seems to indicate the same thing: https://youtu.be/Qd9z48Bo8ZA https://youtu.be/Qd9z48Bo8ZA They say you can search "one website or even a specific topic across multiple websites". This is specifying multiple domains, not a full-index search.

skykooler 11y ago

There was someone on HN a few weeks back who was making a new search engine - I'm curious as to whether they offer a search API.

pajop 11y ago

the first thing that came to mind was DuckDuckGo :) https://duckduckgo.com/api https://duckduckgo.com/api

SwellJoe 11y ago
DDG is not a web search API.
boyter 11y ago
Its not a web index. They are unable to offer that as a lot of their results come from external services such as Yandex and Bing. The results are their zeroclick info snippets and it is very useful for pulling an answer from wikipedia since they have already parsed it.
- chris_wot 11y ago
  But do they consider spidering context problematic? What would stop someone building an API on top of their simple websearch?
jrochkind1 11y ago
It has no web search api.
- skrowl 11y ago
  Their search result HTML is pretty easy to scrape though

boyter 11y ago

Yandex Bing Gigablast Those are the only web scale indexes that have an API I am aware of these days.

DonHopkins 11y ago
Hoolie's search API hasn't been returning any articles critical of Gavin Belson any more, but it sure is faster now that they're using that new ranking system!

dwynings 11y ago

http://gigablast.com/searchfeed.html http://gigablast.com/searchfeed.html