5 ms·

Google Custom Search is one option, but it has a limit of 10,000 queries per day and is very expensive compared to Bing ($5 per 1,000 queries). I've migrated t

by bake 10y ago

Google Custom Search is one option, but it has a limit of 10,000 queries per day and is very expensive compared to Bing ($5 per 1,000 queries). I've migrated to Bing, myself -- no volume limits, generous free tier, and much cheaper than Google.

SwellJoe 10y ago
I'm not seeing how to make GCS perform a full web search. It won't allow creation of a search without at least one schema defined, and I need schemaless data (or rather, I need the whole web, including a variety of schemas). Am I missing something?
jrochkind1 10y ago
Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous URL: https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57-A49D98D29A44 https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57... . I think that's what you're talking about? Note the link on that page to "Bing API FAQ" -- is broken. If this wasn't MS, the poor docs would make me think it was surely a terrible product.
- woodruffw 10y ago
  >Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. You can, refer to this SO answer: http://stackoverflow.com/a/11206266/3354209 http://stackoverflow.com/a/11206266/3354209
  - jrochkind1 10y ago
    Thanks. One of the comments on that SO answer from Aug 2015 says: > WARNING: we did development using the free version, but to upgrade to the paid version (to do more than 100 searches), google forces you to turn off the "search the entire web but emphasize included sites" I remain not very confident that you can really search the whole web with google custom search, or if you can that it's not some kind of a loophole that google might close without warning. But if someone has actually done this succesfully, with a paid account for more than 100/queries a day, I would definitely be interested in hearing about it!
    - hantusk 10y ago
      I am doing this successfully with 1,000 queries in a day, and the limit it 10k queries per day (pay per 1,000 queries) The billing system is confusing. You just have to sign up for billing, and you can raise the limit. The upgrade in the google system, is only if you want site search with no ads.
    - ellyagg 10y ago
      Yes, you can search the whole web. We have been using a pool of credit cards to make up to a million requests per month.
- phatrice 10y ago
  You can check out the new home of Bing Search API here: https://www.microsoft.com/cognitive-services/en-us/bing-web-search-api https://www.microsoft.com/cognitive-services/en-us/bing-web-... You can get key right from there and an API console.
  - jrochkind1 10y ago
    Thanks! Should I chalk up to conspiracy that this page is nearly impossible to find on, um, Google, searching terms that you would reasonably use to find it? But if you have an in with MS, it would probably help an awful lot if they removed the old pages with broken links, unhelpful content, and possibly out of date inaccurate content too. The page you link to has "coming soon" for pricing for Search (which was kind of confusing to find) -- does that mean that the pricing info on the page I found is not accurate, or soon won't be, with no indication on the page I found that that will be the case? MS is really really bad at documenting this service. Only because it's MS am I willing to entertain that the quality of docs may not reflect the quality of service, generally I'd give up on something that markets and explains itself so poorly and figure that if they can't get that right, they probably didn't get the service right either.
  - bake 10y ago
    Here's the old home, with pricing laid out (missing from the new one): https://datamarket.azure.com/dataset/bing/searchweb#terms https://datamarket.azure.com/dataset/bing/searchweb#terms
- bake 10y ago
  Here's the older link to the Bing Web Search API -- while the new page doesn't list actual prices, this one does in nice detail: https://datamarket.azure.com/dataset/bing/searchweb#terms https://datamarket.azure.com/dataset/bing/searchweb#terms
- jpsaroudakis 10y ago
  Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services https://www.microsoft.com/cognitive-services. All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pricing plan if you want to extend the monthly quota. Some folks mentioned this page: https://datamarket.azure.com/dataset/bing/search https://datamarket.azure.com/dataset/bing/search. I want to clarify that the page includes the old Bing Search APIs that are still in use, but will be deprecated in the future. You can send feedback or questions here: https://cognitive.uservoice.com/ https://cognitive.uservoice.com/ [Edit] Here is also a blog post with useful resources: https://blogs.msdn.microsoft.com/bingdevcenter/2016/04/13/recap-bing-at-build-2016/ https://blogs.msdn.microsoft.com/bingdevcenter/2016/04/13/re...
  - SwellJoe 10y ago
    I keep checking out the Bing API every couple months, hoping for sort-by options (I need last changed date, most recent first). Any hope of ever seeing sort-by-date from Bing? Google has it, I don't think anyone else does.
  - tokensimian 10y ago
    Hi, just a quick bit of feedback -- 1k API calls / month is so low as to be a nonstarter. Of the half dozen hobby (non-commercial) projects I am currently working on, this amount is tragically low. Per day would give me pause. I don't mean to rain on your efforts, but I would personally never consider using an API with such limits, even for trivial hobby projects.
    - trakout 10y ago
      Agreed -- 1k/mo is not worth the initial time investment.
    - deleted 10y ago
      [deleted]

by bake 10y ago

SwellJoe 10y ago

I'm not seeing how to make GCS perform a full web search. It won't allow creation of a search without at least one schema defined, and I need schemaless data (or rather, I need the whole web, including a variety of schemas). Am I missing something?

jrochkind1 10y ago

Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous URL: https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57-A49D98D29A44 https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57... . I think that's what you're talking about? Note the link on that page to "Bing API FAQ" -- is broken. If this wasn't MS, the poor docs would make me think it was surely a terrible product.

woodruffw 10y ago
>Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. You can, refer to this SO answer: http://stackoverflow.com/a/11206266/3354209 http://stackoverflow.com/a/11206266/3354209
- jrochkind1 10y ago
  Thanks. One of the comments on that SO answer from Aug 2015 says: > WARNING: we did development using the free version, but to upgrade to the paid version (to do more than 100 searches), google forces you to turn off the "search the entire web but emphasize included sites" I remain not very confident that you can really search the whole web with google custom search, or if you can that it's not some kind of a loophole that google might close without warning. But if someone has actually done this succesfully, with a paid account for more than 100/queries a day, I would definitely be interested in hearing about it!
  - hantusk 10y ago
    I am doing this successfully with 1,000 queries in a day, and the limit it 10k queries per day (pay per 1,000 queries) The billing system is confusing. You just have to sign up for billing, and you can raise the limit. The upgrade in the google system, is only if you want site search with no ads.
  - ellyagg 10y ago
    Yes, you can search the whole web. We have been using a pool of credit cards to make up to a million requests per month.
phatrice 10y ago
You can check out the new home of Bing Search API here: https://www.microsoft.com/cognitive-services/en-us/bing-web-search-api https://www.microsoft.com/cognitive-services/en-us/bing-web-... You can get key right from there and an API console.
- jrochkind1 10y ago
  Thanks! Should I chalk up to conspiracy that this page is nearly impossible to find on, um, Google, searching terms that you would reasonably use to find it? But if you have an in with MS, it would probably help an awful lot if they removed the old pages with broken links, unhelpful content, and possibly out of date inaccurate content too. The page you link to has "coming soon" for pricing for Search (which was kind of confusing to find) -- does that mean that the pricing info on the page I found is not accurate, or soon won't be, with no indication on the page I found that that will be the case? MS is really really bad at documenting this service. Only because it's MS am I willing to entertain that the quality of docs may not reflect the quality of service, generally I'd give up on something that markets and explains itself so poorly and figure that if they can't get that right, they probably didn't get the service right either.
- bake 10y ago
  Here's the old home, with pricing laid out (missing from the new one): https://datamarket.azure.com/dataset/bing/searchweb#terms https://datamarket.azure.com/dataset/bing/searchweb#terms
bake 10y ago
Here's the older link to the Bing Web Search API -- while the new page doesn't list actual prices, this one does in nice detail: https://datamarket.azure.com/dataset/bing/searchweb#terms https://datamarket.azure.com/dataset/bing/searchweb#terms
jpsaroudakis 10y ago
Hi all, I am a program manager in the Bing API team. I am providing some pointers for people that are interested. We released our latest Search APIs in March via the Microsoft Cognitive Services site. You can find them here: https://www.microsoft.com/cognitive-services https://www.microsoft.com/cognitive-services. All APIs are currently offered for free with limited calls per month (for most APIs,the quota is 1,000 calls per month). We will announce soon a pricing plan if you want to extend the monthly quota. Some folks mentioned this page: https://datamarket.azure.com/dataset/bing/search https://datamarket.azure.com/dataset/bing/search. I want to clarify that the page includes the old Bing Search APIs that are still in use, but will be deprecated in the future. You can send feedback or questions here: https://cognitive.uservoice.com/ https://cognitive.uservoice.com/ [Edit] Here is also a blog post with useful resources: https://blogs.msdn.microsoft.com/bingdevcenter/2016/04/13/recap-bing-at-build-2016/ https://blogs.msdn.microsoft.com/bingdevcenter/2016/04/13/re...
- SwellJoe 10y ago
  I keep checking out the Bing API every couple months, hoping for sort-by options (I need last changed date, most recent first). Any hope of ever seeing sort-by-date from Bing? Google has it, I don't think anyone else does.
- tokensimian 10y ago
  Hi, just a quick bit of feedback -- 1k API calls / month is so low as to be a nonstarter. Of the half dozen hobby (non-commercial) projects I am currently working on, this amount is tragically low. Per day would give me pause. I don't mean to rain on your efforts, but I would personally never consider using an API with such limits, even for trivial hobby projects.
  - trakout 10y ago
    Agreed -- 1k/mo is not worth the initial time investment.
  - deleted 10y ago
    [deleted]