4 ms·

In your about page, I see you are using Bing's API. I didn't even know Bing has a search API that everyone can use! How much do you have to pay them for this?

by TekMol 4y ago

gdcbe 4y ago
Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to. Correct me if I’m wrong though, but I’m pretty certain that all other search engines in the same category use one of these as their backend. Eg I’m pretty certain that counts for duckduckgo as well.
- Santosh83 4y ago
  I think Yandex have their own index and some others too like Marginalia, but the latter couldn't be called "in the same category" as the other three.
  - kkielhofner 4y ago
    Yandex also suffered a security breach and their source code is available[0] although utilizing it in any way is ethically and legally dubious (at best). [0] - https://arstechnica.com/information-technology/2023/01/massive-yandex-code-leak-reveals-russian-search-engines-ranking-factors/amp/ https://arstechnica.com/information-technology/2023/01/massi...
- antonok 4y ago
  Brave Search has its own independent index too - https://brave.com/brave-search-beta/ https://brave.com/brave-search-beta/
- quectophoton 4y ago
  > Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to. Money and resources and a dominant-enough position so that your crawlers are not blocked by websites. Unfortunately.
  - rezonant 4y ago
    There's definitely gatekeeping on websites but having done a bunch of crawler work I can say you'd be surprised how rarely a site will outright block you if you just do the right things: have an identifiable user agent with a working URL that explains what your crawler does, respect robots.txt, implement polite crawling. As for actually being able to crawl the whole thing though, yeah it's stupidly expensive :-/
- daoudc 4y ago
  Mwmbl has its own index but it's orders of magnitude smaller than commercial search engines.
- schemescape 4y ago
  What about Common Crawl? https://commoncrawl.org/ https://commoncrawl.org/
anthonyhn 4y ago
It's $4/1000 queries, but the rate is increasing in May to $18/1000 queries. The Bing API is available through Azure.
- danuker 4y ago
  I hope the author knows this, and won't be surprised by a bill more than 4x larger.
  - riidom 4y ago
    That comment was written by the author.
    - deleted 4y ago
      [deleted]
    - danuker 4y ago
      Oh! I am scatterbrained.
- flas9sd 4y ago
  quite a price hike. Bad for the sustainability of the other portals I use that are backed by the bing index. Will either increase their pricing or efforts on monetization. Is it easy for you to rely on more search index providers, what are your options?
  - anthonyhn 4y ago
    >Is it easy for you to rely on more search index providers, what are your options? I have a few options: * Switch index providers. For example, Mojeek has an index with 6bn pages and has a web search API; may be more sustainable to switch to their index in the long run. * Build my own index. This is my preferred option and I've already started to work on this. * Look for funding sources to offset the price hike.
    - 4y ago

In your about page, I see you are using Bing's API. I didn't even know Bing has a search API that everyone can use! How much do you have to pay them for this?

by TekMol 4y ago

In your about page, I see you are using Bing's API. I didn't even know Bing has a search API that everyone can use! How much do you have to pay them for this?

gdcbe 4y ago

Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to. Correct me if I’m wrong though, but I’m pretty certain that all other search engines in the same category use one of these as their backend. Eg I’m pretty certain that counts for duckduckgo as well.

Santosh83 4y ago
I think Yandex have their own index and some others too like Marginalia, but the latter couldn't be called "in the same category" as the other three.
- kkielhofner 4y ago
  Yandex also suffered a security breach and their source code is available[0] although utilizing it in any way is ethically and legally dubious (at best). [0] - https://arstechnica.com/information-technology/2023/01/massive-yandex-code-leak-reveals-russian-search-engines-ranking-factors/amp/ https://arstechnica.com/information-technology/2023/01/massi...
antonok 4y ago
Brave Search has its own independent index too - https://brave.com/brave-search-beta/ https://brave.com/brave-search-beta/
quectophoton 4y ago
> Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to. Money and resources and a dominant-enough position so that your crawlers are not blocked by websites. Unfortunately.
- rezonant 4y ago
  There's definitely gatekeeping on websites but having done a bunch of crawler work I can say you'd be surprised how rarely a site will outright block you if you just do the right things: have an identifiable user agent with a working URL that explains what your crawler does, respect robots.txt, implement polite crawling. As for actually being able to crawl the whole thing though, yeah it's stupidly expensive :-/
daoudc 4y ago
Mwmbl has its own index but it's orders of magnitude smaller than commercial search engines.
schemescape 4y ago
What about Common Crawl? https://commoncrawl.org/ https://commoncrawl.org/

anthonyhn 4y ago

It's $4/1000 queries, but the rate is increasing in May to $18/1000 queries. The Bing API is available through Azure.

danuker 4y ago
I hope the author knows this, and won't be surprised by a bill more than 4x larger.
- riidom 4y ago
  That comment was written by the author.
  - deleted 4y ago
    [deleted]
  - danuker 4y ago
    Oh! I am scatterbrained.
flas9sd 4y ago
quite a price hike. Bad for the sustainability of the other portals I use that are backed by the bing index. Will either increase their pricing or efforts on monetization. Is it easy for you to rely on more search index providers, what are your options?
- anthonyhn 4y ago
  >Is it easy for you to rely on more search index providers, what are your options? I have a few options: * Switch index providers. For example, Mojeek has an index with 6bn pages and has a web search API; may be more sustainable to switch to their index in the long run. * Build my own index. This is my preferred option and I've already started to work on this. * Look for funding sources to offset the price hike.
  - 4y ago