5 ms·
How do new search engines like Perplexity and Kagi avoid getting rate limited or hitting captchas as they build and maintain their indexes? Sites have to make
by ec109685 3y ago
How do new search engines like Perplexity and Kagi avoid getting rate limited or hitting captchas as they build and maintain their indexes?
Sites have to make exceptions for Google, but likely wouldn’t care enough to allow other search engines in.
- geoelectric 3y agoI’m fairly sure Kagi reuses Google and Bing search results, then de-craps and applies your customized preferences to them. I’m not sure it even runs its own spider.
- snewman 3y agoThis does not seem to be correct. I asked Kagi "does kagi build its own index". From the first result (on help.kagi.com): > Kagi Search includes anonymized requests to traditional search indexes including Brave, as well our own non-commercial index (Teclis), news index (TinyGem), and an AI for instant answers. Teclis and TinyGem are a result of our crawl through millions of domains, focusing primarily on non-commercial, high-quality content. [Edit] I guess it is possible that Google or Bing are hiding under "traditional search indexes including Brave".
- JumpCrisscross 3y agoKagi sends “anonymized API calls to traditional search indexes like Google, Yandex, Mojeek and Brave, specialized search engines like Marginalia, and sources of vertical information like Wolfram Alpha, Apple, Wikipedia, Open Meteo, Yelp, TripAdvisor and other APIs,” and also maintains its own “web index (internal name - Teclis) and news index (internal name - TinyGem)” [1]. [1] https://help.kagi.com/kagi/search-details/search-sources.html https://help.kagi.com/kagi/search-details/search-sources.htm...
- aitchnyu 3y agoUmm, So Google and MS can leave Kagi high and dry if it eats into their revenue?
- Moldoteck 3y agothey can, on the other hand, kagi launched recently something that would improve the crawling process
- geoelectric 3y agoThanks for correcting me on this. My take (as a subscriber) is still that the majority of information I'm seeing is assimilated and distilled from the more mainstream sources, but I'm possibly underestimating how much comes from their own indexes. I will admit that for news I tend to use the !gn bang to jump straight to Google News as it tends to be a more complete feed than Kagi's news tab, so wasn't thinking about TinyGem. But I'd also forgotten completely about Kagi's "Small Web" initiative for indexing non-commercial content that's not interlinked enough for other engines to prioritize it. I believe that's what Teclis (and possibly TinyGem as well) concentrates on. IIRC, the few times I tried routing searches into that index I didn't find much, so I set it aside, but that's probably because I'm usually searching for more widely-consumed content.
- mminer237 3y agoI believe Kagi just pays to use Google's and Microsoft's indexes.
- bigyikes 3y agoGoogle sells their index?? (Microsoft famously sells theirs to DuckDuckGo, right?)
- manquer 3y agoBing index has always been licensable , Yahoo was using it too, and they have other customers . I don’t think Google has a publicly known licensing deal ,
- ilaksh 3y agoThey sell API access to Google searching.
- mrweasel 3y agoMicrosoft sells Bing to a ton of search engines, it has been the go to for everyone who wanted to run a search engine. Ecosia have also been relying on Bing, but recently announced that they've now added Google as well.
- metabagel 3y agoI don’t think that’s correct.
- migrane3 3y ago[dead]
- neocritter 3y agoKagi's internal index seems to focus on sites that wouldn't necessarily block spiders. https://help.kagi.com/kagi/search-details/search-sources.html https://help.kagi.com/kagi/search-details/search-sources.htm... I've crawled small web sites (even hosted on major providers) to archive them and never hit a wall.