4 ms·
I believe DuckDuckGo does (or at least they did) this with Bing. Starting a new scraper at a scale that users would need to be useful for what they're used to i
by jjice 3y ago
I believe DuckDuckGo does (or at least they did) this with Bing. Starting a new scraper at a scale that users would need to be useful for what they're used to is such a huge jump. I'm sure if Kagi continue to grow they'd prioritize their own scraping too, but that's just not feasible at first.
- giancarlostoro 3y agoBack in the day I'd suggest doing it via Alexa top sites, but now that Alexa is gone, I'm not sure what strategy I would use, but I would want to hit sites that are like the "top 10000 most popular" first, and scrape every inch I could.
- elaus 3y agoI think Kagi is going in the opposite direction: https://blog.kagi.com/small-web https://blog.kagi.com/small-web They try to highlight small, personal websites instead of the big mainstream sites. (This was a HN submission 2 weeks ago)
- giancarlostoro 3y agoI saw that, but that's kind of useless when you kind of want something like SO or similar results, something Google keeps failing at.