4 ms·
Why are so many search engines trying to piggy back off other search engines? Can they not crawl sites themselves and cut out the extra layer? In a sea of bad w
by arminiusreturns 6y ago
Why are so many search engines trying to piggy back off other search engines? Can they not crawl sites themselves and cut out the extra layer? In a sea of bad websites I feel like if you started with a whitelist approach it could be a good basis.
- hosteur 6y agoCrawling is difficult these days as recaptcha and other measures keep bots out if you are not google or MS
- NetOpWibby 6y agoThat is quite unfortunate. How the hell are new original search engines supposed to compete?
- jameshush 6y agoI’ve worked in this space for over two years. They can’t. It’s simply not possible. The only feature that’s feasible to offer is privacy. You’ll never be able to catch up to Google or Bing with crawling.
- NetOpWibby 6y agoIn your opinion, do you think a new original search engine can survive in this landscape? Of course it would always be behind Google and Bing by about 15 years. I've been reseasrching some ideas.
- crocodiletears 6y agoIt's a pretty herculean task, moreso no than ever before. On top of the sheer volume of the internet, and the technical challenges of performant search, many sites are hostile to unknown scrapers, and automation.