3 ms·
> It is now significantly easier to build a search engine with its own index and crawl the entirety of the Internet than it was ten years ago. ReCAPTCHA, hCapt
by slezyr 6y ago
> It is now significantly easier to build a search engine with its own index and crawl the entirety of the Internet than it was ten years ago.
ReCAPTCHA, hCaptcha, CloudFlare definitely don't make it easier to crawl the web than it was ten years ago
- echelon 6y agoThankfully (1) there just aren't that many useful websites anymore. Most content lives in walled gardens. Negotiate a few deals to get around CloudFlare for those select few and you're set. [1] I'm being sarcastic.
- pbhjpbhj 6y ago'Many a true word was said in jest' as the aphorism goes. IME useful information has become somewhat concentrated into nodes, instead of trawling blogs for info it's all on stackoverflow/Twitter/whatever (depending on subject).
- echelon 6y agoFor certain populations of data and authors, it's true that centralization aids in discovery, distribution, discourse, maintenance, etc. Unfortunately many of these channels are hooked up to hyper-growth valuation needs and this necessitates volume over quality. Look at Quora, Pinterest, Medium, and LinkedIn. And I haven't even mentioned the biggest offenders. I don't think I'm better served in 2020 than I was in 2008.