2 ms·
Just like Google is crawling the whole internet (hitting your server a million times a day) and then, with a straight face, will plug a captcha in your face if
by visarga 3y ago
Just like Google is crawling the whole internet (hitting your server a million times a day) and then, with a straight face, will plug a captcha in your face if you dare search more than 3 times with a quoted string or non-trivial terms. Forget about doing a few million searches to bootstrap your dataset, Google was always hostile - both in access controls and pricing. You want results past 100 or 1000 mark? never possible, not even 20 years ago. But they say 1 bazzilion web pages in their index match your search.
tl;dr Google crawls you, you can't crawl Google. How is that fair? They built their empire on our brain outputs but won't share theirs.
- LadyCailin 3y agoGoogle respects robots.txt though, which is more then they need to. If you put data out in public, what do you expect? If you don’t like Google crawling you though, just restrict them. That’s usually to your own detriment though, but I won’t judge you if you’re into self-flagellation. Just don’t think you’re holier than me because you are.
- visarga 3y agoIt's ok, the wheel is turning and now reality has come to bite them in the ass. CommonCrawl supplied the text and LLMs replace their index, for a large number of requests. A new crop of search engines like phind.com and perplexity.ai have better than Google results.
- richardw 3y agoMy memory of the early ish days is that Facebook heavily leveraged Google contacts (which was allowed by Google) to discover friends, and then blocked others from doing the same. Is that correct or can someone offer better info?