4 ms·
Google respects robots.txt though, which is more then they need to. If you put data out in public, what do you expect? If you don’t like Google crawling you tho
by LadyCailin 3y ago
Google respects robots.txt though, which is more then they need to. If you put data out in public, what do you expect? If you don’t like Google crawling you though, just restrict them. That’s usually to your own detriment though, but I won’t judge you if you’re into self-flagellation. Just don’t think you’re holier than me because you are.
- visarga 3y agoIt's ok, the wheel is turning and now reality has come to bite them in the ass. CommonCrawl supplied the text and LLMs replace their index, for a large number of requests. A new crop of search engines like phind.com and perplexity.ai have better than Google results.