3 ms·
Does robots.txt not work? Is the issue that you want to be indexed but otherwise not have them use what they crawl for other stuff? If I'm interpreting this ri
by version_five 5y ago
Does robots.txt not work? Is the issue that you want to be indexed but otherwise not have them use what they crawl for other stuff?
If I'm interpreting this right, I agree that it's scummy of them to set up a kind of social contract where they can index your stuff and then decide actually they can do whatever they want with it. This is effectively what happened with news sites I think. But at this point, they are at the stage of reaping the rewards of the monopoly they built, so antitrust laws are unfortunately the likely recourse.
- calvcalv 5y agoExactly, I allow Google to crawl me so I can be Indexed, not bypassed.