3 ms·
robots.txt is a convention for those who want to be good 'web citizens' rather than legally binding. It does absolutely nothing to stop someone who ignores you
by blihp 5y ago
robots.txt is a convention for those who want to be good 'web citizens' rather than legally binding. It does absolutely nothing to stop someone who ignores your wishes. For example, there are tons of bots that ignore robots.txt entirely or even go straight for the thing (i.e. 'hey, thanks for telling us where to look!') you're telling them to avoid in robots.txt. While copyright is a mechanism that can be used if you can make the case, and have the means, it will only work for entities that have something to lose and are within a jurisdiction where it matters.