There are robots which follow robots.txt but which site operators prefer to block or content-restrict, for various reasons. See eg the annotated wikipedia robot

There are robots which follow robots.txt but which site operators prefer to block or content-restrict, for various reasons. See eg the annotated wikipedia robots.txt me tioned upthread.