> Anyone can index the web Most, if not all, websites will block your bot after a couple of request even if it respects robots.txt. Also they seem to have dif

> Anyone can index the web Most, if not all, websites will block your bot after a couple of request even if it respects robots.txt. Also they seem to have different rules for GoogleBot and different for yours.