3 ms·You'd end up blocking all traffic then. When was was the last time you pulled robots.txt?by bdcravens 14y agoYou'd end up blocking all traffic then. When was was the last time you pulled robots.txt?