4 ms·Yes, pretty sure this is the way to go. You can even tell which bots are allowed to index and not.by TheChaplain 5y agoYes, pretty sure this is the way to go. You can even tell which bots are allowed to index and not.