3 ms·Do you respect robots.txt? How can I block your crawlers?by xena 1y agoDo you respect robots.txt? How can I block your crawlers?skylerwiernik 1y agoDo you find this to be worse than googlebot somehow?