4 ms·respecting robots.txt and using publicly declared ua come to mind.by avip 7y agorespecting robots.txt and using publicly declared ua come to mind.