We use an auto-throttling algorithm for the crawler to ensure we are a 'polite' crawler! We also send a unique user-agent with every request, so any host can e

We use an auto-throttling algorithm for the crawler to ensure we are a 'polite' crawler! We also send a unique user-agent with every request, so any host can easily block us if required - luckily, this has never happened so far.