Basically we had the same infrastructure back in the day at Trulia - a dedicated “bot cluster” We had two charges from management: let google crawl as fast as

Basically we had the same infrastructure back in the day at Trulia - a dedicated “bot cluster” We had two charges from management: let google crawl as fast as it possibly can, and don’t let it disrupt users.