Rand replied that they don't use AWS for crawling. I would imagine the main bottleneck, by far is the I/O with reading/writing to the database clusters and sea

Rand replied that they don't use AWS for crawling. I would imagine the main bottleneck, by far is the I/O with reading/writing to the database clusters and search index. Crawling is relatively cheap, it's storing that data that's hard.