Yes. I used lib curl's multi interface on one $40/m server around that time. Indeed at any scale the rate limiting becomes the main bottleneck, mainly because a

Yes. I used lib curl's multi interface on one $40/m server around that time. Indeed at any scale the rate limiting becomes the main bottleneck, mainly because a lot of sites are concentrated on certain hosts. Speed isn't the problem and multiple servers aren't really needed.