And of course, CURL deals with multiple protocols as the OP mentioned. I've used libcurl for sending emails and FTP but mainly for HTTP. Using the multi interf

And of course, CURL deals with multiple protocols as the OP mentioned. I've used libcurl for sending emails and FTP but mainly for HTTP. Using the multi interface it's relatively trivial (a few hundred lines of C) to fetch using hundreds or thousands of concurrent connections. There's even some courtesy opts for rate limiting per host. It's so good he/they built c-ares for asynchronous DNS lookups, IIRC something that wasn't so readily available in years gone by.