4 ms·
Does Curl performance really matter? i.e. if it's too performant, doesn't that increase the odds your spider is blocked? Of course, if you're sharding horizon
by jgalt212 3y ago
Does Curl performance really matter? i.e. if it's too performant, doesn't that increase the odds your spider is blocked? Of course, if you're sharding horizontally across targets, then any performance increase is appreciated.
- j16sdiz 3y agolibcurl is the backend for many (RESTful) API library. Improving upload throughput to S3 bucket would be great, right?
- backum 3y ago[flagged]
- zamadatix 3y agoWhat if you're not using curl as a spider? Even if you are I'd recommend some other spider design which doesn't rely on the performance of curl to set the crawling rate.