Something I've never understood is what to do when the units of work are known up front (like, I have to process these 1000 items with 4 stages), but the stages

Something I've never understood is what to do when the units of work are known up front (like, I have to process these 1000 items with 4 stages), but the stages include network requests. So instead of having a fixed setpoint like 0 in the piece, I want smallest processing time. Can this be solved with PID and control theory? For example, say I want to list all the keys in an s3 bucket that start with any of 1000 given prefixes. I can do a lot of calls per second, but of course have limited bandwidth and limited cpu to process the incoming responses, and sometimes s3 can say "too many requests" if other users are querying the same prefixes. How do I do this as fast as possible?