4 ms·

I tried to build a scraper in Rust just a few days ago and got stuck trying to concurrency limit my calls (the website I was scraping appropriately had a rate l

I tried to build a scraper in Rust just a few days ago and got stuck trying to concurrency limit my calls (the website I was scraping appropriately had a rate limit). I couldn’t figure out how to get tokio::stream/tokio_stream to work. Does this fix that problem?

ashish01 6y ago
You can limit the numbers pf concurrent calls using a semaphore and acquiring it before making a http request.
matsche 6y ago
Yes, you can either limit how many request can be send concurrently or enforce a fixed or random delay in between consecutive requests. Basically this a just a futures_timer::Delay [0] that is reset after each request, which is non blocking. [0] https://docs.rs/futures-timer/3.0.2/futures_timer/ https://docs.rs/futures-timer/3.0.2/futures_timer/
- singhrac 6y ago
  How do you set the concurrency limit? I'm down to use your framework (thanks!), just curious how you implement it. I couldn't get Stream::buffered to work correctly.
  - Matthias247 6y ago
    You can use `Futures::StreamExt::buffer_unordered` for this. [1] is an example where I used it for benchmark which creates a certain amount of QUIC connections at a time. But you can also spawn all tasks upfront via `tokio::spawn`, and let them wait on a `tokio::sync::Semaphore` before making the request. The drawback of this is that you might allocate more memory for tasks upfront - but if you don't have an extremely high number it might not matter. [1] https://github.com/quinn-rs/quinn/blob/de627437bc7d836564c3666c8d2885110b74833d/bench/src/bulk.rs#L105-L124 https://github.com/quinn-rs/quinn/blob/de627437bc7d836564c36...
    - singhrac 6y ago
      This is a late comment, but this helped me out a lot. Thank you for the detailed explanation and code example, I appreciate it.
michael_j_ward 6y ago
I assume the website has something like "requests per second" quota, which then you'd want the `governor` crate [0]. It was recently used to implement rate limiting middleware for both tide[1] and actix[2] [0] https://docs.rs/governor/0.3.1/governor/_guide/index.html https://docs.rs/governor/0.3.1/governor/_guide/index.html [1] https://github.com/ohmree/tide-governor https://github.com/ohmree/tide-governor [2] https://github.com/AaronErhardt/actix-governor https://github.com/AaronErhardt/actix-governor

ashish01 6y ago

You can limit the numbers pf concurrent calls using a semaphore and acquiring it before making a http request.

matsche 6y ago

Yes, you can either limit how many request can be send concurrently or enforce a fixed or random delay in between consecutive requests. Basically this a just a futures_timer::Delay [0] that is reset after each request, which is non blocking. [0] https://docs.rs/futures-timer/3.0.2/futures_timer/ https://docs.rs/futures-timer/3.0.2/futures_timer/

singhrac 6y ago
How do you set the concurrency limit? I'm down to use your framework (thanks!), just curious how you implement it. I couldn't get Stream::buffered to work correctly.
- Matthias247 6y ago
  You can use `Futures::StreamExt::buffer_unordered` for this. [1] is an example where I used it for benchmark which creates a certain amount of QUIC connections at a time. But you can also spawn all tasks upfront via `tokio::spawn`, and let them wait on a `tokio::sync::Semaphore` before making the request. The drawback of this is that you might allocate more memory for tasks upfront - but if you don't have an extremely high number it might not matter. [1] https://github.com/quinn-rs/quinn/blob/de627437bc7d836564c3666c8d2885110b74833d/bench/src/bulk.rs#L105-L124 https://github.com/quinn-rs/quinn/blob/de627437bc7d836564c36...
  - singhrac 6y ago
    This is a late comment, but this helped me out a lot. Thank you for the detailed explanation and code example, I appreciate it.

michael_j_ward 6y ago

I assume the website has something like "requests per second" quota, which then you'd want the `governor` crate [0]. It was recently used to implement rate limiting middleware for both tide[1] and actix[2] [0] https://docs.rs/governor/0.3.1/governor/_guide/index.html https://docs.rs/governor/0.3.1/governor/_guide/index.html [1] https://github.com/ohmree/tide-governor https://github.com/ohmree/tide-governor [2] https://github.com/AaronErhardt/actix-governor https://github.com/AaronErhardt/actix-governor