3 ms·

JRuby has been invaluable at scaling out our web scraping. Specifically, we've managed to leverage its concurrency support very effectively. We're scraping ma

JRuby has been invaluable at scaling out our web scraping. Specifically, we've managed to leverage its concurrency support very effectively. We're scraping many millions of pages, so a single-threaded scraper wouldn't be practical. Since most of the time, a given thread is waiting on IO, we get our highest throughput with a few dozen scraper threads running at once. Trying to use multiple threads in MRI is pretty much a big mess. For whatever reason, it starts falling down after three or four. Ruby 1.9's fibers might be a good solution but it wasn't out of beta when we started this project. Using EventMachine with MRI might get us the same effect, but changing a relatively large synchronous codebase to work asynchronously is a big task for an uncertain benefit. With JRuby, we simply pointed our (synchronous) Ruby code to a event-driven HTTP client which uses futures to block each thread. Threads are certainly heavier-weight than some concurrency primitives, but with only a few dozen the JVM is great at holding all the relevant state. We've also been able to leverage tools in java.util.concurrent and use Scala actors with pretty great effect. JRuby (and the JVM) isn't without its faults but when it comes to concurrency it's the Ruby implementation to beat.

bodhi 17y ago
Just curious, for your MRI tests, were you using 1.8? You probably already know, but 1.8 has green threads but 1.9 threads are native. Someone mentioned in a sibling that 1.9 still has a GIL, thus making concurrency a bit painful. I wonder if it is released when waiting on IO?
- steveklabnik 17y ago
  MRI is Ruby 1.8.x, and YARV (or KRI) is Ruby 1.9.x. "MRI 1.9" is a misnomer.
- stephenjudkins 17y ago
  Yeah, MRI is Ruby 1.8. I imagine 1.9 would perform a great deal better. Since we're scraping entirely on single-CPU machines, the GIL probably wouldn't hurt us much. We rewrote the scheduler and our interface to the HTTP client using Scala, however, so now we're pretty wedded to the JVM. However, if we were to use Ruby 1.9, we would have to spend more time investigating HTTP libraries. From our experience, it's worth it to have workers threads NEVER block directly on IO. We saw a huge increase in reliability by having worker threads (1) send a request to an event-based client, running in its own thread and (2) block on a future with a set timeout, waiting for the callback. Having individual threads talk directly to the sockets, in either MRI or Java, wasn't a promising approach. If sockets got wedged (and they do, even using very reputable HTTP libraries) the event-based approach keeps on humming, while the blocking approach grinds to a halt. There exist several event-based HTTP clients for Java that all work well (though some significantly better than others). Compare that with Ruby, where we couldn't find anything that mature. That's not to mention all the concurrency primitives available in Scala, Clojure, and the underrated java.util.concurrent. Even if Ruby 1.9 might look much more competitive to JRuby in synthetic tests of concurrent performance, in the real world the Java ecosystem features tons of libraries that helps one write reliable, predictable code quickly.

bodhi 17y ago

Just curious, for your MRI tests, were you using 1.8? You probably already know, but 1.8 has green threads but 1.9 threads are native. Someone mentioned in a sibling that 1.9 still has a GIL, thus making concurrency a bit painful. I wonder if it is released when waiting on IO?

steveklabnik 17y ago
MRI is Ruby 1.8.x, and YARV (or KRI) is Ruby 1.9.x. "MRI 1.9" is a misnomer.
stephenjudkins 17y ago
Yeah, MRI is Ruby 1.8. I imagine 1.9 would perform a great deal better. Since we're scraping entirely on single-CPU machines, the GIL probably wouldn't hurt us much. We rewrote the scheduler and our interface to the HTTP client using Scala, however, so now we're pretty wedded to the JVM. However, if we were to use Ruby 1.9, we would have to spend more time investigating HTTP libraries. From our experience, it's worth it to have workers threads NEVER block directly on IO. We saw a huge increase in reliability by having worker threads (1) send a request to an event-based client, running in its own thread and (2) block on a future with a set timeout, waiting for the callback. Having individual threads talk directly to the sockets, in either MRI or Java, wasn't a promising approach. If sockets got wedged (and they do, even using very reputable HTTP libraries) the event-based approach keeps on humming, while the blocking approach grinds to a halt. There exist several event-based HTTP clients for Java that all work well (though some significantly better than others). Compare that with Ruby, where we couldn't find anything that mature. That's not to mention all the concurrency primitives available in Scala, Clojure, and the underrated java.util.concurrent. Even if Ruby 1.9 might look much more competitive to JRuby in synthetic tests of concurrent performance, in the real world the Java ecosystem features tons of libraries that helps one write reliable, predictable code quickly.