5 ms·

Rails threading nightmare

9 points by wheels 16y ago

bensummers 16y ago
This article seems to be more about the limitations of the standard Ruby interpreter, than about Rails. Switching to JRuby would be an easy solution to these woes. Most gems are compatible these days, and there's even experimental support for C extensions as a temporary solution during migration.
wheels 16y ago
I posted this when I hit similar problems and ended up largely isolating them to ActiveRecord. I have a class which I use for doing background processing in Ruby: class Future def initialize(postprocessor = nil, &finalize) @postprocessor = postprocessor @future = Thread.new do begin finalize.call rescue => ex warn "Exception in background thread: #{ex}" end end end def method_missing(method, *args, &block) data.send(method, *args, &block) end def to_s data.to_s end private def data @data ||= @postprocessor ? @postprocessor.call(@future.value) : @future.value end end However this was causing random 5 second delays in my Rails app -- even if I never called "data" (and hence never joined). I added the postprocessor so that I could pass in a lambda that does all of the ActiveRecord stuff back in the calling thread, which alleviated the problem. But in contrast to what bensummers said -- this is actually a Rails problem, not a Ruby problem. Ruby's threads suck, but it seems to be ActiveRecord's locking that's at the root of the problem.
pilif 16y ago
I might be old-fashioned, but I would not start any background processing in the context of the process that handles the request. Depending on the web server handling the request, your process (threads are subprocesses) can be killed at any time when it's not serving a request. That could either kill your threads or prevent the webserver from correctly shutting down the process or handling another request on the same process. I would hand the background jobs off to some daemon process or write instructions describing the job into some queue and then have a cron job process them. I would never expect to be able to keep anything running after request processing has finished.
patio11 16y ago
Rails typically scales via adding processes rather than threads, and you can do concurrency the same way. Rather than ever doing anything blocking in the HTTP request/response protocol, just offload your tasks to worker processes and give your clients the page layout with Javascript ready to poll for the interesting stuff. This works, is fairly simple to implement, and kicks concurrency to database programmers, who actually understand it. The only reason I hesitate to give this advice is that substantially all of my downtime can be traced to worker processes either failing silently or going wild and munching my swap until the server died. Productizing the use of them is a little tricky. Concrete recommendation: I like DelayedJob, Thomas likes the Redis queue.
steveklabnik 16y ago
> February 11, 2009 Not that threading is perfect. But really, if you're doing anything intensive, you should be switching to background job + polling, anyway. You don't want to starve your mongrels/unicorn workers! This isn't to say that your background job wouldn't have concurrency... but it doesn't need the whole Rails environment. Ruby threading has changed a lot since 2009. Still not perfect, but Rubinius is supposed to be removing the GIL...
ericb 16y ago
Seems like a flaky architecture that depends too much on the internals of the web server. I would launch a stand-alone ruby process for the communication and connect to it via Drb to get results. Note that this will not work if you have greater than about 40-100 things trying to connect to the object, but it is simple and will work pretty well at small scales-it is less hairy than threads. Alternately, you could put rabbitmq or a messaging queue in the middle, and look at EventMachine for the communicator process. That will scale nicely.
andrewcooke 16y ago
These are green threads, according to the article. That means that they must work cooperatively (they're not system threads). That means that if any of the threads blocks - say on reading a secondary web service - then all threads will block. That's because the thread switching is managed by the VM, not the kernel, so it has no control over kernel level blocking calls. I know nothing about Ruby, so perhaps I am misunderstanding, but this would explain what he is seeing and is consistent with what he describes.
tptacek 16y ago
Do not --- DO NOT --- thread long-running Rails applications. By doing so, you expose yourself to every possible runtime bug, every possible C-code extension bug, and every possible concurrency bug in all of the above plus the whole Rails stack. We had exactly the problem this post describes with Playbook, even without using threads. We suffered random MRI crashes for weeks and weeks. In particular, the SWIG bindings for Subversion (two great tastes that make me want to throw up in my mouth) would randomly corrupt memory, but only when hosted in a process servicing Rails requests. We solve this problem --- remarkably easily --- by creating a shim class in front of our SVN code, hosting the "real" SVN code in a sidecar process, and remoting the requests. In no other mainstream language is it as easy to remote requests; you can literally just Marshal.save({method => args}) and chuck that over a socket. It's even easy to pass exceptions back and forth. If you have a task-based architecture, there's already a Rails best practice solution: use DJ or Resque to factor tasks out into worker classes that run off job queues. Both DJ and Resque are extremely easy to set up. There's no reason to overthink this.

9 points by wheels 16y ago

This article seems to be more about the limitations of the standard Ruby interpreter, than about Rails. Switching to JRuby would be an easy solution to these woes. Most gems are compatible these days, and there's even experimental support for C extensions as a temporary solution during migration.

wheels 16y ago

I posted this when I hit similar problems and ended up largely isolating them to ActiveRecord. I have a class which I use for doing background processing in Ruby: class Future def initialize(postprocessor = nil, &finalize) @postprocessor = postprocessor @future = Thread.new do begin finalize.call rescue => ex warn "Exception in background thread: #{ex}" end end end def method_missing(method, *args, &block) data.send(method, *args, &block) end def to_s data.to_s end private def data @data ||= @postprocessor ? @postprocessor.call(@future.value) : @future.value end end However this was causing random 5 second delays in my Rails app -- even if I never called "data" (and hence never joined). I added the postprocessor so that I could pass in a lambda that does all of the ActiveRecord stuff back in the calling thread, which alleviated the problem. But in contrast to what bensummers said -- this is actually a Rails problem, not a Ruby problem. Ruby's threads suck, but it seems to be ActiveRecord's locking that's at the root of the problem.

pilif 16y ago

I might be old-fashioned, but I would not start any background processing in the context of the process that handles the request. Depending on the web server handling the request, your process (threads are subprocesses) can be killed at any time when it's not serving a request. That could either kill your threads or prevent the webserver from correctly shutting down the process or handling another request on the same process. I would hand the background jobs off to some daemon process or write instructions describing the job into some queue and then have a cron job process them. I would never expect to be able to keep anything running after request processing has finished.

patio11 16y ago

Rails typically scales via adding processes rather than threads, and you can do concurrency the same way. Rather than ever doing anything blocking in the HTTP request/response protocol, just offload your tasks to worker processes and give your clients the page layout with Javascript ready to poll for the interesting stuff. This works, is fairly simple to implement, and kicks concurrency to database programmers, who actually understand it. The only reason I hesitate to give this advice is that substantially all of my downtime can be traced to worker processes either failing silently or going wild and munching my swap until the server died. Productizing the use of them is a little tricky. Concrete recommendation: I like DelayedJob, Thomas likes the Redis queue.

steveklabnik 16y ago

> February 11, 2009 Not that threading is perfect. But really, if you're doing anything intensive, you should be switching to background job + polling, anyway. You don't want to starve your mongrels/unicorn workers! This isn't to say that your background job wouldn't have concurrency... but it doesn't need the whole Rails environment. Ruby threading has changed a lot since 2009. Still not perfect, but Rubinius is supposed to be removing the GIL...

ericb 16y ago

Seems like a flaky architecture that depends too much on the internals of the web server. I would launch a stand-alone ruby process for the communication and connect to it via Drb to get results. Note that this will not work if you have greater than about 40-100 things trying to connect to the object, but it is simple and will work pretty well at small scales-it is less hairy than threads. Alternately, you could put rabbitmq or a messaging queue in the middle, and look at EventMachine for the communicator process. That will scale nicely.

andrewcooke 16y ago

These are green threads, according to the article. That means that they must work cooperatively (they're not system threads). That means that if any of the threads blocks - say on reading a secondary web service - then all threads will block. That's because the thread switching is managed by the VM, not the kernel, so it has no control over kernel level blocking calls. I know nothing about Ruby, so perhaps I am misunderstanding, but this would explain what he is seeing and is consistent with what he describes.

tptacek 16y ago

Do not --- DO NOT --- thread long-running Rails applications. By doing so, you expose yourself to every possible runtime bug, every possible C-code extension bug, and every possible concurrency bug in all of the above plus the whole Rails stack. We had exactly the problem this post describes with Playbook, even without using threads. We suffered random MRI crashes for weeks and weeks. In particular, the SWIG bindings for Subversion (two great tastes that make me want to throw up in my mouth) would randomly corrupt memory, but only when hosted in a process servicing Rails requests. We solve this problem --- remarkably easily --- by creating a shim class in front of our SVN code, hosting the "real" SVN code in a sidecar process, and remoting the requests. In no other mainstream language is it as easy to remote requests; you can literally just Marshal.save({method => args}) and chuck that over a socket. It's even easy to pass exceptions back and forth. If you have a task-based architecture, there's already a Rails best practice solution: use DJ or Resque to factor tasks out into worker classes that run off job queues. Both DJ and Resque are extremely easy to set up. There's no reason to overthink this.