6 ms·
If you fork inside an app server, such as mod_python, you will fork the entire parent process (apache!). This could happen by calling something like os.system("
by patio11 17y ago
If you fork inside an app server, such as mod_python, you will fork the entire parent process (apache!). This could happen by calling something like os.system("mv foo bar") from a python application.
I nominate this post as the most distressingly important bit of information I've ever received at 2:43 AM in the morning.
Now the question: what can I do in Ruby to avoid the four calls a second or so I'm currently making to system(big_command_to_invoke_imagemagick) ?
- polvi 17y agoThe solution to use an image processing library such as RMagick, http://rmagick.rubyforge.org/ http://rmagick.rubyforge.org/
- tptacek 17y agoCalling into RMagick/ImageMagick from inside the request/response cycle is probably even worse than shelling out, because ImageMagick does grievous damage to your runtime.
- polvi 17y agoI guess it all depends how you design it and what you are doing. I would have to agree with others, the out of request cycle image processing solutions are definitely the right way to go overall.
- Pistos2 17y agoLast I tried RMagick, it leaked significantly. Definitely not something I want to use in a long-lived process. I remember having to fork to use it, to work around the memory leak. If you don't need the fancier operations, there are lighter image manipulation gems out there that do just the basics, but without leaking. e.g. ImageScience http://seattlerb.rubyforge.org/ImageScience.html http://seattlerb.rubyforge.org/ImageScience.html
- boundlessdreamz 17y agoUse a queue. You should never be doing time consuming method calls inside a controller anyway.
- idlewords 17y agoYes, yes, yes. Beanstalkd is an easy one to set up, for example, with good Rails integration.
- swombat 17y agoI'd use something like DelayedJob and send_later the call to your image processing stuff, that way the forking happens out of the request path, at least.
- patio11 17y agoYou just described my exact setup. However, my understanding is that Delayed::Job's worker threads have a full Rails environment in them, and if this blog post is correct and I am indeed forking that entire Rails process for every call out to ImageMagick, my vague recollections of what a fork entails suggest to me that the Ghosts of C Programmers Past are going to visit a terrible vengeance upon me.
- swombat 17y agoOnly if you run out of memory. But with DJ at least you should be forking only one call at a time, rather than multiple, like you might from the controller itself. So although you'll end up using more memory, it'll only be one extra rails process, not 4. May not still be ideal... interested to hear other people's ideas.
- simonw 17y agoUse http://github.com/documentcloud/cloud-crowd http://github.com/documentcloud/cloud-crowd
- jeremyw 17y agoThe fork+exec is efficient. The blog post compares things without units. Forks (principally page table copies w/copy-on-write in effect) are measured in microseconds and the exec is your standard binary startup time. While you don't want to put a synchronous fork/exec in the way of 5,000 reqs/sec, it will be a trivial part of your asynchronous imagemagick processing. At scale, you might care about the imagemagick startup latency, but not the forking.
- deleted 17y ago[deleted]
- lincolnq 17y agoFour forks per second is basically nothing. This article is blowing it all out of proportion. You can't sustain forking per web request on a really large site but at this scale it's not going to matter. The author is being stupid: the size of the process that you're forking doesn't really matter (it might start to matter if you didn't call exec() or exit() right after you forked, but that's not the case: you're just execing another program, which replaces the current process in memory). VERY little is copied; fork is defined to have copy-on-write semantics for the process's address space.