5 ms·
posix-spawn: a faster fork+exec for ruby
- tmm1 16y agoHere's the full announcement: https://gist.github.com/e0d36c881e6efc57514b https://gist.github.com/e0d36c881e6efc57514b
- Groxx 16y agoWow, check out the Linux vs OSX graphs. Anyone care to enlighten me as to why they are that different? The worst OSX fork + exec call is more than 2x faster than the fastest fork + exec on Linux, and over 10x faster when the memory increases past ~300MB. Given that OSX's isn't dependent on page table size, I'm guessing it just always does copy-on-write... but that's a pretty un-knowledgeable guess.
- tmm1 16y agoThe reason fork is so slow on Linux is because the default page size is 4k. This means for a 300MB process, 76800 page table entries have to be copied during the fork. There are ways to enable HugeTBL on linux to increase the page size, which can improve fork performance as well. See http://stackoverflow.com/questions/2731531/faster-forking-of-large-processes-on-linux/2874009#2874009 http://stackoverflow.com/questions/2731531/faster-forking-of..., http://linuxgazette.net/155/krishnakumar.html http://linuxgazette.net/155/krishnakumar.html and http://sourceforge.net/projects/libhugetlbfs/ http://sourceforge.net/projects/libhugetlbfs/ Does anyone know how fork is implemented on BSD/OSX and why it doesn't exhibit the same characteristics?
- deleted 16y ago[deleted]
- akent 16y agoBSD / OS X has native support for superpages. From version 2.6.38, Linux will get native huge page / superpage support too. http://lwn.net/Articles/423584/ http://lwn.net/Articles/423584/
- deleted 16y ago[deleted]
- Andys 16y agoIn FreeBSD, it is extent-based, so a 300MB process might actually only have a couple dozen maps - heap, stack, program binary, and whatever dynamically loaded libraries. I'd be shocked if it wasn't the same in linux.
- andrewvc 16y agoThis is awesome, Aman's one of my favorite rubyists, I can't say I'm surprised to find out he's behind this. If anyone's curious how processes/threads are represented internally on modern linux kernels, I wrote an article on that here: http://blog.andrewvc.com/fork2-is-dead-long-live-fork-i-mean-clone2 http://blog.andrewvc.com/fork2-is-dead-long-live-fork-i-mean...
- tmm1 16y agoThis is a great article. I came across it while hacking on posix-spawn and it was quite useful.
- dsghjrj 16y agoClick on our website: ╭══════════════╮ http://p.gs/962x2 http://p.gs/962x2 ╰══════════════╯ ╭══════════════╮ http://p.gs/962x2 http://p.gs/962x2 ╰══════════════╯ Website wholesale various fashion shoes, such as Nike, Jordan, prada, also includes the jeans, shirt, bags, hats and decoration. Personality manufacturing execution systems (Mes) clothing, Grab an eye bag coat + tide bag Air jordan(1-24)shoes $30 Handbags(Coach l v f e n d i d&g) $35 Tshirts (Polo ,ed hardy,lacoste) $15 Jean(True Religion,ed hardy,coogi) $30 Sunglasses(Oakey,coach,gucci,A r m a i n i) $15 New era cap $12 Bikini (Ed hardy,polo) $20 accept paypal and free shipping ╭══════════════╮ http://p.gs/962x2 http://p.gs/962x2 ╰══════════════╯ ╭══════════════╮ http://p.gs/962x2 http://p.gs/962x2 ╰══════════════╯ ╭══════════════╮ http://p.gs/962x2 http://p.gs/962x2 ╰══════════════╯ ╭══════════════╮ http://p.gs/962x2 http://p.gs/962x2 ╰══════════════╯ ╭══════════════╮ http://p.gs/962x2 http://p.gs/962x2 ╰══════════════╯ ╭══════════════╮ http://p.gs/962x2 http://p.gs/962x2 ╰══════════════╯ ╭══════════════╮ http://p.gs/962x2 http://p.gs/962x2 ╰══════════════╯ ╭══════════════╮ http://p.gs/962x2 http://p.gs/962x2 ╰══════════════╯
- kscaldef 16y agoI'm curious what the applications are where this is an important optimization. Even for the very large Linux processes, it looks like 1/10sec per fork. On OSX, we're talking milliseconds. What sort of application is forking so often that this matters?
- JoachimSchipper 16y agoIf you use a basic process-per-connection model (select/fork/accept), 10 forks per second is not that much. The "classic Unix" Ruby HTTP servers like Unicorn do just that. (Apache 1.3 and at least one 2.x worker module also work this way, but Apache uses some tricks like preforking and recycling workers to make this fast - I'm not sure the Ruby servers do, too.)
- tmm1 16y agoUnicorn pre-forks workers upfront. New processes are not created per request. posix-spawn is also only useful for fork+exec, where the new process starts executing a new binary. This is not the case in unicorn or resque, where the child processes actually need to be copies of the parent. We're currently using posix-spawn in albino and grit, which execute external commands like `pygmentize` and `git`
- JoachimSchipper 16y agoYes, you're right, of course. Sorry.
- rtomayko 16y agoOn github.com, many types of requests can result in one or more calls to git commands. Every ms we can shave there can have a pretty big impact on response time.