5 ms·

There's a good reason most Ruby and Python projects don't rely heavily on system calls: portability. If you care at all about portability, it's just easier to n

by mrshoe 17y ago

There's a good reason most Ruby and Python projects don't rely heavily on system calls: portability. If you care at all about portability, it's just easier to not hit the system calls directly, otherwise you'll have to detect host OS and make sure you're using the system calls properly. If threads are "out", then pre-fork is way out. Just look at the history of the Apache project. I realize this all happened before the RoR era, but Apache used to use a pre-fork MPM almost exclusively. In more recent years it has added the threaded MPM and the async MPM. They have also put in the work I mentioned above to achieve cross-platform compatibility. I'm just using Apache as an example here; I'm not suggesting that we should all use Apache. It's just funny to me to see a post that basically says, "All the stuff we've been doing for the last 5 years is out. We should be doing the same stuff they were doing 15 years ago, but in Ruby this time around instead of C." Maybe software trends are like music trends? Everything from 5 years ago is lame, but the stuff from 20 years ago is super groovy, man.

defunkt 17y ago
Counter example: nginx, which uses fork(), and seems to smoke Apache while offering features like binary reloading without dropping connections (not sure if Apache supports this but I seem to remember no - please correct me if I'm wrong). Portability is not always necessary - when talking about fork() and friends you're talking about trade offs. When I'm only ever deploying to Unix environments, I accept the lack of portability in exchange for features I value.
- mrshoe 17y ago
  That's not really a counter example. nginx does indeed fork off child processes, but it uses async I/O in each child process. It doesn't use the traditional pre-fork process-per-connection model that Apache's pre-fork MPM uses or that Unicorn is using.
  - defunkt 17y ago
    nginx relies heavily on system calls at the expense of portability.
    - mrshoe 17y ago
      Oh, you were referring to the syscalls/portability tradeoff. In that case it's still not really a counter example, because Apache made the same decision in its early versions (Unix only). And that's a reasonable decision to make given the tradeoffs, I think. The point that I was more upset with was the "threads are dead, pre-fork is the way to go" section of the article. However, thinking about it more now, even the syscalls vs. portability arguments presented here are another example of forgetting the past (or just never being aware of it). After all the work from the Python devs to encapsulate syscalls in the standard library and provide a portable API to them, Jacob Kaplan-Moss says, "I’m a bit dismayed to see [syscalls] relegated to the dusty corners of our shiny dynamic languages."
    - sovande 17y ago
      Hello!? nginx is a POSIX network application written in C. Of course it uses syscalls. As the previous poster pointed out you are also wrong in that nginx uses anything like a prefork model. It _may_ fork of a process or more if it detect that it is running on a SMP system to take advantage more than one CPU. But each process has a strictly async io architecture handling request/responses within a big loop.
      - gloob 17y ago
        The point of the post you are responding to is that it uses syscalls. I'm not entirely certain what about that warrants a "!?"; could you enlighten me?
        sovande 17y ago
        The '!?' was aimed at Mr. defunkt who said that nginx relies heavily on system calls at the expense of portability Its a oxymoron as you cannot expect a C application that mostly do network and disk i/o not to use syscalls. That said, the author of nginx does an admirable job of reducing the numbers of syscalls and make the application as efficient as possible. In this context a syscall i.e. a kernel call is heavy and something one want to minimize. The blog post on the other hand was talking about using syscalls from Ruby. I can understand Ruby programmers who wrinkle their nose at this. If you want your Ruby application to be portable then using, for instance, fork is not the best idea. In fact, programing anything long lived such as a server in Ruby is not the best idea. The GC in 1.8.x leaks memory over time and it is a common case that Ruby servers has to be restarted often as they consume all memory on the machine they run on. Adding fork on top of this is bad. A fork will copy the whole Ruby interpreter into the new process and you will end up with a lot of top-heavy processes - not a good idea unless you sell RAM. Basically, using Ruby as a systems programming language and for applications that are long lived is a bad idea.
cwp 17y ago
When the author says that threads are out, he means that they are a difficult way to write concurrent code. Pre-fork is just one (classic) example of how to write concurrent code without shared state. Instead of threads communicating via shared state, you have processes communicating over pipes or sockets. That said, you're right about portability. The attraction of Ruby's Thread is that it works identically across operating systems.

by mrshoe 17y ago

defunkt 17y ago

Counter example: nginx, which uses fork(), and seems to smoke Apache while offering features like binary reloading without dropping connections (not sure if Apache supports this but I seem to remember no - please correct me if I'm wrong). Portability is not always necessary - when talking about fork() and friends you're talking about trade offs. When I'm only ever deploying to Unix environments, I accept the lack of portability in exchange for features I value.

mrshoe 17y ago
That's not really a counter example. nginx does indeed fork off child processes, but it uses async I/O in each child process. It doesn't use the traditional pre-fork process-per-connection model that Apache's pre-fork MPM uses or that Unicorn is using.
- defunkt 17y ago
  nginx relies heavily on system calls at the expense of portability.
  - mrshoe 17y ago
    Oh, you were referring to the syscalls/portability tradeoff. In that case it's still not really a counter example, because Apache made the same decision in its early versions (Unix only). And that's a reasonable decision to make given the tradeoffs, I think. The point that I was more upset with was the "threads are dead, pre-fork is the way to go" section of the article. However, thinking about it more now, even the syscalls vs. portability arguments presented here are another example of forgetting the past (or just never being aware of it). After all the work from the Python devs to encapsulate syscalls in the standard library and provide a portable API to them, Jacob Kaplan-Moss says, "I’m a bit dismayed to see [syscalls] relegated to the dusty corners of our shiny dynamic languages."
  - sovande 17y ago
    Hello!? nginx is a POSIX network application written in C. Of course it uses syscalls. As the previous poster pointed out you are also wrong in that nginx uses anything like a prefork model. It _may_ fork of a process or more if it detect that it is running on a SMP system to take advantage more than one CPU. But each process has a strictly async io architecture handling request/responses within a big loop.
    - gloob 17y ago
      The point of the post you are responding to is that it uses syscalls. I'm not entirely certain what about that warrants a "!?"; could you enlighten me?
      - sovande 17y ago
        The '!?' was aimed at Mr. defunkt who said that nginx relies heavily on system calls at the expense of portability Its a oxymoron as you cannot expect a C application that mostly do network and disk i/o not to use syscalls. That said, the author of nginx does an admirable job of reducing the numbers of syscalls and make the application as efficient as possible. In this context a syscall i.e. a kernel call is heavy and something one want to minimize. The blog post on the other hand was talking about using syscalls from Ruby. I can understand Ruby programmers who wrinkle their nose at this. If you want your Ruby application to be portable then using, for instance, fork is not the best idea. In fact, programing anything long lived such as a server in Ruby is not the best idea. The GC in 1.8.x leaks memory over time and it is a common case that Ruby servers has to be restarted often as they consume all memory on the machine they run on. Adding fork on top of this is bad. A fork will copy the whole Ruby interpreter into the new process and you will end up with a lot of top-heavy processes - not a good idea unless you sell RAM. Basically, using Ruby as a systems programming language and for applications that are long lived is a bad idea.

cwp 17y ago

When the author says that threads are out, he means that they are a difficult way to write concurrent code. Pre-fork is just one (classic) example of how to write concurrent code without shared state. Instead of threads communicating via shared state, you have processes communicating over pipes or sockets. That said, you're right about portability. The attraction of Ruby's Thread is that it works identically across operating systems.