6 ms·

Snap: A Haskell Web Framework: Benchmarks

44 points by aliasaria 16y ago

rubyrescue 16y ago
interesting. i would like to see a comparison to yaws or mochiweb (perhaps w/nitrogen).
acangiano 16y ago
Let me state once again that the Rails benchmarks are absolutely way off. Snap would still be faster - much faster - but you can expect several thousands req/s from that Rails app on that kind of hardware.
- wwortiz 16y ago
  It was said in another thread that these were done using development servers: http://news.ycombinator.com/item?id=1371048 http://news.ycombinator.com/item?id=1371048 Edit: Oh he responded to you the last time, but I'll leave this here so others can see
- rufugee 16y ago
  I did my own testing, but I didn't publish them because I left Haskell completely unoptimized and didn't really have the time to try to optimize it. I also added play (http://www.playframework.org http://www.playframework.org) into the mix, because I admire the framework and how closely it mimics Rails, but in Java (and Scala). Still, I saved 'em, so here you go: On a 2.27Ghz Xeon dual quad core with 12 GB ram, using Snap's tests from GitHub: Test Pong: | Rails Webrick | Rails Mongrel | Rails Thin | Rails JRuby (glassfish) | Snap | Play! | | 155.0 req/s | 1940.5 req/s | 619.0 req/s | 3606.0 req/s | 11347.0 req/s | 19602.2 req/s | Test File: | Rails Webrick | Rails Mongrel | Rails Thin | Rails JRuby (glassfish) | Snap | Play! | | Didn't test | 188.3 req/s | 718.0 req/s | 6633.3 req/s | memory leak | 606.6 conn/s | The full results are here: Pong: http://gist.github.com/414724 http://gist.github.com/414724 File: http://gist.github.com/414728 http://gist.github.com/414728 Note, Snap had a mem leak when performing the file test that caused a machine with 12 GB to swap, but that may be because I used the Ubuntu Lucid version of Haskell. Also note, Play is damned fast. And relative to other Rails options, JRuby with glassfish is as well.
  - mark_l_watson 16y ago
    Very interesting - thanks. Scala is a hot language right now, and when Play gets really solid Scala support (does it have it already?) then Play may get to be more popular.
    - rufugee 16y ago
      It supports Scala well, but the API (on the Scala side) is a bit of a moving target. However, I really encourage all of you to give the Java side of play a shot. It's really unlike any Java framework I've ever worked with, and I've been doing Java for far too long. It automatically reloads your changes without a compile...it makes great use of annotations, and it uses Groovy for the templating language, it comes with support by default for the big three IDEs (Netbeans, Eclipse, IntelliJ) although you can easily be happy with vim, emacs, or tm. After a bit of time, it's easy to forget you're actually working with Java. Also, if you know Rails, you essentially know Play. The concepts are the same, the directory structure is the same. You're learning the Java version of Rails, so there are differences, but my years of Rails development experience made picking up play a breeze. edit: clarified API comment
  - mightybyte 16y ago
    You need to look at the average reply rate instead of the request rate. Requests/sec can be misleading. The replies / sec tell quite a different story. Here they are for pong: Rails Webrick: 153.9 Rails Thin: 33.3 Rails Mongrel: 90.7 Rails JRuby (glassfish): 3,350 JRuby Mongrel: 31.3 Snap: 11,309 Play: 19,796
    - rufugee 16y ago
      You're absolutely correct. Sorry. Here it is, updated properly: Test Pong: | Rails Webrick | Rails Mongrel | Rails Thin | Rails JRuby (glassfish) | Snap | Play! | | 153.9 | 90.7 | 33.3 | 3350.5 | 11309.5 | 19796.3 | Test File: | Rails Webrick | Rails Mongrel | Rails Thin | Rails JRuby (glassfish) | Snap | Play! | | Didn't test | 179.9 | 758.0 | 6146.9 | memory leak | 1617.9 |
warfangle 16y ago
Would love to see a comparison to Node.js on here, as well.
- matthijs 16y ago
  Indeed, for example the express framework appears to be optimized for high performance http://expressjs.com/ http://expressjs.com/
JimBastard 16y ago
How are you going to run a benchmark like this and not include nginx and nodejs? I mean, okay, node is new...but not including nginx?
angusgr 16y ago
Although these seem like useful (and impressive) baseline benchmarks, won't the really important benchmarks come from measuring actual dynamic content generation, and/or caching performance for generated dynamic content?
- prb 16y ago
  When I did experiments with RoR versus Haskell via FCGI, Haskell had around a two orders of magnitude performance advantage under load and only required a single OS process.

44 points by aliasaria 16y ago

interesting. i would like to see a comparison to yaws or mochiweb (perhaps w/nitrogen).

Let me state once again that the Rails benchmarks are absolutely way off. Snap would still be faster - much faster - but you can expect several thousands req/s from that Rails app on that kind of hardware.

wwortiz 16y ago
It was said in another thread that these were done using development servers: http://news.ycombinator.com/item?id=1371048 http://news.ycombinator.com/item?id=1371048 Edit: Oh he responded to you the last time, but I'll leave this here so others can see
rufugee 16y ago
I did my own testing, but I didn't publish them because I left Haskell completely unoptimized and didn't really have the time to try to optimize it. I also added play (http://www.playframework.org http://www.playframework.org) into the mix, because I admire the framework and how closely it mimics Rails, but in Java (and Scala). Still, I saved 'em, so here you go: On a 2.27Ghz Xeon dual quad core with 12 GB ram, using Snap's tests from GitHub: Test Pong: | Rails Webrick | Rails Mongrel | Rails Thin | Rails JRuby (glassfish) | Snap | Play! | | 155.0 req/s | 1940.5 req/s | 619.0 req/s | 3606.0 req/s | 11347.0 req/s | 19602.2 req/s | Test File: | Rails Webrick | Rails Mongrel | Rails Thin | Rails JRuby (glassfish) | Snap | Play! | | Didn't test | 188.3 req/s | 718.0 req/s | 6633.3 req/s | memory leak | 606.6 conn/s | The full results are here: Pong: http://gist.github.com/414724 http://gist.github.com/414724 File: http://gist.github.com/414728 http://gist.github.com/414728 Note, Snap had a mem leak when performing the file test that caused a machine with 12 GB to swap, but that may be because I used the Ubuntu Lucid version of Haskell. Also note, Play is damned fast. And relative to other Rails options, JRuby with glassfish is as well.
- mark_l_watson 16y ago
  Very interesting - thanks. Scala is a hot language right now, and when Play gets really solid Scala support (does it have it already?) then Play may get to be more popular.
  - rufugee 16y ago
    It supports Scala well, but the API (on the Scala side) is a bit of a moving target. However, I really encourage all of you to give the Java side of play a shot. It's really unlike any Java framework I've ever worked with, and I've been doing Java for far too long. It automatically reloads your changes without a compile...it makes great use of annotations, and it uses Groovy for the templating language, it comes with support by default for the big three IDEs (Netbeans, Eclipse, IntelliJ) although you can easily be happy with vim, emacs, or tm. After a bit of time, it's easy to forget you're actually working with Java. Also, if you know Rails, you essentially know Play. The concepts are the same, the directory structure is the same. You're learning the Java version of Rails, so there are differences, but my years of Rails development experience made picking up play a breeze. edit: clarified API comment
- mightybyte 16y ago
  You need to look at the average reply rate instead of the request rate. Requests/sec can be misleading. The replies / sec tell quite a different story. Here they are for pong: Rails Webrick: 153.9 Rails Thin: 33.3 Rails Mongrel: 90.7 Rails JRuby (glassfish): 3,350 JRuby Mongrel: 31.3 Snap: 11,309 Play: 19,796
  - rufugee 16y ago
    You're absolutely correct. Sorry. Here it is, updated properly: Test Pong: | Rails Webrick | Rails Mongrel | Rails Thin | Rails JRuby (glassfish) | Snap | Play! | | 153.9 | 90.7 | 33.3 | 3350.5 | 11309.5 | 19796.3 | Test File: | Rails Webrick | Rails Mongrel | Rails Thin | Rails JRuby (glassfish) | Snap | Play! | | Didn't test | 179.9 | 758.0 | 6146.9 | memory leak | 1617.9 |

warfangle 16y ago

Would love to see a comparison to Node.js on here, as well.

matthijs 16y ago
Indeed, for example the express framework appears to be optimized for high performance http://expressjs.com/ http://expressjs.com/

JimBastard 16y ago

How are you going to run a benchmark like this and not include nginx and nodejs? I mean, okay, node is new...but not including nginx?

angusgr 16y ago

Although these seem like useful (and impressive) baseline benchmarks, won't the really important benchmarks come from measuring actual dynamic content generation, and/or caching performance for generated dynamic content?

prb 16y ago
When I did experiments with RoR versus Haskell via FCGI, Haskell had around a two orders of magnitude performance advantage under load and only required a single OS process.