5 ms·

Summary: if you want performance, use Java, Scala, Go, Clojure, Lua, or C++. Honestly, now with all the great Scala frameworks, Clojure, and the ability to run

Summary: if you want performance, use Java, Scala, Go, Clojure, Lua, or C++. Honestly, now with all the great Scala frameworks, Clojure, and the ability to run Rails, plus Cassandra, Storm, etc, I'm a little creeped out that I'm actually strongly considering building my current new project completely on the JVM.

deleted 13y ago
[deleted]
- cgh 13y ago
  Here is the environment info: "As of Round 7, three Intel Sandy Bridge Core i7-2600K workstations with 8 GB memory each (early 2011 vintage) for the i7 tests; database server equipped with Samsung 840 Pro SSD Two Amazon EC2 m1.large instances for the EC2 tests Switched gigabit Ethernet" So 8GB workstations. Even assuming full memory usage, the JVM, Go and C++ frameworks destroyed all comers.
  - rallison 13y ago
    Just for reference, the deleted comment used to say: Raw performance is nothing without memory usage information.
- TomasEkeli 13y ago
  That said - memory is dirt cheap
  - Thaxll 13y ago
    Not really, on a desktop it is, on Amazon it's very expensive.
    - deleted 13y ago
      [deleted]
    - meepmorp 13y ago
      So don't use EC2 with your java apps.
  - mgkimsal 13y ago
    There are still practical limits, and sometimes you have to work within those. I've got a project on a 16G server - the data center doesn't have anything bigger. Moving everything to a custom server or different data center to get to 24G or 32G is... a lot of work. Some time spent finding memory reductions is still worth it. And what if we went to 32G or 64G, but still needed more? "RAM is cheap" doesn't scale. The hardware needed to host a 128G system (or multiple 16/32G systems) isn't cheap.
    - sciurus 13y ago
      Either you're wrong about the cost or we have a different definition of "cheap". The sweet spot for RAM pricing right now seems to be 16GB DIMMs, which cost $150 to $250. I can go to dell.com and configure a R320 with a Xeon CPU and 96GB of RAM for $2,500. That's less than a high-end Macbook Pro! A R420 with two Xeon CPUs and 192GB of RAM costs $4,300. I recently upgraded two Dell Poweredge 2950s from 16 to 32GB of RAM. The cost per server was $300. If I'd needed to, I could have upgraded them to 64GB. That's on a five year old server that sells refurbished for $800.
recuter 13y ago
I'm a fan of Go and an even bigger fan of Lua, and I'm not sure I agree with your summary. Go and OpenResty can push 300,000 "Hello World" plaintext responses a second, and some Java stuff can do x2 that, so JVM frameworks are strictly better? And something called cpoll_cppsp trumps a sane language choice? Don't be cute. :) It is perfectly fine to not squeeze every drop from your hardware and pick Go or Lua. In the same vain, Flask may be x20 slower at a meaningless benchmark but you'd note that the ones that involve actual work, you know, the kind your complex app will actually perform, the more involved it is the more the difference shrinks. All the way down to x4, in other words not an order of magnitutde. The real takeaway summary should probably be: a well made framework like flask doesn't have so much overhead that the productivity gains it offers are not worth trading a bit of performance for. A terribly made framework (I won't name names) will bite you in the ass. Choose carefully.
- mangeletti 13y ago
  I think the idea of a "meaningless benchmark" has become a dogma. Can you possibly know all the complexities that an app will contain when benchmarking a framework? No, but that doesn't make baseline benchmarks useless. Part of the reason for including the most basic (CRUD-ish) operations as separate benchmarks in their tests is so that you can make comparisons between basic "Hello, World" operations and more complex operations, which gives you a bit more insight into the overhead of each framework. I'll take that any day, over guessing which framework is fastest.
  - recuter 13y ago
    Agreed, I should have avoided that particular phrasing.
- piranha 13y ago
  Actually, it's other way around. In 'JSON encoding' tornado and flask-pypy are 4x slower than compojure, for example, but in 'Multiple queries' they are 14x slower.
- integraton 13y ago
  Just for the record, my comment didn't intend to imply that JVM frameworks are strictly better in all cases, and I don't believe that at all. In fact, I'd argue that for the majority of applications, choosing any modern framework purely for benchmark performance is very foolish since all of them perform more than adequately. Rails, for example, despite always having been "slow," performs far more than adequately for the vast majority of web applications, scaling patterns are well established, and it's obviously hugely productive for many, many developers and companies.
thezilch 13y ago
Or PHP, as you're most likely hitting some kind of DB / cache and not just serving plaintext or cached JSON out of your backend... backs away slowly
- fixxer 13y ago
  I'd run.
- ohashi 13y ago
  php-raw performed quite well.
  - andypants 13y ago
    As soon as you throw a php framework on top though, it sinks to the bottom of the benchmarks.
- jol 13y ago
  Actually I was quite surprised that multiple queries benchmark showed more than 1 entry for PHP in top 10. take-away: if you don't do much sophisticated data processing then PHP is ok choice. gets ready to run
  - Joeri 13y ago
    That's because php is a really thin wrapper around the underlying C libraries. The thing that makes it ugly is what makes it fast. Php is still interpreted and not jit'ed, even with the opcode cache used in this benchmark, so it loses out in anything that exercises the php code side of the equation, which is why the big php frameworks (symfony, cake and laravel) are 50 times slower than raw php. I wonder how facebook's hhvm engine would stack up, as it is a jit'ing php engine and supposedly an order of magnitude faster.
    - bhauer 13y ago
      That's my understanding of the situation as well. As a related aside, we have an intention to eventually capture some additional statistics about the implementations such as source lines of code, total number of commits, and possibly lines of code of libraries (where available). These are just additional data points to use as the reader sees fit, but they may provide some insights. For example: (a) developer efficiency, assuming you are comfortable using sloc as a proxy for developer efficiency, which is admittedly a hotly debated matter; (b) commits are a proxy for the level of attention the particular test implementation has received in our project, since a test with only 1 commit may be unrefined and in need of tuning and a test with dozens of commits may be considered highly volatile; (c) where applicable, the performance impact of additional code, perhaps even as granular as a calculated average per-line cost in rps. The last point might be particularly illuminating for languages such as PHP.
hbbio 13y ago
Too bad memory is not measured/displayed as well. For many cloud-hosted servers, you will end up paying for memory, rather than CPU. And Java/Scala are disqualified pretty quickly.
- rsynnott 13y ago
  Java and Scala would have large base memory requirements, but not necessarily large per-concurrent-request memory requirements.
TylerE 13y ago
When ever I feel like that, a few hours with either the Java or Scala persistance frameworks rapidly cures me of it.
- EtienneK 13y ago
  Honestly, JPA is not that bad.
- terhechte 13y ago
  This. In my last project I tried to use scalatra with slick, and while I really liked scalatra, slick made me go nuts. I had to jump over so many hoops that it was just a pain. I'm currently working on a new project (still research phase) where I was pondering going with Spray and a lightweight Postgresql wrapper because I primarily need to read data from a database, do some transformations on it, and write it out as JSON as fast as possible. I had it working in Spray, but I had issues with server crashes and the speed wasn't as I'd have expected. I fiddled with it for one day, and then, out of frustration, decided to give openresty a try. I've never written much Lua in my life, but after only a couple of hours, I had it working, and it was far, far faster than the Spray implementation. I did some research there, and it seems that the database stuff took a whole lot longer in Scala/Spray than in Lua. Now, of course, I loose type safety, so there may be hidden issues in there, but since I'm really just doing simple data transformations, I think I'm fine with lua / openresty. As a first verdict, I really, really like what I've seen of Openresty so far.
- flavor8 13y ago
  mybatis.
arocks 13y ago
or Haskell :)
stesch 13y ago
If you ignore Erlang and run your database on a ramdisk. Sorry, round 7 isn't helping much. Wait for round 8.

deleted 13y ago

[deleted]

cgh 13y ago
Here is the environment info: "As of Round 7, three Intel Sandy Bridge Core i7-2600K workstations with 8 GB memory each (early 2011 vintage) for the i7 tests; database server equipped with Samsung 840 Pro SSD Two Amazon EC2 m1.large instances for the EC2 tests Switched gigabit Ethernet" So 8GB workstations. Even assuming full memory usage, the JVM, Go and C++ frameworks destroyed all comers.
- rallison 13y ago
  Just for reference, the deleted comment used to say: Raw performance is nothing without memory usage information.
TomasEkeli 13y ago
That said - memory is dirt cheap
- Thaxll 13y ago
  Not really, on a desktop it is, on Amazon it's very expensive.
  - deleted 13y ago
    [deleted]
  - meepmorp 13y ago
    So don't use EC2 with your java apps.
- mgkimsal 13y ago
  There are still practical limits, and sometimes you have to work within those. I've got a project on a 16G server - the data center doesn't have anything bigger. Moving everything to a custom server or different data center to get to 24G or 32G is... a lot of work. Some time spent finding memory reductions is still worth it. And what if we went to 32G or 64G, but still needed more? "RAM is cheap" doesn't scale. The hardware needed to host a 128G system (or multiple 16/32G systems) isn't cheap.
  - sciurus 13y ago
    Either you're wrong about the cost or we have a different definition of "cheap". The sweet spot for RAM pricing right now seems to be 16GB DIMMs, which cost $150 to $250. I can go to dell.com and configure a R320 with a Xeon CPU and 96GB of RAM for $2,500. That's less than a high-end Macbook Pro! A R420 with two Xeon CPUs and 192GB of RAM costs $4,300. I recently upgraded two Dell Poweredge 2950s from 16 to 32GB of RAM. The cost per server was $300. If I'd needed to, I could have upgraded them to 64GB. That's on a five year old server that sells refurbished for $800.

recuter 13y ago

I'm a fan of Go and an even bigger fan of Lua, and I'm not sure I agree with your summary. Go and OpenResty can push 300,000 "Hello World" plaintext responses a second, and some Java stuff can do x2 that, so JVM frameworks are strictly better? And something called cpoll_cppsp trumps a sane language choice? Don't be cute. :) It is perfectly fine to not squeeze every drop from your hardware and pick Go or Lua. In the same vain, Flask may be x20 slower at a meaningless benchmark but you'd note that the ones that involve actual work, you know, the kind your complex app will actually perform, the more involved it is the more the difference shrinks. All the way down to x4, in other words not an order of magnitutde. The real takeaway summary should probably be: a well made framework like flask doesn't have so much overhead that the productivity gains it offers are not worth trading a bit of performance for. A terribly made framework (I won't name names) will bite you in the ass. Choose carefully.

mangeletti 13y ago
I think the idea of a "meaningless benchmark" has become a dogma. Can you possibly know all the complexities that an app will contain when benchmarking a framework? No, but that doesn't make baseline benchmarks useless. Part of the reason for including the most basic (CRUD-ish) operations as separate benchmarks in their tests is so that you can make comparisons between basic "Hello, World" operations and more complex operations, which gives you a bit more insight into the overhead of each framework. I'll take that any day, over guessing which framework is fastest.
- recuter 13y ago
  Agreed, I should have avoided that particular phrasing.
piranha 13y ago
Actually, it's other way around. In 'JSON encoding' tornado and flask-pypy are 4x slower than compojure, for example, but in 'Multiple queries' they are 14x slower.
integraton 13y ago
Just for the record, my comment didn't intend to imply that JVM frameworks are strictly better in all cases, and I don't believe that at all. In fact, I'd argue that for the majority of applications, choosing any modern framework purely for benchmark performance is very foolish since all of them perform more than adequately. Rails, for example, despite always having been "slow," performs far more than adequately for the vast majority of web applications, scaling patterns are well established, and it's obviously hugely productive for many, many developers and companies.

thezilch 13y ago

Or PHP, as you're most likely hitting some kind of DB / cache and not just serving plaintext or cached JSON out of your backend... backs away slowly

fixxer 13y ago
I'd run.
ohashi 13y ago
php-raw performed quite well.
- andypants 13y ago
  As soon as you throw a php framework on top though, it sinks to the bottom of the benchmarks.
jol 13y ago
Actually I was quite surprised that multiple queries benchmark showed more than 1 entry for PHP in top 10. take-away: if you don't do much sophisticated data processing then PHP is ok choice. gets ready to run
- Joeri 13y ago
  That's because php is a really thin wrapper around the underlying C libraries. The thing that makes it ugly is what makes it fast. Php is still interpreted and not jit'ed, even with the opcode cache used in this benchmark, so it loses out in anything that exercises the php code side of the equation, which is why the big php frameworks (symfony, cake and laravel) are 50 times slower than raw php. I wonder how facebook's hhvm engine would stack up, as it is a jit'ing php engine and supposedly an order of magnitude faster.
  - bhauer 13y ago
    That's my understanding of the situation as well. As a related aside, we have an intention to eventually capture some additional statistics about the implementations such as source lines of code, total number of commits, and possibly lines of code of libraries (where available). These are just additional data points to use as the reader sees fit, but they may provide some insights. For example: (a) developer efficiency, assuming you are comfortable using sloc as a proxy for developer efficiency, which is admittedly a hotly debated matter; (b) commits are a proxy for the level of attention the particular test implementation has received in our project, since a test with only 1 commit may be unrefined and in need of tuning and a test with dozens of commits may be considered highly volatile; (c) where applicable, the performance impact of additional code, perhaps even as granular as a calculated average per-line cost in rps. The last point might be particularly illuminating for languages such as PHP.

hbbio 13y ago

Too bad memory is not measured/displayed as well. For many cloud-hosted servers, you will end up paying for memory, rather than CPU. And Java/Scala are disqualified pretty quickly.

rsynnott 13y ago
Java and Scala would have large base memory requirements, but not necessarily large per-concurrent-request memory requirements.

TylerE 13y ago

When ever I feel like that, a few hours with either the Java or Scala persistance frameworks rapidly cures me of it.

EtienneK 13y ago
Honestly, JPA is not that bad.
terhechte 13y ago
This. In my last project I tried to use scalatra with slick, and while I really liked scalatra, slick made me go nuts. I had to jump over so many hoops that it was just a pain. I'm currently working on a new project (still research phase) where I was pondering going with Spray and a lightweight Postgresql wrapper because I primarily need to read data from a database, do some transformations on it, and write it out as JSON as fast as possible. I had it working in Spray, but I had issues with server crashes and the speed wasn't as I'd have expected. I fiddled with it for one day, and then, out of frustration, decided to give openresty a try. I've never written much Lua in my life, but after only a couple of hours, I had it working, and it was far, far faster than the Spray implementation. I did some research there, and it seems that the database stuff took a whole lot longer in Scala/Spray than in Lua. Now, of course, I loose type safety, so there may be hidden issues in there, but since I'm really just doing simple data transformations, I think I'm fine with lua / openresty. As a first verdict, I really, really like what I've seen of Openresty so far.
flavor8 13y ago
mybatis.

arocks 13y ago

or Haskell :)

stesch 13y ago

If you ignore Erlang and run your database on a ramdisk. Sorry, round 7 isn't helping much. Wait for round 8.