7 ms·

Python vs. Node vs. PyPy benchmarks

63 points by chanux 14y ago

guilloche 14y ago
I am surprised that node.js performs significantly better than python & pypy. Can anyone have some explanation?
- afandian 14y ago
  Money.
- raphman 14y ago
  It depends on the web framework. As the author of the benchmark shows in a later blog post [1], Python+uWSGI can be faster than Node.js [1] http://blog.kgriffs.com/2012/12/18/uwsgi-vs-gunicorn-vs-node-benchmarks.html http://blog.kgriffs.com/2012/12/18/uwsgi-vs-gunicorn-vs-node... (link originally posted by jdub in this discussion)
- fijal 14y ago
  He carefully chose to use a really bad combination for pypy (or uncarefully, always assume stupidity over malice). Fast options for PyPy are cyclone.io or tornado (or eventlet or gevent on a branch I'm working on, although it's not done).
tferris 14y ago
Node/V8 is a beast and will dominate web development in few years. EDIT: Downvote me, I don't care.
- zokier 14y ago
  That'd be true if performance would trump productivity in web development. On the other hand, I don't see a lot of web-apps written in C...
  - tferris 14y ago
    Give Node s try before judging, Node gives you same productivity as other popular languages used for web development but with C-class performance
    - newman314 14y ago
      Ummm, where are you getting that node has C-class perf?
      - NuZZ 14y ago
        A quick google [1] results in this article. Look for JS:V8 - Node runs off of that. Most of the graphs I glanced at show it's very close to C. I glance because I've properly checked out other benchmarks which confirm this. [1] http://attractivechaos.github.com/plb/ http://attractivechaos.github.com/plb/
- pestaa 14y ago
  If the web development industry moves from PHP to JavaScript as the main backend language... well, from quality perspective nothing will change. IMO.
  - espadrine 14y ago
    If one company gave you a PHP job, and another a JS backend job, and that'd be all you got, would you flip a coin to pick one? Or is this a way to say that you feel more satisfied with your work in Python than with your work in node?
    - pestaa 14y ago
      This is my way to say several points: - If the majority of our industry can make the switch to JavaScript, they'll ruin it in terms of average code quality. - Even good, idiomatic JavaScript code sucks for reasons inherent in the language. - I wish people wrote just good enough Python or Erlang or Haskell or whatever code instead of excellent PHP/JavaScript. So yes, I treat PHP and JavaScript backend jobs equally. [Edit: grammar.]
      - rgbrgb 14y ago
        >> Even good, idiomatic JavaScript code sucks for reasons inherent in the language. Care to elaborate?
        dguaraglia 14y ago
        I'll just throw a couple pet-peeves of mine in here: - No proper packaging/namespacing - Completely idiosyncratic results when operating with mixed data types (10 + 'p' = '10p', 7/0 = 'infinity' and all that) - Variables default to a pseudo-global scope - Prototype based 'inheritance', while cool, is just cumbersome and inelegant compared to languages that have proper object oriented language constructs - Take all those and a syntax that was partially taken from Java, and you end up with verbose code jumble that is way more difficult to read than equivalent Python/Ruby (even C#)
- malkarouri 14y ago
  I actually don't think so, not that I have downvoted you. V8 was such a breakthrough at the time, it blew off the competition. Since then, many competitors have picked up the tricks and as shown in the bennchmark, uWSGI holds its own against Node/V8. As a result, the Node/V8 combination is not going to be so dominant in a few years in terms of performance. Another factors will come into play such as the language and the libraries available (the ecosystem). I would find it much harder to predict the dominance of the Noode/V8 ecosystem compared to that of Python or Java in a few years' time.
  - jessedhillon 14y ago
    I think this is it exactly. The pace at which infrastructure, support and tools in "competing" ecosystems can be improved will outpace the rate at which people's miss can be changed about JS. There are 10-15 years of the language's, most of which were a Very Bad time for it's reputation, and only recently has the community of developers begun to land on solid development principles -- and you can hardly call those universal. I think most people fear callback spaghetti when they think of diving into someone else's JS code.
- lucian1900 14y ago
  V8 and PyPy are similarly fast runtimes, but this benchmark tests libraries against each other. Not particularly relevant, especially as at least two of them are known to have slow IO.
  - tferris 14y ago
    As good as pypy is it's no replacement for python yet. http://stackoverflow.com/questions/12867263/why-wasnt-pypy-included-in-standard-python http://stackoverflow.com/questions/12867263/why-wasnt-pypy-i...
jdub 14y ago
Don't miss this follow-up post, though: http://blog.kgriffs.com/2012/12/18/uwsgi-vs-gunicorn-vs-node-benchmarks.html http://blog.kgriffs.com/2012/12/18/uwsgi-vs-gunicorn-vs-node...
- davyjones 14y ago
  Graph 3 as a log y chart would've been nice.
deleted 14y ago
[deleted]
deleted 14y ago
[deleted]
rdtsc 14y ago
Another set of benchmarks that shows Erlang's cowboy coming ahead on EC instances. It also compares EC2 vs real hardware and throws more contenders in the mix. On physical hardware the race is pretty close (except node, python-gevent, but cluster node did very well). http://eric.themoritzfamily.com/websocket-demo-results-v2.html http://eric.themoritzfamily.com/websocket-demo-results-v2.ht...
- ericmoritz 14y ago
  Yeah, because of cowboy and Erlang's design it is going to out perform a single process evented server hands-down. 1. The evented servers are bound to one CPU, Erlang will use all CPUs 2. any bit of blocking code will stall the other code, even it just for a moment. It is a little unfair to compare the two approaches.
  - rdtsc 14y ago
    I am not sure it will always under-perform with one CPU. It depends if async threads are used and if kernel poll is enabled. Erlang is designed for better low latency response at the expense of _some_ sequential slow down. I wonder if Erlang would scale slower but it will throw less errors as number of requests increases. EDIT: Also it would seem benchmarking Erlang and forcing it to run on a single core just like node.js is deliberately handicapping it. One of the strengths of Erlang is exactly the ability to take advantage of multiple cores.
calpaterson 14y ago
I don't understand why there are no numbers on these graphs. Graphs are pretty pointless if you don't label the axes.
- insaneirish 14y ago
  Isn't labeling axes the first rule of graphing? This is bad.
- rplnt 14y ago
  It's either fixed or there's a problem on your end.
  - oib 14y ago
    Nope, I don't get any numbers/labels either. Seems like the website is incompatible with a bunch of browsers.
    - bhauer 14y ago
      He's just using Flot, which a lot of us use for canvas-based charts. Do they work for you at all? http://www.flotcharts.org/ http://www.flotcharts.org/ If not, what browser are you using? It would be nice to know if Flot is not compatible with something.
      - oib 14y ago
        iceweasel 10.0.12 (basically the same as Firefox 10.0.12) It ships as the default browser for debian wheezy. The home page of flotcharts.org does work correctly. I also tried Konqueror Version 4.8.4 (KDE's browser) and it didn't even display the graphs (not in the blog post or on flotcharts.org).
  - thomasz 14y ago
    works with IE9 and firefox 18, not with chrome 24.
  - calpaterson 14y ago
Egregore 14y ago
For Node.js it's a big difference between 1k and 64k files, what about something in between, like 32k files?
phenom 14y ago
Edit: pointed out in other comment
wyuenho 14y ago
I'm actually quite perplexed as to why the blog author chose to use the WSGIRef which is known to be one of the slowest of the bunch according to this much more thorough benchmark: http://nichol.as/benchmark-of-python-web-servers http://nichol.as/benchmark-of-python-web-servers Even more perplexed, why wasn't the payload and the WSGI apps sources published?
- robert-zaremba 14y ago
  Your observation is good. Author should clarify this. But in his next articles he shows benchmark against other web frameworks, which shows more justified results
  - fijal 14y ago
    That doesn't really answer why use the slowest one with PyPy (and no, uWSGI does not work for PyPy, but a lot of stuff does)
    - unbit 14y ago
      uWSGI works with PyPY (Obviously with lot of limitations on the features). http://projects.unbit.it/uwsgi/wiki/PyPy http://projects.unbit.it/uwsgi/wiki/PyPy The problem is that for the vast majority of the webapps there is (still) no advantage. I only wanted to focus on that as i have invested days on (really complex for me) pypy sources, and i do not want people to ignore it ;)
  - wyuenho 14y ago
    His articles don't justify anything. I don't know what his payloads are or how he produced them in the apps. For all I know these 2 "benchmarks" are unreproducible. The author said he wanted to try science, there's nothing scientific about these 2 articles at all.
est 14y ago
If you are using wsgiref don't forget the patch __import__('BaseHTTPServer').BaseHTTPRequestHandler.address_string = lambda x:x.client_address[0]
- deleted 14y ago
  [deleted]
swah 14y ago
I'd love to see something on the JVM against this.
wheaties 14y ago
This whole test says more to me about the performance of non-blocking MongoDB drivers than it says about the underlying performance of the frameworks/languages. The python tests lacked a non-blocking driver. Why not use the PostGRES drivers ? That would give a much more accurate and fair treatment. And for the record, I don't care one way or the other which is "faster," just make a damned test where one isn't given such a clear leg up.
Yaggo 14y ago
Just FYI, in my own tests I've found Node.js performing about 50% better (req/s) than Tornado/C-python, while Tornado/pypy comes very close to Node.js' performance. Tornado is one of the fastest Python web frameworks and does not use WSGI (by default). The point is that pypy can crunch almost as fast as Node.js, so while Node.js may set the standard, Python is not bad choice either from performance perspective.
mercuryrising 14y ago
Where's the source code? I'd like to see how long it takes someone who doesn't know either language to figure out what's happening in the code (a 'new' kind of response time).
ricardobeat 14y ago
Red and green side-by-side on a chart... noo :( http://jfly.iam.u-tokyo.ac.jp/color/#pallet http://jfly.iam.u-tokyo.ac.jp/color/#pallet http://tools.medialab.sciences-po.fr/iwanthue/ http://tools.medialab.sciences-po.fr/iwanthue/
- swah 14y ago
  IMO its a little bit excessive to ask that every graph generated is color-blind safe... Isn't there software for colorblind people to correct that? (Something that changes the palette of colors displayed...)
  - ricardobeat 14y ago
    Red & green is the stereotypical colorblind test, that's a pretty obvious combination to avoid. It's easy to choose colors that are at right angles in the color wheel, and that's not how accessibility works - you can't require every paraplegic to have a special wheelchair that can climb stairs, despite it existing...
justin_vanw 14y ago
Things like this DO NOT MATTER. Facebook was built on PHP and MySQL, both of which are awful pieces of crap. Twitter went down like every week for a year. It was written on RoR by people who didn't really know RoR. Meanwhile, a bunch of better engineers using better technology got their clocks cleaned while the winners eventually had so much money they could afford to hire an army of engineers to rewrite everything. If you know Javascript really well, use Node (although I feel sorry for you). If you know Python really well, use Python (yay). If you don't know anything, well, pick something. If you are choosing a programming language because of how many QPS you can get on some trivial task, you are an idiot. You know the graph I would like to see? A scatterplot of how good various startup founders and early team were on engineering vs finding a market fit, with the dots labeled with how rich they are now.

63 points by chanux 14y ago

guilloche 14y ago

I am surprised that node.js performs significantly better than python & pypy. Can anyone have some explanation?

afandian 14y ago
Money.
raphman 14y ago
It depends on the web framework. As the author of the benchmark shows in a later blog post [1], Python+uWSGI can be faster than Node.js [1] http://blog.kgriffs.com/2012/12/18/uwsgi-vs-gunicorn-vs-node-benchmarks.html http://blog.kgriffs.com/2012/12/18/uwsgi-vs-gunicorn-vs-node... (link originally posted by jdub in this discussion)
fijal 14y ago
He carefully chose to use a really bad combination for pypy (or uncarefully, always assume stupidity over malice). Fast options for PyPy are cyclone.io or tornado (or eventlet or gevent on a branch I'm working on, although it's not done).

tferris 14y ago

Node/V8 is a beast and will dominate web development in few years. EDIT: Downvote me, I don't care.

zokier 14y ago
That'd be true if performance would trump productivity in web development. On the other hand, I don't see a lot of web-apps written in C...
- tferris 14y ago
  Give Node s try before judging, Node gives you same productivity as other popular languages used for web development but with C-class performance
  - newman314 14y ago
    Ummm, where are you getting that node has C-class perf?
    - NuZZ 14y ago
      A quick google [1] results in this article. Look for JS:V8 - Node runs off of that. Most of the graphs I glanced at show it's very close to C. I glance because I've properly checked out other benchmarks which confirm this. [1] http://attractivechaos.github.com/plb/ http://attractivechaos.github.com/plb/
pestaa 14y ago
If the web development industry moves from PHP to JavaScript as the main backend language... well, from quality perspective nothing will change. IMO.
- espadrine 14y ago
  If one company gave you a PHP job, and another a JS backend job, and that'd be all you got, would you flip a coin to pick one? Or is this a way to say that you feel more satisfied with your work in Python than with your work in node?
  - pestaa 14y ago
    This is my way to say several points: - If the majority of our industry can make the switch to JavaScript, they'll ruin it in terms of average code quality. - Even good, idiomatic JavaScript code sucks for reasons inherent in the language. - I wish people wrote just good enough Python or Erlang or Haskell or whatever code instead of excellent PHP/JavaScript. So yes, I treat PHP and JavaScript backend jobs equally. [Edit: grammar.]
    - rgbrgb 14y ago
      >> Even good, idiomatic JavaScript code sucks for reasons inherent in the language. Care to elaborate?
      - dguaraglia 14y ago
        I'll just throw a couple pet-peeves of mine in here: - No proper packaging/namespacing - Completely idiosyncratic results when operating with mixed data types (10 + 'p' = '10p', 7/0 = 'infinity' and all that) - Variables default to a pseudo-global scope - Prototype based 'inheritance', while cool, is just cumbersome and inelegant compared to languages that have proper object oriented language constructs - Take all those and a syntax that was partially taken from Java, and you end up with verbose code jumble that is way more difficult to read than equivalent Python/Ruby (even C#)
malkarouri 14y ago
I actually don't think so, not that I have downvoted you. V8 was such a breakthrough at the time, it blew off the competition. Since then, many competitors have picked up the tricks and as shown in the bennchmark, uWSGI holds its own against Node/V8. As a result, the Node/V8 combination is not going to be so dominant in a few years in terms of performance. Another factors will come into play such as the language and the libraries available (the ecosystem). I would find it much harder to predict the dominance of the Noode/V8 ecosystem compared to that of Python or Java in a few years' time.
- jessedhillon 14y ago
  I think this is it exactly. The pace at which infrastructure, support and tools in "competing" ecosystems can be improved will outpace the rate at which people's miss can be changed about JS. There are 10-15 years of the language's, most of which were a Very Bad time for it's reputation, and only recently has the community of developers begun to land on solid development principles -- and you can hardly call those universal. I think most people fear callback spaghetti when they think of diving into someone else's JS code.
lucian1900 14y ago
V8 and PyPy are similarly fast runtimes, but this benchmark tests libraries against each other. Not particularly relevant, especially as at least two of them are known to have slow IO.
- tferris 14y ago
  As good as pypy is it's no replacement for python yet. http://stackoverflow.com/questions/12867263/why-wasnt-pypy-included-in-standard-python http://stackoverflow.com/questions/12867263/why-wasnt-pypy-i...

jdub 14y ago

Don't miss this follow-up post, though: http://blog.kgriffs.com/2012/12/18/uwsgi-vs-gunicorn-vs-node-benchmarks.html http://blog.kgriffs.com/2012/12/18/uwsgi-vs-gunicorn-vs-node...

davyjones 14y ago
Graph 3 as a log y chart would've been nice.

deleted 14y ago

[deleted]

deleted 14y ago

[deleted]

rdtsc 14y ago

Another set of benchmarks that shows Erlang's cowboy coming ahead on EC instances. It also compares EC2 vs real hardware and throws more contenders in the mix. On physical hardware the race is pretty close (except node, python-gevent, but cluster node did very well). http://eric.themoritzfamily.com/websocket-demo-results-v2.html http://eric.themoritzfamily.com/websocket-demo-results-v2.ht...

ericmoritz 14y ago
Yeah, because of cowboy and Erlang's design it is going to out perform a single process evented server hands-down. 1. The evented servers are bound to one CPU, Erlang will use all CPUs 2. any bit of blocking code will stall the other code, even it just for a moment. It is a little unfair to compare the two approaches.
- rdtsc 14y ago
  I am not sure it will always under-perform with one CPU. It depends if async threads are used and if kernel poll is enabled. Erlang is designed for better low latency response at the expense of _some_ sequential slow down. I wonder if Erlang would scale slower but it will throw less errors as number of requests increases. EDIT: Also it would seem benchmarking Erlang and forcing it to run on a single core just like node.js is deliberately handicapping it. One of the strengths of Erlang is exactly the ability to take advantage of multiple cores.

calpaterson 14y ago

I don't understand why there are no numbers on these graphs. Graphs are pretty pointless if you don't label the axes.

insaneirish 14y ago
Isn't labeling axes the first rule of graphing? This is bad.
rplnt 14y ago
It's either fixed or there's a problem on your end.
- oib 14y ago
  Nope, I don't get any numbers/labels either. Seems like the website is incompatible with a bunch of browsers.
  - bhauer 14y ago
    He's just using Flot, which a lot of us use for canvas-based charts. Do they work for you at all? http://www.flotcharts.org/ http://www.flotcharts.org/ If not, what browser are you using? It would be nice to know if Flot is not compatible with something.
    - oib 14y ago
      iceweasel 10.0.12 (basically the same as Firefox 10.0.12) It ships as the default browser for debian wheezy. The home page of flotcharts.org does work correctly. I also tried Konqueror Version 4.8.4 (KDE's browser) and it didn't even display the graphs (not in the blog post or on flotcharts.org).
- thomasz 14y ago
  works with IE9 and firefox 18, not with chrome 24.
- calpaterson 14y ago

Egregore 14y ago

For Node.js it's a big difference between 1k and 64k files, what about something in between, like 32k files?

phenom 14y ago

Edit: pointed out in other comment

wyuenho 14y ago

I'm actually quite perplexed as to why the blog author chose to use the WSGIRef which is known to be one of the slowest of the bunch according to this much more thorough benchmark: http://nichol.as/benchmark-of-python-web-servers http://nichol.as/benchmark-of-python-web-servers Even more perplexed, why wasn't the payload and the WSGI apps sources published?

robert-zaremba 14y ago
Your observation is good. Author should clarify this. But in his next articles he shows benchmark against other web frameworks, which shows more justified results
- fijal 14y ago
  That doesn't really answer why use the slowest one with PyPy (and no, uWSGI does not work for PyPy, but a lot of stuff does)
  - unbit 14y ago
    uWSGI works with PyPY (Obviously with lot of limitations on the features). http://projects.unbit.it/uwsgi/wiki/PyPy http://projects.unbit.it/uwsgi/wiki/PyPy The problem is that for the vast majority of the webapps there is (still) no advantage. I only wanted to focus on that as i have invested days on (really complex for me) pypy sources, and i do not want people to ignore it ;)
- wyuenho 14y ago
  His articles don't justify anything. I don't know what his payloads are or how he produced them in the apps. For all I know these 2 "benchmarks" are unreproducible. The author said he wanted to try science, there's nothing scientific about these 2 articles at all.

est 14y ago

If you are using wsgiref don't forget the patch __import__('BaseHTTPServer').BaseHTTPRequestHandler.address_string = lambda x:x.client_address[0]

deleted 14y ago
[deleted]

swah 14y ago

I'd love to see something on the JVM against this.

wheaties 14y ago

This whole test says more to me about the performance of non-blocking MongoDB drivers than it says about the underlying performance of the frameworks/languages. The python tests lacked a non-blocking driver. Why not use the PostGRES drivers ? That would give a much more accurate and fair treatment. And for the record, I don't care one way or the other which is "faster," just make a damned test where one isn't given such a clear leg up.

Yaggo 14y ago

Just FYI, in my own tests I've found Node.js performing about 50% better (req/s) than Tornado/C-python, while Tornado/pypy comes very close to Node.js' performance. Tornado is one of the fastest Python web frameworks and does not use WSGI (by default). The point is that pypy can crunch almost as fast as Node.js, so while Node.js may set the standard, Python is not bad choice either from performance perspective.

mercuryrising 14y ago

Where's the source code? I'd like to see how long it takes someone who doesn't know either language to figure out what's happening in the code (a 'new' kind of response time).

ricardobeat 14y ago

Red and green side-by-side on a chart... noo :( http://jfly.iam.u-tokyo.ac.jp/color/#pallet http://jfly.iam.u-tokyo.ac.jp/color/#pallet http://tools.medialab.sciences-po.fr/iwanthue/ http://tools.medialab.sciences-po.fr/iwanthue/

swah 14y ago
IMO its a little bit excessive to ask that every graph generated is color-blind safe... Isn't there software for colorblind people to correct that? (Something that changes the palette of colors displayed...)
- ricardobeat 14y ago
  Red & green is the stereotypical colorblind test, that's a pretty obvious combination to avoid. It's easy to choose colors that are at right angles in the color wheel, and that's not how accessibility works - you can't require every paraplegic to have a special wheelchair that can climb stairs, despite it existing...

justin_vanw 14y ago

Things like this DO NOT MATTER. Facebook was built on PHP and MySQL, both of which are awful pieces of crap. Twitter went down like every week for a year. It was written on RoR by people who didn't really know RoR. Meanwhile, a bunch of better engineers using better technology got their clocks cleaned while the winners eventually had so much money they could afford to hire an army of engineers to rewrite everything. If you know Javascript really well, use Node (although I feel sorry for you). If you know Python really well, use Python (yay). If you don't know anything, well, pick something. If you are choosing a programming language because of how many QPS you can get on some trivial task, you are an idiot. You know the graph I would like to see? A scatterplot of how good various startup founders and early team were on engineering vs finding a market fit, with the dots labeled with how rich they are now.