14 ms·

Tornado Without a GIL on PyPy STM

160 points by fijal 12y ago

error54 12y ago
Note: If anyone was wondering what STM was, it stands for Software Transactional Memory. The read the docs page gives a good overview[1] of what pypy-stm is. 1-http://pypy.readthedocs.org/en/latest/stm.html http://pypy.readthedocs.org/en/latest/stm.html
amelius 12y ago
I'm wondering, how does PyPy compare to nodejs?
- Fede_V 12y ago
  Like apples compare to oranges :) Cheeky comment aside, PyPy is an alternative interpreter for Python and, more broadly, a meta-frame work to make writing tracing JIT for the language of your choice easy. It's much faster than regular Python. The only downside (which, admittedly, is very, very big) is that the PyPy developers released their own FFI library for interfacing with C-code (which works brilliantly), but using c-types and the c-python c-api is very kludgy. This means that a lot of very important python libraries which are basically thin wrappers around c/fortran code (SciPy, NumPy, etc) are not really working. There is some effort around reimplementing NumPy in PyPy, but it's going very slowly. Edit: PyPy is also a heroic effort by a small team of developers who receive some donations. JS is probably the language that received the most amount of money and attention into making it run fast from Google/Mozilla, etc - in terms of results/money, PyPy is incredible.
  - fiedzia 12y ago
    To be a bit more specific, pypy can be much faster, for certain usecases. For others it may not be, or it may be even significanlty slower - writing JIT for python is not an easy task.
  - TillE 12y ago
    > but using c-types and the c-python c-api is very kludgy It's easy enough to use CPython extension modules via cpyext, but it's really slow.
- rdtsc 12y ago
  The same way LLVM compares to haproxy ;-) PyPy is a project to bring some optimization to Python. Basically make Python run faster. It is also effectively another implementation of Python. CPython is the default one (the one you download at python.org). There is also Jython (running Python on the JVM), and PyPy, and a few others. NodeJS is the marriage of the V8 Javascript interpreter and JIT with an asynchronous IO library (libuv) + a large ecosystem of modules. You might want to compare nodejs with PyPY+Tornado or with PyPy+eventlet. Read about STM and the idea behind it. STM lets you take advantage of multiple cores. nodejs is single threaded. In practice nodejs might be faster currently just because V8 is very good and depending on workload if most of the stuff it does is just proxying data from one stream to another, it might do pretty well. But if you start doing a large number of concurrent requests where each request has do to some logic, PyPy might come out on top. In general anywhere with complicated business logic or a large number of steps needed in the backend to handle requests, I wouldn't use nodejs. I never liked the callback/errback paradigm for large concurrent applications. That works for demos and short web tutorials, in practice, I don't like how it looks. I like green threads, lightweight processes better.
  - fijal 12y ago
    "In practice nodejs might be faster currently just because V8 is very good" and I presume you claim PyPy is not so good? Well if so, I would really like to say [citation needed], especially for workloads and not say computer language shootout, which has little to do with performance of doing any actual workload.
    - rdtsc 12y ago
      > and I presume you claim PyPy is not so good? Of course not, Maciej ;-) PyPy is awesome and thank you and the whole team you have been doing a great job so far. Just looking at performance graphs and speedups gained over the years. It looks very impressive. Yeah I don't have a citation but from experience, I have noticed where there are a few steps in each request processing (think proxies) solutions based event loops (epoll, kqueue and friends) can outperform those that spawn a thread/process/context. For example haproxy is certainly a very well done fast proxy, it is single threaded and it seems to work for it. Again sorry for misunderstanding, I was just speculating without any benchmarks or even particular applications in mind.
- illumen 12y ago
  Node.js has a pretty good Garbage Collector for a web browser.
- glibgil 12y ago
  Sigh, I'll give you the only honest answer it seems you will get here. Node.js runs JavaScript faster than anything runs Python, PyPy included. If you want speed and you like JavaScript just as much as Python, then use Node.js. When I say "faster" I mean you will use less servers/cores for your application with Node.js than you will with CPython or PyPy. V8 is just a better and faster VM for its target language.
  - BuckRogers 12y ago
    But that wouldn't be a true statement. Google's engineers didn't laugh at Node.js for using V8 due to arrogance. V8 is limited to a single CPU thread per instance. It's great for I/O and a typical webpage, but a CPU task stops everything. PyPy with STM does not have this limitation. Once this project is out of beta, V8 wouldn't compare to PyPy/STM. A better comparison would be the JVM or CLR.
    - pjmlp 12y ago
      There are plenty of JVMs to compare to.
  - fijal 12y ago
    "Node.js runs JavaScript faster than anything runs Python, PyPy included" that's a [citation needed] right here. Do you have some facts that I don't happen to have? Please share. That said, I don't care about a recursive fibonacci or computer language shootout problems.
    - glibgil 12y ago
      Benchmarks are all you need to know unless a language runtime has IO problems. Fewer instructions and less memory mean fewer servers. Benchmarks are really good at predicting general performance. Languages with slow benchmarks need more servers to run your app. Languages with fast benchmarks need far fewer servers to run your app. This fact is so obvious and testable that I don't know why slow-language people bother throwing up the argument that benchmarks aren't everything.
bwlandstreet 12y ago
Sometimes I forget the number of great contributions / projects that Bret Taylor keystroked. It's a motivating reminder to build.
- fijal 12y ago
  I think your comment is moderately off-topic. This is really not about tornado at all, rather than you can take medium-sized program and get STM in PyPy not to crash
pjmlp 12y ago
Maybe one day, PyPy will eventually become the canonical implementation of Python.
- fijal 12y ago
  PyPy does not really try to do that. CPython is great for a lot of things, small, easy to install on a myriad of platforms and is, after all, Python. PyPy instead tries to push what's possible to do with Python as a language and with dynamic language interpreters. You can do real time image processing using only Python with PyPy, now STM, fast numerics in the future etc. The goal is to expand Python ecosystem, not to "replace" CPython.
  - joelthelion 12y ago
    Having multiple competing implementations is hurting the ecosystem, with duplicated work, compatibility problems, etc.
    - glibgil 12y ago
      CPython is hurting the ecosystem by being slow. I'll let you in on a little secret. People used to using C, C++, D even Java and C# laugh at the slowness of languages like Python and Ruby. Blah, blah, blah productivity gains. We laugh. We think, "I could write this in Java and go to production with three servers or I could write it in Python and use eleven." People don't like to say it to your face, but Python is often derided and considered to be pretty silly. Oh, and its syntax is like Lego. Neat at first until you want something like a multiline lambda.
      - ubernostrum 12y ago
        C, C++, D, Java and C# are hurting the ecosystem by being slow. I'll let you in on a little secret. People used to coding assembly laugh at the slowness of languages like C and C++. Blah, blah, blah productivity gains. We laugh. We think, "I could write this in assembly and go to production on an Arduino or I could write it in C and have to use expensive servers." People don't like to say it to your face, but C is often derided and considered to be pretty silly. Oh, and its syntax is like Lego. Neat at first until you want to understand what something is a pointer to.
reduce 12y ago
Am I reading this correctly? Looking at the benchmark code, Why not do a comparison versus the default Tornado setup, which is to fork one process per core? So STM Tornado is allowed to use multiple cores in this benchmark, but vanilla Tornado is not allowed to? http_server = HTTPServer(Application(), xheaders=True, ) http_server.bind(port) http_server.start(0) # Forks one sub-process per core [1] https://bitbucket.org/kostialopuhin/tornado-stm-bench/src/65144cda7a1f8e56db6d3d9057d7d39a1652e622/utils.py?at=default https://bitbucket.org/kostialopuhin/tornado-stm-bench/src/65...
- fijal 12y ago
  The problem with multiple processes is the "share nothing" model. It works for some problems, but it blatantly fails for a whole variety of other problems. STM tries to address those problems where "share nothing" does not work, e.g. because there is interesting data to be shared (albeit with few conflicts) or the memory overhead of N processes is just too much.
nickbauman 12y ago
I have found personally A* is particularly difficult to scale across cores because it's never a shared-nothing problem. For two reasons. One, each core has to know about the other cores' search space and avoid it (to avoid duplicating effort) so you will contend on some sort of 'visited node' cache. Two, the graph your building itself must be shared, obvs. Are most of the interesting problems to solve always limited by Amdahl's Law? Will we never see the gains of single-core speed we saw in the last century again?
- desdiv 12y ago
  One, each core has to know about the other cores' search space and avoid it (to avoid duplicating effort) so you will contend on some sort of 'visited node' cache. Just make the visited node cache public and immutable. Two, the graph your building itself must be shared, obvs. If the graph is immutable then there's zero problem with it being shared.
  - jerven 12y ago
    You can't make the cache immutable because then it will be empty at start and stays that way ;) The cache has to mutate and be shared as that's the work completed list. As each thread completes a bit of work (visits a node) it needs to communicate it with the other threads.
    - desdiv 12y ago
      In Scala, its: var cache = Seq(1,2,3) cache :+= 4 The cache is immutable and freely shareable. Any other thread and come in and read it and be guaranteed that its current state is valid.
      - jerven 12y ago
        :+= Returns A copy of this sequence with an element appended. Meaning the cache is no longer shared as all threads end up having a thread local cache. You can't update shared changing state and have immutability at the same time. Immutability is great when one can have it but sometimes its not possible. Shared changing state is something to be avoid as much as possible but sometimes we need it.
haberman 12y ago
I recently read this paper about Julia (http://arxiv.org/abs/1411.1607 http://arxiv.org/abs/1411.1607). Now when I hear about PyPy I can't help but think about this quote from that paper: New users also want a quick explanation as to why Julia is fast, and whether somehow the same “magic dust” could also be sprinkled on their traditional scientific computing language. [...] Julia is fast because we, the designers, developed it that way for us, the users. Performance is fragile, like accuracy, one arithmetic error can ruin an entire otherwise correct computation. We do not believe that a language can be designed for the human, and retrofitted for the computer. Rather a language must be designed from the start for the human and the computer. To me, Python and Ruby are both perfect examples of languages that were designed for the human, and ever since have seen extensive effort to retrofit them for fast execution by computers. I respect the work of the PyPy team, particularly given the raving reviews I've seen lately of how RPython is a boon to language designers who can use it to prototype their languages and get a decently-performing VM in not very much time: http://tratt.net/laurie/blog/entries/fast_enough_vms_in_fast_enough_time http://tratt.net/laurie/blog/entries/fast_enough_vms_in_fast... But I can't help but think that languages like Python and Ruby will start to fall to languages like Swift, Julia, Go, etc. that were designed with performance in mind. I'm not saying this will happen soon, but these languages are showing that you can have your cake and eat it too. I'm not sure how JavaScript and Lua fit into this analysis. They weren't specifically designed for performance, but have been very successfully optimized. Lua is a very simple language and Mike Pall is a genius, so LuaJIT has been very successful at speeding up Lua. JavaScript is a little more complicated, but has received an immense amount of resources into optimizing it, and has also been quite successful at getting fast.
- tiffanyh 12y ago
  Agreed. Where does Lua then fit into your analogy?
  - justincormack 12y ago
    Lua was designed, and has things taken out, and has its semantics simplified (and well documented). And it is small.
fijal 12y ago
While we're here, please donate to PyPy STM, it's purely a crowdfunded effort http://pypy.org/tmdonate2.html http://pypy.org/tmdonate2.html
- tshepang 12y ago
  Please create a PyPy team on Gratipay.
  - fijal 12y ago
    here is mine - https://gratipay.com/fijal/ https://gratipay.com/fijal/ - not exactly killing it
    - tshepang 12y ago
      Better it be the team account, to avoid having donors try decide "who contributes how much"... let the team decide. Also, please offer it as an alternative to PayPal on the project website.
      - fijal 12y ago
        We can't really be creating accounts everywhere for minor donations. I support your sentiment, but the official PyPy bookkeeping has to be done in a proper way via the Software Freedom Conservancy, so going through all those services for a few $$$ is simply not worth it.
        tshepang 12y ago
        Why doubt the amount that can be received from Gratipay? What do you need to see to consider using it for PyPy funding? Would a promise of, say $500/week, be enough to make it worth a bother?
zaphar 12y ago
I like that they instrumented the STM code enough that you can debug slowdowns when writing your code. Never understimate the power of well instrumented languages.
exacube 12y ago
This is some great progress! But I should say that there is still too much overhead in using STM; you will still be able to very easily (and by a large margin) outperform STM-4 by running 4 instances of Tornado with HAProxy or some other lightweight router ontop. A comparison graph for this should have been the benchmark.

160 points by fijal 12y ago

Note: If anyone was wondering what STM was, it stands for Software Transactional Memory. The read the docs page gives a good overview[1] of what pypy-stm is. 1-http://pypy.readthedocs.org/en/latest/stm.html http://pypy.readthedocs.org/en/latest/stm.html

amelius 12y ago

I'm wondering, how does PyPy compare to nodejs?

Fede_V 12y ago
Like apples compare to oranges :) Cheeky comment aside, PyPy is an alternative interpreter for Python and, more broadly, a meta-frame work to make writing tracing JIT for the language of your choice easy. It's much faster than regular Python. The only downside (which, admittedly, is very, very big) is that the PyPy developers released their own FFI library for interfacing with C-code (which works brilliantly), but using c-types and the c-python c-api is very kludgy. This means that a lot of very important python libraries which are basically thin wrappers around c/fortran code (SciPy, NumPy, etc) are not really working. There is some effort around reimplementing NumPy in PyPy, but it's going very slowly. Edit: PyPy is also a heroic effort by a small team of developers who receive some donations. JS is probably the language that received the most amount of money and attention into making it run fast from Google/Mozilla, etc - in terms of results/money, PyPy is incredible.
- fiedzia 12y ago
  To be a bit more specific, pypy can be much faster, for certain usecases. For others it may not be, or it may be even significanlty slower - writing JIT for python is not an easy task.
- TillE 12y ago
  > but using c-types and the c-python c-api is very kludgy It's easy enough to use CPython extension modules via cpyext, but it's really slow.
rdtsc 12y ago
The same way LLVM compares to haproxy ;-) PyPy is a project to bring some optimization to Python. Basically make Python run faster. It is also effectively another implementation of Python. CPython is the default one (the one you download at python.org). There is also Jython (running Python on the JVM), and PyPy, and a few others. NodeJS is the marriage of the V8 Javascript interpreter and JIT with an asynchronous IO library (libuv) + a large ecosystem of modules. You might want to compare nodejs with PyPY+Tornado or with PyPy+eventlet. Read about STM and the idea behind it. STM lets you take advantage of multiple cores. nodejs is single threaded. In practice nodejs might be faster currently just because V8 is very good and depending on workload if most of the stuff it does is just proxying data from one stream to another, it might do pretty well. But if you start doing a large number of concurrent requests where each request has do to some logic, PyPy might come out on top. In general anywhere with complicated business logic or a large number of steps needed in the backend to handle requests, I wouldn't use nodejs. I never liked the callback/errback paradigm for large concurrent applications. That works for demos and short web tutorials, in practice, I don't like how it looks. I like green threads, lightweight processes better.
- fijal 12y ago
  "In practice nodejs might be faster currently just because V8 is very good" and I presume you claim PyPy is not so good? Well if so, I would really like to say [citation needed], especially for workloads and not say computer language shootout, which has little to do with performance of doing any actual workload.
  - rdtsc 12y ago
    > and I presume you claim PyPy is not so good? Of course not, Maciej ;-) PyPy is awesome and thank you and the whole team you have been doing a great job so far. Just looking at performance graphs and speedups gained over the years. It looks very impressive. Yeah I don't have a citation but from experience, I have noticed where there are a few steps in each request processing (think proxies) solutions based event loops (epoll, kqueue and friends) can outperform those that spawn a thread/process/context. For example haproxy is certainly a very well done fast proxy, it is single threaded and it seems to work for it. Again sorry for misunderstanding, I was just speculating without any benchmarks or even particular applications in mind.
illumen 12y ago
Node.js has a pretty good Garbage Collector for a web browser.
glibgil 12y ago
Sigh, I'll give you the only honest answer it seems you will get here. Node.js runs JavaScript faster than anything runs Python, PyPy included. If you want speed and you like JavaScript just as much as Python, then use Node.js. When I say "faster" I mean you will use less servers/cores for your application with Node.js than you will with CPython or PyPy. V8 is just a better and faster VM for its target language.
- BuckRogers 12y ago
  But that wouldn't be a true statement. Google's engineers didn't laugh at Node.js for using V8 due to arrogance. V8 is limited to a single CPU thread per instance. It's great for I/O and a typical webpage, but a CPU task stops everything. PyPy with STM does not have this limitation. Once this project is out of beta, V8 wouldn't compare to PyPy/STM. A better comparison would be the JVM or CLR.
  - pjmlp 12y ago
    There are plenty of JVMs to compare to.
- fijal 12y ago
  "Node.js runs JavaScript faster than anything runs Python, PyPy included" that's a [citation needed] right here. Do you have some facts that I don't happen to have? Please share. That said, I don't care about a recursive fibonacci or computer language shootout problems.
  - glibgil 12y ago
    Benchmarks are all you need to know unless a language runtime has IO problems. Fewer instructions and less memory mean fewer servers. Benchmarks are really good at predicting general performance. Languages with slow benchmarks need more servers to run your app. Languages with fast benchmarks need far fewer servers to run your app. This fact is so obvious and testable that I don't know why slow-language people bother throwing up the argument that benchmarks aren't everything.

bwlandstreet 12y ago

Sometimes I forget the number of great contributions / projects that Bret Taylor keystroked. It's a motivating reminder to build.

fijal 12y ago
I think your comment is moderately off-topic. This is really not about tornado at all, rather than you can take medium-sized program and get STM in PyPy not to crash

pjmlp 12y ago

Maybe one day, PyPy will eventually become the canonical implementation of Python.

fijal 12y ago
PyPy does not really try to do that. CPython is great for a lot of things, small, easy to install on a myriad of platforms and is, after all, Python. PyPy instead tries to push what's possible to do with Python as a language and with dynamic language interpreters. You can do real time image processing using only Python with PyPy, now STM, fast numerics in the future etc. The goal is to expand Python ecosystem, not to "replace" CPython.
- joelthelion 12y ago
  Having multiple competing implementations is hurting the ecosystem, with duplicated work, compatibility problems, etc.
  - glibgil 12y ago
    CPython is hurting the ecosystem by being slow. I'll let you in on a little secret. People used to using C, C++, D even Java and C# laugh at the slowness of languages like Python and Ruby. Blah, blah, blah productivity gains. We laugh. We think, "I could write this in Java and go to production with three servers or I could write it in Python and use eleven." People don't like to say it to your face, but Python is often derided and considered to be pretty silly. Oh, and its syntax is like Lego. Neat at first until you want something like a multiline lambda.
    - ubernostrum 12y ago
      C, C++, D, Java and C# are hurting the ecosystem by being slow. I'll let you in on a little secret. People used to coding assembly laugh at the slowness of languages like C and C++. Blah, blah, blah productivity gains. We laugh. We think, "I could write this in assembly and go to production on an Arduino or I could write it in C and have to use expensive servers." People don't like to say it to your face, but C is often derided and considered to be pretty silly. Oh, and its syntax is like Lego. Neat at first until you want to understand what something is a pointer to.

reduce 12y ago

Am I reading this correctly? Looking at the benchmark code, Why not do a comparison versus the default Tornado setup, which is to fork one process per core? So STM Tornado is allowed to use multiple cores in this benchmark, but vanilla Tornado is not allowed to? http_server = HTTPServer(Application(), xheaders=True, ) http_server.bind(port) http_server.start(0) # Forks one sub-process per core [1] https://bitbucket.org/kostialopuhin/tornado-stm-bench/src/65144cda7a1f8e56db6d3d9057d7d39a1652e622/utils.py?at=default https://bitbucket.org/kostialopuhin/tornado-stm-bench/src/65...

fijal 12y ago
The problem with multiple processes is the "share nothing" model. It works for some problems, but it blatantly fails for a whole variety of other problems. STM tries to address those problems where "share nothing" does not work, e.g. because there is interesting data to be shared (albeit with few conflicts) or the memory overhead of N processes is just too much.

nickbauman 12y ago

I have found personally A* is particularly difficult to scale across cores because it's never a shared-nothing problem. For two reasons. One, each core has to know about the other cores' search space and avoid it (to avoid duplicating effort) so you will contend on some sort of 'visited node' cache. Two, the graph your building itself must be shared, obvs. Are most of the interesting problems to solve always limited by Amdahl's Law? Will we never see the gains of single-core speed we saw in the last century again?

desdiv 12y ago
One, each core has to know about the other cores' search space and avoid it (to avoid duplicating effort) so you will contend on some sort of 'visited node' cache. Just make the visited node cache public and immutable. Two, the graph your building itself must be shared, obvs. If the graph is immutable then there's zero problem with it being shared.
- jerven 12y ago
  You can't make the cache immutable because then it will be empty at start and stays that way ;) The cache has to mutate and be shared as that's the work completed list. As each thread completes a bit of work (visits a node) it needs to communicate it with the other threads.
  - desdiv 12y ago
    In Scala, its: var cache = Seq(1,2,3) cache :+= 4 The cache is immutable and freely shareable. Any other thread and come in and read it and be guaranteed that its current state is valid.
    - jerven 12y ago
      :+= Returns A copy of this sequence with an element appended. Meaning the cache is no longer shared as all threads end up having a thread local cache. You can't update shared changing state and have immutability at the same time. Immutability is great when one can have it but sometimes its not possible. Shared changing state is something to be avoid as much as possible but sometimes we need it.

haberman 12y ago

I recently read this paper about Julia (http://arxiv.org/abs/1411.1607 http://arxiv.org/abs/1411.1607). Now when I hear about PyPy I can't help but think about this quote from that paper: New users also want a quick explanation as to why Julia is fast, and whether somehow the same “magic dust” could also be sprinkled on their traditional scientific computing language. [...] Julia is fast because we, the designers, developed it that way for us, the users. Performance is fragile, like accuracy, one arithmetic error can ruin an entire otherwise correct computation. We do not believe that a language can be designed for the human, and retrofitted for the computer. Rather a language must be designed from the start for the human and the computer. To me, Python and Ruby are both perfect examples of languages that were designed for the human, and ever since have seen extensive effort to retrofit them for fast execution by computers. I respect the work of the PyPy team, particularly given the raving reviews I've seen lately of how RPython is a boon to language designers who can use it to prototype their languages and get a decently-performing VM in not very much time: http://tratt.net/laurie/blog/entries/fast_enough_vms_in_fast_enough_time http://tratt.net/laurie/blog/entries/fast_enough_vms_in_fast... But I can't help but think that languages like Python and Ruby will start to fall to languages like Swift, Julia, Go, etc. that were designed with performance in mind. I'm not saying this will happen soon, but these languages are showing that you can have your cake and eat it too. I'm not sure how JavaScript and Lua fit into this analysis. They weren't specifically designed for performance, but have been very successfully optimized. Lua is a very simple language and Mike Pall is a genius, so LuaJIT has been very successful at speeding up Lua. JavaScript is a little more complicated, but has received an immense amount of resources into optimizing it, and has also been quite successful at getting fast.

tiffanyh 12y ago
Agreed. Where does Lua then fit into your analogy?
- justincormack 12y ago
  Lua was designed, and has things taken out, and has its semantics simplified (and well documented). And it is small.

fijal 12y ago

While we're here, please donate to PyPy STM, it's purely a crowdfunded effort http://pypy.org/tmdonate2.html http://pypy.org/tmdonate2.html

tshepang 12y ago
Please create a PyPy team on Gratipay.
- fijal 12y ago
  here is mine - https://gratipay.com/fijal/ https://gratipay.com/fijal/ - not exactly killing it
  - tshepang 12y ago
    Better it be the team account, to avoid having donors try decide "who contributes how much"... let the team decide. Also, please offer it as an alternative to PayPal on the project website.
    - fijal 12y ago
      We can't really be creating accounts everywhere for minor donations. I support your sentiment, but the official PyPy bookkeeping has to be done in a proper way via the Software Freedom Conservancy, so going through all those services for a few $$$ is simply not worth it.
      - tshepang 12y ago
        Why doubt the amount that can be received from Gratipay? What do you need to see to consider using it for PyPy funding? Would a promise of, say $500/week, be enough to make it worth a bother?

zaphar 12y ago

I like that they instrumented the STM code enough that you can debug slowdowns when writing your code. Never understimate the power of well instrumented languages.

exacube 12y ago

This is some great progress! But I should say that there is still too much overhead in using STM; you will still be able to very easily (and by a large margin) outperform STM-4 by running 4 instances of Tornado with HAProxy or some other lightweight router ontop. A comparison graph for this should have been the benchmark.