7 ms·

Ian Bicking: My Unsolicited Advice For PyPy

121 points by jnoller 16y ago

ominous_prime 16y ago
I agree. I think there needs to be an effort to push towards concurrency in Pypy. They were optimistic that the GIL would be easier to factor out in their runtime, and I think that could be a great start. I'm all for new methods for concurrency, but I feel being able to make efficient use of native threads would bring a lot of momentum to the project. Even though it's not as big a deal as most people make it out to be, Python gets a bad rap for the well publicized GIL problems.
- jnoller 16y ago
  Everyone is optimistic that they can factor out the GIL (see: Unladen Swallow). Then they find out the problem is much, much harder then they initially thought. That said, all it requires is a fundamental change (e.g. dropping) the ref counting system: http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Global_Interpreter_Lock http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Gl... So, perhaps PyPy can do it, and I hope they do - from an evolutionary standpoint, they're the interpreter with the most velocity inside of the interpreter core development, so if it's going to happen, it's going to happen there. CPython will stay conservative, and much as it is today for some time.
  - ianb 16y ago
    For a while I had it in my mind that maybe you could save some memory while having safety just by using os.fork() after pre-importing a bunch of stuff; but then reference counting was pointed out to me and I realized it would be pointless -- the act of merely looking at an object changes the object, so copy-on-write means copy-everything. Reference counting seems like it could be removed... except for all the C extensions that are specifically handling reference counting internally, removal of which I'm guessing would cause a lot of breakage. But then... it seems like those C extensions are exactly the problem, so if PyPy tries to keep them is it messing itself up? (Though I'm guessing it would somehow use proxy objects or some other trick to keep its internal object representation flexible.)
    - jjs 16y ago
      Have your cake and eat it too: open a socket to a CPython process and let it handle the legacy C extensions.
      - jnoller 16y ago
        http://morepypy.blogspot.com/2010/04/using-cpython-extension-modules-with.html http://morepypy.blogspot.com/2010/04/using-cpython-extension...
    - jnoller 16y ago
      Ah! But you see, it's possible to trick CPython extensions into thinking you do support refcounting: http://code.google.com/p/ironclad/ http://code.google.com/p/ironclad/ is proof. So you can be compatible, yet advanced :)
    - 1amzave 16y ago
      > ...reference counting was pointed out to me and I realized it would be pointless -- the act of merely looking at an object changes the object, so copy-on-write means copy-everything. Hmm...might it be worthwhile to work around this by storing reference counts in a separate set of pages, and having each object instead hold a constant pointer to its refcount? Presumably the extra level of indirection would have some degree of performance impact, but if it allows CoW to actually work in a meaningful way... Though I guess that would probably break every C extension in existence, no? (Note: I know absolutely nothing about Python's C extension API, just guessing.)
hyperbovine 16y ago
Ahh, but PyPy is not being written by web developers -- or for them. The main guy behind the project is a research computer scientist (http://cfbolz.de/ http://cfbolz.de/) and most of the people who are excited about it are also doing scientific computing. I am one of them, and the ability to write scientific code in Python that runs as fast (or faster!) than C would be so awesome that I can hardly put it into words. Unlike web development, performance is still a huge bottleneck in this field; people still mill about for 3 weeks waiting for their job to finish just like it was the 1970s. Consequently, most code written for research projects consists of lot of arcane, difficult-to-read C, C++, or even (gasp!) FORTRAN. The benefits of switching to Python in terms of development time, ease of debugging, reusability, potential for collaboration, reproducibility, etc. are immense, but the performance penalty is such that it rarely happens. Even in spite of that, Python already has a large academic following; if PyPy manages to pull off its stated goals, it would be revolutionary.
- joe_the_user 16y ago
  A lot of scientific code also runs on the (crufty, spiffy) Matlab, which is fast but not super-fast, is less portable than most things but has good usability and lots of history. A python that could run incrementally, with robust threads, in the ground might, just possibly, make SAGE a better environment and so increase migration from MATLAB (my personal scientific computing experience was with Matlab and anything that escape this monster is great in my book). I mean, Python won't become faster than optimized C so however convincing or unconvincing the faster iteration and usability argument is, that, in the end is going have to be what Python makes.
- justincormack 16y ago
  Right now it looks like LuaJIT is your best option for this type of performance. PyPy might get there eventually though...
  - ominous_prime 16y ago
    LuaJIT is awesome, but python already has a large volume of scientific libraries. I haven't heard mention of scientific researchers using Lua (yet).
btmorex 16y ago
The first two points are describing fork(), which python already has on unix-like systems. If he means something different, then he needs to explain more how exactly it would work (especially regarding hardware/os support).
- ianb 16y ago
  Interpreter-level fork() is very much what I mean ("green processes" would be another phrasing I guess?) Of course the Python interpreter always acts as an intermediary for any access to resources, so I feel like you can kind of punt on that, call it an interpreter implementation detail. fork() isn't awesome with CPython, as its reference counting kind of destroys the utility of copy-on-write (reading an object creates a reference, which increments the object's reference count, which is a write). And there's really a lot of shared data that you can have between processes, since every module you import is data (and expensive data!) -- I'd like to see it feasible to create extremely short-lived processes, for instance (why have a worker pool at all?) Once you've created a new microprocess I suppose the actual concurrency would be up to the interpreter to best decide (or at least there would be a lot of flexibility because that concurrency would not be so overtly exposed to the Python program) -- it would be reasonable for a Python microprocess to be an OS-level thread, for instance, or for it to simply be cooperative multitasking at the VM level, or even be a real OS fork (though without an OS fork it seems reasonable to share immutable objects, but once forked the objects must be immutable and serializable).
  - sanxiyn 16y ago
    Eh, PyPy does not use reference counting. Are you saying PyPy + fork() is what you want?
    - vak 16y ago
      BTW, PyPy (unlike CPython) indeed great for multiprocessing. CPython's ref-counting forces copy-on-write in RAM even when no write operation have been even applied at all. So, if you have a huge read-only object in RAM forget about multiprocessing. The object will be cloned for each process.
haberman 16y ago
"What do I want? [...] Shared objects across [micro-]processes with copy-on-write; then you can efficiently share objects (like modules!) across concurrent processes without the danger of shared state, but without the overhead of copying everything you want to share." Is overhead of modules really an issue on servers with GB of RAM? More importantly, almost every multi-threaded string library (like C++ STL) has moved away from sharing across threads and COW because the cost of the atomic operations is too high. See Herb Sutter's "Optimizations That Aren't (In a Multithreaded World)." http://www.gotw.ca/publications/optimizations.htm http://www.gotw.ca/publications/optimizations.htm Also, to say that speed is "really uninteresting" but latency is important seems like a contradiction. Latency is absolutely gated by speed for non-parallelizable problems. And even if the problem can be parallelized, speed is a much easier and simpler way to decrease latency than parallelizing. I guess this reads to me more like a list of theoretically interesting ideas than a set of features that will actually help anyone in the real world.
- ianb 16y ago
  RAM seems relatively constrained to me -- at least, running a multiprocess server without a few GB or RAM is hard to do (and memory seems to be the most expensive part of servers). And when it goes wrong (you reach whatever your limit is) then things tend to fail in less than ideal ways. Tools around memory usage are also quite poor, so while performance gets optimized memory seldom does. And a culture of cavalier memory usage doesn't help either -- too many people are borrowing memory to get performance, compounded in the case of an application that is typically an aggregation of many people's work. The amount of code involved in systems is also continuing to go up, so that the amount of memory you use before you've done anything (but when you've loaded up all the code) is getting constantly higher. In the world of static/compiled languages this might not be as notable as there's a clear sense of "code" and "data" -- Python has no such distinction, so if you don't share anything then you don't share anything. WRT speed-vs-latency -- definitely related, but most benchmarking seems to specifically remove latency from the benchmark and instead test throughput. E.g., tests frequently throw away the slowest run, even though abnormally slow runs exist in real programs and can have an effect. (Of course they aren't predictable and might be affected by other things on the system -- which is a kind of bias to not optimize things that are hard to measure). But latency is mostly just simplicity of implementation, so no, I wouldn't expect parallelizing to help.
baltcode 16y ago
In addition to speed and the points raised by the OP, I am interested in static/duck typing as it relates to both speed and static error-checking and documentation. i.e., making debugging easier and faster. I somehow think this can happen while keeping almost all of Python's strengths and ease of development.
- ianb 16y ago
  Python 3 added type annotations, but I've yet to see much interest in actually using it. Traits (http://code.enthought.com/projects/traits/ http://code.enthought.com/projects/traits/) pursues the error-checking side of type declarations, while Cython pursues the speed side. If you want both then too bad ;)
mrshoe 16y ago
It seems like he could get most of what he wants without a new runtime. Diesel (http://dieselweb.org http://dieselweb.org) provides async without callbacks or Deferreds. Diesel1 used Python 2.5's yield expression, and Diesel2 switched to using greenlets. As long as your modules and your applications don't rely on global state (which they shouldn't anyway), you essentially get everything on this list. For web programming, cooperative multitasking usually works perfectly well and no state is generally shared between any 2 http requests, so I'm not sure how much a web programmer stands to gain from preemptive multitasking and completely separate address spaces with CoW. The one thing Diesel is missing is good runtime introspection and more primitives to encourage reliability (a la Erlang), but if you hang out in #diesel you'll see that jamwt has already implemented a lot of that at Bump and he plans on pushing it down into the framework soon (probably for Diesel3).
- ianb 16y ago
  All these systems are share-everything, which I'm not excited about -- if I want share-everything I can just use threads, and they work far better than most people seem willing to admit.
  - mrshoe 16y ago
    Well threads certainly don't work well for high concurrency, which I read as your biggest impetus for a new runtime. I suppose if you mean true (multi-core) concurrency then you're right that Python has some big limitations, but that pretty much boils down to the age old "remove the GIL" proposal.
    - ianb 16y ago
      Threads work pretty darn well, actually. Depending on the libraries you use the GIL may not get in the way (e.g., database libraries, stuff like lxml that releases the GIL). If you have blocking stuff like I/O (socket or file) then threads help. The only time when the GIL really gets in the way is with multiple processors and CPU-bound routines. Even with CPU-bound routines you are often better off with threads as the machinery does share the CPU transparently and relatively fairly. Threads are a little half-assed, sure, but so are most of the currently offered "replacements" for threads, including cooperative multitasking (which also doesn't do anything with multiple processors!)
n_are_q 16y ago
For what it's worth, another 2c from another web developer - for me memory isolation and COW for module dependencies would be less useful than a single process and a lack of GIL (hold the flamethrowers for one second please) - allowing me to share on a multicore server resources like those same modules, as well as database connections, as well as any significant static data i may read in at start up, decreasing my operational complexity, etc. In a web server environment shared memory isn't particularly scary, to me anyway. Am I misunderstanding the reason for COW? Also, the most important characteristic to me is latency. I don't care about throughput that much as long as it's "good enough", hardware is getting really cheap really quickly anyway (in terms of cores and RAM/$). Latency on the other hand is tough. While IO is normally going to be the biggest factor here, if the runtime allows me to consistently shave 5ms off of every request - that's still a great improvement. In this sense i would say that raw speed IS important to me. Perhaps I haven't encountered the use cases Ian is talking about, it was difficult to figure them out from the blog post (i'm a relative novice in Python). But I am hoping PyPy can at some point help with the concerns I have above, even lower latency by itself may be compelling enough to switch.
- ianb 16y ago
  Copy-on-write lets you efficiently share objects across boundaries -- not copying everything right away (and everything in Python means everything), while also safely avoiding the sharing of state. If you are cool with threads then it's not an issue. If you can spare the overhead of a few processes then the GIL isn't an issue. I'd rather it not be a matter of tradeoffs, and I don't think either OS processes or threads are it.

121 points by jnoller 16y ago

I agree. I think there needs to be an effort to push towards concurrency in Pypy. They were optimistic that the GIL would be easier to factor out in their runtime, and I think that could be a great start. I'm all for new methods for concurrency, but I feel being able to make efficient use of native threads would bring a lot of momentum to the project. Even though it's not as big a deal as most people make it out to be, Python gets a bad rap for the well publicized GIL problems.

jnoller 16y ago
Everyone is optimistic that they can factor out the GIL (see: Unladen Swallow). Then they find out the problem is much, much harder then they initially thought. That said, all it requires is a fundamental change (e.g. dropping) the ref counting system: http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Global_Interpreter_Lock http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Gl... So, perhaps PyPy can do it, and I hope they do - from an evolutionary standpoint, they're the interpreter with the most velocity inside of the interpreter core development, so if it's going to happen, it's going to happen there. CPython will stay conservative, and much as it is today for some time.
- ianb 16y ago
  For a while I had it in my mind that maybe you could save some memory while having safety just by using os.fork() after pre-importing a bunch of stuff; but then reference counting was pointed out to me and I realized it would be pointless -- the act of merely looking at an object changes the object, so copy-on-write means copy-everything. Reference counting seems like it could be removed... except for all the C extensions that are specifically handling reference counting internally, removal of which I'm guessing would cause a lot of breakage. But then... it seems like those C extensions are exactly the problem, so if PyPy tries to keep them is it messing itself up? (Though I'm guessing it would somehow use proxy objects or some other trick to keep its internal object representation flexible.)
  - jjs 16y ago
    Have your cake and eat it too: open a socket to a CPython process and let it handle the legacy C extensions.
    - jnoller 16y ago
      http://morepypy.blogspot.com/2010/04/using-cpython-extension-modules-with.html http://morepypy.blogspot.com/2010/04/using-cpython-extension...
  - jnoller 16y ago
    Ah! But you see, it's possible to trick CPython extensions into thinking you do support refcounting: http://code.google.com/p/ironclad/ http://code.google.com/p/ironclad/ is proof. So you can be compatible, yet advanced :)
  - 1amzave 16y ago
    > ...reference counting was pointed out to me and I realized it would be pointless -- the act of merely looking at an object changes the object, so copy-on-write means copy-everything. Hmm...might it be worthwhile to work around this by storing reference counts in a separate set of pages, and having each object instead hold a constant pointer to its refcount? Presumably the extra level of indirection would have some degree of performance impact, but if it allows CoW to actually work in a meaningful way... Though I guess that would probably break every C extension in existence, no? (Note: I know absolutely nothing about Python's C extension API, just guessing.)

hyperbovine 16y ago

Ahh, but PyPy is not being written by web developers -- or for them. The main guy behind the project is a research computer scientist (http://cfbolz.de/ http://cfbolz.de/) and most of the people who are excited about it are also doing scientific computing. I am one of them, and the ability to write scientific code in Python that runs as fast (or faster!) than C would be so awesome that I can hardly put it into words. Unlike web development, performance is still a huge bottleneck in this field; people still mill about for 3 weeks waiting for their job to finish just like it was the 1970s. Consequently, most code written for research projects consists of lot of arcane, difficult-to-read C, C++, or even (gasp!) FORTRAN. The benefits of switching to Python in terms of development time, ease of debugging, reusability, potential for collaboration, reproducibility, etc. are immense, but the performance penalty is such that it rarely happens. Even in spite of that, Python already has a large academic following; if PyPy manages to pull off its stated goals, it would be revolutionary.

joe_the_user 16y ago
A lot of scientific code also runs on the (crufty, spiffy) Matlab, which is fast but not super-fast, is less portable than most things but has good usability and lots of history. A python that could run incrementally, with robust threads, in the ground might, just possibly, make SAGE a better environment and so increase migration from MATLAB (my personal scientific computing experience was with Matlab and anything that escape this monster is great in my book). I mean, Python won't become faster than optimized C so however convincing or unconvincing the faster iteration and usability argument is, that, in the end is going have to be what Python makes.
justincormack 16y ago
Right now it looks like LuaJIT is your best option for this type of performance. PyPy might get there eventually though...
- ominous_prime 16y ago
  LuaJIT is awesome, but python already has a large volume of scientific libraries. I haven't heard mention of scientific researchers using Lua (yet).

btmorex 16y ago

The first two points are describing fork(), which python already has on unix-like systems. If he means something different, then he needs to explain more how exactly it would work (especially regarding hardware/os support).

ianb 16y ago
Interpreter-level fork() is very much what I mean ("green processes" would be another phrasing I guess?) Of course the Python interpreter always acts as an intermediary for any access to resources, so I feel like you can kind of punt on that, call it an interpreter implementation detail. fork() isn't awesome with CPython, as its reference counting kind of destroys the utility of copy-on-write (reading an object creates a reference, which increments the object's reference count, which is a write). And there's really a lot of shared data that you can have between processes, since every module you import is data (and expensive data!) -- I'd like to see it feasible to create extremely short-lived processes, for instance (why have a worker pool at all?) Once you've created a new microprocess I suppose the actual concurrency would be up to the interpreter to best decide (or at least there would be a lot of flexibility because that concurrency would not be so overtly exposed to the Python program) -- it would be reasonable for a Python microprocess to be an OS-level thread, for instance, or for it to simply be cooperative multitasking at the VM level, or even be a real OS fork (though without an OS fork it seems reasonable to share immutable objects, but once forked the objects must be immutable and serializable).
- sanxiyn 16y ago
  Eh, PyPy does not use reference counting. Are you saying PyPy + fork() is what you want?
  - vak 16y ago
    BTW, PyPy (unlike CPython) indeed great for multiprocessing. CPython's ref-counting forces copy-on-write in RAM even when no write operation have been even applied at all. So, if you have a huge read-only object in RAM forget about multiprocessing. The object will be cloned for each process.

haberman 16y ago

"What do I want? [...] Shared objects across [micro-]processes with copy-on-write; then you can efficiently share objects (like modules!) across concurrent processes without the danger of shared state, but without the overhead of copying everything you want to share." Is overhead of modules really an issue on servers with GB of RAM? More importantly, almost every multi-threaded string library (like C++ STL) has moved away from sharing across threads and COW because the cost of the atomic operations is too high. See Herb Sutter's "Optimizations That Aren't (In a Multithreaded World)." http://www.gotw.ca/publications/optimizations.htm http://www.gotw.ca/publications/optimizations.htm Also, to say that speed is "really uninteresting" but latency is important seems like a contradiction. Latency is absolutely gated by speed for non-parallelizable problems. And even if the problem can be parallelized, speed is a much easier and simpler way to decrease latency than parallelizing. I guess this reads to me more like a list of theoretically interesting ideas than a set of features that will actually help anyone in the real world.

ianb 16y ago
RAM seems relatively constrained to me -- at least, running a multiprocess server without a few GB or RAM is hard to do (and memory seems to be the most expensive part of servers). And when it goes wrong (you reach whatever your limit is) then things tend to fail in less than ideal ways. Tools around memory usage are also quite poor, so while performance gets optimized memory seldom does. And a culture of cavalier memory usage doesn't help either -- too many people are borrowing memory to get performance, compounded in the case of an application that is typically an aggregation of many people's work. The amount of code involved in systems is also continuing to go up, so that the amount of memory you use before you've done anything (but when you've loaded up all the code) is getting constantly higher. In the world of static/compiled languages this might not be as notable as there's a clear sense of "code" and "data" -- Python has no such distinction, so if you don't share anything then you don't share anything. WRT speed-vs-latency -- definitely related, but most benchmarking seems to specifically remove latency from the benchmark and instead test throughput. E.g., tests frequently throw away the slowest run, even though abnormally slow runs exist in real programs and can have an effect. (Of course they aren't predictable and might be affected by other things on the system -- which is a kind of bias to not optimize things that are hard to measure). But latency is mostly just simplicity of implementation, so no, I wouldn't expect parallelizing to help.

baltcode 16y ago

In addition to speed and the points raised by the OP, I am interested in static/duck typing as it relates to both speed and static error-checking and documentation. i.e., making debugging easier and faster. I somehow think this can happen while keeping almost all of Python's strengths and ease of development.

ianb 16y ago
Python 3 added type annotations, but I've yet to see much interest in actually using it. Traits (http://code.enthought.com/projects/traits/ http://code.enthought.com/projects/traits/) pursues the error-checking side of type declarations, while Cython pursues the speed side. If you want both then too bad ;)

mrshoe 16y ago

It seems like he could get most of what he wants without a new runtime. Diesel (http://dieselweb.org http://dieselweb.org) provides async without callbacks or Deferreds. Diesel1 used Python 2.5's yield expression, and Diesel2 switched to using greenlets. As long as your modules and your applications don't rely on global state (which they shouldn't anyway), you essentially get everything on this list. For web programming, cooperative multitasking usually works perfectly well and no state is generally shared between any 2 http requests, so I'm not sure how much a web programmer stands to gain from preemptive multitasking and completely separate address spaces with CoW. The one thing Diesel is missing is good runtime introspection and more primitives to encourage reliability (a la Erlang), but if you hang out in #diesel you'll see that jamwt has already implemented a lot of that at Bump and he plans on pushing it down into the framework soon (probably for Diesel3).

ianb 16y ago
All these systems are share-everything, which I'm not excited about -- if I want share-everything I can just use threads, and they work far better than most people seem willing to admit.
- mrshoe 16y ago
  Well threads certainly don't work well for high concurrency, which I read as your biggest impetus for a new runtime. I suppose if you mean true (multi-core) concurrency then you're right that Python has some big limitations, but that pretty much boils down to the age old "remove the GIL" proposal.
  - ianb 16y ago
    Threads work pretty darn well, actually. Depending on the libraries you use the GIL may not get in the way (e.g., database libraries, stuff like lxml that releases the GIL). If you have blocking stuff like I/O (socket or file) then threads help. The only time when the GIL really gets in the way is with multiple processors and CPU-bound routines. Even with CPU-bound routines you are often better off with threads as the machinery does share the CPU transparently and relatively fairly. Threads are a little half-assed, sure, but so are most of the currently offered "replacements" for threads, including cooperative multitasking (which also doesn't do anything with multiple processors!)

n_are_q 16y ago

For what it's worth, another 2c from another web developer - for me memory isolation and COW for module dependencies would be less useful than a single process and a lack of GIL (hold the flamethrowers for one second please) - allowing me to share on a multicore server resources like those same modules, as well as database connections, as well as any significant static data i may read in at start up, decreasing my operational complexity, etc. In a web server environment shared memory isn't particularly scary, to me anyway. Am I misunderstanding the reason for COW? Also, the most important characteristic to me is latency. I don't care about throughput that much as long as it's "good enough", hardware is getting really cheap really quickly anyway (in terms of cores and RAM/$). Latency on the other hand is tough. While IO is normally going to be the biggest factor here, if the runtime allows me to consistently shave 5ms off of every request - that's still a great improvement. In this sense i would say that raw speed IS important to me. Perhaps I haven't encountered the use cases Ian is talking about, it was difficult to figure them out from the blog post (i'm a relative novice in Python). But I am hoping PyPy can at some point help with the concerns I have above, even lower latency by itself may be compelling enough to switch.

ianb 16y ago
Copy-on-write lets you efficiently share objects across boundaries -- not copying everything right away (and everything in Python means everything), while also safely avoiding the sharing of state. If you are cool with threads then it's not an issue. If you can spare the overhead of a few processes then the GIL isn't an issue. I'd rather it not be a matter of tradeoffs, and I don't think either OS processes or threads are it.