8 ms·

So you want to write a fast Python?

262 points by ericflo 15y ago

wbhart 15y ago
I have some questions: * Is the Jit in PyPy standalone, i.e. can it be used by other projects independently of Python? Is it documented? * How does PyPy compare with highly optimised C? * Assuming that the benchmarks that used to be at the Computer Benchmarks Game for PyPy are somewhat representative of overall performance, why is there still such an enormous difference with optimised C? * There used to be this thing called restricted python. If one limits oneself to using that only, are benchmarks much better? * I read somewhere that some projects have merged or combined forces. There used to be this thing called Unladen Swallow. There was also Psyco. Have either of these merged with/been absorbed by PyPy? Are any other such projects still going? * Cython is another fast project. If it became part of mainline Python, would PyPy become irrelevant?
- llambda 15y ago
  Maybe I'm confused, but isn't Cython just a tool for writing C types and C functions directly in Python? Whereas PyPy, Jython, etc, are full runtime environments.
- kingkilr 15y ago
  I have some answers! * Yes, the JIT is part of the RPython translation toolchain (more on this in a few) and can be used in other interpreters, for example we have Scheme, Javascript, Prolog, and Haskell in varying level of completeness (Prolog being the most complete). * Highly optimized C? Not spectacularly, normal C, not bad, on numerical code we often hover around gcc -O1. * They weren't representative, most of that code was heavily optimized for CPython, which has very different performance characteristics. * RPython is the language the PyPy interpreter is written in, it's not really meant for general purpose code, it's designed mostly for VMs, but it does run at basically C-like speed. * Unladen Swallow was Google's fork of CPython, it's basically dead at this point, retrospective here (http://qinsb.blogspot.com/2011/03/unladen-swallow-retrospective.html http://qinsb.blogspot.com/2011/03/unladen-swallow-retrospect...), Psyco was an extension module for CPython that added a JIT, it's no longer maintained (but still works), it's creator is the lead developer of PyPy. * Cython isn't Python (this could be a meme or something), it does not implement the entire language and thus isn't directly comparable (although we still often compare, because it is a competitor in the space of "making Python-like stuff faster").
  - wbhart 15y ago
    Thanks for the concise and extremely helpful answers. I have some followups: * What is the Jit itself written in? * You mention that for numeric code PyPy hovers around gcc -O1. What are the other classes of code that you see PyPy as having to deal with? And how would it perform compared to normal C on those? * Very interesting comments re the benchmark game. Did anyone ever optimise some of these tasks specifically for PyPy? * Just a clarification re Cython. Yes, I believe it was forked from Pyrex which was mainly for nailing up C code I think. Though since that time my impression is it has become its own language, similar in parts to C/C++ and similar in parts to Python. They appear to have put a fair bit of work into making it fast. It is compiled though (to C), not interpreted. There was some talk about them trying to get it into CPython. But I'm not sure what was meant by that. One final followup: * Do you know if there are any plans to develop a new language built on top of the PyPy VM similar to Python but extended in some way? For example would it be feasible to implement a statically typed language with or without type inference on top of the same technology? By the way, if you google virtual machine you get pages and pages of links to LLVM, JVM, Parrot and Dalvik (go figure) and the occasional reference to a few others. I'm really surprised that the PyPy VM is not better known!
    - kingkilr 15y ago
      * The JIT is written in RPython. * Comparing code with C is pretty difficult for all but the easiest 1-1 ports (i.e. numeric stuff), so I don't have good comparison numbers for it. * Yes, I tried to improve the submissions to the benchmark game, this led down a long path which resulted in the benchmark game only having a single implementation for each language (e.g. tracemonkey is no longer on there, only v8) * We don't plan on writing a new language, we kind of like Python, but someone absolutely could. * PyPy doesn't have a VM, we have a toolkit for writing VMs and adding JITs to them. Perhaps these two blog posts will make it more clear: http://morepypy.blogspot.com/2011/04/tutorial-writing-interpreter-with-pypy.html http://morepypy.blogspot.com/2011/04/tutorial-writing-interp... http://morepypy.blogspot.com/2011/04/tutorial-part-2-adding-jit.html http://morepypy.blogspot.com/2011/04/tutorial-part-2-adding-...
      - 15y ago
- koenigdavidmj 15y ago
  * Yes...I have already seen it for some "build your own language interpreter" examples. Not sure about the documentation quality. * http://geetduggal.wordpress.com/2010/11/25/speed-up-your-python-unladen-vs-shedskin-vs-pypy-vs-c/ http://geetduggal.wordpress.com/2010/11/25/speed-up-your-pyt... lists an example where PyPy is in fact faster than C++ with STL. Not sure how it would compare to well-optimized C, but remember that JIT warmup time will hurt you for short lived jobs. * Can't answer this one, sorry. * RPython is still around---PyPy is written in it! Certainly it would be more friendly toward the JIT, which does best when the types of variables are consistent (which is required by RPython). Plus, RPython can be translated directly to native code and you can get rid of the JIT entirely. * Unladen Swallow tried to do a lot of the same stuff that PyPy is doing, but on top of LLVM. It never did much better than CPython (within an order of magnitude), but if it did they were planning to merge it into CPython (of which it was a branch). PyPy is a followup to Psyco (which only ran on 32 bit x86) by the same people, but Psyco ran under CPython. * Cython does not interpret Python---it compiles from a language that looks a lot like Python but with C-like semantics (for example, optional strong typing, and the ability to directly call C functions), and turns it into C or C++ code that can be compiled into a Python module. As such, it targets a slightly different use case.
  - wbhart 15y ago
    " http://geetduggal.wordpress.com/2010/11/25/speed-up-your-pyt.. http://geetduggal.wordpress.com/2010/11/25/speed-up-your-pyt.... lists an example where PyPy is in fact faster than C++ with STL. Not sure how it would compare to well-optimized C, but remember that JIT warmup time will hurt you for short lived jobs." But note a little further down the page. In the hands of a C++ coder (or indeed Cython coder) things look very different.
- luismgz 15y ago
  1) Is the JIT standalone? No easy answer. Pypy is a complex project, not just a python implementation. Pypy is a complex toolchain, a framework, for implementing dynamic languages. With pypy, you could write your own implementation of ruby, scheme, php, whatever... And once your interpreter is complete, pypy will generate the JIT automatically for you (you won't need to implement it yourself, you just have to write the interpreter, and pypy will do the rest). Pypy is writen in Rpython (restricted python) but you can implement any vm with it, for any language you may want. 2) How does it compare to c? It's pretty competitive in highly algorithmic code, cpu intensive and numerical tasks. For anything else, it depends. Overall, it on average up to 4 times faster than regular python. 3) Why there's still a difference? Pypy is a work in progress. You should compare it to other projects such as v8 or tracemonkey. 4) Restricted python: It's the static subset of python used to implement pypy. It takes the place of c in cpython. Yes, it's much faster, but also more limited and less flexible. It's nicer than c, but not as cool as full python. You can also find Shedskin and Cython. Shedskin is a true static subset of python that compiles to c++. Cython is a python-like language that adds type declarations to the language to make it closer to c. Unladen Swallow was a separate project and it's dead now. Psyco is the predecessor of pypy. It is no longer maintained.
DrJ 15y ago
If anything, PyPy's blogs and updates are really interesting to read. (not to mention the awesome work those guys are doing!)
callahad 15y ago
Yeah, that's it. Time to start using PyPy as the primary implementation for some toy projects and see how things go. For folks that want to test their projects against various version of CPython, PyPy, Jython, etc., check out tox: http://tox.testrun.org/ http://tox.testrun.org/
- timtadh 15y ago
  I tried using it a month or so back and had a great time. Caveats: * You have to use a special patched version of virtualenv. * Numpy is unavailable for Pypy. For various reasons the pypy folks are just going to have to re-implement it. The lack of numpy turned out to be a deal breaker for me in the end. I look forward to using it again once there version of numpy is ready for consumption.
  - kingkilr 15y ago
    no longer true, virtualenv 1.6.1 supports pypy nicely!
schiptsov 15y ago
Perfect is the enemy of good enough. The huge goal of CPython (especially 3.x - those who still stuck on 2.x are, well, just stuck on 2.x) is that it is good enough, was designed to be good enough and simple. Need speed? Write extension is C. It is simple and it was designed to be so. Most of people still didn't get it. CPython 3.x is good enough for its purpose and its goals. It's evolving according to its philosophy (http://www.python.org/dev/peps/pep-0020/ http://www.python.org/dev/peps/pep-0020/) and it is really really good (If you understand some general principles like The Middle Way, Being Good Enough, Divide Et Impera and so on). Making things too complicated is as bad as making them naively oversimplified. ^_^ And porting everything to JVM is just some kind of sport. Being simple extendable and at the same time close enough to a hardware and using optimizations provided by an OS is much better.
- masklinn 15y ago
  What are you talking about? And why are you even mentioning the JVM?
  - schiptsov 15y ago
    Yeah, I was influenced by my previous comment. Original article didn't mention JVM. But there is a bigger view - if most of the modules are mere wrappers around plain C libraries it is very ineffective approach to try to use some complicated VM while you must do zillions of FFI calls. That is of no use. So, in my opinion, for a scripting language fast and efficient module calls are more important that any modern JIT stuff, while your modules mostly are mere plain .so btw, this is yet another point where Java sucks. If you are re-implementing everything in Java, that is probably OK (if you don't care about performance. NIO2 is still just a spec), but if you wish to call any code outside JVM - it sucks. The approach itself is deeply flawed. Look what mess JDBC is. It is so obvious, that I really disappointed by the level of discussions on HN.
    - scott_s 15y ago
      If the scripting language itself is fast, then the game changes completely. That is obvious to the rest of us, and motivates this work.
thristian 15y ago
Dear PyPy developers: while I'm happy to download binaries and run them out of my home directory for something I use all the time (like Firefox), for less-frequently used tools I'd much rather set up an Ubuntu PPA to install things system-wide and have them kept up-to-date without my having to think too hard about it. Unfortunately, the only PyPy PPA I can find on launchpad.net[1] hasn't been updated for a year. Please set up a new PPA, or update the one you have - a lot of curious Python tinkerers would love to try out PyPy on their pet projects! [1] https://launchpad.net/~pypy/+archive/ppa https://launchpad.net/~pypy/+archive/ppa
- rat 15y ago
  or get into the debian repos.
  - thristian 15y ago
    I discovered that PyPy used to be in Debian, back in the days of PyPy 1.0, but it was removed because it took so long to build and because it wasn't yet generally useful. The latest releases of PyPy seem to be a lot more useful, and hopefully the build-time is being addressed...
    - jcoby 15y ago
      I tried building pypy on my macbook pro w/4 4gb of ram and 2.4 GHz i5. After over an hour I had to give up and cancel it. It took another fifteen minutes for the system to recover enough to where I could start using it again. I don't know what it was doing but there was a python process that was using every bit of CPU available and a ton of swap. This same machine can build ruby 1.9.2 in under 10 minutes. It took less time to download and refresh all of my installed macports (including postgresql 9.0) than it took to attempt to build pypy.
      - sirn 15y ago
        Even though the documentation[1] says you need at least 4GB of RAM to build 64-bit PyPy, I've never successfully building PyPy on a 4GB machine. It took about 20 minutes to successfully build on a machine with 8GB RAM, though. [1]: http://pypy.org/download.html http://pypy.org/download.html
ohyes 15y ago
You can not write a static compiler for python, because python does not have static typing, and it is a dynamic language. So that part is a tautology. I suspect the author is trying to say that you can not write a compiler to machine code for python. This is wrong. Compiling dynamic languages to machine code has been done dozens of times in languages with equivalent or greater dynamic properties (Common Lisp, Scheme, and Smalltalk, for example). I guess an example of proof of concept here is that Python was implemented in Common Lisp as a DSL (macros), and it works on the machine code lisp implementations. (http://common-lisp.net/project/clpython/manual.html#compiling-before-running http://common-lisp.net/project/clpython/manual.html#compilin...) The truth is that no one can be bothered to do it, because there is little to be gained from a faster python implementation. All of the slow code is in little parts that can be rewritten in C (or whatever faster language... so almost any language). The mentioned Python compiler projects are all 'research,' as far as I can tell. Doing something that is actually known to work and is difficult would be of no use to someone who is interested in tenure.
- rabidsnail 15y ago
  I mostly agree with you, but there are cases where the problem isn't that "thing X is too slow", but "after a long period of running the garbage collector gets bogged down with long-lived cycles and dies". So I think garbage collector improvements would be really helpful. Core interpreter improvements, not so much.
  - ohyes 15y ago
    Absolutely, improving garbage collection is definitely more interesting/important than code that runs 'faster' for arbitrary benchmarks.
- stiff 15y ago
  A static compiler has nothing to do with static typing, it is just a traditional compiler that passes through the source code and generates the final result of the compilation in one shot, as opposed to a byte-code JIT compiler. I guess the point is that you can not write an _efficient_ static compiler, because there is too little information available at compile-time. I don't think your examples give a counter-proof here, most Smalltalk compilers are JIT compilers, so it's really an argument in favour of the authors orginal thesis, and Common Lisp introduces a lot of extra type annotations to achieve good compilation results. Racket (PLT Scheme), the most popular Scheme implementation, uses a JIT compiler as well.
sqrt17 15y ago
This reminds me that most of what PyPy delivers today - 10x speedup, full range of the Python language, was available within CPython in a module called Psyco that you could easily install. Like PyPy for a long time, Psyco was only available on x86. Psyco's author, Armin Rigo, then got disgusted with Psyco and went on to work on PyPy (possibly for more sanity and better funding). So, yes, I'd be quite happy to use PyPy, even by default, if it was as easy to install as CPython (or ghc, for that matter, which is a bit larger than CPython but also quite easy to install) and worked out of the box. I definitely was when it came to Psyco.
lysol 15y ago
I started messing around with PyPy and love the idea of it. Is there a good Postgres lib for PyPy yet? I've been hooked on Psycopg2 since I got into Python and would love to take advantage of the speed boost from PyPy for my surrounding code.
- fijal 15y ago
  https://bitbucket.org/alex_gaynor/pypy-postgresql/overview https://bitbucket.org/alex_gaynor/pypy-postgresql/overview unfortunately a fork (a well maintained one at least)
  - StavrosK 15y ago
    I was going to ask "why 'unfortunately'?" and then I realized that you mean a fork of pypy, not of psycopg2. Unfortunately indeed...
scorpion032 15y ago
What is the Pypy's plan of action for Py3k?
glenjamin 15y ago
Does anyone know what the state of PyPy with regards to Python 3 is? I had a quick google around but could only find an april fool's joke from 2008!
mattlong 15y ago
So I really like python and would love to contribute to a project like PyPy, but I don't have much (aka any) experience writing interpreters, compilers, etc. What's the best way to start learning about these things?

262 points by ericflo 15y ago

I have some questions: * Is the Jit in PyPy standalone, i.e. can it be used by other projects independently of Python? Is it documented? * How does PyPy compare with highly optimised C? * Assuming that the benchmarks that used to be at the Computer Benchmarks Game for PyPy are somewhat representative of overall performance, why is there still such an enormous difference with optimised C? * There used to be this thing called restricted python. If one limits oneself to using that only, are benchmarks much better? * I read somewhere that some projects have merged or combined forces. There used to be this thing called Unladen Swallow. There was also Psyco. Have either of these merged with/been absorbed by PyPy? Are any other such projects still going? * Cython is another fast project. If it became part of mainline Python, would PyPy become irrelevant?

llambda 15y ago
Maybe I'm confused, but isn't Cython just a tool for writing C types and C functions directly in Python? Whereas PyPy, Jython, etc, are full runtime environments.
kingkilr 15y ago
I have some answers! * Yes, the JIT is part of the RPython translation toolchain (more on this in a few) and can be used in other interpreters, for example we have Scheme, Javascript, Prolog, and Haskell in varying level of completeness (Prolog being the most complete). * Highly optimized C? Not spectacularly, normal C, not bad, on numerical code we often hover around gcc -O1. * They weren't representative, most of that code was heavily optimized for CPython, which has very different performance characteristics. * RPython is the language the PyPy interpreter is written in, it's not really meant for general purpose code, it's designed mostly for VMs, but it does run at basically C-like speed. * Unladen Swallow was Google's fork of CPython, it's basically dead at this point, retrospective here (http://qinsb.blogspot.com/2011/03/unladen-swallow-retrospective.html http://qinsb.blogspot.com/2011/03/unladen-swallow-retrospect...), Psyco was an extension module for CPython that added a JIT, it's no longer maintained (but still works), it's creator is the lead developer of PyPy. * Cython isn't Python (this could be a meme or something), it does not implement the entire language and thus isn't directly comparable (although we still often compare, because it is a competitor in the space of "making Python-like stuff faster").
- wbhart 15y ago
  Thanks for the concise and extremely helpful answers. I have some followups: * What is the Jit itself written in? * You mention that for numeric code PyPy hovers around gcc -O1. What are the other classes of code that you see PyPy as having to deal with? And how would it perform compared to normal C on those? * Very interesting comments re the benchmark game. Did anyone ever optimise some of these tasks specifically for PyPy? * Just a clarification re Cython. Yes, I believe it was forked from Pyrex which was mainly for nailing up C code I think. Though since that time my impression is it has become its own language, similar in parts to C/C++ and similar in parts to Python. They appear to have put a fair bit of work into making it fast. It is compiled though (to C), not interpreted. There was some talk about them trying to get it into CPython. But I'm not sure what was meant by that. One final followup: * Do you know if there are any plans to develop a new language built on top of the PyPy VM similar to Python but extended in some way? For example would it be feasible to implement a statically typed language with or without type inference on top of the same technology? By the way, if you google virtual machine you get pages and pages of links to LLVM, JVM, Parrot and Dalvik (go figure) and the occasional reference to a few others. I'm really surprised that the PyPy VM is not better known!
  - kingkilr 15y ago
    * The JIT is written in RPython. * Comparing code with C is pretty difficult for all but the easiest 1-1 ports (i.e. numeric stuff), so I don't have good comparison numbers for it. * Yes, I tried to improve the submissions to the benchmark game, this led down a long path which resulted in the benchmark game only having a single implementation for each language (e.g. tracemonkey is no longer on there, only v8) * We don't plan on writing a new language, we kind of like Python, but someone absolutely could. * PyPy doesn't have a VM, we have a toolkit for writing VMs and adding JITs to them. Perhaps these two blog posts will make it more clear: http://morepypy.blogspot.com/2011/04/tutorial-writing-interpreter-with-pypy.html http://morepypy.blogspot.com/2011/04/tutorial-writing-interp... http://morepypy.blogspot.com/2011/04/tutorial-part-2-adding-jit.html http://morepypy.blogspot.com/2011/04/tutorial-part-2-adding-...
    - 15y ago
koenigdavidmj 15y ago
* Yes...I have already seen it for some "build your own language interpreter" examples. Not sure about the documentation quality. * http://geetduggal.wordpress.com/2010/11/25/speed-up-your-python-unladen-vs-shedskin-vs-pypy-vs-c/ http://geetduggal.wordpress.com/2010/11/25/speed-up-your-pyt... lists an example where PyPy is in fact faster than C++ with STL. Not sure how it would compare to well-optimized C, but remember that JIT warmup time will hurt you for short lived jobs. * Can't answer this one, sorry. * RPython is still around---PyPy is written in it! Certainly it would be more friendly toward the JIT, which does best when the types of variables are consistent (which is required by RPython). Plus, RPython can be translated directly to native code and you can get rid of the JIT entirely. * Unladen Swallow tried to do a lot of the same stuff that PyPy is doing, but on top of LLVM. It never did much better than CPython (within an order of magnitude), but if it did they were planning to merge it into CPython (of which it was a branch). PyPy is a followup to Psyco (which only ran on 32 bit x86) by the same people, but Psyco ran under CPython. * Cython does not interpret Python---it compiles from a language that looks a lot like Python but with C-like semantics (for example, optional strong typing, and the ability to directly call C functions), and turns it into C or C++ code that can be compiled into a Python module. As such, it targets a slightly different use case.
- wbhart 15y ago
  " http://geetduggal.wordpress.com/2010/11/25/speed-up-your-pyt.. http://geetduggal.wordpress.com/2010/11/25/speed-up-your-pyt.... lists an example where PyPy is in fact faster than C++ with STL. Not sure how it would compare to well-optimized C, but remember that JIT warmup time will hurt you for short lived jobs." But note a little further down the page. In the hands of a C++ coder (or indeed Cython coder) things look very different.
luismgz 15y ago
1) Is the JIT standalone? No easy answer. Pypy is a complex project, not just a python implementation. Pypy is a complex toolchain, a framework, for implementing dynamic languages. With pypy, you could write your own implementation of ruby, scheme, php, whatever... And once your interpreter is complete, pypy will generate the JIT automatically for you (you won't need to implement it yourself, you just have to write the interpreter, and pypy will do the rest). Pypy is writen in Rpython (restricted python) but you can implement any vm with it, for any language you may want. 2) How does it compare to c? It's pretty competitive in highly algorithmic code, cpu intensive and numerical tasks. For anything else, it depends. Overall, it on average up to 4 times faster than regular python. 3) Why there's still a difference? Pypy is a work in progress. You should compare it to other projects such as v8 or tracemonkey. 4) Restricted python: It's the static subset of python used to implement pypy. It takes the place of c in cpython. Yes, it's much faster, but also more limited and less flexible. It's nicer than c, but not as cool as full python. You can also find Shedskin and Cython. Shedskin is a true static subset of python that compiles to c++. Cython is a python-like language that adds type declarations to the language to make it closer to c. Unladen Swallow was a separate project and it's dead now. Psyco is the predecessor of pypy. It is no longer maintained.

DrJ 15y ago

If anything, PyPy's blogs and updates are really interesting to read. (not to mention the awesome work those guys are doing!)

callahad 15y ago

Yeah, that's it. Time to start using PyPy as the primary implementation for some toy projects and see how things go. For folks that want to test their projects against various version of CPython, PyPy, Jython, etc., check out tox: http://tox.testrun.org/ http://tox.testrun.org/

timtadh 15y ago
I tried using it a month or so back and had a great time. Caveats: * You have to use a special patched version of virtualenv. * Numpy is unavailable for Pypy. For various reasons the pypy folks are just going to have to re-implement it. The lack of numpy turned out to be a deal breaker for me in the end. I look forward to using it again once there version of numpy is ready for consumption.
- kingkilr 15y ago
  no longer true, virtualenv 1.6.1 supports pypy nicely!

schiptsov 15y ago

Perfect is the enemy of good enough. The huge goal of CPython (especially 3.x - those who still stuck on 2.x are, well, just stuck on 2.x) is that it is good enough, was designed to be good enough and simple. Need speed? Write extension is C. It is simple and it was designed to be so. Most of people still didn't get it. CPython 3.x is good enough for its purpose and its goals. It's evolving according to its philosophy (http://www.python.org/dev/peps/pep-0020/ http://www.python.org/dev/peps/pep-0020/) and it is really really good (If you understand some general principles like The Middle Way, Being Good Enough, Divide Et Impera and so on). Making things too complicated is as bad as making them naively oversimplified. ^_^ And porting everything to JVM is just some kind of sport. Being simple extendable and at the same time close enough to a hardware and using optimizations provided by an OS is much better.

masklinn 15y ago
What are you talking about? And why are you even mentioning the JVM?
- schiptsov 15y ago
  Yeah, I was influenced by my previous comment. Original article didn't mention JVM. But there is a bigger view - if most of the modules are mere wrappers around plain C libraries it is very ineffective approach to try to use some complicated VM while you must do zillions of FFI calls. That is of no use. So, in my opinion, for a scripting language fast and efficient module calls are more important that any modern JIT stuff, while your modules mostly are mere plain .so btw, this is yet another point where Java sucks. If you are re-implementing everything in Java, that is probably OK (if you don't care about performance. NIO2 is still just a spec), but if you wish to call any code outside JVM - it sucks. The approach itself is deeply flawed. Look what mess JDBC is. It is so obvious, that I really disappointed by the level of discussions on HN.
  - scott_s 15y ago
    If the scripting language itself is fast, then the game changes completely. That is obvious to the rest of us, and motivates this work.

thristian 15y ago

Dear PyPy developers: while I'm happy to download binaries and run them out of my home directory for something I use all the time (like Firefox), for less-frequently used tools I'd much rather set up an Ubuntu PPA to install things system-wide and have them kept up-to-date without my having to think too hard about it. Unfortunately, the only PyPy PPA I can find on launchpad.net[1] hasn't been updated for a year. Please set up a new PPA, or update the one you have - a lot of curious Python tinkerers would love to try out PyPy on their pet projects! [1] https://launchpad.net/~pypy/+archive/ppa https://launchpad.net/~pypy/+archive/ppa

rat 15y ago
or get into the debian repos.
- thristian 15y ago
  I discovered that PyPy used to be in Debian, back in the days of PyPy 1.0, but it was removed because it took so long to build and because it wasn't yet generally useful. The latest releases of PyPy seem to be a lot more useful, and hopefully the build-time is being addressed...
  - jcoby 15y ago
    I tried building pypy on my macbook pro w/4 4gb of ram and 2.4 GHz i5. After over an hour I had to give up and cancel it. It took another fifteen minutes for the system to recover enough to where I could start using it again. I don't know what it was doing but there was a python process that was using every bit of CPU available and a ton of swap. This same machine can build ruby 1.9.2 in under 10 minutes. It took less time to download and refresh all of my installed macports (including postgresql 9.0) than it took to attempt to build pypy.
    - sirn 15y ago
      Even though the documentation[1] says you need at least 4GB of RAM to build 64-bit PyPy, I've never successfully building PyPy on a 4GB machine. It took about 20 minutes to successfully build on a machine with 8GB RAM, though. [1]: http://pypy.org/download.html http://pypy.org/download.html

ohyes 15y ago

You can not write a static compiler for python, because python does not have static typing, and it is a dynamic language. So that part is a tautology. I suspect the author is trying to say that you can not write a compiler to machine code for python. This is wrong. Compiling dynamic languages to machine code has been done dozens of times in languages with equivalent or greater dynamic properties (Common Lisp, Scheme, and Smalltalk, for example). I guess an example of proof of concept here is that Python was implemented in Common Lisp as a DSL (macros), and it works on the machine code lisp implementations. (http://common-lisp.net/project/clpython/manual.html#compiling-before-running http://common-lisp.net/project/clpython/manual.html#compilin...) The truth is that no one can be bothered to do it, because there is little to be gained from a faster python implementation. All of the slow code is in little parts that can be rewritten in C (or whatever faster language... so almost any language). The mentioned Python compiler projects are all 'research,' as far as I can tell. Doing something that is actually known to work and is difficult would be of no use to someone who is interested in tenure.

rabidsnail 15y ago
I mostly agree with you, but there are cases where the problem isn't that "thing X is too slow", but "after a long period of running the garbage collector gets bogged down with long-lived cycles and dies". So I think garbage collector improvements would be really helpful. Core interpreter improvements, not so much.
- ohyes 15y ago
  Absolutely, improving garbage collection is definitely more interesting/important than code that runs 'faster' for arbitrary benchmarks.
stiff 15y ago
A static compiler has nothing to do with static typing, it is just a traditional compiler that passes through the source code and generates the final result of the compilation in one shot, as opposed to a byte-code JIT compiler. I guess the point is that you can not write an _efficient_ static compiler, because there is too little information available at compile-time. I don't think your examples give a counter-proof here, most Smalltalk compilers are JIT compilers, so it's really an argument in favour of the authors orginal thesis, and Common Lisp introduces a lot of extra type annotations to achieve good compilation results. Racket (PLT Scheme), the most popular Scheme implementation, uses a JIT compiler as well.

sqrt17 15y ago

This reminds me that most of what PyPy delivers today - 10x speedup, full range of the Python language, was available within CPython in a module called Psyco that you could easily install. Like PyPy for a long time, Psyco was only available on x86. Psyco's author, Armin Rigo, then got disgusted with Psyco and went on to work on PyPy (possibly for more sanity and better funding). So, yes, I'd be quite happy to use PyPy, even by default, if it was as easy to install as CPython (or ghc, for that matter, which is a bit larger than CPython but also quite easy to install) and worked out of the box. I definitely was when it came to Psyco.

lysol 15y ago

I started messing around with PyPy and love the idea of it. Is there a good Postgres lib for PyPy yet? I've been hooked on Psycopg2 since I got into Python and would love to take advantage of the speed boost from PyPy for my surrounding code.

fijal 15y ago
https://bitbucket.org/alex_gaynor/pypy-postgresql/overview https://bitbucket.org/alex_gaynor/pypy-postgresql/overview unfortunately a fork (a well maintained one at least)
- StavrosK 15y ago
  I was going to ask "why 'unfortunately'?" and then I realized that you mean a fork of pypy, not of psycopg2. Unfortunately indeed...

scorpion032 15y ago

What is the Pypy's plan of action for Py3k?

glenjamin 15y ago

Does anyone know what the state of PyPy with regards to Python 3 is? I had a quick google around but could only find an april fool's joke from 2008!

mattlong 15y ago

So I really like python and would love to contribute to a project like PyPy, but I don't have much (aka any) experience writing interpreters, compilers, etc. What's the best way to start learning about these things?