7 ms·

Design of CPython’s Compiler

52 points by sharmajai 15y ago

sharmajai 15y ago
This is best read with CPython's source code available on the side for reference IMHO.
eliben 15y ago
A shameless plug: This is a good companion article to that design doc - http://eli.thegreenplace.net/2010/06/30/python-internals-adding-a-new-statement-to-python/ http://eli.thegreenplace.net/2010/06/30/python-internals-add...
oconnore 15y ago
Seeing how the internals of Python are designed, I can't help but wonder how programmers don't flock to lisp compilers. The dynamic language without a GIL problem has already been solved, as has the optimization problem. The amount of work needed to build libraries like Django or Rails on common lisp or scheme is a fraction of the work needed to improve cpython/pypy/ruby to the level of sbcl, ccl, or one of the many high quality scheme's. Do people just not care about performance? Is GIL locked performance just good enough? Do developers hate parens enough to justify the huge sacrifice in everything else?
- slurgfest 15y ago
  Python's pseudocode-like readability is a big asset which you are selling short when you pretend like the appeal's just some kind of irrational paren hate. Have you found some devastating performance problem with Python that you want to share? Because I'm not aware of one, as long as you don't try to write Java code in Python. If you want lots of concurrency for some special purpose, I can recommend gevent as a very nice option. If I have a performance problem it's usually due to a tight inner loop and in the rare event that it can't be fixed based on judicious profiling, it is not really a huge problem to write a little code in C. I don't understand why I would have to switch to LISP to deal with a rare performance problem.
  - oconnore 15y ago
    > Because I'm not aware of one, as long as you don't try to write Java code in Python. That means you are aware of one. Anyways, I guess I have my answer, performance problems just aren't very common for many programmers. Keep in mind that profiling is only a solution to certain problems, due to Amdahl's law. To say that all interesting problems fit in that mould is disingenuous. > it is not really a huge problem to write a little code in C. Sure, but any language can do that. GCC is fast, but I thought we wanted to write code in something higher level? Or did we just give up?
    - slurgfest 15y ago
      Whatever language you use, there are going to be many ways of writing slow code. The reason people use threads very heavily in (say) Java is that it is the way things are done effectively on that platform, because there isn't really anything better. It's not because it is globally the best way to do things in every language and condition. So it doesn't make sense to insist that exactly the same pattern be the 'best' solution in every other language. If I naively apply a Java "best practice" in LISP and get bad performance, does that mean LISP is suddenly unsuitable and fraught with horrible performance problems? Probably not: rather, it means that I'm newer to LISP than I am to Java, and that is why my program is not performing as well. Again - I am not aware of any devastating performance problem with Python - and in general I don't hear anything about the GIL except when someone is trashing Python from a list of talking points. You suggest I'm being disingenuous by attributing to me the statement that "all interesting problems" are solved by profiling. But this is really dishonest, as I didn't say anything like that to begin with. Nobody was even discussing "all interesting problems". The right way to deal with Python programs that perform poorly, in PRACTICE, is to start by measuring - and then focus your optimization effort on the parts of the code which are actually creating the problem. The vast majority of the time, there would be no need to switch off of Python, but even in those incredibly rare cases it is certainly possible to write a little function in C and call it a day. Not to start by switching to LISP. LISP is not a miracle panacea. Ruby is not. Python is not. For the purposes of optimizing highly performance-sensitive parts of a program, I prefer writing C to writing LISP. For the purpose of writing new software, I would in many cases choose Python and leave performance concerns to when they are actually demonstrably important, because I tend to get more-than-acceptable performance from Python in most cases. I like C just fine, I have written C for many years - in the event I am ever facing a problem which is magically impossible to solve in Python, I will not hesitate to use C. That is not "giving up". That is simply saying that there really is NOT any devastating reason why someone should avoid Python out of some evidence-free anxiety about performance. In the worst case, you can still use C to handle the nasty bits. So I have NO reason to switch to LISP, least of all some perceived performance advantage due to a lot of hand-waving about the GIL. But anyone who wants to use it should feel free, I am not about to mount some kind of advocacy campaign against tools I don't happen to use.
- killedbydeath 15y ago
  There is a very large number of problem where it is very easy to just run a few independent instances of Python. Web servers definitely fall into this category, splitting requests between processes is trivial. GIL would be an issue for a task which cannot be parallelized and assuming the problem is performance-critical, I would go with C anyway.
- machrider 15y ago
  > The amount of work needed to build libraries like Django or Rails on common lisp or scheme is a fraction of the work needed to improve cpython/pypy/ruby to the level of sbcl, ccl, or one of the many high quality scheme's. So where are they?
  - oconnore 15y ago
    The premise of that quote is that something is missing in both languages. In that context, your question is sort of absurd.
- Nate75Sanders 15y ago
  It's true that a ton of people do not like all the parentheses. This cannot be understated and it's particularly important to get this point across to lispers. It's tough to accept that if you, yourself, can't see any problem at all with them and in fact may well consider them a blessing for delineating pieces of syntax. But most people don't like them. This means you don't have as large of a following. Which means you don't have as many libraries. Which makes new programmers even more likely to choose python over lisp. Additionally python looks more like whatever languages people learn in intro CS classes. This further increases python's following....and its libraries...and it keeps snowballing. And to answer your other questions, yeah, most people don't care about performance and GIL-locked performance is just good enough for what they're doing. Most people these days are gluing things together and essentially doing plumbing (all of it single-threaded), and python is great for that. Machines are fast enough these days that most tasks don't call for multiple cores/threads or even really efficient single-thread performance. It's just the way things are.
- nyellin 15y ago
  Before I was familiar with lisp, I remember being shocked to see a suggestion in pypy's documentation that everything could just be compiled to lisp. I couldn't understand why anyone would do that. I don't know why lisp failed to become mainstream, but I wish the GNU vision of using guile as a universal extension language had succeeded. How different using computers would be today! Instead of the superficial changes in GNOME 3 and KDE 4, GNU would have a unified desktop vision, where all programs are as extensible as Emacs. We would have personal computing environments that evolve to handle our workflow. Instead, we keep confusing simplicity with lack of extensibility for regular users. We keep creating incomplete scripting environments from scratch. No one writes one off Firefox extensions or OpenOffice plugins the way they do Emacs macros... But it didn't have to be this way. It's 2011 and we should all be scripting our computers, but instead we are slaves to software and its needs. We are forced to repeat more and more repetitive tasks as our programs improve. This comment was automatically written by M-x commentator
  - fijal 15y ago
    I'm happy to see the world where not all programs are like emacs. Don't get me wrong - I like and use emacs, but I would not like to do some M-x magic in a text buffer to get some web experience.
    - nyellin 15y ago
      I don't think Emacs has the perfect approach either. But I would like the option to get under the hood and script any app with guile.
- fijal 15y ago
  Can you provide a citation for a reasonable benchmark where lisp outperforms pypy by a fair margin? Or LuaJIT? Or did you just pull that out of thin air and gossip?

52 points by sharmajai 15y ago

This is best read with CPython's source code available on the side for reference IMHO.

A shameless plug: This is a good companion article to that design doc - http://eli.thegreenplace.net/2010/06/30/python-internals-adding-a-new-statement-to-python/ http://eli.thegreenplace.net/2010/06/30/python-internals-add...

oconnore 15y ago

Seeing how the internals of Python are designed, I can't help but wonder how programmers don't flock to lisp compilers. The dynamic language without a GIL problem has already been solved, as has the optimization problem. The amount of work needed to build libraries like Django or Rails on common lisp or scheme is a fraction of the work needed to improve cpython/pypy/ruby to the level of sbcl, ccl, or one of the many high quality scheme's. Do people just not care about performance? Is GIL locked performance just good enough? Do developers hate parens enough to justify the huge sacrifice in everything else?

slurgfest 15y ago
Python's pseudocode-like readability is a big asset which you are selling short when you pretend like the appeal's just some kind of irrational paren hate. Have you found some devastating performance problem with Python that you want to share? Because I'm not aware of one, as long as you don't try to write Java code in Python. If you want lots of concurrency for some special purpose, I can recommend gevent as a very nice option. If I have a performance problem it's usually due to a tight inner loop and in the rare event that it can't be fixed based on judicious profiling, it is not really a huge problem to write a little code in C. I don't understand why I would have to switch to LISP to deal with a rare performance problem.
- oconnore 15y ago
  > Because I'm not aware of one, as long as you don't try to write Java code in Python. That means you are aware of one. Anyways, I guess I have my answer, performance problems just aren't very common for many programmers. Keep in mind that profiling is only a solution to certain problems, due to Amdahl's law. To say that all interesting problems fit in that mould is disingenuous. > it is not really a huge problem to write a little code in C. Sure, but any language can do that. GCC is fast, but I thought we wanted to write code in something higher level? Or did we just give up?
  - slurgfest 15y ago
    Whatever language you use, there are going to be many ways of writing slow code. The reason people use threads very heavily in (say) Java is that it is the way things are done effectively on that platform, because there isn't really anything better. It's not because it is globally the best way to do things in every language and condition. So it doesn't make sense to insist that exactly the same pattern be the 'best' solution in every other language. If I naively apply a Java "best practice" in LISP and get bad performance, does that mean LISP is suddenly unsuitable and fraught with horrible performance problems? Probably not: rather, it means that I'm newer to LISP than I am to Java, and that is why my program is not performing as well. Again - I am not aware of any devastating performance problem with Python - and in general I don't hear anything about the GIL except when someone is trashing Python from a list of talking points. You suggest I'm being disingenuous by attributing to me the statement that "all interesting problems" are solved by profiling. But this is really dishonest, as I didn't say anything like that to begin with. Nobody was even discussing "all interesting problems". The right way to deal with Python programs that perform poorly, in PRACTICE, is to start by measuring - and then focus your optimization effort on the parts of the code which are actually creating the problem. The vast majority of the time, there would be no need to switch off of Python, but even in those incredibly rare cases it is certainly possible to write a little function in C and call it a day. Not to start by switching to LISP. LISP is not a miracle panacea. Ruby is not. Python is not. For the purposes of optimizing highly performance-sensitive parts of a program, I prefer writing C to writing LISP. For the purpose of writing new software, I would in many cases choose Python and leave performance concerns to when they are actually demonstrably important, because I tend to get more-than-acceptable performance from Python in most cases. I like C just fine, I have written C for many years - in the event I am ever facing a problem which is magically impossible to solve in Python, I will not hesitate to use C. That is not "giving up". That is simply saying that there really is NOT any devastating reason why someone should avoid Python out of some evidence-free anxiety about performance. In the worst case, you can still use C to handle the nasty bits. So I have NO reason to switch to LISP, least of all some perceived performance advantage due to a lot of hand-waving about the GIL. But anyone who wants to use it should feel free, I am not about to mount some kind of advocacy campaign against tools I don't happen to use.
killedbydeath 15y ago
There is a very large number of problem where it is very easy to just run a few independent instances of Python. Web servers definitely fall into this category, splitting requests between processes is trivial. GIL would be an issue for a task which cannot be parallelized and assuming the problem is performance-critical, I would go with C anyway.
machrider 15y ago
> The amount of work needed to build libraries like Django or Rails on common lisp or scheme is a fraction of the work needed to improve cpython/pypy/ruby to the level of sbcl, ccl, or one of the many high quality scheme's. So where are they?
- oconnore 15y ago
  The premise of that quote is that something is missing in both languages. In that context, your question is sort of absurd.
Nate75Sanders 15y ago
It's true that a ton of people do not like all the parentheses. This cannot be understated and it's particularly important to get this point across to lispers. It's tough to accept that if you, yourself, can't see any problem at all with them and in fact may well consider them a blessing for delineating pieces of syntax. But most people don't like them. This means you don't have as large of a following. Which means you don't have as many libraries. Which makes new programmers even more likely to choose python over lisp. Additionally python looks more like whatever languages people learn in intro CS classes. This further increases python's following....and its libraries...and it keeps snowballing. And to answer your other questions, yeah, most people don't care about performance and GIL-locked performance is just good enough for what they're doing. Most people these days are gluing things together and essentially doing plumbing (all of it single-threaded), and python is great for that. Machines are fast enough these days that most tasks don't call for multiple cores/threads or even really efficient single-thread performance. It's just the way things are.
nyellin 15y ago
Before I was familiar with lisp, I remember being shocked to see a suggestion in pypy's documentation that everything could just be compiled to lisp. I couldn't understand why anyone would do that. I don't know why lisp failed to become mainstream, but I wish the GNU vision of using guile as a universal extension language had succeeded. How different using computers would be today! Instead of the superficial changes in GNOME 3 and KDE 4, GNU would have a unified desktop vision, where all programs are as extensible as Emacs. We would have personal computing environments that evolve to handle our workflow. Instead, we keep confusing simplicity with lack of extensibility for regular users. We keep creating incomplete scripting environments from scratch. No one writes one off Firefox extensions or OpenOffice plugins the way they do Emacs macros... But it didn't have to be this way. It's 2011 and we should all be scripting our computers, but instead we are slaves to software and its needs. We are forced to repeat more and more repetitive tasks as our programs improve. This comment was automatically written by M-x commentator
- fijal 15y ago
  I'm happy to see the world where not all programs are like emacs. Don't get me wrong - I like and use emacs, but I would not like to do some M-x magic in a text buffer to get some web experience.
  - nyellin 15y ago
    I don't think Emacs has the perfect approach either. But I would like the option to get under the hood and script any app with guile.
fijal 15y ago
Can you provide a citation for a reasonable benchmark where lisp outperforms pypy by a fair margin? Or LuaJIT? Or did you just pull that out of thin air and gossip?