10 ms·

The Python I Would Like To See

327 points by rachbelaid 12y ago

mpdehaan2 12y ago
I think most of the things I'd want to see first would come in the standard library and in unicode handling. I'd probably argue that most uses of metaclasses are a reason to step back and make the code simpler. There are a few cases where they come in very usefully, but at least in open source code, they create a barrier to understanding and increase complexity. ython starts to get ugly when you start writing code that does "automagical things", and that's somewhat tolerated because it's not a language for building automagical things. Or, to say it another way, if you have to think about how the interpreter works, your python has jumped off the idiomatic wagon a while ago. That being said __slots__ as a performance enhancement is crazy useful - and I'd like to see more things in that area. Though for most people, the thing that would lead to cleaner code would be a more powerful and elegant standard library.
- Dylan16807 12y ago
  >That being said __slots__ as a performance enhancement is crazy useful Can you be more specific? The article claims that it's not a performance boost at all. Where is it useful?
  - ejstronge 12y ago
    The article mentions, a bit unclearly, that there are two types of slots: the slots used to implement method dispatch in the interpreter and __slots__ [1-2]. Armin's article is about method dispatch. __slots__ are useful to reduce memory usage when you're instantiating lots (tens of thousands or millions) of Python objects. Since objects defined in Python code support the addition of arbitrary attributes, their members are internally represented as a dictionary which isn't necessary when your tens of thousands of objects have a limited set of attributes. Defining a sequence __slots__ tells the interpreter that your Python class will only need memory for the members defined in that sequence and restricts your ability to arbitrarily add new members (see [1] for details). [1] https://docs.python.org/2/reference/datamodel.html#slots https://docs.python.org/2/reference/datamodel.html#slots [2] http://code.activestate.com/lists/python-list/531365/ http://code.activestate.com/lists/python-list/531365/
    - TheLoneWolfling 12y ago
      Note that in PyPy __slots__ is entirely unnecessary: http://morepypy.blogspot.ca/2010/11/efficiently-implementing-python-objects.html http://morepypy.blogspot.ca/2010/11/efficiently-implementing...
- DasIch 12y ago
  > That being said __slots__ as a performance enhancement is crazy useful - and I'd like to see more things in that area. Though for most people, the thing that would lead to cleaner code would be a more powerful and elegant standard library. __slots__ as PyPy shows is completely unnecessary.
- masklinn 12y ago
  1. __slots__ and TFA have nothing to do with one another 2. __slots__ are not actually useful, Pypy does what they do by default and deoptimizes as needed, CPython probably could as well 3. the only thing __slots__ do is lower memory usage (by not allocating a dict per instance) and PEP 412 has already improved that __slots__ is not "crazy useful" by any definition of the expression, it is somewhat useful when you have a huge living number of simple objects.
  - PythonicAlpha 12y ago
    Sounds as, that you did not understand, that the author did not mean the __slots__ system, that allows to avoid the __dict__ on objects, but the internal slots-system that is intended to speed up the dispatching of special methods.
    - thu 12y ago
      You missed his point 1.
      - PythonicAlpha 12y ago
        Ok. Then my answer belongs to the first in the sub-tree. What I actually did not understand: What means TFA?
        deleted 12y ago
        [deleted]
        stewartbutler 12y ago
        The Freaking Article.
        PythonicAlpha 12y ago
        Thanks! That helps.
        12y ago
rattray 12y ago
It seems pretty clear to me that we're ready for Python 4 by now. Experienced Python devs know the problems. We even have some solutions. So I think it's about time that leaders of the community like Ronacher, Rietz, Gaynor, etc come together to talk about the next generator of Python. We have PyPy now; why not use the tremendous advances there in the next Python? We have Go and Javascript to compete with and we understand a lot of why they're winning; it's time Pythonistas fought back instead of just leaving. Python 3 was supposed to be "the next thing" five years ago; it fizzled. Let's make the real "next thing" for 2016. Python 4.
- pixelmonkey 12y ago
  Go and Javascript are "winning"? That's news to me. They might be buzzier right now, but their compiler/interpreters only appeared in 2009 and Python has been around since 1991. Is it really such a surprise that the hype period is over for Python as a language spec? Now people are just getting a lot of stuff done with this productive language and nobody needs to be convinced any longer how good a language it is. Most people already know. Now, we're onto the good work of building a huge ecosystem of open source modules atop it. Call me when Go and JavaScript have the equivalent of the PyData stack and rock-solid drivers for every database technology on the planet. Then maybe I'll look at them for anything other than niche use cases. (Obviously JavaScript is still the only game in town inside the browser and Go seems like a pretty interesting alternative to C for UNIX system utilities.)
  - rattray 12y ago
    Thanks; I should clarify. I agree Python is, in my subjective opinion, the best language spec and community in programming. It's simply the most fun, productive environment. But people are leaving; important people and newcomers and loyal stalwarts. Im on the go right now so can't access links, but there have been a handful of posts here in the past 6mo or so of prominent members leaving, or sharing their disenchantment. Many other "I'm moving to go" posts. And the velocity of npm/go are undeniable (look at the stats). Remember, npm is largely for the server, where js shouldn't have an advantage. I love Python. I want it to win. Maybe "lose" is too strong of a word, but it's certainly slipping.
orf 12y ago
I switched to Python 3 this year and I haven't looked back -there are some niggles, but overall its a great improvement. A great feature that's not really talked about is the __prepare__ function in metaclasses: you can supply a custom type that stores all class members. You could whip up multiple-dispatch using this (it lets you handle classes with duplicate property names) in conjunction with signature annotations, which I think is pretty neat.
- Sami_Lehtinen 12y ago
  This has been one of the reasons why I'm not currently doing anything for Google App Engine, they still lack Python 3 support. It feels awkward to code in Python 2 after Python 3.
pixelmonkey 12y ago
It's an interesting article, as a tour of some Python internals. But it comes off as a rant without a real suggestion. Python 3 was an internal interpreter cleanup. That's actually part of its lack of widespread popularity. The core developers didn't add a ton of unique functionality (and most of the new stuff was backported to 2.7 anyway), but they fixed some annoying problems in the CPython codebase and broke some things in the name of cleanup/unification of concepts. They made Python easier to maintain and easier to build new features atop. They sharpened the axe. Yes, they didn't clean up Armin's pet thing -- slots. And they created new problems for library developers in their bytes vs unicode changes. But they cleaned up a whole lot of other stuff. I personally think with the maturity of pypy and the stability of both the 2.7 and 3.4 lines of development, the Python ecosystem has never been more exciting. The advances in pypy make Python attractive for CPU-bound work and the inclusion of asyncio in the stdlib will make it more and more attractive for I/O-bound work over time. Python has long been a winner for mixed workloads, and the ecosystem around Python, especially pydata utilities like numpy and pandas, keep getting better. Stop complaining -- let's just build an awesome future atop this marvelous language.
- the_mitsuhiko 12y ago
  > But it comes off as a rant without a real suggestion. How exactly is that a rant? It's an exploration about a design decision / mistake that was carried through Python for 25 years and has left an impact. There is no complaint and there is no suggestion for Python. As I said in the early paragraph it's something that's interesting for people that are interested in language design.
  - ZenoArrow 12y ago
    It's a rant because the solution was ill defined. If the title of the article is 'The Python I Would Like To See', you should expect more detail of how to implement the solution.
    - the_mitsuhiko 12y ago
      I hope it is valid to describe a situation and voice a wish for a situation that would look different without having to provide a concrete step towards that. The removal of the slot system would be a backwards incompatible change with very little direct benefit for developers. It could be interesting as a general goal for a hypothetical Python 4 but I do not believe it makes any sense to discuss this as anything more than a hypothetical case for the next few years.
nakovet 12y ago
I follow Ronacher's work, open source and posts and I agree with his arguments. This post remember me of Spolsky's one (http://www.joelonsoftware.com/articles/LeakyAbstractions.html http://www.joelonsoftware.com/articles/LeakyAbstractions.htm...) though. When you get to a point where you get to know what's under the hood and why it's not working the way you are trying to use, however in a majority of cases you are just fine and don't need to know what's happening under the hood and maybe if this majority is big enough that's just fine.
- PhantomGremlin 12y ago
  I got a totally different impression from the post. What I thought of was (http://en.wikipedia.org/wiki/How_many_angels_can_dance_on_the_head_of_a_pin%3F http://en.wikipedia.org/wiki/How_many_angels_can_dance_on_th...). In particular: In modern usage, this question also serves as a metaphor for wasting time debating topics of no practical value, or questions whose answers hold no intellectual consequence. Python will (or won't) be used for a particular application regardless of whether some contrived test takes 0.158 usec or takes 0.256 usec per iteration. This sort of misunderstanding comes up so often that there are a plethora of cliches for it. Here's another one: "Missing the forest for the trees.".
hosay123 12y ago
This post is surprisingly confused, it is phrased as a complaint about the language, then immediately degrades into CPython implementation specifics that have little bearing on the usability of the language itself. Ronacher should also know better than to post microbenchmarks like the one provided here, especially without corresponding (C) profiler output. At the C level, slots allow the implementation constant-time access to the most common code paths for an object, and especially when you have C code calling other C code via the type system (IMHO the primary use for Python, and still its strongest use case), "interpreter overhead" is reduced to a few extra memory indirection operations. In the alternative world, sure, perhaps some microbenchmark may behave faster, but now systemically, and for e.g. "reduce(operator.add, range(1000))" requires more hash table lookups than I can count. Python is all about providing a lightweight way to compose bits of fast code (the kernel, network stack, NumPy, MySQL, whatever). Unfortunately somewhere along the way solutions like Django got popular, which are almost the antithesis to this old viewpoint. Ronacher seems to be advocating that we should punish the CPython implementation's traditional strong areas in favour of.. actually, he didn't even describe any vision that was impacted by his complaints. He just seems to want the CPython implementation to suffer. Perhaps his complaint about new-style __getattribute__ would be better suited as a bug report, it seems the only substantial observation made about the language itself in this post.
- the_mitsuhiko 12y ago
  > This post is surprisingly confused, it is phrased as a complaint about the language, then immediately degrades into CPython implementation-specifics that have little bearing on the actual usability of the language itself. You might think that, but you are very wrong and I should probably make that point in another blog post. These CPython specifics are enshrined in the language. While PyPy does not have the actual structs, it needs to implement the same user exposed API as it slots were used. PyPy cannot just say "a + b" means "a.__add__(b)", it needs to implement the exact same dispatch logic that CPython has because people's code depends on it. //EDIT: > Ronacher seems to be advocating that we should punish the CPython implementation's traditional strong areas in favour of.. actually, he didn't even describe any vision that was impacted by his complaints. Maybe I did not make my point very clear but the whole last paragraph advocates about trying a version of Python that abolishes the internal slot system.
futuredale 12y ago
Ah nice. I was always curious why __getattribute__ didn't fire for specific dunder methods. Made subclassing pandas objects much more complicated.
alberth 12y ago
From the OP: >>"In recent years there is a clear trend of making Python more complex as a language. I would like to see the inverse of that trend. I would like to see an internal interpreter design could be based on interpreters that work independent of each other, with local base types and more, similar to how JavaScript works." Sounds like OP should investigate Lua (and LuaJIT in particular). He might like Lua more than Python. (Given that OP created Flask, I'd love to see a Flask equivalent developed in Lua) Edit: typo
- scrollaway 12y ago
  Good job not reading the article and skipping to the end, I guess? >> This is in fact how many other dynamic languages work. For instance this is how lua implementations operate, how javascript engines work etc. The clear advantage is that you can have two interpreters. What a novel concept.
PythonicAlpha 12y ago
I really would like to see, when the suggested, simpler system would be tried. It is sometimes amazing, how by simplification gains could be reached against common wisdom. And even, when no gains or little losses would be there with the new, simpler system, it could be worthwhile to change, since the current system has his huddles, that I also already stumbled about, and which are not clearly communicated, since it is such a deep implementation detail of CPython. Also it was mentioned, that a simpler system could be even more powerful.
xorcist 12y ago
Armin for BDFL! (Just a joke, people. Just a joke.) Anyway, I wonder why Armin's not more involved with core Python things, his perspective should be valuable. As someone doing some Python stuff on the side his articles are always worth reading carefully. The Python community has always been the best part of the language, but the important subcommunities like NumPy, Twisted and PyPy has always seemed like they are a bit outside looking in. I don't know why. Perhaps the language would have evolved differently if these projects were a bit more involved in the actual language development.
philh 12y ago
>>> original = 42 >>> class FooProxy: ... def __getattr__(self, x): ... return getattr(original, x) ... >>> proxy = FooProxy() >>> proxy 42 For anyone else who found this as confusing as I did ("wtf, how can proxy actually be 42?"), what's going on here is that it's calling `proxy.__repr__()` when it attempts to display `proxy`, which in turn calls `42.__repr__()`. (Similarly, `proxy + 1` calls `42.__add__(1)`.)
fndrplayer13 12y ago
Thanks for the very informative post. I've been a full-time Python developer for only the past 7 or 8 months now. I had not in that time learned about the slot system or the CPython implementation. Also, <3 Flask. So good.
GnarfGnarf 12y ago
I'm learning Python and I must say it is a wonderful language. Being able to concatenate strings by simply saying "a + b" is a great productivity boost (coming from C++). Python libraries are powerful. I can read an Excel spreadsheet with one line of code. I can create plots in PDF format with a half-dozen lines. Amazing! However, I am disappointed with the difficulty of turning a program into a Windows EXE. I wrote a small program (couple of thousand lines), tried Py2exe which failed to handle the Numpy (or Matplotlib, I forget which) imbroglio. PyInstaller works, except that the EXE is 85MB, and takes one minute to start up. Not practical for customer distribution. I can't expect my customers to install the Python runtime. In contrast, my 500KLOC C++ program, with all its third-party libraries, is 19MB. Yes, I know, Python needs everything including the kitchen sink. Still, 85MB is not practical. Too bad. Not all programs run on a server.
- alexchamberlain 12y ago
  I love Python too... but, I love C++ as well... http://en.cppreference.com/w/cpp/string/basic_string/operator%2B http://en.cppreference.com/w/cpp/string/basic_string/operato...
- theseoafs 12y ago
  > Being able to concatenate strings by simply saying "a + b" is a great productivity boost (coming from C++) C++ has had this operator overloaded for strings for decades. With regard to the size of compiled executables, I can't really say much except "that's not what it's made for". If you need to ship compiled executables to people, Python is an extraordinarily inappropriate choice; stick with C++.
  - dkersten 12y ago
    I've often heard it said that if you're not using std::string for strings in C++, you're (arguably) not using C++
    - GnarfGnarf 12y ago
      I just did a test using std:: and old-fashioned char[]. The fancy std:: is fifteen times slower than the strcat(). In a loop with intensive string manipulation, this could cause the program to get back to you in 15 seconds instead of 1 second. You don't mind waiting? Here's the code (Visual Studio 2010): #include <string> #include <time.h> char buf[64], buf2[64]; clock_t start = clock(); for(i = 0; i < 100000; i++) { _snprintf(buf, sizeof(buf), "%d", i); _snprintf(buf2, sizeof(buf2), "%d", i * i); #if defined FAST strcat(buf, buf2); #else std::string s1 = buf; std::string s2 = buf2; std::string s3 = s1 + s2; #endif } _snprintf(buf, sizeof(buf), "%.4f\n", (float)(clock() - start) / (float)CLK_TCK); OutputDebugString(buf);
notastartup 12y ago
I moved from Javascript to Python 2.7 and haven't looked back since. I now strictly use Python in all of my projects, Javascript only because browser doesn't support it. I wish that Python would introduce a few things from the Javascript world. Easy async, and a real-time Meteor like framework.
- lmm 12y ago
  Improved async is the biggest new feature in 3.4
berdario 12y ago
I was a bit puzzled by the "lightweight old style classes" remark, so I tried it out: >>> sys.getsizeof(OldStyleClass), sys.getsizeof(NewStyleClass) (104, 904) >>> sys.getsizeof(OldStyleClass()), sys.getsizeof(NewStyleClass()) (72, 64) I agree that OldStyleClasses might be simpler (while less featureful), but I think I'd care more for the footprint of instances, rather than class objects themselves
scardine 12y ago
tldr; method calling is unnecessarily expensive in current Python due to debatable design decisions.

327 points by rachbelaid 12y ago

I think most of the things I'd want to see first would come in the standard library and in unicode handling. I'd probably argue that most uses of metaclasses are a reason to step back and make the code simpler. There are a few cases where they come in very usefully, but at least in open source code, they create a barrier to understanding and increase complexity. ython starts to get ugly when you start writing code that does "automagical things", and that's somewhat tolerated because it's not a language for building automagical things. Or, to say it another way, if you have to think about how the interpreter works, your python has jumped off the idiomatic wagon a while ago. That being said __slots__ as a performance enhancement is crazy useful - and I'd like to see more things in that area. Though for most people, the thing that would lead to cleaner code would be a more powerful and elegant standard library.

Dylan16807 12y ago
>That being said __slots__ as a performance enhancement is crazy useful Can you be more specific? The article claims that it's not a performance boost at all. Where is it useful?
- ejstronge 12y ago
  The article mentions, a bit unclearly, that there are two types of slots: the slots used to implement method dispatch in the interpreter and __slots__ [1-2]. Armin's article is about method dispatch. __slots__ are useful to reduce memory usage when you're instantiating lots (tens of thousands or millions) of Python objects. Since objects defined in Python code support the addition of arbitrary attributes, their members are internally represented as a dictionary which isn't necessary when your tens of thousands of objects have a limited set of attributes. Defining a sequence __slots__ tells the interpreter that your Python class will only need memory for the members defined in that sequence and restricts your ability to arbitrarily add new members (see [1] for details). [1] https://docs.python.org/2/reference/datamodel.html#slots https://docs.python.org/2/reference/datamodel.html#slots [2] http://code.activestate.com/lists/python-list/531365/ http://code.activestate.com/lists/python-list/531365/
  - TheLoneWolfling 12y ago
    Note that in PyPy __slots__ is entirely unnecessary: http://morepypy.blogspot.ca/2010/11/efficiently-implementing-python-objects.html http://morepypy.blogspot.ca/2010/11/efficiently-implementing...
DasIch 12y ago
> That being said __slots__ as a performance enhancement is crazy useful - and I'd like to see more things in that area. Though for most people, the thing that would lead to cleaner code would be a more powerful and elegant standard library. __slots__ as PyPy shows is completely unnecessary.
masklinn 12y ago
1. __slots__ and TFA have nothing to do with one another 2. __slots__ are not actually useful, Pypy does what they do by default and deoptimizes as needed, CPython probably could as well 3. the only thing __slots__ do is lower memory usage (by not allocating a dict per instance) and PEP 412 has already improved that __slots__ is not "crazy useful" by any definition of the expression, it is somewhat useful when you have a huge living number of simple objects.
- PythonicAlpha 12y ago
  Sounds as, that you did not understand, that the author did not mean the __slots__ system, that allows to avoid the __dict__ on objects, but the internal slots-system that is intended to speed up the dispatching of special methods.
  - thu 12y ago
    You missed his point 1.
    - PythonicAlpha 12y ago
      Ok. Then my answer belongs to the first in the sub-tree. What I actually did not understand: What means TFA?
      - deleted 12y ago
        [deleted]
      - stewartbutler 12y ago
        The Freaking Article.
        PythonicAlpha 12y ago
        Thanks! That helps.
      - 12y ago

rattray 12y ago

It seems pretty clear to me that we're ready for Python 4 by now. Experienced Python devs know the problems. We even have some solutions. So I think it's about time that leaders of the community like Ronacher, Rietz, Gaynor, etc come together to talk about the next generator of Python. We have PyPy now; why not use the tremendous advances there in the next Python? We have Go and Javascript to compete with and we understand a lot of why they're winning; it's time Pythonistas fought back instead of just leaving. Python 3 was supposed to be "the next thing" five years ago; it fizzled. Let's make the real "next thing" for 2016. Python 4.

pixelmonkey 12y ago
Go and Javascript are "winning"? That's news to me. They might be buzzier right now, but their compiler/interpreters only appeared in 2009 and Python has been around since 1991. Is it really such a surprise that the hype period is over for Python as a language spec? Now people are just getting a lot of stuff done with this productive language and nobody needs to be convinced any longer how good a language it is. Most people already know. Now, we're onto the good work of building a huge ecosystem of open source modules atop it. Call me when Go and JavaScript have the equivalent of the PyData stack and rock-solid drivers for every database technology on the planet. Then maybe I'll look at them for anything other than niche use cases. (Obviously JavaScript is still the only game in town inside the browser and Go seems like a pretty interesting alternative to C for UNIX system utilities.)
- rattray 12y ago
  Thanks; I should clarify. I agree Python is, in my subjective opinion, the best language spec and community in programming. It's simply the most fun, productive environment. But people are leaving; important people and newcomers and loyal stalwarts. Im on the go right now so can't access links, but there have been a handful of posts here in the past 6mo or so of prominent members leaving, or sharing their disenchantment. Many other "I'm moving to go" posts. And the velocity of npm/go are undeniable (look at the stats). Remember, npm is largely for the server, where js shouldn't have an advantage. I love Python. I want it to win. Maybe "lose" is too strong of a word, but it's certainly slipping.

orf 12y ago

I switched to Python 3 this year and I haven't looked back -there are some niggles, but overall its a great improvement. A great feature that's not really talked about is the __prepare__ function in metaclasses: you can supply a custom type that stores all class members. You could whip up multiple-dispatch using this (it lets you handle classes with duplicate property names) in conjunction with signature annotations, which I think is pretty neat.

Sami_Lehtinen 12y ago
This has been one of the reasons why I'm not currently doing anything for Google App Engine, they still lack Python 3 support. It feels awkward to code in Python 2 after Python 3.

pixelmonkey 12y ago

It's an interesting article, as a tour of some Python internals. But it comes off as a rant without a real suggestion. Python 3 was an internal interpreter cleanup. That's actually part of its lack of widespread popularity. The core developers didn't add a ton of unique functionality (and most of the new stuff was backported to 2.7 anyway), but they fixed some annoying problems in the CPython codebase and broke some things in the name of cleanup/unification of concepts. They made Python easier to maintain and easier to build new features atop. They sharpened the axe. Yes, they didn't clean up Armin's pet thing -- slots. And they created new problems for library developers in their bytes vs unicode changes. But they cleaned up a whole lot of other stuff. I personally think with the maturity of pypy and the stability of both the 2.7 and 3.4 lines of development, the Python ecosystem has never been more exciting. The advances in pypy make Python attractive for CPU-bound work and the inclusion of asyncio in the stdlib will make it more and more attractive for I/O-bound work over time. Python has long been a winner for mixed workloads, and the ecosystem around Python, especially pydata utilities like numpy and pandas, keep getting better. Stop complaining -- let's just build an awesome future atop this marvelous language.

the_mitsuhiko 12y ago
> But it comes off as a rant without a real suggestion. How exactly is that a rant? It's an exploration about a design decision / mistake that was carried through Python for 25 years and has left an impact. There is no complaint and there is no suggestion for Python. As I said in the early paragraph it's something that's interesting for people that are interested in language design.
- ZenoArrow 12y ago
  It's a rant because the solution was ill defined. If the title of the article is 'The Python I Would Like To See', you should expect more detail of how to implement the solution.
  - the_mitsuhiko 12y ago
    I hope it is valid to describe a situation and voice a wish for a situation that would look different without having to provide a concrete step towards that. The removal of the slot system would be a backwards incompatible change with very little direct benefit for developers. It could be interesting as a general goal for a hypothetical Python 4 but I do not believe it makes any sense to discuss this as anything more than a hypothetical case for the next few years.

nakovet 12y ago

I follow Ronacher's work, open source and posts and I agree with his arguments. This post remember me of Spolsky's one (http://www.joelonsoftware.com/articles/LeakyAbstractions.html http://www.joelonsoftware.com/articles/LeakyAbstractions.htm...) though. When you get to a point where you get to know what's under the hood and why it's not working the way you are trying to use, however in a majority of cases you are just fine and don't need to know what's happening under the hood and maybe if this majority is big enough that's just fine.

PhantomGremlin 12y ago
I got a totally different impression from the post. What I thought of was (http://en.wikipedia.org/wiki/How_many_angels_can_dance_on_the_head_of_a_pin%3F http://en.wikipedia.org/wiki/How_many_angels_can_dance_on_th...). In particular: In modern usage, this question also serves as a metaphor for wasting time debating topics of no practical value, or questions whose answers hold no intellectual consequence. Python will (or won't) be used for a particular application regardless of whether some contrived test takes 0.158 usec or takes 0.256 usec per iteration. This sort of misunderstanding comes up so often that there are a plethora of cliches for it. Here's another one: "Missing the forest for the trees.".

hosay123 12y ago

This post is surprisingly confused, it is phrased as a complaint about the language, then immediately degrades into CPython implementation specifics that have little bearing on the usability of the language itself. Ronacher should also know better than to post microbenchmarks like the one provided here, especially without corresponding (C) profiler output. At the C level, slots allow the implementation constant-time access to the most common code paths for an object, and especially when you have C code calling other C code via the type system (IMHO the primary use for Python, and still its strongest use case), "interpreter overhead" is reduced to a few extra memory indirection operations. In the alternative world, sure, perhaps some microbenchmark may behave faster, but now systemically, and for e.g. "reduce(operator.add, range(1000))" requires more hash table lookups than I can count. Python is all about providing a lightweight way to compose bits of fast code (the kernel, network stack, NumPy, MySQL, whatever). Unfortunately somewhere along the way solutions like Django got popular, which are almost the antithesis to this old viewpoint. Ronacher seems to be advocating that we should punish the CPython implementation's traditional strong areas in favour of.. actually, he didn't even describe any vision that was impacted by his complaints. He just seems to want the CPython implementation to suffer. Perhaps his complaint about new-style __getattribute__ would be better suited as a bug report, it seems the only substantial observation made about the language itself in this post.

the_mitsuhiko 12y ago
> This post is surprisingly confused, it is phrased as a complaint about the language, then immediately degrades into CPython implementation-specifics that have little bearing on the actual usability of the language itself. You might think that, but you are very wrong and I should probably make that point in another blog post. These CPython specifics are enshrined in the language. While PyPy does not have the actual structs, it needs to implement the same user exposed API as it slots were used. PyPy cannot just say "a + b" means "a.__add__(b)", it needs to implement the exact same dispatch logic that CPython has because people's code depends on it. //EDIT: > Ronacher seems to be advocating that we should punish the CPython implementation's traditional strong areas in favour of.. actually, he didn't even describe any vision that was impacted by his complaints. Maybe I did not make my point very clear but the whole last paragraph advocates about trying a version of Python that abolishes the internal slot system.

futuredale 12y ago

Ah nice. I was always curious why __getattribute__ didn't fire for specific dunder methods. Made subclassing pandas objects much more complicated.

alberth 12y ago

From the OP: >>"In recent years there is a clear trend of making Python more complex as a language. I would like to see the inverse of that trend. I would like to see an internal interpreter design could be based on interpreters that work independent of each other, with local base types and more, similar to how JavaScript works." Sounds like OP should investigate Lua (and LuaJIT in particular). He might like Lua more than Python. (Given that OP created Flask, I'd love to see a Flask equivalent developed in Lua) Edit: typo

scrollaway 12y ago
Good job not reading the article and skipping to the end, I guess? >> This is in fact how many other dynamic languages work. For instance this is how lua implementations operate, how javascript engines work etc. The clear advantage is that you can have two interpreters. What a novel concept.

PythonicAlpha 12y ago

I really would like to see, when the suggested, simpler system would be tried. It is sometimes amazing, how by simplification gains could be reached against common wisdom. And even, when no gains or little losses would be there with the new, simpler system, it could be worthwhile to change, since the current system has his huddles, that I also already stumbled about, and which are not clearly communicated, since it is such a deep implementation detail of CPython. Also it was mentioned, that a simpler system could be even more powerful.

xorcist 12y ago

Armin for BDFL! (Just a joke, people. Just a joke.) Anyway, I wonder why Armin's not more involved with core Python things, his perspective should be valuable. As someone doing some Python stuff on the side his articles are always worth reading carefully. The Python community has always been the best part of the language, but the important subcommunities like NumPy, Twisted and PyPy has always seemed like they are a bit outside looking in. I don't know why. Perhaps the language would have evolved differently if these projects were a bit more involved in the actual language development.

philh 12y ago

>>> original = 42 >>> class FooProxy: ... def __getattr__(self, x): ... return getattr(original, x) ... >>> proxy = FooProxy() >>> proxy 42 For anyone else who found this as confusing as I did ("wtf, how can proxy actually be 42?"), what's going on here is that it's calling `proxy.__repr__()` when it attempts to display `proxy`, which in turn calls `42.__repr__()`. (Similarly, `proxy + 1` calls `42.__add__(1)`.)

fndrplayer13 12y ago

Thanks for the very informative post. I've been a full-time Python developer for only the past 7 or 8 months now. I had not in that time learned about the slot system or the CPython implementation. Also, <3 Flask. So good.

GnarfGnarf 12y ago

I'm learning Python and I must say it is a wonderful language. Being able to concatenate strings by simply saying "a + b" is a great productivity boost (coming from C++). Python libraries are powerful. I can read an Excel spreadsheet with one line of code. I can create plots in PDF format with a half-dozen lines. Amazing! However, I am disappointed with the difficulty of turning a program into a Windows EXE. I wrote a small program (couple of thousand lines), tried Py2exe which failed to handle the Numpy (or Matplotlib, I forget which) imbroglio. PyInstaller works, except that the EXE is 85MB, and takes one minute to start up. Not practical for customer distribution. I can't expect my customers to install the Python runtime. In contrast, my 500KLOC C++ program, with all its third-party libraries, is 19MB. Yes, I know, Python needs everything including the kitchen sink. Still, 85MB is not practical. Too bad. Not all programs run on a server.

alexchamberlain 12y ago
I love Python too... but, I love C++ as well... http://en.cppreference.com/w/cpp/string/basic_string/operator%2B http://en.cppreference.com/w/cpp/string/basic_string/operato...
theseoafs 12y ago
> Being able to concatenate strings by simply saying "a + b" is a great productivity boost (coming from C++) C++ has had this operator overloaded for strings for decades. With regard to the size of compiled executables, I can't really say much except "that's not what it's made for". If you need to ship compiled executables to people, Python is an extraordinarily inappropriate choice; stick with C++.
- dkersten 12y ago
  I've often heard it said that if you're not using std::string for strings in C++, you're (arguably) not using C++
  - GnarfGnarf 12y ago
    I just did a test using std:: and old-fashioned char[]. The fancy std:: is fifteen times slower than the strcat(). In a loop with intensive string manipulation, this could cause the program to get back to you in 15 seconds instead of 1 second. You don't mind waiting? Here's the code (Visual Studio 2010): #include <string> #include <time.h> char buf[64], buf2[64]; clock_t start = clock(); for(i = 0; i < 100000; i++) { _snprintf(buf, sizeof(buf), "%d", i); _snprintf(buf2, sizeof(buf2), "%d", i * i); #if defined FAST strcat(buf, buf2); #else std::string s1 = buf; std::string s2 = buf2; std::string s3 = s1 + s2; #endif } _snprintf(buf, sizeof(buf), "%.4f\n", (float)(clock() - start) / (float)CLK_TCK); OutputDebugString(buf);

notastartup 12y ago

I moved from Javascript to Python 2.7 and haven't looked back since. I now strictly use Python in all of my projects, Javascript only because browser doesn't support it. I wish that Python would introduce a few things from the Javascript world. Easy async, and a real-time Meteor like framework.

lmm 12y ago
Improved async is the biggest new feature in 3.4

berdario 12y ago

I was a bit puzzled by the "lightweight old style classes" remark, so I tried it out: >>> sys.getsizeof(OldStyleClass), sys.getsizeof(NewStyleClass) (104, 904) >>> sys.getsizeof(OldStyleClass()), sys.getsizeof(NewStyleClass()) (72, 64) I agree that OldStyleClasses might be simpler (while less featureful), but I think I'd care more for the footprint of instances, rather than class objects themselves

scardine 12y ago

tldr; method calling is unnecessarily expensive in current Python due to debatable design decisions.