12 ms·
Redesigning Python's named tuples
- hasenj 9y agoWhy don't languages like Python have the concept of a C-like struct? Seems like it should be straight forward to have with no downsides that I can think of.
- bkul 9y agoSeems to me that classes generally replace structs in such languages. I know they're not the same thing, but classes have the features of structs (and much more) at the cost of performance.
- smitherfield 9y ago> classes have the features of structs (and much more) at the cost of performance. Not true; C++ classes have zero performance overhead over C/C++ structs (because they're just another word for the same thing). Of course Python classes have a lot of costly features that C++ classes don't have, but the notion of a "class" (with methods, constructors etc) is not one of them.
- kronos29296 9y agoAnd also makes them look more complicated with more syntax. C++ has all those features that I will never know without adding performance. But people don't use Python for performance but ease of use. In C++ ease of use is given up for the sake of performance. In python it is the other way.
- hasenj 9y agoDon't C++ classes (and therefore structs) have a hidden vtable?
- bpicolo 9y agoA lot of people (my whole company, for sure) are using namedtuples essentially just as immutable structs. This is a case where classes aren't sufficing. To be fair, it's at least a little bit the functional-programming, immutability bandwagon, but either way it's helping maintain large python apps with many devs. Python36 gradual typing also helps here though, and am definitely interested to see if that changes the game a bit
- chubot 9y agoI had this question for awhile too, but I realized after hacking on the Python interpreter that it breaks the execution model: - Python compiles a single module to bytecode at a time (.py to .pyc) - It interleaves execution and compilation -- imports are dynamic, you can put arbitrary computation between them, and conditionally import, etc. If you had structs, then you would need to resolve names to offsets at compile time -- e.g p.x is offset 0, p.y is offset 4. That's a form of type information. Types can be defined in one module and used in another, and Python really has no infrastructure for that. Nor do any other dynamic languages like Ruby, Perl, JavaScript, Lua, or PHP, as far as I can tell. They are all based on hash tables because deferring name lookup until runtime means that I can start executing "now" rather than looking all over the program for types. It probably helps a bit with the REPL too. The need for type information in a translation unit is also what gives you C's header file mess, so it's not a trivial problem.
- pjmlp 9y agoCommon Lisp, Dylan and Julia do offer such support. It is a matter of caring about it, and having JIT/AOT support as part of the standard tooling. C header file mess is caused by its designers ignoring the work outside AT&T and not bothering to implement a module system.
- agumonkey 9y agomodules are unloved children, I recently used scheme48 which has a very nice module system (funny considering how old, but anyway) even though other don't have one..
- nnwedwe 9y agoCan you point us to a public repository that contains your output? It would be very instructive.
- flavio81 9y ago> C header file mess is caused by its designers ignoring the work outside AT&T and not bothering to implement a module system. To be fair to them, they wanted to create a language whose compiler did fit into the limited hardware they had. It's not their fault that UNIX had so much success and the world largely went there ignoring most of the progress made in programming tools and languages during the 70s.
- pjmlp 9y agoIf you mean dynamic languages, Common Lisp, Dylan and Julia do have such concept.
- ubernostrum 9y agoPython does have the ability to create and use packed byte objects which are interoperable with C structs. The interface is even in a module in the standard library called... 'struct'. The main issues are: 1. The C-struct interoperability is cumbersome, since really it's just byte objects with some helper functions to pack and unpack them according to a given definition (so you need both the bytes and the correct format definition in order to turn it back into native Python objects), and 2. Most people already are using one of a few different struct-like things depending on their exact use case and preference, and getting everyone to agree on a One True Struct Object to build in to Python or its standard library is probably never going to happen. Just look at namedtuples -- they were an attempt to take the most struct-like thing in Python (the tuple) and add extra optional struct-like behavior to appease people who were frustrated with the lack of structs, but of course people rant about them for having a weird internal implementation and for having behavior that makes them backwards-compatible with plain tuples.
- eesmith 9y agoctypes can also be used to create C-like structs. It has the advantage over the struct module that the elements can be accessed and modified in place.
- Pxtl 9y agoThe tuple was Python's answer to the C-Struct, really. If I want to duct-tape 4 values together and hand them around as 1, the convention was to use a tuple. Tuples don't name values, but that's actually important for performance - Python is an old-fashioned dynamic language that still uses hashtable access for everything, but tuple indexing is a simple array offset. Considering Python's age, reducing that endlessly-nested series of hashtable hits was important back in the day.
- Too 9y agoIt seems like the python maintainers always try to complicate things for themselves to cater for some obscure use case that nobody use. I mean really, when was the last time you ever accessed a namedtuple by index?? (For vectors i do see a benefit that you can throw it right into a matrix multiplication that is designed for arrays but really, you can do that equally easy in other ways). And constructing them on the fly doesn't make much sense either as it defats the benefits of a shared type across the code base. All people ever wanted was just a way to write in one line ´Point3D = struct(x, y, z)´ that translates into the exact equivalent of: class Point3D: def __init__(x, y, z): self.x = x self.y = y self.z = z What would be the startup performance of parsing the above 5 line class compared to a namedtuple? Surely it should be faster to create it with a shorthand form built into the language, if the functionality is equivalent?
- zodiac 9y ago> What would be the startup performance of parsing the above 5 line class compared to a namedtuple? I think https://news.ycombinator.com/item?id=15135147 https://news.ycombinator.com/item?id=15135147 explains why you can't optimize namedtuple like that, since dynamic dispatch makes calls to them not "the exact equivalent" of that code. The proposal to move them to a top level data structure could solve this problem though, I think.
- joshuamorton 9y ago>It seems like the python maintainers always try to complicate things for themselves to cater for some obscure use case that nobody use. I mean really, when was the last time you ever accessed a namedtuple by index?? I do things like for x, y in iterable_of_ntuples: #process all the time
- asragab 9y agoThis article is ostensibly about how the current implementation of namedtuples has had serious consequences for the startup time of Python, because namedtuples are used in the compilation of classes (roughly). However, somehow this buries the lede...the most "interesting" discussion is kicked off by the first comment: Issues around the performance of Python and programs written in it have far wider consequences than startup time. During all the time any Python program is running, its host machine is consuming power that typically depends on pumping CO2 into the atmosphere. If most of that power is wasted, the effects go far beyond extra money to buy it, or to operate extra servers, or users who wait a little longer. The carbon footprint of a Python program that runs throughout a data center, or many data centers, adds up. There was an article earlier on HN about the energy consumption pattern of Bitcoin/Ethereum and presumably any blockchain that implements a proof-of-work protocol/scheme, and between that article and this comment - I've started to notice a growing unease (I am probably waaay behind on the uptake) about the "world-eating" capacity of software. I wonder how quantifiable implementation decisions like the ones exhibited by namedtuples in Python, which one might argue is an unfortunate/accidental side-effect vis-a-vis energy consumption, versus ones like proof-of-work, which I would argue are explicitly designed to be expensive. And if anything should come of that quantification, namely, does optimizing code really become a moral imperative, and if so are there some usability and refactorability metrics that are often held in high regard that we ought to consider abandoning in the name of "energy efficient" software. Obviously, this isn't a simple tradeoff, software that is difficult to write because it is highly optimized is difficult to maintain, and it might be the case that performance derived energy savings are outweighed by the energy cost of maintenance (literally, the energy cost of debugging and testing).
- slavik81 9y agoIf the cost of energy accurately reflected its effect on the environment, a lot of these sorts of questions would be easier to answer.
- alphaalpha101 9y agoMy PC is powered, I think, entirely by hydroelectric power stations. Almost all of New Zealand's power is.
- lngnmn 9y agoObviously, practicality should beat purity and very clever and reasobable syntax (x=1, y=2) which could be used/generalized for a procedure arguments and effecient implementation in C (we have C-based arrays (lists) and hash-tables, so why not generalized records/tuples?) should be accepted. The problem with a "pure democracy" is that majority cannot be smarter than the top 5% individuals, so really good ideas almost never getting thorough the bulk of a bell curve.
- joejev 9y agoWhy is (x=1, y=2) more practical than nt(x=1, y=2)? Also, we have a very efficient implementation for tuples. collections.namedtuple is quite fast and the article shows cnamedtuple (which I am the author of) which is a C implementation that is even faster. None of this needs a change to the language itself.
- rplnt 9y ago> Why is (x=1, y=2) more practical than nt(x=1, y=2)? It was outlined in the linked article. Order of named arguments should not matter so new syntax would be better.
- joshuamorton 9y agoI think a strong argument against that syntax is that namedtuples are, essentially, an abstract class (or maybe a metaclass). An anonymous namedtuple doesn't really make sense. You want the type information. As of python 3.6, this is already possible: class Point2D(typing.NamedTuple): x: int y: int p = Point2D(x=2, y=4) If you allow anonymous namedtuples you lose one of the big values of a namedtuple, which is that if I want a Point2D, you can be sure I'm getting a Point2D, and not a Point3D. With anonymous namedtuples, there's nothing stopping you from passing a (x=2, y=3, z=4), when you wanted a (x=2, y=3). And maybe that will work fine, but maybe not (or the reverse). All this is to say, an anonymous namedtuple is an oxymoron. NamedTuples should be named. This isn't a "really good idea".
- adambyrtek 9y ago> The problem with a "pure democracy" is that majority cannot be smarter than the top 5% individuals, so really good ideas almost never getting thorough the bulk of a bell curve. Python is governed as "benevolent dictatorship" (with Guido as BDFL and maintainers as your "top 5%"), definitely not "pure democracy", so your argument doesn't really hold. When it comes to the syntax, just because something is "very clever" doesn't mean that it's necessarily a good idea. Syntax changes should be carefully considered and introduced only if the benefit clearly outweighs the cost of extra complexity. If you still believe that it's such a good idea then feel free to draft a PEP: https://www.python.org/dev/peps/ https://www.python.org/dev/peps/
- alphaalpha101 9y ago> Either way (or both) would be implemented in C for speed. It would allow named tuples to be created without having to describe them up front, as is done now. But it would also remove one of the principles that guided the design of named tuples, as Tim Peters said: > > How do you propose that the resulting object T know that T.x is 1. T.y is 0, and T.z doesn't make sense? Declaring a namedtuple up front allows the _class_ to know that all of its instances map attribute "x" to index 0 and attribute "y" to index 1. The instances know nothing about that on their own, and consume no more memory than a plain tuple. If your `ntuple()` returns an object implementing its own mapping, it loses a primary advantage (0 memory overhead) of namedtuples. Post-decree, Ethan Furman moved the discussion to python-ideas and suggested looking at his aenum module as a possible source for a new named tuple. But that implementation uses metaclasses, which could lead to problems when subclassing as Van Rossum pointed out. > Jim Jewett's suggestion to make named tuples simply be a view into a dictionary ran aground on too many incompatibilities with the existing implementation. Python dictionaries are now ordered by default and are optimized for speed, so they might be a reasonable choice, Jewett said. As Greg Ewing and others noted, though, that would lose many of the attributes that are valued for named tuples, including low memory overhead, access by index, and being a subclass of tuple. > Rodolà revived his proposal for named tuples without a declaration, but there are a number of problems with that approach. One of the main stumbling blocks is the type of these on-the-fly named tuples—effectively each one created would have its own type even if it had the same names in the same order. That is wasteful of memory, as is having each instance know about the mapping from indexes to names; the current implementation puts that in the class, which can be reused. There might be ways to cache these on-the-fly named tuple types to avoid some of the wasted memory, however. Those problems and concern that it would be abused led Van Rossum to declare the "bare" syntax (e.g. (x=1, y=0)) proposal as dead. From what I've read, v8's implementation of objects in Javascript goes basically like this: when you call a constructor function and assign properties to your object, it makes up struct types and ties the object to that type or something. Like this: function Point2D(x, y) { this.x = x; this.y = y; } let p = new Point2D(1.0, 2.0) initially you'll have an empty object, which will be an empty struct. 'this.x = x' will change the type to the 'X' struct, and 'this.y = y' will change the type to the 'XY' struct. If you do this again with another object, they'll share these underlying structs. Now this is perhaps easier with a JIT, and perhaps not. But it bears thinking about. Why not just make it so that (x: 1, y: 0) - which would be the best syntax IMO as it fills out the {set, dict; tuple, ???} square - creates an object that shares its class with every other namedtuple that has exactly the x and y properties in exactly that order? It really frustrates me when I read 'Those problems and concern that it would be abused led Van Rossum to declare the "bare" syntax (e.g. (x=1, y=0)) proposal as dead.' I mean come on, I know it's a different environment in Python than in V8, but seriously this is a solved problem. Those problems? Those problems are a solved problem that a solution was already proposed for in the thread. Just do that. >He elaborated on the ordering problem by giving an example of a named tuple that stored the attributes of elementary particles (e.g. flavor, spin, charge) which do not have an automatic ordering. That argument seemed to resonate with several thread participants. I don't want to be too harsh, but this is nonsensical rubbish. Dictionaries preserve order in Python. This ship sailed a long time ago. Namedtuples also already preserve order. Tuples preserve order. Lists preserve order. Dictionaries preserve order. What doesn't preserve order? Like, I get that it's not strictly defined that dictionaries preserve order, but they do, and people do rely on that, and so it's never going to actually be changed. >This is exactly why I scream at relational databases. If you can't tell the difference between a set and a list, and especially if you want to store a list in a set-based paradigm, you are going to have ALL SORTS of grief ... Unrelated but I found this comment funny. This guy has heard of an index, right?
- vanni 9y agoPrevious LWN article on the subject: "Reducing Python's startup time" (Aug 16, 2017) https://lwn.net/Articles/730915/ https://lwn.net/Articles/730915/ Related HN discussion: https://news.ycombinator.com/item?id=15131981 https://news.ycombinator.com/item?id=15131981
- tveita 9y agoI'd be happy to see it optimized - namedtuple looks like a convenient way to quick-and-dirtily define a data structure, but several times I've ended up changing back to plain tuples because using namedtuple was much slower, especially when pickling. Maybe they could add a variant that isn't a tuple as well.
- zimablue 9y agoThe subtext is jealousy of JavaScript object creation/destructing, it's the thing I like most about JavaScript I think, really makes your code feel fluid. I couldn't tell you what they preserve order though. There's even precompiler extensions to destructure immutable.js Maps.
- alexchamberlain 9y agoCould you expand on the syntax you're referring to?
- zimablue 9y agoNot well but here is a link, the object destructing section https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Operators/Destructuring_assignment https://developer.mozilla.org/en/docs/Web/JavaScript/Referen...
- terragon 9y agoSeconded. When using JavaScript Es6, I really love how object and array destructuring, spread operator and default values work together to make my code so dense.
- ryanx435 9y agoI have yet to encounter a time when a list of 2 items didn't suffice for tuples. I'm sure applications exist, as tuples are immutable, but I have never encountered the need for them.