6 ms·
I have grown to seriously dislike 'namedtuple'. It seems great, and it's so easy to use, but there's so few places it can be used without dragging in future bac
by eesmith 9y ago
I have grown to seriously dislike 'namedtuple'. It seems great, and it's so easy to use, but there's so few places it can be used without dragging in future backwards-compatibility baggage.
Let's say you have a light-weight object, like a "full name" which can be represented with a "forename" and an "surname". (Yes, there are many names which don't fit this category. This is a toy example.) So you write:
from collections import namedtuple
FullName = namedtuple("FullName", "forename surname")
name = FullName("Edward", "Smith")
So simple! So clean! No "self.forename = forename" or need to make your own __repr__.
Except you also inherit the tuple behavior, because this isn't really light-weight object but a ... let's call it a heavy-weight tuple.
>>> name[0]
'Edward'
>>> name[1]
'Smith'
This mean people may (okay, are likely) do things like:
forename, surname = name
or even
for forename, surname in names:
...
Even if that's not what you want in your API. If you care about backwards compatibility, then if you replace the namedtuple with its own class then you'll have to re-implement __getitem__, _replace, __len__, and count - methods which make no sense for a simple name class.
Or, if you decided to change the API, to make it be:
FullName = namedtuple("FullName", "forename middlename surname suffix")
then you'll again break code which used the full indexing API that you originally implicitly promised, simply by saying this is a namedtuple.
'namedtuple' has one good use - migration from a tuple object to a real object while allowing backwards compatability to the tuple API. For example, the old tuples once returned from os.stat() and time.gmtime().
It's also fine if it will only be used by code where you have full control over its use and can either prevent inappropriate API use like destructuring assignment, or can update all uses if you want to change the API. For example, a light-weight object used in a test-suite or internal helper function.
Otherwise, don't use it. Spend the extra few minutes so in the future you don't have to worry about backwards compatibility issues.
- sevensor 9y agoI disagree -- namedtuple has a crucial property that I've found incredibly useful. Namedtuples are immutable. This makes it impossible (well, not really, but hard) for a namedtuple to change its state without telling me about it, because if it changes state it's not the same namedtuple any more. I agree that namedtuple being a sequence type is problematic, but deciding not to use a lightweight immutable data structure because your colleagues might use it badly is throwing out the baby with the bathwater.
- eesmith 9y agoI was not speaking of in-house software were only "your colleagues might use it badly". In that case, you have access to the entire source code and aren't so worried about backwards compatibility. I was referring to publishing a library which uses a namedtuple as part of the public API. Your argument seems to be that there are good things about namedtuple - which I agree with - so therefore I shouldn't be so dismissive of it. That might be reasonable if there were no other easy way to construct a light-weight object with immutable attributes. But the attr package exists, as js2 pointed out in a sibling branch. See http://www.attrs.org/en/stable/examples.html#immutability http://www.attrs.org/en/stable/examples.html#immutability .
- kemitche 9y agoIf you're writing a python library, you should already be aware that there's only _so much_ you can do to prevent the library's users from using it incorrectly or oddly - this is Python after all, there's no private/protected. From that perspective, I don't see the problem with returning a namedtuple, if you document it and write your changelogs appropriately when/if you add a field.
- eesmith 9y agoWe must assume people are being reasonable. If someone uses a module where the top-level docstring says "not part of the public API", and which starts with an "_", then that person is not being reasonable, by well-established Python developer standards. I have no objections to breaking compatibility for people who reach into a private API. It's different if I hand them an object and say "use it". Even if I don't document anything, it's implicit that this is part of the public API. These are two different things, though yes, there can be ambiguity. "if you document it and write your changelogs appropriately when/if you add a field." I tried to point this out earlier. It takes only a few (boring) minutes to write the essentially boilerplate code. While over the lifetime of the project it will likely take more time to write the supporting documentation and deal with the effects of breaking compatibility.
- sevensor 9y ago
- nerpderp83 9y agoI think you are missing the best aspects of namedtuple. The positional properties are there to ease interaction with other APIs. The vast majority of access should be through `.member`
- abecedarius 9y agoNo, I agree with eesmith: I've been bitten multiple times by bugs from namedtuple objects providing the tuple interface which I'd never intended to use. So I stopped using namedtuple in favor of a new more-restricted module I wrote to replace it.
- nerdponx 9y agoOut of curiosity, what kind of bugs have you see that arose from "providing the tuple interface" inappropriately?
- abecedarius 9y agoAll I remember clearly still is the baffled frustration, like you feel when a C compiler optimizes out undefined behavior you didn't know about. I was using namedtuple in generally this kind of way: https://github.com/darius/tinyhiss/blob/master/terp.py#L90 https://github.com/darius/tinyhiss/blob/master/terp.py#L90 (this was a trampolined interpreter for a dialect of Smalltalk). One thing that could happen is, when you might expect to see a type error if you take one of these objects x and mistakenly write x[0], it'd just continue on with some field value. Or if you mistakenly try to iterate over x, you can. Or maybe you were generically walking a tree of objects and didn't intend for an instance of one of these classes to have children. The bafflement I ran into was weirder than these; maybe it had to do with the fact that inheritance was mixed in here too? And I think I'd defined `__getitem__` in some of these classes one of those times. Sorry I can't remember more. To write https://github.com/darius/tailbiter https://github.com/darius/tailbiter I had to know quite a bit about Python, but it's a big language! It was a mistake to assume I could use namedtuple as just a convenient way to define a struct type without grokking all its details.
- 9y ago
- deleted 9y ago[deleted]
- hchasestevens 9y agoThis is, in fact, the exact reason for the _source attribute which is being discussed in this article - you can use namedtuple to easily create a structure, get its source, and then edit it to suit your individual project needs. E.g.: >>> from collections import namedtuple >>> FullName = namedtuple("FullName", "forename surname") >>> print(FullName._source) from builtins import property as _property, tuple as _tuple from operator import itemgetter as _itemgetter from collections import OrderedDict class FullName(tuple): 'FullName(forename, surname)' __slots__ = () _fields = ('forename', 'surname') def __new__(_cls, forename, surname): 'Create new instance of FullName(forename, surname)' return _tuple.__new__(_cls, (forename, surname)) @classmethod def _make(cls, iterable, new=tuple.__new__, len=len): 'Make a new FullName object from a sequence or iterable' result = new(cls, iterable) if len(result) != 2: raise TypeError('Expected 2 arguments, got %d' % len(result)) return result def _replace(_self, **kwds): 'Return a new FullName object replacing specified fields with new values' result = _self._make(map(kwds.pop, ('forename', 'surname'), _self)) if kwds: raise ValueError('Got unexpected field names: %r' % list(kwds)) return result def __repr__(self): 'Return a nicely formatted representation string' return self.__class__.__name__ + '(forename=%r, surname=%r)' % self def _asdict(self): 'Return a new OrderedDict which maps field names to their values.' return OrderedDict(zip(self._fields, self)) def __getnewargs__(self): 'Return self as a plain tuple. Used by copy and pickle.' return tuple(self) forename = _property(_itemgetter(0), doc='Alias for field number 0') surname = _property(_itemgetter(1), doc='Alias for field number 1') You can then pretty simply remove the inheritance from tuple and change the properties to get the behavior you describe.
- nerdponx 9y agoThis is a nice touch but insisting that it should be part of the stock functionality for a basic data structure makes no sense to me.
- eesmith 9y agoI do not think it's simple to remove the __getitem__ lookup behavior from this template. To start with, forename depends on _itemgetter(0). I might as well re-write it as: class FullName(object): __slots__ = ("_forename", "_surname") def __init__(self, forename, surname): self._forename = forename self._surname = surname def __repr__(self): return "%s(forename=%r, surname=%r)" % ( self.__class__.__name__, self._forename, self._surname) @property def forename(self): return self._forename @property def surname(self): return self._surname That took about two minutes to write. How long does it take you to rewrite the _source so it doesn't support __getitem__ or __len__?
- js2 9y ago> Except you also inherit the tuple behavior. It sounds like you might be happier with attrs if you want an easy to construct class that doesn't have tuple behavior: http://www.attrs.org/ http://www.attrs.org/ http://www.attrs.org/en/stable/why.html#namedtuples http://www.attrs.org/en/stable/why.html#namedtuples
- sametmax 9y agoOnly if one is misusing namedtuple: it should be used for iterable things, like csv lines, db rows, coordinates, etc. but not for a person. That makes no sense. If what you need is a cheap, quick container, then you want types.SimpleNamespace.
- eesmith 9y ago"but not for a person. That makes no sense." I wrote 'toy example' for a reason. While it does not make domain sense, it's easy to understand and it highlights the problems I wanted to show. "it should be used for iterable things" Really? This article concerned the use of namedtuple in functools.py 21:from collections import namedtuple 403:_CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"]) The cache_info() method returns a _CacheInfo instance to the user. Where/what are the "iterable things"? If a future version adds another attribute, it will break most code which uses destructuring assignment. (For that matter, I can't see why it make sense to use a namedtuple for this case. Why not just: class _CacheInfo: def __init__(self, hits, misses, maxsize, currsize): self.hits = hits ... ? Who cares about immutability here?) Or for another case, here's a Python function which returns a class derived from a namedtuple: >>> from urllib import parse >>> result = parse.urlparse("http://www.python.org/") >>> for term in result: ... print(result) ... >>> for i, term in enumerate(result): ... print(i, repr(term)) ... 0 'http' 1 'www.python.org' 2 '/' 3 '' 4 '' 5 '' Again, where is the "iterable thing"? I mean, the ParseResult instance is iterable, but my point is that it makes no sense to iterate over those fragments. If the Python standard library gets it wrong, what is the right solution? "If what you need is a cheap, quick container, then you want types.SimpleNamespace. " You mean the one where the documentation at https://docs.python.org/3/library/types.html#types.SimpleNamespace https://docs.python.org/3/library/types.html#types.SimpleNam... says "SimpleNamespace may be useful as a replacement for class NS: pass. However, for a structured record type use namedtuple() instead." ? I do not think you are correct.
- sametmax 9y agoThere are plenty of issues with the stdlib. Part of it don't respect the PEP8, there are inconsistencies with argument types and orders, some of it is even unusable for real life usages (ex: csv in python 2.7) and we keep it for legacy reasons. The fact is, the use cases you are showing are wrong, and the one I'm talking about are the proper ones. It's ok to make mistakes, we all do a lot of them. And we will have to pay for those. If you are interested in the subject, last month we had a huge debate on the Python mailing list about a way to have a proper official immutable but non interable structured record to replace named tuple and avoid those abuses. And also to maybe include something like attrs in the stdlib.
- icebraining 9y agoI agree that namedtuples shouldn't be used if you don't actually want a tuple.
- raymondh 9y agoI concur :-) Otherwise, use a dict, types.SimpleNamespace, python-attrs, or a custom class. FWIW, the origin of namedtuple() was that some variation of it had been re-invented many times and this tool brought together the common features of those tools (substitutability for regular tuples, compact storage, usability as a dictionary key, named fields, a clear __repr__, etc.)
- xapata 9y agoNametuple is designed for backwards compatibility with tuples. It's to make an easy transition from using tuples as structs to using something with attributes, without breaking all the code you just demonstrated.
- eesmith 9y agoYes, that was the "one good use" I mentioned. However, in practice it's not used that way. This specific LWN article concerns the following code from functools.py: _CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"]) It is used like this: def decorating_function(user_function): wrapper = _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo) ... def _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo): ... def cache_info(): """Report cache statistics""" with lock: return _CacheInfo(hits, misses, maxsize, cache_len()) The _lru_cache_wrapper is then replaced by a C extension in _functoolsmodule.c which only uses the cache_info_type as: static PyObject * lru_cache_cache_info(lru_cache_object *self, PyObject *unused) { return PyObject_CallFunction(self->cache_info_type, "nnOn", self->hits, self->misses, self->maxsize_O, PyDict_GET_SIZE(self->cache)); } This is basically the same code, just in C. There is no reason for this to expose a tuple API. And yet it does. >>> @functools.lru_cache(maxsize=None) ... def fib(i): ... if i <= 1: return 1 ... return fib(i-1)+fib(i-2) ... >>> for i in range(10): ... print(i, fib(i)) ... 0 1 1 1 2 2 3 3 4 5 5 8 6 13 7 21 8 34 9 55 >>> fib.cache_info() CacheInfo(hits=17, misses=10, maxsize=None, currsize=10) >>> for x in fib.cache_info(): ... print(x) ... 17 10 None 10 A spot check shows no uses of namedtuple in the Python standard library which are used to keep backwards compatibility with tuples, though I only looked at a few. So while I agree with you, it appears that the core developers do not agree with us.
- xapata 9y agoQuoting the code as it is doesn't show its development history. The cache_info could have been a tuple originally then updated to a namedtuple. Regardless, through conversation I am confident that at least one of the core devs thinks about namedtuple in that manner. Not as namedtuple's only purpose, but as one major benefit. Further, the core devs are not a single entity, but many people who don't always agree with each other. Edit: note Raymond's comment elsewhere on this page.
- ubernostrum 9y agoThis is Python, where lots of things are iterable/indexable. If for some reason you don't want an iterable/indexable object, then don't make one. But don't blame a tool other people use for the fact that it wasn't the tool you wanted it to be.
- eesmith 9y agoI am pointing out a long-term negative with using namedtuple that is not obvious at first use, in hopes that others might gain some insight on when to use it and when not to use it. Other people can and do use tools incorrectly. I use tools incorrectly. That doesn't mean we shouldn't learn from the mistakes of others.
- yen223 9y agoIf you want an immutable object with a fixed set of fields - not an unreasonable request, mind you - the Python standard library doesn't give you much to work with. NamedTuples are the closest thing we have, but it's so clunky to use that it's not widely used. It's incredibly frustrating, because this leads to external libraries abusing mutable objects everywhere, Django being one of the worse offenders.