4 ms·

That post is from 2010, before even 2.7 was released. It is extremely out of date. Re-running the same tests I get: In [1]: from typing import NamedTuple

by gipp 9y ago

That post is from 2010, before even 2.7 was released. It is extremely out of date. Re-running the same tests I get: In [1]: from typing import NamedTuple In [2]: class A(NamedTuple): ...: a = 1 ...: b = 2 ...: c = 3 ...: In [3]: a = A() In [4]: class B: ...: __slots__ = ('a', 'b', 'c') ...: def __init__(self, a=1, b=2, c=3): ...: self.a = a ...: self.b = b ...: self.c = c ...: In [5]: b = B() In [6]: c = dict(a=1, b=2, c=3) In [7]: d = (1,2,3) In [8]: e = [1,2,3] In [9]: key = 2 In [10]: %timeit z = a.c 38.2 ns ± 0.07 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) In [11]: %timeit z = b.c 48.1 ns ± 0.0461 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) In [12]: %timeit z = c['c'] 38 ns ± 0.062 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) In [13]: %timeit z = d[2] 38.8 ns ± 0.0425 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) In [14]: %timeit z = e[2] 39.8 ns ± 0.0641 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) In [15]: %timeit z = d[key] 48.6 ns ± 0.0713 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) In [16]: %timeit z = e[key] 49.4 ns ± 0.0369 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) Namedtuple looks faster than slots here. And for comparison's sake: In [17]: class F: ...: def __init__(self, a=1, b=2, c=3): ...: self.a = a ...: self.b = b ...: self.c = c ...: In [18]: f = F() In [19]: %timeit z = f.c 53.7 ns ± 0.265 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

gshulegaard 9y ago
The question was asked in 2010, but the benchmark post I was referencing is from 2014. It doesn't appear as if `namedtuple` has changed much: https://hg.python.org/cpython/file/tip/Lib/collections/__init__.py#l302 https://hg.python.org/cpython/file/tip/Lib/collections/__ini... Running the same tests from the post, I get similar results: from collections import namedtuple STest = namedtuple("TEST", "a b c") a = STest(a=1,b=2,c=3) class Test(object): __slots__ = ["a","b","c"] a=1 b=2 c=3 b = Test() c = {'a':1, 'b':2, 'c':3} d = (1,2,3) e = [1,2,3] f = (1,2,3) g = [1,2,3] key = 2 if __name__ == '__main__': from timeit import timeit print("Named tuple with a, b, c:") print(timeit("z = a.c", "from __main__ import a")) print("Named tuple, using index:") print(timeit("z = a[2]", "from __main__ import a")) print("Class using __slots__, with a, b, c:") print(timeit("z = b.c", "from __main__ import b")) print("Dictionary with keys a, b, c:") print(timeit("z = c['c']", "from __main__ import c")) print("Tuple with three values, using a constant key:") print(timeit("z = d[2]", "from __main__ import d")) print("List with three values, using a constant key:") print(timeit("z = e[2]", "from __main__ import e")) print("Tuple with three values, using a local key:") print(timeit("z = d[key]", "from __main__ import d, key")) print("List with three values, using a local key:") print(timeit("z = e[key]", "from __main__ import e, key")) Output: Named tuple with a, b, c: 0.147349834442 Named tuple, using index: 0.0364408493042 Class using __slots__, with a, b, c: 0.0334758758545 Dictionary with keys a, b, c: 0.0339260101318 Tuple with three values, using a constant key: 0.0425179004669 List with three values, using a constant key: 0.0301761627197 Tuple with three values, using a local key: 0.0436310768127 List with three values, using a local key: 0.0309669971466
- aaronchall 9y ago
  I just noticed your slotted object uses class variables. That is completely wrong - so your analysis is probably wrong. In Python, I think it's important to do what's directly semantically correct. However, we can compare apples and oranges on what they have in common, so here's a bit of my own analysis: >>> from timeit import repeat >>> from collections import namedtuple >>> class Slotted: __slots__ = 'a', 'b', 'c', 'd', 'e' ... >>> NT = namedtuple('NT', 'a b c d e') >>> >>> nt = NT(1,2,3,4,5) >>> s = Slotted() >>> s.a, s.b, s.c, s.d, s.e = 1,2,3,4,5 >>> min(repeat(lambda: (s.a, s.b, s.c, s.d, s.e))) 0.2850431999977445 >>> min(repeat(lambda: (nt.a, nt.b, nt.c, nt.d, nt.e))) 0.418955763001577 I'm showing the namedtuple to be a little slower, but I think the thing to remember is to do the semantically correct thing first. If you've met your technical requirements, you're done. Only if you're too slow, then you look for ways to optimize for bottlenecks.
  - gshulegaard 9y ago
    I was just copying the benchmark I was quoting as a sanity check since I was just trying to validate references. I also feel compelled to point out that __slots__ changes the way objects are initialized [1]: > By default, instances of both old and new-style classes have a dictionary for attribute storage. This wastes space for objects having very few instance variables. The space consumption can become acute when creating large numbers of instances. > > The default can be overridden by defining __slots__ in a new-style class definition. The __slots__ declaration takes a sequence of instance variables and reserves just enough space in each instance to hold a value for each variable. Space is saved because __dict__ is not created for each instance. Which means that, in actuality, simply defining __slots__ changes the nature of the object. Quick trial demonstrates this: >>> class mySlottedObj(object): ... __slots__ = ('a', 'b') ... c = 1 ... >>> x = mySlottedObj() >>> x <mySlottedObj object at 0x10e6a3510> >>> x.c 1 >>> x.__dict__ Traceback (most recent call last): File "<input>", line 1, in <module> AttributeError: 'mySlottedObj' object has no attribute '__dict__' >>> x.__weakref__ Traceback (most recent call last): File "<input>", line 1, in <module> AttributeError: 'mySlottedObj' object has no attribute '__weakref__' >>> x.a = 1 >>> x.c = 2 Traceback (most recent call last): File "<input>", line 1, in <module> AttributeError: 'mySlottedObj' object attribute 'c' is read-only The benefit of namedtuples, in the context of Python is reducing the object footprint to being no greater than a tuple. The performance hit, which I am trying to demonstrate, is in lookup and is shown in the source (linked above). _field_template = '''\ {name} = _property(_itemgetter({index:d}), doc='Alias for field number {index:d}') ''' As I understand it this means when you access using __getitem__, a named tuple first maps to a property that then calls a __getitem__ using the aliased index on the base `tuple` object that `namedtuple` is storing. But at any rate, the benchmark script was not mine and refining the test to be as close to the same code path is welcome. And I wholeheartedly agree: > Only if you're too slow, then you look for ways to optimize for bottlenecks. I was merely responding to the original assertion: > This provides the same performance benefits, and is very similar in a lot of ways to a Scala case class -- aside from removing the boilerplate of stating each attribute twice in __init__ and again in __slots__, And I think at this point it is quite clear that namedtuple has a bit more caveats than just "removing the boilerplate" of __slots__.

by gipp 9y ago

gshulegaard 9y ago

The question was asked in 2010, but the benchmark post I was referencing is from 2014. It doesn't appear as if `namedtuple` has changed much: https://hg.python.org/cpython/file/tip/Lib/collections/__init__.py#l302 https://hg.python.org/cpython/file/tip/Lib/collections/__ini... Running the same tests from the post, I get similar results: from collections import namedtuple STest = namedtuple("TEST", "a b c") a = STest(a=1,b=2,c=3) class Test(object): __slots__ = ["a","b","c"] a=1 b=2 c=3 b = Test() c = {'a':1, 'b':2, 'c':3} d = (1,2,3) e = [1,2,3] f = (1,2,3) g = [1,2,3] key = 2 if __name__ == '__main__': from timeit import timeit print("Named tuple with a, b, c:") print(timeit("z = a.c", "from __main__ import a")) print("Named tuple, using index:") print(timeit("z = a[2]", "from __main__ import a")) print("Class using __slots__, with a, b, c:") print(timeit("z = b.c", "from __main__ import b")) print("Dictionary with keys a, b, c:") print(timeit("z = c['c']", "from __main__ import c")) print("Tuple with three values, using a constant key:") print(timeit("z = d[2]", "from __main__ import d")) print("List with three values, using a constant key:") print(timeit("z = e[2]", "from __main__ import e")) print("Tuple with three values, using a local key:") print(timeit("z = d[key]", "from __main__ import d, key")) print("List with three values, using a local key:") print(timeit("z = e[key]", "from __main__ import e, key")) Output: Named tuple with a, b, c: 0.147349834442 Named tuple, using index: 0.0364408493042 Class using __slots__, with a, b, c: 0.0334758758545 Dictionary with keys a, b, c: 0.0339260101318 Tuple with three values, using a constant key: 0.0425179004669 List with three values, using a constant key: 0.0301761627197 Tuple with three values, using a local key: 0.0436310768127 List with three values, using a local key: 0.0309669971466

aaronchall 9y ago
I just noticed your slotted object uses class variables. That is completely wrong - so your analysis is probably wrong. In Python, I think it's important to do what's directly semantically correct. However, we can compare apples and oranges on what they have in common, so here's a bit of my own analysis: >>> from timeit import repeat >>> from collections import namedtuple >>> class Slotted: __slots__ = 'a', 'b', 'c', 'd', 'e' ... >>> NT = namedtuple('NT', 'a b c d e') >>> >>> nt = NT(1,2,3,4,5) >>> s = Slotted() >>> s.a, s.b, s.c, s.d, s.e = 1,2,3,4,5 >>> min(repeat(lambda: (s.a, s.b, s.c, s.d, s.e))) 0.2850431999977445 >>> min(repeat(lambda: (nt.a, nt.b, nt.c, nt.d, nt.e))) 0.418955763001577 I'm showing the namedtuple to be a little slower, but I think the thing to remember is to do the semantically correct thing first. If you've met your technical requirements, you're done. Only if you're too slow, then you look for ways to optimize for bottlenecks.
- gshulegaard 9y ago
  I was just copying the benchmark I was quoting as a sanity check since I was just trying to validate references. I also feel compelled to point out that __slots__ changes the way objects are initialized [1]: > By default, instances of both old and new-style classes have a dictionary for attribute storage. This wastes space for objects having very few instance variables. The space consumption can become acute when creating large numbers of instances. > > The default can be overridden by defining __slots__ in a new-style class definition. The __slots__ declaration takes a sequence of instance variables and reserves just enough space in each instance to hold a value for each variable. Space is saved because __dict__ is not created for each instance. Which means that, in actuality, simply defining __slots__ changes the nature of the object. Quick trial demonstrates this: >>> class mySlottedObj(object): ... __slots__ = ('a', 'b') ... c = 1 ... >>> x = mySlottedObj() >>> x <mySlottedObj object at 0x10e6a3510> >>> x.c 1 >>> x.__dict__ Traceback (most recent call last): File "<input>", line 1, in <module> AttributeError: 'mySlottedObj' object has no attribute '__dict__' >>> x.__weakref__ Traceback (most recent call last): File "<input>", line 1, in <module> AttributeError: 'mySlottedObj' object has no attribute '__weakref__' >>> x.a = 1 >>> x.c = 2 Traceback (most recent call last): File "<input>", line 1, in <module> AttributeError: 'mySlottedObj' object attribute 'c' is read-only The benefit of namedtuples, in the context of Python is reducing the object footprint to being no greater than a tuple. The performance hit, which I am trying to demonstrate, is in lookup and is shown in the source (linked above). _field_template = '''\ {name} = _property(_itemgetter({index:d}), doc='Alias for field number {index:d}') ''' As I understand it this means when you access using __getitem__, a named tuple first maps to a property that then calls a __getitem__ using the aliased index on the base `tuple` object that `namedtuple` is storing. But at any rate, the benchmark script was not mine and refining the test to be as close to the same code path is welcome. And I wholeheartedly agree: > Only if you're too slow, then you look for ways to optimize for bottlenecks. I was merely responding to the original assertion: > This provides the same performance benefits, and is very similar in a lot of ways to a Scala case class -- aside from removing the boilerplate of stating each attribute twice in __init__ and again in __slots__, And I think at this point it is quite clear that namedtuple has a bit more caveats than just "removing the boilerplate" of __slots__.