5 ms·

The dict shrinkage(25% less!) in memory is pretty sweet! I'm not sure how many of you abuse dicts to hold [just about] everything, but this is a big boost for

The dict shrinkage(25% less!) in memory is pretty sweet! I'm not sure how many of you abuse dicts to hold [just about] everything, but this is a big boost for those of us that do.

virgil_disgr4ce 10y ago
"Dict shrinkage"
Alex3917 10y ago
I've been trying to break this habit and get into using namedtuples, but the problem is I still need to look up the API every time I need to use them.
- erokar 10y ago
  Besides immutability, what are the advantages of using namedtuples instead of dicts?
  - burntsushi 10y ago
    - A clear single point of truth describing your data. - Probably more memory efficient.
    - i80and 10y ago
      We can test memory usage in CPython (3.5)! >>> import collections >>> import sys >>> class A: ... def __init__(self, x, y): ... self.x = x ... self.y = y ... >>> class B: ... __slots__ = ('x', 'y') ... def __init__(self, x, y): ... self.x = x ... self.y = y ... >>> C = collections.namedtuple('C', ('x', 'y')) >>> a = A(1, 1) >>> b = B(1, 1) >>> c = C(1, 1) >>> d = {'x': 1, 'y': 1} >>> sys.getsizeof(a) + sys.getsizeof(a.__dict__) # Plain object 344 >>> sys.getsizeof(b) + sys.getsizeof(b.__slots__) # Plain object with slots 120 >>> sys.getsizeof(c) # Named Tuple 64 >>> sys.getsizeof(d) # Dict 288 EDIT: Fixed a slew of measurement issues EDIT2: Included size of __slots__ tuple
      - SEJeff 10y ago
        If you're using Python 3.5, use tracemalloc, it is much nicer for this sort of thing. https://docs.python.org/3/library/tracemalloc.html https://docs.python.org/3/library/tracemalloc.html
    - lqdc13 10y ago
      It is an order of magnitude slower and not less memory efficient. Also, for me it is never easier to understand what is going on when named tuples are used instead of dicts Proof: def make_dicts(): res = [] for i in range(1000000): res.append({'x': i, 'y': i}) return res def make_tuples(): templ = namedtuple('Point', ('x', 'y')) res = [] for i in range(1000000): res.append(templ(i, i)) return res from ipython_memory_usage import ipython_memory_usage as imu imu.start_watching_memory() used 0.2305 MiB RAM in 4.00s, peaked 0.00 MiB above current, total RAM usage 86.41 MiB %time tuples = make_tuples() CPU times: user 780 ms, sys: 32 ms, total: 812 ms Wall time: 816 ms used 109.0625 MiB RAM in 0.92s, peaked 0.00 MiB above current, total RAM usage 195.47 MiB %time dicts = make_dicts() CPU times: user 160 ms, sys: 32 ms, total: 192 ms Wall time: 194 ms used 99.5117 MiB RAM in 0.30s, peaked 0.00 MiB above current, total RAM usage 524.33 MiB
      - burntsushi 10y ago
        Your sibling got opposite results. I haven't dissected either program, but I don't see any obvious flaws. I find your results particularly surprising. Can you explain them? Are there optimisations going on in your benchmark that you haven't accounted for? It should not be controversial that namedtuples are clearer for the reason I stated in my initial comment.
        lqdc13 10y ago
        sys.getsizeof() on a code object returns the size of the code struct, not all the other objects it references. To measure real usage, you have to use another method. I use ipython_memory_usage because it's easy to use. You can test this out easily by making a large dict and see how much your OS thinks python is using vs what sys.getsizeof is reporting.
  - mixmastamyk 10y ago
    - Dot syntax: foo.length rather than foo['length']
  - lgunsch 10y ago
    I gives a name and expected properties to common data structures, or unspoken unnamed types. Many times when a dict is used, it's expected to have certain attributes, and may have expected ways of interacting with those attributes. Named tuples is a simple way to formalize these, which makes them not so mysterious to you future-you, or to other developers who inherit the project 6 months down the road.
- mixmastamyk 10y ago
  When something is moderately awkward I often make a "snippet" in my editor, e.g.: pr<TAB> --> print('%cursor%', %cursor%) nt<TAB> --> namedtuple('%cursor%', ['%cursor%', …])
- deniska 10y ago
  Be careful with named tuples: >>> from collections import namedtuple >>> Time = namedtuple('Time', ['hours', 'minutes']) >>> Point = namedtuple('Point', ['x', 'y']) >>> t = Time(hours=7, minutes=40) >>> p = Point(x=7, y=40) >>> p == t True They're still tuples. They're meant more as a tool for making an already existing tuple mess slightly more managable rather than creating a new mess. For making plain data objects take less boilerplate to create have a look at attrs: https://attrs.readthedocs.io https://attrs.readthedocs.io
  - deleted 10y ago
    [deleted]
  - SEJeff 10y ago
    Strong +1 for attrs. It is a really nice library and quite easy to use.
    - tedmiston 10y ago
      Interesting find with attrs. My main use case for namedtuples is a class as a data container which doesn't require any special methods. This might be a better alternative.
  - thijsvandien 10y ago
    Ugh, that's terrible... Thanks for pointing it out!
- kchoudhu 10y ago
  Classes work for this as well, I've found. I use a lot of classes. It feels like Java sometimes, but it does make debugging easier...
  - CardenB 10y ago
    In my experience, namedtuples have significantly less overhead than classes.
    - kchoudhu 10y ago
      Yes for the increased overhead, but I find that the fact that they're tuples underneath results in odd semantics (I think the comment above mine[1] does a decent job pointing out why I am leary of using them. Tradeoffs, I guess. [1] https://news.ycombinator.com/item?id=13124501 https://news.ycombinator.com/item?id=13124501
    - skrause 10y ago
      I doubt that because defining a namedtuple just creates a new class internally: https://github.com/python/cpython/blob/master/Lib/collections/__init__.py#L304 https://github.com/python/cpython/blob/master/Lib/collection... Class creation the very expensive in Python, so Guido van Rossum explicitly recommends against using dynamically created namedtuples: https://mail.python.org/pipermail//python-ideas/2016-April/039867.html https://mail.python.org/pipermail//python-ideas/2016-April/0...
      - tcbawo 10y ago
        I believe that objects of namedtuple have less runtime overhead than a class. Guido seems to point out that the process of instantiating a namedtuple class is expensive.
    - benhoyt 10y ago
      Normally, yes. But if you define __slots__ on the class, it's the same overhead as a namedtuple (no __dict__ per instance).
halflings 10y ago
I'd expect it to also improve things in regular code since class attributes are saved in __dict__, unless you use __slots__. I might be wrong though (remember seeing that talk by Heroku about how objects are treated differently even in CPython).
timehastoldme 10y ago
Wow, the Python community sure is impressed with its small dicts. Have they tried pooling?

The dict shrinkage(25% less!) in memory is pretty sweet! I'm not sure how many of you abuse dicts to hold [just about] everything, but this is a big boost for those of us that do.

virgil_disgr4ce 10y ago

"Dict shrinkage"

Alex3917 10y ago

I've been trying to break this habit and get into using namedtuples, but the problem is I still need to look up the API every time I need to use them.

erokar 10y ago
Besides immutability, what are the advantages of using namedtuples instead of dicts?
- burntsushi 10y ago
  - A clear single point of truth describing your data. - Probably more memory efficient.
  - i80and 10y ago
    We can test memory usage in CPython (3.5)! >>> import collections >>> import sys >>> class A: ... def __init__(self, x, y): ... self.x = x ... self.y = y ... >>> class B: ... __slots__ = ('x', 'y') ... def __init__(self, x, y): ... self.x = x ... self.y = y ... >>> C = collections.namedtuple('C', ('x', 'y')) >>> a = A(1, 1) >>> b = B(1, 1) >>> c = C(1, 1) >>> d = {'x': 1, 'y': 1} >>> sys.getsizeof(a) + sys.getsizeof(a.__dict__) # Plain object 344 >>> sys.getsizeof(b) + sys.getsizeof(b.__slots__) # Plain object with slots 120 >>> sys.getsizeof(c) # Named Tuple 64 >>> sys.getsizeof(d) # Dict 288 EDIT: Fixed a slew of measurement issues EDIT2: Included size of __slots__ tuple
    - SEJeff 10y ago
      If you're using Python 3.5, use tracemalloc, it is much nicer for this sort of thing. https://docs.python.org/3/library/tracemalloc.html https://docs.python.org/3/library/tracemalloc.html
  - lqdc13 10y ago
    It is an order of magnitude slower and not less memory efficient. Also, for me it is never easier to understand what is going on when named tuples are used instead of dicts Proof: def make_dicts(): res = [] for i in range(1000000): res.append({'x': i, 'y': i}) return res def make_tuples(): templ = namedtuple('Point', ('x', 'y')) res = [] for i in range(1000000): res.append(templ(i, i)) return res from ipython_memory_usage import ipython_memory_usage as imu imu.start_watching_memory() used 0.2305 MiB RAM in 4.00s, peaked 0.00 MiB above current, total RAM usage 86.41 MiB %time tuples = make_tuples() CPU times: user 780 ms, sys: 32 ms, total: 812 ms Wall time: 816 ms used 109.0625 MiB RAM in 0.92s, peaked 0.00 MiB above current, total RAM usage 195.47 MiB %time dicts = make_dicts() CPU times: user 160 ms, sys: 32 ms, total: 192 ms Wall time: 194 ms used 99.5117 MiB RAM in 0.30s, peaked 0.00 MiB above current, total RAM usage 524.33 MiB
    - burntsushi 10y ago
      Your sibling got opposite results. I haven't dissected either program, but I don't see any obvious flaws. I find your results particularly surprising. Can you explain them? Are there optimisations going on in your benchmark that you haven't accounted for? It should not be controversial that namedtuples are clearer for the reason I stated in my initial comment.
      - lqdc13 10y ago
        sys.getsizeof() on a code object returns the size of the code struct, not all the other objects it references. To measure real usage, you have to use another method. I use ipython_memory_usage because it's easy to use. You can test this out easily by making a large dict and see how much your OS thinks python is using vs what sys.getsizeof is reporting.
- mixmastamyk 10y ago
  - Dot syntax: foo.length rather than foo['length']
- lgunsch 10y ago
  I gives a name and expected properties to common data structures, or unspoken unnamed types. Many times when a dict is used, it's expected to have certain attributes, and may have expected ways of interacting with those attributes. Named tuples is a simple way to formalize these, which makes them not so mysterious to you future-you, or to other developers who inherit the project 6 months down the road.
mixmastamyk 10y ago
When something is moderately awkward I often make a "snippet" in my editor, e.g.: pr<TAB> --> print('%cursor%', %cursor%) nt<TAB> --> namedtuple('%cursor%', ['%cursor%', …])
deniska 10y ago
Be careful with named tuples: >>> from collections import namedtuple >>> Time = namedtuple('Time', ['hours', 'minutes']) >>> Point = namedtuple('Point', ['x', 'y']) >>> t = Time(hours=7, minutes=40) >>> p = Point(x=7, y=40) >>> p == t True They're still tuples. They're meant more as a tool for making an already existing tuple mess slightly more managable rather than creating a new mess. For making plain data objects take less boilerplate to create have a look at attrs: https://attrs.readthedocs.io https://attrs.readthedocs.io
- deleted 10y ago
  [deleted]
- SEJeff 10y ago
  Strong +1 for attrs. It is a really nice library and quite easy to use.
  - tedmiston 10y ago
    Interesting find with attrs. My main use case for namedtuples is a class as a data container which doesn't require any special methods. This might be a better alternative.
- thijsvandien 10y ago
  Ugh, that's terrible... Thanks for pointing it out!
kchoudhu 10y ago
Classes work for this as well, I've found. I use a lot of classes. It feels like Java sometimes, but it does make debugging easier...
- CardenB 10y ago
  In my experience, namedtuples have significantly less overhead than classes.
  - kchoudhu 10y ago
    Yes for the increased overhead, but I find that the fact that they're tuples underneath results in odd semantics (I think the comment above mine[1] does a decent job pointing out why I am leary of using them. Tradeoffs, I guess. [1] https://news.ycombinator.com/item?id=13124501 https://news.ycombinator.com/item?id=13124501
  - skrause 10y ago
    I doubt that because defining a namedtuple just creates a new class internally: https://github.com/python/cpython/blob/master/Lib/collections/__init__.py#L304 https://github.com/python/cpython/blob/master/Lib/collection... Class creation the very expensive in Python, so Guido van Rossum explicitly recommends against using dynamically created namedtuples: https://mail.python.org/pipermail//python-ideas/2016-April/039867.html https://mail.python.org/pipermail//python-ideas/2016-April/0...
    - tcbawo 10y ago
      I believe that objects of namedtuple have less runtime overhead than a class. Guido seems to point out that the process of instantiating a namedtuple class is expensive.
  - benhoyt 10y ago
    Normally, yes. But if you define __slots__ on the class, it's the same overhead as a namedtuple (no __dict__ per instance).

halflings 10y ago

I'd expect it to also improve things in regular code since class attributes are saved in __dict__, unless you use __slots__. I might be wrong though (remember seeing that talk by Heroku about how objects are treated differently even in CPython).

timehastoldme 10y ago

Wow, the Python community sure is impressed with its small dicts. Have they tried pooling?