6 ms·
>Rows are represented as standard Python dictionaries That seems inefficient. You're always going to have the same columns in each row, so storing column names
by Grue3 11y ago
>Rows are represented as standard Python dictionaries
That seems inefficient. You're always going to have the same columns in each row, so storing column names for each row is a huge waste of memory. Django's "cursor.fetchall" returns a list of tuples, which seems to be a saner way to represent db rows in Python.
- r0muald 11y agoNamed tuples seem to offer the best of both worlds, as they're immutable.
- cwmma 11y agoprobably so you don't have to remember what order they were returned in
- siddhant 11y agoI'm also a fan of NamedTupleCursor. Not sure how NamedTuples compare to raw tuples in terms of memory consumption, but they make code look a lot nicer.
- stuartaxelowen 11y agoThey're essentially the same footprint IIRC.
- glifchits 11y agoThis memory waste argument is only true if you want to fetch all rows, which you could possibly avoid with different techniques. Otherwise, a dictionary seems like a very simple, convenient, Pythonic, and intuitive representation of a row. This lib also looks brand new, so there is plenty of time for the community to optimize the hell out of it.
- nathancahill 11y agoI also "voted" for NamedTuples https://twitter.com/nathancahill/status/696146948567359488 https://twitter.com/nathancahill/status/696146948567359488
- nvader 11y agoOne argument I've seen for using dictionaries is that namedtuples restrict you to column names that are valid Python identifiers, which is a smaller set than postgres allows for column names (postgres allows the '$' and non-Latin characters).
- hobarrera 11y agoSo does python: >>> d = {'ñanáöß$': 1} >>> d {'ñanáöß$': 1} >>> d.keys() dict_keys(['ñanáöß$']) >>>
- kilink 11y agoThe parent was talking about namedtuple. >>> collections.namedtuple("test", "$") Traceback (most recent call last): ... ValueError: Type names and field names can only contain alphanumeric characters and underscores: '$'
- hobarrera 11y agoThat's because attribute names can't be called "$", so `sometuple.$` would be a syntax violation.
- EmanueleAina 11y agoI guess everyone is aware of that, thanks, The parent comments were discussing why choosing dicts over namedtuples for db-originated columns, and the fact that field names are restricted on namedtuples but not on dicts is a very compelling point. Making examples usind dicts and explaining why namedtuples have restrictions is completely missing the point.
- pyre 11y agoHN supports "<pre><code>" blocks in a similar manor to Markdown, but using a two-space prefix (rather than Markdown's 4-space prefix): >>> d = {'ñanáöß$': 1} d {'ñanáöß$': 1} >>> d.keys() dict_keys(['ñanáöß$'])
- glifchits 11y agoReitz has a public Facebook post on this. Here are his comments: Harry Percival Looks nice! Y u no namedtuple? Kenneth Reitz Dictionaries are a much more predictable interface ;) Kenneth Reitz Just because you can do something, doesn't mean you should!
- kenneth_reitz 11y agoWell, that's creepy.
- mattdeboard 11y agoI learned from a coworker recently that namedtuple is a much lower memory usage than a dictionary, so if you're dealing with lots and lots of data regularly, and lots of processes on a single machine, (i.e. memory is a concern) namedtuple might be a better choice.
- kilink 11y agoIf this were a big deal, it could be optimized using something like a key-sharing dict implementation [1]. You can keep the dict interface to accommodate column names that aren't valid identifiers, and avoid most of the memory overhead. [1] https://www.python.org/dev/peps/pep-0412/ https://www.python.org/dev/peps/pep-0412/
- jstimpfle 11y agopython-sqlite3's RowFactory does this as well.
- bjt 11y agoI've made a pull request implementing namedtuple support at https://github.com/kennethreitz/records/pull/6 https://github.com/kennethreitz/records/pull/6.
- tcoder 11y agoNamedtuple CPU-speed performance for some common operations is TERRIBLE compared to dictionaries, at least on 2.7. Orders of magnitude difference. So bad that it really matters. It's shocking how bad the implementors got this.
- dangoldin 11y agoCurious what the operations you're talking about are. Get?
- pyre 11y agoIf you look at the the StackOverflow in my sibling comment to yours, running 'obj.attrname' on a namedtuple takes longer because it needs to translate 'attrname' to an integer index and then run (e.g.) 'obj[0]'. Outside of that, I'm not sure.
- tveita 11y agoPickling, for one. >>> nt = namedtuple('nt', 'a b') >>> nt10k = [nt(1, 2) for i in range(10000)] >>> dict10k = [{'a':1, 'b':2} for i in range(10000)] >>> timeit pickle.dumps(nt10k) 10 loops, best of 3: 26.3 ms per loop >>> timeit pickle.dumps(dict10k) 100 loops, best of 3: 2.49 ms per loop
- pyre 11y agonamedtuple is just a thin wrapper around tuple. It literally constructs a class definition that sub-classes tuple as a string and then executes it. You can see the string template that it uses here[1]. If you're interested in something like namedtuple there are other[2] things[3] you can use depending on your use-case. [1] https://github.com/python/cpython/blob/master/Lib/collections/__init__.py#L304 https://github.com/python/cpython/blob/master/Lib/collection... [2] http://stackoverflow.com/a/2648186 http://stackoverflow.com/a/2648186 [3] https://pypi.python.org/pypi/frozendict/ https://pypi.python.org/pypi/frozendict/
- Redoubts 11y agoIt's really weird that immutable tuples take a performance hit, compared to mutable lists (almost 50% on access!?!).