3 ms·
The source code for that function (groupby in toolz) is bizarre! Creating a defaultdict where the entries are append-to-list functions, calling them, then going
by joppy 6y ago
The source code for that function (groupby in toolz) is bizarre! Creating a defaultdict where the entries are append-to-list functions, calling them, then going over the dictionary again to extract the underling list objects. Does anyone know what this pattern is for, and why one wouldn’t just create a defaultdict(list)?
The function: https://toolz.readthedocs.io/en/latest/_modules/toolz/itertoolz.html#groupby https://toolz.readthedocs.io/en/latest/_modules/toolz/iterto...
- pdonis 6y ago> why one wouldn’t just create a defaultdict(list)? One shouldn't even do that. groupby is supposed to assume that the input is already sorted by the given key, so it can be implemented as a generator. What's more, it should be implemented as a generator (that's the way the Python stlib's itertools.groupby does it), to avoid having to realize the entire iterable at once.
- ciupicri 6y agoThe Toolz version has a different purpose than the one from Python's stdlib (itertools). One is unsorted input, while the other one is for sorted input. The Toolz version is not a replacement and the documentation states > Not to be confused with ``itertools.groupby``
- ciupicri 6y agoIt's the avoiding dots pattern [1]. The idea is that referencing the `append` method in the for grouping loop, i.e. doing d[key(item)].append(item) takes more time than rebuilding the _rv_ groups dictionary with lists instead of `append` methods. Of course, a benchmark should be run and see how much longer the input sequence needs to be than the resulting groups, for this to happen. [1]: https://wiki.python.org/moin/PythonSpeed/PerformanceTips#Avoiding_dots https://wiki.python.org/moin/PythonSpeed/PerformanceTips#Avo......