7 ms·
The worst part of python is the lack of utility functions for munging collections. But it sits at a slightly higher level than this - things that are idiomatic
by zmmmmm 5y ago
The worst part of python is the lack of utility functions for munging collections. But it sits at a slightly higher level than this - things that are idiomatic in other scripting languages like groupBy(fn), sortBy(fn), countBy(fn), collate, are all inexplicably like IKEA self assembled furniture in Python instead of first class entities. It makes lots of routine data munging and algorithmic work much more annoying than it needs to be (comparing to Groovy, Ruby, etc).
- BurningFrog 5y agoIt will take Python 10 more years to allow a flatten()...
- staticautomatic 5y agomore_itertools ftw
- masklinn 5y ago10 years is very optimistic: flatten has little use without a functional pipeline (something like clojure’s threading macro) as in other contexts it’s a trivial comprehension.
- joshuamorton 5y agoDo you need the recursive form? Because a single flatten is itertools.chain.from_iterable
- nerdponx 5y agoI have written this in serious code before, and I don't regret it: from itertools import chain flatten = chain.from_iterable def lflatten(x): return list(flatten(x)) def sflatten(x): return set(flatten(x))
- BurningFrog 5y agoI know it exists, but the universally known name is 80% of the utility of having a standard flatten() function. I could tolerate a slightly different name, but `itertools.chain.from_iterable` is absurd to me.
- gsinclair 5y agoAnd it's simply amazing that functools doesn't fill this gap.
- smallnamespace 5y agoNot sure what collate is supposed to be, but these idioms are hardly difficult if you're used to Python: - groupBy is itertools.groupBy(lst, fn) - sortBy is just lst.sort(key=fn) - countBy is collections.Counter(map(fn, lst)) - Sibling comment mentioned flatten, which is just [item for sublist for sublist in lst] More esoteric needs are usually met by itertools.
- masklinn 5y ago> - sortBy is just lst.sort(key=fn) You mean sorted(). list.sort only works on list and in-place.
- gfaure 5y agoAnnoyingly, itertools is one of those packages in the Python standard library with wordsruntogether-named functions, so it's really known as itertools.groupby instead.
- bobbylarrybobby 5y agoThe real issue is that python doesn’t really make it easy to define anonymous functions. If you want one with more than one line, you need to name it and move its definition outside the point of use. Quite annoying.
- diarrhea 5y agoI always thought of that as a feature.
- mrweasel 5y agoExcessive usage of anonymous functions is a major pain. JavaScript developers for some reason almost refuse to name their functions, even though it would make reading and reusing code much easier. You can make lambdas in Python, but naming function is in my mind very much a "feature".
- 5y ago
- goodside 5y agoI think the motivation there historically was to encourage use of generators and generator expressions — it’s definitely intentional, because things like `apply` and `reduce` used to be builtins but now they’re imports. If you reach for itertools imports often in an interactive REPL, you might be interested in Pyflyby: https://labs.quansight.org/blog/2021/07/pyflyby-improving-efficiency-of-jupyter-interactive-sessions/ https://labs.quansight.org/blog/2021/07/pyflyby-improving-ef...
- steve_gh 5y agoYes - the description of writing Python as like assembling Ikea furniture is absolutely spot-on. Yes, you can do it, and the results can be nice, but by God it is sometimes such a pain. The comment showing where groipBy, sortBy etc can be found just shows the problem - they are all in different libraries.That's just plain annoying! And don't get me started on the pain of trying to build an Ordered Dictionary with a default initial value!
- mixmastamyk 5y agoLike defaultdict? All dicts have been ordered since 3.6.
- theelous3 5y agoSmall note: they've been sorted as an implementation detail of cpython since 3.6, but as language spec since 3.7.
- robertlagrant 5y ago> And don't get me started on the pain of trying to build an Ordered Dictionary with a default initial value! >>> from collections import defaultdict >>> from datetime import datetime >>> >>> d = defaultdict(datetime.now) >>> d[1], d[2], d[3], d[4], d[0] (datetime.datetime(2021, 7, 9, 15, 50, 52, 87605), datetime.datetime(2021, 7, 9, 15, 50, 52, 87613), datetime.datetime(2021, 7, 9, 15, 50, 52, 87614), datetime.datetime(2021, 7, 9, 15, 50, 52, 87615), datetime.datetime(2021, 7, 9, 15, 50, 52, 87616)) >>> for k,v in d.items(): ... print(k,v) ... 1 2021-07-09 15:50:52.087605 2 2021-07-09 15:50:52.087613 3 2021-07-09 15:50:52.087614 4 2021-07-09 15:50:52.087615 0 2021-07-09 15:50:52.087616 >>> Seems pretty good to me?
- roenxi 5y ago> ...like groupBy(fn), sortBy(fn), countBy(fn), collate... What leaps out at me is that these 3 functions are all straight out of relational algebra style worldview. Python the language doesn't support relational algebra as a first class concept. The reason it feels like IKEA self assembly is probably because you are implicitly implementing a data model that isn't how Python thinks about collections.
- quietbritishjim 5y agoExactly. Pythonic versions: groupBy(fn): {fn(x): x for x in foo} sortBy(fn): sorted(foo, key=fn) countBy(fn): Counter(fn(x) for x in foo) flatten: [x for y in foo for x in y] filter(fn): [x for x in foo if fn(x)] All the "for x in blah" can seem like a lot of boilerplate for a non-Pythonista, but actually it becomes subconscious once you're used to it and actually helps you feel the structure (a bit like indentation isn't necessary in C but it still helps you to see it). For compound operations (e.g. merge two lists, filter and flatten), I find the code to be a lot more easier to "feel" than if you'd combined several functional style functions, where you have to read the function names instead of just seeing the structure.
- unishark 5y agoSpeaking of the list and set comprehensions, they also mimic formal mathematical notation e.g. {f(x) | x>0, x in reals} or whatever
- cwilkes 5y agoThat groupBy doesn’t look like it does what you want to do. For example if “def fn(i): return i//2” You probably want to get an output like {0: [0,1]} but you’ll only get a scalar for the value (1) and it will be the last one evaluated. It would be nice if there was a way to do this on one line but you’ll have to do ret=collections.defaultdict(list) For i In foo: ret[fn(i)].append(i) Return ret I suppose it isn’t that bad as it will be in the groupBy method. Or you could add it to a method on a list. I think that’s easier to do with Rust.
- 5y ago
- emmelaich 5y agoSome of those are given as examples in the functools / itertools doc pages.
- publicola1990 5y agoAlso many python std library collection methods modify their inputs.
- hcrisp 5y agoIt's a package, not a built in, but pytoolz [0] is a very complete solution to the type of functional programming munging you are taking about. I wish more people knew about it (no one here seems to have mentioned it). And it has been Cython optimized in another package called cytoolz [1]. The author explains in the Heritage section how it is an extension of and plays nice with itertools and functools [2]. For Python coders new to functional programming, and how it can make working with data easier, I highly recommend reading the following sections of the pytoolz docs: Composability, Function Purity, Laziness, and Control Flow [0]. [0] https://toolz.readthedocs.io/en/latest/ https://toolz.readthedocs.io/en/latest/ [1] https://github.com/pytoolz/cytoolz https://github.com/pytoolz/cytoolz [2] https://toolz.readthedocs.io/en/latest/heritage.html https://toolz.readthedocs.io/en/latest/heritage.html