12 ms·
Show HN: Python library for functional programming
- merlincorey 9y ago> Prepare for 1.0 next release So this will soon be a stable API? Great work. The documentation is extensive. I am personally a big fan of toolz[0], which is clojure inspired. This object method chaining Scala inspired approach is different, and interesting. It might be fun to make sort of comparison matrix of the difference in approach between this library, toolz, and the venerable fn[1] which also has a Scala influence (and apparently already implements the '_' lambda form). [0] http://toolz.readthedocs.io/en/latest/ http://toolz.readthedocs.io/en/latest/ [1] https://github.com/kachayev/fn.py https://github.com/kachayev/fn.py
- emmelaich 9y ago... and functools/itertools for completeness?
- catnaroek 9y agoFrom https://github.com/kachayev/fn.py https://github.com/kachayev/fn.py : > There are theoretical and practical advantages to the functional style: > - Formal provability This is bollocks. To prove things about your programs, you need a formal semantics for the language you're using.
- bionoid 9y ago> This [Formal provability] is bollocks. How precisely does modularity, composability and ease of debugging improve from this module? Not at all. It may have benefits, sure. But to claim these as general attributes is just sophistry in my personal opinion.
- catnaroek 9y agoAgreed. I just felt a stronger need to comment on the zeroth bullet point in the list.
- lmm 9y agoFunctional style makes it easier to have a natural semantics that corresponds directly to the language proper: you can use a small-step semantics where evaluation is just reduction, and you then have a very simple and natural denotational semantics (the denotation of an expression is just the value it evaluates to) that obviously corresponds directly to your operational semantics. All that is much harder to do with programming styles that make use of pervasive, implicit state.
- catnaroek 9y agoA semantics is a property a programming language, not the style you happen to program in. --- @lmm > In practice no practical language provides formal semantics for the entire language Programming languages are mathematical objects and always have a formal semantics, regardless of whether someone has taken the trouble to write it down or not. What I'm saying is - the formal semantics that most languages have preclude the existence of compound values. Without those, you can't define interesting functions, and thus you can't program in a functional style.
- lmm 9y agoIn practice no practical language provides formal semantics for the entire language, only for restricted subsets of it i.e. particular styles within the language.
- lmm 9y agoResponding to edits: > Programming languages are mathematical objects and always have a formal semantics, regardless of whether someone has taken the trouble to write it down or not. Not true for the usual plain-english meaning of "programming language". Plenty of things we call programming languages take a "the implementation is the spec" view, or simply can't be assigned any useful semantics in a way that's at all aligned with the defining characteristics of the language in question. Even in, say, Haskell, io explicitly lacks formal semantics. > What I'm saying is - the formal semantics that most languages have preclude the existence of compound values. Without those, you can't define interesting functions, and thus you can't program in a functional style. Python tuples are perfectly cromulent compound values. Python has a bunch of builtin classes for which the equality operators that are available in the language are all kind of useless, sure, but that's true in almost all languages.
- entilzha 9y agoAuthor here. 1.0 has been out for a while now, and I'm very committed to API stability. 1.1 (or 1.0.1) should be out as soon as I have some time with mostly minor fixes here and there. I hadn't considered doing a comparison matrix, but I haven't done a '_' lambda since it seems redundent with the one in fn.
- bryanwb 9y agoshameless plug, if you like this you may also like my library tryme which ports the Either and Maybe monads to Python http://tryme.readthedocs.io/en/latest/ http://tryme.readthedocs.io/en/latest/
- catnaroek 9y agoWithout exhaustiveness checks for all your case analyses, what exactly is the point?
- bryanwb 9y agoThe point is that you can indicate whether an operation was successful or not by wrapping that value w/ a Success or Failure, rather returning a boolean, status code, and some subset of information about the operation.
- catnaroek 9y agoAnd how is that useful in the absence of pattern matching? If you write: if foo.is_success(): ... else: # foo.is_failure() ... Then you're back to branching on booleans. And we haven't even addressed the “lack of exhaustiveness checks” objection.
- ztjio 9y agoThese libraries in languages like python remind me of when Windows 95 was new and a bunch of Win3.1 compatible clones of the win95 task bar appeared. People are often satisfied by the veneer of something better, because, they were told it was better but don't truly understand why. This is super common in the programming world, I'm afraid.
- catnaroek 9y agoAgreed. While algebraic data types are obviously a good idea, the best way to use Python is to accept it for what it is: a dynamically typed object-oriented language that's not very good at manipulating user-defined values (or even defining them to begin with), but has some pretty cool tricks of its own, like runtime metaprogramming with decorators and metaclasses.
- flor1s 9y agoThis is quite cool, though it seems similar to RxPy (https://github.com/ReactiveX/RxPY https://github.com/ReactiveX/RxPY)
- vog 9y agoCurious question: Is there also something like Mobx [1] for Python? I'd love to have a reactive state libraries like Mobx on server-side. Mobx is also reactive, but has a different focus: It is not so much avout the low-level event dispatching, but reacting on data changes, recalculating derived values (but only those affected, and being lazy on non-observed dependent values). And Mobx determines dependencies of calculated values automatically. Also, Vue [2] has Mobx functionality built-in, with an even better syntax, but closely tied to the framework. Alas, both are JavaScript and targeted at client-side / user interface. Is there something similar for Python? (See also my question on SO [3].) [1] https://mobx.js.org/ https://mobx.js.org/ [2] https://vuejs.org/ https://vuejs.org/ [3] https://stackoverflow.com/q/47808562/19163 https://stackoverflow.com/q/47808562/19163
- imiric 9y agoI'm not deeply familiar with Mobx, but there's a few Redux implementations for Python. See pydux[0] and aioredux[1]. Not the same thing, of course, since Mobx is much more sophisticated than Redux, but in my team we're currently experimenting with aioredux for state management of client data, which drives the React+Redux frontend. It's working surprisingly well so far, although the biggest hurdle is team familiarity with functional aspects of Redux, and general structuring and responsibility of reducers. The library is no longer maintained, but considering the simplicity of the implementation and of Redux itself, we felt it wasn't a show stopper since we could always take over maintenance ourselves. I'd also be interested to know of Mobx Python implementations. [0]: https://github.com/usrlocalben/pydux https://github.com/usrlocalben/pydux [1]: https://github.com/ariddell/aioredux https://github.com/ariddell/aioredux
- vog 9y agoThanks for the pointer, although that's not what I'm looking for. The fundamental difference between reactive state management (Mobx) and the Redux approach is that Mobx has some automatisms that are vital to what I'm looking for: - automatic recognition of actual value dependencies (i.e. minimal update chains) - automatic determination of the order of updates (so you don't have to think about and design your reducers along their dependencies ... because this is an annoying task which can and hence should be automated ... it is prone to inefficiencies, and sometimes errors, when done by hand.)
- cwyers 9y agoDefinitely approve of the name EntilZha for your GitHub user name. I was interested to see that coexisting with Pandas is supported, but the documentation seems either scant or unfinished, I can't tell which: https://github.com/EntilZha/PyFunctional/blob/master/examples/PyFunctional-pandas-tutorial.ipynb https://github.com/EntilZha/PyFunctional/blob/master/example...
- fredley 9y agoThere seems to be a lot of similar functionality (although in a very different paradigm) to Pandas, I'd be very interested to see performance benchmarks between the two on some common pipelines (csv -> filter -> aggregate -> SQL)
- entilzha 9y agoFor operations where vectorization pandas will be much faster since its based on numpy. That being said, there are things which are very awkard to do in pandas which are very easy in pyfunctional
- entilzha 9y agoThanks on name! Fair enough on docs being a bit scarce. The main intent is to being able to easily make a pandas dataframe a sequence of tuple/namedtuples, and convert a sequence of tuple/namedtuple to a pandas dataframes.
- rajaravivarma_r 9y agoPython should somewhat provide a way of chaining these iterators. Coming from Ruby background, I really miss the ability to chain iterators. And the backslashes shouldn't be mandatory for wrapping method chains.
- make3 9y agoIs this like what your talking about https://docs.python.org/3/library/itertools.html#itertools.chain https://docs.python.org/3/library/itertools.html#itertools.c... ?
- r41nbowdash 9y agoTrailing backslashes are so ugly
- abhsag24 9y agoIs it just me or this seems too much like ruby?
- wodenokoto 9y agoReminds me of dplyr. How well does this work with pandas?
- entilzha 9y agoData interchange between the two is relatively seeemless. seq(df) converts a pandas dataframe into a sequence of tuples/namedtuples. Calling .to_pandas(columns=cols) will convert a sequence of tuples/namedtuples to a dataframe with cols optionally being the column names
- andybak 9y agoSorry to digress - but the comment in the first code example: > # or if you don't like backslash continuation got me thinking. I've always felt backslash continuation is a Python wart - but using brackets in this way only feels like a minor improvement. It's nearly always this scenario that makes me want to break a line - i.e. splitting a long chained method call on a "." Now - starting a line of code with a "." is not valid Python as far as I'm aware and therefore allowing it would not introduce any ambiguity or significantly complicate Python's parsing rules. "If a line begins with whitespace followed by "." then consider it as a continuation of the previous line. Can anyone comment on whether this is a terrible idea or not?
- andreareina 9y agoThis is a peeve of mine, and one of the reasons that I find myself preferring to write javascript these days rather than python. Never would have thought I'd say that. I'm not up to snuff on my compiler theory, but I think at the least this would require upgrading to a two-token lookahead parser, and it might also upgrade the python grammar from context-free to context-sensitive. Accordingly, I wouldn't expect this to get implemented -- IIRC ease of parsing was/is one of the big design goals.
- adrianN 9y agoLine continuations could be treated by the Lexer.
- Doxin 9y agoIn python line continuations are a pre-processing step as far as I'm aware (together with inserting INDENT and DEDENT tokens). In the same vein you could just do a simple loop over the token list and remove the appropriate NEWLINE tokens before going to the actual parsing.
- yen223 9y agoWhitespace-sensitive blocks aren't easily parsed either.
- baldfat 9y ago
- yedpodtrzitko 9y agoI'm not into functional programming in Python, but pipetools[0] looks way more handy. [0] https://github.com/0101/pipetools https://github.com/0101/pipetools
- danpalmer 9y agoIt looks like theres not a huge amount of overlap. Pipetools is all about implementing a "clever" syntax for data flows, whereas PyFunctional looks like it's much more about providing a library of functions for manipulating data in functional ways. I would expect that PyFunctional could probably implement the pipe syntax if they wanted, as just a layer of syntactic sugar on top of their core, however that would break OR semantics in places so I'm not convinced that syntax is a good choice, as "clever" as it may be.
- g4k 9y agoAlso Coconut looks interesting: http://coconut-lang.org/ http://coconut-lang.org/
- andreareina 9y agoFirst off, well done. My big issue with method chaining is exactly the one of requiring the line-continuation-escape, or enclosing the whole thing in parens. You also need to wrap the initial value, then possibly unwrap it at the end, and you're limited to the methods you define (mind you, this seems like a pretty comprehensive list). For this reason I decided to emulate clojure's threading macros instead, as a pair of functions: pipe_{first,last}( init_val, some_unary_callable, # so far so familiar (some_n_ary_callable, *args), # puts the piped value first or last, depending on {first/last} ... ) Unfinished is the version that makes `(callable, ...args..., _, ...more_args...)` work, is that what you mean by the _ lambda operator?
- senand 9y agoAre you aware of the thread_first and thread_last functions from toolz?
- andreareina 9y agoI wasn't thanks for the pointer!
- entilzha 9y agoNot OP, but author. Line continuation is pretty annoying. Adding custom methods wouldn't be hard (https://github.com/EntilZha/PyFunctional/issues/113 https://github.com/EntilZha/PyFunctional/issues/113), I just don't have time right now to add it. With respect to _, this would make these two equivalent: a = lambda x: x + 1 b = _ + 1 a(1) == b(1) Is there an expanded example I could look at for the pipe stuff, I haven't used clojure before
- valw 9y agoThis looks a lot like Clojure's standard library, and you could probably steal more ideas from it, e.g transducers https://clojure.org/reference/transducers https://clojure.org/reference/transducers
- amelius 9y agoAm I the only one who thinks Python's syntax is not really suited for functional programming? Creating an ad-hoc closure that spans multiple lines, for example, seems awkward. PS: I'm not aware of any syntax changes in this area since Python 2.7, so I could be wrong.
- marshray 9y agoI’ll say it: Python sucks for Functional Programming. It’s just not the right tool for the job (or job for the tool, depending on your point of view.) It’s a great language for most programmers to get things done in, but it belongs to the school of ‘limited abstraction is for your own good’. For example, no multi line lambda expressions. The argument given is that it would mess up whitespace syntax. Haskell handles this situation, but it does indeed introduce a bit more syntax. If you need to FP in a general purpose scripting language, I would be looking at Elixir or Javascript right now.
- entilzha 9y agoTrue, but for better or worse python has support for spark/sklearn/pandas/numpy/plotting/deep learning/web which makes it a good jack of all trades language for data/research science. Its strength is that it can do all of those at levels varying from proficiently to best-in-class.
- gigatexal 9y agoThank you to the author: another day another. Wat library to tinker with.
- fermigier 9y agoShameless plug: https://github.com/sfermigier/awesome-functional-python https://github.com/sfermigier/awesome-functional-python (a list of documents and projects related to FP in Python, including, already, the OP's project). I'm personally using Toolz so far, probably because it's the most popular of the maintained ones. Would be happy to see both some standardisation (a PEP?) and consolidation between the various projects.
- entilzha 9y agoIt would be great to have more native support for FP (flat map is a glaring example I run into a lot). It would help if there were 1) a less verbose anonymous closure syntax 2) support for a multi-line closure.
- flavio81 9y agoVery nice. I just wished Python would have a proper lambda operator. As far as i know they still have that "one-line" lambda keyword which is very restrictive for functional programming.
- sevensor 9y agoI'm no expert in functional programming, but this feels a bit inside-out and backwards to me. It's writing f(g(h(x))) as x.h().g().f(), which doesn't read like function composition to me. If anything, the comprehensions already built into Python look more natural and more functional: (f(v) for v in (g(u) for u in (h(w) for w in x))) The dots in the x.h().g().f() notation also suggest mutation (since we're calling methods on objects here) in a way that makes me uncomfortable. But like I said, I'm no FP expert. My objections are basically aesthetic. Maybe if I used it, I'd learn to like it.
- chasedehan 9y agoThe way you read x.h().g().f() is essentially the same way you read it in F# (with |>), Haskell(with . ,although it goes the opposite direction) and R(with %>%). It is one of the most aestetically pleasing attributes of use in the functional languages
- sevensor 9y agoSo the thing that looks less like the function composition I know from algebra is more idiomatic in FP and more pleasing to the FP cognoscenti? I fear my familiarity with FP may be even more deficient than I thought. I could understand writing f . g . h, but what we're talking about here is written more like Unix pipelines. In which case, I could forgive overloading the bitwise or operator, which is already left-associative. You could then do x | h | g | f. Again, this is a purely aesthetic argument -- the intent of the library seems unclear to me from its syntax. If it's beautiful and clear to you, I'm not going to object to your using it. Aesthetics aside, I don't think there's anything wrong with this library.
- sevensor 9y agoHere's how you could hack up a pipe operator in Python: class Funk: def __init__(self, f): self._f = f def __or__(self, other): return Funk(lambda *_, **__: other(self(*_, **__))) def __call__(self, *args, **kwargs): return self._f(*args, **kwargs) This lets you do fun stuff like: pipeline = Funk(h) | Funk(g) | Funk(f) pipeline(x) pipeline(y) And that makes me happy that I participated in this discussion. I'm kind of surprised nobody's done this before. Unless they have. Any pointers to more fully fleshed out implementations of this idea?
- kccqzy 9y agoHow does its parallelization feature work with lazy generators? Or does its seq function not return generators?
- entilzha 9y agoSeq doesn't return a generator. At its core there is a concept of Lineage taken from Apache Spark. Essentially, when you do something like seq(data).map(func) it builds on a list of operations to execute and holds a copy of the base data. When asked for a value only then will it compute the result. So for parallelization operations that are embarassingly parallel (map/filter etc) are paralelized with multiprocessing. If the data is heavy and serialization is expensive it might be slow, but for operations where the bulk of the work is done in parallel it can help a lot.
- jjuel 9y agoIs there a reason to use this over Coconut? http://coconut-lang.org/ http://coconut-lang.org/
- brightsize 9y agoCoconut is a compile-to-python(2/3) language, not just a library, so if you use it you have to be OK with using a transpiler in your project. Here's one way you might write one of the examples in Coconut: ([1, 2, 3, 4] |> map$(-> _ * 2) |> filter$(-> _ > 4) |> reduce$((x, y) -> x + y) ) You may freely mix Coconut and Python in a project, even in the same file, but any Coconut code still needs to be run through the compiler.