8 ms·

Monads in Python (with nice syntax)

92 points by llambda 14y ago

hxseven 14y ago
BTW: This was already submitted to HN. http://www.hnsearch.com/search#request/all&q=Monads+in+Python&start=0 http://www.hnsearch.com/search#request/all&q=Monads+in+P... But that doesn't make it less interesting ;)
temac 14y ago
This is very interesting, very intelligent. Indeed this is too much intelligent. There is a huge amount of highly valuable programmers that won't be able to understand all that mess, and the gain is questionable at least given that there are alternate constructs that are way better compromise between expressiveness and maintainability. Use the right tool in its intended way: don't try to retrofit constructs in languages were they don't fit, just for an accessory characteristic. (Every line of an imperative language is conceptually kind of a monad, so the article here is just trying to convince us that yet another form of hidden goto is good, by an ad-hoc extension in a language were more than half of monad necessity is missing)
- beering 14y ago
  It might require an intelligent programmer to create, but as a library, can be used by regular, unintelligent programmers. If he documents his monads well and shows how to use it, and it's effective, then it's not such a big deal. Or, you know, this is more of an attempt to show programmers stuck in simpler languages what other PL features have to offer.
- dllthomas 14y ago
  "Every line of an imperative language is conceptually kind of a monad[.]" Every line of an imperative language is conceptually in the IO monad. That doesn't mean that there's never reason to run things in another monad - many do things IO can't.
crucialfelix 14y ago
The problem with Monads is that the people who understand them are the least qualified to explain them to other people.
- dons 14y ago
  This is specious. Many people understand monads, and their ability to teach and explain varies wildly.
- ufo 14y ago
  I disagree. I think the only big problems with teaching monads is that they are a mathematically inspired abstract interface (while most people only need to bother about the concrete implementations of this inteface) and that Haskell forced them too soon on people, since you need monads to do IO.
- StavrosK 14y ago
  The corollary would mean that the people who can explain well can't understand monads, which means that some people just can't understand them. We're doomed!
  - ibotty 14y ago
    not that i agree with the top comment, but your reasoning only works for a static view of people (i.e. they cannot learn something and at the same time forget something).
freework 14y ago
This is missing the one thing that all other "let me explain monads to you" articles is missing: a real example of what problems monads solve. As far as I know, a monad is some kind of mathematical construct that is very useful when solving mathematical proofs, but has no use for the every day programmer. The examples in these kinds of articles are always completely abstract. You can very easily describe a database by using the "customer/phone number" example, yet I've never seen such an example with monads.
- dons 14y ago
  http://www.haskell.org/haskellwiki/Monad#Interesting_monads http://www.haskell.org/haskellwiki/Monad#Interesting_monads Nothing to do with proofs. Monads are a generic interface -- an API -- for changing up the computational rules of your programming language. Just think about that for a second -- its a big deal. Maybe you want to work with non-deterministic threads, or with continuations, or quantum computations, or backtracking search, or exceptions or partial results, or some other notion of computation. But you want to still write in your usual language. So you embed that computational style as a monad, and now you have a really nice API for programming in that different environment -- inside your everyday language. Its simple, but profound. With (good support for) monads you aren't constrained to the default computational rules of your programming language anymore. You are free.
  - lhnz 14y ago
    The problem with this description is that I don't know what kind of benefits I can get from using different computational rules... Can you show me?
    - dons 14y ago
      The standard example is adding checked exceptions. You can refactor nested checks for null into straight line code in the Either (or Maybe or Option) monad, that does the same thing. Consider this pseudo/Haskell code. Functions might fail, so we have to check the return value using nested switches: modify s f = do ev <- get s case ev of Left e -> return (Left e) Right v -> do let u = f v ev <- set s u case ev of Left e -> return (Left e) Right __-> return (Right (v,u)) Lots of nested checks -- makes the code more unreadable, and harder to maintain. Equivalent to null checks in other languages. The 'case' stuff however, is the monadic 'semicolon' between each statement. So we can refactor to: modify s f = do get s >>= \v -> do let u = f v set s u >>= \v -> do return (v,u) Using the error monad definition of bind / >>= - http://hackage.haskell.org/packages/archive/category-extras/latest/doc/html/src/Control-Monad-Either.html#EitherT http://hackage.haskell.org/packages/archive/category-extras/... So then, if you language supports monadic syntax, you can actually use semicolon instead of 'bind': modify s f = do { v <- get s ; let u = f v ; v <- set s u ; return (v,u) ; } and if you are in a whitespace-sensitive language, the error checking disappears into the newline: modify s f = do v <- get s let u = f v v <- set s u return (v,u) So the above code, in the Either monad, is equivalent to the original code, however, we now have shorter, cleaner code, and a compile-time guarantee that all return values are checked. This example is my "programmable semicolon" motivator, from http://donsbot.files.wordpress.com/2009/01/semicolon.pdf http://donsbot.files.wordpress.com/2009/01/semicolon.pdf I chose the computational environment I wanted my code to run in -- one with checked errors -- and did this by overriding my language's default 'semicolon' behavior using the monad interface to do so. I reprogrammed my programming language, to make it possible to write better code. That's real power.
dustingetz 14y ago
i've worked through this article a couple times and his implementation never really sat well with me, the "with nice syntax for monad comprehensions" bit really obfuscates the implementation. here's python source to a monadic lisp interpreter, which i wrote to follow along the paper "monad transformers and modular interpreters" [1]. i think this is a much simpler implementation of python monads than provided in the ValuedLessons article. https://github.com/dustingetz/monadic-interpreter https://github.com/dustingetz/monadic-interpreter Study of this implementation will teach you why nobody actually uses monads in python for non-toy projects. A literal port of this code to clojure would feel so much more idiomatic and not hacky at all. here are some smaller code dumps demonstrating the fundamental concepts to that monadic lisp interpreter: http://www.dustingetz.com/2012/10/02/reader-writer-state-monad-in-python.html http://www.dustingetz.com/2012/10/02/reader-writer-state-mon... http://www.dustingetz.com/2012/10/07/monads-in-python-identity-maybe-error.html http://www.dustingetz.com/2012/10/07/monads-in-python-identi... [1] http://haskell.cs.yale.edu/wp-content/uploads/2011/02/POPL96-Modular-interpreters.pdf http://haskell.cs.yale.edu/wp-content/uploads/2011/02/POPL96...
- dbaupp 14y ago
  I think the point of this article is to make a "nice" syntax for monadic operations in python (i.e. avoiding all the lambdas and extra parens in the "bind(mx, lambda x: my)"/"mx >> (lambda x: my)" pattern), no matter how unclear and non-pedagogical the implementation ends up.
almost 14y ago
Can anyone explain to me how the continuation Monad is working there? As far as I know there's no way to return from the same yield twice, which is what a callcc would need to do. I'm going to carry on trying to puzzle it out but a pointer would be helpful! EDIT: I just tried out the code and it doesn't support multiple returns. Isn't that pretty much the thing that defines the Continuation Monad or am I just not getting it?
- qznc 14y ago
  Technically, it is still a continuation, even if it does not match the behavior of Haskell's continuation monad. http://c2.com/cgi/wiki?SingleUseContinuation http://c2.com/cgi/wiki?SingleUseContinuation
  - almost 14y ago
    I didn't know that, thanks. Seems less useful though. What's the correct term for a continuation as I understand them? (eg: as they are in Haskell, Ruby, Scheme etc)
    - ufo 14y ago
      Continuations that are not single use are usually called multi-shot continuations. Single shot continuations are very useful though. They are enough for a very large percentage of the situations you would use continuations for while being much more efficient to implement (no need to save the stack)while also being more reasonable theretically in other situations(call/cc can be used to break some important invariants when it interacts with mutablestate and other things like that. Its kind of subtle)
klibertp 14y ago
I went through this article some time ago and, while it's certainly neat implementation and worth understanding (that was the first recursive generator that made sense to me - thanks to this I had much easier time understanding Scheme's call/cc and it's uses), it needs to be noted that it is not a generic monad implementation. You can not, for example, implement list monad (I tried very hard and couldn't, I saw the same said in comments below the article, but if it's possible please let me know!) using this implementation and syntax, because you can not "rewind" Python's generators. So, while it's nice implementation for some kinds of monads it's not nearly general enough to be (IMHO) called "monads in Python" - it is possible to implement fully general monads in Python, but you need nested lambdas and there is no getting away from it. Still, very nice hack and worth spending an evening (or two) to understand how it works.
andymatuschak 14y ago
This approach to the "do" syntax requires that the function implementer know the monad type a priori. That defeats some of the purpose of monad as general abstraction. One of the big ideas here, as far as I'm concerned, is that monads represent an abstraction over a common structure of programs. Lists, Maybes, threading models, etc, all share this same language of structuring computations. This is significant because it means that one can build functions which manipulate those structures, and then those structures can be deployed generically. Depending on how you're counting, that means third-order abstraction. For instance, consider liftM from Control.Monad: -- | Promote a function to a monad. liftM :: (Monad m) => (a1 -> r) -> m a1 -> m r liftM f m1 = do { x1 <- m1; return (f x1) } This represents a common idea across e.g. both List and Maybe. I've got a function of values. I want to turn it into a function of Lists / Maybes. Well, liftM recognizes the similarity of List's and Maybe's structure and provides a single implementation suitable for both places. The decorator described in this article cannot function at liftM's level of abstraction: at least as far as I can tell (as a non-Python programmer), it requires a monad type parameter. If that's true, I think it's missing a key idea behind the power of monads.
- ufo 14y ago
  Not really. Python doesnt have typeclasses so liftM would be a fucntion receiving the monad dictionary as an explicit parameter. At that point, you can pass that dictionary to the decorator: def liftM(m, f, x): @do(m) def doNotationBody(): ... ...
csense 14y ago
The first line of his first example of using his monad implementation is: @do(Maybe) Ctrl+F Maybe. It's not defined anywhere! I was excited to see that someone could maybe finally explain to me, in Python, why people are so excited about Monads. I looked at Haskell once. It looks impossible to understand without understanding Monads. (And near-impossible to understand even if you do understand Monads -- the syntax is a nightmare.) I thought maybe this article would help me, by using the syntax of language I do understand -- Python -- with a concept I don't understand -- Monads. But this excitement was misplaced, since this isn't a usable demo. Please, if you're the author of this article or someone who understands this code, post a Github repo, or a tarball on your HTTP server, or something with a runnable, simple example of code that uses the article's approach to monads.

92 points by llambda 14y ago

BTW: This was already submitted to HN. http://www.hnsearch.com/search#request/all&q=Monads+in+Python&start=0 http://www.hnsearch.com/search#request/all&q=Monads+in+P... But that doesn't make it less interesting ;)

temac 14y ago

This is very interesting, very intelligent. Indeed this is too much intelligent. There is a huge amount of highly valuable programmers that won't be able to understand all that mess, and the gain is questionable at least given that there are alternate constructs that are way better compromise between expressiveness and maintainability. Use the right tool in its intended way: don't try to retrofit constructs in languages were they don't fit, just for an accessory characteristic. (Every line of an imperative language is conceptually kind of a monad, so the article here is just trying to convince us that yet another form of hidden goto is good, by an ad-hoc extension in a language were more than half of monad necessity is missing)

beering 14y ago
It might require an intelligent programmer to create, but as a library, can be used by regular, unintelligent programmers. If he documents his monads well and shows how to use it, and it's effective, then it's not such a big deal. Or, you know, this is more of an attempt to show programmers stuck in simpler languages what other PL features have to offer.
dllthomas 14y ago
"Every line of an imperative language is conceptually kind of a monad[.]" Every line of an imperative language is conceptually in the IO monad. That doesn't mean that there's never reason to run things in another monad - many do things IO can't.

crucialfelix 14y ago

The problem with Monads is that the people who understand them are the least qualified to explain them to other people.

dons 14y ago
This is specious. Many people understand monads, and their ability to teach and explain varies wildly.
ufo 14y ago
I disagree. I think the only big problems with teaching monads is that they are a mathematically inspired abstract interface (while most people only need to bother about the concrete implementations of this inteface) and that Haskell forced them too soon on people, since you need monads to do IO.
StavrosK 14y ago
The corollary would mean that the people who can explain well can't understand monads, which means that some people just can't understand them. We're doomed!
- ibotty 14y ago
  not that i agree with the top comment, but your reasoning only works for a static view of people (i.e. they cannot learn something and at the same time forget something).

freework 14y ago

This is missing the one thing that all other "let me explain monads to you" articles is missing: a real example of what problems monads solve. As far as I know, a monad is some kind of mathematical construct that is very useful when solving mathematical proofs, but has no use for the every day programmer. The examples in these kinds of articles are always completely abstract. You can very easily describe a database by using the "customer/phone number" example, yet I've never seen such an example with monads.

dons 14y ago
http://www.haskell.org/haskellwiki/Monad#Interesting_monads http://www.haskell.org/haskellwiki/Monad#Interesting_monads Nothing to do with proofs. Monads are a generic interface -- an API -- for changing up the computational rules of your programming language. Just think about that for a second -- its a big deal. Maybe you want to work with non-deterministic threads, or with continuations, or quantum computations, or backtracking search, or exceptions or partial results, or some other notion of computation. But you want to still write in your usual language. So you embed that computational style as a monad, and now you have a really nice API for programming in that different environment -- inside your everyday language. Its simple, but profound. With (good support for) monads you aren't constrained to the default computational rules of your programming language anymore. You are free.
- lhnz 14y ago
  The problem with this description is that I don't know what kind of benefits I can get from using different computational rules... Can you show me?
  - dons 14y ago
    The standard example is adding checked exceptions. You can refactor nested checks for null into straight line code in the Either (or Maybe or Option) monad, that does the same thing. Consider this pseudo/Haskell code. Functions might fail, so we have to check the return value using nested switches: modify s f = do ev <- get s case ev of Left e -> return (Left e) Right v -> do let u = f v ev <- set s u case ev of Left e -> return (Left e) Right __-> return (Right (v,u)) Lots of nested checks -- makes the code more unreadable, and harder to maintain. Equivalent to null checks in other languages. The 'case' stuff however, is the monadic 'semicolon' between each statement. So we can refactor to: modify s f = do get s >>= \v -> do let u = f v set s u >>= \v -> do return (v,u) Using the error monad definition of bind / >>= - http://hackage.haskell.org/packages/archive/category-extras/latest/doc/html/src/Control-Monad-Either.html#EitherT http://hackage.haskell.org/packages/archive/category-extras/... So then, if you language supports monadic syntax, you can actually use semicolon instead of 'bind': modify s f = do { v <- get s ; let u = f v ; v <- set s u ; return (v,u) ; } and if you are in a whitespace-sensitive language, the error checking disappears into the newline: modify s f = do v <- get s let u = f v v <- set s u return (v,u) So the above code, in the Either monad, is equivalent to the original code, however, we now have shorter, cleaner code, and a compile-time guarantee that all return values are checked. This example is my "programmable semicolon" motivator, from http://donsbot.files.wordpress.com/2009/01/semicolon.pdf http://donsbot.files.wordpress.com/2009/01/semicolon.pdf I chose the computational environment I wanted my code to run in -- one with checked errors -- and did this by overriding my language's default 'semicolon' behavior using the monad interface to do so. I reprogrammed my programming language, to make it possible to write better code. That's real power.

dustingetz 14y ago

i've worked through this article a couple times and his implementation never really sat well with me, the "with nice syntax for monad comprehensions" bit really obfuscates the implementation. here's python source to a monadic lisp interpreter, which i wrote to follow along the paper "monad transformers and modular interpreters" [1]. i think this is a much simpler implementation of python monads than provided in the ValuedLessons article. https://github.com/dustingetz/monadic-interpreter https://github.com/dustingetz/monadic-interpreter Study of this implementation will teach you why nobody actually uses monads in python for non-toy projects. A literal port of this code to clojure would feel so much more idiomatic and not hacky at all. here are some smaller code dumps demonstrating the fundamental concepts to that monadic lisp interpreter: http://www.dustingetz.com/2012/10/02/reader-writer-state-monad-in-python.html http://www.dustingetz.com/2012/10/02/reader-writer-state-mon... http://www.dustingetz.com/2012/10/07/monads-in-python-identity-maybe-error.html http://www.dustingetz.com/2012/10/07/monads-in-python-identi... [1] http://haskell.cs.yale.edu/wp-content/uploads/2011/02/POPL96-Modular-interpreters.pdf http://haskell.cs.yale.edu/wp-content/uploads/2011/02/POPL96...

dbaupp 14y ago
I think the point of this article is to make a "nice" syntax for monadic operations in python (i.e. avoiding all the lambdas and extra parens in the "bind(mx, lambda x: my)"/"mx >> (lambda x: my)" pattern), no matter how unclear and non-pedagogical the implementation ends up.

almost 14y ago

Can anyone explain to me how the continuation Monad is working there? As far as I know there's no way to return from the same yield twice, which is what a callcc would need to do. I'm going to carry on trying to puzzle it out but a pointer would be helpful! EDIT: I just tried out the code and it doesn't support multiple returns. Isn't that pretty much the thing that defines the Continuation Monad or am I just not getting it?

qznc 14y ago
Technically, it is still a continuation, even if it does not match the behavior of Haskell's continuation monad. http://c2.com/cgi/wiki?SingleUseContinuation http://c2.com/cgi/wiki?SingleUseContinuation
- almost 14y ago
  I didn't know that, thanks. Seems less useful though. What's the correct term for a continuation as I understand them? (eg: as they are in Haskell, Ruby, Scheme etc)
  - ufo 14y ago
    Continuations that are not single use are usually called multi-shot continuations. Single shot continuations are very useful though. They are enough for a very large percentage of the situations you would use continuations for while being much more efficient to implement (no need to save the stack)while also being more reasonable theretically in other situations(call/cc can be used to break some important invariants when it interacts with mutablestate and other things like that. Its kind of subtle)

klibertp 14y ago

I went through this article some time ago and, while it's certainly neat implementation and worth understanding (that was the first recursive generator that made sense to me - thanks to this I had much easier time understanding Scheme's call/cc and it's uses), it needs to be noted that it is not a generic monad implementation. You can not, for example, implement list monad (I tried very hard and couldn't, I saw the same said in comments below the article, but if it's possible please let me know!) using this implementation and syntax, because you can not "rewind" Python's generators. So, while it's nice implementation for some kinds of monads it's not nearly general enough to be (IMHO) called "monads in Python" - it is possible to implement fully general monads in Python, but you need nested lambdas and there is no getting away from it. Still, very nice hack and worth spending an evening (or two) to understand how it works.

andymatuschak 14y ago

This approach to the "do" syntax requires that the function implementer know the monad type a priori. That defeats some of the purpose of monad as general abstraction. One of the big ideas here, as far as I'm concerned, is that monads represent an abstraction over a common structure of programs. Lists, Maybes, threading models, etc, all share this same language of structuring computations. This is significant because it means that one can build functions which manipulate those structures, and then those structures can be deployed generically. Depending on how you're counting, that means third-order abstraction. For instance, consider liftM from Control.Monad: -- | Promote a function to a monad. liftM :: (Monad m) => (a1 -> r) -> m a1 -> m r liftM f m1 = do { x1 <- m1; return (f x1) } This represents a common idea across e.g. both List and Maybe. I've got a function of values. I want to turn it into a function of Lists / Maybes. Well, liftM recognizes the similarity of List's and Maybe's structure and provides a single implementation suitable for both places. The decorator described in this article cannot function at liftM's level of abstraction: at least as far as I can tell (as a non-Python programmer), it requires a monad type parameter. If that's true, I think it's missing a key idea behind the power of monads.

ufo 14y ago
Not really. Python doesnt have typeclasses so liftM would be a fucntion receiving the monad dictionary as an explicit parameter. At that point, you can pass that dictionary to the decorator: def liftM(m, f, x): @do(m) def doNotationBody(): ... ...

csense 14y ago

The first line of his first example of using his monad implementation is: @do(Maybe) Ctrl+F Maybe. It's not defined anywhere! I was excited to see that someone could maybe finally explain to me, in Python, why people are so excited about Monads. I looked at Haskell once. It looks impossible to understand without understanding Monads. (And near-impossible to understand even if you do understand Monads -- the syntax is a nightmare.) I thought maybe this article would help me, by using the syntax of language I do understand -- Python -- with a concept I don't understand -- Monads. But this excitement was misplaced, since this isn't a usable demo. Please, if you're the author of this article or someone who understands this code, post a Github repo, or a tarball on your HTTP server, or something with a runnable, simple example of code that uses the article's approach to monads.