6 ms·

Lambdas and Functions in Python

62 points by mastro35 8y ago

pbreit 8y ago
“Isn’t it better?” No!!!!!!!! The original code looks by far the best to me. Why over-complicate?
- flomble 8y ago
  Because it's almost the same thing copy-pasted repeatedly; textbook boilerplate. I know it's a matter of taste, but the last one also looks by far the best to me.
- mastro35 8y ago
  Well, actually I find the last code far better and easier than the original one. As I wrote in the article, if you want to add a factorial method you just need to change the catalog of the function provided by adding this line “!”: lambda x: math.factorial(x), However, it’s probably a matter of taste... but I do like the last code more. :)
  - adjkant 8y ago
    I'm a big fan of first-class functions (or similar) and functional code, but lambdas are in the end just syntactic sugar. Why not just write: "!": math.factorial, The lambda wrapper isn't doing anything here.
    - guitarbill 8y ago
      I also thought that, but `math.factorial` is a builtin, and so you cannot get the signature of it, and the code can't determine how many operands the command needs. This is similar to the `self.stack.clear()` example. (`signature(math.factorial)` gives "ValueError: no signature found for builtin <built-in function factorial>")
- jewbacca 8y ago
  Most of the code by volume in this post's final product (and most of the ugliness you're probably seeing in it) comes from dealing separately with the arity of different operations (`compute_operation_with_one_operand`, `compute_operation_with_two_operands`, etc) -- which retains a whole lot of copy-paste-but-change-one-small-thing-in-the-middle boilerplate. That can be taken further and generalized out. It's been a while since I did much Python metaprogramming-navel-gazing, but if you can't do something to automatically detect the arity of the operations' lambdas/functions, you can at least explicitly annotate the operations data with their arity -- which is already implicitly being done (as code instead of data) in `compute`'s 3 if-statements + the 3 "compute_with_N_operands" functions. From there, you'll only have 1 case once, instead of 3 cases with each one manifesting in 2 different places. And just about half of the code disappears. ---- edit: since adjkant already covered the "automatically detect the arity of the operations' lambdas/functions" approach to removing all the cases, (https://news.ycombinator.com/item?id=17329772 https://news.ycombinator.com/item?id=17329772), here is the gist of what I meant by "explicitly annotate the operations data with their arity": import math, operator class rpn_engine: def __init__(self): self.stack = [] self.catalog = {"+": (2, operator.add), "-": (2, operator.sub), "*": (2, operator.mul), "/": (2, operator.truediv), "^2": (1, lambda x: x * x), "SQRT": (1, math.sqrt), "C": (0, self.stack.pop), "AC": (0, self.stack.clear)} def compute(self, operation): (arity, op) = self.catalog[operation] operands = reversed([self.stack.pop() for _ in range(arity)]) return self.stack.push(op(*operands))
  - deleted 8y ago
    [deleted]
  - jewbacca 8y ago
    Someone replied with and then unfortunately deleted an extremely good suggestion. Instead of each operation taking a variable number of numbers and then either returning a value or directly changing the state of the stack (which was also a bug in my previous gist, in pushing the result of C and AC): homogenize the types of the operations by making all of them take a stack and return a stack. You could also wrap all of the basic arithmetic functions with a higher-level function so that, eg, `add` would still not actually have to be aware of the stack: import math, operator def stackFunction(arity, function): return lambda stack: stack[:-arity] + [function(*stack[-arity:])] class rpn_engine: def __init__(self): self.stack = [] self.catalog = {"+": stackFunction(2, operator.add), "-": stackFunction(2, operator.sub), "*": stackFunction(2, operator.mul), "/": stackFunction(2, operator.truediv), "^2": stackFunction(1, lambda x: x * x), "SQRT": stackFunction(1, math.sqrt), "C": lambda stack: stack[:-1], "AC": lambda stack: []} def compute(self, operation): self.stack = self.catalog[operation](self.stack) def push(self, number): self.stack.append(number)
- andrepd 8y ago
  The last version looks much better to me, and much more extensible.
chriswarbo 8y ago
It's interesting to note that a dictionary ("catalog") of values is essentially the same concept as an object. This is clearer in Javascript, where `{...}` notation is called an "object" rather than a "dictionary" and the `.foo` and `["foo"]` notations are mostly interchangable. Storing functions in a dictionary is very close to storing methods in an object. The only differences are sharing state via a `self` (or `this`) argument, and inheritance chains (either from classes or prototypes). Also interesting to note that many of the author's functions, like `lambda x, y: x + y`, only make sense because the underlying language (Python) thought that constructs like `+` shouldn't be functions for some reason. Many languages seem to do this, e.g. here's a bug report I raised for PHP https://bugs.php.net/bug.php?id=66368 https://bugs.php.net/bug.php?id=66368 When we introduce arbitrary distinctions, like dictionary vs object, named vs anonymous functions, functions vs operators, etc. then we end up with all of the choices and complications like those shown in this article. One of the principles I try to follow when programming is to avoid making distinctions, or breaking symmetries, unless my hand is forced by the underlying dynamics. Experience has taught me that introducing distinctions in one place just introduces complication elsewhere (e.g. more cases to handle when metaprogramming).
- Felk 8y ago
  I don't know why operators are the way they are in Python, but to me it looks like an oversight, which ultimately let to the "operator" module, exposing all operators as functions
  - guitarbill 8y ago
    At the risk of oversimplifying/getting the subtleties wrong, operators are syntactic sugar to call magic methods. You could still call the magic method itself: >>> a = 1 >>> a.__add__(2) 3 >>> b = [] >>> b.__len__() 0 It's rare I have to use the "operator" module, and I find Python pretty well thought out w.r.t. consistency. All in all, seems fine to me.
    - chriswarbo 8y ago
      > You could still call the magic method itself Only if you have the first argument available. As a silly example, we might want to do the following (in analogy to the built-in `sum` function): product = lambda vals: reduce(operators.__mult__, vals, 1) We can't use a bound `__mult__` method for situations like this. More generally, looking up methods from particular objects forces our code to be "first order", i.e. we have to explicitly deal with intermediate results which don't appear in higher-order programming. For the `product` case we could do: def product(vals): result = 1 for val in vals: result *= val return result 3/5 of these lines deal with `result`, which doesn't appear at all in the `reduce` version. The same thing happens with stuff like composition too, e.g. the `x` in `lambda x: foo(bar(x))` when we could just `compose(foo, bar)`. When code is bound up in methods, we're often forced to expose these intermediate values, in order to call their methods. Of course, we can abstract over such boilerplate too, with functions like `lambda o, m: o.__getattribute__(m)`, etc. but at that point we're fighting against the language. PS: Yes, I know Guido doesn't like `reduce` and moved it out into `functools` for Python 3.
guitarbill 8y ago
That code seems suspect. Throwing and catching BaseException is bad/has some weird consequences and probably unintentional - I assume the author meant to use Exception? Pathologically avoiding `elif`. Misspelling `inspect` in the code examples which are ostensibly verbatim Python sessions. In general, lambda functions are less necessary in Python because functions are first-class. The lambda syntax is useful for extremely trivial functions, e.g. `sorted(a, key=lambda value: value.foo_bar())`, but otherwise "normal" functions work just fine/better - especially since you can define a function wherever you want. (This is how it's supposed to be. E.g. in Java or Javascript, code using many lambdas can be a royal pain to debug.) Maybe related, I dislike the use of `map` and `filter`. Almost all real-world cases are more readable using list/dict/set comprehension or generator expressions. The only thing missing now is assignment expressions (PEP 572, [0]) so I can do a map + filter in one comprehension. [0] https://www.python.org/dev/peps/pep-0572/ https://www.python.org/dev/peps/pep-0572/
- danso 8y ago
  Moving from Ruby to Python, for me one of the initial frustrations was being unable to define multi-line anonymous functions, though eventually I accepted the wisdom of Python's conservative philosophy over Ruby's "Do what makes you happy". But I'm glad that Guido et. al decided to keep lambda around. It feels like the right tradeoff with syntactic sugar, as one-line anonymous functions offer just enough utility in a limited scope. In contrast, having to formally define trivial functions feels like too much overhead and clutter.
- mastro35 8y ago
  Hi guitarbill, the article is intended to point out how to review the original code. I figured out a bad code written by anyone else and I tried to make it better. About the lambdas you are right, they’re good just if you have trivial functions, like in other languages. If you need more “power” just use normal functions. PS: if I mispelled “inspect” or anything else... sorry, I’m not a native speaker, I wish I could speak and write better but the truth is that my English sucks... :(
  - RussianCow 8y ago
adjkant 8y ago
This code can look a lot better by replacing the compute function with this: https://pastebin.com/fMtc8Git https://pastebin.com/fMtc8Git
- mastro35 8y ago
  Yep! Good call! ;)
- abecedarius 8y ago
  It looks like that reverses the arguments? You could do args = self.stack[-n:] del self.stack[-n:] self.push(function_requested(*args)) or maybe self.stack[-n:] = [function_requested(*args)]
  - adjkant 8y ago
    Yeah oops coded that up quick. Either way you fix it, it’s far better than a function for calling each number of args and keeps the code readable.
    - adjkant 8y ago
      For fun, an updated version that cleans up other things as well as the ordering, not relying on signatures of lambdas, etc. https://pastebin.com/HF41J8KQ https://pastebin.com/HF41J8KQ
Waterluvian 8y ago
Has anyone ever experienced a time where assigning member variables on a function made sense? The thing I love about python is the freedom to do whatever I want and the responsibility to almost never do it. I crave examples of exceptional cases.
- guitarbill 8y ago
  The most common cases are decorator functions, e.g. one that counts the number of invocations: https://stackoverflow.com/questions/44968004/python-decorators-count-function-call https://stackoverflow.com/questions/44968004/python-decorato...
roenxi 8y ago
Python's lambda also has an interesting design flaw that anyone using it will come up against sooner or later [1]. The way it interacts with Python's scoping rules is quite broken. This hinders using it as freely as is possible in, eg, Clojure. >>> fs = [(lambda n: i + n) for i in range(10)] >>> [f(4) for f in fs] [13, 13, 13, 13, 13, 13, 13, 13, 13, 13] [1] http://math.andrej.com/2009/04/09/pythons-lambda-is-broken/ http://math.andrej.com/2009/04/09/pythons-lambda-is-broken/
- UncleEntity 8y ago
  Python stores the captured variables in __globals__ which is shared with all the different lambdas for some odd reason: >>> fs[0] == fs[1] False >>> fs[0].__globals__ == fs[1].__globals__ True >>> fs[0].__globals__['i'], fs[1].__globals__['i'] (9, 9) Kind of weird? --edit-- >>> def f(i): ... return lambda n: i + n >>> fs = [f(i) for i in range(10)] >>> [f(4) for f in fs] [4, 5, 6, 7, 8, 9, 10, 11, 12, 13] It works if it has somewhere to store the captured variables so just don't pollute the global environment and all is fine...
  - abecedarius 8y ago
    I think you're looking at Python 2, because in py3 the list comprehensions create a nested scope for the loop variable to be local in. -- i.e. 'i' would be in a local environment, not in globals. This doesn't change the behavior of these particular closures, since i is still mutated in the loop.
    - UncleEntity 8y ago
      Python 3.6.5 (default, Apr 4 2018, 15:01:18) ...even weirder: >>> fs[0].__closure__ == fs[1].__closure__ True
      - abecedarius 8y ago
        Since I don't have py3 installed at the moment, I can't check anything, but I'd guess this is two closures with distinct ids but the same environments, and == is doing a structural comparison rather than an identity comparison. Wish I could look into it right now.

62 points by mastro35 8y ago

pbreit 8y ago

“Isn’t it better?” No!!!!!!!! The original code looks by far the best to me. Why over-complicate?

flomble 8y ago
Because it's almost the same thing copy-pasted repeatedly; textbook boilerplate. I know it's a matter of taste, but the last one also looks by far the best to me.
mastro35 8y ago
Well, actually I find the last code far better and easier than the original one. As I wrote in the article, if you want to add a factorial method you just need to change the catalog of the function provided by adding this line “!”: lambda x: math.factorial(x), However, it’s probably a matter of taste... but I do like the last code more. :)
- adjkant 8y ago
  I'm a big fan of first-class functions (or similar) and functional code, but lambdas are in the end just syntactic sugar. Why not just write: "!": math.factorial, The lambda wrapper isn't doing anything here.
  - guitarbill 8y ago
    I also thought that, but `math.factorial` is a builtin, and so you cannot get the signature of it, and the code can't determine how many operands the command needs. This is similar to the `self.stack.clear()` example. (`signature(math.factorial)` gives "ValueError: no signature found for builtin <built-in function factorial>")
jewbacca 8y ago
Most of the code by volume in this post's final product (and most of the ugliness you're probably seeing in it) comes from dealing separately with the arity of different operations (`compute_operation_with_one_operand`, `compute_operation_with_two_operands`, etc) -- which retains a whole lot of copy-paste-but-change-one-small-thing-in-the-middle boilerplate. That can be taken further and generalized out. It's been a while since I did much Python metaprogramming-navel-gazing, but if you can't do something to automatically detect the arity of the operations' lambdas/functions, you can at least explicitly annotate the operations data with their arity -- which is already implicitly being done (as code instead of data) in `compute`'s 3 if-statements + the 3 "compute_with_N_operands" functions. From there, you'll only have 1 case once, instead of 3 cases with each one manifesting in 2 different places. And just about half of the code disappears. ---- edit: since adjkant already covered the "automatically detect the arity of the operations' lambdas/functions" approach to removing all the cases, (https://news.ycombinator.com/item?id=17329772 https://news.ycombinator.com/item?id=17329772), here is the gist of what I meant by "explicitly annotate the operations data with their arity": import math, operator class rpn_engine: def __init__(self): self.stack = [] self.catalog = {"+": (2, operator.add), "-": (2, operator.sub), "*": (2, operator.mul), "/": (2, operator.truediv), "^2": (1, lambda x: x * x), "SQRT": (1, math.sqrt), "C": (0, self.stack.pop), "AC": (0, self.stack.clear)} def compute(self, operation): (arity, op) = self.catalog[operation] operands = reversed([self.stack.pop() for _ in range(arity)]) return self.stack.push(op(*operands))
- deleted 8y ago
  [deleted]
- jewbacca 8y ago
  Someone replied with and then unfortunately deleted an extremely good suggestion. Instead of each operation taking a variable number of numbers and then either returning a value or directly changing the state of the stack (which was also a bug in my previous gist, in pushing the result of C and AC): homogenize the types of the operations by making all of them take a stack and return a stack. You could also wrap all of the basic arithmetic functions with a higher-level function so that, eg, `add` would still not actually have to be aware of the stack: import math, operator def stackFunction(arity, function): return lambda stack: stack[:-arity] + [function(*stack[-arity:])] class rpn_engine: def __init__(self): self.stack = [] self.catalog = {"+": stackFunction(2, operator.add), "-": stackFunction(2, operator.sub), "*": stackFunction(2, operator.mul), "/": stackFunction(2, operator.truediv), "^2": stackFunction(1, lambda x: x * x), "SQRT": stackFunction(1, math.sqrt), "C": lambda stack: stack[:-1], "AC": lambda stack: []} def compute(self, operation): self.stack = self.catalog[operation](self.stack) def push(self, number): self.stack.append(number)
andrepd 8y ago
The last version looks much better to me, and much more extensible.

chriswarbo 8y ago

It's interesting to note that a dictionary ("catalog") of values is essentially the same concept as an object. This is clearer in Javascript, where `{...}` notation is called an "object" rather than a "dictionary" and the `.foo` and `["foo"]` notations are mostly interchangable. Storing functions in a dictionary is very close to storing methods in an object. The only differences are sharing state via a `self` (or `this`) argument, and inheritance chains (either from classes or prototypes). Also interesting to note that many of the author's functions, like `lambda x, y: x + y`, only make sense because the underlying language (Python) thought that constructs like `+` shouldn't be functions for some reason. Many languages seem to do this, e.g. here's a bug report I raised for PHP https://bugs.php.net/bug.php?id=66368 https://bugs.php.net/bug.php?id=66368 When we introduce arbitrary distinctions, like dictionary vs object, named vs anonymous functions, functions vs operators, etc. then we end up with all of the choices and complications like those shown in this article. One of the principles I try to follow when programming is to avoid making distinctions, or breaking symmetries, unless my hand is forced by the underlying dynamics. Experience has taught me that introducing distinctions in one place just introduces complication elsewhere (e.g. more cases to handle when metaprogramming).

Felk 8y ago
I don't know why operators are the way they are in Python, but to me it looks like an oversight, which ultimately let to the "operator" module, exposing all operators as functions
- guitarbill 8y ago
  At the risk of oversimplifying/getting the subtleties wrong, operators are syntactic sugar to call magic methods. You could still call the magic method itself: >>> a = 1 >>> a.__add__(2) 3 >>> b = [] >>> b.__len__() 0 It's rare I have to use the "operator" module, and I find Python pretty well thought out w.r.t. consistency. All in all, seems fine to me.
  - chriswarbo 8y ago
    > You could still call the magic method itself Only if you have the first argument available. As a silly example, we might want to do the following (in analogy to the built-in `sum` function): product = lambda vals: reduce(operators.__mult__, vals, 1) We can't use a bound `__mult__` method for situations like this. More generally, looking up methods from particular objects forces our code to be "first order", i.e. we have to explicitly deal with intermediate results which don't appear in higher-order programming. For the `product` case we could do: def product(vals): result = 1 for val in vals: result *= val return result 3/5 of these lines deal with `result`, which doesn't appear at all in the `reduce` version. The same thing happens with stuff like composition too, e.g. the `x` in `lambda x: foo(bar(x))` when we could just `compose(foo, bar)`. When code is bound up in methods, we're often forced to expose these intermediate values, in order to call their methods. Of course, we can abstract over such boilerplate too, with functions like `lambda o, m: o.__getattribute__(m)`, etc. but at that point we're fighting against the language. PS: Yes, I know Guido doesn't like `reduce` and moved it out into `functools` for Python 3.

guitarbill 8y ago

That code seems suspect. Throwing and catching BaseException is bad/has some weird consequences and probably unintentional - I assume the author meant to use Exception? Pathologically avoiding `elif`. Misspelling `inspect` in the code examples which are ostensibly verbatim Python sessions. In general, lambda functions are less necessary in Python because functions are first-class. The lambda syntax is useful for extremely trivial functions, e.g. `sorted(a, key=lambda value: value.foo_bar())`, but otherwise "normal" functions work just fine/better - especially since you can define a function wherever you want. (This is how it's supposed to be. E.g. in Java or Javascript, code using many lambdas can be a royal pain to debug.) Maybe related, I dislike the use of `map` and `filter`. Almost all real-world cases are more readable using list/dict/set comprehension or generator expressions. The only thing missing now is assignment expressions (PEP 572, [0]) so I can do a map + filter in one comprehension. [0] https://www.python.org/dev/peps/pep-0572/ https://www.python.org/dev/peps/pep-0572/

danso 8y ago
Moving from Ruby to Python, for me one of the initial frustrations was being unable to define multi-line anonymous functions, though eventually I accepted the wisdom of Python's conservative philosophy over Ruby's "Do what makes you happy". But I'm glad that Guido et. al decided to keep lambda around. It feels like the right tradeoff with syntactic sugar, as one-line anonymous functions offer just enough utility in a limited scope. In contrast, having to formally define trivial functions feels like too much overhead and clutter.
mastro35 8y ago
Hi guitarbill, the article is intended to point out how to review the original code. I figured out a bad code written by anyone else and I tried to make it better. About the lambdas you are right, they’re good just if you have trivial functions, like in other languages. If you need more “power” just use normal functions. PS: if I mispelled “inspect” or anything else... sorry, I’m not a native speaker, I wish I could speak and write better but the truth is that my English sucks... :(
- RussianCow 8y ago

adjkant 8y ago

This code can look a lot better by replacing the compute function with this: https://pastebin.com/fMtc8Git https://pastebin.com/fMtc8Git

mastro35 8y ago
Yep! Good call! ;)
abecedarius 8y ago
It looks like that reverses the arguments? You could do args = self.stack[-n:] del self.stack[-n:] self.push(function_requested(*args)) or maybe self.stack[-n:] = [function_requested(*args)]
- adjkant 8y ago
  Yeah oops coded that up quick. Either way you fix it, it’s far better than a function for calling each number of args and keeps the code readable.
  - adjkant 8y ago
    For fun, an updated version that cleans up other things as well as the ordering, not relying on signatures of lambdas, etc. https://pastebin.com/HF41J8KQ https://pastebin.com/HF41J8KQ

Waterluvian 8y ago

Has anyone ever experienced a time where assigning member variables on a function made sense? The thing I love about python is the freedom to do whatever I want and the responsibility to almost never do it. I crave examples of exceptional cases.

guitarbill 8y ago
The most common cases are decorator functions, e.g. one that counts the number of invocations: https://stackoverflow.com/questions/44968004/python-decorators-count-function-call https://stackoverflow.com/questions/44968004/python-decorato...

roenxi 8y ago

Python's lambda also has an interesting design flaw that anyone using it will come up against sooner or later [1]. The way it interacts with Python's scoping rules is quite broken. This hinders using it as freely as is possible in, eg, Clojure. >>> fs = [(lambda n: i + n) for i in range(10)] >>> [f(4) for f in fs] [13, 13, 13, 13, 13, 13, 13, 13, 13, 13] [1] http://math.andrej.com/2009/04/09/pythons-lambda-is-broken/ http://math.andrej.com/2009/04/09/pythons-lambda-is-broken/

UncleEntity 8y ago
Python stores the captured variables in __globals__ which is shared with all the different lambdas for some odd reason: >>> fs[0] == fs[1] False >>> fs[0].__globals__ == fs[1].__globals__ True >>> fs[0].__globals__['i'], fs[1].__globals__['i'] (9, 9) Kind of weird? --edit-- >>> def f(i): ... return lambda n: i + n >>> fs = [f(i) for i in range(10)] >>> [f(4) for f in fs] [4, 5, 6, 7, 8, 9, 10, 11, 12, 13] It works if it has somewhere to store the captured variables so just don't pollute the global environment and all is fine...
- abecedarius 8y ago
  I think you're looking at Python 2, because in py3 the list comprehensions create a nested scope for the loop variable to be local in. -- i.e. 'i' would be in a local environment, not in globals. This doesn't change the behavior of these particular closures, since i is still mutated in the loop.
  - UncleEntity 8y ago
    Python 3.6.5 (default, Apr 4 2018, 15:01:18) ...even weirder: >>> fs[0].__closure__ == fs[1].__closure__ True
    - abecedarius 8y ago
      Since I don't have py3 installed at the moment, I can't check anything, but I'd guess this is two closures with distinct ids but the same environments, and == is doing a structural comparison rather than an identity comparison. Wish I could look into it right now.