11 ms·

The unknown features of Python's `operator` module

88 points by feross 5y ago

stuaxo 5y ago
I've used one or two of these before, would be interested in more patterns that use, methodcaller (I think in the example given str.strip would have worked too).
- mjw1007 5y ago
  In my experience methodcaller is an occasionally-useful "mid-refactoring feature". That is, sometimes when I'm making a largeish modification there's a point in the commit series where the natural way to express what's going on involves using methodcaller, then by the time the work is finished the use has gone away again.
- klyrs 5y ago
  methodcaller can be useful in functional programming, like close_file = methodcaller('close') exhaust(map(close_file, files)) where exhaust is a function that consumes an iterator. Obviously this is a contrived example, but it's a weird transformation -- it turns map from 'call this function on each of a sequence of arguments' to, roughly 'call each of this sequence of functions on a fixed argument'
  - xrisk 5y ago
    exhaust((file.close() for file in files)) works.
    - klyrs 5y ago
      Like I said, contrived example. Though, it's a specific example of a more general pattern where you have a bunch of objects that you need to send a message to. You might use this for event handling in the guts of a game engine.
    - elteto 5y ago
      I believe exhaust(file.close() for file in files) should work as well, no need for the extra parens.
  - wruza 5y ago
    It’s higher order for the sake of higher order, and unreadable as hell when you return to five full pages of this nonsense a year later. These getter-caller generators are cool to show off, but irl the most naturally readable form is either a plain anonymous function, or a special form suggested by a sibling commenter. Both by a compiler easily readable by a human unreadable out of purely functional languages by design this high fashion came.
    - klyrs 5y ago
      Sometimes you need to do higher order stuff (or, maybe you don't? But it exists for good reason) Do you think it's better to use a conceptually difficult example to showcase a high order function? Or do you think it's better to show an example where the reader can accurately understand the concept and focus on what the function does?
blumomo 5y ago
In this article, some listed operators are: operator.length_hint operator.itemgetter operator.countOf What's the (historical?) reason the operator module uses different casing standards?
- O5vYtytb 5y ago
  The operator module is very old: https://github.com/python/cpython/blob/ea4d3f0804a3aad5afaa66a91012325e2067a0af/Modules/operator.c https://github.com/python/cpython/blob/ea4d3f0804a3aad5afaa6... PEP8 was introduced in 2001: https://www.python.org/dev/peps/pep-0008/ https://www.python.org/dev/peps/pep-0008/
- wirrbel 5y ago
  typically in CPython its just a sign that the code is fairly old, or at times (as with the unittest module), that an API was ported from CamlCase language at least partially.
O5vYtytb 5y ago
The article makes a mistake in comparing lambda vs operator.add performance. The lambda example is recreating the function each time. $ python -m timeit "(lambda x,y: x + y)(12, 15)" 10000000 loops, best of 3: 0.124 usec per loop $ python -m timeit -s "from operator import add" "add(12, 15)" 10000000 loops, best of 3: 0.0565 usec per loop Instead, it should create a function in the setup and use that for the loop: $ python -m timeit -s "func = lambda x,y: x + y" "func(12, 15)" 10000000 loops, best of 3: 0.0779 usec per loop
- blester 5y ago
  Generally it is un-pythonic to assign lambda to a variable. Lots of linters will complain if you do this.
  - sharikous 5y ago
    So do a "def" instead, it can be even inlined. def add_fun(x,y): return x+y But I wouldn't mind irritating the linter with an assigned lambda. I fail to see what's the problem with it
    - ViViDboarder 5y ago
      But if you’re doing that, why not just import the identical function?
      - sharikous 5y ago
        One major point of OP was that the functions defined in operators are faster. Grandparent comment argued that there is a measurement error that invalidates this point, since properly defined lambdas (or defs) are just as fast. Of course the point regarding the simplicity of importing a function instead of defining it still stands. I personally wouldn't mind defining those functions as they are very immediate. So from the article I would take only the part about the more exotic "operator".
        5y ago
nemetroid 5y ago
From the article: (lambda x, y: x ^ y)(7, 10) from operator import xor xor(7, 10) > Clearly the second option is more readable. Clearly, the readable option here is ”7 ^ 10”. The examples are a bit contrived, so readability can’t really be gauged from them.
- MajorBee 5y ago
  Can't say I agree with you. Off the top of my head, I couldn't remember what the symbol for xor was (since it hasn't come up once in many years of my programming), but I certainly do know what the word xor stands for.
- klyrs 5y ago
  Yes, articles that explain features with simple examples are almost always full of contrived examples. It's unfortunate that people nitpick them, because the alternative is long and complicated examples that can't be understood at a glance. If you want to do a parity check on a string: parity = reduce(xor, map(ord, str))%2 Except Python3 took reduce away in favor of more boilerplate. And yes, there are multiple alternative implementations of the above.
  - kazinator 5y ago
    > parity = reduce(xor, map(ord, str))%2 "% 2" will not calculate the parity of the reduced value; it just extracts the least significant bit.
    - klyrs 5y ago
      Can you show me an integer whose parity differs from its least significant bit?
      - mikevin 5y ago
        Maybe a stupid question but I've never learned much about parity and this sounds like a good opportunity. Can you give some definitions? Are you talking about error checking parity or 'evenness' parity? I'm guessing the last one since that's what fits your LSB comment(when using 'evenness' not 'oddness'). But in that case I'm not sure your example is doing, my python might be too rusty but is it something like: get the unicode representation(int) of each character in a string, xor the results and get the LSB?
samatman 5y ago
The existence of methodcaller confirms for me the good choices made in the design of Lua. Tables are simple key-value maps, with an optimization for dense arrays, while symbols are strings with a parser-enforced grammar. A method call is syntax sugar: table:method(...) is converted into table.method(table, ...). This means that "methodcaller" is simply table[string](table, ...), which I've used in a few places to good effect.
- ris 5y ago
  Except now you can't have advanced methods on your container types without intermingling the keys of methods and data. This is why javascript moved away from their "super user-friendly" concept of objects-as-maps towards having a real `Map` type - because mixing data and program can get weird (and if it's user-controllable data, dangerous). I suppose the question is - is it possible for a table in lua to contain separate entries for the string "foo" and the symbol "foo", or will one happily overwrite the other?
  - slaymaker1907 5y ago
    I thought the main reason for `Map` was mainly for non-string keys?
  - ledauphin 5y ago
    I'm not sure I've observed a lot of folks in the JavaScript world using Map. I agree there are occasions when it's very useful, but for my money those have more to do with cases where you need non-string keys. For everything else, as far as I can tell, Objects remain entirely idiomatic despite their idiosyncrasies.
    - ris 5y ago
      That's interesting - I must have just been lucky in what I've encountered ES-wise as of late. It relieved me greatly to no longer have the horror of mixing user data and method/"special" names.
sdrabing 5y ago
I've been a weirdo with my Python code recently, and tend towards a more functional programming style. So, the `operator` module is essential for doing some cools things with maps. For instance, I'm writing a library to simulate the behavior of vectors in R, and that means vectorized operators (the operators runs the length of two paired vectors). All you need to do is make a special tuple subclass that maps the operator along another tuple (vector) and returns your vector subclass: https://github.com/shanedrabing/pyrat/blob/e0049213152f615e41f6d189560edc9317fd5fc1/pyrat/base.py#L523 https://github.com/shanedrabing/pyrat/blob/e0049213152f615e4... Anyways, this is all to say that I love `operator` and try my best not to use lamdbas much anymore (except where they are honestly simpler).
- 6gvONxR4sf7o 5y ago
  I try that, but it makes tracebacks harder frequently. Adjusting my programming style based on error message availability is a pet peeve of mine.

88 points by feross 5y ago

I've used one or two of these before, would be interested in more patterns that use, methodcaller (I think in the example given str.strip would have worked too).

mjw1007 5y ago
In my experience methodcaller is an occasionally-useful "mid-refactoring feature". That is, sometimes when I'm making a largeish modification there's a point in the commit series where the natural way to express what's going on involves using methodcaller, then by the time the work is finished the use has gone away again.
klyrs 5y ago
methodcaller can be useful in functional programming, like close_file = methodcaller('close') exhaust(map(close_file, files)) where exhaust is a function that consumes an iterator. Obviously this is a contrived example, but it's a weird transformation -- it turns map from 'call this function on each of a sequence of arguments' to, roughly 'call each of this sequence of functions on a fixed argument'
- xrisk 5y ago
  exhaust((file.close() for file in files)) works.
  - klyrs 5y ago
    Like I said, contrived example. Though, it's a specific example of a more general pattern where you have a bunch of objects that you need to send a message to. You might use this for event handling in the guts of a game engine.
  - elteto 5y ago
    I believe exhaust(file.close() for file in files) should work as well, no need for the extra parens.
- wruza 5y ago
  It’s higher order for the sake of higher order, and unreadable as hell when you return to five full pages of this nonsense a year later. These getter-caller generators are cool to show off, but irl the most naturally readable form is either a plain anonymous function, or a special form suggested by a sibling commenter. Both by a compiler easily readable by a human unreadable out of purely functional languages by design this high fashion came.
  - klyrs 5y ago
    Sometimes you need to do higher order stuff (or, maybe you don't? But it exists for good reason) Do you think it's better to use a conceptually difficult example to showcase a high order function? Or do you think it's better to show an example where the reader can accurately understand the concept and focus on what the function does?

blumomo 5y ago

In this article, some listed operators are: operator.length_hint operator.itemgetter operator.countOf What's the (historical?) reason the operator module uses different casing standards?

O5vYtytb 5y ago
The operator module is very old: https://github.com/python/cpython/blob/ea4d3f0804a3aad5afaa66a91012325e2067a0af/Modules/operator.c https://github.com/python/cpython/blob/ea4d3f0804a3aad5afaa6... PEP8 was introduced in 2001: https://www.python.org/dev/peps/pep-0008/ https://www.python.org/dev/peps/pep-0008/
wirrbel 5y ago
typically in CPython its just a sign that the code is fairly old, or at times (as with the unittest module), that an API was ported from CamlCase language at least partially.

O5vYtytb 5y ago

The article makes a mistake in comparing lambda vs operator.add performance. The lambda example is recreating the function each time. $ python -m timeit "(lambda x,y: x + y)(12, 15)" 10000000 loops, best of 3: 0.124 usec per loop $ python -m timeit -s "from operator import add" "add(12, 15)" 10000000 loops, best of 3: 0.0565 usec per loop Instead, it should create a function in the setup and use that for the loop: $ python -m timeit -s "func = lambda x,y: x + y" "func(12, 15)" 10000000 loops, best of 3: 0.0779 usec per loop

blester 5y ago
Generally it is un-pythonic to assign lambda to a variable. Lots of linters will complain if you do this.
- sharikous 5y ago
  So do a "def" instead, it can be even inlined. def add_fun(x,y): return x+y But I wouldn't mind irritating the linter with an assigned lambda. I fail to see what's the problem with it
  - ViViDboarder 5y ago
    But if you’re doing that, why not just import the identical function?
    - sharikous 5y ago
      One major point of OP was that the functions defined in operators are faster. Grandparent comment argued that there is a measurement error that invalidates this point, since properly defined lambdas (or defs) are just as fast. Of course the point regarding the simplicity of importing a function instead of defining it still stands. I personally wouldn't mind defining those functions as they are very immediate. So from the article I would take only the part about the more exotic "operator".
      - 5y ago

nemetroid 5y ago

From the article: (lambda x, y: x ^ y)(7, 10) from operator import xor xor(7, 10) > Clearly the second option is more readable. Clearly, the readable option here is ”7 ^ 10”. The examples are a bit contrived, so readability can’t really be gauged from them.

MajorBee 5y ago
Can't say I agree with you. Off the top of my head, I couldn't remember what the symbol for xor was (since it hasn't come up once in many years of my programming), but I certainly do know what the word xor stands for.
klyrs 5y ago
Yes, articles that explain features with simple examples are almost always full of contrived examples. It's unfortunate that people nitpick them, because the alternative is long and complicated examples that can't be understood at a glance. If you want to do a parity check on a string: parity = reduce(xor, map(ord, str))%2 Except Python3 took reduce away in favor of more boilerplate. And yes, there are multiple alternative implementations of the above.
- kazinator 5y ago
  > parity = reduce(xor, map(ord, str))%2 "% 2" will not calculate the parity of the reduced value; it just extracts the least significant bit.
  - klyrs 5y ago
    Can you show me an integer whose parity differs from its least significant bit?
    - mikevin 5y ago
      Maybe a stupid question but I've never learned much about parity and this sounds like a good opportunity. Can you give some definitions? Are you talking about error checking parity or 'evenness' parity? I'm guessing the last one since that's what fits your LSB comment(when using 'evenness' not 'oddness'). But in that case I'm not sure your example is doing, my python might be too rusty but is it something like: get the unicode representation(int) of each character in a string, xor the results and get the LSB?

samatman 5y ago

The existence of methodcaller confirms for me the good choices made in the design of Lua. Tables are simple key-value maps, with an optimization for dense arrays, while symbols are strings with a parser-enforced grammar. A method call is syntax sugar: table:method(...) is converted into table.method(table, ...). This means that "methodcaller" is simply table[string](table, ...), which I've used in a few places to good effect.

ris 5y ago
Except now you can't have advanced methods on your container types without intermingling the keys of methods and data. This is why javascript moved away from their "super user-friendly" concept of objects-as-maps towards having a real `Map` type - because mixing data and program can get weird (and if it's user-controllable data, dangerous). I suppose the question is - is it possible for a table in lua to contain separate entries for the string "foo" and the symbol "foo", or will one happily overwrite the other?
- slaymaker1907 5y ago
  I thought the main reason for `Map` was mainly for non-string keys?
- ledauphin 5y ago
  I'm not sure I've observed a lot of folks in the JavaScript world using Map. I agree there are occasions when it's very useful, but for my money those have more to do with cases where you need non-string keys. For everything else, as far as I can tell, Objects remain entirely idiomatic despite their idiosyncrasies.
  - ris 5y ago
    That's interesting - I must have just been lucky in what I've encountered ES-wise as of late. It relieved me greatly to no longer have the horror of mixing user data and method/"special" names.

sdrabing 5y ago

I've been a weirdo with my Python code recently, and tend towards a more functional programming style. So, the `operator` module is essential for doing some cools things with maps. For instance, I'm writing a library to simulate the behavior of vectors in R, and that means vectorized operators (the operators runs the length of two paired vectors). All you need to do is make a special tuple subclass that maps the operator along another tuple (vector) and returns your vector subclass: https://github.com/shanedrabing/pyrat/blob/e0049213152f615e41f6d189560edc9317fd5fc1/pyrat/base.py#L523 https://github.com/shanedrabing/pyrat/blob/e0049213152f615e4... Anyways, this is all to say that I love `operator` and try my best not to use lamdbas much anymore (except where they are honestly simpler).

6gvONxR4sf7o 5y ago
I try that, but it makes tracebacks harder frequently. Adjusting my programming style based on error message availability is a pet peeve of mine.