8 ms·

Generators are one of the concepts that devs from other languages under use in Python. It's a natural solution to a ton of problems, since it lazily evaluates,

Generators are one of the concepts that devs from other languages under use in Python. It's a natural solution to a ton of problems, since it lazily evaluates, and only stores one result out of the entire sequence at a time, and its interface is compatible with the iterator protocol. It's a mini state machine with it's own little memory world as well. If you are used to call backs, promises, tick() calls or closures, it will be natural to reach for them a lot, and be disappointed at their state in the Python ecosystem. But while I do miss better anonymous functions support sometimes in Python, it is rarely a problem because it comes with other tools to arrive to the same result. Generators are one of them. Of course, you'll have to realize that generators are way more that a lazy way of generating data. You can use them for control flow, hold state, hide complexity or delegate behavior using a common interface. The send() method is very often completely ignored by most devs, and the yield from underused. But beyond generators are a lot more things that, in the same vein, will bring you to python enlightenment once you start thinking with them instead of trying to make a Python fits into a square hole: - the functools module - the itertools module - context managers - any/all/zip/iter/next/enumerate - decorators (it can do so much more than you imagine) - type hints (take a look at fast api for awesomeness)

drcongo 5y ago
I don't know many languages, do decorators show up in any others? I _love_ them in Python.
- BiteCode_dev 5y ago
  Yes. In fact, E.G: C# and JS have them, although they don't have exactly the same features. edit, sorry I've meant generators. Incidentally, it's also true for js decorators.
  - k__ 5y ago
    How do Python generators differ from JavaScript generators?
    - BiteCode_dev 5y ago
      Yield from comes to mind.
      - k__ 5y ago
        Isn't JS yield* the same as yield from in Python
        BiteCode_dev 5y ago
        Unless I'm mistaken, it propagate pulling, but not pushing with send(). I haven't checked though.
  - goatlover 5y ago
    Decorators are still a pending proposal in ECMAScript, so they would only be available to transpilers. That makes them non-standard until the proposal is approved, where the implementation could change.
- useerup 5y ago
  In C#: static IEnumerable<int> Fibonacci() { int first = 0; int second = 1; yield return first; yield return second; while (true) { (first, second) = (second, second + first); yield return second; } }
  - whoisthemachine 5y ago
    Interesting use of tuples I had never considered before, a lot of state is packed in that `(first, second)...` block! Pretty fun though.
- b3morales 5y ago
  Emacs Lisp* has a system called "advice" that's maybe a bit clumsier than Python's, but is also more powerful. The relationship between decorator and decorated [is configurable][0]. For example, the decorated function could be called conditionally on the result of applying the decorator to the arguments, something like: def _combined(args): return decorator(args) and original(args) It can also be applied retroactively, not just at the decorated function's definition. This is extremely handy in the context of configuring your text editor, but I admit I wouldn't like it in a program I was working on with other people. (Basically a form of monkeypatching.) * Granted not really used for general-purpose programming like Python. [0]:https://www.gnu.org/software/emacs/manual/html_node/elisp/Advice-Combinators.html https://www.gnu.org/software/emacs/manual/html_node/elisp/Ad...
  - creamytaco 5y ago
    Emacs Lisp has macros which are a lot more elegant (not to mention powerful) than Python decorators which comparatively, are nothing more than a kludge.
- krick 5y ago
  Depends on what you mean. I often hear claims that you can use decorators in language X, which are formally true, since decorator is just a function accepting and returning another function, so any language that has anonymous functions (which is a lot of languages by now) "has decorators". But for me this is kind of false marketing, because there's nothing impressive about that, and the killer feature of Python's decorators is the syntactic sugar, which makes them so easy to use. And in this sense not so many languages "have decorators". You can do something similar with annotations in PHP, but I've never actually done this in a real project, because that feels like abusing the language.
ajuc 5y ago
Generators are such a game-changer for clean separation of data traversal from data processing. It's my favorite trick in python and I miss it dearly in other languages. In languages without generators it's easy to write code that traverses some data structure in hard-coded way and does something configurable on every node, but it's hard to write code that traverses a data structure in configurable way and does some hardcoded thing on every element. In Python it's trivial both ways and even both at once.
- idle_zealot 5y ago
  > it's hard to write code that traverses a data structure in configurable way and does some hardcoded thing on every element Wouldn’t this just be a function that operates on an iterator? I suppose generators make the creation of lazy iterators easier, but generally the solution for languages without generators is to have your traversal build a list. So you lose the laziness but the code remains simple. Then map your per-node processing to the list.
  - ajuc 5y ago
    > generally the solution for languages without generators is to have your traversal build a list Yes, but then you move from O(1) to O(n) in memory usage. And if you want to optimize it you have to restructure your code, when in python lists and generators are drop-in replacements.
    - justinpombrio 5y ago
      Also, producing a list doesn't work very well as a substitute for infinite generators.
    - greggyb 5y ago
      Until you want to walk the generator twice -- stateful generators are a design flaw, in my opinion.
- muxator 5y ago
  The C++20 coroutines has the potential of being this and much more. As with many things C++, its adoption will be very slow (they are supported by latest compiler versions, but there is no support in the standard libraries yet).
andrewstuart 5y ago
>> If you are used to call backs, promises, tick() calls or closures, it will be natural to reach for them a lot, and be disappointed at their state in the Python ecosystem. It's a real disappointment that Python has no answer for JavaScript fat arrow. The fat arrow is the one construct that completely changes the structure of all the programs I write - it makes things extremely terse and I use it in nearly every function/method in some way. I'm a big fan of Python's whitespace structure but it seems to be an obstacle to creating single line functions that are both powerful and get multiple things done, which sometimes makes a program easier to read and more understandable. Sure there's lambda functions but I find them clumsy and hard to work with compared to the beauty of the fat arrow.
- BiteCode_dev 5y ago
  The rare case where you do need a fat arrow, it is indeed elegant once misses from Python. But I also know by experience that people from other languages are reaching for anonymous functions way too often instead of using the tools that are suitable for the use case in python. E.G: if you use a lot of map() and filter() in python, you are doing it wrong.
anonymousDan 5y ago
One thing that really annoys me about PySpark is Python's lack of tuple unpacking in lambdas. It turns what should be relatively readable pipelines of transformations into an unreadable mess.
- canjobear 5y ago
  Alas, Python 2 had tuple unpacking, but they removed it in Python 3 :(
valenterry 5y ago
> If you are used to call backs, promises, tick() calls or closures, it will be natural to reach for them a lot, and be disappointed at their state in the Python ecosystem. Maybe - but at the same time, generators are side-effectful and quite general. That means, when someone uses a generator, now I have to understand what happens by looking at the code. If someone uses a promise or a closure/lambda, then the scope of what can happen is much more limited. It's like looping vs. mapping. You can use "for ... in ..." syntax or you can do "map(..., ...)" but I generally prefer the latter (minus the syntax overhead in python), simply because I easily see at a glance what the intend is.
omegalulw 5y ago
A big problem with many of these features is that they almost always end up getting used in ways that makes the code more convoluted when there are simpler alternatives. needing to store state - use a class needing lazy evalutaion - use a class (though this is a common enough use case to be a primitive, e.g. Kotlin) decorators - if you pass decorated methods around -> now you have to remember the context the methods hold. Instead, refactor your code to be more direct so that you don't need decorators. type hints - they are not fully enforceable at compile time, unless everything in your codebase has type hints. This is never the case and type hints can easily lull you into a false sense of security. functools module - generally I'm ok with this but sonetimes you have to be aware of when you need to deep copy vs when you don't etc.

drcongo 5y ago

I don't know many languages, do decorators show up in any others? I _love_ them in Python.

BiteCode_dev 5y ago
Yes. In fact, E.G: C# and JS have them, although they don't have exactly the same features. edit, sorry I've meant generators. Incidentally, it's also true for js decorators.
- k__ 5y ago
  How do Python generators differ from JavaScript generators?
  - BiteCode_dev 5y ago
    Yield from comes to mind.
    - k__ 5y ago
      Isn't JS yield* the same as yield from in Python
      - BiteCode_dev 5y ago
        Unless I'm mistaken, it propagate pulling, but not pushing with send(). I haven't checked though.
- goatlover 5y ago
  Decorators are still a pending proposal in ECMAScript, so they would only be available to transpilers. That makes them non-standard until the proposal is approved, where the implementation could change.
useerup 5y ago
In C#: static IEnumerable<int> Fibonacci() { int first = 0; int second = 1; yield return first; yield return second; while (true) { (first, second) = (second, second + first); yield return second; } }
- whoisthemachine 5y ago
  Interesting use of tuples I had never considered before, a lot of state is packed in that `(first, second)...` block! Pretty fun though.
b3morales 5y ago
Emacs Lisp* has a system called "advice" that's maybe a bit clumsier than Python's, but is also more powerful. The relationship between decorator and decorated [is configurable][0]. For example, the decorated function could be called conditionally on the result of applying the decorator to the arguments, something like: def _combined(args): return decorator(args) and original(args) It can also be applied retroactively, not just at the decorated function's definition. This is extremely handy in the context of configuring your text editor, but I admit I wouldn't like it in a program I was working on with other people. (Basically a form of monkeypatching.) * Granted not really used for general-purpose programming like Python. [0]:https://www.gnu.org/software/emacs/manual/html_node/elisp/Advice-Combinators.html https://www.gnu.org/software/emacs/manual/html_node/elisp/Ad...
- creamytaco 5y ago
  Emacs Lisp has macros which are a lot more elegant (not to mention powerful) than Python decorators which comparatively, are nothing more than a kludge.
krick 5y ago
Depends on what you mean. I often hear claims that you can use decorators in language X, which are formally true, since decorator is just a function accepting and returning another function, so any language that has anonymous functions (which is a lot of languages by now) "has decorators". But for me this is kind of false marketing, because there's nothing impressive about that, and the killer feature of Python's decorators is the syntactic sugar, which makes them so easy to use. And in this sense not so many languages "have decorators". You can do something similar with annotations in PHP, but I've never actually done this in a real project, because that feels like abusing the language.

ajuc 5y ago

Generators are such a game-changer for clean separation of data traversal from data processing. It's my favorite trick in python and I miss it dearly in other languages. In languages without generators it's easy to write code that traverses some data structure in hard-coded way and does something configurable on every node, but it's hard to write code that traverses a data structure in configurable way and does some hardcoded thing on every element. In Python it's trivial both ways and even both at once.

idle_zealot 5y ago
> it's hard to write code that traverses a data structure in configurable way and does some hardcoded thing on every element Wouldn’t this just be a function that operates on an iterator? I suppose generators make the creation of lazy iterators easier, but generally the solution for languages without generators is to have your traversal build a list. So you lose the laziness but the code remains simple. Then map your per-node processing to the list.
- ajuc 5y ago
  > generally the solution for languages without generators is to have your traversal build a list Yes, but then you move from O(1) to O(n) in memory usage. And if you want to optimize it you have to restructure your code, when in python lists and generators are drop-in replacements.
  - justinpombrio 5y ago
    Also, producing a list doesn't work very well as a substitute for infinite generators.
  - greggyb 5y ago
    Until you want to walk the generator twice -- stateful generators are a design flaw, in my opinion.
muxator 5y ago
The C++20 coroutines has the potential of being this and much more. As with many things C++, its adoption will be very slow (they are supported by latest compiler versions, but there is no support in the standard libraries yet).

andrewstuart 5y ago

>> If you are used to call backs, promises, tick() calls or closures, it will be natural to reach for them a lot, and be disappointed at their state in the Python ecosystem. It's a real disappointment that Python has no answer for JavaScript fat arrow. The fat arrow is the one construct that completely changes the structure of all the programs I write - it makes things extremely terse and I use it in nearly every function/method in some way. I'm a big fan of Python's whitespace structure but it seems to be an obstacle to creating single line functions that are both powerful and get multiple things done, which sometimes makes a program easier to read and more understandable. Sure there's lambda functions but I find them clumsy and hard to work with compared to the beauty of the fat arrow.

BiteCode_dev 5y ago
The rare case where you do need a fat arrow, it is indeed elegant once misses from Python. But I also know by experience that people from other languages are reaching for anonymous functions way too often instead of using the tools that are suitable for the use case in python. E.G: if you use a lot of map() and filter() in python, you are doing it wrong.

anonymousDan 5y ago

One thing that really annoys me about PySpark is Python's lack of tuple unpacking in lambdas. It turns what should be relatively readable pipelines of transformations into an unreadable mess.

canjobear 5y ago
Alas, Python 2 had tuple unpacking, but they removed it in Python 3 :(

valenterry 5y ago

> If you are used to call backs, promises, tick() calls or closures, it will be natural to reach for them a lot, and be disappointed at their state in the Python ecosystem. Maybe - but at the same time, generators are side-effectful and quite general. That means, when someone uses a generator, now I have to understand what happens by looking at the code. If someone uses a promise or a closure/lambda, then the scope of what can happen is much more limited. It's like looping vs. mapping. You can use "for ... in ..." syntax or you can do "map(..., ...)" but I generally prefer the latter (minus the syntax overhead in python), simply because I easily see at a glance what the intend is.

omegalulw 5y ago

A big problem with many of these features is that they almost always end up getting used in ways that makes the code more convoluted when there are simpler alternatives. needing to store state - use a class needing lazy evalutaion - use a class (though this is a common enough use case to be a primitive, e.g. Kotlin) decorators - if you pass decorated methods around -> now you have to remember the context the methods hold. Instead, refactor your code to be more direct so that you don't need decorators. type hints - they are not fully enforceable at compile time, unless everything in your codebase has type hints. This is never the case and type hints can easily lull you into a false sense of security. functools module - generally I'm ok with this but sonetimes you have to be aware of when you need to deep copy vs when you don't etc.