3 ms·

The whole article properly the best explanation of generators I have come across. This quote stuck out: > Generators are a special type of function that can re

The whole article properly the best explanation of generators I have come across. This quote stuck out: > Generators are a special type of function that can return multiple pieces of data during its execution. Traditional functions can return multiple data by using structures like Arrays and Objects, but Generators return data whenever the caller asks for it, and they pause execution until they are asked to continue to generate and return more data. Applications of generators? I have only used Redux-Saga[1]. Can't even think of other libraries that use them, but would be interested in learning. [1]: https://redux-saga.js.org/ https://redux-saga.js.org/

littlestymaar 3y ago
I think seeing generators as special functions makes them feel more magic than what they really are: generators are nothing more than a syntactic sugar for objects with a “call” method that will change the values of the private fields in this object, that's why you can call it repeatedly with different results. Some patterns become easier to write with the generator syntax, but it's not really adding expressive power. (Unlike futures for instance)
- heavenlyblue 3y ago
  You could also have an object with on_success, on_failure attributes so Futures are the same syntax sugar
  - littlestymaar 3y ago
    In JavaScript, where you have first class functions and an event loop built-in, yes. But for languages that lack one of those, then no.
- btown 3y ago
  IMO this is like saying a high-level language is syntactic sugar for assembly. Generators allow you to write a state machine as a procedural function whose local variables are automatically suspended when you yield and restored when the caller returns control to the generator. They compile this down to, as you said, an object with a call method. But that transformation is as nontrivial as any compilation/transpilation project.
Zecc 3y ago
You can use them in any place where you want to return a bunch of values, but not necessarily all in one go. This goes for bytes in a Uint8Array, points you want to draw in canvas, DOM elements you are selecting or generating on the fy, data objects you've downloaded from somewhere, etc. The advantage of generators over arrays is that you don't have to allocate a large array, so you can handle large numbers of elements while keeping a lower memory footprint. They can also hide batching/chunking logic. You could have code that repeatedly downloads of separate JSON files with 200, or 500, or 5000 elements at a time, then return a generator that goes through each element at a time. The consuming code will be none the wiser.
heavyset_go 3y ago
Use them wherever you need an iterable or lazy-loading. You can build pipelines of generators to interleave work, too.
masklinn 3y ago
Generators are not used much in the JS ecosystem for some reason, but they're used a lot in Python. The most common application is iteration (lazy producers or transformers), but they're also used for their ability to suspend and resume function execution (interruptible functions) e.g. - pytest fixtures use generators to setup, allow the test to run (yielding a value or not), and teardown - contextlib.contextmanager uses generators for similar purposes of entering a context, allowing the wrapped code to run, and then exiting the context They can also act as cheap state managers, especially with the ability to inject data back into the generator, so they can be used to implement a multi-step function where the user needs multiple points of interaction, without needing a bunch of callbacks, or complicated state management (especially without the benefit of sum types or static state machines)
- athanagor2 3y ago
  I used generators in the way you described in your last paragraph. And I added the possibility to "replay" the generators with the previous inputs except the last one. It provides a cheap way to have a kind of multi-step input process with cancellation.
roggenilsson 3y ago
You can also think of generators as a native implementation of Observables from rx (except you can't replay a generator), especially async generators. You can implement basic operators like map, filter, take, etc. over generators to create pipelines of operations. Very neat abstraction to work with, but like rx, can quickly get hard to reason about. Recently I wrote some tooling to read, do some operations, and write hundreds of thousands of files locally. Using generators solved having to think about not loading too much stuff into memory since it only yields files when consumed. Also allows you simply implement stuff like batching, like running X requests to a server at a time, and only starting the next batch once the first one is done.
- Tomuus 3y ago
  Observables and Generators (iterators) are fundamentally different. Observables are push-based (like a promise) whereas iterators are pull-based (like a function). Glossing over this fact leads to a flawed understanding, not a deeper one.
qudat 3y ago
redux-saga maintainer here! Before async/await, generators were the only way to express async flow control in a synchronous way. Libraries like “co” were really popular at the time. There are some really downside to async/await which is why redux-saga continued to use generators. Even in 2023, me and a couple others are experimenting with deliminited continuations using generators as the foundation. A couple of libraries we have built: - https://github.com/thefrontside/continuation https://github.com/thefrontside/continuation - https://github.com/thefrontside/effection/tree/v3 https://github.com/thefrontside/effection/tree/v3 - https://github.com/neurosnap/starfx https://github.com/neurosnap/starfx The last one intends to replace redux-saga using DCs. Here’s a presentation I gave recently talking about DCs in typescript: https://youtu.be/uRbqLGj_6mI?si=XI0JNMKMoO2VHMvM https://youtu.be/uRbqLGj_6mI?si=XI0JNMKMoO2VHMvM
davidmurdoch 3y ago
I use generators to stringify a huge amount (several gigabytes) of generated JSON for transmission over http/ws. Node can only handle about 1-2 gigs in a single string or Buffer -- it's a hard-coded limit depending on CPU architecture, AFAIK. So these huge JSON blobs need to be chunked. A generator is nice here because it makes it easy for the code that converts the JSON for transmission to wait until the OS has transmitted the previous chunks -- which helps avoid catastrophic back pressure and runaway memory use.
mistercow 3y ago
Generators are a super clean way to do stream processing (in fact, in modern node, streams are async iterators, and can be used with “for await”). The really nice thing is that you can chain together a whole pipeline of generators to do transformations on a stream which are much more complex than map. For example, do you have a stream of chunks, and want to turn it into a stream of lines? The implementation of this with generators is super clean and readable, because it’s like you’re just iterating over a list.
emadda 3y ago
A nice use of async generators is when you have an API with paginated results. The generator function keeps track of the current page. This allows you to use a `for await` loop which is quite concise. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... You can do it without generator functions, but generator functions allows the caller to not have to know about the underlying pagination API.

littlestymaar 3y ago

I think seeing generators as special functions makes them feel more magic than what they really are: generators are nothing more than a syntactic sugar for objects with a “call” method that will change the values of the private fields in this object, that's why you can call it repeatedly with different results. Some patterns become easier to write with the generator syntax, but it's not really adding expressive power. (Unlike futures for instance)

heavenlyblue 3y ago
You could also have an object with on_success, on_failure attributes so Futures are the same syntax sugar
- littlestymaar 3y ago
  In JavaScript, where you have first class functions and an event loop built-in, yes. But for languages that lack one of those, then no.
btown 3y ago
IMO this is like saying a high-level language is syntactic sugar for assembly. Generators allow you to write a state machine as a procedural function whose local variables are automatically suspended when you yield and restored when the caller returns control to the generator. They compile this down to, as you said, an object with a call method. But that transformation is as nontrivial as any compilation/transpilation project.

Zecc 3y ago

You can use them in any place where you want to return a bunch of values, but not necessarily all in one go. This goes for bytes in a Uint8Array, points you want to draw in canvas, DOM elements you are selecting or generating on the fy, data objects you've downloaded from somewhere, etc. The advantage of generators over arrays is that you don't have to allocate a large array, so you can handle large numbers of elements while keeping a lower memory footprint. They can also hide batching/chunking logic. You could have code that repeatedly downloads of separate JSON files with 200, or 500, or 5000 elements at a time, then return a generator that goes through each element at a time. The consuming code will be none the wiser.

heavyset_go 3y ago

Use them wherever you need an iterable or lazy-loading. You can build pipelines of generators to interleave work, too.

masklinn 3y ago

Generators are not used much in the JS ecosystem for some reason, but they're used a lot in Python. The most common application is iteration (lazy producers or transformers), but they're also used for their ability to suspend and resume function execution (interruptible functions) e.g. - pytest fixtures use generators to setup, allow the test to run (yielding a value or not), and teardown - contextlib.contextmanager uses generators for similar purposes of entering a context, allowing the wrapped code to run, and then exiting the context They can also act as cheap state managers, especially with the ability to inject data back into the generator, so they can be used to implement a multi-step function where the user needs multiple points of interaction, without needing a bunch of callbacks, or complicated state management (especially without the benefit of sum types or static state machines)

athanagor2 3y ago
I used generators in the way you described in your last paragraph. And I added the possibility to "replay" the generators with the previous inputs except the last one. It provides a cheap way to have a kind of multi-step input process with cancellation.

roggenilsson 3y ago

You can also think of generators as a native implementation of Observables from rx (except you can't replay a generator), especially async generators. You can implement basic operators like map, filter, take, etc. over generators to create pipelines of operations. Very neat abstraction to work with, but like rx, can quickly get hard to reason about. Recently I wrote some tooling to read, do some operations, and write hundreds of thousands of files locally. Using generators solved having to think about not loading too much stuff into memory since it only yields files when consumed. Also allows you simply implement stuff like batching, like running X requests to a server at a time, and only starting the next batch once the first one is done.

Tomuus 3y ago
Observables and Generators (iterators) are fundamentally different. Observables are push-based (like a promise) whereas iterators are pull-based (like a function). Glossing over this fact leads to a flawed understanding, not a deeper one.

qudat 3y ago

redux-saga maintainer here! Before async/await, generators were the only way to express async flow control in a synchronous way. Libraries like “co” were really popular at the time. There are some really downside to async/await which is why redux-saga continued to use generators. Even in 2023, me and a couple others are experimenting with deliminited continuations using generators as the foundation. A couple of libraries we have built: - https://github.com/thefrontside/continuation https://github.com/thefrontside/continuation - https://github.com/thefrontside/effection/tree/v3 https://github.com/thefrontside/effection/tree/v3 - https://github.com/neurosnap/starfx https://github.com/neurosnap/starfx The last one intends to replace redux-saga using DCs. Here’s a presentation I gave recently talking about DCs in typescript: https://youtu.be/uRbqLGj_6mI?si=XI0JNMKMoO2VHMvM https://youtu.be/uRbqLGj_6mI?si=XI0JNMKMoO2VHMvM

davidmurdoch 3y ago

I use generators to stringify a huge amount (several gigabytes) of generated JSON for transmission over http/ws. Node can only handle about 1-2 gigs in a single string or Buffer -- it's a hard-coded limit depending on CPU architecture, AFAIK. So these huge JSON blobs need to be chunked. A generator is nice here because it makes it easy for the code that converts the JSON for transmission to wait until the OS has transmitted the previous chunks -- which helps avoid catastrophic back pressure and runaway memory use.

mistercow 3y ago

Generators are a super clean way to do stream processing (in fact, in modern node, streams are async iterators, and can be used with “for await”). The really nice thing is that you can chain together a whole pipeline of generators to do transformations on a stream which are much more complex than map. For example, do you have a stream of chunks, and want to turn it into a stream of lines? The implementation of this with generators is super clean and readable, because it’s like you’re just iterating over a list.

emadda 3y ago

A nice use of async generators is when you have an API with paginated results. The generator function keeps track of the current page. This allows you to use a `for await` loop which is quite concise. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... You can do it without generator functions, but generator functions allows the caller to not have to know about the underlying pagination API.