22 ms·

Announcing GenStage

224 points by jbernardo95 10y ago

lucidstack 10y ago
This looks incredible, congratulations to the Elixir team! Perhaps more exciting than the first part of the post is the second bit about the future. It's fantastic to see such a clear path forward for concurrency in Elixir. Definitely looking forward to GenStage.Flow
lpgauth 10y ago
Curious, how is the back pressure actually implemented? ACK message or ETS public table?
- strmpnk 10y ago
  It's demand based. The consumer will request a fixed size of input and then the producer will send up to but no more than that until more demand is made.
  - josevalim 10y ago
    In addition to that, demand is asynchronous: the consumer can request for more events, at any amount, any time it wants to. The default implementation requests more when half of the current demand is processed. So if you set :max_demand to 100, when the consumer has processed 50 events, we request 50 more.
  - lpgauth 10y ago
    Sure it's demand based, but how is it implemented? How is the demand communicated between the processes?
    - querulous 10y ago
      i believe it's similar to window updates in http2. consumers send an async message to producers with a count of messages they are willing to accept and it's up to producers to not exceed the count
    - josevalim 10y ago
      I am on my phone (sorry) but the documentation for the GenStage module has a section on the message protocol between consumers and producers. GenStage is simply one implementation of this message protocol.
poorman 10y ago
Super excited about GenStage.Flow. Might have to start another Elixir project just to try it out.
losvedir 10y ago
Congrats to the Elixir team. As I understand it GenServer is kind of a wrapper over erlang's underlying OTP genserver abstraction. (Is that correct?) What's GenStage's relationship to erlang? Does erlang have an equivalent?
- ssijak 10y ago
  It does not have it, this is new in Elixir.
- toolz 10y ago
  All OTP abstractions are based on process creation/supervision/messaging primitives. So while erlang doesn't have this abstraction, they are all based on the same primitives that both erlang and elixir use.
thibaut_barrere 10y ago
I was already diving into Elixir for ETL-based work before discovering GenStage, but now even more. Thanks José et al. for the hard work on this!
robbles 10y ago
If the back-pressure is abstracted away by this interface, how do you monitor it?
- felixgallo 10y ago
  You'd still have the regular BEAM reduction-counting backpressure in place, and it would be trivial to have your flow components report their work rates and/or demand requests to whatever monitoring service (statsd, etc.) that you might care to set up. Or to respond to state query messages like sys:get_state/1.
Dangeranger 10y ago
Could this be used to allow Elixir to load balance a third party server? For example: You have an Elixir load balancer that manages requests for three other servers and distributes load based on the 'demand' that the consumer communicates back to the balancer?
- felixgallo 10y ago
  yep. The neat thing about this is that it's composable primitives, so you could imagine having an arbitrarily shaped tree of demand-creating producer-consumers and consumers. And with distribution, it'd be possible to scale this out pretty far.
  - Dangeranger 10y ago
    That's great. Are there any plans to use these primitives to communicate with servers not written in Elixir? For example can you foresee Elixir being used to manage and coordinate heterogeneous architectures?
    - felixgallo 10y ago
      I'm on the erlang rather than the elixir side of the BEAM, but it's pretty easy and normal in the BEAM to communicate with heterogenous processes, either by implementing a dirty or regular NIF, opening a port, building a C Node or using JInterface with java, using CORBA haha, or just opening a socket.
windor 10y ago
It's definitely worth trying! I have used akka-stream once, and always expect something like that in erlang world. Finally, here it is. Thanks for the Elixir team.
rpazyaquian 10y ago
I'm not quite at the point where I can appreciate how useful this is, because I don't really understand concurrency itself. What is a good resource from which to learn about concurrency, and especially Elixir/Erlang's approach to it?
- iamd3vil 10y ago
  If you are interested in books, "Elixir in Action" is quite a nice introduction to OTP.
- macintux 10y ago
  Fred Hebert's Learn You Some Erlang is quite approachable. http://learnyousomeerlang.com/the-hitchhikers-guide-to-concurrency http://learnyousomeerlang.com/the-hitchhikers-guide-to-concu...
- OWaz 10y ago
  If you want to watch a video, check out Joe Armstrong (one of the creators of Erlang) talk about concurrency and Erlang[0]. There are a bunch of Erlang/Elixir videos [1] posted by Erlang Solutions on YouTube. [0]: https://www.youtube.com/watch?v=YaUPdgtUYko https://www.youtube.com/watch?v=YaUPdgtUYko [1]: https://www.youtube.com/c/erlangsolutions/videos https://www.youtube.com/c/erlangsolutions/videos
- vemv 10y ago
  Before diving into Erlang/Elixir, I'd make sure to learn the basics of concurrency/parallelism with more basic languages such as Java, Python or Ruby. i.e. start with the easy stuff, then with that strong foundation choose as you prefer!
  - dragonwriter 10y ago
    Honestly, learning actor-model concurrency with Erlang (and probably Elixir) is a lot easier than dealing with concurrency in the "basic" languages you list. Concurrency in Java, Python, or Ruby isn't "the easy stuff" compared to concurrency in Erlang/Elixir.
hosh 10y ago
Yep, I wished I had had access to this back when I was working on something in the Rails side. Ingesting large amounts of data from Shopify across multiple oauth access points while being limited to the PostgreSQL backend and Shopify rate limit restrictions was a pain to setup with Redis and Sidekiq. I ended up forking Sidekiq to accomplish that -- essentially using Sidekiq as a concurrency framework to build in a different set of behavior. Reading through the GenStage document, I could have implemented something better. The way I implemented it (with Ruby) left gaps of idle time, and huge amounts of data gets staged through the queues. I couldn't figure out how to make it better. Since I only had a week to come up with something while the infrastructure was melting down, at the time, it was acceptable. Looking at this though, the demand-based backpressure would work very well. There are a couple problems I think I can use this in my current work -- this is great work!
- danvayn 10y ago
  Install the Bullet gem and see if your queries look sharp!
  - hosh 10y ago
    Already did. That wasn't the issue. I get that you're trying to help, but you don't really understand the problem and why GenStage would help it. The issue isn't N+1 queries but in concurrency and reasoning with concurrency, something that Rails, and many Rails developers seem to avoid. See also: https://news.ycombinator.com/item?id=12074425 https://news.ycombinator.com/item?id=12074425

224 points by jbernardo95 10y ago

This looks incredible, congratulations to the Elixir team! Perhaps more exciting than the first part of the post is the second bit about the future. It's fantastic to see such a clear path forward for concurrency in Elixir. Definitely looking forward to GenStage.Flow

lpgauth 10y ago

Curious, how is the back pressure actually implemented? ACK message or ETS public table?

strmpnk 10y ago
It's demand based. The consumer will request a fixed size of input and then the producer will send up to but no more than that until more demand is made.
- josevalim 10y ago
  In addition to that, demand is asynchronous: the consumer can request for more events, at any amount, any time it wants to. The default implementation requests more when half of the current demand is processed. So if you set :max_demand to 100, when the consumer has processed 50 events, we request 50 more.
- lpgauth 10y ago
  Sure it's demand based, but how is it implemented? How is the demand communicated between the processes?
  - querulous 10y ago
    i believe it's similar to window updates in http2. consumers send an async message to producers with a count of messages they are willing to accept and it's up to producers to not exceed the count
  - josevalim 10y ago
    I am on my phone (sorry) but the documentation for the GenStage module has a section on the message protocol between consumers and producers. GenStage is simply one implementation of this message protocol.

poorman 10y ago

Super excited about GenStage.Flow. Might have to start another Elixir project just to try it out.

losvedir 10y ago

Congrats to the Elixir team. As I understand it GenServer is kind of a wrapper over erlang's underlying OTP genserver abstraction. (Is that correct?) What's GenStage's relationship to erlang? Does erlang have an equivalent?

ssijak 10y ago
It does not have it, this is new in Elixir.
toolz 10y ago
All OTP abstractions are based on process creation/supervision/messaging primitives. So while erlang doesn't have this abstraction, they are all based on the same primitives that both erlang and elixir use.

thibaut_barrere 10y ago

I was already diving into Elixir for ETL-based work before discovering GenStage, but now even more. Thanks José et al. for the hard work on this!

robbles 10y ago

If the back-pressure is abstracted away by this interface, how do you monitor it?

felixgallo 10y ago
You'd still have the regular BEAM reduction-counting backpressure in place, and it would be trivial to have your flow components report their work rates and/or demand requests to whatever monitoring service (statsd, etc.) that you might care to set up. Or to respond to state query messages like sys:get_state/1.

Dangeranger 10y ago

Could this be used to allow Elixir to load balance a third party server? For example: You have an Elixir load balancer that manages requests for three other servers and distributes load based on the 'demand' that the consumer communicates back to the balancer?

felixgallo 10y ago
yep. The neat thing about this is that it's composable primitives, so you could imagine having an arbitrarily shaped tree of demand-creating producer-consumers and consumers. And with distribution, it'd be possible to scale this out pretty far.
- Dangeranger 10y ago
  That's great. Are there any plans to use these primitives to communicate with servers not written in Elixir? For example can you foresee Elixir being used to manage and coordinate heterogeneous architectures?
  - felixgallo 10y ago
    I'm on the erlang rather than the elixir side of the BEAM, but it's pretty easy and normal in the BEAM to communicate with heterogenous processes, either by implementing a dirty or regular NIF, opening a port, building a C Node or using JInterface with java, using CORBA haha, or just opening a socket.

windor 10y ago

It's definitely worth trying! I have used akka-stream once, and always expect something like that in erlang world. Finally, here it is. Thanks for the Elixir team.

rpazyaquian 10y ago

I'm not quite at the point where I can appreciate how useful this is, because I don't really understand concurrency itself. What is a good resource from which to learn about concurrency, and especially Elixir/Erlang's approach to it?

iamd3vil 10y ago
If you are interested in books, "Elixir in Action" is quite a nice introduction to OTP.
macintux 10y ago
Fred Hebert's Learn You Some Erlang is quite approachable. http://learnyousomeerlang.com/the-hitchhikers-guide-to-concurrency http://learnyousomeerlang.com/the-hitchhikers-guide-to-concu...
OWaz 10y ago
If you want to watch a video, check out Joe Armstrong (one of the creators of Erlang) talk about concurrency and Erlang[0]. There are a bunch of Erlang/Elixir videos [1] posted by Erlang Solutions on YouTube. [0]: https://www.youtube.com/watch?v=YaUPdgtUYko https://www.youtube.com/watch?v=YaUPdgtUYko [1]: https://www.youtube.com/c/erlangsolutions/videos https://www.youtube.com/c/erlangsolutions/videos
vemv 10y ago
Before diving into Erlang/Elixir, I'd make sure to learn the basics of concurrency/parallelism with more basic languages such as Java, Python or Ruby. i.e. start with the easy stuff, then with that strong foundation choose as you prefer!
- dragonwriter 10y ago
  Honestly, learning actor-model concurrency with Erlang (and probably Elixir) is a lot easier than dealing with concurrency in the "basic" languages you list. Concurrency in Java, Python, or Ruby isn't "the easy stuff" compared to concurrency in Erlang/Elixir.

hosh 10y ago

Yep, I wished I had had access to this back when I was working on something in the Rails side. Ingesting large amounts of data from Shopify across multiple oauth access points while being limited to the PostgreSQL backend and Shopify rate limit restrictions was a pain to setup with Redis and Sidekiq. I ended up forking Sidekiq to accomplish that -- essentially using Sidekiq as a concurrency framework to build in a different set of behavior. Reading through the GenStage document, I could have implemented something better. The way I implemented it (with Ruby) left gaps of idle time, and huge amounts of data gets staged through the queues. I couldn't figure out how to make it better. Since I only had a week to come up with something while the infrastructure was melting down, at the time, it was acceptable. Looking at this though, the demand-based backpressure would work very well. There are a couple problems I think I can use this in my current work -- this is great work!

danvayn 10y ago
Install the Bullet gem and see if your queries look sharp!
- hosh 10y ago
  Already did. That wasn't the issue. I get that you're trying to help, but you don't really understand the problem and why GenStage would help it. The issue isn't N+1 queries but in concurrency and reasoning with concurrency, something that Rails, and many Rails developers seem to avoid. See also: https://news.ycombinator.com/item?id=12074425 https://news.ycombinator.com/item?id=12074425