4 ms·

Has anyone used gen stage / flow for their jobs / small - medium projects ? What are some examples where this could be used ? Game server ? I found an example o

Has anyone used gen stage / flow for their jobs / small - medium projects ? What are some examples where this could be used ? Game server ? I found an example on YouTube about using it in an automation warehouse but that seems too niche to implement

qohen 9y ago
Peter Hastie, of sports-news site Bleacher Report, gave a talk at Erlang & Elixir Factory (2017) about GenStage (and Flow) -- there's a description of the talk and an embedded YouTube video at the bottom: http://www.erlang-factory.com/sfbay2017/peter-hastie.html http://www.erlang-factory.com/sfbay2017/peter-hastie.html
tortilla 9y ago
https://www.youtube.com/watch?v=Aa--NDjL9SI https://www.youtube.com/watch?v=Aa--NDjL9SI In the talk by Chris Bell, (29:15) they use Flow to migrate millions of records.
Subetai 9y ago
Flow - used it at work for an odd use case where had to merge data from 3 database sources (given to me in csv format) against other data to migrate it. It involved 100s of millions of rows and lots of unknowns so flow was perfect for processing and adaptability whereas spark or many similar solutions felt a bit heavy for the use case. GenStage specifically: we needed an internal load tester and GenStage plus easy concurrency allowed to quickly build an effective and reliable tool with rate control
ShaneWilton 9y ago
We use Elixir at Tinfoil Security for our API Security Scanner, and it uses GenStage quite a bit under the hood. It's a big simplification, but we basically have a producer that emits a stream of "scan tasks" to be performed (things like "scan this endpoint for SQL injection"), and then those are consumed by individual worker processes. It lets us super easily manage things like rate limiting and throttling, while providing backpressure so that an influx of scans (or scans on large APIs) won't overload our infrastructure.

qohen 9y ago

Peter Hastie, of sports-news site Bleacher Report, gave a talk at Erlang & Elixir Factory (2017) about GenStage (and Flow) -- there's a description of the talk and an embedded YouTube video at the bottom: http://www.erlang-factory.com/sfbay2017/peter-hastie.html http://www.erlang-factory.com/sfbay2017/peter-hastie.html

tortilla 9y ago

https://www.youtube.com/watch?v=Aa--NDjL9SI https://www.youtube.com/watch?v=Aa--NDjL9SI In the talk by Chris Bell, (29:15) they use Flow to migrate millions of records.

Subetai 9y ago

Flow - used it at work for an odd use case where had to merge data from 3 database sources (given to me in csv format) against other data to migrate it. It involved 100s of millions of rows and lots of unknowns so flow was perfect for processing and adaptability whereas spark or many similar solutions felt a bit heavy for the use case. GenStage specifically: we needed an internal load tester and GenStage plus easy concurrency allowed to quickly build an effective and reliable tool with rate control

ShaneWilton 9y ago

We use Elixir at Tinfoil Security for our API Security Scanner, and it uses GenStage quite a bit under the hood. It's a big simplification, but we basically have a producer that emits a stream of "scan tasks" to be performed (things like "scan this endpoint for SQL injection"), and then those are consumed by individual worker processes. It lets us super easily manage things like rate limiting and throttling, while providing backpressure so that an influx of scans (or scans on large APIs) won't overload our infrastructure.