3 ms·

While the page does mention alternatives (Flink, Storm, Samza, Spark Streaming) it does not compare Faust to those alternatives. And it does not answer the most

While the page does mention alternatives (Flink, Storm, Samza, Spark Streaming) it does not compare Faust to those alternatives. And it does not answer the most important question for a prospective user: why would I use this instead? Ex ante Python seems like a poor choice for a large distributed system where network and CPU efficiency are important. Existing solutions have good performance, are easy to use, and thanks to broad adoption and community support, are very well maintained.

tomnipotent 8y ago
> Ex ante Python seems like a poor choice for a large distributed system where network and CPU efficiency are important Not every component in a distributed system needs to be "efficient". There is a broad class of software solving real problems that can benefit from Python having a stream processor on top of robust infrastructure.
- wirrbel 8y ago
  I would consider myself to be fairly agnostic when it comes to languages (although Python happens to be the language I work most with, which has a lot to do with what my coworkers prefer). What I can say is, that quite often, those large distributed system solutions has high constant offsets / penalties if you aren't building the BIG solution. so everything, starting from functional testing to those small tasks that happen more often than one would like take ages. I wouldn't be surprised if a python stream processing framework could provide a good developer-happiness -- efficiency ratio, just by being responsive.
wenc 8y ago
> why would I use this instead? I can't speak for the developer, but a few things stood out to me: 1) It's Python. Which means there's no impedance mismatch in using numerical/data science libraries like Numpy and the like. For data engineers trying to productionize data science workloads, this is quite compelling -- no need to throw away all of the Python code written by data scientists. This also lowers the barrier to entry for streaming code. 2) I had the same reservations about CPU efficiency, but it looks like they're using best-of-class libraries (RocksDB is C++, uvloop is C/Cython, etc.). I was at a PyCon talk where the speaker demo'ed Dask (a distributed computation system similar to Spark) running on Kubernetes and it was very impressive. Scalability didn't seem to be an issue. Dask actually outperformed Spark in some instances. I wonder if Kubernetes is the key to making these types of solutions competitive with traditional JVM type distributed applications like Spark Streaming, etc. 3) Not all streaming data is real-time. In fact, streaming just means unbounded data with no inherent stipulation of near real-time SLAs. Real-time is actually a much stricter requirement.
- asksol 8y ago
  My name is Ask and I'm co-creator on this project, along with Vineet Goel. This answer sums up why we wanted to use Python for this project: it's the most popular language for data science and people can learn it quickly. Performance is not really a problem either, with Python 3 and asyncio we can process tens of thousands of events/s. I have seen 50k events/s, and there are still many optimizations that can be made. That said, Faust is not just for data science. We use it to write backend services that serve WebSockets and HTTP from the same Faust worker instances that process the stream.
alienreborn 8y ago
Because if your topics' throughput is low (say <1000 per sec), all those tools might be overkill as most of them require a new cluster, significant setup and using new framework. This library might help quickly build consumers with little to no extra setup/infra. Also, Robinhood using this in production says this performs reasonably well and battle tested to handle some operations behind a popular app such as theirs. Definitely excited to try this out to see how this fares compared to Kafka Streams and Spark Structured streaming, both of which we use.
- vineetgoel1992 8y ago
  Not having to use systems such as Yarn/Mesos was a big motivation for us. Faust works well with any existing deployment systems (we currently use supervisor with EC2 instances) you may have and definitely made it easy for teams to start building/deploying apps. Further since Faust is a library, it can work with any existing python projects/tools - we use it both independently and alongside Django.

tomnipotent 8y ago

> Ex ante Python seems like a poor choice for a large distributed system where network and CPU efficiency are important Not every component in a distributed system needs to be "efficient". There is a broad class of software solving real problems that can benefit from Python having a stream processor on top of robust infrastructure.

wirrbel 8y ago
I would consider myself to be fairly agnostic when it comes to languages (although Python happens to be the language I work most with, which has a lot to do with what my coworkers prefer). What I can say is, that quite often, those large distributed system solutions has high constant offsets / penalties if you aren't building the BIG solution. so everything, starting from functional testing to those small tasks that happen more often than one would like take ages. I wouldn't be surprised if a python stream processing framework could provide a good developer-happiness -- efficiency ratio, just by being responsive.

wenc 8y ago

> why would I use this instead? I can't speak for the developer, but a few things stood out to me: 1) It's Python. Which means there's no impedance mismatch in using numerical/data science libraries like Numpy and the like. For data engineers trying to productionize data science workloads, this is quite compelling -- no need to throw away all of the Python code written by data scientists. This also lowers the barrier to entry for streaming code. 2) I had the same reservations about CPU efficiency, but it looks like they're using best-of-class libraries (RocksDB is C++, uvloop is C/Cython, etc.). I was at a PyCon talk where the speaker demo'ed Dask (a distributed computation system similar to Spark) running on Kubernetes and it was very impressive. Scalability didn't seem to be an issue. Dask actually outperformed Spark in some instances. I wonder if Kubernetes is the key to making these types of solutions competitive with traditional JVM type distributed applications like Spark Streaming, etc. 3) Not all streaming data is real-time. In fact, streaming just means unbounded data with no inherent stipulation of near real-time SLAs. Real-time is actually a much stricter requirement.

asksol 8y ago
My name is Ask and I'm co-creator on this project, along with Vineet Goel. This answer sums up why we wanted to use Python for this project: it's the most popular language for data science and people can learn it quickly. Performance is not really a problem either, with Python 3 and asyncio we can process tens of thousands of events/s. I have seen 50k events/s, and there are still many optimizations that can be made. That said, Faust is not just for data science. We use it to write backend services that serve WebSockets and HTTP from the same Faust worker instances that process the stream.

alienreborn 8y ago

Because if your topics' throughput is low (say <1000 per sec), all those tools might be overkill as most of them require a new cluster, significant setup and using new framework. This library might help quickly build consumers with little to no extra setup/infra. Also, Robinhood using this in production says this performs reasonably well and battle tested to handle some operations behind a popular app such as theirs. Definitely excited to try this out to see how this fares compared to Kafka Streams and Spark Structured streaming, both of which we use.

vineetgoel1992 8y ago
Not having to use systems such as Yarn/Mesos was a big motivation for us. Faust works well with any existing deployment systems (we currently use supervisor with EC2 instances) you may have and definitely made it easy for teams to start building/deploying apps. Further since Faust is a library, it can work with any existing python projects/tools - we use it both independently and alongside Django.