3 ms·
> why would I use this instead? I can't speak for the developer, but a few things stood out to me: 1) It's Python. Which means there's no impedance mismatch i
by wenc 8y ago
> why would I use this instead?
I can't speak for the developer, but a few things stood out to me:
1) It's Python. Which means there's no impedance mismatch in using numerical/data science libraries like Numpy and the like. For data engineers trying to productionize data science workloads, this is quite compelling -- no need to throw away all of the Python code written by data scientists. This also lowers the barrier to entry for streaming code.
2) I had the same reservations about CPU efficiency, but it looks like they're using best-of-class libraries (RocksDB is C++, uvloop is C/Cython, etc.). I was at a PyCon talk where the speaker demo'ed Dask (a distributed computation system similar to Spark) running on Kubernetes and it was very impressive. Scalability didn't seem to be an issue. Dask actually outperformed Spark in some instances.
I wonder if Kubernetes is the key to making these types of solutions competitive with traditional JVM type distributed applications like Spark Streaming, etc.
3) Not all streaming data is real-time. In fact, streaming just means unbounded data with no inherent stipulation of near real-time SLAs. Real-time is actually a much stricter requirement.
- asksol 8y agoMy name is Ask and I'm co-creator on this project, along with Vineet Goel. This answer sums up why we wanted to use Python for this project: it's the most popular language for data science and people can learn it quickly. Performance is not really a problem either, with Python 3 and asyncio we can process tens of thousands of events/s. I have seen 50k events/s, and there are still many optimizations that can be made. That said, Faust is not just for data science. We use it to write backend services that serve WebSockets and HTTP from the same Faust worker instances that process the stream.