4 ms·
Dask might be lightweight internally but resorting to it just to solve a simple task that requires concurrency is not "simple". Streamz looks nice! However: "
by cgarciae 8y ago
Dask might be lightweight internally but resorting to it just to solve a simple task that requires concurrency is not "simple".
Streamz looks nice! However:
"Streamz relies on the Tornado framework for concurrency. This allows us to handle many concurrent operations cheaply and consistently within a SINGLE THREAD."
Apparently you can set it up to use Dask to escape the single threads but that is kind of a global config. With Pypeline you can mix and match between using Processes, Threads, and asyncio.Tasks where it makes sense, resource management per stage is simple and explicit. If you have some understanding of the multiprocessing, threading and asyncio modules, Pypeline will save you tons of time.
Still, will keep an eye on Streamz, its a very nice work, lots of features, it should get more visibility.
- bayesian_horse 8y agoPypeline doesn't seem that simple itself.
- unethical_ban 8y agoWhy not? It seems to be a boilerplate remover for simple parallel processing tasks.
- bayesian_horse 8y agoMaybe I'm just not quite able to get why "lightweight" is a thing. I also prefer Django over Flask for even the simplest of server software...