3 ms·
Looks very interesting guys, thank you a lot to release that. I was working on something similar currently. Any data pipeline complex enough finish soon or la
by adev_ 6y ago
Looks very interesting guys, thank you a lot to release that.
I was working on something similar currently.
Any data pipeline complex enough finish soon or later multi-lingual (Python, R, C++ + MPI) and multi-runtime (JVM, native, python) and it becomes quickly impossible to execute everything in one single process space without problems.
You are right in your design, shared memory node-to-node data distribution is the answer to that to avoid the classical / inefficient data dump-load-dump pattern that we find usually in most heterogeneous pipeline.
- sighingnow 6y agoThanks for liking this project! Your understanding is correct. If you want to give vineyard a try, you are welcome to drop me an email (sighingnow@gmail.com), I can help you set it up.