4 ms·

What would be a typical use case for this kind of technology?

sighingnow 6y ago
Let's say someone want to use python first to create a few tensors with numpy, and then use another python package networkx to create a graph from the tensors just created and next to do some graph analysis with networkx. It is easy to do it on a single machine in a single python runtime because data structures like tensors can be easily shared within a single python runtime. It will become a bit harder to share the data across processes/Python runtime efficiently without expensive IO or serialization (doable with the help of things like plasma in Apache Arrow). However, if we want to deal with "big data" that cannot be handled on a single machine, it will become very hard to share data without IO/serialization. Vineyard can be used for such cases. It makes sharing big distributed data structures easy for different runtime. There will be some added benefits with vineyard: 1. vineyard can handle the the IO, sharding/partitioning, failover and data migrations for the applications. 2. vineyard supports many out-of-the-box highly efficient data formats (such as Apache Arrow) and high-level abstractions which ease the difficulties of developing big data applications. 3. With stream data support, there is a possibility of enabling cross-process optimizations. 4. Fits K8s very well, where common data structures can live in a separated pod/container with its own resource limit.
streetcat1 6y ago
I would assume that any workload which decouple compute from storage. Most of the new data systems (e.g. Snowflake) are build on top of a data lake (E.g. S3 bucket) and a scale out JIT compute nodes. By using such tool, you can read the data from S3 once, and avoid loading the data into memory when you add nodes. A specific use case that I am working on now is Auto ML. Imagine training 100's of models on the same data.
- sighingnow 6y ago
  Thanks for your interest on this project. Training multiple models using the same data is truely a typical use case for vineyard. We will release more integeration with machine learning engines (tensorflow and pytorch) in the next major release.

Let's say someone want to use python first to create a few tensors with numpy, and then use another python package networkx to create a graph from the tensors just created and next to do some graph analysis with networkx. It is easy to do it on a single machine in a single python runtime because data structures like tensors can be easily shared within a single python runtime. It will become a bit harder to share the data across processes/Python runtime efficiently without expensive IO or serialization (doable with the help of things like plasma in Apache Arrow). However, if we want to deal with "big data" that cannot be handled on a single machine, it will become very hard to share data without IO/serialization. Vineyard can be used for such cases. It makes sharing big distributed data structures easy for different runtime. There will be some added benefits with vineyard: 1. vineyard can handle the the IO, sharding/partitioning, failover and data migrations for the applications. 2. vineyard supports many out-of-the-box highly efficient data formats (such as Apache Arrow) and high-level abstractions which ease the difficulties of developing big data applications. 3. With stream data support, there is a possibility of enabling cross-process optimizations. 4. Fits K8s very well, where common data structures can live in a separated pod/container with its own resource limit.

streetcat1 6y ago

I would assume that any workload which decouple compute from storage. Most of the new data systems (e.g. Snowflake) are build on top of a data lake (E.g. S3 bucket) and a scale out JIT compute nodes. By using such tool, you can read the data from S3 once, and avoid loading the data into memory when you add nodes. A specific use case that I am working on now is Auto ML. Imagine training 100's of models on the same data.

sighingnow 6y ago
Thanks for your interest on this project. Training multiple models using the same data is truely a typical use case for vineyard. We will release more integeration with machine learning engines (tensorflow and pytorch) in the next major release.