3 ms·

It'd be interesting to know how this compares with alternative solutions. I might not understand the benefit proposition correctly, and I'm not specifically in

It'd be interesting to know how this compares with alternative solutions. I might not understand the benefit proposition correctly, and I'm not specifically into Python for data work, but I immediately thought of things like feather[1], fst[2], disk.frame[3] and even DuckDB[4]. Some of these are on disk rather than in memory, but I'd still be interested in performance and use case comparisons. [1] https://github.com/wesm/feather https://github.com/wesm/feather [2] https://www.fstpackage.org/fst/ https://www.fstpackage.org/fst/ [3] https://diskframe.com/ https://diskframe.com/ [4] https://duckdb.org/ https://duckdb.org/

sighingnow 6y ago
Vineyard addresses on sharing "distributed abstracted immutable data" in memory, e.g., dask[1] generates a distributed dataframe (since the size of data cannot be fit into a single machine), and feed it to tensorflow as features for distributed training. Vineyard aims at sharing the "DataFrame" between these two engines, by memory sharing. + feather: as well as arrow, are focused on interoperable data stroage format. feather and arrow define a IPC serailziation schema for common data structures, e.g., tensor and DataFrame. Rather than do serailization for zero-copy (when possible) deserialization, vineyard organized data as a "metadata" and a set of "blobs", the metadata decides how to interpret those blobs and blobs are shared in a zero-copy fashion. Moreover, vineyard could manages large that cannot be fit into a single machine as a "GlobalObject" and global objects can be shared efficiently as well. + fst: fst is also a serailization/stroage format, but vineyard is not such a thing. + diskframe: diskframe is similar to fst, and works for data that cannot be fit into memory. Vineyard shares in-memory data that cannot be fit into a single machine between different engines (might be implemented in different languages). + duckdb: vineyard is not a SQL execution engine.

sighingnow 6y ago

Vineyard addresses on sharing "distributed abstracted immutable data" in memory, e.g., dask[1] generates a distributed dataframe (since the size of data cannot be fit into a single machine), and feed it to tensorflow as features for distributed training. Vineyard aims at sharing the "DataFrame" between these two engines, by memory sharing. + feather: as well as arrow, are focused on interoperable data stroage format. feather and arrow define a IPC serailziation schema for common data structures, e.g., tensor and DataFrame. Rather than do serailization for zero-copy (when possible) deserialization, vineyard organized data as a "metadata" and a set of "blobs", the metadata decides how to interpret those blobs and blobs are shared in a zero-copy fashion. Moreover, vineyard could manages large that cannot be fit into a single machine as a "GlobalObject" and global objects can be shared efficiently as well. + fst: fst is also a serailization/stroage format, but vineyard is not such a thing. + diskframe: diskframe is similar to fst, and works for data that cannot be fit into memory. Vineyard shares in-memory data that cannot be fit into a single machine between different engines (might be implemented in different languages). + duckdb: vineyard is not a SQL execution engine.