3 ms·
I would assume that any workload which decouple compute from storage. Most of the new data systems (e.g. Snowflake) are build on top of a data lake (E.g. S3 bu
by streetcat1 6y ago
I would assume that any workload which decouple compute from storage.
Most of the new data systems (e.g. Snowflake) are build on top of a data lake (E.g. S3 bucket) and a scale out JIT compute nodes.
By using such tool, you can read the data from S3 once, and avoid loading the data into memory when you add nodes.
A specific use case that I am working on now is Auto ML. Imagine training 100's of models on the same data.
- sighingnow 6y agoThanks for your interest on this project. Training multiple models using the same data is truely a typical use case for vineyard. We will release more integeration with machine learning engines (tensorflow and pytorch) in the next major release.