3 ms·

I have used this quite a bit, really enjoy it, but use it for a very limited use case. Use case is basically saving versioned datasets for supervised ML. Featur

I have used this quite a bit, really enjoy it, but use it for a very limited use case. Use case is basically saving versioned datasets for supervised ML. Features I would like to be added (or features I don't know exist): Get multiple datasets into one place, i.e. I want datasets A, B, and C in one place for use, rather than downloading them all separately and then combining them, mix and match! Also, some built in meta about the dataset, class distribution/class map, things like that, so I can intelligently pick what datasets I might need at any given time. Combined these two things would be like a model zoo or something, for data.

chatmasta 6y ago
(Disclaimer: Splitgraph co-founder) > I want datasets A, B, and C in one place for use, rather than downloading them all separately and then combining them, mix and match This is the use case we're focused on at Splitgraph [0]. We think data is most interesting when combined with other data, so it's important to be able to reference multiple datasets in one query. The Splitgraph engine is powered by Postgres and allows you to build versioned "data images," which are snapshots of databases (like Docker images are snapshots of filesystems). You can "mount" multiple databases in one engine [1] and query them with regular SQL, or you can write a Splitfile [2] that references both of them when "importing" data into an image. This can be really useful at times. We have a blog post about mounting multiple databases and using dbt for cross-DB transformations. [3] Ultimately, our goal is to build a "data delivery network" -- a single SQL endpoint where you can query any databases connected to Splitgraph (whether live, or cached using Splitgraph images). [0] https://www.splitgraph.com https://www.splitgraph.com [1] https://www.splitgraph.com/docs/ingesting-data/foreign-data-wrappers/introduction https://www.splitgraph.com/docs/ingesting-data/foreign-data-... [2] https://www.splitgraph.com/docs/concepts/splitfiles https://www.splitgraph.com/docs/concepts/splitfiles [3] https://splitgraph.com/blog/dbt https://splitgraph.com/blog/dbt
- dmpetrov 6y ago
  It is important to know what are A and B. Splitgraph\dbt\Dolt can work if these are structured tables in SQL database while DVC works if A and B are files or directories.
dmpetrov 6y ago
Thanks you for warm words about DVC :) `dvc import` can connect multiple datasets in a repository. It also preserves the "link" to the dataset that you can update/downgrade. Class/labels distributions for a dataset can be implemented with `dvc metrics` and new feature `dvc plots`. However, you will need to introduce a dvc pipeline, like an dataset evaluation step. Re the model zoo for data - yes, we're working on "higher level" data artifacts (like ML models on top of the basic data management layer). Stay tuned for a release soon. PS: I'm one of the DVC creators.

chatmasta 6y ago

(Disclaimer: Splitgraph co-founder) > I want datasets A, B, and C in one place for use, rather than downloading them all separately and then combining them, mix and match This is the use case we're focused on at Splitgraph [0]. We think data is most interesting when combined with other data, so it's important to be able to reference multiple datasets in one query. The Splitgraph engine is powered by Postgres and allows you to build versioned "data images," which are snapshots of databases (like Docker images are snapshots of filesystems). You can "mount" multiple databases in one engine [1] and query them with regular SQL, or you can write a Splitfile [2] that references both of them when "importing" data into an image. This can be really useful at times. We have a blog post about mounting multiple databases and using dbt for cross-DB transformations. [3] Ultimately, our goal is to build a "data delivery network" -- a single SQL endpoint where you can query any databases connected to Splitgraph (whether live, or cached using Splitgraph images). [0] https://www.splitgraph.com https://www.splitgraph.com [1] https://www.splitgraph.com/docs/ingesting-data/foreign-data-wrappers/introduction https://www.splitgraph.com/docs/ingesting-data/foreign-data-... [2] https://www.splitgraph.com/docs/concepts/splitfiles https://www.splitgraph.com/docs/concepts/splitfiles [3] https://splitgraph.com/blog/dbt https://splitgraph.com/blog/dbt

dmpetrov 6y ago
It is important to know what are A and B. Splitgraph\dbt\Dolt can work if these are structured tables in SQL database while DVC works if A and B are files or directories.

dmpetrov 6y ago

Thanks you for warm words about DVC :) `dvc import` can connect multiple datasets in a repository. It also preserves the "link" to the dataset that you can update/downgrade. Class/labels distributions for a dataset can be implemented with `dvc metrics` and new feature `dvc plots`. However, you will need to introduce a dvc pipeline, like an dataset evaluation step. Re the model zoo for data - yes, we're working on "higher level" data artifacts (like ML models on top of the basic data management layer). Stay tuned for a release soon. PS: I'm one of the DVC creators.