4 ms·
(Disclaimer: Splitgraph co-founder) > I want datasets A, B, and C in one place for use, rather than downloading them all separately and then combining them, mi
by chatmasta 6y ago
(Disclaimer: Splitgraph co-founder)
> I want datasets A, B, and C in one place for use, rather than downloading them all separately and then combining them, mix and match
This is the use case we're focused on at Splitgraph [0]. We think data is most interesting when combined with other data, so it's important to be able to reference multiple datasets in one query.
The Splitgraph engine is powered by Postgres and allows you to build versioned "data images," which are snapshots of databases (like Docker images are snapshots of filesystems). You can "mount" multiple databases in one engine [1] and query them with regular SQL, or you can write a Splitfile [2] that references both of them when "importing" data into an image.
This can be really useful at times. We have a blog post about mounting multiple databases and using dbt for cross-DB transformations. [3]
Ultimately, our goal is to build a "data delivery network" -- a single SQL endpoint where you can query any databases connected to Splitgraph (whether live, or cached using Splitgraph images).
[0] https://www.splitgraph.com https://www.splitgraph.com
[1] https://www.splitgraph.com/docs/ingesting-data/foreign-data-wrappers/introduction https://www.splitgraph.com/docs/ingesting-data/foreign-data-...
[2] https://www.splitgraph.com/docs/concepts/splitfiles https://www.splitgraph.com/docs/concepts/splitfiles
[3] https://splitgraph.com/blog/dbt https://splitgraph.com/blog/dbt
- dmpetrov 6y agoIt is important to know what are A and B. Splitgraph\dbt\Dolt can work if these are structured tables in SQL database while DVC works if A and B are files or directories.