2 ms·
We're building something similar at Splitgraph [0], although we actually try to avoid reductive descriptors like "Git for data" and "GitHub for data." Forcing t
by chatmasta 6y ago
We're building something similar at Splitgraph [0], although we actually try to avoid reductive descriptors like "Git for data" and "GitHub for data." Forcing the Git workflow "for data" is the wrong approach, and we don't believe it's what analytics engineers actually want to use.
Our solution is a "data delivery network" [1] that looks like a big Postgres database with 40,000+ datasets in it. So you can connect to it [2] with any SQL client that is compatible with the Postgres wire protocol. You can query individual datasets, or join across them, as if they were tables in a Postgres database. Those datasets can be "live" (in which case we forward queries to the upstream data source, e.g. a government data portal), or they can be versioned snapshots called "data images." Data images are a lot like Docker images, and you can build them with declarative recipes called Splitfiles, all using our open source tool. [3] Then you can push the images to Splitgraph and query them alongside all the other data on the platform.
The public platform is mostly community data right now. Soon you'll be able to connect arbitrary upstreams to it (e.g. connect directly to a Mongo/Postgres/MySQL database by providing its read-only credentials in the web UI). As an example, we have a repository that forwards queries to the OxCOVID19 database, which is a Postgres database maintained by a lab at Oxford. [4]
We're also building an enterprise program for hosting private deployments of Splitgraph. We have a pilot program where customers can use Splitgraph as a data cataloging and governance solution for their internal company data. We've also raised a round of funding but haven't announced it yet. If anyone is interested in joining the pilot program, get in touch. You can also read more details about how we dogfood Splitgraph in our own analytics stack. [5]
[0] https://www.splitgraph.com/ https://www.splitgraph.com/
[1] https://www.splitgraph.com/blog/data-delivery-network-launch https://www.splitgraph.com/blog/data-delivery-network-launch
[2] https://www.splitgraph.com/connect https://www.splitgraph.com/connect
[3] https://www.github.com/splitgraph/splitgraph https://www.github.com/splitgraph/splitgraph
[4] https://www.splitgraph.com/splitgraph/oxcovid19 https://www.splitgraph.com/splitgraph/oxcovid19
[5] https://www.splitgraph.com/blog/splitgraph-matomo-elasticsearch-metabase https://www.splitgraph.com/blog/splitgraph-matomo-elasticsea...