25 ms·

Okay, thanks! I think I got the idea. Have you seen Quilt btw? They had the same message initially - Docker for Data, packaging data, etc. Implementation was ve

Okay, thanks! I think I got the idea. Have you seen Quilt btw? They had the same message initially - Docker for Data, packaging data, etc. Implementation was very different though. I somewhat don't like this analogy btw, see how you mentioned Docker changed life for DevOps in the first place (vs engineers), the same here - data scientists don't care about packaging data - there should be strong incentive to do so. A few specific questions: 1. where does query execution happen - always client or remote as well? 2. in a global "Github for data" case is there some discovery mechanism for existing data? 3. do you provide a public storage to cover the case for Github for data? or is it now more like torrent - peers host and pay for data storage? Btw, what is major direction for you - Github (public collaboration and sharing) or internal versioned warehouses (or some other internal case?).

chatmasta 6y ago
Excellent questions, thank you. > Have you seen Quilt btw? Yes, we have. It seems in this space that everything has been done or pitched before, but in our opinion nobody has hit the exact right execution yet. A big problem with a lot of existing tools is that they disrupt your workflow, or are otherwise hard to adopt without major adjacent changes. Our core philosophy with Splitgraph is to stay out of the way. As long as we can keep this up, and as long as we can continue building on a core set of simple abstractions, we think we stand a pretty good chance. > data scientists don't care about packaging data Indeed. It's worth noting that packaging data with Splitfiles is entirely optional. You can also run ad-hoc queries against a database with change-tracking enabled (meaning, Splitgraph audit triggers are installed), and periodically commit or checkout different versions as you see fit. This workflow would be more similar to a git workflow. But we encourage the use of Splitfiles because of the advantages they add; namely reproducibility due to provenance. It's sort of like how you can build a docker image by running arbitrary commands in a container and then `docker commit`. The problem with that workflow is that you lose all the benefits of Dockerfiles. The same logic applies to `sgr commit` and Splitfiles. Our bet is that data scientists will find Splitfiles to be the path of least resistance to accomplishing their goals. > where does query execution happen - always client or remote as well? At the moment, most of it happens on the client. But in Splitgraph Cloud, we do have the capability to execute queries on the remote. In a public setting, it's obviously more desirable to push down query execution to the client (or, if it's done remotely, to charge them for it). But in a corporate setting, you could imagine a shared remote cluster that executes queries on behalf of thin clients. So, it's possible to support both, but at the moment we're focused on the client. > in a global "Github for data" case is there some discovery mechanism for existing data Splitgraph Cloud includes discovery mechanisms including search and topics. We'll be adding a lot more features around this. We intend for the "data catalog" to be a core part of our offering. > do you provide a public storage to cover the case for Github for data? or is it now more like torrent - peers host and pay for data storage? At the moment, for simplicity and while we're in beta, Splitgraph Cloud is providing storage at our discretion. However, Splitgraph is designed so that data storage is decoupled from metadata storage. You can configure `sgr` to upload objects to any S3 compatible store. Currently it's configured to upload to object storage at Splitgraph Cloud, but there is no reason we could not introduce some kind of federation protocol where users can upload to independent silos of S3-compatible storage. But this raises a lot of questions with reliability and responsibility, so we have not fully explored it yet. In the near term, the easier solution will probably be charging clients for storage at Splitgraph Cloud. But, federation is something that is technically possible and at least academically interesting. Also, note that Splitgraph Cloud does not host all the data it includes in its index. For example, the 40,000+ datasets currently in the Splitgraph index are not hosted by Splitgraph [0], but we index them, and provide value added services like a REST API that does some remote execution of queries on your behalf. Currently these use the Socrata mount point, but you could imagine a situation in a corporate environment where the catalog might index lots of databases that are not Splitgraph images, but can be mounted with an FDW in the same way. > what is major direction for you - Github (public collaboration and sharing) or internal versioned warehouses (or some other internal case?). Most likely, both. We will probably follow the GitHub model of offering a public and on-premise version of the same product. In an ideal world, companies or universities might pay to license an on-premise version of Splitgraph Cloud that includes all the same features as the public version. We've done a lot of work on our backend to make deployments like this possible, so it's an appealing direction for us. [0] https://www.splitgraph.com/docs/splitgraph-cloud/external-repositories https://www.splitgraph.com/docs/splitgraph-cloud/external-re...

chatmasta 6y ago

Excellent questions, thank you. > Have you seen Quilt btw? Yes, we have. It seems in this space that everything has been done or pitched before, but in our opinion nobody has hit the exact right execution yet. A big problem with a lot of existing tools is that they disrupt your workflow, or are otherwise hard to adopt without major adjacent changes. Our core philosophy with Splitgraph is to stay out of the way. As long as we can keep this up, and as long as we can continue building on a core set of simple abstractions, we think we stand a pretty good chance. > data scientists don't care about packaging data Indeed. It's worth noting that packaging data with Splitfiles is entirely optional. You can also run ad-hoc queries against a database with change-tracking enabled (meaning, Splitgraph audit triggers are installed), and periodically commit or checkout different versions as you see fit. This workflow would be more similar to a git workflow. But we encourage the use of Splitfiles because of the advantages they add; namely reproducibility due to provenance. It's sort of like how you can build a docker image by running arbitrary commands in a container and then `docker commit`. The problem with that workflow is that you lose all the benefits of Dockerfiles. The same logic applies to `sgr commit` and Splitfiles. Our bet is that data scientists will find Splitfiles to be the path of least resistance to accomplishing their goals. > where does query execution happen - always client or remote as well? At the moment, most of it happens on the client. But in Splitgraph Cloud, we do have the capability to execute queries on the remote. In a public setting, it's obviously more desirable to push down query execution to the client (or, if it's done remotely, to charge them for it). But in a corporate setting, you could imagine a shared remote cluster that executes queries on behalf of thin clients. So, it's possible to support both, but at the moment we're focused on the client. > in a global "Github for data" case is there some discovery mechanism for existing data Splitgraph Cloud includes discovery mechanisms including search and topics. We'll be adding a lot more features around this. We intend for the "data catalog" to be a core part of our offering. > do you provide a public storage to cover the case for Github for data? or is it now more like torrent - peers host and pay for data storage? At the moment, for simplicity and while we're in beta, Splitgraph Cloud is providing storage at our discretion. However, Splitgraph is designed so that data storage is decoupled from metadata storage. You can configure `sgr` to upload objects to any S3 compatible store. Currently it's configured to upload to object storage at Splitgraph Cloud, but there is no reason we could not introduce some kind of federation protocol where users can upload to independent silos of S3-compatible storage. But this raises a lot of questions with reliability and responsibility, so we have not fully explored it yet. In the near term, the easier solution will probably be charging clients for storage at Splitgraph Cloud. But, federation is something that is technically possible and at least academically interesting. Also, note that Splitgraph Cloud does not host all the data it includes in its index. For example, the 40,000+ datasets currently in the Splitgraph index are not hosted by Splitgraph [0], but we index them, and provide value added services like a REST API that does some remote execution of queries on your behalf. Currently these use the Socrata mount point, but you could imagine a situation in a corporate environment where the catalog might index lots of databases that are not Splitgraph images, but can be mounted with an FDW in the same way. > what is major direction for you - Github (public collaboration and sharing) or internal versioned warehouses (or some other internal case?). Most likely, both. We will probably follow the GitHub model of offering a public and on-premise version of the same product. In an ideal world, companies or universities might pay to license an on-premise version of Splitgraph Cloud that includes all the same features as the public version. We've done a lot of work on our backend to make deployments like this possible, so it's an appealing direction for us. [0] https://www.splitgraph.com/docs/splitgraph-cloud/external-repositories https://www.splitgraph.com/docs/splitgraph-cloud/external-re...