Great comment! We're pretty much in agreement with all of this. > can I query the same dataset at two different PIT using layered querying in SQL Yes. We have

Great comment! We're pretty much in agreement with all of this. > can I query the same dataset at two different PIT using layered querying in SQL Yes. We have a query that does this on our home page (it's the second in the carousel at the top): SELECT new.domain AS domain, new.id AS id, new.name AS name FROM "splitgraph/socrata:20200809".datasets old RIGHT OUTER JOIN "splitgraph/socrata:20200810".datasets new This is querying two versions of the same image. > are you storing the entire dataset as new or only the diff between versions Basically we store the diffs between versions, yes. The simplified explanation is that data images are an initial snapshot + chain of diffs ("delta-compressed"). You can read more in the concepts section of the docs. [0] > Git-like semantics (pull, push, checkout, commit) are poorly suited for versioned, immutable datasets We pretty much agree with this. Some similar projects take the wrong approach and try to implement 100% of git commands "for data." But it's a square peg trying to fit into a round hole. The use cases are not the same. There are some similarities, but they're limited. Our goal was to step back and think from first principles: What do we like about our coding tools? How can we get the same benefits with data? > Versions should be at least partially ordered and monotonically increasing. Hashes doesn't convey the information necessary to decide if dataset:de4d is an earlier version of dataset:123a, or not. Sure. If you want monotonically increasing versions, you can do that by tagging an image with `sgr tag`. (FWIW, image hashes are currently random, but the hashes of objects that comprise them are not -- they're actually content addressable representations of the changeset from the root to that version using an algorithm called LTHash.) > Tracing a derived dataset provenance will only work if you can assert that the "code" or transformations applied to the original dataset is deterministic (side-effect free). Yes, definitely. Dataset provenance works best when you use Splitfiles [1] to construct images. It's still possible to run into problems with non-determinism, but for the OLAP use case, we think those situations should be rare. [0] https://www.splitgraph.com/docs/concepts/images https://www.splitgraph.com/docs/concepts/images [1] https://www.splitgraph.com/docs/concepts/splitfiles https://www.splitgraph.com/docs/concepts/splitfiles