3 ms·

As someone who tried, and almost succeeded, to get rid of pachyderm for the last two years, I like what I just read. Something is not entirely clear to me righ

by Fiahil 6y ago

As someone who tried, and almost succeeded, to get rid of pachyderm for the last two years, I like what I just read. Something is not entirely clear to me right now: An image is an immutable snapshot of a dataset at a given point-in-time - great - but, can I query the same dataset at two different PIT using layered querying in SQL ? Something like this: SELECT * FROM dataset:version-1, dataset:version-2 Also, are you storing the entire dataset as new or only the diff between versions (and later reconstruct the full image) ? Now, onto the things that could be improved... - Git-like semantics (pull, push, checkout, commit) are poorly suited for versioned, immutable datasets. Just (intelligently) abstract fetching and sending datasets by looking at the SQL query (dataset:version-2, above) - Versions should be at least partially ordered and monotonically increasing. Hashes doesn't convey the information necessary to decide if dataset:de4d is an earlier version of dataset:123a, or not. - Tracing a derived dataset provenance will only work if you can assert that the "code" or transformations applied to the original dataset is deterministic (side-effect free). So, either you have your own ETL language that you can execute in a sandbox and add a myriad of useless stuff for creating and scheduling pipelines (please don't do that!), or you just let it go and don't end up becoming Pachyderm (sounds great!).

eadan 6y ago
Is something like dolthub [0] what you have in mind? [0] https://www.dolthub.com/ https://www.dolthub.com/
- Fiahil 6y ago
  No. They embraced the "Git for data" mindset, like Pachyderm and DVC. This design is restricted by git semantics when we need to go further. Think more about Kafka's topic/message objects, with each message being an entire table, and their Stream API
chatmasta 6y ago
Great comment! We're pretty much in agreement with all of this. > can I query the same dataset at two different PIT using layered querying in SQL Yes. We have a query that does this on our home page (it's the second in the carousel at the top): SELECT new.domain AS domain, new.id AS id, new.name AS name FROM "splitgraph/socrata:20200809".datasets old RIGHT OUTER JOIN "splitgraph/socrata:20200810".datasets new This is querying two versions of the same image. > are you storing the entire dataset as new or only the diff between versions Basically we store the diffs between versions, yes. The simplified explanation is that data images are an initial snapshot + chain of diffs ("delta-compressed"). You can read more in the concepts section of the docs. [0] > Git-like semantics (pull, push, checkout, commit) are poorly suited for versioned, immutable datasets We pretty much agree with this. Some similar projects take the wrong approach and try to implement 100% of git commands "for data." But it's a square peg trying to fit into a round hole. The use cases are not the same. There are some similarities, but they're limited. Our goal was to step back and think from first principles: What do we like about our coding tools? How can we get the same benefits with data? > Versions should be at least partially ordered and monotonically increasing. Hashes doesn't convey the information necessary to decide if dataset:de4d is an earlier version of dataset:123a, or not. Sure. If you want monotonically increasing versions, you can do that by tagging an image with `sgr tag`. (FWIW, image hashes are currently random, but the hashes of objects that comprise them are not -- they're actually content addressable representations of the changeset from the root to that version using an algorithm called LTHash.) > Tracing a derived dataset provenance will only work if you can assert that the "code" or transformations applied to the original dataset is deterministic (side-effect free). Yes, definitely. Dataset provenance works best when you use Splitfiles [1] to construct images. It's still possible to run into problems with non-determinism, but for the OLAP use case, we think those situations should be rare. [0] https://www.splitgraph.com/docs/concepts/images https://www.splitgraph.com/docs/concepts/images [1] https://www.splitgraph.com/docs/concepts/splitfiles https://www.splitgraph.com/docs/concepts/splitfiles
- Fiahil 6y ago
  Ah ! I initially missed the splitfiles page ! Declarative, cacheable... It's a great approach you have here. I think you could go even further and infer some SQL transformations from the schema definition. Anyway, I'm curious to see where your project will end up. You put much more efforts in the design than most "for data" tools ! Cheers !
ManWith2Plans 6y ago
Maybe a JCR implementation like Apache Jackrabbit Oak would be a good tool for this data. See here for sql semantics: https://jackrabbit.apache.org/oak/docs/query/grammar-sql2.html https://jackrabbit.apache.org/oak/docs/query/grammar-sql2.ht...
didip 6y ago
What's wrong with Pachyderm? Sounds like you are having a lot of pain with them. I am not affiliated at all with Pachyderm, I am just curious about this problem-space.
- Fiahil 6y ago
  There is a good number of specific reasons, but it really comes down to this : they have too many features.

by Fiahil 6y ago

eadan 6y ago

Is something like dolthub [0] what you have in mind? [0] https://www.dolthub.com/ https://www.dolthub.com/

Fiahil 6y ago
No. They embraced the "Git for data" mindset, like Pachyderm and DVC. This design is restricted by git semantics when we need to go further. Think more about Kafka's topic/message objects, with each message being an entire table, and their Stream API

chatmasta 6y ago

Great comment! We're pretty much in agreement with all of this. > can I query the same dataset at two different PIT using layered querying in SQL Yes. We have a query that does this on our home page (it's the second in the carousel at the top): SELECT new.domain AS domain, new.id AS id, new.name AS name FROM "splitgraph/socrata:20200809".datasets old RIGHT OUTER JOIN "splitgraph/socrata:20200810".datasets new This is querying two versions of the same image. > are you storing the entire dataset as new or only the diff between versions Basically we store the diffs between versions, yes. The simplified explanation is that data images are an initial snapshot + chain of diffs ("delta-compressed"). You can read more in the concepts section of the docs. [0] > Git-like semantics (pull, push, checkout, commit) are poorly suited for versioned, immutable datasets We pretty much agree with this. Some similar projects take the wrong approach and try to implement 100% of git commands "for data." But it's a square peg trying to fit into a round hole. The use cases are not the same. There are some similarities, but they're limited. Our goal was to step back and think from first principles: What do we like about our coding tools? How can we get the same benefits with data? > Versions should be at least partially ordered and monotonically increasing. Hashes doesn't convey the information necessary to decide if dataset:de4d is an earlier version of dataset:123a, or not. Sure. If you want monotonically increasing versions, you can do that by tagging an image with `sgr tag`. (FWIW, image hashes are currently random, but the hashes of objects that comprise them are not -- they're actually content addressable representations of the changeset from the root to that version using an algorithm called LTHash.) > Tracing a derived dataset provenance will only work if you can assert that the "code" or transformations applied to the original dataset is deterministic (side-effect free). Yes, definitely. Dataset provenance works best when you use Splitfiles [1] to construct images. It's still possible to run into problems with non-determinism, but for the OLAP use case, we think those situations should be rare. [0] https://www.splitgraph.com/docs/concepts/images https://www.splitgraph.com/docs/concepts/images [1] https://www.splitgraph.com/docs/concepts/splitfiles https://www.splitgraph.com/docs/concepts/splitfiles

Fiahil 6y ago
Ah ! I initially missed the splitfiles page ! Declarative, cacheable... It's a great approach you have here. I think you could go even further and infer some SQL transformations from the schema definition. Anyway, I'm curious to see where your project will end up. You put much more efforts in the design than most "for data" tools ! Cheers !

ManWith2Plans 6y ago

Maybe a JCR implementation like Apache Jackrabbit Oak would be a good tool for this data. See here for sql semantics: https://jackrabbit.apache.org/oak/docs/query/grammar-sql2.html https://jackrabbit.apache.org/oak/docs/query/grammar-sql2.ht...

didip 6y ago

What's wrong with Pachyderm? Sounds like you are having a lot of pain with them. I am not affiliated at all with Pachyderm, I am just curious about this problem-space.

Fiahil 6y ago
There is a good number of specific reasons, but it really comes down to this : they have too many features.