5 ms·

I work as a SWE at a large AI consultancy. We've been experimenting with "git for data" products for a while, and we've been trying to get rid of them (notably

by Fiahil 6y ago

I work as a SWE at a large AI consultancy. We've been experimenting with "git for data" products for a while, and we've been trying to get rid of them (notably Pachyderm) for -at least- 2 years. What they all share is a) awful performances, and b) bad design. Git semantics, ("branche", "merge", "commit") are not well suited for data, because merging dataframes and creating "branches" often leads to misunderstandings and delays. Time travel is very nice to have, but it's often the case where you would like to consume your input datasets at different point in time in the same repository (unless you do one dataset per repository, but then, what's the point ?). Performances are bad, because all updates needs to go through some kind of coordination mechanism (etcd, zookeeper, or raft directly). In a single instance scenario, you often end-up flooding it or needing additional memory to cope with the load. However, you could deliver high throughput and high availability by using proper sharding and distributing updates to specific masters (like you would do in any actor-based architecture). As a replacement, we're now using a custom event-sourcing framework on top of AWS S3/Azure blob. It's faster, more reliable, and most importantly, better designed.

LukeEF 6y ago
That's very interesting. This is why we think (Luke from TerminusDB again) designing your own full featured graph DB is the best appraoch to the problem - you can work from the ground up and ensure performance is satisfactory for the operations you want to deliver. I don't agree that Git semantics are not well suited to data, but you do have to be very careful about how you advance. I was reading through these source code branching patterns and you can easily imagine (if design and performance is right) how they would apply to data - but as the author says 'branching is easy, merging is harder.' https://martinfowler.com/articles/branching-patterns.html https://martinfowler.com/articles/branching-patterns.html
LukeEF 6y ago
Was discussing your point with a colleague and the comment about time travel is very interesting. Could you give a little more detail the the point? With us, everyone can be talking about a different time point in a branch whenever they like, and it has nothing to do with head. Two clients can look at different time points in our database and neither have to know about each other at all. Also, for TerminusDB we don't use a multimaster coordination mechanism - we actually use the same sort of git approach.
- Fiahil 6y ago
  First, I need to say that I never tried TerminusDB, so I can't claim having a strong opinion on your approach :) Back to the time-travel. One of the most evident architecture when dealing with AI/ML/Optimisation is to design your application as a mesh of small, deterministic, steps (or scripts) reading input data and outputting results. As you would expect, output of one step is reusable by another one. Example: Script A is reading Sales data from source S, Weather data from source W; writing its result to A. Script B is reading data from source A, and Calendar from C; writing its result to B. In this example, we want to be able to do two things: 1) run a previous version of A with S and W from 2 weeks ago and assert the result it produced now is exactly identical to the one it produced at the time 2) run a _newer_ version of A with S and W from 2 weeks ago and compare its result from the one it previously produced. Of course, in the real-world, S, W, C, progress at different speed : new sales could be inserted by the minute, but the weather data would likely change by the day. So, you need a system that would allow you to read S@2fabbg and W@4c4490 while being in the same "repository". That's why git semantics are not a good fit: you need to have only one "branch" to ensure consistency and limit misunderstandings, but you want to "commit" datasets in the same repository at different pace. For that purpose, event sourcing is much better :) (BTW, git at its core, is basically event-sourcing) Kafka's architecture is actually the best solution.
  - LukeEF 6y ago
    You'll have to try :) Very interesting - thanks for the additional detail. Will have to think about how we might best represent in Terminus. We did a bunch of work for retailers in exactly the situation you describe.
  - zachmu 6y ago
    Dolt actually addresses this need exactly. You can query tables at two different revisions without checking out the two different branches. E.g.: `SELECT * FROM S AS OF '2fabbg';` `SELECT * FROM W AS OF '4c4490';` (Branch names work as well as commit hashes for the above). You can even do joins between the tables as they existed at those revisions: `SELECT * FROM S AS OF '2fabbg' JOIN W AS OF '4c4490' ON ...` As long as your data is actually relational, it's a pretty good fit.
buttersbrian 6y ago
We do a similar revisioning system that just isn't performant enough. Can you give me some more detail about what you are doing? We have a a large road ahead, and I am genuinely curious about the path you've taken.
- Fiahil 6y ago
  In short, something very similar to https://www.snowflake.com/ https://www.snowflake.com/. An event-based system sitting on top of s3, indexed with a postgresql, influxDB, or anything else.
zachmu 6y ago
We're building a very literal "git for data" product, Dolt. (doltdb.com). I'm very curious about this criticism: > Git semantics, ("branche", "merge", "commit") are not well suited for data, because merging dataframes and creating "branches" often leads to misunderstandings and delays. Can you give a concrete example of what you mean? I'm wondering if this is a failing of the tool you're using or the model itself? > Time travel is very nice to have, but it's often the case where you would like to consume your input datasets at different point in time in the same repository Dolt supports doing exactly this. See my reply to a child comment below. > Awful performance It's not obvious to me why this needs to be true in the general case, unless it's caused by b) bad design. Are you mostly talking about write performance, or are reads also a problem?
- Fiahil 6y ago
  I don't want you to take it personally, but, from my point of view, Dolt falls in the same bucket as DVC, Pachyderm and some others. The shortcomings are not from the tools themselves, it's a fundamental issue with the design. First, branching and merging. In git, branching allows you to make uncoordinated parallel progress for the price of a reconciliation step. In a datastore, you want the exact opposite: A single, consistent, available source of truth. Having different branches of the same dataset bring more confusion while solving zero problem. Then, commits. In git, a commit represent a snapshot of the entire state of your repository. This is particularly attractive because it guarantees that your code will build no matter what kind of update will follow (without incidence: editing a readme ; severely destructive: removing an src folder). In a datastore, this is nice but unnecessary. As I mentioned it in this thread, datasets move at different speeds, and attaching an new hash to something that didn't change doesn't add value. However, I have to recognize, I failed to mention earlier that datasets are often unrelated and not relational. This would be to reconsider if it were the case, of course. Most of the time, a dataset is represented as a single dataframe (or a single collection of dataframes). There some points where git semantics make sense: immutability of commits, linearizability within branches. Both are extremely important if you want to enable reproducibility of your pipeline. These are traits coming from Event Sourcing. Reproducibility is also claimed by DVC and Pachyderm, but their issue here is more a problem of trying to do too much things at once but not managing to do it right. Running code within Pachyderm pipelines was a recipe for disaster and the first thing we got rid of. As for performances, the write side is where it matters, because it needs to be coordinated. Reads almost never are an issue with good caching. In any case, it should be robust enough to fill the gap between csv files sent to s3 and a full kafka cluster, eg: not complaining for a few TB. To my knowledge, the only multi-leader datastore suitable for storing sharded datasets as a continuous log is Kafka.

by Fiahil 6y ago

LukeEF 6y ago

That's very interesting. This is why we think (Luke from TerminusDB again) designing your own full featured graph DB is the best appraoch to the problem - you can work from the ground up and ensure performance is satisfactory for the operations you want to deliver. I don't agree that Git semantics are not well suited to data, but you do have to be very careful about how you advance. I was reading through these source code branching patterns and you can easily imagine (if design and performance is right) how they would apply to data - but as the author says 'branching is easy, merging is harder.' https://martinfowler.com/articles/branching-patterns.html https://martinfowler.com/articles/branching-patterns.html

LukeEF 6y ago

Was discussing your point with a colleague and the comment about time travel is very interesting. Could you give a little more detail the the point? With us, everyone can be talking about a different time point in a branch whenever they like, and it has nothing to do with head. Two clients can look at different time points in our database and neither have to know about each other at all. Also, for TerminusDB we don't use a multimaster coordination mechanism - we actually use the same sort of git approach.

Fiahil 6y ago
First, I need to say that I never tried TerminusDB, so I can't claim having a strong opinion on your approach :) Back to the time-travel. One of the most evident architecture when dealing with AI/ML/Optimisation is to design your application as a mesh of small, deterministic, steps (or scripts) reading input data and outputting results. As you would expect, output of one step is reusable by another one. Example: Script A is reading Sales data from source S, Weather data from source W; writing its result to A. Script B is reading data from source A, and Calendar from C; writing its result to B. In this example, we want to be able to do two things: 1) run a previous version of A with S and W from 2 weeks ago and assert the result it produced now is exactly identical to the one it produced at the time 2) run a _newer_ version of A with S and W from 2 weeks ago and compare its result from the one it previously produced. Of course, in the real-world, S, W, C, progress at different speed : new sales could be inserted by the minute, but the weather data would likely change by the day. So, you need a system that would allow you to read S@2fabbg and W@4c4490 while being in the same "repository". That's why git semantics are not a good fit: you need to have only one "branch" to ensure consistency and limit misunderstandings, but you want to "commit" datasets in the same repository at different pace. For that purpose, event sourcing is much better :) (BTW, git at its core, is basically event-sourcing) Kafka's architecture is actually the best solution.
- LukeEF 6y ago
  You'll have to try :) Very interesting - thanks for the additional detail. Will have to think about how we might best represent in Terminus. We did a bunch of work for retailers in exactly the situation you describe.
- zachmu 6y ago
  Dolt actually addresses this need exactly. You can query tables at two different revisions without checking out the two different branches. E.g.: `SELECT * FROM S AS OF '2fabbg';` `SELECT * FROM W AS OF '4c4490';` (Branch names work as well as commit hashes for the above). You can even do joins between the tables as they existed at those revisions: `SELECT * FROM S AS OF '2fabbg' JOIN W AS OF '4c4490' ON ...` As long as your data is actually relational, it's a pretty good fit.

buttersbrian 6y ago

We do a similar revisioning system that just isn't performant enough. Can you give me some more detail about what you are doing? We have a a large road ahead, and I am genuinely curious about the path you've taken.

Fiahil 6y ago
In short, something very similar to https://www.snowflake.com/ https://www.snowflake.com/. An event-based system sitting on top of s3, indexed with a postgresql, influxDB, or anything else.

zachmu 6y ago

We're building a very literal "git for data" product, Dolt. (doltdb.com). I'm very curious about this criticism: > Git semantics, ("branche", "merge", "commit") are not well suited for data, because merging dataframes and creating "branches" often leads to misunderstandings and delays. Can you give a concrete example of what you mean? I'm wondering if this is a failing of the tool you're using or the model itself? > Time travel is very nice to have, but it's often the case where you would like to consume your input datasets at different point in time in the same repository Dolt supports doing exactly this. See my reply to a child comment below. > Awful performance It's not obvious to me why this needs to be true in the general case, unless it's caused by b) bad design. Are you mostly talking about write performance, or are reads also a problem?

Fiahil 6y ago
I don't want you to take it personally, but, from my point of view, Dolt falls in the same bucket as DVC, Pachyderm and some others. The shortcomings are not from the tools themselves, it's a fundamental issue with the design. First, branching and merging. In git, branching allows you to make uncoordinated parallel progress for the price of a reconciliation step. In a datastore, you want the exact opposite: A single, consistent, available source of truth. Having different branches of the same dataset bring more confusion while solving zero problem. Then, commits. In git, a commit represent a snapshot of the entire state of your repository. This is particularly attractive because it guarantees that your code will build no matter what kind of update will follow (without incidence: editing a readme ; severely destructive: removing an src folder). In a datastore, this is nice but unnecessary. As I mentioned it in this thread, datasets move at different speeds, and attaching an new hash to something that didn't change doesn't add value. However, I have to recognize, I failed to mention earlier that datasets are often unrelated and not relational. This would be to reconsider if it were the case, of course. Most of the time, a dataset is represented as a single dataframe (or a single collection of dataframes). There some points where git semantics make sense: immutability of commits, linearizability within branches. Both are extremely important if you want to enable reproducibility of your pipeline. These are traits coming from Event Sourcing. Reproducibility is also claimed by DVC and Pachyderm, but their issue here is more a problem of trying to do too much things at once but not managing to do it right. Running code within Pachyderm pipelines was a recipe for disaster and the first thing we got rid of. As for performances, the write side is where it matters, because it needs to be coordinated. Reads almost never are an issue with good caching. In any case, it should be robust enough to fill the gap between csv files sent to s3 and a full kafka cluster, eg: not complaining for a few TB. To my knowledge, the only multi-leader datastore suitable for storing sharded datasets as a continuous log is Kafka.