5 ms·

Has anybody tried using Dolt? It looks quite young. I think this idea is really valuable, but I usually see it implemented as a time-series extension on top of

by gen220 6y ago

Has anybody tried using Dolt? It looks quite young. I think this idea is really valuable, but I usually see it implemented as a time-series extension on top of Postgres or MySQL, like SQLAlchemy-Continuum or TimescaleDB. i.e. you caan get most of the useful git-like time-travel semantics (modulo schema migrations) out of timeseries data with a separate transaction history table. I'm curious what Dolt's performance profile looks like (i.e. how reads and writes scale vs "branch"-count, row-count, and how they handle indices across schema migrations), since the aforementioned solutions on Postgres are building on an already-very-performant core. edit: TerminusDB also looks very cool, although it's approaching this problem from a graph-database angle rather than a relational one. Their architecture (prolog server on top of a rust persistence core) also seems super fascinating, I'd love to read more on how they chose it.

LukeEF 6y ago
Thanks! Luke from TerminusDB here. We think a graph is the best way to approach a graphy problem! We like Prolog for Querying, constraint checking and user interaction. Rust is great for low-level data manipulation. Prolog is a superpower - very simple but very powerful, but quite far removed from the hardware and uses abstractions not under our control so not good at nitty-gritty bit manipulation. We like Rust as a low-level memory-safe language (and it has a great community). Much more background on why we made those choices: https://news.ycombinator.com/item?id=22867767 https://news.ycombinator.com/item?id=22867767
timsehn 6y ago
CEO of Dolt here. You are indeed correct, we are quite young. We are also focused at this stage on the data distribution use case (ie. use Dolt instead of putting a CSV or JSON file on GitHub). So, we haven't spent much time on query performance or scale. The current model is that the database and all of its history have to fit on the user's hard drive. The format is easily distributed (ie. put content addressed chunks on different servers, optimize for fewer network hops) but that's not how it works today. That being said, we think architecturally we can eventually get pretty close to parity with other RDBMS on the read path. We will be slower on the write path given the need to build the Merkle DAG on writes. Long ways to go though, we only launched open source last August. On a suite of MySQL ~6M correctness benchmarks, we currently execute 3-4X more slowly. These aren't large sets either so we suspect we'll run into some non-linearities in performance. This is just normal SQL. We haven't really tested the limits of how many branches we can handle or how long branch creation or merge takes at scale. Not because we don't want to but because it's not the use case we're focused on. If you are willing to take the journey with us, we'll work with you to get Dolt where it needs to be (if it can, reads should be fine, writes will be more expensive). But we're just at the beginning so there will be bumps along the road. If that's not for you, we completely understand.
- bduerst 6y ago
  Looks great. Are users able to query commits/historical values to a cell inside their SQL query? i.e. "Select population.master, population.branch2, population.commit123.diff from state_populations"
  - zachmu 6y ago
    You can do historical queries two different ways: 1) `SELECT * FROM state_populations AS OF 'master'` to get the repo as it existed at a particular timestamp / ref name 2) `SELECT * FROM dolt_diff_state_populations WHERE ...` to examine to/from column value pairs between any two commits you choose. There are a bunch of different supported system tables that provide interesting functionality: https://www.dolthub.com/docs/reference/sql/#dolt-system-tables https://www.dolthub.com/docs/reference/sql/#dolt-system-tabl...
zachmu 6y ago
> you caan get most of the useful git-like time-travel semantics (modulo schema migrations) out of timeseries data with a separate transaction history table Well, not really. Dolt isn't just time travel. If all you want is time travel (or append-only data), you can do that with Postgres or MySQL pretty easily and get great performance. What Dolt brings to the table is actual Git semantics, a real commit graph you can branch, merge, fork, clone. You can inspect the historical values of each cell and examine the author and commit message for each change. Two people can clone a database, both insert / update / delete rows, and then merge each other's changes. If they touched the same rows, they'll have a merge conflict they have to resolve. If you find somebody's data they've published in the Dolt format has an error, you can fork it, correct the data, and submit them a PR to merge back, just like you do with code on Github. It's a database built for collaboration from the ground up. Instead of authorizing each person you want to make edits specific grants on particular tables / subsets of the data, you just let them clone it, then submit a PR that you can examine and approve. Or if they disagree with your policies, they can fork the data and still merge in updates from your master source as necessary. Git demonstrated what a powerful model the commit graph is for code. Dolt brings it to data, with SQL built in. To answer your question about indexes across schema migrations, indexes are versioned with the schema of the table. This means they can be efficiently synced during a pull operation, but it means that previous versions of the data don't have indexes on them. We're considering adding the ability to add an index to the data from the beginning of the commit history as a rebase operation, but haven't implemented that yet.
- xiaodai 6y ago
  Dolt doesn't work for large data right?
  - zachmu 6y ago
    Depends what you mean by "work" and "large," I guess. We have repos in the 250 GB range that definitely push the limits of functionality, but everything still works (pull, push, merge, clone, etc., and queries return the right results, if slowly).
Fiahil 6y ago
I work as a SWE at a large AI consultancy. We've been experimenting with "git for data" products for a while, and we've been trying to get rid of them (notably Pachyderm) for -at least- 2 years. What they all share is a) awful performances, and b) bad design. Git semantics, ("branche", "merge", "commit") are not well suited for data, because merging dataframes and creating "branches" often leads to misunderstandings and delays. Time travel is very nice to have, but it's often the case where you would like to consume your input datasets at different point in time in the same repository (unless you do one dataset per repository, but then, what's the point ?). Performances are bad, because all updates needs to go through some kind of coordination mechanism (etcd, zookeeper, or raft directly). In a single instance scenario, you often end-up flooding it or needing additional memory to cope with the load. However, you could deliver high throughput and high availability by using proper sharding and distributing updates to specific masters (like you would do in any actor-based architecture). As a replacement, we're now using a custom event-sourcing framework on top of AWS S3/Azure blob. It's faster, more reliable, and most importantly, better designed.
- LukeEF 6y ago
  That's very interesting. This is why we think (Luke from TerminusDB again) designing your own full featured graph DB is the best appraoch to the problem - you can work from the ground up and ensure performance is satisfactory for the operations you want to deliver. I don't agree that Git semantics are not well suited to data, but you do have to be very careful about how you advance. I was reading through these source code branching patterns and you can easily imagine (if design and performance is right) how they would apply to data - but as the author says 'branching is easy, merging is harder.' https://martinfowler.com/articles/branching-patterns.html https://martinfowler.com/articles/branching-patterns.html
- LukeEF 6y ago
  Was discussing your point with a colleague and the comment about time travel is very interesting. Could you give a little more detail the the point? With us, everyone can be talking about a different time point in a branch whenever they like, and it has nothing to do with head. Two clients can look at different time points in our database and neither have to know about each other at all. Also, for TerminusDB we don't use a multimaster coordination mechanism - we actually use the same sort of git approach.

by gen220 6y ago

LukeEF 6y ago

Thanks! Luke from TerminusDB here. We think a graph is the best way to approach a graphy problem! We like Prolog for Querying, constraint checking and user interaction. Rust is great for low-level data manipulation. Prolog is a superpower - very simple but very powerful, but quite far removed from the hardware and uses abstractions not under our control so not good at nitty-gritty bit manipulation. We like Rust as a low-level memory-safe language (and it has a great community). Much more background on why we made those choices: https://news.ycombinator.com/item?id=22867767 https://news.ycombinator.com/item?id=22867767

timsehn 6y ago

CEO of Dolt here. You are indeed correct, we are quite young. We are also focused at this stage on the data distribution use case (ie. use Dolt instead of putting a CSV or JSON file on GitHub). So, we haven't spent much time on query performance or scale. The current model is that the database and all of its history have to fit on the user's hard drive. The format is easily distributed (ie. put content addressed chunks on different servers, optimize for fewer network hops) but that's not how it works today. That being said, we think architecturally we can eventually get pretty close to parity with other RDBMS on the read path. We will be slower on the write path given the need to build the Merkle DAG on writes. Long ways to go though, we only launched open source last August. On a suite of MySQL ~6M correctness benchmarks, we currently execute 3-4X more slowly. These aren't large sets either so we suspect we'll run into some non-linearities in performance. This is just normal SQL. We haven't really tested the limits of how many branches we can handle or how long branch creation or merge takes at scale. Not because we don't want to but because it's not the use case we're focused on. If you are willing to take the journey with us, we'll work with you to get Dolt where it needs to be (if it can, reads should be fine, writes will be more expensive). But we're just at the beginning so there will be bumps along the road. If that's not for you, we completely understand.

bduerst 6y ago
Looks great. Are users able to query commits/historical values to a cell inside their SQL query? i.e. "Select population.master, population.branch2, population.commit123.diff from state_populations"
- zachmu 6y ago
  You can do historical queries two different ways: 1) `SELECT * FROM state_populations AS OF 'master'` to get the repo as it existed at a particular timestamp / ref name 2) `SELECT * FROM dolt_diff_state_populations WHERE ...` to examine to/from column value pairs between any two commits you choose. There are a bunch of different supported system tables that provide interesting functionality: https://www.dolthub.com/docs/reference/sql/#dolt-system-tables https://www.dolthub.com/docs/reference/sql/#dolt-system-tabl...

zachmu 6y ago

> you caan get most of the useful git-like time-travel semantics (modulo schema migrations) out of timeseries data with a separate transaction history table Well, not really. Dolt isn't just time travel. If all you want is time travel (or append-only data), you can do that with Postgres or MySQL pretty easily and get great performance. What Dolt brings to the table is actual Git semantics, a real commit graph you can branch, merge, fork, clone. You can inspect the historical values of each cell and examine the author and commit message for each change. Two people can clone a database, both insert / update / delete rows, and then merge each other's changes. If they touched the same rows, they'll have a merge conflict they have to resolve. If you find somebody's data they've published in the Dolt format has an error, you can fork it, correct the data, and submit them a PR to merge back, just like you do with code on Github. It's a database built for collaboration from the ground up. Instead of authorizing each person you want to make edits specific grants on particular tables / subsets of the data, you just let them clone it, then submit a PR that you can examine and approve. Or if they disagree with your policies, they can fork the data and still merge in updates from your master source as necessary. Git demonstrated what a powerful model the commit graph is for code. Dolt brings it to data, with SQL built in. To answer your question about indexes across schema migrations, indexes are versioned with the schema of the table. This means they can be efficiently synced during a pull operation, but it means that previous versions of the data don't have indexes on them. We're considering adding the ability to add an index to the data from the beginning of the commit history as a rebase operation, but haven't implemented that yet.

xiaodai 6y ago
Dolt doesn't work for large data right?
- zachmu 6y ago
  Depends what you mean by "work" and "large," I guess. We have repos in the 250 GB range that definitely push the limits of functionality, but everything still works (pull, push, merge, clone, etc., and queries return the right results, if slowly).

Fiahil 6y ago

I work as a SWE at a large AI consultancy. We've been experimenting with "git for data" products for a while, and we've been trying to get rid of them (notably Pachyderm) for -at least- 2 years. What they all share is a) awful performances, and b) bad design. Git semantics, ("branche", "merge", "commit") are not well suited for data, because merging dataframes and creating "branches" often leads to misunderstandings and delays. Time travel is very nice to have, but it's often the case where you would like to consume your input datasets at different point in time in the same repository (unless you do one dataset per repository, but then, what's the point ?). Performances are bad, because all updates needs to go through some kind of coordination mechanism (etcd, zookeeper, or raft directly). In a single instance scenario, you often end-up flooding it or needing additional memory to cope with the load. However, you could deliver high throughput and high availability by using proper sharding and distributing updates to specific masters (like you would do in any actor-based architecture). As a replacement, we're now using a custom event-sourcing framework on top of AWS S3/Azure blob. It's faster, more reliable, and most importantly, better designed.

LukeEF 6y ago
That's very interesting. This is why we think (Luke from TerminusDB again) designing your own full featured graph DB is the best appraoch to the problem - you can work from the ground up and ensure performance is satisfactory for the operations you want to deliver. I don't agree that Git semantics are not well suited to data, but you do have to be very careful about how you advance. I was reading through these source code branching patterns and you can easily imagine (if design and performance is right) how they would apply to data - but as the author says 'branching is easy, merging is harder.' https://martinfowler.com/articles/branching-patterns.html https://martinfowler.com/articles/branching-patterns.html
LukeEF 6y ago
Was discussing your point with a colleague and the comment about time travel is very interesting. Could you give a little more detail the the point? With us, everyone can be talking about a different time point in a branch whenever they like, and it has nothing to do with head. Two clients can look at different time points in our database and neither have to know about each other at all. Also, for TerminusDB we don't use a multimaster coordination mechanism - we actually use the same sort of git approach.