13 ms·
Dolt is Git for Data
- arealaccount 4y agoI’ve been looking for something like this where you can bring your own DB. The ability to version schema and metadata is much more interesting over a saas DB host.
- pdpi 4y agoGiven how many databases already implement MVCC, some form of versioning is already baked into DB systems. It seems obvious in hindsight that this could be reified into its own feature instead of a book-keeping detail. Cool to see a project actually tackle that challenge.
- memorable 4y agoSeems interesting! Will keep an eye on this.
- MontyCarloHall 4y agoNo mention of performance on their GitHub homepage. Anyone know how this compares to a standard MySQL distribution?
- jpnc 4y agoThey say it's slower but are trying to close the gap [1] [1] - https://docs.dolthub.com/sql-reference/benchmarks/latency https://docs.dolthub.com/sql-reference/benchmarks/latency
- NickHoff 4y agoIn this [1] blog post, they say it's "between 2 and 20 times slower than MySQL (but committed to getting down to 4x slower)". [1]: https://www.dolthub.com/blog/2021-03-09-dolt-use-cases-in-the-wild/ https://www.dolthub.com/blog/2021-03-09-dolt-use-cases-in-th...
- deleted 4y ago[deleted]
- cosmic_quanta 4y agoThat looks awesome. One of the listed use-cases is 'time-travel': https://dolthub.com/blog/2021-03-09-dolt-use-cases-in-the-wild/#other https://dolthub.com/blog/2021-03-09-dolt-use-cases-in-the-wi... I wish we could use this at work. We're trying to predict time-series stuff. However, there's a lot of infrastructure complexity which is there to ensure that when we're training on data from years ago, that we're not using data that would be in the future from this point (future data leaking into the past). Using Dolt, as far as I understand it, we could simply set the DB to a point in the past where the 'future' data wasn't available. Very cool
- wodenokoto 4y agoFor at least your use-case, this is a fairly common feature in enterprise databases. For a lot of types of data, auditing requires history. Have a look at MSSQLs history table, for example: https://docs.microsoft.com/en-us/sql/relational-databases/tables/temporal-tables?view=sql-server-ver16 https://docs.microsoft.com/en-us/sql/relational-databases/ta...
- kortex 4y agoHave you looked at dvc www.dvc.org? Takes a little bit to figure out how you want to handle the backing store (usually s3) but then it's very straightforward. You could do a similar pattern: have a data repository and simply move the git HEAD to the desired spot and dvc automatically adds/removes the data files based on what's in the commit. You can even version binaries, without blowing up your .git tree.
- isolli 4y agoI'm looking into DVC right now, and I feel like the code history (in git) and the data history are too intertwined. If you move the git HEAD back, then you get the old data back, but you also get the old code back. I wish there was a way to move the two "heads" independently. Or is there? Edit: I can always revert the contents of the .dvc folder to a previous commit, but I wonder if there's a more natural way of doing it.
- 4y ago
- bjarneh 4y agoThis looks super handy, probably worth the performance penalty in most cases to make every command undo-able (except drop database).
- timsehn 4y agoWe're trying to close the performance gap. Current performance on sysbench can be found here: https://docs.dolthub.com/sql-reference/benchmarks/latency https://docs.dolthub.com/sql-reference/benchmarks/latency We're even slower on heavy transactional use cases. We're dedicating the year to it and we think we can get under 2X MySQL. We have customers who switched from other No-SQL DBs and actually got faster so getting close to MySQL is a great accomplishment.
- throwaway787544 4y agoAw man, why did they have to saddle themselves with Oracle? :(
- timsehn 4y agoCreator here... Only need Oracle clients but we will eventually have a Postgres foreign data wrapper for the storage engine. I explain how we ended up MySQL in this blog post if you're curious. https://www.dolthub.com/blog/2022-03-28-have-postgres-want-dolt/ https://www.dolthub.com/blog/2022-03-28-have-postgres-want-d...
- bestouff 4y agoI don't find the name very nice: there's too much potential of mixing it with "doit" with a capital "d" and a capital "i".
- moffkalast 4y agoDew it. Good, good. It was too dangerous to not keep a backup.
- LandStander 4y agoI blame font designers.
- cal85 4y agoThe font designer designed a font, not a logo.
- CPLX 4y agoI mean forget about mixing it up, what about the actual meaning of the word: dolt noun \ ˈdōlt \ Definition of dolt : a stupid person https://www.merriam-webster.com/dictionary/dolt https://www.merriam-webster.com/dictionary/dolt
- graedus 4y agoI thought this was an odd name too, but I'm guessing it's intentional, as git also is a derogatory word with a similar meaning. : a foolish or worthless person https://www.merriam-webster.com/dictionary/git https://www.merriam-webster.com/dictionary/git
- timsehn 4y agoOne of the creators here... Indeed. Intentional. https://docs.dolthub.com/other/faq#why-is-it-called-dolt-are-you-calling-me-dumb https://docs.dolthub.com/other/faq#why-is-it-called-dolt-are...
- deleted 4y ago
- pgt 4y agoXTDB is a general-purpose bitemporal database with graph query capabilities: https://xtdb.com/ https://xtdb.com/
- infogulch 4y agoThere was a short talk about xtdb in the recent 'rubbing a database on it' conference: https://www.hytradboi.com/2022/baking-in-time-at-the-bottom-of-the-database https://www.hytradboi.com/2022/baking-in-time-at-the-bottom-... The talk expands on what "bitemporal" means in this context, namely, separate "valid" and "audit" time dimensions. The discussion upthread is a good example: if you accidentally mark down inventory in the wrong location and you find out later, you should backdate the fix to when you did the count -- this is the 'valid' dimension, but the fact that you changed it today should also be recorded so it can be reviewed later -- this is the 'audit' dimension.
- alex-treebeard 4y agogreat name
- dmd 4y agoSee also: https://www.datalad.org/ https://www.datalad.org/
- wonderwonder 4y agovery cool. Would love to know the size of the team that built this. Like learning about people that create interesting new products.
- zachmu 4y agohttps://www.dolthub.com/team https://www.dolthub.com/team
- wonderwonder 4y agolol, that would probably be the best place for me to have looked. Appreciate it!
- layer8 4y agoOnly MySQL-compatible, apparently?
- sitkack 4y agohttps://github.com/dolthub/go-mysql-server https://github.com/dolthub/go-mysql-server 170kloc of pure Go
- timsehn 4y agoCreator here. You only connect with a MySQL-client. There is no MySQL code in it. Here's the story if you're curious. https://www.dolthub.com/blog/2022-03-28-have-postgres-want-dolt/ https://www.dolthub.com/blog/2022-03-28-have-postgres-want-d... We will eventually build a Postgres Foreign Data Wrapper.
- melvinroest 4y agoShoutout to Smalltalk [1] for using doit [2] (do it) to execute code! Name collisions happen, it's handy to know about them ;-) [1] https://en.wikipedia.org/wiki/Smalltalk https://en.wikipedia.org/wiki/Smalltalk [2] https://twitter.com/worrydream/status/1424952628207702022 https://twitter.com/worrydream/status/1424952628207702022 <-- Brett Victor talking about doit
- allknowingfrog 4y agoI think you mistook the "l" for an "i". This is "dolt", not "doit".
- melvinroest 4y agodolt DoIt Ah... whoops, my bad
- patrickdevivo 4y agoa very cool project they also maintain is a MySQL server framework for arbitrary backends (in Go): https://github.com/dolthub/go-mysql-server https://github.com/dolthub/go-mysql-server You can define a "virtual" table (schema, how to retrieve rows/columns) and then a MySQL client can connect and execute arbitrary queries on your table (which could just be an API or other source)
- cgio 4y agoI have been interested in this space, but have failed to understand how these versioning solutions for data work in the context of environments. There are aspects of time travel that line up better e.g. with data modelling approaches (such as bitemporal DBs, xtdb etc.) others more with git-like use cases (e.g. schema evolution, backfilling) some combinations. The challenge is, with data I don’t see how you’d like to have all environments in same place/repo and there may be additional considerations coupled with directionality of moves, such as anonymisation for moving from prod to non-prod , back filling for moving from non-prod to prod etc. Keen to read more on other people experiences in this space and how they might be combining different solutions.
- nerdponx 4y agoThese tools aren't really meant for developers. They are meant for researchers, analysts, and other "offline" users and managers of data sources. Data science research workflows generally don't need the same "dev/test/prod" kind of environment setup.
- zachmu 4y agoI won't speak for other data versioning products, but Dolt is definitely for developers. Our customers are software engineers writing database applications that need version control features.
- ollien 4y agoI've only used Dolt once, but it was very helpful when I was working in an ML class on a group project. Previously we would commit gigantic CSVs to git, which sucked. Putting it in Dolt made a lot of the data exploration and sharing easier, and separated our data from the code.
- anyfactor 4y agoI use dolthub's public database to practice queries. I gave their bounty program a shot but I quit when I was prompted to download/clone a 8gb database. So the best thing would be to have a dolthub account and develop the data from there. I think if you do a little bit of git hacking you can create a branch with just an empty or a dummy database based on the schema, clone only that branch and make a PR to append to the original database. But I couldn't figure that out.
- ascar 4y agoCould you elaborate what's so bad about 8GB of test data? This is a database project after all and many things need a certain amount of data to show up.
- anyfactor 4y agoI tried downloading the data and my network connection glitched at 6gb, I was thrown an error and the processed exited. I believe I was recommended to restart the entire process again as there was no way continue the process where I left out. I didn’t try again. I think there could be a better design to clone/download the database on local machine. I understand people don't actually download databases to their local machine but I expected a better solution.
- zachmu 4y agoDefinitely. We have plans to allow clone to be a shallow operation, so that the original data stays remote and only your changes are local. Will make queries slower obviously, but that's the right trade-off for some people.
- adamw2k 4y agoWas going to say it reminded me of noms, only to realize it's a fork... https://github.com/attic-labs/noms https://github.com/attic-labs/noms
- timsehn 4y agoCreator here... We are a fork of Noms. Lots of incremental changes and we're in the process of a major storage engine overhaul (what we use Noms for) for performance as we speak.
- deleted 4y ago[deleted]
- Hakeemmidan 4y agothis looks really fun
- aratob 4y agoPrevious discussions: - Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge (2021) - 177 comments https://news.ycombinator.com/item?id=26370572 https://news.ycombinator.com/item?id=26370572 - Dolt is Git for data (2020) - 191 comments https://news.ycombinator.com/item?id=22731928 https://news.ycombinator.com/item?id=22731928
- amelius 4y agoThe nice thing about Git is that it is batteries-included. I don't want to deal with databases and everything that comes with it. Just give me something that works with the filesystem directly.
- throwaway892238 4y agoThis is the future of databases, but nobody seems to realize it yet. One of the biggest problems with databases (particularly SQL ones) is they're a giant pile of mutable state. The whole idea of "migrations" exists because it is impossible to "just" revert any arbitrary change to a database, diff changes automatically, merge changes automatically. You need some kind of intelligent tool or framework to generate DDL, DML, DCL, they have to be applied in turn, something has to check if they've already been applied, etc. And of course you can't roll back a change once it's been applied, unless you create even more program logic to figure out how to do that. It's all a big hack. By treating a database as version-controlled, you can treat any operation as immutable. Make any change you want and don't worry about conflicts. You can always just go back to the last working version, revert a specific change, merge in one or more changes from different working databases. Make a thousand changes a day, and when one breaks, revert it. No snapshotting and slowly restoring the whole database due to a non-reversible change. Somebody dropped the main table in prod? Just revert the drop. Need to make a change to the prod database but the staging database is different? Branch the prod database, make a change, test it, merge back into prod. The effect is going to be as radical as the popularization of containers. Whether you like them or not, they are revolutionizing an industry and are a productivity force multiplier.
- jahnu 4y agoDoesn’t Datomic do all this for some years now?
- zachmu 4y agoLots of databases offer time travel / historical querying, including datomic, MySQL, Postgres, etc (plugins required in some cases). Dolt's unique functionality isn't time travel, although it has that. It's version control, i.e. branch and merge, push and pull, fork and clone. A bunch of database products give you some of this for schema migrations, but Dolt is the only one that does it for table data as well.
- rapnie 4y ago
- mriet 4y agoDolt is git for databases. Nessie is git for data.
- nailer 4y agoI thought dat was git for data. But it looks like it's dead now. Docs site is still up: https://docs.dat.foundation/ https://docs.dat.foundation/
- smm11 4y agoData? rsync
- henrydark 4y agoI get "git for data" by putting iceberg snapshot files _in_ git. That's it. It's literally git for data.
- remram 4y agoWhat kind of merge actually happens, e.g. how does it deal with conflicts? Does it merge at the SQL command level or at the changed tuple level? If I have name job age remram dev 32 and I concurrently do UPDATE table SET age = age + 2 WHERE job = 'dev'; UPDATE table SET job = 'mgr' WHERE age = 32; Do I get a conflict? Do I get something consistent with the two transactions serialized, e.g. dev 34 or op 32)? Can I get something that no ordering of transaction could have given me, like mgr 34?
- zachmu 4y agoMerge is cell-wise. If the two heads being merged edited disjoint cells, there's no conflict, they merge together. If they touched the same cell, or if one branch deleted a row another added, then you get a conflict. Merge conflicts are stored in a special system table so you can decide what to do with them, kind of analogous to conflict markers in a source file. More details here: https://docs.dolthub.com/sql-reference/version-control/merges#conflicts https://docs.dolthub.com/sql-reference/version-control/merge... The situation you're talking about with two transactions isn't a merge operation, it's just normal database transaction isolation level stuff. Dolt supports REPEATABLE_READ right now, with others coming in future releases. So in the example above, whichever transaction committed last would fail and get rolled back (assuming they touched the same rows).
- remram 4y ago> The situation you're talking about with two transactions isn't a merge operation, it's just normal database transaction isolation level stuff I mean, arguably. It's not like there is a standard definition for "merge operation" on data. Even Git tries to do more than line-level changes, taking context into account, and turning one line change into two line changes if merging across a file copy for example. Dolt markets itself as "a version controlled SQL database", so I think it is perfectly reasonable to consider the standard that already exists for concurrent changes to a SQL database, and that's transaction isolation. I guess anything more complex than this would be pretty unwieldy though, with probably little benefits. I am struggling to come up with a good example for the kind of anomaly I imagine.
- iRomain 4y agoAlso in the same vein, check out https://lakefs.io/ https://lakefs.io/
- chirau 4y agoThey have a pretty interesting data bounty program as well. https://www.dolthub.com/bounties https://www.dolthub.com/bounties I participated in one a few months back. I had to stop midway inbetween because work was calling but it was a lot of fun. I know I earned some dollars, lol, but probably too few to count for the competition. I am actually trying to convince my manager to put up a few bounties later this year. Their Discord support is also super impressive.
- gniap 4y agoIt's quite similar to https://qri.io https://qri.io functionality, but unfortunately Qri stopped providing their Cloud service(https://qri.io/winding_down https://qri.io/winding_down).
- deleted 4y ago[deleted]
- magicalhippo 4y agoSomewhat unrelated but... docs $ dolt conflicts cat docs +-----+--------+----+----+ | | | pk | c1 | +-----+--------+----+----+ | | base | 1 | 1 | | * | ours | 1 | 10 | | * | theirs | 1 | 0 | +-----+--------+----+----+ For some reason my brain just can't seem to remember what's "ours" and what's "theirs" when resolving merge conflicts. Like which one represents what was in the working copy and which one represents the data I'm trying to merge. I'd much prefer absolute terms rather than those relative ones.
- zachmu 4y agoYeah I'm kinda the same way. Ultimately we decided to copy git's terminology for this rather than making users learn new terms. (ours is the working copy, theirs is the thing you're merging in)
- magicalhippo 4y agoYeah I totally get keeping the nomenclature given Dolt tries to be Git-like.
- nibab 4y agoGit-like functionality on top of existing RDBMS solutions would significantly reduce the overhead of managing ephemeral test environments for us. It would be awesome to see Dolt as a thin layer on top of MSSQL or Postgres that would allow my team to easily test changes before deploying to prod. Today, this means creating a snapshot of the DB, creating a new DB instance, copying the data over, doing a deploy, testing changes, teardown and cleanup. Branching with copy-on-write (very similar to what neon.tech offers) on top of on-prem instances would be lifechanging for teams operating in legacy dbs.
- mildbyte 4y agoWe might have what you're looking for with sgr [0] (I'm the co-founder) which works by running SQL commands on top of a PostgreSQL instance and uses PostgreSQL audit triggers for versioning, though note it currently only works on top of a custom PostgreSQL Docker image, since it requires some extensions. [0] https://www.splitgraph.com/docs/sgr-advanced/getting-started/introduction https://www.splitgraph.com/docs/sgr-advanced/getting-started...