16 ms·
Dolt is Git for data
- rad_gruchalski 7y agoSeems like a lot of work went into this and there are very smart people behind it. However, I can’t help the feeling that this will lead to so many unintentional data leaks. Nevertheless, starred. Let’s see what does it give.
- IanCal 7y agoLooks interesting, depending on performance this could neatly cover a few use-cases I have at the moment without needing to build as much myself. At least dolt on its own, whether we would need the hub is another matter but I guess it depends on uptake.
- kdamica 7y agoIt's a cool idea. There's also https://quiltdata.com/ https://quiltdata.com/ but I haven't heard anything about them in a long time.
- timsehn 7y agoThis blog compares us to Quilt and others: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-data/ https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-...
- oldgregg 7y agoReally interesting. Would be nice to see documentation. All their examples show modifying the database by running command line sql queries, does it turn up a normal mysql instance or just emulate? Are hooks available in Go? Surprised they don't market it as a blockchain database. I'm building a Dapp right now and this could be really useful.
- timsehn 7y agoYou can spin up a MySQL compatible server using `dolt sql-server`. It does not implement everything right now but we'll get there.
- dominotw 7y ago> Dolt is the only database with branches. datomic has branching too afaik.
- benatkin 7y agoSo does git. I know, I know, git doesn't provide the features you expect from a full fledged database, but neither does Dolt. Dolt even has some content in a GitHub wiki, which uses git as a database backend for a web app: https://github.com/liquidata-inc/dolt/wiki https://github.com/liquidata-inc/dolt/wiki
- jonhohle 7y agoAmazon has an internal k/v store with branches (as well as schema versioning and hierarchical keys). It was primarily used for config management, which it excelled at.
- Random_ernest 7y agoDatomic also has change history, and you can query against a specific point in time afaik.
- timdorr 7y agoAny reason or history behind the name? It means "a stupid person", which seems like a bad choice IMHO: https://www.merriam-webster.com/dictionary/dolt https://www.merriam-webster.com/dictionary/dolt
- hinkley 7y agoit's of a piece with git: https://www.merriam-webster.com/dictionary/git https://www.merriam-webster.com/dictionary/git
- nmstoker 7y agoThat definition misses a significant portion of the commonly used meaning: someone who is unpleasant (as in mean, annoying) For instance: stop being such a git - this is not imploring someone not to be stupid, it's saying don't be annoying, awkward etc. https://www.collinsdictionary.com/amp/english/git https://www.collinsdictionary.com/amp/english/git
- StavrosK 7y agoWhat does git mean?
- dnautics 7y agohonestly, maybe this reflects my americanness but I presumed it was derived from the (western film/culture) word, a corruption of 'get'. Today I learned that it means something else in british.
- kthejoker2 7y agoLove that the name "rhymes" with Git (both are insults), potentially a good fit for MLOps to version your training splits.
- sytse 7y agoVery cool! The world needs better version control for data. How does this compare to something like Pachyderm? How does it work under the covers? What is a splice and what does it mean when it overlaps? https://github.com/liquidata-inc/dolt/blob/84d9eded517167eb2b1f76073df88e85665eec1d/go/store/merge/three_way_list_test.go#L137 https://github.com/liquidata-inc/dolt/blob/84d9eded517167eb2... Is it feasible to use Conflict-free Replicated Data Types (CRDT) for this?
- timsehn 7y agoHere is an earlier blog we published on comparison's to Pachyderm: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-data/ https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-... We got a blog on the storage system coming on Wednesday. It's a mashup of a Merkle DAG and a B-tree called a Prolly Tree. It comes from an open source package called Noms (https://github.com/attic-labs/noms https://github.com/attic-labs/noms). I'm not familiar with CRDT. Will read up on that.
- jdoliner 7y agoWeighing in as Pachyderm founder. The post Tim links here is a very apt description of what Pachyderm does. We're designed for version controlling data pipelines, as well as the data they input and output. Pachyderm's filesystem, pfs, is the component that's most similar to dolt. Pfs is a filesystem, rather than a database, so it tends to be used for bigger data formats like videos, genomics files, sometimes databases dumps. And the main reason people do that is so they can run pipelines on top of those data files. Under the hood the datastructures are actually very similar though, we use a Merkle Tree, rather than a DAG. But the overall algorithm is very similar. Dolt, I think, is a great approach to version controlling SQL style data and access. Noms was a really cool idea that didn't seem to quite find its groove. Whereas dolt seems to have taken the algorithm and made it into more of a tool with practical uses.
- visarga 7y agoHow does Pachyderm deal with GDPR requests. Is it possible to remove a file not just from the present but also from the history? It would be no use to delete a file on GPDR request from the current version while still keeping it around in past commits.
- hinkley 7y agoMaybe not a killer app, but there are certain kinds of collaborative 'CRUD' apps that could benefit greatly from having versioning built into the database as a service. For instance, how much of a functional wiki could one assemble from off-the-shelf parts? Edit, display, account management, templating, etc could all be handled with existing libraries in a wide array of programming languages. The logic around the edit history is likely to contain the plurality if not the majority of the custom code.
- quickthrower2 7y agoI think data (as in raw, collected / measured / surveyed data) doesn't really change, but you get more of it. Some data may occasionally supersede old data. Maybe the schema of the data changes, so your first set of data is in one form, and subsequent data might have more information, or recorded in a different way.
- jon_richards 7y agoOne really important feature of time series data is the preservation of what the dataset looked like at each point in time. Financial data providers will make a mistake (off by order of magnitude, missed a stock split, etc) and then go back and correct it. This means you end up training models entirely on corrected data, but trade based on uncorrected data.
- cbenz 7y agoThat's exactly the reason why we created https://db.nomics.world https://db.nomics.world
- quickthrower2 7y agoThanks, I didn't consider training of models, this is a great use case for a tool.
- timsehn 7y agoWe think this is the assumption (that data does not change) but we don't think it holds and data diffs help show you when something you did not expect did change.
- cjbprime 7y agoComparison to Dat? https://docs.dat.foundation/docs/intro https://docs.dat.foundation/docs/intro
- timsehn 7y agoDat is more of a distributed data sharing protocol. It does not do branch/merge which Dolt does.
- fleetside72 7y agoAs far as I can tell the only way to use this is to push everything into a mysql instance. def some pros and cons there.
- timsehn 7y agoYou can export to whatever you like. In our imagination, data versioning would sit upstream of production just like source code versioning. You take the data out and do what you need with it in a "compile" step.
- peteforde 7y agoOnly 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data contests with The Economist - but at the end of the day, we couldn't find anyone with an urgent problem that they were willing to pay to solve. I wish these folks luck! Perhaps things have changed; we were part of a flock of 5 or 10 similar projects and I'm pretty sure the only one still around today is Kaggle. https://www.youtube.com/watch?v=EWMjQhhxhQ4 https://www.youtube.com/watch?v=EWMjQhhxhQ4
- sjtindell 7y agoJust want to say, really appreciate this food for thought. Where do you go and see someone say "my company tried it and...". This site is a godsend.
- austinjp 7y agoAgreed, it's always refreshing to read candid posts like this. A useful term for Google is "startup post mortem" although you still have to do plenty of sifting.
- timsehn 7y agoI'll check it out. We think the world is a little more ready for this now, given how widely Git is adopted and the advances in other data tooling (like ML). But, as we're all aware, starting a business is hard :-)
- zyang 7y agoThe need is definitely there. My day job involves such need. But we simply cannot trust a drive-by startup to fill the gap. It's safer just to roll your own.
- 7y ago
- honksillet 7y agoPardon my ignorance, but is data copy writable? Or can it be owned? Obviously someone can get into trouble upload propriety code to git. Is there proprietary data?
- timsehn 7y agoWe wrote a couple blog post on data licensing. We're not lawyers but we did a bit of research: https://www.dolthub.com/blog/2020-02-24-data-licenses/ https://www.dolthub.com/blog/2020-02-24-data-licenses/ https://www.dolthub.com/blog/2020-02-26-copyrightable-material/ https://www.dolthub.com/blog/2020-02-26-copyrightable-materi...
- perfect_wave 7y agoCan you give some more information about what you're doing with your cloud infrastructure? Would be intrigued to hear about what you're running.
- timsehn 7y agoWe use AWS but will switch over to Google or multi-cloud when we exhaust our credits. The system is still pretty simple. The main cost is the storage for the blobs in the Dolt repos pushed to DoltHub. We use S3 for that. There is an API that receives pushes and writes any other metadata (user, permissions, etc) into an RDS instance that stores metadata for DoltHub. That instance is also used to cache some critical things. Then it's just a set of web servers and a GraphQL sitting on top serving our React app.
- flashman 7y agoSo, we ingest a third-party dataset that changes daily. One of our problems is that we need to retrospectively measure arbitrary metrics (how many X had condition Y on days 1 through 180 of the current year?). Imagine the external data like this: UUID,CategoryA,CategoryACount,CategoryB,CategoryBCount,BooleanC,BooleanD...etc When we ingest a new UUID, we add a column "START_DATE" which is the first date the UUID's metrics were valid. When any of the metric counts changes, we add "END_DATE" to the row and add a new row for that UUID with an updated START_DATE. It works, but it sucks to analyse because you have to partition the database by the days each row was valid and do your aggregations on those partitions. And it sucks to get a snapshot of how a dataset looked on a particular day. It would be much easier if we could just access the daily diffs, which seems like a task Dolt would accomplish. I mean it has a better chance of working than getting the third party to implement versioning on their data feed.
- jamesblonde 7y agoYou can accomplish this using time-travel queries in frameworks like Apache Hudi and Databricks Delta that i mentioned in more detail in an earlier comment. They only work for Spark-based data pipelines.
- aabbcc1241 7y agoIt claimed to be the first database that support versioning. How does it compare to the revision mechanism in couchdb?
- matthewbauer 7y agoI thought Git was Git for data.
- AlexCoventry 7y agoSpotted the lisp fan.
- fb03 7y agoSpotted the lisp fan.
- timsehn 7y agoYou can make a solid case and I tried: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-data/ https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-... But there are some limitations for large tables. Diffs only work if you order the file in the same order. Plus, you have to import the data into some other tool to do anything with it.
- brachi 7y ago> With Dolt, you can view a human-readable diff of the data you received last time versus the data you received this time. How is this accomplished if the data is binary? Also, how does this compare to git lfs?
- timsehn 7y agoNo data diffs is the data is binary. But diffs are cell-wise like so: https://www.dolthub.com/repositories/Liquidata/corona-virus/compare/qebrs5sdd9p9qjh8p2sbmf7lbnb81afh https://www.dolthub.com/repositories/Liquidata/corona-virus/... git-lfs lets you put store large files on GitHub. With Dolt, we offer a similar a similar utility called git-dolt. Both these allow you to store large things on GitHub as a reference to another storage system, not the object itself.
- rburhum 7y agoEh, I worked on database with branches in 2002 for 3 years while I was at ESRI. It is called a versioned system... Here is how it works from an answer I gave several years back on gis.stackexchange https://gis.stackexchange.com/questions/15203/when-versioning-with-arcsde-can-posted-edits-be-cancelled-or-rejected https://gis.stackexchange.com/questions/15203/when-versionin...
- hypewatch 7y ago> As far as we can tell, Dolt is the only database with branches What about pachyderm?
- timsehn 7y agoSee the comment down below from a cofounder of Pachyderm :-)
- aantix 7y agoSlightly related - how does ML track new data input and ensure that the data hasn't introduced a regression? I would assume there's an automated test suite, but also some way of diffing large amounts of input data and visualizing those input additions relative to model classifications? What are the common tools for this?
- visarga 7y agoYou generally can't analyse the accuracy of an ML system by each individual piece of data in the training set. Each batch of examples slightly changes the model making their updates interact and combine during the training process, so it becomes extremely difficult to assign the contribution of individual examples. Of course you could retrain the model leaving one example out, but that would be exceedingly slow and the result would be inconclusive from a single run because the stochastic noise of the training process is larger than the effect of removing or adding one example. Related areas are confidence calibration, active learning and hard example detection during training. Another approach is to synthesise a new, much smaller dataset that would train a neural net to the same accuracy of the original larger dataset.
- samatman 7y ago> Dolt is the only database with branches There's also litetree, whose slogan is simply "SQLite with branches": https://github.com/aergoio/litetree https://github.com/aergoio/litetree
- timsehn 7y agoVery cool. Have not heard of this one. Will check it out.
- chrismorgan 7y agoAlso noms, which I had high hopes for three years ago, but it seems dead these days (Salesforce acquired their company, and that was that): https://github.com/attic-labs/noms https://github.com/attic-labs/noms (Judging by other comments in this thread, Dolt may be a descendant, partially or completely, of Noms?)
- aboodman 7y agoYep, dolt is a fork of noms: https://github.com/liquidata-inc/dolt#credits-and-license https://github.com/liquidata-inc/dolt#credits-and-license (and sorry :( -- yay open source?)
- lifty 7y agoI can't tell you how happy I was when I discovered noms several days ago, and then the subsequent disappointment that it is not developed anymore. Anyway, now that it's in the open maybe someone with the technical chops to develop such a thing will continue development. Here's to hoping! And good luck in your new endeavour, which also looks like a very cool project.
- aboodman 7y agoThank you! For the record, Replicache uses Noms internally, so I think ... something ... will still become of Noms. We're just not sure how to move forward with it at this point. And I personally find Noms incredibly satisfying mental model to work with, so I hope that eventually some others will too.
- nerdponx 7y agoHow does Dolt compare to DVC?
- timsehn 7y agoWe compared DVC and a bunch of other Git for data" tools here:https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-data/ https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-...
- Noumenon72 7y agoGreat start page. Very persuasive writing, tells me what the project will do for me and not just what it is.
- hypewatch 7y agoDoes Dolt have any benchmarks against other databases at scale? I would think that a git SQL database would not be very snappy at scale
- timsehn 7y agoWe're working on building performance benchmarks right now. We started with correctness. You can read about our correctness journey here: https://www.dolthub.com/blog/2019-12-17-one-nine-of-sql-correctness/ https://www.dolthub.com/blog/2019-12-17-one-nine-of-sql-corr... We think over time (like years) we can achieve read performance parity with MySQL or PosgreSQL. Architecturally, we will always be slower on write than other SQL databases, given the versioned storage engine. Right now, Dolt is built to be used offline for data sharing. And in that use case, the data and all of its history needs to fit on a single logical storage system. The biggest Dolt repository we have right now is 300Gb. It tickles some performance bottlenecks. In the long run, if we get traction we imagine building "big dolt" which is a distributed version of Dolt, where the network cuts happen at logical points in the Merkle DAG. Thus, you could run an arbitrarily large storage and compute cluster to power it.
- ChrisFoster 7y agoA year or so I looked into "git for data" for medical research data curation. At the time I found a couple of promising solutions based on wrapping git and git annex: GIN: https://gin.g-node.org/ https://gin.g-node.org/ datalad: https://www.datalad.org/ https://www.datalad.org/ At the time GIN looked really promising as something potentially simple enough for end users in the lab but with a lot of power behind it. (Unfortunately we never got it deployed due to organizational constraints... but that's a separate story.)
- pnut 7y agoCame here to say the same regarding datalad.
- DocSavage 7y agoYou might want to look at some of the work we're doing in "git for connectomics": http://dvid.io http://dvid.io There will be a new backend released soon that should really improve our ability to transfer and backup versioned data.
- deleted 7y ago[deleted]
- gitgud 7y agoGit tracks changes in logic (software). Tracking changes in Data is simply called a database...
- databeetle 7y agoI use a Python based CMS called CodeRedCMS for my website. They store all their content in a file called db.sqlite3. I use PythonAnywhere for hosting the site and they read the website-files from GitHub. So whenever I update my site (including the blog), I just push the latest version of the db.sqlite3 file to GitHub and pull it into PythonAnywhere. So, as I understand, as long as the DB can be converted into files, it will work as anything else on Git and GitHub. What am I missing?
- deleted 7y ago[deleted]
- BiteCode_dev 7y agoDolt is not Git for data. Git take existing files, and allow you to version them. Git for data would take existing tables or rows, and allow you to version them. A uniform, drop in, open source way to have an history or row, merge them, restore them, etc. that works for Postgres, Mysql or Oracle in the same way. And is compatible with migrations. You can have an history if you use big table or couchdb, not need for Dolt if it's about using a specific product.
- tobr 7y agoYou can nitpick any comparison like that by pointing out the different ways the metaphor breaks.
- BiteCode_dev 7y agoIt's not a metaphor. It's a sell pitch.
- zby 7y agoNon binary data can be saved as text - for example you can have an SQL database dump. You can put that text into git. What does this solution add to that simple idea?
- teraku 7y agoI think they could find funding and use-cases, if they had something like lincensing and terms of use backed into data to track lineage. E.g. "this columns contains emails" and is revokable. Or when you publish data, "this column needs hashing/anonymizing/...". And if you track data across versions and can version relations, you can create lineage. Overall seen many of these lately, waiting for one to really shine. But not because I think it's a grand problem, as I can version my DDL/DML even/code, but I see some need for it because I have a lot of non-tech people working with data throwing it left and right and expecting me to clean up after them.
- olliej 7y agoWow, this really emphasizes that the rationale for choosing “Ok” rather than “Do It” for buttons was correct. :-/
- ralfebert 7y agoSince Wil Shipleys presentation "Git as a Document Format" (AltConf, 2015, [1]) the idea of using git to track data has stuck with me. Cool to see another approach at this. From the first look, I miss the representation of data as plain-old-text-files, but I guess that's a little bit in competition with the goal of getting performance for larger data sets. Anyway, I am wondering, did somebody here try using plain git like a database to track data in a repository? [1] https://academy.realm.io/posts/altconf-wil-shipley-git-document-format/ https://academy.realm.io/posts/altconf-wil-shipley-git-docum...
- apichat 7y agoit look like Daff (align and compare tables) https://github.com/paulfitz/daff https://github.com/paulfitz/daff and Coopy (distributed spreadsheets with intelligent merges) https://github.com/paulfitz/coopy https://github.com/paulfitz/coopy
- chrisweekly 7y agoWe all know naming things is hard, but "dolt" -- as in "idiot" or "imbecile" -- is a head-scratcher.
- ComodoHacker 7y agoThe idea is good, the product may be good too (can't find any whitepapers or something about underlying technology). But some of their marketing is suspiciously unprofessional. Like "Better Database Backups". In DB world, you can't call a "backup" anything that can't restore all of your DB files bit-for-bit, anything non-deterministic. You can call it "dump", "export" or whatever, but not backup. I don't think they plan to compete on DB backups storage market. So please don't mislead you potential customers.
- fiatjaf 7y agoWhat happened to Dat?
- pedro1976 7y agoRecently i was working with some open-data data and i was in need for a tool that transforms those csv/jsons to something standardized, that i can run queries against and patch the data. Maybe this is a use case for dolt.
- deleted 7y ago[deleted]
- tgb 7y agoAn example use case that "git for data" seems to break: storing data for medical research where the participants are allowed to withdraw from the study after the fact. Then their data must be deleted retroactively, not just in the head node. I don't know of a good methodology for dealing with this at all as it breaks backups, for example. The problem extends beyond medical research due to privacy laws like the GDPR. A participant or user must be able to delete their data not merely hide it so as to protect themselves from data breaches. Suggestions welcome.
- kspacewalk2 7y agoIn principle, you should be able to 'rewrite history' in the same way you can already do with git. It is clunky to remove a file from all versions using git itself but easy using tools like bfg[0]. [0] https://rtyley.github.io/bfg-repo-cleaner https://rtyley.github.io/bfg-repo-cleaner
- zachmu 7y agoYou can rebase to change the history. As with git, if you do this, everyone with a clone will need to clone a fresh copy, as they can no longer merge with the remote HEAD.
- heinrichhartman 7y agoLooks like they are a fork of noms (https://github.com/attic-labs/noms https://github.com/attic-labs/noms). The object store has the telling name `.dolt/noms`. Inside are a bunch of binary files. It would be interesting to know more about the on-disk layout of the stored tables. I was not able to find any documentation. Does someone know more about this? Pointers would be appreciated.
- russfink 7y agoI can't tell from the font - is it DOLT - delta Oscar lima tango - or DOIT delta Oscar India tango?
- enriquto 7y agodelta zero one tango
- jnbiche 7y agoIs there a way to page sql results? Also, it would be awesome if I could use rlwrap with `dolt sql`, so I can use the shortcuts I'm used to in an REPL environment.
- zachmu 7y agoYeah, the SQL shell needs some work, including its readline implementation which is kind of broken. Filed an issue for this: https://github.com/liquidata-inc/dolt/issues/505 https://github.com/liquidata-inc/dolt/issues/505
- jnbiche 7y agoAwesome. Yeah, my question wasn't clear, but I meant paging in the shell, as you correctly assumed. In a pinch, I can run page an inline `dolt sql -q` query in the OS shell. But it would be idea to be able to page results in the dolt shell, as we can in most SQL database shells. BTW, I should have written it above, but dolthub/dolt is quite impressive. I hope you all make it, because it's a great product that I would love to use at work if I eventually shift back over to a data science position (right now, working as a software dev).
- aerovistae 7y agoReally bad name lol. A dolt is an idiot.
- readarticle 7y agoA git is an ass.
- mst 7y agoAnd a git is an asshole. That's the joke.
- stared 7y agoWhetever it works or not, I find the introduction confusing. Compare and contrast it with the clarity of these introductions: - https://git-lfs.github.com/ https://git-lfs.github.com/ (Git Large File Storage) - http://paulfitz.github.io/daff/ http://paulfitz.github.io/daff/ ("data diff for tables")
- amolo 7y agoI'm curious. So what is Kaggle?
- danzig13 7y agoCan I get a git for excel?
- senorsmile 7y agoThis solves an immediate need that I was considering noms for. Thank you!
- fmajid 7y agoDelphix has provided for years branching and test database functionality for real databases people actually use like Oracle.