5 ms·

It's outrageous how little tooling support there is for version control in data compared to code. Every mainstream database forgets history with updates, don't

It's outrageous how little tooling support there is for version control in data compared to code. Every mainstream database forgets history with updates, don't support distributed workflows, don't support commit ids as first class objects, or most other basic features of VCSs. Databases just aren't a solution to version control. I can't imagine a future where we don't treat data version control like a necessity in the same was as code version control. I hope Quilt can fill the much-needed role of "Github for data".

xyzzy_plugh 9y ago
I've often thought about this problem space. Imagine you have a database with all the bells and whistles you suggest. At first, it's great. But at some point, when you start experiencing growth (compute/storage) pressure (many petabytes) all of this metadata adds up. Does it remain cost effective at scale? All the source code in the world is a drop in the bucket compared to the raw data collected by a large business.
- smilliken 9y ago
  Well, there's human-generated data, and there's computer-generated data. The former is tiny but more dynamic, and where we want all of these nice creature comforts. The latter can be large, but workflows tend to be static so there's less need for these features.
- hinkley 9y ago
  If the user of the data insists on a time horizon of 'since our founding', then yeah, there's an opportunity cost to keeping all of the data. The one people usually miss is that it takes log(n) time to update or insert a record because you have to update all of the indexes. So when you have 1000 times as much data, every insert takes 10x as long as it did at the beginning. Or you use partial indexes and it gets maybe 2 times slower which the users probably won't notice. So people just give up and exfiltrate all of the data to another server to run their reports on. But as a user I still want to be able to figure out 'did I do that thing in February, or was it March?' fairly often. One thing I've always wished database replication systems did (universally) was allow you to run different indexes on different replicas, so you don't have to export the data at all. Your insert time would still be a function of network delay + worst case index update time, but you could segregate traffic based on kind and continue to scale to a fairly large company before anyone had to mention data warehousing or data lakes.
Lightbody 9y ago
I just joined a company that solves this very problem: https://www.delphix.com/ https://www.delphix.com/ What you described is literally what we offer -- the ability to go backwards and forwards in time and create branches, bookmarks, and refreshes with very little overhead. [Edit: This is all for databases that by default, as you noted, don't do any of this stuff.] We aren't (yet) super well known, but we've got a number of things in flight that should change that :) And if you're curious how this magic works, the hint is: we have a number of really smart ZFS contributors on the engineering team.
- smilliken 9y ago
  How does it work? Is it filesystem snapshots? I know it's unreasonable to expect, but do you support forking/merging? What about incremental or logical replication?
  - Lightbody 9y ago
    Filesystem snapshots are part of it. Forking yes. Merging no. You can configure it to ingest incremental backups as well as realtime transaction logs, which means you can go backwards/forward in time all the way down to the individual transaction. It's quite cool!
falsedan 9y ago
A lot of the time, people get confused by source control & think, “I need to refer to this data as it was at a specific point in time, I’ll use git”. Then they get all the overhead of a DVCS without getting most of the benefits (blame, bisect, branches, merge resolution) when what they wanted was versioned releases of data packages…
hinkley 9y ago
And most databases can't tell you who put that bad data into the database. Nearly every place I've worked at we've had to try to create our own version history and audit information. This one of the examples I use when people start talking about how we have a shortage of engineers. No, we have a shortage of good tools. Nobody should have been writing their own data versioning code in 2007, let alone 2017.
hn_throwaway_99 9y ago
Have you seen data.world? "GitHub for data" is basically their entire model.
- akarve 9y ago
  Yeah. Important differences: Quilt offers unlimited public storage, users do not need to log in to use public data, offers on-premise solution, handles serialization, works offline with local builds, etc. You'll see more differentiation as we roll out new product features.
justinclift 9y ago
As a thought, keep an eye on DBHub.io: https://github.com/sqlitebrowser/dbhub.io https://github.com/sqlitebrowser/dbhub.io Still in early dev stages, and I'm on holidays for a few weeks atm, but we'll have something useful online in a few months. Supporting forking, branching, merging, etc.

xyzzy_plugh 9y ago

I've often thought about this problem space. Imagine you have a database with all the bells and whistles you suggest. At first, it's great. But at some point, when you start experiencing growth (compute/storage) pressure (many petabytes) all of this metadata adds up. Does it remain cost effective at scale? All the source code in the world is a drop in the bucket compared to the raw data collected by a large business.

smilliken 9y ago
Well, there's human-generated data, and there's computer-generated data. The former is tiny but more dynamic, and where we want all of these nice creature comforts. The latter can be large, but workflows tend to be static so there's less need for these features.
hinkley 9y ago
If the user of the data insists on a time horizon of 'since our founding', then yeah, there's an opportunity cost to keeping all of the data. The one people usually miss is that it takes log(n) time to update or insert a record because you have to update all of the indexes. So when you have 1000 times as much data, every insert takes 10x as long as it did at the beginning. Or you use partial indexes and it gets maybe 2 times slower which the users probably won't notice. So people just give up and exfiltrate all of the data to another server to run their reports on. But as a user I still want to be able to figure out 'did I do that thing in February, or was it March?' fairly often. One thing I've always wished database replication systems did (universally) was allow you to run different indexes on different replicas, so you don't have to export the data at all. Your insert time would still be a function of network delay + worst case index update time, but you could segregate traffic based on kind and continue to scale to a fairly large company before anyone had to mention data warehousing or data lakes.

Lightbody 9y ago

I just joined a company that solves this very problem: https://www.delphix.com/ https://www.delphix.com/ What you described is literally what we offer -- the ability to go backwards and forwards in time and create branches, bookmarks, and refreshes with very little overhead. [Edit: This is all for databases that by default, as you noted, don't do any of this stuff.] We aren't (yet) super well known, but we've got a number of things in flight that should change that :) And if you're curious how this magic works, the hint is: we have a number of really smart ZFS contributors on the engineering team.

smilliken 9y ago
How does it work? Is it filesystem snapshots? I know it's unreasonable to expect, but do you support forking/merging? What about incremental or logical replication?
- Lightbody 9y ago
  Filesystem snapshots are part of it. Forking yes. Merging no. You can configure it to ingest incremental backups as well as realtime transaction logs, which means you can go backwards/forward in time all the way down to the individual transaction. It's quite cool!

falsedan 9y ago

A lot of the time, people get confused by source control & think, “I need to refer to this data as it was at a specific point in time, I’ll use git”. Then they get all the overhead of a DVCS without getting most of the benefits (blame, bisect, branches, merge resolution) when what they wanted was versioned releases of data packages…

hinkley 9y ago

And most databases can't tell you who put that bad data into the database. Nearly every place I've worked at we've had to try to create our own version history and audit information. This one of the examples I use when people start talking about how we have a shortage of engineers. No, we have a shortage of good tools. Nobody should have been writing their own data versioning code in 2007, let alone 2017.

hn_throwaway_99 9y ago

Have you seen data.world? "GitHub for data" is basically their entire model.

akarve 9y ago
Yeah. Important differences: Quilt offers unlimited public storage, users do not need to log in to use public data, offers on-premise solution, handles serialization, works offline with local builds, etc. You'll see more differentiation as we roll out new product features.

justinclift 9y ago

As a thought, keep an eye on DBHub.io: https://github.com/sqlitebrowser/dbhub.io https://github.com/sqlitebrowser/dbhub.io Still in early dev stages, and I'm on holidays for a few weeks atm, but we'll have something useful online in a few months. Supporting forking, branching, merging, etc.