5 ms·

Very interesting, I think we need a git for data. What is the performance of diffs and merges? What data size does it become too slow?

by rpedela 10y ago

aeharding 10y ago
If by "Git for data" you mean accumulate-only (or append-only), immutable data stores... There are already many existing solutions. It's always good to see alternatives, though!
- chubot 10y ago
  Which ones are you thinking of? The nice thing about git is that it doesn't really require much of the server. I run my own "git hosting" with linode, apt-get, and ssh. Most of the solutions for immutable big data don't have that level of convenience, as far as I know. Anything which requires a lot of sysadmin and ops work will eventually become a commercial cloud service... git on the other hand is useful by itself, and good companies can obviously be built on top of it as well.
  - TylerE 10y ago
    Most binary formats don't diff well.
- rpedela 10y ago
  No not really. CVS and git are basically the same thing but git is a lot better in many ways. I don't know if noms is really the git for data but those other tools are more like CVS in this analogy: clunky and slow.
  - emmelaich 10y ago
    They are so different it's hard to know what you mean by "basically the same thing"
    - rpedela 10y ago
      Yes there are significant differences in terms of feature, implementation, etc but they have the same goal: manage source code versions.
      - emmelaich 10y ago
        That's how Git is used but I think that by design, it is foremost a userlevel append-only content-addressable file system. The difference between this and a version control system is a major part of why the UI is so awful.
- rspeer 10y ago
  Can you point me at some? Because I've tried a few immutable data stores and been disappointed every time. Given about 10 GB of JSON structures, I keep finding things that can't outperform the boring combo of: * Convert the versioned data to tab-separated values * COPY it into Postgres every time * Hope Postgres can act immutable enough even though it wasn't designed to be The closest I've come to improving this situation was Kyoto Cabinet (unusable license) and rolling my own damn hashtable (it worked okay but adding new kinds of indexes was just unmaintainable, there's a reason databases should be made by experts).
  - ykler 10y ago
    I was curious about Camlistore and Tahoe-LAFS at one point. I didn't investigate them very thoroughly, however. How do they compare?
  - deleted 10y ago
    [deleted]
  - sjezewski 10y ago
    (Disclaimer - I work at pachyderm) http://pachyderm.io http://pachyderm.io Pachyderm is git for data. We work hard to make sure we can store data of different types (binary, text, json) efficiently. We also work hard to give you good mechanisms to read the data in a distributed way. I'd be curious how this suits your purposes.
    - rspeer 10y ago
      Just started looking at Pachyderm. While I can see how a git-based filesystem can help with some use cases, does it do any kind of indexing at all? I see that the FAQ recommends exporting the data from Pachyderm into PostgreSQL, which leaves me where I am now.
AvenueIngres 10y ago
So.. CRDTs?
- aboodman 10y ago
  https://github.com/attic-labs/noms/blob/master/doc/faq.md#why-dont-you-just-use-crdts https://github.com/attic-labs/noms/blob/master/doc/faq.md#wh...
  - ergl 10y ago
    Those reasons are a bit outdated, you can certainly have add-and-remove crdts with or-sets, and you can do so with almost no garbage. One of the Tomtom engineers explained one approach at Strangeloop last month: https://www.youtube.com/watch?v=veeWamWy8dk https://www.youtube.com/watch?v=veeWamWy8dk
aboodman 10y ago
Noms performs diff and merge in time proportional to size of the diff. The size of the source data is not really relevant. One way to think of Noms is that is an index optimized for computing diffs. Here's a screencast that shows off Noms diffing things fast: https://www.youtube.com/watch?v=Zeg9CY3BMes https://www.youtube.com/watch?v=Zeg9CY3BMes
dwhitena 10y ago
I definitely agree here. As a data scientist, sometimes it seems like we are in the wild west as far as reproducibility and versioning of our analyses. This seems like an interesting project that tackles some of the data versioning stuff. However, I believe that, at least in data science, we need data versioning closely tied to the analyses themselves for complete reproducibility. That is, we need the versioning tied to the inputs/outputs of data pipeline stages, such that we can reproduce pipeline runs at any time and incrementally improve and run pipelines based on diffs in data. As mentioned elsewhere in the comments, Pachyderm (http://pachyderm.io/ http://pachyderm.io/) does exactly this. Working both as git for data, but also enabling data pipelining and analyses with the data versioning.

Very interesting, I think we need a git for data. What is the performance of diffs and merges? What data size does it become too slow?

by rpedela 10y ago

Very interesting, I think we need a git for data. What is the performance of diffs and merges? What data size does it become too slow?

aeharding 10y ago

If by "Git for data" you mean accumulate-only (or append-only), immutable data stores... There are already many existing solutions. It's always good to see alternatives, though!

chubot 10y ago
Which ones are you thinking of? The nice thing about git is that it doesn't really require much of the server. I run my own "git hosting" with linode, apt-get, and ssh. Most of the solutions for immutable big data don't have that level of convenience, as far as I know. Anything which requires a lot of sysadmin and ops work will eventually become a commercial cloud service... git on the other hand is useful by itself, and good companies can obviously be built on top of it as well.
- TylerE 10y ago
  Most binary formats don't diff well.
rpedela 10y ago
No not really. CVS and git are basically the same thing but git is a lot better in many ways. I don't know if noms is really the git for data but those other tools are more like CVS in this analogy: clunky and slow.
- emmelaich 10y ago
  They are so different it's hard to know what you mean by "basically the same thing"
  - rpedela 10y ago
    Yes there are significant differences in terms of feature, implementation, etc but they have the same goal: manage source code versions.
    - emmelaich 10y ago
      That's how Git is used but I think that by design, it is foremost a userlevel append-only content-addressable file system. The difference between this and a version control system is a major part of why the UI is so awful.
rspeer 10y ago
Can you point me at some? Because I've tried a few immutable data stores and been disappointed every time. Given about 10 GB of JSON structures, I keep finding things that can't outperform the boring combo of: * Convert the versioned data to tab-separated values * COPY it into Postgres every time * Hope Postgres can act immutable enough even though it wasn't designed to be The closest I've come to improving this situation was Kyoto Cabinet (unusable license) and rolling my own damn hashtable (it worked okay but adding new kinds of indexes was just unmaintainable, there's a reason databases should be made by experts).
- ykler 10y ago
  I was curious about Camlistore and Tahoe-LAFS at one point. I didn't investigate them very thoroughly, however. How do they compare?
- deleted 10y ago
  [deleted]
- sjezewski 10y ago
  (Disclaimer - I work at pachyderm) http://pachyderm.io http://pachyderm.io Pachyderm is git for data. We work hard to make sure we can store data of different types (binary, text, json) efficiently. We also work hard to give you good mechanisms to read the data in a distributed way. I'd be curious how this suits your purposes.
  - rspeer 10y ago
    Just started looking at Pachyderm. While I can see how a git-based filesystem can help with some use cases, does it do any kind of indexing at all? I see that the FAQ recommends exporting the data from Pachyderm into PostgreSQL, which leaves me where I am now.

AvenueIngres 10y ago

So.. CRDTs?

aboodman 10y ago
https://github.com/attic-labs/noms/blob/master/doc/faq.md#why-dont-you-just-use-crdts https://github.com/attic-labs/noms/blob/master/doc/faq.md#wh...
- ergl 10y ago
  Those reasons are a bit outdated, you can certainly have add-and-remove crdts with or-sets, and you can do so with almost no garbage. One of the Tomtom engineers explained one approach at Strangeloop last month: https://www.youtube.com/watch?v=veeWamWy8dk https://www.youtube.com/watch?v=veeWamWy8dk

aboodman 10y ago

Noms performs diff and merge in time proportional to size of the diff. The size of the source data is not really relevant. One way to think of Noms is that is an index optimized for computing diffs. Here's a screencast that shows off Noms diffing things fast: https://www.youtube.com/watch?v=Zeg9CY3BMes https://www.youtube.com/watch?v=Zeg9CY3BMes

dwhitena 10y ago

I definitely agree here. As a data scientist, sometimes it seems like we are in the wild west as far as reproducibility and versioning of our analyses. This seems like an interesting project that tackles some of the data versioning stuff. However, I believe that, at least in data science, we need data versioning closely tied to the analyses themselves for complete reproducibility. That is, we need the versioning tied to the inputs/outputs of data pipeline stages, such that we can reproduce pipeline runs at any time and incrementally improve and run pipelines based on diffs in data. As mentioned elsewhere in the comments, Pachyderm (http://pachyderm.io/ http://pachyderm.io/) does exactly this. Working both as git for data, but also enabling data pipelining and analyses with the data versioning.