6 ms·

Git as a NoSQL datastore

84 points by joshfraser 16y ago

silentbicycle 16y ago
Well, sure, git provides an append/log-based distributed hash store. It's probably not what you're looking for, though: It doesn't have a library for efficient in-process access, so you need to spawn a git shell command per operation, with somewhat opinionated semantics. It's also GPL'd. It's good for prototyping, though. (And I hear it's good at managing changes for source code!)
- WALoeIII 16y ago
  "It doesn't have a library for efficient in-process access" https://github.com/mojombo/grit https://github.com/mojombo/grit
  - timf 16y ago
    There is a Python port, too: http://packages.python.org/GitPython/0.3.1/ http://packages.python.org/GitPython/0.3.1/ ( http://gitorious.org/git-python http://gitorious.org/git-python )
    - deleted 16y ago
      [deleted]
  - silentbicycle 16y ago
    Ah, via Ruby. RUBY?!??! Ah. Delightful. YECCH So, uh, can't we just make an efficient, well-designed, C library for accessing those data structures, write BSD-licensed wrappers as consenting adults, and get on with our lives? I'm speechless. Is writing an append-only log data-structure system really that hard? Ok, then.
    - technoweenie 16y ago
      Sure: https://github.com/libgit2/libgit2 https://github.com/libgit2/libgit2.
      - deleted 16y ago
        [deleted]
    - holman 16y ago
      We use Grit extensively all over GitHub. It works quite well. I've yet to have a seizure from it. At the same time, we're also funding work on libgit2, which is an efficient, well-designed C library for accessing those data structures. There's also wrappers for consenting adults. http://libgit2.github.com http://libgit2.github.com
      - silentbicycle 16y ago
        Thanks for your informative response. Are you interested in a Lua wrapper?
        pygy_ 16y ago
        I would love to see this happening. I could then port gitmodel on top of your lib and get a NoSQL DB, easy to sync with a single user desktop app for the iOS platform (in conjunction with Wax).
        silentbicycle 16y ago
        I may get to it this weekend. I'll keep you posted.
    - aditya 16y ago
      What's stopping you?
      - deleted 16y ago
        [deleted]
      - deleted 16y ago
        [deleted]
      - silentbicycle 16y ago
        Git itself works fine, for my purposes. If I wrote a C library for managing git's data structures, it would basically be out of spite. I may make a Lua wrapper for libgit2, though - it's more what I had in mind (though it's too bad it's GPL'd). I have a library for managing distributed graphs on top of an append-only log file, but it's got different design trade-offs than git does - It's for a specific C/Lua project, and the semantics of the data don't match git's.
        asb 16y ago
        libgit2 is GPLed, but importantly also has a libgcc-style exception.
  - pauldowman 16y ago
    Yep, GitModel uses Grit.
trjordan 16y ago
I want ACID in my databases: - Atomic - Yep, but done with a global lock. - Consistency - Yep. Since it's NoSQL, there's no referential checks, so this is easy. Git itself probably wants index / commit tree has to be readable, and all attributes.json files be valid. I don't see any reason it wouldn't be. - Isolated - Sure, works fine, due to that global lock. - Durable - Check. Once it's written, it's written. So, git doesn't get you any of this for free, and getting Atomic / Isolated is done by introducing a big ugly global lock. I guess I don't see the appeal of using git (or even a model like git) because it doesn't get you any interesting necessary features for free. I see the backup and history thing as a secondary feature -- neat, but not worth sacrificing ACID for. That said, cool idea -- there's not a lot of code you need to get to a reasonably small codebase.
- bergie 16y ago
  Wouldn't atomicity come from doing things on per-commit basis? Also, you're not taking into account the things that Git does give you that typical databases don't, like branching and merges
  - technoweenie 16y ago
    Git doesn't give you merges, it gives you merge conflicts. I find it similar to how Riak uses vector clocks (http://blog.basho.com/2010/01/29/why-vector-clocks-are-easy/ http://blog.basho.com/2010/01/29/why-vector-clocks-are-easy/). It knows enough to tell you there's a conflict, but requires you (or the app) to tell it which version is the right one.
    - pauldowman 16y ago
      Right, it's either a trivial merge when you commit the transaction, or it's a conflict (i.e. two transactions updated the same attribute of the same entity). There's no automatically fixing a conflict so the second transaction fails. Since there's full isolation that's pretty standard optimistic locking: http://en.wikipedia.org/wiki/Optimistic_locking http://en.wikipedia.org/wiki/Optimistic_locking In most cases the transaction would succeed if it was automatically retried.
po 16y ago
Also of interest and in a similar sort of spirit: Bup: Git based backups - https://github.com/apenwarr/bup https://github.com/apenwarr/bup
- pygy_ 16y ago
  In the same vein: Gibak. http://eigenclass.org/hiki/gibak-backup-system-introduction http://eigenclass.org/hiki/gibak-backup-system-introduction http://eigenclass.org/hiki/gibak-0.3.0 http://eigenclass.org/hiki/gibak-0.3.0 https://github.com/pangloss/gibak https://github.com/pangloss/gibak for an improved fork (last commit in 2008, but still...)
- pauldowman 16y ago
  Bup looks awesome. All the benefits of rsync plus the ability to do incremental backups. "Even when a backup is incremental, you don't have to worry about restoring the full backup, then each of the incrementals in turn; an incremental backup acts as if it's a full backup, it just takes less disk space" .... nice!
epynonymous 16y ago
have you done any performance tests? how well does git scale in general for when you have billions of files and versions? this gives me some ideas, good stuff.
- zoomzoom 16y ago
  i have seen that these huge deployments of git run into problems, that is why git is less than perfect as a system-wide backup tool.
Sirupsen 16y ago
This could be interesting for projects where keeping revisions would be handy, for instance a note-taking or document platform.
- rlpb 16y ago
  Yes - how about Tomboy/gnote integration?
derefr 16y ago
Huh, I was half-way to creating something in the same vein, while trying to use Git as the "world-state synchronization protocol" for a distributed MOO. Back to the (much-simplified) drawing-board :)
yatsyk 16y ago
I also thought about similar project. Git based active model has some drawbacks and not for every project for sure. It's not so good if there are many users making a lot of changes to data (like facebook). But it's very interesting for applications when few users make changes on site and site looks like static for rest of visitors (applications like any CMS). And we are getting a lot of cool git features for free.
Vitaly 16y ago
I don't think its intended to be used for frequently changed models like 'user'. It is much more suited for low change frequency document like models. For example 'content' model in a blog or cms. having history might not be a big deal, but having branches IS!. It is great for stuff like testing some new set of pages on a staging server and then 'adding' them to the currently running production server which kept changing while you were working on the stage. Try that with SQL!
- pauldowman 16y ago
  Yes, it would be a lot more appropriate for things that don't change that frequently. I actually started writing a blogging app for coders, and then succumbed to a bout of extraction distraction by generalizing the part that read the pages/posts from a Git repo: https://github.com/pauldowman/balisong https://github.com/pauldowman/balisong
JeffJenkins 16y ago
I wrote code to do this in python for a project. The biggest problem I ran into was that if you wanted the full history of X/Y/Z.json to contain the full history of Z.json, even if it had been moved, it ended up requiring two parallel structures. One structure with directories for the tree data, one for the raw data of that node. My intent was to use it in a document system using Operational Transforms, which side-stepped the issue of concurrent access; only the canonical representation of client data would need to be written, so it was serial writing
9ec4c12949a4f3 16y ago
I can't be the only one who felt sick when I saw the thread, can i?
uriel 16y ago
In a somewhat related note, see also Venti: http://doc.cat-v.org/plan_9/4th_edition/papers/venti/ http://doc.cat-v.org/plan_9/4th_edition/papers/venti/

84 points by joshfraser 16y ago

Well, sure, git provides an append/log-based distributed hash store. It's probably not what you're looking for, though: It doesn't have a library for efficient in-process access, so you need to spawn a git shell command per operation, with somewhat opinionated semantics. It's also GPL'd. It's good for prototyping, though. (And I hear it's good at managing changes for source code!)

WALoeIII 16y ago
"It doesn't have a library for efficient in-process access" https://github.com/mojombo/grit https://github.com/mojombo/grit
- timf 16y ago
  There is a Python port, too: http://packages.python.org/GitPython/0.3.1/ http://packages.python.org/GitPython/0.3.1/ ( http://gitorious.org/git-python http://gitorious.org/git-python )
  - deleted 16y ago
    [deleted]
- silentbicycle 16y ago
  Ah, via Ruby. RUBY?!??! Ah. Delightful. YECCH So, uh, can't we just make an efficient, well-designed, C library for accessing those data structures, write BSD-licensed wrappers as consenting adults, and get on with our lives? I'm speechless. Is writing an append-only log data-structure system really that hard? Ok, then.
  - technoweenie 16y ago
    Sure: https://github.com/libgit2/libgit2 https://github.com/libgit2/libgit2.
    - deleted 16y ago
      [deleted]
  - holman 16y ago
    We use Grit extensively all over GitHub. It works quite well. I've yet to have a seizure from it. At the same time, we're also funding work on libgit2, which is an efficient, well-designed C library for accessing those data structures. There's also wrappers for consenting adults. http://libgit2.github.com http://libgit2.github.com
    - silentbicycle 16y ago
      Thanks for your informative response. Are you interested in a Lua wrapper?
      - pygy_ 16y ago
        I would love to see this happening. I could then port gitmodel on top of your lib and get a NoSQL DB, easy to sync with a single user desktop app for the iOS platform (in conjunction with Wax).
        silentbicycle 16y ago
        I may get to it this weekend. I'll keep you posted.
  - aditya 16y ago
    What's stopping you?
    - deleted 16y ago
      [deleted]
    - deleted 16y ago
      [deleted]
    - silentbicycle 16y ago
      Git itself works fine, for my purposes. If I wrote a C library for managing git's data structures, it would basically be out of spite. I may make a Lua wrapper for libgit2, though - it's more what I had in mind (though it's too bad it's GPL'd). I have a library for managing distributed graphs on top of an append-only log file, but it's got different design trade-offs than git does - It's for a specific C/Lua project, and the semantics of the data don't match git's.
      - asb 16y ago
        libgit2 is GPLed, but importantly also has a libgcc-style exception.
- pauldowman 16y ago
  Yep, GitModel uses Grit.

trjordan 16y ago

I want ACID in my databases: - Atomic - Yep, but done with a global lock. - Consistency - Yep. Since it's NoSQL, there's no referential checks, so this is easy. Git itself probably wants index / commit tree has to be readable, and all attributes.json files be valid. I don't see any reason it wouldn't be. - Isolated - Sure, works fine, due to that global lock. - Durable - Check. Once it's written, it's written. So, git doesn't get you any of this for free, and getting Atomic / Isolated is done by introducing a big ugly global lock. I guess I don't see the appeal of using git (or even a model like git) because it doesn't get you any interesting necessary features for free. I see the backup and history thing as a secondary feature -- neat, but not worth sacrificing ACID for. That said, cool idea -- there's not a lot of code you need to get to a reasonably small codebase.

bergie 16y ago
Wouldn't atomicity come from doing things on per-commit basis? Also, you're not taking into account the things that Git does give you that typical databases don't, like branching and merges
- technoweenie 16y ago
  Git doesn't give you merges, it gives you merge conflicts. I find it similar to how Riak uses vector clocks (http://blog.basho.com/2010/01/29/why-vector-clocks-are-easy/ http://blog.basho.com/2010/01/29/why-vector-clocks-are-easy/). It knows enough to tell you there's a conflict, but requires you (or the app) to tell it which version is the right one.
  - pauldowman 16y ago
    Right, it's either a trivial merge when you commit the transaction, or it's a conflict (i.e. two transactions updated the same attribute of the same entity). There's no automatically fixing a conflict so the second transaction fails. Since there's full isolation that's pretty standard optimistic locking: http://en.wikipedia.org/wiki/Optimistic_locking http://en.wikipedia.org/wiki/Optimistic_locking In most cases the transaction would succeed if it was automatically retried.

po 16y ago

Also of interest and in a similar sort of spirit: Bup: Git based backups - https://github.com/apenwarr/bup https://github.com/apenwarr/bup

pygy_ 16y ago
In the same vein: Gibak. http://eigenclass.org/hiki/gibak-backup-system-introduction http://eigenclass.org/hiki/gibak-backup-system-introduction http://eigenclass.org/hiki/gibak-0.3.0 http://eigenclass.org/hiki/gibak-0.3.0 https://github.com/pangloss/gibak https://github.com/pangloss/gibak for an improved fork (last commit in 2008, but still...)
pauldowman 16y ago
Bup looks awesome. All the benefits of rsync plus the ability to do incremental backups. "Even when a backup is incremental, you don't have to worry about restoring the full backup, then each of the incrementals in turn; an incremental backup acts as if it's a full backup, it just takes less disk space" .... nice!

epynonymous 16y ago

have you done any performance tests? how well does git scale in general for when you have billions of files and versions? this gives me some ideas, good stuff.

zoomzoom 16y ago
i have seen that these huge deployments of git run into problems, that is why git is less than perfect as a system-wide backup tool.

Sirupsen 16y ago

This could be interesting for projects where keeping revisions would be handy, for instance a note-taking or document platform.

rlpb 16y ago
Yes - how about Tomboy/gnote integration?

derefr 16y ago

Huh, I was half-way to creating something in the same vein, while trying to use Git as the "world-state synchronization protocol" for a distributed MOO. Back to the (much-simplified) drawing-board :)

yatsyk 16y ago

I also thought about similar project. Git based active model has some drawbacks and not for every project for sure. It's not so good if there are many users making a lot of changes to data (like facebook). But it's very interesting for applications when few users make changes on site and site looks like static for rest of visitors (applications like any CMS). And we are getting a lot of cool git features for free.

Vitaly 16y ago

I don't think its intended to be used for frequently changed models like 'user'. It is much more suited for low change frequency document like models. For example 'content' model in a blog or cms. having history might not be a big deal, but having branches IS!. It is great for stuff like testing some new set of pages on a staging server and then 'adding' them to the currently running production server which kept changing while you were working on the stage. Try that with SQL!

pauldowman 16y ago
Yes, it would be a lot more appropriate for things that don't change that frequently. I actually started writing a blogging app for coders, and then succumbed to a bout of extraction distraction by generalizing the part that read the pages/posts from a Git repo: https://github.com/pauldowman/balisong https://github.com/pauldowman/balisong

JeffJenkins 16y ago

I wrote code to do this in python for a project. The biggest problem I ran into was that if you wanted the full history of X/Y/Z.json to contain the full history of Z.json, even if it had been moved, it ended up requiring two parallel structures. One structure with directories for the tree data, one for the raw data of that node. My intent was to use it in a document system using Operational Transforms, which side-stepped the issue of concurrent access; only the canonical representation of client data would need to be written, so it was serial writing

9ec4c12949a4f3 16y ago

I can't be the only one who felt sick when I saw the thread, can i?

uriel 16y ago

In a somewhat related note, see also Venti: http://doc.cat-v.org/plan_9/4th_edition/papers/venti/ http://doc.cat-v.org/plan_9/4th_edition/papers/venti/