9 ms·

I want ACID in my databases: - Atomic - Yep, but done with a global lock. - Consistency - Yep. Since it's NoSQL, there's no referential checks, so thi

I want ACID in my databases: - Atomic - Yep, but done with a global lock. - Consistency - Yep. Since it's NoSQL, there's no referential checks, so this is easy. Git itself probably wants index / commit tree has to be readable, and all attributes.json files be valid. I don't see any reason it wouldn't be. - Isolated - Sure, works fine, due to that global lock. - Durable - Check. Once it's written, it's written. So, git doesn't get you any of this for free, and getting Atomic / Isolated is done by introducing a big ugly global lock. I guess I don't see the appeal of using git (or even a model like git) because it doesn't get you any interesting necessary features for free. I see the backup and history thing as a secondary feature -- neat, but not worth sacrificing ACID for. That said, cool idea -- there's not a lot of code you need to get to a reasonably small codebase.

bergie 16y ago
Wouldn't atomicity come from doing things on per-commit basis? Also, you're not taking into account the things that Git does give you that typical databases don't, like branching and merges
- technoweenie 16y ago
  Git doesn't give you merges, it gives you merge conflicts. I find it similar to how Riak uses vector clocks (http://blog.basho.com/2010/01/29/why-vector-clocks-are-easy/ http://blog.basho.com/2010/01/29/why-vector-clocks-are-easy/). It knows enough to tell you there's a conflict, but requires you (or the app) to tell it which version is the right one.
  - pauldowman 16y ago
    Right, it's either a trivial merge when you commit the transaction, or it's a conflict (i.e. two transactions updated the same attribute of the same entity). There's no automatically fixing a conflict so the second transaction fails. Since there's full isolation that's pretty standard optimistic locking: http://en.wikipedia.org/wiki/Optimistic_locking http://en.wikipedia.org/wiki/Optimistic_locking In most cases the transaction would succeed if it was automatically retried.
po 16y ago
Maybe you should just think of it as the NoSQL version of sqlite3 but with added version control.
- jemfinch 16y ago
  What does that even mean?
  - deleted 16y ago
    [deleted]
  - JoachimSchipper 16y ago
    I think "easy to set up" combined with "less crappy than expected". Although SQLite isn't very crappy at all...
  - chopsueyar 16y ago
    Because sqlite stores everything in a flatfile, so your are just doing version control on flatfile changes.
  - po 16y ago
    Sqlite is often used as a quick and dirty database by projects that don't want to burden their developers with the setup overhead of mysql or postgres. It's a tiny, rock-solid executable that stores data in a single binary file and supports sql access. Access to the file is globally locked. It's good enough for testing out a project on a single developer's machine but the global lock makes it hard to use in a multi-user production scenario. In a similar vein I could see git as a widely available, rock-solid, "quick and dirty" way for a project to store and retrieve documents.
- deleted 16y ago
  [deleted]
technoweenie 16y ago
What global lock? Are you talking about the index? There are options for writing the index to separate directories. You can write each blob/tree separately, build an index in a unique temporary directory, and then the commit. Is there something else I'm missing?
- fhars 16y ago
  Concurrent updates to the same object. You could get around some of the limitations by creating a branch for every session (though probably not on a high volume site), but then you will have lots of interesting problems once a session ends and you have to merge all changes back into master.
  - technoweenie 16y ago
    Ahh, yes this would be a problem on GitModel. I was thinking of Git at a lower level, since you can't update objects without also changing the SHA.
- pauldowman 16y ago
  GitModel itself has a global lock (for now), here: https://github.com/pauldowman/gitmodel/blob/master/lib/gitmodel/transaction.rb#L18 https://github.com/pauldowman/gitmodel/blob/master/lib/gitmo... I think it's easily fixed with a merge, as I described here: http://news.ycombinator.com/item?id=1929691 http://news.ycombinator.com/item?id=1929691
vog 16y ago
> - Consistency - > Since it's NoSQL, there's no referential checks, so this is easy. Consistency is quite easily achieved by putting the consistency checks into appropriate Git hooks. That way, Git would not allow any commit/pull of inconsistent data. A "rollback" is then trivial, as this only has to be done in the local working directory. So regarding consistency, Git provides everything that SQL databases provide, not just the limited NoSQL interpretation of "consistency". Also note that this already happens in the traditional use of Git: There, consistency checks are called "building the application and running the test suite". It is a vital part of continuous integration systems. So consistency checks are not only theoretically possible, but already actively practiced with Git.
pauldowman 16y ago
I'm pretty sure I can fix that global lock, but I'm focusing on getting the basics solid before focusing on performance. My idea was that if there are concurrent transactions, then you just need a merge when the second one commits. This is "optimistic locking": since they're probably updating different subdirectories it would be a trivial merge. If they update the same attribute of the same entity (i.e. they both change the same line of the same attributes.json file) then there's a conflict and the second commit fails. This still requires a lock but only a tiny one, just at the moment of actually updating refs/heads/<branchname> (whereas right now it's around the whole transaction which obviously sucks). Are there any flaws in that? The reason I haven't done it yet even though it sounds pretty trivial is that with Grit (the Ruby Git library) the merge will need to use the working directory and that's a problem for concurrency (and also because I want the working directory to be useable by a human). I was thinking that with a bit of extra work I could make it happen in memory somehow. Paul
chopsueyar 16y ago
I feel it is similair to the motivation when one attempts to install Linux on some random device. I expect to see a lot more 'interesting' projects that create NoSQL datastores in creative ways.

bergie 16y ago

Wouldn't atomicity come from doing things on per-commit basis? Also, you're not taking into account the things that Git does give you that typical databases don't, like branching and merges

technoweenie 16y ago
Git doesn't give you merges, it gives you merge conflicts. I find it similar to how Riak uses vector clocks (http://blog.basho.com/2010/01/29/why-vector-clocks-are-easy/ http://blog.basho.com/2010/01/29/why-vector-clocks-are-easy/). It knows enough to tell you there's a conflict, but requires you (or the app) to tell it which version is the right one.
- pauldowman 16y ago
  Right, it's either a trivial merge when you commit the transaction, or it's a conflict (i.e. two transactions updated the same attribute of the same entity). There's no automatically fixing a conflict so the second transaction fails. Since there's full isolation that's pretty standard optimistic locking: http://en.wikipedia.org/wiki/Optimistic_locking http://en.wikipedia.org/wiki/Optimistic_locking In most cases the transaction would succeed if it was automatically retried.

po 16y ago

Maybe you should just think of it as the NoSQL version of sqlite3 but with added version control.

jemfinch 16y ago
What does that even mean?
- deleted 16y ago
  [deleted]
- JoachimSchipper 16y ago
  I think "easy to set up" combined with "less crappy than expected". Although SQLite isn't very crappy at all...
- chopsueyar 16y ago
  Because sqlite stores everything in a flatfile, so your are just doing version control on flatfile changes.
- po 16y ago
  Sqlite is often used as a quick and dirty database by projects that don't want to burden their developers with the setup overhead of mysql or postgres. It's a tiny, rock-solid executable that stores data in a single binary file and supports sql access. Access to the file is globally locked. It's good enough for testing out a project on a single developer's machine but the global lock makes it hard to use in a multi-user production scenario. In a similar vein I could see git as a widely available, rock-solid, "quick and dirty" way for a project to store and retrieve documents.
deleted 16y ago
[deleted]

technoweenie 16y ago

What global lock? Are you talking about the index? There are options for writing the index to separate directories. You can write each blob/tree separately, build an index in a unique temporary directory, and then the commit. Is there something else I'm missing?

fhars 16y ago
Concurrent updates to the same object. You could get around some of the limitations by creating a branch for every session (though probably not on a high volume site), but then you will have lots of interesting problems once a session ends and you have to merge all changes back into master.
- technoweenie 16y ago
  Ahh, yes this would be a problem on GitModel. I was thinking of Git at a lower level, since you can't update objects without also changing the SHA.
pauldowman 16y ago
GitModel itself has a global lock (for now), here: https://github.com/pauldowman/gitmodel/blob/master/lib/gitmodel/transaction.rb#L18 https://github.com/pauldowman/gitmodel/blob/master/lib/gitmo... I think it's easily fixed with a merge, as I described here: http://news.ycombinator.com/item?id=1929691 http://news.ycombinator.com/item?id=1929691

vog 16y ago

> - Consistency - > Since it's NoSQL, there's no referential checks, so this is easy. Consistency is quite easily achieved by putting the consistency checks into appropriate Git hooks. That way, Git would not allow any commit/pull of inconsistent data. A "rollback" is then trivial, as this only has to be done in the local working directory. So regarding consistency, Git provides everything that SQL databases provide, not just the limited NoSQL interpretation of "consistency". Also note that this already happens in the traditional use of Git: There, consistency checks are called "building the application and running the test suite". It is a vital part of continuous integration systems. So consistency checks are not only theoretically possible, but already actively practiced with Git.

pauldowman 16y ago

I'm pretty sure I can fix that global lock, but I'm focusing on getting the basics solid before focusing on performance. My idea was that if there are concurrent transactions, then you just need a merge when the second one commits. This is "optimistic locking": since they're probably updating different subdirectories it would be a trivial merge. If they update the same attribute of the same entity (i.e. they both change the same line of the same attributes.json file) then there's a conflict and the second commit fails. This still requires a lock but only a tiny one, just at the moment of actually updating refs/heads/<branchname> (whereas right now it's around the whole transaction which obviously sucks). Are there any flaws in that? The reason I haven't done it yet even though it sounds pretty trivial is that with Grit (the Ruby Git library) the merge will need to use the working directory and that's a problem for concurrency (and also because I want the working directory to be useable by a human). I was thinking that with a bit of extra work I could make it happen in memory somehow. Paul

chopsueyar 16y ago

I feel it is similair to the motivation when one attempts to install Linux on some random device. I expect to see a lot more 'interesting' projects that create NoSQL datastores in creative ways.