14 ms·

Git as a NoSql Database (2016)

143 points by EntICOnc 6y ago

chmaynard 6y ago
Fascinating article. Looks like it was posted to HN at least six times before today and got no traction. In my browser, the examples are sometimes a concatenation of the command and the output. Does anyone else see this?
- escanor 6y ago
  yes, i see this as well. at first it was confusing, but later i understood it was the output.
cschulee 6y ago
"Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should."
cryptonector 6y ago
I should write a SQL as a NoSQL database blog post, showing a trivial one-table schema that does all you could ever want in a NoSQL, especially after choosing to ignore that it still has SQL.
- devoutsalsa 6y ago
  Someone wrote a SQL wrapper around InnoDB & called it MySQL XD
  - 10000truths 6y ago
    They should sell that to a large tech corp, I bet they'd make a killing.
- richardwhiuk 6y ago
  sqlite and json.1 gets you pretty close to emulating a more powerful nosql database.
  - cryptonector 6y ago
    I'm aware. My comment was a joke, but evidently it's been well-received.
- munk-a 6y ago
  This may or may not be the case anymore[1] - but for a long while Postgres' managed to outperform MongoDB when restricted to just using single JSON column tables. One of the larger relational DBs beating the OG NoSQL will always fill me with amusement. 1. Ed. looks like this is still the case, but please remember this is MongoDB specifically. AFAICT MongoDB is mostly a dead branch at this point and a comparison against Redis or something else that's had continued investment might be more fair. I'm just happy chilling in my SQL world - I'm not super up-to-date on the NoSQL market.
  - Cir0X 6y ago
    I can't find any sources for this. Can you provide one?
    - munk-a 6y ago
      Here's one (read the article in detail, apparently this is up to pretty recently...) https://www.enterprisedb.com/news/new-benchmarks-show-postgres-dominating-mongodb-varied-workloads https://www.enterprisedb.com/news/new-benchmarks-show-postgr... Also, here's SiSense sensationalizing it for easy digest: https://www.sisense.com/blog/postgres-vs-mongodb-for-storing-json-data/ https://www.sisense.com/blog/postgres-vs-mongodb-for-storing...
- Lorin 6y ago
  check out RedBeanPHP - it does some interesting stuff for schema liquidity
- dmingod666 6y ago
  Postgres has good support for JSON.
- wingspan 6y ago
  You joke, but this is basically how Facebook uses MySQL for the main user table and for the majority of Entities: https://blog.yugabyte.com/facebooks-user-db-is-it-sql-or-nosql/ https://blog.yugabyte.com/facebooks-user-db-is-it-sql-or-nos...
math-dev 6y ago
Thank you, was a nice read (particularly the conclusion, which was insightful and made sense)
Spivak 6y ago
I don't know the name of this type of learning style but I would eat up a tutorial on Git that started with the plumbing commands like this and worked their way up. I think this is oddly one of the clearest explanations of Git I've ever read. I know that wasn't really the point of the post but still.
- bacon_waffle 6y ago
  This doesn't start with the plumbing commands, but it's along those lines and seemed helpful to some new-to-git folks at work when we started switching to it last year: https://eagain.net/articles/git-for-computer-scientists/ https://eagain.net/articles/git-for-computer-scientists/ If you're already comfortable with git, the Git Internals section of the book may be better: https://git-scm.com/book/en/v2/Git-Internals-Git-Objects https://git-scm.com/book/en/v2/Git-Internals-Git-Objects
- cryptonector 6y ago
  My attempt: https://gist.github.com/nicowilliams/a6e5c9131767364ce2f4b3996549748d https://gist.github.com/nicowilliams/a6e5c9131767364ce2f4b39...
- chmaynard 6y ago
  This article/video by Mary Rose Cook is helpful: https://maryrosecook.com/blog/post/git-from-the-inside-out https://maryrosecook.com/blog/post/git-from-the-inside-out
- finnthehuman 6y ago
  Write yourself a git https://wyag.thb.lt/ https://wyag.thb.lt/
- weaksauce 6y ago
  many people have done it but here's another one https://jwiegley.github.io/git-from-the-bottom-up/ https://jwiegley.github.io/git-from-the-bottom-up/
- bspammer 6y ago
  There’s nothing like writing it yourself for understanding something. This tutorial goes through reimplementing the basics of git in a few lines of Python, it’s surprisingly simple and doable in an afternoon: https://wyag.thb.lt/ https://wyag.thb.lt/
zachmu 6y ago
Or for a SQL database with Git versioning semantics: https://github.com/dolthub/dolt https://github.com/dolthub/dolt
simon_acca 6y ago
Interesting exercise! Some pointers to related ideas: https://en.m.wikipedia.org/wiki/Persistent_data_structure https://en.m.wikipedia.org/wiki/Persistent_data_structure - https://docs.datomic.com/cloud/index.html https://docs.datomic.com/cloud/index.html - https://opencrux.com/ https://opencrux.com/ - https://github.com/attic-labs/noms https://github.com/attic-labs/noms - https://researcher.watson.ibm.com/researcher/files/us-leejinho/Multiversion_support_for_property_graph_databases___BigData_2017.pdf https://researcher.watson.ibm.com/researcher/files/us-leejin...
hungryhobo 6y ago
Most noSQL can guarantee at least two of consistency, availability and partition tolerance, this seem to drop the ball on all 3.
- ses1984 6y ago
  I can't tell if this is sarcasm.
  - hungryhobo 6y ago
    In what ways does git implement CAP better than a NoSQL database?
    - ninkendo 6y ago
      No system can get all three of CAP, that’s the whole point of CAP. Hence why your comment sounds like a joke of some sort.
- macintux 6y ago
  > Most noSQL can guarantee at least two of consistency, availability and partition tolerance... You mean "at most" two.
  - hungryhobo 6y ago
    ah yes, thank you. brain fart on my part.
fiddlerwoaroof 6y ago
Tangentially I started a re-implementation of git in Common Lisp[1], and have completed parsers for most of the file formats except delta encoded objects. Does anyone happen to know of an implementation or tests for delta encoding I could consult that is available under an MIT-like license? (BSD, Apache v2, etc.) [1]: https://github.com/fiddlerwoaroof/cl-git https://github.com/fiddlerwoaroof/cl-git
- ori_b 6y ago
  I just used git-fsck, and my implemention is here (both read and write, hosted using git9 on 9front): http://shithub.us/ori/git9/724c516a6eda0063439457a6701ef0d7ef37eb93/pack.c/f.html http://shithub.us/ori/git9/724c516a6eda0063439457a6701ef0d7e... http://shithub.us/ori/git9/724c516a6eda0063439457a6701ef0d7ef37eb93/delta.c/f.html http://shithub.us/ori/git9/724c516a6eda0063439457a6701ef0d7e... as far as I'm aware, the OpenBSD implemention of git, Game of trees, is adapting this code too.
  - fiddlerwoaroof 6y ago
    Thanks, delta encoding stumped me a bit because the relevant git documentation was a bit ambiguous. Maybe I’ll implement this tonight.
ioquatix 6y ago
For the lolz: https://github.com/ioquatix/relaxo https://github.com/ioquatix/relaxo
- pokstad 6y ago
  Looks like it is inspired by CouchDB
  - ioquatix 6y ago
    Yes, it was. It was originally a front end and back end (query server) for CouchDB. But CouchDB 2 really went in a different direction to what I expected with query servers so I gave up and decided to pivot in a different direction with the code base. https://github.com/ioquatix/relaxo-query-server https://github.com/ioquatix/relaxo-query-server FYI.
- jaredcwhite 6y ago
  No lolz here! I actually need exactly this as I'll soon be starting work on a CMS-style editor for Git-based static sites, and it's all Ruby to boot. Awesome sauce!
  - 0xbadcafebee 6y ago
    lolz!
  - nojvek 6y ago
    Have you played with netlifycms.org? It's a Git based CMS that interfaces directly with GitHub apis.
  - FrontAid 6y ago
    Maybe you can take some inspiration from FrontAid CMS https://frontaid.io/ https://frontaid.io/
simonw 6y ago
In almost every database-backed application I've ever built someone, at some point, inevitable asks for the ability to see what changes were made when and by whom. My current preferred strategy for dealing with this (at least for any table smaller than a few GBs) is to dump the entire table contents to a git repository on a schedule. I've run this for a few toy projects and it seems to work really well. I'm ready to try it with something production-scale the next time the opportunity presents itself.
- sedeki 6y ago
  So each commit is exactly one change to a relevant table? I guess you can't just plug this into a common system of rotating logs easily, as there might be several changes beteeen the log rotation. Also, I guess you'd need a user friendlier interface to actually display who made the change from the git repo. Anyway, interesting solution.
- lisper 6y ago
  > My current preferred strategy for dealing with this (at least for any table smaller than a few GBs) is to dump the entire table contents to a git repository on a schedule. How does that solve the problem? All this would do is give you snapshots of the state of the DB at particular points in time. The only way to know who changed what when is to keep track of those changes as they are made, and the best way to do that is to stick the changes in an eventlog table.
  - zikzak 6y ago
    A developer joined a team I was on once. The boss was infamous for hiring anyone with a pulse and assigning a small client project. Sink or swim. As the senior dev it was my job to provide info as needed and subtly assess the new, potential rockstar code ninjas. Anyway, this new guy says "I store all my dates in sql as text in iso format so I can easily sort them".
    - com2kid 6y ago
      That is... almost not horrible? If he had said anything other than "in sql" it would be the correct answer. "I store all dates in spreadsheets in ISO format" "I prepend ISO dates to marketing campaign names so I can easily sort them in the horrible web frontends that doesn't have a sort by date option" "I start all my blog entries with ISO dates..." I mean, just... so close!
skybrian 6y ago
I haven't used it myself, but based on their blog, Dolt looks like a nicer way to share tables of data in a git-like way, since you get both branches and merges and SQL support. https://www.dolthub.com/ https://www.dolthub.com/
- schaefer 6y ago
  A big part of the magic of git is that I can run it locally. And sync up with my team much later. I’m not seeing a way to do anything similar with dolthub.
  - timsehn 6y ago
    That's Dolt. https://github.com/dolthub/dolt https://github.com/dolthub/dolt
chubot 6y ago
I just started using git annex, which is an extension for large files. https://git-annex.branchable.com/ https://git-annex.branchable.com/ I like it a lot so far, and I think it could be used for "cloud" stuff, not just backups. I'd like to see Debian/Docker/PyPI/npm repositories in git annex, etc. It has lazy checkouts which fits that use case. By default you just sync the metadata with 'git annex sync', and then you can get content with 'git annex get FILE', or git annex sync --content. Can anyone see a reason why not? Those kinds of repositories all seem to have weird custom protocols. I'd rather just sync the metadata and do the query/package resolution locally. It might be a bigger that way, but you only have fully sync it once, and the rest are incremental.
- GordonS 6y ago
  I'm a little confused about what git annex is - I think it's perhaps a basic file sync utility that uses git behind the scenes?
  - chubot 6y ago
    Yeah it solves the problem of putting big binaries that don't compress well inside git. If you've ever tried that, then git will start to choke around a few gigabytes (I think it's the packing/diffing algorithms). Github recommends that you keep repos less than 1 GB and definitely less than 5 GB, and they probably have a hard limit. So what git annex does is simply store symlinks to big files inside .git/annex, and then it has algorithms for managing and syncing the big files. I don't love symlinks and neither does the author, but it seems to work fine. I just do ls -L -l instead of ls -l to follow the symlinks. I think package repos are something like 300 GB, which should be easily manageable by git annex. And again you don't have to check out everything eagerly. I'm also pretty certain that git annex could support 3TB or 30TB repos if the file system has enough space. For container images, I think you could simple store layers as files which will save some space for many versions. There's also git LFS, which github supports, but git annex seems more truly distributed, which I like.
    - warp 6y ago
lisper 6y ago
> The reason why it says stupid in the man-pages is that it makes no assumptions about what content you store in it. That's not true. The assumption that you are storing text files is very much built in to the design of git, and specifically, into the design of its diff and merge algorithm. Those algorithms treat line breaks as privileged markers of the structure of the underlying data. This is the reason git does not play well with large binary files.
- mediocregopher 6y ago
  The diffing/merging stuff that you see when you use the git tool is really just some sugar on top of the underlying data which is being stored. If you look at what a commit actually contains, it's just a hash of the tree root, one or more parent commits identified by their hash, and some other metadata (author, date, etc). There's nothing about the commit that cares about the contents of its tree or its parents' trees. It's the git tooling on top of those structures which does. In git world this distinction is referred to as the "porcelain" vs "plumbing", the plumbing being the underlying structures and porcelain being the stuff most people actually use (diffs, merges, rebases, etc...)
  - lisper 6y ago
    Yes, I get that. But: > one or more parent commits identified by their hash The reason that matters is because you need that information to do merges. So merges are integral to git. They are not "porcelain". They are, arguably, the whole point.
    - tehbeard 6y ago
      Nothing in the "plumbing" defines what merge/diff algorithm is used or is even needed. You can pick any of the merge strategies, or plug in any diff utility. The plumbing cares only about one thing, that you have a valid data structure, a data structure that among many things tracks "what was/were the previous state/s before this commit?" as a pointer to the previous commit/s, nothing more. I can construct a git tree by hand / shell script just piping in full text/binary files, and git's porcelain will happily spit out "diffs" of the structure. Your porcelain on top of that decides what to do with that information, standard builds admittedly just provide diff tooling that focuses on , more expansive UIs can diff some binary data (e.g. GitHub shows diffs of images using a UI more suited for that data, with side by side, swiping, or fading to allow someone to examine the changes) > The assumption that you are storing text files is very much built in to the design of git I haven't dived deep in a while to git's source code, but the last time I did, this argument would only really hold true for packfiles / where delta encoding is used to compact the .git directory's contents as that does indeed seem tuned for generic text content instead of binary data. That isn't related to the diff/merge functionality of git though; that is a feature used to reduce disk space by reducing loose objects (full copies of the previous versions of files), trading increased compute time (rebuilding a previous version of a file from a set of diffs) for reduced disk space.
0xbadcafebee 6y ago
How about StackExchangeQL? Write your data as comments on StackExchange questions. To update the data, reply to your comment. It's like NoSQL because somebody else is hosting it for you and there's no schema.
bachmeier 6y ago
> You can query by key … and that’s about it. The only piece of good news here is that you can structure your data in folders in such a way that you can easily get content by prefix, but that’s about it. Any other query is off limits, unless you want to do a full recursive search. The only option here is to build indices specifically for querying. You can do this on a scheduled basis if staleness is of no concern or you can use git hooks to update indices as soon as a commit happens. Isn't the point of a database the ability to query? Why else would you want "Git as a NoSql database?" If this is really what you're after, maybe you should be using Fossil, for which the repo is an sqlite database that you can query like any sqlite database.
- js2 6y ago
  The point of a database is to hold data, and there are many key/value databases for which you can only query by key unless you add a secondary index. (e.g. Bigtable, DynamoDB, Riak, Redis, etc).
- MuffinFlavored 6y ago
  > Isn't the point of a database the ability to query? I was thinking about this in terms of the new CloudFlare Durable Objects open beta: https://blog.cloudflare.com/durable-objects-open-beta/ https://blog.cloudflare.com/durable-objects-open-beta/ Storing data is really only half the battle. How could you reimplement the poorest of a poor man's SQL on top of a key/value store? The idea I came up with is: whatever fields you want to query by need an index as a separate key. Maybe this would allow JOINs? Probably not?
no_wizard 6y ago
Isn't the problem attempting to be solved here solved by event sourcing[0]? Which lets you 're-create' your changes and is - ultimately- suppose to allow you effectively log each change in such a way its re-playable backwards and forwards [0]: https://martinfowler.com/eaaDev/EventSourcing.html https://martinfowler.com/eaaDev/EventSourcing.html
- hcarvalhoalves 6y ago
  Kind of, but there’s more to it. “Event sourcing” is the pattern being implemented by git’s immutable object database, but there’s also all the hashing and reference management.
patrickdevivo 6y ago
I've been very curious to explore this type of use case with askgit (https://github.com/augmentable-dev/askgit https://github.com/augmentable-dev/askgit) which was designed for running simple "slice and dice" queries and aggregations on git history (and change stats) for basic analytical purposes. I've been curious about how this could be applied to a small text+git based "db". Say, for a regular json or CSV dumps. This also reminds me of Dolt: https://github.com/dolthub/dolt https://github.com/dolthub/dolt which I believe has been on HN a couple times
rrdharan 6y ago
For completeness here’s Git as a SQL Database (2018): https://opensource.com/article/18/11/gitbase https://opensource.com/article/18/11/gitbase
blazeeboy 6y ago
You don't even need to write your key/value in a file. Git accept committing empty commit. and allow any message to be committed with it so content of your database can be the git commit message. I wrote about that concept before and tried to implement it but never went to the full potential (too many side projects) https://www.emadelsaid.com/git-as-messages-queue/ https://www.emadelsaid.com/git-as-messages-queue/ https://github.com/emad-elsaid/gitmq https://github.com/emad-elsaid/gitmq
abathologist 6y ago
Related https://irmin.org/ https://irmin.org/ (and in development since 2013) > A distributed database built on the same principles as Git (and which can use git as a backend)
- senorsmile 6y ago
  Is Irmin more like a database engine? I.e. you have to write code in OCaml in order to use it???
zimbatm 6y ago
Gerrit is using Git as a NoSql database. All the users, projects, reviews, ... are stored in Git repositories[1]. One of the main advantage of this approach is that replication becomes trivial, it's just a matter of running `git fetch` on all the repositories. The main disadvantage is that NodeDB only has one implementation (Java), and no CLI to play with it. [1]: https://gerrit-review.googlesource.com/Documentation/note-db.html https://gerrit-review.googlesource.com/Documentation/note-db...

143 points by EntICOnc 6y ago

Fascinating article. Looks like it was posted to HN at least six times before today and got no traction. In my browser, the examples are sometimes a concatenation of the command and the output. Does anyone else see this?

escanor 6y ago
yes, i see this as well. at first it was confusing, but later i understood it was the output.

cschulee 6y ago

"Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should."

cryptonector 6y ago

I should write a SQL as a NoSQL database blog post, showing a trivial one-table schema that does all you could ever want in a NoSQL, especially after choosing to ignore that it still has SQL.

devoutsalsa 6y ago
Someone wrote a SQL wrapper around InnoDB & called it MySQL XD
- 10000truths 6y ago
  They should sell that to a large tech corp, I bet they'd make a killing.
richardwhiuk 6y ago
sqlite and json.1 gets you pretty close to emulating a more powerful nosql database.
- cryptonector 6y ago
  I'm aware. My comment was a joke, but evidently it's been well-received.
munk-a 6y ago
This may or may not be the case anymore[1] - but for a long while Postgres' managed to outperform MongoDB when restricted to just using single JSON column tables. One of the larger relational DBs beating the OG NoSQL will always fill me with amusement. 1. Ed. looks like this is still the case, but please remember this is MongoDB specifically. AFAICT MongoDB is mostly a dead branch at this point and a comparison against Redis or something else that's had continued investment might be more fair. I'm just happy chilling in my SQL world - I'm not super up-to-date on the NoSQL market.
- Cir0X 6y ago
  I can't find any sources for this. Can you provide one?
  - munk-a 6y ago
    Here's one (read the article in detail, apparently this is up to pretty recently...) https://www.enterprisedb.com/news/new-benchmarks-show-postgres-dominating-mongodb-varied-workloads https://www.enterprisedb.com/news/new-benchmarks-show-postgr... Also, here's SiSense sensationalizing it for easy digest: https://www.sisense.com/blog/postgres-vs-mongodb-for-storing-json-data/ https://www.sisense.com/blog/postgres-vs-mongodb-for-storing...
Lorin 6y ago
check out RedBeanPHP - it does some interesting stuff for schema liquidity
dmingod666 6y ago
Postgres has good support for JSON.
wingspan 6y ago
You joke, but this is basically how Facebook uses MySQL for the main user table and for the majority of Entities: https://blog.yugabyte.com/facebooks-user-db-is-it-sql-or-nosql/ https://blog.yugabyte.com/facebooks-user-db-is-it-sql-or-nos...

math-dev 6y ago

Thank you, was a nice read (particularly the conclusion, which was insightful and made sense)

Spivak 6y ago

I don't know the name of this type of learning style but I would eat up a tutorial on Git that started with the plumbing commands like this and worked their way up. I think this is oddly one of the clearest explanations of Git I've ever read. I know that wasn't really the point of the post but still.

bacon_waffle 6y ago
This doesn't start with the plumbing commands, but it's along those lines and seemed helpful to some new-to-git folks at work when we started switching to it last year: https://eagain.net/articles/git-for-computer-scientists/ https://eagain.net/articles/git-for-computer-scientists/ If you're already comfortable with git, the Git Internals section of the book may be better: https://git-scm.com/book/en/v2/Git-Internals-Git-Objects https://git-scm.com/book/en/v2/Git-Internals-Git-Objects
cryptonector 6y ago
My attempt: https://gist.github.com/nicowilliams/a6e5c9131767364ce2f4b3996549748d https://gist.github.com/nicowilliams/a6e5c9131767364ce2f4b39...
chmaynard 6y ago
This article/video by Mary Rose Cook is helpful: https://maryrosecook.com/blog/post/git-from-the-inside-out https://maryrosecook.com/blog/post/git-from-the-inside-out
finnthehuman 6y ago
Write yourself a git https://wyag.thb.lt/ https://wyag.thb.lt/
weaksauce 6y ago
many people have done it but here's another one https://jwiegley.github.io/git-from-the-bottom-up/ https://jwiegley.github.io/git-from-the-bottom-up/
bspammer 6y ago
There’s nothing like writing it yourself for understanding something. This tutorial goes through reimplementing the basics of git in a few lines of Python, it’s surprisingly simple and doable in an afternoon: https://wyag.thb.lt/ https://wyag.thb.lt/

zachmu 6y ago

Or for a SQL database with Git versioning semantics: https://github.com/dolthub/dolt https://github.com/dolthub/dolt

simon_acca 6y ago

Interesting exercise! Some pointers to related ideas: https://en.m.wikipedia.org/wiki/Persistent_data_structure https://en.m.wikipedia.org/wiki/Persistent_data_structure - https://docs.datomic.com/cloud/index.html https://docs.datomic.com/cloud/index.html - https://opencrux.com/ https://opencrux.com/ - https://github.com/attic-labs/noms https://github.com/attic-labs/noms - https://researcher.watson.ibm.com/researcher/files/us-leejinho/Multiversion_support_for_property_graph_databases___BigData_2017.pdf https://researcher.watson.ibm.com/researcher/files/us-leejin...

hungryhobo 6y ago

Most noSQL can guarantee at least two of consistency, availability and partition tolerance, this seem to drop the ball on all 3.

ses1984 6y ago
I can't tell if this is sarcasm.
- hungryhobo 6y ago
  In what ways does git implement CAP better than a NoSQL database?
  - ninkendo 6y ago
    No system can get all three of CAP, that’s the whole point of CAP. Hence why your comment sounds like a joke of some sort.
macintux 6y ago
> Most noSQL can guarantee at least two of consistency, availability and partition tolerance... You mean "at most" two.
- hungryhobo 6y ago
  ah yes, thank you. brain fart on my part.

fiddlerwoaroof 6y ago

Tangentially I started a re-implementation of git in Common Lisp[1], and have completed parsers for most of the file formats except delta encoded objects. Does anyone happen to know of an implementation or tests for delta encoding I could consult that is available under an MIT-like license? (BSD, Apache v2, etc.) [1]: https://github.com/fiddlerwoaroof/cl-git https://github.com/fiddlerwoaroof/cl-git

ori_b 6y ago
I just used git-fsck, and my implemention is here (both read and write, hosted using git9 on 9front): http://shithub.us/ori/git9/724c516a6eda0063439457a6701ef0d7ef37eb93/pack.c/f.html http://shithub.us/ori/git9/724c516a6eda0063439457a6701ef0d7e... http://shithub.us/ori/git9/724c516a6eda0063439457a6701ef0d7ef37eb93/delta.c/f.html http://shithub.us/ori/git9/724c516a6eda0063439457a6701ef0d7e... as far as I'm aware, the OpenBSD implemention of git, Game of trees, is adapting this code too.
- fiddlerwoaroof 6y ago
  Thanks, delta encoding stumped me a bit because the relevant git documentation was a bit ambiguous. Maybe I’ll implement this tonight.

ioquatix 6y ago

For the lolz: https://github.com/ioquatix/relaxo https://github.com/ioquatix/relaxo

pokstad 6y ago
Looks like it is inspired by CouchDB
- ioquatix 6y ago
  Yes, it was. It was originally a front end and back end (query server) for CouchDB. But CouchDB 2 really went in a different direction to what I expected with query servers so I gave up and decided to pivot in a different direction with the code base. https://github.com/ioquatix/relaxo-query-server https://github.com/ioquatix/relaxo-query-server FYI.
jaredcwhite 6y ago
No lolz here! I actually need exactly this as I'll soon be starting work on a CMS-style editor for Git-based static sites, and it's all Ruby to boot. Awesome sauce!
- 0xbadcafebee 6y ago
  lolz!
- nojvek 6y ago
  Have you played with netlifycms.org? It's a Git based CMS that interfaces directly with GitHub apis.
- FrontAid 6y ago
  Maybe you can take some inspiration from FrontAid CMS https://frontaid.io/ https://frontaid.io/

simonw 6y ago

In almost every database-backed application I've ever built someone, at some point, inevitable asks for the ability to see what changes were made when and by whom. My current preferred strategy for dealing with this (at least for any table smaller than a few GBs) is to dump the entire table contents to a git repository on a schedule. I've run this for a few toy projects and it seems to work really well. I'm ready to try it with something production-scale the next time the opportunity presents itself.

sedeki 6y ago
So each commit is exactly one change to a relevant table? I guess you can't just plug this into a common system of rotating logs easily, as there might be several changes beteeen the log rotation. Also, I guess you'd need a user friendlier interface to actually display who made the change from the git repo. Anyway, interesting solution.
lisper 6y ago
> My current preferred strategy for dealing with this (at least for any table smaller than a few GBs) is to dump the entire table contents to a git repository on a schedule. How does that solve the problem? All this would do is give you snapshots of the state of the DB at particular points in time. The only way to know who changed what when is to keep track of those changes as they are made, and the best way to do that is to stick the changes in an eventlog table.
- zikzak 6y ago
  A developer joined a team I was on once. The boss was infamous for hiring anyone with a pulse and assigning a small client project. Sink or swim. As the senior dev it was my job to provide info as needed and subtly assess the new, potential rockstar code ninjas. Anyway, this new guy says "I store all my dates in sql as text in iso format so I can easily sort them".
  - com2kid 6y ago
    That is... almost not horrible? If he had said anything other than "in sql" it would be the correct answer. "I store all dates in spreadsheets in ISO format" "I prepend ISO dates to marketing campaign names so I can easily sort them in the horrible web frontends that doesn't have a sort by date option" "I start all my blog entries with ISO dates..." I mean, just... so close!

skybrian 6y ago

I haven't used it myself, but based on their blog, Dolt looks like a nicer way to share tables of data in a git-like way, since you get both branches and merges and SQL support. https://www.dolthub.com/ https://www.dolthub.com/

schaefer 6y ago
A big part of the magic of git is that I can run it locally. And sync up with my team much later. I’m not seeing a way to do anything similar with dolthub.
- timsehn 6y ago
  That's Dolt. https://github.com/dolthub/dolt https://github.com/dolthub/dolt

chubot 6y ago

I just started using git annex, which is an extension for large files. https://git-annex.branchable.com/ https://git-annex.branchable.com/ I like it a lot so far, and I think it could be used for "cloud" stuff, not just backups. I'd like to see Debian/Docker/PyPI/npm repositories in git annex, etc. It has lazy checkouts which fits that use case. By default you just sync the metadata with 'git annex sync', and then you can get content with 'git annex get FILE', or git annex sync --content. Can anyone see a reason why not? Those kinds of repositories all seem to have weird custom protocols. I'd rather just sync the metadata and do the query/package resolution locally. It might be a bigger that way, but you only have fully sync it once, and the rest are incremental.

GordonS 6y ago
I'm a little confused about what git annex is - I think it's perhaps a basic file sync utility that uses git behind the scenes?
- chubot 6y ago
  Yeah it solves the problem of putting big binaries that don't compress well inside git. If you've ever tried that, then git will start to choke around a few gigabytes (I think it's the packing/diffing algorithms). Github recommends that you keep repos less than 1 GB and definitely less than 5 GB, and they probably have a hard limit. So what git annex does is simply store symlinks to big files inside .git/annex, and then it has algorithms for managing and syncing the big files. I don't love symlinks and neither does the author, but it seems to work fine. I just do ls -L -l instead of ls -l to follow the symlinks. I think package repos are something like 300 GB, which should be easily manageable by git annex. And again you don't have to check out everything eagerly. I'm also pretty certain that git annex could support 3TB or 30TB repos if the file system has enough space. For container images, I think you could simple store layers as files which will save some space for many versions. There's also git LFS, which github supports, but git annex seems more truly distributed, which I like.
  - warp 6y ago

lisper 6y ago

> The reason why it says stupid in the man-pages is that it makes no assumptions about what content you store in it. That's not true. The assumption that you are storing text files is very much built in to the design of git, and specifically, into the design of its diff and merge algorithm. Those algorithms treat line breaks as privileged markers of the structure of the underlying data. This is the reason git does not play well with large binary files.

mediocregopher 6y ago
The diffing/merging stuff that you see when you use the git tool is really just some sugar on top of the underlying data which is being stored. If you look at what a commit actually contains, it's just a hash of the tree root, one or more parent commits identified by their hash, and some other metadata (author, date, etc). There's nothing about the commit that cares about the contents of its tree or its parents' trees. It's the git tooling on top of those structures which does. In git world this distinction is referred to as the "porcelain" vs "plumbing", the plumbing being the underlying structures and porcelain being the stuff most people actually use (diffs, merges, rebases, etc...)
- lisper 6y ago
  Yes, I get that. But: > one or more parent commits identified by their hash The reason that matters is because you need that information to do merges. So merges are integral to git. They are not "porcelain". They are, arguably, the whole point.
  - tehbeard 6y ago
    Nothing in the "plumbing" defines what merge/diff algorithm is used or is even needed. You can pick any of the merge strategies, or plug in any diff utility. The plumbing cares only about one thing, that you have a valid data structure, a data structure that among many things tracks "what was/were the previous state/s before this commit?" as a pointer to the previous commit/s, nothing more. I can construct a git tree by hand / shell script just piping in full text/binary files, and git's porcelain will happily spit out "diffs" of the structure. Your porcelain on top of that decides what to do with that information, standard builds admittedly just provide diff tooling that focuses on , more expansive UIs can diff some binary data (e.g. GitHub shows diffs of images using a UI more suited for that data, with side by side, swiping, or fading to allow someone to examine the changes) > The assumption that you are storing text files is very much built in to the design of git I haven't dived deep in a while to git's source code, but the last time I did, this argument would only really hold true for packfiles / where delta encoding is used to compact the .git directory's contents as that does indeed seem tuned for generic text content instead of binary data. That isn't related to the diff/merge functionality of git though; that is a feature used to reduce disk space by reducing loose objects (full copies of the previous versions of files), trading increased compute time (rebuilding a previous version of a file from a set of diffs) for reduced disk space.

0xbadcafebee 6y ago

How about StackExchangeQL? Write your data as comments on StackExchange questions. To update the data, reply to your comment. It's like NoSQL because somebody else is hosting it for you and there's no schema.

bachmeier 6y ago

> You can query by key … and that’s about it. The only piece of good news here is that you can structure your data in folders in such a way that you can easily get content by prefix, but that’s about it. Any other query is off limits, unless you want to do a full recursive search. The only option here is to build indices specifically for querying. You can do this on a scheduled basis if staleness is of no concern or you can use git hooks to update indices as soon as a commit happens. Isn't the point of a database the ability to query? Why else would you want "Git as a NoSql database?" If this is really what you're after, maybe you should be using Fossil, for which the repo is an sqlite database that you can query like any sqlite database.

js2 6y ago
The point of a database is to hold data, and there are many key/value databases for which you can only query by key unless you add a secondary index. (e.g. Bigtable, DynamoDB, Riak, Redis, etc).
MuffinFlavored 6y ago
> Isn't the point of a database the ability to query? I was thinking about this in terms of the new CloudFlare Durable Objects open beta: https://blog.cloudflare.com/durable-objects-open-beta/ https://blog.cloudflare.com/durable-objects-open-beta/ Storing data is really only half the battle. How could you reimplement the poorest of a poor man's SQL on top of a key/value store? The idea I came up with is: whatever fields you want to query by need an index as a separate key. Maybe this would allow JOINs? Probably not?

no_wizard 6y ago

Isn't the problem attempting to be solved here solved by event sourcing[0]? Which lets you 're-create' your changes and is - ultimately- suppose to allow you effectively log each change in such a way its re-playable backwards and forwards [0]: https://martinfowler.com/eaaDev/EventSourcing.html https://martinfowler.com/eaaDev/EventSourcing.html

hcarvalhoalves 6y ago
Kind of, but there’s more to it. “Event sourcing” is the pattern being implemented by git’s immutable object database, but there’s also all the hashing and reference management.

patrickdevivo 6y ago

I've been very curious to explore this type of use case with askgit (https://github.com/augmentable-dev/askgit https://github.com/augmentable-dev/askgit) which was designed for running simple "slice and dice" queries and aggregations on git history (and change stats) for basic analytical purposes. I've been curious about how this could be applied to a small text+git based "db". Say, for a regular json or CSV dumps. This also reminds me of Dolt: https://github.com/dolthub/dolt https://github.com/dolthub/dolt which I believe has been on HN a couple times

rrdharan 6y ago

For completeness here’s Git as a SQL Database (2018): https://opensource.com/article/18/11/gitbase https://opensource.com/article/18/11/gitbase

blazeeboy 6y ago

You don't even need to write your key/value in a file. Git accept committing empty commit. and allow any message to be committed with it so content of your database can be the git commit message. I wrote about that concept before and tried to implement it but never went to the full potential (too many side projects) https://www.emadelsaid.com/git-as-messages-queue/ https://www.emadelsaid.com/git-as-messages-queue/ https://github.com/emad-elsaid/gitmq https://github.com/emad-elsaid/gitmq

abathologist 6y ago

Related https://irmin.org/ https://irmin.org/ (and in development since 2013) > A distributed database built on the same principles as Git (and which can use git as a backend)

senorsmile 6y ago
Is Irmin more like a database engine? I.e. you have to write code in OCaml in order to use it???

zimbatm 6y ago

Gerrit is using Git as a NoSql database. All the users, projects, reviews, ... are stored in Git repositories[1]. One of the main advantage of this approach is that replication becomes trivial, it's just a matter of running `git fetch` on all the repositories. The main disadvantage is that NodeDB only has one implementation (Java), and no CLI to play with it. [1]: https://gerrit-review.googlesource.com/Documentation/note-db.html https://gerrit-review.googlesource.com/Documentation/note-db...