3 ms·

Would using Git for this be performant?

• simonw 5y ago
  That's why I'm using it to build a SQLite database. I've run the script against ~20,000 commits and it took an hour to run... but then on subsequent runs when there are only a few new commits it finishes in a few seconds.
  • CyberShadow 5y ago
    Git can be very performant if you use a good interface to it. Unfortunately, the CLI interfaces are slow, and e.g. libgit2 is a bit awkward to use. However, there is the object reader and writer interface, which allows reading and writing raw Git objects to its database - with it, you can build some nice high-performance tools. For example, here is a program which, given a file's SHA1 and path, finds the last commit to have that file version at that path: https://github.com/cybershadow/misc/blob/master/git-find-file-with-sha1.d https://github.com/cybershadow/misc/blob/master/git-find-fil...
• cerved 5y ago
  It's unlikely to be the most performant since git is first and foremost designed to version control plain text source code for developing software. This is version controlling JSON data. Git stores every version of a file as distinct compressed copies (objects). Periodically, it packs these objects into a pack file. To save space, delta encoding is used in this process. As far as I recall, this is done on a line by line level. In this case, simply dumping this JSON data is going to hamper the delta encoding and the repository will grow substantially in size each time there is a change in data. Then there's the issue of retrieving the data. The basic implementation of reading objects using git checkout and rewriting the objects to disc but there are other ways to read the raw file contents into memory and do your stuff there. It's not going to be the most performant or best way of version controlling data but in many cases it will do a decent job.