5 ms·

Yet another file in the .git directory. The work is impressive and certainly helpful, but I can already hear Fossil proponents say "just use SQLite", which is g

by rakoo 8y ago

Yet another file in the .git directory. The work is impressive and certainly helpful, but I can already hear Fossil proponents say "just use SQLite", which is getting more and more true.

rtcoolaid 8y ago
Is that what you got from the article? Or you're the guy that knows the names of all he latest tools but understands the underlying concept of none and comments this making HN a waste of time to actually learn anything.
- naturalgradient 8y ago
  I think you should refrain from making such comments and follow the best-intention assumption on HN.
Aissen 8y ago
I had to bite… The sqlite repository has only 20795 commits since May 2000 at the time of this writing: https://www.sqlite.org/src/timeline?udc=1&ss=m&n=100000&y=ci https://www.sqlite.org/src/timeline?udc=1&ss=m&n=100000&y=ci This is the amount of commits that goes into Linux every ~5months. Has anyone done any meaningful performance comparison between fossil and git?
- adrianN 8y ago
  The number of commits is not a very meaningful metric.
  - masklinn 8y ago
    It's very much a meaningful metric when the entire point of TFA is caching the commits graph. This is only an issue when you actually have lots of commits, and even more so a very branchy graph.
    - adrianN 8y ago
      It's not a meaningful metric when discussing whether to use sqlite as storage for git instead of files in the .git directory.
      - jasode 8y ago
        >It's not a meaningful metric when discussing whether to use sqlite as storage for git instead of files in the .git There are 2 different conversations happening. You seemed to be responding to ggp (rakoo) "sqlite db vs files". However masklinn was responding to gp (Aissen) question of "Fossil vs Git" performance and a charitable reading would be comparing the Fossil algorithms (also affected by combination with underlying SQLite db engine algorithm) for commit searches, graph traversal, etc. In that case, the high number of commits and total repository size to stress test Fossil/Git would be very relevant. An example post about Fossil scaling: https://www.fossil-scm.org/xfer/technote/be8f2f3447ef2ea3344f8058b6733aa08c08336f https://www.fossil-scm.org/xfer/technote/be8f2f3447ef2ea3344...
- reificator 8y ago
  How does the number of commits in the SQLite repository make it unsuitable as alternate store for git metadata?
  - Aissen 8y ago
    @jasode's reply in this thread made a good summary of the two parallel discussions/quid pro quo: - on filesystem vs sqlite (put git files in sqlite): there's a good benchmark on https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html claiming sqlite is up to 35% faster than fs. I'd like to see the same benchmark with git's file pattern; also, it's a known issue with git that it was written for linux first, hence optimized against Linux (relatively) good fs performance (vs Windows and Mac at the time). Same with most OSS build systems that (over)use process forking, which is also very optimized in Linux. - on Fossil vs git (why bother putting git files in sqlite and not directly jump to fossil?): that was my comment, and it relates to subject of this article (the commit-graph). I'm wondering if Fossil has seen the optimization that git has, with regards to number of commits, considering that sqlite is the only high-profile project that uses it. Maybe performance is supposed to be taken care of by the sqlite database itself ? Edit: see https://www.fossil-scm.org/xfer/doc/trunk/www/stats.wiki https://www.fossil-scm.org/xfer/doc/trunk/www/stats.wiki for a fossil-provided performance analysis, that does a comparison with CVS (!).
    - deleted 8y ago
      [deleted]
superflyguy 8y ago
Use sqllite instead of git? Or git should use sqllite? If the latter then one problem is you'd need to keep your own fork forever as they don't accept patches. I'm not sure if that's a price worth paying to reduce the number of files git uses. Why is this a problem for you, anyway?
- girvo 8y ago
  They mean to use SQLite instead of the .git folder and file structures, I believe.
- rakoo 8y ago
  It's not a problem for me, because all I see is the different commands that _use_ the underlying infrastructure. It's more about the design that was chosen: if you want to speed up things with git you have to implement specific logic in application code that will write a file and will need to update it periodically to keep it up-to-date, instead of using a querying engine made specifically for this purpose. I don't see git ever changing its file format, but I do see another tool that imports everything from git and gives you a read-only sqlite db where you can do whatever you want, including displaying a graph quickly as the post advertises.
  - kyberias 8y ago
    I think you don't fully understand what you are proposing. The storage engine (file system or SQLite) has little to do with git graph algorithm performance. SQLite doesn't magically "display a graph quickly".
    - rakoo 8y ago
      What I'm saying is, the iteration step from "existing dataset" (what we have today) to "faster data traversal" (what the article proposes) is a custom file with a custom format on one side, and the appropriate query/index on the other side; one is definitely more understandable, portable and maintainable than the other.
rhencke 8y ago
I love SQLite, and Fossil is very cool, but I don't see the fundamental difference between Git adding another file in the .git directory, and Fossil adding another table or index in the SQLite database.
- rakoo 8y ago
  This is about having a single, unified interface for all operations. This is all explained in great details by SQLite itself at https://sqlite.org/appfileformat.html https://sqlite.org/appfileformat.html
  - grive 8y ago
    As usual with the SQLite / Fossil developer argumentation, it just seems very biased and far-fetched. Just one example: > Pile-of-Files Formats. Sometimes the application state is stored as a hierarchy of files. Git is a prime example of this, though the phenomenon occurs frequently in one-off and bespoke applications. A pile-of-files format essentially uses the filesystem as a key/value database, storing small chunks of information into separate files. This gives the advantage of making the content more accessible to common utility programs such as text editors or "awk" or "grep". But even if many of the files in a pile-of-files format are easily readable, there are usually some files that have their own custom format (example: Git "Packfiles") and are hence "opaque blobs" that are not readable or writable without specialized tools. It is also much less convenient to move a pile-of-files from one place or machine to another, than it is to move a single file. And it is hard to make a pile-of-files document into an email attachment, for example. Finally, a pile-of-files format breaks the "document metaphor": there is no one file that a user can point to that is "the document". More precisely: > But even if many of the files in a pile-of-files format are easily readable, there are usually some files that have their own custom format (example: Git "Packfiles") and are hence "opaque blobs" that are not readable or writable without specialized tools. What is advocated here is to transform the pile-of-files in a single SQLite database accessed through SQL queries. So instead of having only a few binary blob, transform everything in a binary blob and force the use of one specialized tool for everything. > It is also much less convenient to move a pile-of-files from one place or machine to another, than it is to move a single file. This is not true. > And it is hard to make a pile-of-files document into an email attachment, for example. I would not trust someone that had just sent his git repo over email. > Finally, a pile-of-files format breaks the "document metaphor": there is no one file that a user can point to that is "the document". A VCS will track source files. Maybe their argument is true for other applications, but for a VCS this is plain useless. Indeed having only an SQL connector accessing a database is a unified interface to the file. But unifying this to the user means that you have to move the complexity further down, as explained: > But an SQLite database is not limited to a simple key/value structure like a pile-of-files database. An SQLite database can have dozens or hundreds or thousands of different tables, with dozens or hundreds or thousands of fields per table, each with different datatypes and constraints and particular meanings, all cross-referencing each other, appropriately and automatically indexed for rapid retrieval, and all stored efficiently and compactly in a single disk file. And all of this structure is succinctly documented for humans by the SQL schema. Yeah, and I don't want to have this complexity managed by a single "entity", I want to have several different tools available to do whichever kind of work I need to do. If I'm working on graphs and need to store them, I would prefer having the ability to read my file directly in my other tools for graph analysis / debugging without having to take the intermediate step of connecting to the SQL database, or redefining a way to work with the SQL paradigm to adapt my file format to the "dozens or hunders or thousands of different tables, fields per table, each with different datatypes". This point is even more salient regarding grep / awk. The author obviously prefer using the query language of his choice and disregards the variety of tools to work on text, but there are many, many tools available to do all kind of work on it, and believing that > An SQLite database file is not an opaque blob. It is true that command-line tools such as text editors or "grep" or "awk" are not useful on an SQLite database, but the SQL query language is a much more powerful and convenient way for examining the content, so the inability to use "grep" and "awk" and the like is not seen as a loss. Is just nonsense. Passing on the file edition conveniently put under the rug, querying the text is usually only the beginning, usually someone wants to parse the output and act upon it, maybe even put back some modified version (sed), and so on. The author just seems close-minded and living in his own world, unable to imagine that other people might want to work differently. This reminds me a lot of his rant against git and for fossil, with the exact same bad faith arguments and lack of knowledge about other ways to do things.

by rakoo 8y ago

Yet another file in the .git directory. The work is impressive and certainly helpful, but I can already hear Fossil proponents say "just use SQLite", which is getting more and more true.

rtcoolaid 8y ago

Is that what you got from the article? Or you're the guy that knows the names of all he latest tools but understands the underlying concept of none and comments this making HN a waste of time to actually learn anything.

naturalgradient 8y ago
I think you should refrain from making such comments and follow the best-intention assumption on HN.

Aissen 8y ago

I had to bite… The sqlite repository has only 20795 commits since May 2000 at the time of this writing: https://www.sqlite.org/src/timeline?udc=1&ss=m&n=100000&y=ci https://www.sqlite.org/src/timeline?udc=1&ss=m&n=100000&y=ci This is the amount of commits that goes into Linux every ~5months. Has anyone done any meaningful performance comparison between fossil and git?

adrianN 8y ago
The number of commits is not a very meaningful metric.
- masklinn 8y ago
  It's very much a meaningful metric when the entire point of TFA is caching the commits graph. This is only an issue when you actually have lots of commits, and even more so a very branchy graph.
  - adrianN 8y ago
    It's not a meaningful metric when discussing whether to use sqlite as storage for git instead of files in the .git directory.
    - jasode 8y ago
      >It's not a meaningful metric when discussing whether to use sqlite as storage for git instead of files in the .git There are 2 different conversations happening. You seemed to be responding to ggp (rakoo) "sqlite db vs files". However masklinn was responding to gp (Aissen) question of "Fossil vs Git" performance and a charitable reading would be comparing the Fossil algorithms (also affected by combination with underlying SQLite db engine algorithm) for commit searches, graph traversal, etc. In that case, the high number of commits and total repository size to stress test Fossil/Git would be very relevant. An example post about Fossil scaling: https://www.fossil-scm.org/xfer/technote/be8f2f3447ef2ea3344f8058b6733aa08c08336f https://www.fossil-scm.org/xfer/technote/be8f2f3447ef2ea3344...
reificator 8y ago
How does the number of commits in the SQLite repository make it unsuitable as alternate store for git metadata?
- Aissen 8y ago
  @jasode's reply in this thread made a good summary of the two parallel discussions/quid pro quo: - on filesystem vs sqlite (put git files in sqlite): there's a good benchmark on https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html claiming sqlite is up to 35% faster than fs. I'd like to see the same benchmark with git's file pattern; also, it's a known issue with git that it was written for linux first, hence optimized against Linux (relatively) good fs performance (vs Windows and Mac at the time). Same with most OSS build systems that (over)use process forking, which is also very optimized in Linux. - on Fossil vs git (why bother putting git files in sqlite and not directly jump to fossil?): that was my comment, and it relates to subject of this article (the commit-graph). I'm wondering if Fossil has seen the optimization that git has, with regards to number of commits, considering that sqlite is the only high-profile project that uses it. Maybe performance is supposed to be taken care of by the sqlite database itself ? Edit: see https://www.fossil-scm.org/xfer/doc/trunk/www/stats.wiki https://www.fossil-scm.org/xfer/doc/trunk/www/stats.wiki for a fossil-provided performance analysis, that does a comparison with CVS (!).
  - deleted 8y ago
    [deleted]

superflyguy 8y ago

Use sqllite instead of git? Or git should use sqllite? If the latter then one problem is you'd need to keep your own fork forever as they don't accept patches. I'm not sure if that's a price worth paying to reduce the number of files git uses. Why is this a problem for you, anyway?

girvo 8y ago
They mean to use SQLite instead of the .git folder and file structures, I believe.
rakoo 8y ago
It's not a problem for me, because all I see is the different commands that _use_ the underlying infrastructure. It's more about the design that was chosen: if you want to speed up things with git you have to implement specific logic in application code that will write a file and will need to update it periodically to keep it up-to-date, instead of using a querying engine made specifically for this purpose. I don't see git ever changing its file format, but I do see another tool that imports everything from git and gives you a read-only sqlite db where you can do whatever you want, including displaying a graph quickly as the post advertises.
- kyberias 8y ago
  I think you don't fully understand what you are proposing. The storage engine (file system or SQLite) has little to do with git graph algorithm performance. SQLite doesn't magically "display a graph quickly".
  - rakoo 8y ago
    What I'm saying is, the iteration step from "existing dataset" (what we have today) to "faster data traversal" (what the article proposes) is a custom file with a custom format on one side, and the appropriate query/index on the other side; one is definitely more understandable, portable and maintainable than the other.

rhencke 8y ago

I love SQLite, and Fossil is very cool, but I don't see the fundamental difference between Git adding another file in the .git directory, and Fossil adding another table or index in the SQLite database.

rakoo 8y ago
This is about having a single, unified interface for all operations. This is all explained in great details by SQLite itself at https://sqlite.org/appfileformat.html https://sqlite.org/appfileformat.html
- grive 8y ago
  As usual with the SQLite / Fossil developer argumentation, it just seems very biased and far-fetched. Just one example: > Pile-of-Files Formats. Sometimes the application state is stored as a hierarchy of files. Git is a prime example of this, though the phenomenon occurs frequently in one-off and bespoke applications. A pile-of-files format essentially uses the filesystem as a key/value database, storing small chunks of information into separate files. This gives the advantage of making the content more accessible to common utility programs such as text editors or "awk" or "grep". But even if many of the files in a pile-of-files format are easily readable, there are usually some files that have their own custom format (example: Git "Packfiles") and are hence "opaque blobs" that are not readable or writable without specialized tools. It is also much less convenient to move a pile-of-files from one place or machine to another, than it is to move a single file. And it is hard to make a pile-of-files document into an email attachment, for example. Finally, a pile-of-files format breaks the "document metaphor": there is no one file that a user can point to that is "the document". More precisely: > But even if many of the files in a pile-of-files format are easily readable, there are usually some files that have their own custom format (example: Git "Packfiles") and are hence "opaque blobs" that are not readable or writable without specialized tools. What is advocated here is to transform the pile-of-files in a single SQLite database accessed through SQL queries. So instead of having only a few binary blob, transform everything in a binary blob and force the use of one specialized tool for everything. > It is also much less convenient to move a pile-of-files from one place or machine to another, than it is to move a single file. This is not true. > And it is hard to make a pile-of-files document into an email attachment, for example. I would not trust someone that had just sent his git repo over email. > Finally, a pile-of-files format breaks the "document metaphor": there is no one file that a user can point to that is "the document". A VCS will track source files. Maybe their argument is true for other applications, but for a VCS this is plain useless. Indeed having only an SQL connector accessing a database is a unified interface to the file. But unifying this to the user means that you have to move the complexity further down, as explained: > But an SQLite database is not limited to a simple key/value structure like a pile-of-files database. An SQLite database can have dozens or hundreds or thousands of different tables, with dozens or hundreds or thousands of fields per table, each with different datatypes and constraints and particular meanings, all cross-referencing each other, appropriately and automatically indexed for rapid retrieval, and all stored efficiently and compactly in a single disk file. And all of this structure is succinctly documented for humans by the SQL schema. Yeah, and I don't want to have this complexity managed by a single "entity", I want to have several different tools available to do whichever kind of work I need to do. If I'm working on graphs and need to store them, I would prefer having the ability to read my file directly in my other tools for graph analysis / debugging without having to take the intermediate step of connecting to the SQL database, or redefining a way to work with the SQL paradigm to adapt my file format to the "dozens or hunders or thousands of different tables, fields per table, each with different datatypes". This point is even more salient regarding grep / awk. The author obviously prefer using the query language of his choice and disregards the variety of tools to work on text, but there are many, many tools available to do all kind of work on it, and believing that > An SQLite database file is not an opaque blob. It is true that command-line tools such as text editors or "grep" or "awk" are not useful on an SQLite database, but the SQL query language is a much more powerful and convenient way for examining the content, so the inability to use "grep" and "awk" and the like is not seen as a loss. Is just nonsense. Passing on the file edition conveniently put under the rug, querying the text is usually only the beginning, usually someone wants to parse the output and act upon it, maybe even put back some modified version (sed), and so on. The author just seems close-minded and living in his own world, unable to imagine that other people might want to work differently. This reminds me a lot of his rant against git and for fossil, with the exact same bad faith arguments and lack of knowledge about other ways to do things.