4 ms·

Since it uses SQLite under the hood, Fossil as a version control system might be able to scale much more gracefully than git for huge monorepos. I think about t

Since it uses SQLite under the hood, Fossil as a version control system might be able to scale much more gracefully than git for huge monorepos. I think about this often but haven't found a way to effectively use this knowledge yet, since I haven't worked in a giant monorepo shop yet, nor have I found any third parties talking about their experience with this.

deleted 3y ago
[deleted]
sesm 3y ago
For a large monorepo network filesystem approach is inevitable. Every local storage approach has limits and breaks down eventually.
zie 3y ago
OpenBSD tried to import their CVS tree a while ago, it didn't go well. Now some OpenBSD devs are essentially re-implementing Git as their hopeful long-term solution to getting off CVS at some point.
- gecko 3y ago
  I love the concept of Fossil being in SQLite, but there's a reason that Mercurial invented revlogs and Git tries to keep related objects close to each other in packfiles. Sometimes, you really do need a dedicated file format optimized for specific use cases. I'm completely unsurprised OpenBSD wasn't able to pull this off. (Kiln split the difference by storing metadata in SQL Server, but keeping all the actual source data in their native formats. This works great, but is only really viable if you can guarantee things never get out of sync, which is basically impossible for random local Git repos.)
- tiffanyh 3y ago
  Any mailing list threads in this topic. Was able to find any and would be curious to read more.
  - camgunz 3y ago
    I think it's called Game of Trees (GoT): https://gameoftrees.org/ https://gameoftrees.org/
  - tsegratis 3y ago
    I found this for NetBSD from 2011 https://2011.eurobsdcon.org/papers/sonnenberger/fossilizing.pdf https://2011.eurobsdcon.org/papers/sonnenberger/fossilizing.... TLDR imported cvs2fossil, then fossil to git and mercurial. fossil was less optimized at scale. No dvcs was perfect. At scale things don't always import easily "The number of artificial commits and conversion glitches could be minimized by cleaning up various issues. This was made easy by exploiting the database and writing small Python scripts calling “rcs” and related programs. Various tests show that the Fossil repository and CVS give exactly the same output for individual revisions. Differences when comparing working copies of specific branches are accounted for. The performance of Fossil is competitive. Some areas like “fossil pull” need further work, but the majority of the work can shift to improving the user interface."
  - stsp 3y ago
    Many years ago I was using fossil for OpenBSD development to manage my patches. Around that time I tried to import the entire OpenBSD src repository into fossil, by importing the CVS-to-git conversion of src, as published on Github. I was following the official git->fossil migration guide. I left this running for a week (or two?) at which point the fossil git loader was loading OpenBSD commits from somewhere around the 2000s. At that point I stopped the process. Performance might be better today, I don't know. And perhaps post-conversion run-time performance is much better, but I never got that far. Anyone can try to reproduce these results by running the same conversion today. I don't think I ever talked about my attempts with fossil to anyone at the time. But I recall the topic coming up somewhere when the Game of Trees project became public, and someone suggested I should be using fossil instead. I am now using Game of Trees for all my OpenBSD development work and I am happy with it.
bachmeier 3y ago
They tried using Fossil for netbsd (and may still use it for some things, I'm not sure). Fossil was not designed for large repositories: https://fossil-scm.org/forum/forumpost/5ad122663cbae98f?t=h https://fossil-scm.org/forum/forumpost/5ad122663cbae98f?t=h It could be optimized for that case, but since there doesn't seem to be a lot of interest, I don't know that anyone is going to do the work.
chokolad 3y ago
Microsoft uses git for Windows repository pretty successfully. It started with custom VFS implementation and fork of git but by now everthing is upstreamed and VFS is not used anymore. https://blog.gitbutler.com/git-tips-3-really-large-repositories/ https://blog.gitbutler.com/git-tips-3-really-large-repositor...
- 0cf8612b2e1e 3y ago
  Doesn’t that speak to parent’s point? Git original implementation did not scale to huge sizes. Needed specific development work to change assumptions. Presumably similar design improvements could change fossil.
zitterbewegung 3y ago
Python and obviously the Linux kernel are large git monorepos. SQLite uses Fossil because they want to not only eat their own dog food but they want a complete app that can do most of the work that GitHub can do.

deleted 3y ago

[deleted]

sesm 3y ago

For a large monorepo network filesystem approach is inevitable. Every local storage approach has limits and breaks down eventually.

zie 3y ago

OpenBSD tried to import their CVS tree a while ago, it didn't go well. Now some OpenBSD devs are essentially re-implementing Git as their hopeful long-term solution to getting off CVS at some point.

gecko 3y ago
I love the concept of Fossil being in SQLite, but there's a reason that Mercurial invented revlogs and Git tries to keep related objects close to each other in packfiles. Sometimes, you really do need a dedicated file format optimized for specific use cases. I'm completely unsurprised OpenBSD wasn't able to pull this off. (Kiln split the difference by storing metadata in SQL Server, but keeping all the actual source data in their native formats. This works great, but is only really viable if you can guarantee things never get out of sync, which is basically impossible for random local Git repos.)
tiffanyh 3y ago
Any mailing list threads in this topic. Was able to find any and would be curious to read more.
- camgunz 3y ago
  I think it's called Game of Trees (GoT): https://gameoftrees.org/ https://gameoftrees.org/
- tsegratis 3y ago
  I found this for NetBSD from 2011 https://2011.eurobsdcon.org/papers/sonnenberger/fossilizing.pdf https://2011.eurobsdcon.org/papers/sonnenberger/fossilizing.... TLDR imported cvs2fossil, then fossil to git and mercurial. fossil was less optimized at scale. No dvcs was perfect. At scale things don't always import easily "The number of artificial commits and conversion glitches could be minimized by cleaning up various issues. This was made easy by exploiting the database and writing small Python scripts calling “rcs” and related programs. Various tests show that the Fossil repository and CVS give exactly the same output for individual revisions. Differences when comparing working copies of specific branches are accounted for. The performance of Fossil is competitive. Some areas like “fossil pull” need further work, but the majority of the work can shift to improving the user interface."
- stsp 3y ago
  Many years ago I was using fossil for OpenBSD development to manage my patches. Around that time I tried to import the entire OpenBSD src repository into fossil, by importing the CVS-to-git conversion of src, as published on Github. I was following the official git->fossil migration guide. I left this running for a week (or two?) at which point the fossil git loader was loading OpenBSD commits from somewhere around the 2000s. At that point I stopped the process. Performance might be better today, I don't know. And perhaps post-conversion run-time performance is much better, but I never got that far. Anyone can try to reproduce these results by running the same conversion today. I don't think I ever talked about my attempts with fossil to anyone at the time. But I recall the topic coming up somewhere when the Game of Trees project became public, and someone suggested I should be using fossil instead. I am now using Game of Trees for all my OpenBSD development work and I am happy with it.

bachmeier 3y ago

They tried using Fossil for netbsd (and may still use it for some things, I'm not sure). Fossil was not designed for large repositories: https://fossil-scm.org/forum/forumpost/5ad122663cbae98f?t=h https://fossil-scm.org/forum/forumpost/5ad122663cbae98f?t=h It could be optimized for that case, but since there doesn't seem to be a lot of interest, I don't know that anyone is going to do the work.

chokolad 3y ago

Microsoft uses git for Windows repository pretty successfully. It started with custom VFS implementation and fork of git but by now everthing is upstreamed and VFS is not used anymore. https://blog.gitbutler.com/git-tips-3-really-large-repositories/ https://blog.gitbutler.com/git-tips-3-really-large-repositor...

0cf8612b2e1e 3y ago
Doesn’t that speak to parent’s point? Git original implementation did not scale to huge sizes. Needed specific development work to change assumptions. Presumably similar design improvements could change fossil.

zitterbewegung 3y ago

Python and obviously the Linux kernel are large git monorepos. SQLite uses Fossil because they want to not only eat their own dog food but they want a complete app that can do most of the work that GitHub can do.