5 ms·
The Fossil Sync Protocol
- hiAndrewQuinn 3y agoSince it uses SQLite under the hood, Fossil as a version control system might be able to scale much more gracefully than git for huge monorepos. I think about this often but haven't found a way to effectively use this knowledge yet, since I haven't worked in a giant monorepo shop yet, nor have I found any third parties talking about their experience with this.
- deleted 3y ago[deleted]
- sesm 3y agoFor a large monorepo network filesystem approach is inevitable. Every local storage approach has limits and breaks down eventually.
- zie 3y agoOpenBSD tried to import their CVS tree a while ago, it didn't go well. Now some OpenBSD devs are essentially re-implementing Git as their hopeful long-term solution to getting off CVS at some point.
- gecko 3y agoI love the concept of Fossil being in SQLite, but there's a reason that Mercurial invented revlogs and Git tries to keep related objects close to each other in packfiles. Sometimes, you really do need a dedicated file format optimized for specific use cases. I'm completely unsurprised OpenBSD wasn't able to pull this off. (Kiln split the difference by storing metadata in SQL Server, but keeping all the actual source data in their native formats. This works great, but is only really viable if you can guarantee things never get out of sync, which is basically impossible for random local Git repos.)
- tiffanyh 3y agoAny mailing list threads in this topic. Was able to find any and would be curious to read more.
- camgunz 3y agoI think it's called Game of Trees (GoT): https://gameoftrees.org/ https://gameoftrees.org/
- tsegratis 3y agoI found this for NetBSD from 2011 https://2011.eurobsdcon.org/papers/sonnenberger/fossilizing.pdf https://2011.eurobsdcon.org/papers/sonnenberger/fossilizing.... TLDR imported cvs2fossil, then fossil to git and mercurial. fossil was less optimized at scale. No dvcs was perfect. At scale things don't always import easily "The number of artificial commits and conversion glitches could be minimized by cleaning up various issues. This was made easy by exploiting the database and writing small Python scripts calling “rcs” and related programs. Various tests show that the Fossil repository and CVS give exactly the same output for individual revisions. Differences when comparing working copies of specific branches are accounted for. The performance of Fossil is competitive. Some areas like “fossil pull” need further work, but the majority of the work can shift to improving the user interface."
- stsp 3y agoMany years ago I was using fossil for OpenBSD development to manage my patches. Around that time I tried to import the entire OpenBSD src repository into fossil, by importing the CVS-to-git conversion of src, as published on Github. I was following the official git->fossil migration guide. I left this running for a week (or two?) at which point the fossil git loader was loading OpenBSD commits from somewhere around the 2000s. At that point I stopped the process. Performance might be better today, I don't know. And perhaps post-conversion run-time performance is much better, but I never got that far. Anyone can try to reproduce these results by running the same conversion today. I don't think I ever talked about my attempts with fossil to anyone at the time. But I recall the topic coming up somewhere when the Game of Trees project became public, and someone suggested I should be using fossil instead. I am now using Game of Trees for all my OpenBSD development work and I am happy with it.
- bachmeier 3y agoThey tried using Fossil for netbsd (and may still use it for some things, I'm not sure). Fossil was not designed for large repositories: https://fossil-scm.org/forum/forumpost/5ad122663cbae98f?t=h https://fossil-scm.org/forum/forumpost/5ad122663cbae98f?t=h It could be optimized for that case, but since there doesn't seem to be a lot of interest, I don't know that anyone is going to do the work.
- chokolad 3y agoMicrosoft uses git for Windows repository pretty successfully. It started with custom VFS implementation and fork of git but by now everthing is upstreamed and VFS is not used anymore. https://blog.gitbutler.com/git-tips-3-really-large-repositories/ https://blog.gitbutler.com/git-tips-3-really-large-repositor...
- 0cf8612b2e1e 3y agoDoesn’t that speak to parent’s point? Git original implementation did not scale to huge sizes. Needed specific development work to change assumptions. Presumably similar design improvements could change fossil.
- zitterbewegung 3y agoPython and obviously the Linux kernel are large git monorepos. SQLite uses Fossil because they want to not only eat their own dog food but they want a complete app that can do most of the work that GitHub can do.
- rcarmo 3y agoFossil is really nice. I still have some projects (from the pre-Dropbox days) that use it--and that are now archived on it, with 100% fidelity. Having a single file repository is just amazing.
- web3-is-a-scam 3y agoI use fossil in the post-Dropbox era by keeping my singular repository file inside Dropbox and my working copies outside of Dropbox. Works great for me. Works infinitely better than a git or mercurial repository where the .git/.hg folders get corrupted when there are file conflicts.
- wakawaka28 3y agoJust set up a git server somewhere. It's not hard. Way better than using Dropbox too.
- throwaway918274 3y agoWhy would I do that when Fossil and Dropbox already works for me :) Sidenote: I use git at work and I absolutely despise it.
- wakawaka28 3y agoBecause syncing a monolithic binary file is trickier and more intensive than syncing a bunch of little files, once your repo is big enough. If your git repo was corrupted, you might recover some data. If you corrupt a binary database, good luck with that. Dropbox is also harvesting your data for various purposes. I hope your code isn't valuable, because it could be leaked or stolen by employees of Dropbox. There's also something to be said about using what other people use instead of bizarre niche tools. Disclaimer: I have used Fossil and it's a decent system for small applications where everyone is on board. But I would not recommend it due to the reasons above.
- 3y ago
- sesm 3y agoDear Jetbrains, if you read this, please reanimate your Fossil plugin. I tried to do it myself, but the learning curve for IJ plugin development is too steep.
- mdaniel 3y agoI readily admit I am not familiar enough with fossil to know about the impedance mismatch, but I'll point out that https://github.com/JetBrains/intellij-plugins/tree/idea/241.13688.18/PerforceIntegration https://github.com/JetBrains/intellij-plugins/tree/idea/241.... https://github.com/JetBrains/intellij-community/tree/idea/241.13688.18/plugins/git4idea https://github.com/JetBrains/intellij-community/tree/idea/24... https://github.com/JetBrains/intellij-community/tree/idea/241.13688.18/plugins/hg4idea https://github.com/JetBrains/intellij-community/tree/idea/24... https://github.com/JetBrains/intellij-community/tree/idea/241.13688.18/plugins/svn4idea https://github.com/JetBrains/intellij-community/tree/idea/24... may a long way toward finding how they think about those operations and you may already know about it but https://github.com/irengrig/fossil4idea https://github.com/irengrig/fossil4idea (Apache 2) is the one listed in https://plugins.jetbrains.com/plugin/7479-fossil-integration https://plugins.jetbrains.com/plugin/7479-fossil-integration although as you pointed out, it's very stale
- sesm 3y agoThanks for the code pointers! Yes, fossil4idea was the plugin I tried to update. I might give it another shot at some point.
- sebstefan 3y ago> Each artifact is identified by a cryptographic hash of its content, expressed as a lower-case hexadecimal string [...] It is assumed that the hash names of artifacts are unique - that every artifact has a different hash Isn't that kinda scary with the birthday problem paradox for people who use Fossil to manage a large number of artifacts
- MathMonkeyMan 3y agoI don't think it will be a problem: <https://fossil-scm.org/home/doc/trunk/www/hashpolicy.wiki https://fossil-scm.org/home/doc/trunk/www/hashpolicy.wiki>
- eesmith 3y agoAs a rough approximation, you need something like sqrt(N) entities before the birthday paradox becomes an issue. The Wikipedia article at https://en.wikipedia.org/wiki/Birthday_problem#Square_approximation https://en.wikipedia.org/wiki/Birthday_problem#Square_approx... gives the approximation as: n = sqrt(2 * m * p(n)) where p(n) is the probability of having a collision. Let p(n) be 1e-9 (a 1 in a billion chance). With fossil's old default of SHA1's 160 bits you'll need roughly sqrt(2 * 2^80 * 1e-9) = 49 million artifacts before the likely duplicate. A 1 in a billion chance might still be scary (if everyone on Earth has their own 50 million repo, each with unique artifacts, then there will be duplicates!), but SHA1 use in fossil is mostly a historical issue. With fossil's new default (since v2.10 in 2019) of SHA3-256, you'll need something like 1.5e34 artifacts before there is a 1-to-a-billion chance of a duplicate. As MathMonkeyMan's link points out, you can configure your repository to "shun-sha1" and only allow sha3.
- cloudhead 3y ago> The "bag of artifacts" data model used by Fossil is apparently an implementation of a particular Conflict-Free Replicated Datatype (CRDT) called a "G-Set" or "Grow-only Set". Isn’t this just by virtue of content addressing objects? In that sense Git’s odb is also a G-set.
- topaz0 3y agoIf you look at the definition of CRDTS, it seems trivial that all distributed scms must satisfy CRDT properties. Some observations are trivial once you think about it, but worth saying to connect to different audiences with different interests or backgrounds. I wouldn't be surprised if they added this because they got the question many times. The rest of the paragraph makes clear that they did not develop fossil in the context of CRDTs, and nothing in the rest of the document depends on this observation.
- account-5 3y agoThis is beyond me to understand properly, so stupid question time: Is this, or could this be modified to be, able to sync an sqlite database (or really the data it contains) between devices? I'm aware of other ways to do this but if there was an internal sqlite way to do this...
- dchest 3y agoNo, this protocol doesn't deal with SQLite. In fact, SQLite in Fossil is just an implementation detail, the design doesn't depend on it.
- XorNot 3y agoThe problem with single file formats is they don't play nicely with file sync tools in what I'd call "common" scenarios. Due to the way it's laid out, a git bare repository will work quite well with a per-file syncing system because references are stored as files pointing at the hashed object store. So the worst you can do even without central coordination is have a reference conflict. I'm sure bad things can happen, but for the purposes of "keeping repos synced via syncthing" it works well enough (note: this is different to trying to keep a working repository synced, which I wouldn't recommend).