3 ms·

Can you summarize what they maybe introduced/fixed to quell your old review? > over the last week, syncthing has corrupted my git repository twice. I reported

Can you summarize what they maybe introduced/fixed to quell your old review? > over the last week, syncthing has corrupted my git repository twice. I reported this as an issue, but the issue was immediately closed with an explanation I disagree with. > I no longer trust syncthing until this gets resolved, unfortunately. https://news.ycombinator.com/item?id=23660438 https://news.ycombinator.com/item?id=23660438 — I would absolutely be syncing directories containing git repos, for better or worse

stjohnswarts 5y ago
I trust syncthing for singular/regular files but trusting something as complicated and interconnected as a git repo on it seems like a bad idea, especially if it's possible for two syncthing computers to act on the same git repo... I never really thought about trying it.
JackMcMack 5y ago
I sync a folder containing multiple git repos, never had a corruption happen. @jwr can you link the issue you reported? The only major issue I had was performance on TrueNAS/FreeBSD, Syncthing tends to keep a lot of files open. Increasing maxfiles/maxvnodes fixed it. [0] On a sidenote, what's the point of these limits if your entire system becomes unusable if you reach them? [0] https://jira.ixsystems.com/browse/NAS-104534 https://jira.ixsystems.com/browse/NAS-104534
WesolyKubeczek 5y ago
I had the exact same thing, granted, it was ten years ago already, with dropbox. Something something file versioning, ending up with neighboring files of conflicting revisions, the disease spreading over to all instances. My takeaway was „never keep git repos in a system that does its own distributed versioning”. I just do git push around so my remotes are up to date. Git is sort of dropbox with a manual gearbox, after all (and more, but that’s not the point). I’m really unsure that this is a kind of easily solvable problem for either dropbox or syncthing. Unless your syncing product gets smart enough to understand what a git repository is and how some files have to be checked in in lockstep, that is. Otherwise, expect race conditions.
- banana_giraffe 5y ago
  So, if you really want to use dropbox for syncing git repos, this plugin solves the problem well: https://github.com/anishathalye/git-remote-dropbox https://github.com/anishathalye/git-remote-dropbox For me, it solved all of the corruption problems with using Dropbox for git storage. In the end, I decided it was too heavy weight, and wanted to be able to clone my repo on machines that might not have anything other than "git" installed, but until I reached that point, I was a happy user of it. I'm not aware of a similar tool for syncthing.
dpedu 5y ago
FWIW I sync my ~/code directory, which I work on quite regularly, between a few macs and linux machines using syncthing. I've never faced corruption and I've been using this setup for 2 or 3 years now. The directory holds about 150 git repos / 50GB / 500k files/folders.
chias 5y ago
Doing a quick search of reported issues around git repos, basically the trouble is that a git repo folder does not expect to be sync'ed at a file level, because repos are expected to be in different states on different devices, and the state of a repo is represented in the filesystem via the .git directory. As an example problem-case, say your first device checks out branch A and your second device checks out branch B. Some files conflict, which is likely reasonably handled as any filesync application needs to handle conflicting updates. But any files that are changed in one and not in the other end up getting sync'ed over into the other, including inside your GIT directory, which will very likely lead to an illegal GIT directory state. As a toy example, pretend that the .git internals require that at any given time, `.git/activebranch/` must contain exactly one file whose filename is the name of the active branch, and git maintains this invariant. By syncing changed files, any filesync utility would break this invariant by resulting in `.git/activebranch/` containing two files, due to each side seeing a new file that the other side needs to acquire. Depending on how such a problem might manifest in real life as a result of performing a file-based sync on two instances of a git directory, this could lead to your .git directory entering a state in which git doesn't know how to interpret it. -- This isn't a syncthing problem as such, it's a problem of trying to use a file syncing application on the internal state of an application on two different machines with two different states, and expecting the result to be a valid state.
- throw10920 5y ago
  > it's a problem of trying to use a file syncing application on the internal state of an application on two different machines with two different states, and expecting the result to be a valid state. I think that we can narrow the problem down to the fact that git's internal state is represented in a way that is transparent to syncthing (a folder of files), that it will attempt to sync, but only partially succeed. If git's internal state was a single database file, you wouldn't get corruption, just a syncthing conflict file.
  - chias 5y ago
    That is a much better way of phrasing it, thank you!
jwr 5y ago
Something must have been fixed in syncthing because this problem never reappeared, and after several months I started trusting it again.

stjohnswarts 5y ago

I trust syncthing for singular/regular files but trusting something as complicated and interconnected as a git repo on it seems like a bad idea, especially if it's possible for two syncthing computers to act on the same git repo... I never really thought about trying it.

JackMcMack 5y ago

I sync a folder containing multiple git repos, never had a corruption happen. @jwr can you link the issue you reported? The only major issue I had was performance on TrueNAS/FreeBSD, Syncthing tends to keep a lot of files open. Increasing maxfiles/maxvnodes fixed it. [0] On a sidenote, what's the point of these limits if your entire system becomes unusable if you reach them? [0] https://jira.ixsystems.com/browse/NAS-104534 https://jira.ixsystems.com/browse/NAS-104534

WesolyKubeczek 5y ago

I had the exact same thing, granted, it was ten years ago already, with dropbox. Something something file versioning, ending up with neighboring files of conflicting revisions, the disease spreading over to all instances. My takeaway was „never keep git repos in a system that does its own distributed versioning”. I just do git push around so my remotes are up to date. Git is sort of dropbox with a manual gearbox, after all (and more, but that’s not the point). I’m really unsure that this is a kind of easily solvable problem for either dropbox or syncthing. Unless your syncing product gets smart enough to understand what a git repository is and how some files have to be checked in in lockstep, that is. Otherwise, expect race conditions.

banana_giraffe 5y ago
So, if you really want to use dropbox for syncing git repos, this plugin solves the problem well: https://github.com/anishathalye/git-remote-dropbox https://github.com/anishathalye/git-remote-dropbox For me, it solved all of the corruption problems with using Dropbox for git storage. In the end, I decided it was too heavy weight, and wanted to be able to clone my repo on machines that might not have anything other than "git" installed, but until I reached that point, I was a happy user of it. I'm not aware of a similar tool for syncthing.

dpedu 5y ago

FWIW I sync my ~/code directory, which I work on quite regularly, between a few macs and linux machines using syncthing. I've never faced corruption and I've been using this setup for 2 or 3 years now. The directory holds about 150 git repos / 50GB / 500k files/folders.

chias 5y ago

Doing a quick search of reported issues around git repos, basically the trouble is that a git repo folder does not expect to be sync'ed at a file level, because repos are expected to be in different states on different devices, and the state of a repo is represented in the filesystem via the .git directory. As an example problem-case, say your first device checks out branch A and your second device checks out branch B. Some files conflict, which is likely reasonably handled as any filesync application needs to handle conflicting updates. But any files that are changed in one and not in the other end up getting sync'ed over into the other, including inside your GIT directory, which will very likely lead to an illegal GIT directory state. As a toy example, pretend that the .git internals require that at any given time, `.git/activebranch/` must contain exactly one file whose filename is the name of the active branch, and git maintains this invariant. By syncing changed files, any filesync utility would break this invariant by resulting in `.git/activebranch/` containing two files, due to each side seeing a new file that the other side needs to acquire. Depending on how such a problem might manifest in real life as a result of performing a file-based sync on two instances of a git directory, this could lead to your .git directory entering a state in which git doesn't know how to interpret it. -- This isn't a syncthing problem as such, it's a problem of trying to use a file syncing application on the internal state of an application on two different machines with two different states, and expecting the result to be a valid state.

throw10920 5y ago
> it's a problem of trying to use a file syncing application on the internal state of an application on two different machines with two different states, and expecting the result to be a valid state. I think that we can narrow the problem down to the fact that git's internal state is represented in a way that is transparent to syncthing (a folder of files), that it will attempt to sync, but only partially succeed. If git's internal state was a single database file, you wouldn't get corruption, just a syncthing conflict file.
- chias 5y ago
  That is a much better way of phrasing it, thank you!

jwr 5y ago

Something must have been fixed in syncthing because this problem never reappeared, and after several months I started trusting it again.