5 ms·

If you had 10's thousands or millions of files, and several terabytes, is syncthing a good option for this? I haven't seen a ton of great options unfortunately

If you had 10's thousands or millions of files, and several terabytes, is syncthing a good option for this? I haven't seen a ton of great options unfortunately :/

em-bee 3y ago
that is a hard problem. it depends on how frequently the data changes, because syncthing or any other backup solution may be to slow to keep up. one of my clients eventually switched from rsync to gluster. instead of making backups, gluster writes multiple storage servers in parallel when files are changed, thus ensuring that there are multiple copies right away. it's not the perfect solution though because when one server is offline for a while and needs to catch up that too is very slow. the only solution there i find is to have enough redundant servers that you can afford to have a server go offline and not worry about getting it back online quickly.
vorpalhex 3y ago
I've been successful with syncthing up to 2tb (never tried higher) with the caveat that I only have it function between two hosts and only one host is the writer.
- nine_k 3y ago
  I have a 2 TB setup with multiple writers; works fine.
karlicoss 3y ago
I sync about two hundred thousands of files without any problems, especially if your aren't changing them all the time. The only issue I can imagine is initial sync with so many files might take a while, even if total size isn't huge. As for several terabytes, for me it's a bit less than a terabyte in total for all synced folders, but also can't see why it wouldn't scale up. I guess similar, initial hashing might take some time, but otherwise it should handle it well.
icelancer 3y ago
I actually have this exact use case. About 25MM files, ~20 TB total. Lot of directories. Many servers (~10), only a few that have the exact same synching file/folder setup (in other words, some servers have only video, some have only code, some have it all, etc, you get it). It works... pretty well. I won't lie and say it works perfectly with all of our other file utilities and scripts we run, but 95-97% of the time I never think about it. I use it with SyncTrazor on Windows Server deployments as well as Debian servers and yeah, I could be happier, but I highly doubt there's anything out there that can beat SyncThing overall. Just know that with that many files the initial scans and DB building will be brutal, and the periodic syncs and rescans will be taxing. Highly recommend getting cheap 1-2U servers with dual sockets and lots of cores/threads - multiprocessing goes a long way in doing the file hashing. EDIT: I will add that I only use it on servers and not on any client/desktop machines, so I can't vouch for its use there.

If you had 10's thousands or millions of files, and several terabytes, is syncthing a good option for this? I haven't seen a ton of great options unfortunately :/

em-bee 3y ago

that is a hard problem. it depends on how frequently the data changes, because syncthing or any other backup solution may be to slow to keep up. one of my clients eventually switched from rsync to gluster. instead of making backups, gluster writes multiple storage servers in parallel when files are changed, thus ensuring that there are multiple copies right away. it's not the perfect solution though because when one server is offline for a while and needs to catch up that too is very slow. the only solution there i find is to have enough redundant servers that you can afford to have a server go offline and not worry about getting it back online quickly.

vorpalhex 3y ago

I've been successful with syncthing up to 2tb (never tried higher) with the caveat that I only have it function between two hosts and only one host is the writer.

nine_k 3y ago
I have a 2 TB setup with multiple writers; works fine.

karlicoss 3y ago

I sync about two hundred thousands of files without any problems, especially if your aren't changing them all the time. The only issue I can imagine is initial sync with so many files might take a while, even if total size isn't huge. As for several terabytes, for me it's a bit less than a terabyte in total for all synced folders, but also can't see why it wouldn't scale up. I guess similar, initial hashing might take some time, but otherwise it should handle it well.

icelancer 3y ago

I actually have this exact use case. About 25MM files, ~20 TB total. Lot of directories. Many servers (~10), only a few that have the exact same synching file/folder setup (in other words, some servers have only video, some have only code, some have it all, etc, you get it). It works... pretty well. I won't lie and say it works perfectly with all of our other file utilities and scripts we run, but 95-97% of the time I never think about it. I use it with SyncTrazor on Windows Server deployments as well as Debian servers and yeah, I could be happier, but I highly doubt there's anything out there that can beat SyncThing overall. Just know that with that many files the initial scans and DB building will be brutal, and the periodic syncs and rescans will be taxing. Highly recommend getting cheap 1-2U servers with dual sockets and lots of cores/threads - multiprocessing goes a long way in doing the file hashing. EDIT: I will add that I only use it on servers and not on any client/desktop machines, so I can't vouch for its use there.