4 ms·

I might be wrong but I want to hear more from you if you're a Borg developer. My understanding is that you may be able to have multiple clients uploading chunk

I might be wrong but I want to hear more from you if you're a Borg developer. My understanding is that you may be able to have multiple clients uploading chunks at the same time, but you won't be able to exploit cross-client deduplication if different clients have a similar set of files (OS files or a large code base for instance). Moreover, if your implementation require locks then it would be very hard to extend to cloud services.

dom0 9y ago
Yes, that's right, concurrent addition of the same chunks would generally mean that some work is wasted; so concurrent long running jobs would not synchronize well in this model, and lock-free performs clearly better there. The only operation which inherently has to be guarded by a lock in Borg is inserting the archive pointer into the manifest (root object, see https://borgbackup.readthedocs.io/en/latest/internals/data-structures.html#the-object-graph https://borgbackup.readthedocs.io/en/latest/internals/data-s...). I suppose it would be possible to work around that without locking or to use the usual hacks around EC, put/get/check/get/check?put/get/check?put etc. until it's "probably there". Deleting / pruning archives would still require a full lock due to the same conceptual issues that your two-phase GC avoids. The same goes for "check".