4 ms·

Borg being single-threaded is painful in the era of consumer 12 and 16 core CPUs, and even prosumer 64-core.

by cma 5y ago

sigio 5y ago
Don't notice this much.... disk is usually the bottleneck, and otherwise it will be the network to the remote backup location. Still backups complete in seconds: Repository: ssh://backup/./backups/mungedhostname.borg Archive name: 20220327-2201 Archive fingerprint: 8b710144579c8d531e7c4a0192304323081b14a71445557608d859494bbe84b6 Time (start): Sun, 2022-03-27 22:01:35 Time (end): Sun, 2022-03-27 22:01:52 Duration: 17.28 seconds Number of files: 45678 Original size Compressed size Deduplicated size This archive: 24.22 GB 9.35 GB 47.78 MB All archives: 363.60 GB 140.91 GB 7.81 GB Unique chunks Total chunks Chunk index: 59857 852257
- sliken 5y ago
  Sure, for spinning disks, but deduplication, compression, and encryption can easily be the bottleneck if you have some NVMEs.
  - cma 5y ago
    And you can turn the compression way up to where it is very expensive but gives much better space savings.
silon42 5y ago
I like that it runs in the background without heating my laptop much.
mekster 5y ago
You could spread the backup into several isolated borg repos and run in parallel if CPU is the bottleneck.
ThomasWaldmann 5y ago
I agree that a multithreaded borg could utilise resources better, esp. IF you have a lot of changed / new data to back up. But OTOH, for many users this is primarily the case for the first backup, but not for their daily / hourly backups when most files are unchanged - for that, I guess speed is I/O bound and all your CPU cores won't help you with that. So for N-1 backup runs, it just works good enough already for many users. And for that 1 initial backup, some patience helps. :-) Implementing MT is planned since long, but due to the above, other stuff had higher priority.

Borg being single-threaded is painful in the era of consumer 12 and 16 core CPUs, and even prosumer 64-core.

by cma 5y ago

Borg being single-threaded is painful in the era of consumer 12 and 16 core CPUs, and even prosumer 64-core.

Don't notice this much.... disk is usually the bottleneck, and otherwise it will be the network to the remote backup location. Still backups complete in seconds: Repository: ssh://backup/./backups/mungedhostname.borg Archive name: 20220327-2201 Archive fingerprint: 8b710144579c8d531e7c4a0192304323081b14a71445557608d859494bbe84b6 Time (start): Sun, 2022-03-27 22:01:35 Time (end): Sun, 2022-03-27 22:01:52 Duration: 17.28 seconds Number of files: 45678 Original size Compressed size Deduplicated size This archive: 24.22 GB 9.35 GB 47.78 MB All archives: 363.60 GB 140.91 GB 7.81 GB Unique chunks Total chunks Chunk index: 59857 852257

sliken 5y ago
Sure, for spinning disks, but deduplication, compression, and encryption can easily be the bottleneck if you have some NVMEs.
- cma 5y ago
  And you can turn the compression way up to where it is very expensive but gives much better space savings.

silon42 5y ago

I like that it runs in the background without heating my laptop much.

mekster 5y ago

You could spread the backup into several isolated borg repos and run in parallel if CPU is the bottleneck.

ThomasWaldmann 5y ago

I agree that a multithreaded borg could utilise resources better, esp. IF you have a lot of changed / new data to back up. But OTOH, for many users this is primarily the case for the first backup, but not for their daily / hourly backups when most files are unchanged - for that, I guess speed is I/O bound and all your CPU cores won't help you with that. So for N-1 backup runs, it just works good enough already for many users. And for that 1 initial backup, some patience helps. :-) Implementing MT is planned since long, but due to the above, other stuff had higher priority.