3 ms·
Compressing large files/backups using all available cores at the highest setting is a bliss. Sadly the choke point now is encryption. Hopefully some kind soul w
by alecco 6y ago
Compressing large files/backups using all available cores at the highest setting is a bliss. Sadly the choke point now is encryption. Hopefully some kind soul will some day add this to gpg or equivalent. There are parallelizable cipher modes like CTR and GCM.
https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
- Algent 6y agoIf your need of encryption is related to backups, look into "Borg backup". It can handle both encryption and zstd, also do dedupe natively since it's storing data as blocks.
- formerly_proven 6y agoBorg only utilizes one thread and does not support multithreaded zstd, so the write throughput will likely be (much) lower than ZFS -- typically 60-120 MB/s. Since ZFS can also dedupe, Borg might not even be faster for heavily deduped backups.
- doublepg23 6y agoZFS dedupe is broken and I don’t recommend using it unless you really know what you’re getting into.
- formerly_proven 6y agoBroken in what way?
- tatersolid 6y agoIt uses a ludicrous amount of RAM and a fixed block size and offset boundry, making it very slow and fairly ineffective.