4 ms·

The deduping in restic is just on the edge of acceptable for me, making me think I'd have trouble with a lot more data. Basically the one a month "prune" operat

The deduping in restic is just on the edge of acceptable for me, making me think I'd have trouble with a lot more data. Basically the one a month "prune" operation takes about 36h (to B2) . I feel I could be tuning something but also it works and I don't want to touch it.

sandgiant 3y ago
Curious how much you backup, which version of restic you're running and why you think the deduplication is borderline unacceptable. There were several major (orders of magnitudes) improvements made to pruning within the past ~1 year, that's why I'm interested.
- mattbee 3y ago
  A straight upgrade, that I can do :) It's been running for years without one. I was only edgy about it because when it takes 36h it blocks the next daily backup, and I wondered whether that was going to get worse (it hasn't).
  - chibea 3y ago
    The max-unused percentage feature is well worth it to 80/20 the prune process and only prune the data which is easiest to prune away (i.e. not try to remove small files big packs but focus on packs which have lots of garbage). In general, there's an unavoidable trade-off between creating many small packs (harder on metadata throughout the system, inside restic and on the backing store but more efficient to prune) versus creating big packs which are more easy on the metadata but might create big repack cost. I guess a bit more intelligent repacking could avoid some of that cost by packing stuff together that might be more likely to get pruned together.
aborsy 3y ago
I backup around 2TB with Restic, also tried locally with Borg. The size is nearly the same. Sadly, I can’t even test with Tarsnap! (absurd pricing for 2TB).
- justsomehnguy 3y ago
  > absurd pricing for 2TB Well, it can't be that ba.. $0.25 x 2000 = $500 Yikes. And this is without BW costs. At $500/M you can just rent a dedicated physical server with a lot of HDDs and still have money left for your favourite pumpkin latte. For comparison rsync.net says it's $0.015 per GB/Mo, for 2TBs that's $30/m and no BW costs.
  - frankjr 3y ago
    Not in any way affiliated but I'm a happy user of Scaleway's Object Storage [0] together with S3QL [1]. It's not the fastest but they give you 75GB of storage for free so that's a fair trade [2]. [0] https://www.scaleway.com/en/object-storage https://www.scaleway.com/en/object-storage [1] https://github.com/s3ql/s3ql https://github.com/s3ql/s3ql [2] https://www.scaleway.com/en/pricing/?tags=storage https://www.scaleway.com/en/pricing/?tags=storage
  - ilyt 3y ago
    I'm renting $15/mo 2TB atom machine from OVH/kimsufi as second target for backups. Now that I think about it... some kind of micro-distributed backup server (throw on few of your machines, auto-replicate between) would be a neat project...
    - justsomehnguy 3y ago
      It's not even that neat. Just slap rsync/syncthing to the backup dir.
      - ilyt 3y ago
        I do use syncthing on NAS + remote cheapo server for my day to day stuff, and bareos for rest. It's just PITA to add another instance.

sandgiant 3y ago

Curious how much you backup, which version of restic you're running and why you think the deduplication is borderline unacceptable. There were several major (orders of magnitudes) improvements made to pruning within the past ~1 year, that's why I'm interested.

mattbee 3y ago
A straight upgrade, that I can do :) It's been running for years without one. I was only edgy about it because when it takes 36h it blocks the next daily backup, and I wondered whether that was going to get worse (it hasn't).
- chibea 3y ago
  The max-unused percentage feature is well worth it to 80/20 the prune process and only prune the data which is easiest to prune away (i.e. not try to remove small files big packs but focus on packs which have lots of garbage). In general, there's an unavoidable trade-off between creating many small packs (harder on metadata throughout the system, inside restic and on the backing store but more efficient to prune) versus creating big packs which are more easy on the metadata but might create big repack cost. I guess a bit more intelligent repacking could avoid some of that cost by packing stuff together that might be more likely to get pruned together.

aborsy 3y ago

I backup around 2TB with Restic, also tried locally with Borg. The size is nearly the same. Sadly, I can’t even test with Tarsnap! (absurd pricing for 2TB).

justsomehnguy 3y ago
> absurd pricing for 2TB Well, it can't be that ba.. $0.25 x 2000 = $500 Yikes. And this is without BW costs. At $500/M you can just rent a dedicated physical server with a lot of HDDs and still have money left for your favourite pumpkin latte. For comparison rsync.net says it's $0.015 per GB/Mo, for 2TBs that's $30/m and no BW costs.
- frankjr 3y ago
  Not in any way affiliated but I'm a happy user of Scaleway's Object Storage [0] together with S3QL [1]. It's not the fastest but they give you 75GB of storage for free so that's a fair trade [2]. [0] https://www.scaleway.com/en/object-storage https://www.scaleway.com/en/object-storage [1] https://github.com/s3ql/s3ql https://github.com/s3ql/s3ql [2] https://www.scaleway.com/en/pricing/?tags=storage https://www.scaleway.com/en/pricing/?tags=storage
- ilyt 3y ago
  I'm renting $15/mo 2TB atom machine from OVH/kimsufi as second target for backups. Now that I think about it... some kind of micro-distributed backup server (throw on few of your machines, auto-replicate between) would be a neat project...
  - justsomehnguy 3y ago
    It's not even that neat. Just slap rsync/syncthing to the backup dir.
    - ilyt 3y ago
      I do use syncthing on NAS + remote cheapo server for my day to day stuff, and bareos for rest. It's just PITA to add another instance.