3 ms·

I have a hand-coded backup system for my photo library that writes to S3. It runs every night at 2AM. The one feature I have that's important to me is this: i

by timmg 5y ago

I have a hand-coded backup system for my photo library that writes to S3. It runs every night at 2AM. The one feature I have that's important to me is this: it will figure out what files need to be uploaded and then upload as many as possible for an hour then stop. That means that it runs for at-most an hour a night. The reason I need/wanted this feature is that I might come home from a trip with (eg) 30G worth of photos. My (cable) internet will upload at around 1G an hour. I don't want this thing to saturate my internet for 30 hours straight. Instead, it backs up a small amount every night for 30 days. Am I the only one that wants a feature like this? I've never seen it in any other backup system. (At alternative might be to have configurable bandwidth for uploads.)

crecker 5y ago
I wonder if you can obtain that behavior with a bash script, but it's boring to write scripts and I do not know if SIGTERM can exit Restic gracely.
- lucgommans 5y ago
  If you've ever ctrl+c'd restic, you'll know the message > signal interrupt received, cleaning up
  - crecker 5y ago
    Thanks! Will try tonight to play with Restic
lucgommans 5y ago
Makes perfect sense. Restic kind-of supports this because you can just kill the client after an hour and, tomorrow, it'll see which objects are there already. I'm not deep enough into the project to know whether this is like an officially supported use-case, but restic was of course made with the idea that interruptions can happen (your computer can crash) and should be handled safely, and for the deduplication it'll cut files up in a deterministic way and thus (as I understand it) store those chunks in a deterministic place.
lykron 5y ago
You could do this with the `timeout` command. `timeout -s SIGINT 1h restic...` That would let restic run for one hour, and then once the hour elapses send a SIGINT which will stop the process (see https://github.com/restic/restic/blob/a29777f46794ea4e35548fce7dd9412b45ef572c/cmd/restic/cleanup.go#L58= https://github.com/restic/restic/blob/a29777f46794ea4e35548f...)
- sydney6 5y ago
  And restic would then resume the backup next time it runs. [1] https://restic.readthedocs.io/en/latest/faq.html#will-restic-resume-an-interrupted-backup https://restic.readthedocs.io/en/latest/faq.html#will-restic...
teaudel 5y ago
My internet upload speed is bad so I do want something like that. I would also like to be able to "stage" a backup: figure out what needs to be transmitted and then create the data files that need to be transmitted without actually immediately transmitting it. That would allow me to do things like backup my laptop to another computer in my house that can upload the files over my slow connection overnight when my laptop isn't on; and to let me bring the backup files to a place (work/university/library) with a fast connection so large backups don't take days or weeks (especially initial backup).
- pdimitar 5y ago
  Take a look at NNCP[0]. Haven't tried it yet but read several articles about it and I am looking forward to finding time to use it seriously. [0] https://www.complete.org/nncp/ https://www.complete.org/nncp/
- funcDropShadow 5y ago
  I am using restic to backup my laptop and workstation to my nas. At night rclone syncs the restic repositories to S3. I can restore both from my nas as well as from s3.
deleted 5y ago
[deleted]
seized 5y ago
Rclone will do exactly what you want, upload to S3 and the --max-duration will stop new transfer from starting after a given duration. There are also throttle options for bandwidth. I use that combined with Node-Red and a smart plug on my monitors, if monitor power draw exceeds a threshold then the upload throttle is changed via the rclone API.
- timmg 5y ago
  Ok, that looks like a nice tool. Thanks!
  - n8henrie 5y ago
    Was just going to suggest rclone as well, but for its easily toggle-able bandwidth limiter. I have slow rural internet; restic backs up locally every night, rclone then syncs it offsite. A systemd timer throttles it during walking hours and lets all 6 mbps rip overnight or when out of town.
- aborsy 5y ago
  Also, Restic has rclone backend!
sydney6 5y ago
Restic has support for rate limiting built-in.
eliaspro 5y ago
On Linux, this can be easily achieved by using RuntimeMaxSec= [1] in the corresponding service unit. [1] https://www.freedesktop.org/software/systemd/man/systemd.service.html#RuntimeMaxSec= https://www.freedesktop.org/software/systemd/man/systemd.ser...

by timmg 5y ago

crecker 5y ago

I wonder if you can obtain that behavior with a bash script, but it's boring to write scripts and I do not know if SIGTERM can exit Restic gracely.

lucgommans 5y ago
If you've ever ctrl+c'd restic, you'll know the message > signal interrupt received, cleaning up
- crecker 5y ago
  Thanks! Will try tonight to play with Restic

lucgommans 5y ago

Makes perfect sense. Restic kind-of supports this because you can just kill the client after an hour and, tomorrow, it'll see which objects are there already. I'm not deep enough into the project to know whether this is like an officially supported use-case, but restic was of course made with the idea that interruptions can happen (your computer can crash) and should be handled safely, and for the deduplication it'll cut files up in a deterministic way and thus (as I understand it) store those chunks in a deterministic place.

lykron 5y ago

You could do this with the `timeout` command. `timeout -s SIGINT 1h restic...` That would let restic run for one hour, and then once the hour elapses send a SIGINT which will stop the process (see https://github.com/restic/restic/blob/a29777f46794ea4e35548fce7dd9412b45ef572c/cmd/restic/cleanup.go#L58= https://github.com/restic/restic/blob/a29777f46794ea4e35548f...)

sydney6 5y ago
And restic would then resume the backup next time it runs. [1] https://restic.readthedocs.io/en/latest/faq.html#will-restic-resume-an-interrupted-backup https://restic.readthedocs.io/en/latest/faq.html#will-restic...

teaudel 5y ago

My internet upload speed is bad so I do want something like that. I would also like to be able to "stage" a backup: figure out what needs to be transmitted and then create the data files that need to be transmitted without actually immediately transmitting it. That would allow me to do things like backup my laptop to another computer in my house that can upload the files over my slow connection overnight when my laptop isn't on; and to let me bring the backup files to a place (work/university/library) with a fast connection so large backups don't take days or weeks (especially initial backup).

pdimitar 5y ago
Take a look at NNCP[0]. Haven't tried it yet but read several articles about it and I am looking forward to finding time to use it seriously. [0] https://www.complete.org/nncp/ https://www.complete.org/nncp/
funcDropShadow 5y ago
I am using restic to backup my laptop and workstation to my nas. At night rclone syncs the restic repositories to S3. I can restore both from my nas as well as from s3.

deleted 5y ago

[deleted]

seized 5y ago

Rclone will do exactly what you want, upload to S3 and the --max-duration will stop new transfer from starting after a given duration. There are also throttle options for bandwidth. I use that combined with Node-Red and a smart plug on my monitors, if monitor power draw exceeds a threshold then the upload throttle is changed via the rclone API.

timmg 5y ago
Ok, that looks like a nice tool. Thanks!
- n8henrie 5y ago
  Was just going to suggest rclone as well, but for its easily toggle-able bandwidth limiter. I have slow rural internet; restic backs up locally every night, rclone then syncs it offsite. A systemd timer throttles it during walking hours and lets all 6 mbps rip overnight or when out of town.
aborsy 5y ago
Also, Restic has rclone backend!

sydney6 5y ago

Restic has support for rate limiting built-in.

eliaspro 5y ago

On Linux, this can be easily achieved by using RuntimeMaxSec= [1] in the corresponding service unit. [1] https://www.freedesktop.org/software/systemd/man/systemd.service.html#RuntimeMaxSec= https://www.freedesktop.org/software/systemd/man/systemd.ser...