4 ms·

I recently moved on to using Google Drive as a backup drive for my computer, the tool I'm using for it, Restic[1], does file deduplication at file block level.

I recently moved on to using Google Drive as a backup drive for my computer, the tool I'm using for it, Restic[1], does file deduplication at file block level. Restic allow multiple hosts to upload to the same backend/repository. So far I already have 25k files, even though I started backing up a only few weeks ago. I imagine that if you setup multiple servers/personal computers backups that could scale significantly. [1]: https://restic.net/ https://restic.net/

pretext-1 4y ago
Restic + rclone + GCP Service Account. This is the way to go. Files will count towards the Service Account, not your account. If you use Google Workspace I strongly recommend using a Shared Drive (instead of sharing a folder in “My Drive”) so the files will be owned by your org, not by the Service Account (otherwise deleting the SA would result in the files being deleted as well, because in “My Drive” the creator of a file keeps ownership even when creating in a shared folder). If you have problems with the 400,000 files limit in a Shared Drive: create a new one, rclone has a “union” backend. It can be set to create new files in the new Shared Drive while still showing files from the other drive(s). Also Shared Drives have their own recent activity views so it doesn’t clutter your “my drive” view. Create a SA in GCP and generate a JSON credentials file. Create a folder or Shared Drive via web and share it with the SA. I’m using this setup for years with zero problems. Example rclone config: https://pastebin.com/KdsFQz5K https://pastebin.com/KdsFQz5K
Gigachad 4y ago
> Restic currently defaults to a 16 MiB pack size. Given this, it is impossible to exceed the 5M file limit with the top plan google offers of 30TB. You'd have to lower the pack size to 6MB or less.

4 ms·

I recently moved on to using Google Drive as a backup drive for my computer, the tool I'm using for it, Restic[1], does file deduplication at file block level.

by augusto-moura 4y ago

pretext-1 4y ago
Restic + rclone + GCP Service Account. This is the way to go. Files will count towards the Service Account, not your account. If you use Google Workspace I strongly recommend using a Shared Drive (instead of sharing a folder in “My Drive”) so the files will be owned by your org, not by the Service Account (otherwise deleting the SA would result in the files being deleted as well, because in “My Drive” the creator of a file keeps ownership even when creating in a shared folder). If you have problems with the 400,000 files limit in a Shared Drive: create a new one, rclone has a “union” backend. It can be set to create new files in the new Shared Drive while still showing files from the other drive(s). Also Shared Drives have their own recent activity views so it doesn’t clutter your “my drive” view. Create a SA in GCP and generate a JSON credentials file. Create a folder or Shared Drive via web and share it with the SA. I’m using this setup for years with zero problems. Example rclone config: https://pastebin.com/KdsFQz5K https://pastebin.com/KdsFQz5K
Gigachad 4y ago
> Restic currently defaults to a 16 MiB pack size. Given this, it is impossible to exceed the 5M file limit with the top plan google offers of 30TB. You'd have to lower the pack size to 6MB or less.