5 ms·

I can understand in theory why they wouldn't want to back up .git folders as-is. Git has a serious object count bloat problem if you have any repository with a

I can understand in theory why they wouldn't want to back up .git folders as-is. Git has a serious object count bloat problem if you have any repository with a good amount of commit history, which causes a lot of unnecessary overhead in just scanning the folder for files alone. I don't quite understand why it's still like this; it's probably the biggest reason why git tends to play poorly with a lot of filesystem tools (not just backups). If it'd been something like an SQLite database instead (just an example really), you wouldn't get so much unnecessary inode bloat. At the same time Backblaze is a backup solution. The need to back up everything is sort of baked in there. They promise to be the third backup solution in a three layer strategy (backup directly connected, backup in home, backup external), and that third one is probably the single most important one of them all since it's the one you're going to be touching the least in an ideal scenario. They really can't be excluding any files whatsoever. The cloud service exclusion is similarly bad, although much worse. Imagine getting hit by a cryptoworm. Your cloud storage tool is dutifully going to sync everything encrypted, junking up your entire storage across devices and because restoring old versions is both ass and near impossible at scale, you need an actual backup solution for that situation. Backblaze excluding files in those folders feels like a complete misunderstanding of what their purpose should be.

ciupicri 6mo ago
> If it'd been something like an SQLite database instead (just an example really) See Fossil (https://fossil-scm.org/ https://fossil-scm.org/) P.S. There's also (https://www.sourcegear.com/vault/ https://www.sourcegear.com/vault/) > SourceGear Vault Pro is a version control and bug tracking solution for professional development teams. Vault Standard is for those who only want version control. Vault is based on a client / server architecture using technologies such as Microsoft SQL Server and IIS Web Services for increased performance, scalability, and security.
maalhamdan 6mo ago
I think they shouldn't back up git objects individually because git handles the versioning information. Just compress the .git folder itself and back it up as a single unit.
- pkaeding 6mo ago
  This is a good point, but you might expect them to back up untracked and modified files in the backup, along with everything else on your filesystem.
  - pixl97 6mo ago
    Eh, you really shouldn't do that for any kind of file that acts like a (an impromptu) database. This is how you get corruption. Especially when change information can be split across more than one file.
    - pkaeding 6mo ago
      Sorry, what are you saying shouldn't be done? Backing up untracked/modified files in a bit repo? Or compressing the .git folder and backing it up as a unit?
      - pixl97 6mo ago
        > Backing up untracked/modified files in a bit repo? This. It's best to do this in an atomic operation, such as a VSS style snapshot that then is consistent and done with no or paused operations on the files. Something like a zip is generally better because it takes less time on the file system than the upload process typically takes.
        pkaeding 6mo ago
        I see what you mean, but isn't this an issue with any filesystem backup tool? Or is there something about untracked files in a git workspace that is different, that I'm not seeing?
- willis936 6mo ago
  Better yet, include dedpulication, incremental versioning, verification, and encryption. Wait, that's borg / restic. This is a joke, but honestly anyone here shouldn't be directly backing up their filesystems and should instead be using the right tool for the job. You'll make the world a more efficient place, have more robust and quicker to recover backups, and save some money along the way.
rmccue 6mo ago
I think it's understandable for both Backblaze and most users, but surely the solution is to add `.git` to their default exclusion list which the user can manage.
rcxdude 6mo ago
It's probably primarily because Linus is a kernel and filesystem nerd, not a database nerd, so he preferred to just use the filesystem which he understood the performance characteristics of well (at least on linux).
grumbelbart2 6mo ago
Git packs objects into pack-files on a regular basis. If it doesn't, check your configuration, or do it manually with 'git repack'.
- noirscape 6mo ago
  I decided to look into this (git gc should also be doing this), and I think I figured out why it's such a consistent issue with git in particular. Running git gc does properly pack objects together and reduce inode count to something much more manageable. It's the same reason why the postgres autovacuum daemon tends to be borderline useless unless you retune it[0]: the defaults are barmy. git gc only runs if there's 6700 loose unpacked objects[1]. Most typical filesystem tools tend to start balking at traversing ~1000 files in a structure (depends a bit on the filesystem/OS as well, Windows tends to get slower a good bit earlier than Linux). To fix it, running > git config --global gc.auto 1000 should retune it and any subsequent commit to your repo's will trigger garbage collection properly when there's around 1000 loose files. Pack file management seems to be properly tuned by default; at more than 50 packs, gc will repack into a larger pack. [0]: For anyone curious, the default postgres autovacuum setting runs only when 10% of the table consists of dead tuples (roughly: deleted+every revision of an updated row). If you're working with a beefy table, you're never hitting 10%. Either tune it down or create an external cronjob to run vacuum analyze more frequently on the tables you need to keep speedy. I'm pretty sure the defaults are tuned solely to ensure that Postgres' internal tables are fast, since those seem to only have active rows to a point where it'd warrant autovacuum. [1]: https://git-scm.com/docs/git-gc https://git-scm.com/docs/git-gc
  - LetTheSmokeOut 6mo ago
    I needed to use > git config --global gc.auto 1000 with the long option name, and no `=`.
  - Dylan16807 6mo ago
    A few thousand files shouldn't be a problem to a program designed to scan entire drives of files. Even in a single folder and considering sloppy programs I wouldn't worry just yet, and git's not putting them in a single folder.
adithyassekhar 6mo ago
I don’t think this is the right way to see this. Why should a file backup solution adapt to work with git? Or any application? It should not try to understand what a git object is. I’m paying to copy files from a folder to their servers just do that. No matter what the file is. Stay at the filesystem level not the application level.
- noirscape 6mo ago
  I'm not saying Backblaze should adapt to git; the issue isn't application related (besides git being badly configured by default; there's a solution with git gc, it's just that git gc basically never runs). It's that to back up a folder on a filesystem, you need to traverse that folder and check every file in that folder to see if it's changed. Most filesystem tools usually assume a fairly low file count for these operations. Git, rather unusually, tends to produce a lot of files in regular use; before packing, every commit/object/branch is simply stored as a file on the filesystem (branches only as pointers). Packing fixes that by compressing commit and object files together, but it's not done by default (only after an initial clone or when the garbage collector runs). Iterating over a .git folder can take a lot of time in a place that's typically not very well optimized (since most "normal" people don't have thousands of tiny files in their folders that contain sprawled out application state.) The correct solution here is either for git to change, or for Backblaze to implement better iteration logic (which will probably require special handling for git..., so it'd be more "correct" to fix up git, since Backblaze's tools aren't the only ones with this problem.)
  - masfuerte 6mo ago
    7za (the compression app) does blazingly fast iteration over any kind of folder. This doesn't require special code for git. Backblaze's backup app could do the same but rather than fix their code they excluded .git folders. When I backup my computer the .git folders are among the most important things on there. Most of my personal projects aren't pushed to github or anywhere else. Fortunately I don't use Backblaze. I guess the moral is don't use a backup solution where the vendor has an incentive to exclude things.
yangm97 6mo ago
You don’t see ZFS/BTRFS block based snapshot replication choking on git or any sort of dataset. Use the right job for the tool or something.
deleted 6mo ago
[deleted]
Ajedi32 6mo ago
FWIW some other people in this thread are saying the article is wrong about .git folders not being backed up: https://news.ycombinator.com/item?id=47765788 https://news.ycombinator.com/item?id=47765788 That's a really important fact that's getting buried so I'd like to highlight it here.
- Ajedi32 6mo ago
  Well, I checked and it looks like none of my .git repos are backed up. All attempts to restore only restore the working copy. -_- I'm not sure why it was working for the person in the comment I linked.
  - Ajedi32 6mo ago
    Update: Deleting C:\Programdata\Backblaze\bzdata\bzexcluderules_mandatory.xml resolved the problem for me. Seems like at one point[1] they started excluding .git directories by default, got a bunch of backlash, reverted that change, but never changed the setting back for some users (like me). [1]: https://www.reddit.com/r/backblaze/comments/1cgy93n/i_did_a_full_restore_test_some_files_did_not_get/ https://www.reddit.com/r/backblaze/comments/1cgy93n/i_did_a_...
stebalien 6mo ago
I've actually spent some time debugging why git causes so many issues with the backup software I use (restic). Ironically, I believe you have it backwards: pack files, git's solution to the "too many tiny files" problem, are the issue here; not the tiny files themselves. In my experience, incremental backup software works best with many small files that never change. Scanning is usually just a matter of checking modification times and moving on. This isn't fast, but it's fast enough for backups and can be optimized by monitoring for file changes in a long-running daemon. However, lots of mostly identical files ARE an issue for filesystems as they tend to waste a lot of space. Git solves this issue by packing these small objects into larger pack files, then compressing them. Unfortunately, it's those pack files that cause issues for backup software: any time git "garbage collects" and creates new pack files, it ends up deleting and creating a bunch of large files filled with what looks like random data (due to compression). Constantly creating/deleting large files filled with random data wreaks havoc on incremental/deduplicating backup systems.

ciupicri 6mo ago

> If it'd been something like an SQLite database instead (just an example really) See Fossil (https://fossil-scm.org/ https://fossil-scm.org/) P.S. There's also (https://www.sourcegear.com/vault/ https://www.sourcegear.com/vault/) > SourceGear Vault Pro is a version control and bug tracking solution for professional development teams. Vault Standard is for those who only want version control. Vault is based on a client / server architecture using technologies such as Microsoft SQL Server and IIS Web Services for increased performance, scalability, and security.

maalhamdan 6mo ago

I think they shouldn't back up git objects individually because git handles the versioning information. Just compress the .git folder itself and back it up as a single unit.

pkaeding 6mo ago
This is a good point, but you might expect them to back up untracked and modified files in the backup, along with everything else on your filesystem.
- pixl97 6mo ago
  Eh, you really shouldn't do that for any kind of file that acts like a (an impromptu) database. This is how you get corruption. Especially when change information can be split across more than one file.
  - pkaeding 6mo ago
    Sorry, what are you saying shouldn't be done? Backing up untracked/modified files in a bit repo? Or compressing the .git folder and backing it up as a unit?
    - pixl97 6mo ago
      > Backing up untracked/modified files in a bit repo? This. It's best to do this in an atomic operation, such as a VSS style snapshot that then is consistent and done with no or paused operations on the files. Something like a zip is generally better because it takes less time on the file system than the upload process typically takes.
      - pkaeding 6mo ago
        I see what you mean, but isn't this an issue with any filesystem backup tool? Or is there something about untracked files in a git workspace that is different, that I'm not seeing?
willis936 6mo ago
Better yet, include dedpulication, incremental versioning, verification, and encryption. Wait, that's borg / restic. This is a joke, but honestly anyone here shouldn't be directly backing up their filesystems and should instead be using the right tool for the job. You'll make the world a more efficient place, have more robust and quicker to recover backups, and save some money along the way.

rmccue 6mo ago

I think it's understandable for both Backblaze and most users, but surely the solution is to add `.git` to their default exclusion list which the user can manage.

rcxdude 6mo ago

It's probably primarily because Linus is a kernel and filesystem nerd, not a database nerd, so he preferred to just use the filesystem which he understood the performance characteristics of well (at least on linux).

grumbelbart2 6mo ago

Git packs objects into pack-files on a regular basis. If it doesn't, check your configuration, or do it manually with 'git repack'.

noirscape 6mo ago
I decided to look into this (git gc should also be doing this), and I think I figured out why it's such a consistent issue with git in particular. Running git gc does properly pack objects together and reduce inode count to something much more manageable. It's the same reason why the postgres autovacuum daemon tends to be borderline useless unless you retune it[0]: the defaults are barmy. git gc only runs if there's 6700 loose unpacked objects[1]. Most typical filesystem tools tend to start balking at traversing ~1000 files in a structure (depends a bit on the filesystem/OS as well, Windows tends to get slower a good bit earlier than Linux). To fix it, running > git config --global gc.auto 1000 should retune it and any subsequent commit to your repo's will trigger garbage collection properly when there's around 1000 loose files. Pack file management seems to be properly tuned by default; at more than 50 packs, gc will repack into a larger pack. [0]: For anyone curious, the default postgres autovacuum setting runs only when 10% of the table consists of dead tuples (roughly: deleted+every revision of an updated row). If you're working with a beefy table, you're never hitting 10%. Either tune it down or create an external cronjob to run vacuum analyze more frequently on the tables you need to keep speedy. I'm pretty sure the defaults are tuned solely to ensure that Postgres' internal tables are fast, since those seem to only have active rows to a point where it'd warrant autovacuum. [1]: https://git-scm.com/docs/git-gc https://git-scm.com/docs/git-gc
- LetTheSmokeOut 6mo ago
  I needed to use > git config --global gc.auto 1000 with the long option name, and no `=`.
- Dylan16807 6mo ago
  A few thousand files shouldn't be a problem to a program designed to scan entire drives of files. Even in a single folder and considering sloppy programs I wouldn't worry just yet, and git's not putting them in a single folder.

adithyassekhar 6mo ago

I don’t think this is the right way to see this. Why should a file backup solution adapt to work with git? Or any application? It should not try to understand what a git object is. I’m paying to copy files from a folder to their servers just do that. No matter what the file is. Stay at the filesystem level not the application level.

noirscape 6mo ago
I'm not saying Backblaze should adapt to git; the issue isn't application related (besides git being badly configured by default; there's a solution with git gc, it's just that git gc basically never runs). It's that to back up a folder on a filesystem, you need to traverse that folder and check every file in that folder to see if it's changed. Most filesystem tools usually assume a fairly low file count for these operations. Git, rather unusually, tends to produce a lot of files in regular use; before packing, every commit/object/branch is simply stored as a file on the filesystem (branches only as pointers). Packing fixes that by compressing commit and object files together, but it's not done by default (only after an initial clone or when the garbage collector runs). Iterating over a .git folder can take a lot of time in a place that's typically not very well optimized (since most "normal" people don't have thousands of tiny files in their folders that contain sprawled out application state.) The correct solution here is either for git to change, or for Backblaze to implement better iteration logic (which will probably require special handling for git..., so it'd be more "correct" to fix up git, since Backblaze's tools aren't the only ones with this problem.)
- masfuerte 6mo ago
  7za (the compression app) does blazingly fast iteration over any kind of folder. This doesn't require special code for git. Backblaze's backup app could do the same but rather than fix their code they excluded .git folders. When I backup my computer the .git folders are among the most important things on there. Most of my personal projects aren't pushed to github or anywhere else. Fortunately I don't use Backblaze. I guess the moral is don't use a backup solution where the vendor has an incentive to exclude things.

yangm97 6mo ago

You don’t see ZFS/BTRFS block based snapshot replication choking on git or any sort of dataset. Use the right job for the tool or something.

deleted 6mo ago

[deleted]

Ajedi32 6mo ago

FWIW some other people in this thread are saying the article is wrong about .git folders not being backed up: https://news.ycombinator.com/item?id=47765788 https://news.ycombinator.com/item?id=47765788 That's a really important fact that's getting buried so I'd like to highlight it here.

Ajedi32 6mo ago
Well, I checked and it looks like none of my .git repos are backed up. All attempts to restore only restore the working copy. -_- I'm not sure why it was working for the person in the comment I linked.
- Ajedi32 6mo ago
  Update: Deleting C:\Programdata\Backblaze\bzdata\bzexcluderules_mandatory.xml resolved the problem for me. Seems like at one point[1] they started excluding .git directories by default, got a bunch of backlash, reverted that change, but never changed the setting back for some users (like me). [1]: https://www.reddit.com/r/backblaze/comments/1cgy93n/i_did_a_full_restore_test_some_files_did_not_get/ https://www.reddit.com/r/backblaze/comments/1cgy93n/i_did_a_...

stebalien 6mo ago

I've actually spent some time debugging why git causes so many issues with the backup software I use (restic). Ironically, I believe you have it backwards: pack files, git's solution to the "too many tiny files" problem, are the issue here; not the tiny files themselves. In my experience, incremental backup software works best with many small files that never change. Scanning is usually just a matter of checking modification times and moving on. This isn't fast, but it's fast enough for backups and can be optimized by monitoring for file changes in a long-running daemon. However, lots of mostly identical files ARE an issue for filesystems as they tend to waste a lot of space. Git solves this issue by packing these small objects into larger pack files, then compressing them. Unfortunately, it's those pack files that cause issues for backup software: any time git "garbage collects" and creates new pack files, it ends up deleting and creating a bunch of large files filled with what looks like random data (due to compression). Constantly creating/deleting large files filled with random data wreaks havoc on incremental/deduplicating backup systems.