4 ms·

I have a local clone of a git repo with many local branches. I lost an object file (and I'm not certain how). Now git gc always fails, complaining about the o

I have a local clone of a git repo with many local branches. I lost an object file (and I'm not certain how). Now git gc always fails, complaining about the object: Counting objects: 100% (11785289/11785289), done. Delta compression using up to 64 threads Compressing objects: 100% (4116944/4116944), done. fatal: unable to read 1cae71a9d5b24991c0d632b45186ca8a250e5d52 fatal: failed to run repack I've cloned the repo again, and that object does not appear in the new clone, so I assume it must be from a commit to a local branch. The odd thing is that I think I locally cloned this repo, and saw no complaints about the missing object. Is there any way to tell what branch and/or file(s) were referred to by the object? And, assuming its from a stale branch, just delete the branch and thereby fix my repo?

masklinn 3y ago
You can try to git fsck in case that identifies the missing object. But I’m not sure there’s any ready made command to identify through which path a missing object is reached. It could actually be the reflog. Pruning the reflog then running “git gc —prune=now” might do the trick.
- drewg123 3y ago
  Thank you. I've tried git fsck in the past, and it complains about the missing object as well. And I just tried "git gc —prune=now", and sadly it still fails the same way. I'm afraid I'm going to have to bite the bullet and clean up my 30 or so worktrees and re-clone the repo, and re-create the worktrees.
  - williamcotton 3y ago
    Try a grep for that missing hash in the fatal message you posted and point it at a commit/object that does exist or just delete that line/file and see if that fixes it!
    - drewg123 3y ago
      This is more or less what I've been trying to do for quite a while. How do I grep for it? And once I do that, how can I "point it at a commit/object that does exist ?"
      - williamcotton 3y ago
        If it's a ref somewhere then it'll just be a string of the hash deadbeef style. Just open the file up and change it to another commit hash or just delete the ref completely. Try something like: grep -r "1cae71a9" ./.git/ And see if it comes up somewhere. My guess is that it's a ref for some old branch pointing at a commit that isn't around.
        Denvercoder9 3y ago
        That won't work, as loose objects are stored zlib-compressed in .git, and packed objects can be stored in a delta format. You'd need something like this: git cat-file --unordered --batch-check='%(objectname)' --batch-all-objects | xargs -n1 -- git cat-file -p | grep 1cae71a9d5b24991c0d632b45186ca8a250e5d52
        drewg123 3y ago
        Thank you! This has been running for several hours now. If this allows me to fix it, I'll be very grateful. I've been delaying re-rooting all those worktrees for 6mo or more.. :)
  - pcthrowaway 3y ago
    You might be able to recreate the git history without that object (or commit). Look into commands like filter-branch. I'd be surprised if there's not a way to recover from this situation
Borg3 3y ago
Thats interesting.. Did you had any system crash recently? Or maybe disk full? The only moment I think this is the only way to loose object like this. Anyway, If this is your only repo containing those branches then its gone. The only way you can try is to clone fresh repo from trusted source and then reconstruct every branch manually, basically squashing commits. And I bet, one of those branches will be broken.
hddqsb 3y ago
I'd first check whether `git fsck` gives you more information. There isn't a simple and fast way to find parent objects, but here is a slow hack: git_find_references() { if [ "$#" -ne 1 ] || [ "${1#-}" != "$1" ]; then echo "usage: git_find_references <HASH>" >&2; return 1 fi local target="$1" git cat-file --batch-check --batch-all-objects --unordered | while read -r hash type size; do if [ "$type" != "blob" ] && git cat-file -p "$hash" | grep -F -q "$target"; then printf "%s %s\n" "$hash" "$type" fi done } # example: git_find_references 1cae71a9d5b24991c0d632b45186ca8a250e5d52 This will output the hashes and types of the objects that refer to the target. (The command is similar to the one posted by @Denvercoder9, but is slightly more sophisticated.) Note that this function will be very slow on large repositories. I can't think of a way to make it faster without writing actual code. (My only tip is that you can search for multiple targets at once by changing the grep command to `grep -E -q "$target"` and then using `git_find_references "HASH1|HASH2|HASH3"`.) Once you find the referring objects, you can use `git cat-file -p <hash>` on them to see their content, and you can repeatedly invoke this function to walk up a directory tree until you get to the enclosing commit. --- To actually fix the corruption, you have a couple of options: - If the corruption is in a branch that you don't care about, just delete the branch. - Find a good copy of the object, e.g. from another repository (you said the object did not appear, but I just want to point out that you should check using `git cat-file -p <hash>` rather than by looking in .git/objects/, because objects can also be stored in packs). Or you might be able to work out what the content was (if the corrupt object is a blob, aka file, you might be able to work out the content based on the previous and next versions of that file; `git hash-object <file>` will tell you if it's exactly right, but won't give you any hints if it's slightly wrong). - Rewrite the history to remove/replace the corrupt object. You can use a tool such as `git-filter-repo` (docs: https://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html https://htmlpreview.github.io/?https://github.com/newren/git..., in particular see --strip-blobs-with-ids). You can also use `git replace` to temporarily replace the corrupt object with a placeholder; it won't fix the commit history, but it might be useful if the corruption causes commands to fail.
- hddqsb 3y ago
yencabulator 3y ago
Have you run git fsck? It should tell you what kind of an object is missing -- and might say who pointed there, I can't remember. Regardless, that's the next step to figuring out who points to it. For this: > I've cloned the repo again, and that object does not appear in the new clone, so I assume it must be from a commit to a local branch. I hope you realize it can be within a pack, not just as a loose object. git cat-file -t 1cae71a9d5b24991c0d632b45186ca8a250e5d52 in the other clones.

masklinn 3y ago

You can try to git fsck in case that identifies the missing object. But I’m not sure there’s any ready made command to identify through which path a missing object is reached. It could actually be the reflog. Pruning the reflog then running “git gc —prune=now” might do the trick.

drewg123 3y ago
Thank you. I've tried git fsck in the past, and it complains about the missing object as well. And I just tried "git gc —prune=now", and sadly it still fails the same way. I'm afraid I'm going to have to bite the bullet and clean up my 30 or so worktrees and re-clone the repo, and re-create the worktrees.
- williamcotton 3y ago
  Try a grep for that missing hash in the fatal message you posted and point it at a commit/object that does exist or just delete that line/file and see if that fixes it!
  - drewg123 3y ago
    This is more or less what I've been trying to do for quite a while. How do I grep for it? And once I do that, how can I "point it at a commit/object that does exist ?"
    - williamcotton 3y ago
      If it's a ref somewhere then it'll just be a string of the hash deadbeef style. Just open the file up and change it to another commit hash or just delete the ref completely. Try something like: grep -r "1cae71a9" ./.git/ And see if it comes up somewhere. My guess is that it's a ref for some old branch pointing at a commit that isn't around.
      - Denvercoder9 3y ago
        That won't work, as loose objects are stored zlib-compressed in .git, and packed objects can be stored in a delta format. You'd need something like this: git cat-file --unordered --batch-check='%(objectname)' --batch-all-objects | xargs -n1 -- git cat-file -p | grep 1cae71a9d5b24991c0d632b45186ca8a250e5d52
        drewg123 3y ago
        Thank you! This has been running for several hours now. If this allows me to fix it, I'll be very grateful. I've been delaying re-rooting all those worktrees for 6mo or more.. :)
- pcthrowaway 3y ago
  You might be able to recreate the git history without that object (or commit). Look into commands like filter-branch. I'd be surprised if there's not a way to recover from this situation

Borg3 3y ago

Thats interesting.. Did you had any system crash recently? Or maybe disk full? The only moment I think this is the only way to loose object like this. Anyway, If this is your only repo containing those branches then its gone. The only way you can try is to clone fresh repo from trusted source and then reconstruct every branch manually, basically squashing commits. And I bet, one of those branches will be broken.

hddqsb 3y ago

I'd first check whether `git fsck` gives you more information. There isn't a simple and fast way to find parent objects, but here is a slow hack: git_find_references() { if [ "$#" -ne 1 ] || [ "${1#-}" != "$1" ]; then echo "usage: git_find_references <HASH>" >&2; return 1 fi local target="$1" git cat-file --batch-check --batch-all-objects --unordered | while read -r hash type size; do if [ "$type" != "blob" ] && git cat-file -p "$hash" | grep -F -q "$target"; then printf "%s %s\n" "$hash" "$type" fi done } # example: git_find_references 1cae71a9d5b24991c0d632b45186ca8a250e5d52 This will output the hashes and types of the objects that refer to the target. (The command is similar to the one posted by @Denvercoder9, but is slightly more sophisticated.) Note that this function will be very slow on large repositories. I can't think of a way to make it faster without writing actual code. (My only tip is that you can search for multiple targets at once by changing the grep command to `grep -E -q "$target"` and then using `git_find_references "HASH1|HASH2|HASH3"`.) Once you find the referring objects, you can use `git cat-file -p <hash>` on them to see their content, and you can repeatedly invoke this function to walk up a directory tree until you get to the enclosing commit. --- To actually fix the corruption, you have a couple of options: - If the corruption is in a branch that you don't care about, just delete the branch. - Find a good copy of the object, e.g. from another repository (you said the object did not appear, but I just want to point out that you should check using `git cat-file -p <hash>` rather than by looking in .git/objects/, because objects can also be stored in packs). Or you might be able to work out what the content was (if the corrupt object is a blob, aka file, you might be able to work out the content based on the previous and next versions of that file; `git hash-object <file>` will tell you if it's exactly right, but won't give you any hints if it's slightly wrong). - Rewrite the history to remove/replace the corrupt object. You can use a tool such as `git-filter-repo` (docs: https://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html https://htmlpreview.github.io/?https://github.com/newren/git..., in particular see --strip-blobs-with-ids). You can also use `git replace` to temporarily replace the corrupt object with a placeholder; it won't fix the commit history, but it might be useful if the corruption causes commands to fail.

hddqsb 3y ago

yencabulator 3y ago

Have you run git fsck? It should tell you what kind of an object is missing -- and might say who pointed there, I can't remember. Regardless, that's the next step to figuring out who points to it. For this: > I've cloned the repo again, and that object does not appear in the new clone, so I assume it must be from a commit to a local branch. I hope you realize it can be within a pack, not just as a loose object. git cat-file -t 1cae71a9d5b24991c0d632b45186ca8a250e5d52 in the other clones.