5 ms·

If you have a Raspberry Pi wasting away in a drawer[0], I strongly recommend installing Gitea or Forgejo and mirroring all the repos you like (i.e. the ones you

If you have a Raspberry Pi wasting away in a drawer[0], I strongly recommend installing Gitea or Forgejo and mirroring all the repos you like (i.e. the ones you contribute(d) to and/or starred, not just on Github too!). You set it up once and it will sync in upstream changes as often as you like (default is daily) 0. Or a homelab, or a cheap 256MB VM, or a NAS that can run docker containers, or an old Chromebook: anything that can execute a Go binary, x64, arm6 or even mips

kitkat_new 4y ago
I'll look for a tool that mirrors all my repositories (i.e. only forks). If I can't find one, I'll write it myself.
- jmondi 4y ago
  I wrote a little tool to mirror my repositories to my gitea instance. It has been months to potentially a year or two since I ran this, but it does what you’re asking. You can mirror repositories of users and repos users have starred. It definitely needs some love. https://github.com/jasonraimondi/deno-mirror-to-gitea https://github.com/jasonraimondi/deno-mirror-to-gitea
  - mckn1ght 4y ago
    I wrote a similar one I called Forgery that’s in a repo of the same name in my github account (which can be found in my user profile… just created this account and didn’t want to post a link in my first post, thought it might get automodded). Very similar to yours, but also does forks, which I’m not sure yours does by a quick glance at your readme. Although, mine doesn’t automatically mirror from another forge, just clones everything locally. I’ll have to add a TODO to add mirroring. Please have a look at how I handle them and consider adding a link to my project in your “similar tools” section, and I’ll do the same for you!
    - jmondi 4y ago
      If I remember correctly, I believe that mine will do forks also, they are just included in in your repositories. I just looked over your project, and it seems pretty cool. A little bit different than mine, since mine is specifically github to a gitea mirror repository. I don't mind adding a link to your project in mine. Thanks for sharing!
- pabs3 4y ago
  https://github-backup.branchable.com/ https://github-backup.branchable.com/
- drudoo 4y ago
  This is build in to Gitea. I mirror multiple github repos to my private gitea instance.
remram 4y ago
I wonder if there is an efficient way to do direct incremental git-to-S3 backups, or if you have to do this, run a Git mirror and do regular filesystem-level backups of it.
- anshumankmr 4y ago
  a cron job ought to do this. Try something like Cloud Scheduler that can automate this for you.( I am not sure what the equivalent is in AWS)
  - remram 4y ago
    My question is not how to run it, but what to run. If your scheduled task does a full clone every time to upload as ZIP to S3, it is massively inefficient. Even if you use something like Restic, because the Git pack file will have nothing in common with the previous one.
    - saltcured 4y ago
      Clone each repo locally. Periodically do "git fetch -p" for each repo to update the local copy of upstream content. Run some periodic task like restic or rclone (depending on whether you want point in time snapshots or just a mirror of latest state) to mirror these local repos into your S3 bucket. The local clones should evolve incrementally due to "git fetch", and then the restic or rclone task should figure out how to make incremental updates to the S3 content.
      - remram 4y ago
        But that's... exactly what I described and asked how to avoid...
        saltcured 4y ago
        I had trouble parsing your earliest comment, so I only tried to address the incremental backup concern. I may not have understood the conversation, but it seemed like you claimed that a filesystem level backup of a clone was not going to produce incremental backup IO in practice. A periodic fetch into a persistent cloned repo will be incremental unless the upstream is doing something crazy with frequent branch deletions and repacks. In practice, most upstream repos I encounter behave relatively monotonically. They accumulate new commits and branch/tag heads but do not often create garbage or need repacking. A periodic backup of the cloned repo will also be incremental if using an appropriate tool like restic or rclone-copy. Also, since the clone only changes during the fetch, you can serialize these in one periodic job and be confident that you are making a consistent snapshot of the repo. The advantage of this approach is its simplicity. It is easy to reason about and easy to work with the backups to restore a repo without having to learn about other tools. It's the kind of thing I could feel comfortable setting up and running for years on end with little supervision. A more sophisticated approach that integrates with git hooks, e.g. to do event-driven rather than periodic backup, is plausible but I think could quickly get in the way of itself. And if working with a hosted upstream, you would need to integrate with their proprietary hooks, e.g. GitHub actions, and deal with other restrictions of the hosting environment. Such a solution likely brings new failure modes and may not be a worthwhile tradeoff...

kitkat_new 4y ago

I'll look for a tool that mirrors all my repositories (i.e. only forks). If I can't find one, I'll write it myself.

jmondi 4y ago
I wrote a little tool to mirror my repositories to my gitea instance. It has been months to potentially a year or two since I ran this, but it does what you’re asking. You can mirror repositories of users and repos users have starred. It definitely needs some love. https://github.com/jasonraimondi/deno-mirror-to-gitea https://github.com/jasonraimondi/deno-mirror-to-gitea
- mckn1ght 4y ago
  I wrote a similar one I called Forgery that’s in a repo of the same name in my github account (which can be found in my user profile… just created this account and didn’t want to post a link in my first post, thought it might get automodded). Very similar to yours, but also does forks, which I’m not sure yours does by a quick glance at your readme. Although, mine doesn’t automatically mirror from another forge, just clones everything locally. I’ll have to add a TODO to add mirroring. Please have a look at how I handle them and consider adding a link to my project in your “similar tools” section, and I’ll do the same for you!
  - jmondi 4y ago
    If I remember correctly, I believe that mine will do forks also, they are just included in in your repositories. I just looked over your project, and it seems pretty cool. A little bit different than mine, since mine is specifically github to a gitea mirror repository. I don't mind adding a link to your project in mine. Thanks for sharing!
pabs3 4y ago
https://github-backup.branchable.com/ https://github-backup.branchable.com/
drudoo 4y ago
This is build in to Gitea. I mirror multiple github repos to my private gitea instance.

remram 4y ago

I wonder if there is an efficient way to do direct incremental git-to-S3 backups, or if you have to do this, run a Git mirror and do regular filesystem-level backups of it.

anshumankmr 4y ago
a cron job ought to do this. Try something like Cloud Scheduler that can automate this for you.( I am not sure what the equivalent is in AWS)
- remram 4y ago
  My question is not how to run it, but what to run. If your scheduled task does a full clone every time to upload as ZIP to S3, it is massively inefficient. Even if you use something like Restic, because the Git pack file will have nothing in common with the previous one.
  - saltcured 4y ago
    Clone each repo locally. Periodically do "git fetch -p" for each repo to update the local copy of upstream content. Run some periodic task like restic or rclone (depending on whether you want point in time snapshots or just a mirror of latest state) to mirror these local repos into your S3 bucket. The local clones should evolve incrementally due to "git fetch", and then the restic or rclone task should figure out how to make incremental updates to the S3 content.
    - remram 4y ago
      But that's... exactly what I described and asked how to avoid...
      - saltcured 4y ago
        I had trouble parsing your earliest comment, so I only tried to address the incremental backup concern. I may not have understood the conversation, but it seemed like you claimed that a filesystem level backup of a clone was not going to produce incremental backup IO in practice. A periodic fetch into a persistent cloned repo will be incremental unless the upstream is doing something crazy with frequent branch deletions and repacks. In practice, most upstream repos I encounter behave relatively monotonically. They accumulate new commits and branch/tag heads but do not often create garbage or need repacking. A periodic backup of the cloned repo will also be incremental if using an appropriate tool like restic or rclone-copy. Also, since the clone only changes during the fetch, you can serialize these in one periodic job and be confident that you are making a consistent snapshot of the repo. The advantage of this approach is its simplicity. It is easy to reason about and easy to work with the backups to restore a repo without having to learn about other tools. It's the kind of thing I could feel comfortable setting up and running for years on end with little supervision. A more sophisticated approach that integrates with git hooks, e.g. to do event-driven rather than periodic backup, is plausible but I think could quickly get in the way of itself. And if working with a hosted upstream, you would need to integrate with their proprietary hooks, e.g. GitHub actions, and deal with other restrictions of the hosting environment. Such a solution likely brings new failure modes and may not be a worthwhile tradeoff...