3 ms·

Using Git-annex for Data Archiving

69 points by jandeboevrie 3y ago

rsync 3y ago
https://git-annex.branchable.com/design/assistant/blog/day_73__rsync.net_configurator/ https://git-annex.branchable.com/design/assistant/blog/day_7... "Now finished building a special configurator for rsync.net. While this is just a rsync remote to git-annex, there are some tricky bits to setting up the ssh key using rsync.net's restricted shell. The configurator automates that nicely."
cdogl 3y ago
I saw a talk at linux.conf.au about a decade ago by the then-main (only?) Git Annex developer. He was living way out in the styx with no internet connection and surviving off community contributions. It seemed like the choice of Haskell, while novel and interesting, made the barrier to entry quite high even for this talented and experienced developer. All power to him, but it’s a peculiar choice for essentially a piece of plumbing software. I shudder to think what this means for casual contributors. I’m on board with functional programming and Haskell is very interesting, but I’ve still never understood the decision.
- whateveracct 3y ago
  hah if you consider yourself "talented" or "experience" and can't contribute to a project you want to contribute to just because it's Haskell, you might have misevaluated yourself on those axes [1] i never knew git annex was Haskell actually! maybe this is a marketing failure. I'd love to contribute. Haskell projects are trivial to contribute to compared to all other languages (speaking as a Haskeller) > I’m on board with functional programming and Haskell is very interesting, but I’ve still never understood the decision. I see this take often with HNers. Have you considered that the author just likes a Haskell? That it's fun? And that without it, they wouldn't have bothered spending the time on the project period? [1] I learned enough Haskell to contribute to your average codebase as an undergrad at a state school who never learned a higher level language than C in school. Neither talent nor experience are needed. But maybe they're detriments given the interplay with ego.
  - cdogl 3y ago
    > hah if you consider yourself "talented" or "experience" and can't contribute to a project you want to contribute to just because it's Haskell, you might have misevaluated yourself on those axes I was describing the Git Annex lead developer as talented and experienced, not myself.
  - ithkuil 3y ago
    > speaking as a Haskeller Perhaps that's what makes it harder for you to put yourself in other people's shoes. Having to learn a new language/stack/... for a contribution (especially a "casual" contribution) objectively raises the barrier to entry. That's not the same as to say "I can't do it because it's too hard". That's more like "I have better uses of my time and I choose to contribute to something else". The net effect of many many people doing this little choice over time is that projects use too unfamiliar stacks will receive less contributions. On the other hand, the contributions they will get will be made by people who truly care and possibly that will mean the contributions will be higher quality.
    - whateveracct 3y ago
      Put yourself in an author's shoes. git annex doesn't exist in the world - but you have the idea for it. Haskell is your strongest language, and honestly you don't really like programming in other languages. So you write it in Haskell and it works and becomes a somewhat popular piece of plumbing. For projects I write at least, I'm that author. My projects wouldn't even exist if I didn't use Haskell. So thinking about optimizing for long tail contributions by anonymous users is illogical on the face of it. It sounds to me you wouldn't think any open source project should be written in Haskell for these reasons. That seems silly to me. Also - I've contributed, debugged, etc projects in many languages I barely know. It's not that hard if you're actually talented and experienced. I actually fixed a bug in a small, single-author Haskell library for my first ever project. I hadn't even finished LYAH let alone fully understand what I had read. So when a supposedly "very experienced" software engineer "cannot learn Haskell" enough to make small changes, I raise my eyebrow. You can't do better than 21 year old me with 0 professional experience and 0 formal FP education?
ErikBjare 3y ago
I love git annex so much. Solves all of the difficult parts of data archival for me (but boy, what a learning curve, even if decent at git). Joey Hess is a legend for building it.
znxster 3y ago
Internet Archive (IABAK) offer this as well. https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK/git-annex_implementation https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK/g...

69 points by jandeboevrie 3y ago

https://git-annex.branchable.com/design/assistant/blog/day_73__rsync.net_configurator/ https://git-annex.branchable.com/design/assistant/blog/day_7... "Now finished building a special configurator for rsync.net. While this is just a rsync remote to git-annex, there are some tricky bits to setting up the ssh key using rsync.net's restricted shell. The configurator automates that nicely."

cdogl 3y ago

I saw a talk at linux.conf.au about a decade ago by the then-main (only?) Git Annex developer. He was living way out in the styx with no internet connection and surviving off community contributions. It seemed like the choice of Haskell, while novel and interesting, made the barrier to entry quite high even for this talented and experienced developer. All power to him, but it’s a peculiar choice for essentially a piece of plumbing software. I shudder to think what this means for casual contributors. I’m on board with functional programming and Haskell is very interesting, but I’ve still never understood the decision.

whateveracct 3y ago
hah if you consider yourself "talented" or "experience" and can't contribute to a project you want to contribute to just because it's Haskell, you might have misevaluated yourself on those axes [1] i never knew git annex was Haskell actually! maybe this is a marketing failure. I'd love to contribute. Haskell projects are trivial to contribute to compared to all other languages (speaking as a Haskeller) > I’m on board with functional programming and Haskell is very interesting, but I’ve still never understood the decision. I see this take often with HNers. Have you considered that the author just likes a Haskell? That it's fun? And that without it, they wouldn't have bothered spending the time on the project period? [1] I learned enough Haskell to contribute to your average codebase as an undergrad at a state school who never learned a higher level language than C in school. Neither talent nor experience are needed. But maybe they're detriments given the interplay with ego.
- cdogl 3y ago
  > hah if you consider yourself "talented" or "experience" and can't contribute to a project you want to contribute to just because it's Haskell, you might have misevaluated yourself on those axes I was describing the Git Annex lead developer as talented and experienced, not myself.
- ithkuil 3y ago
  > speaking as a Haskeller Perhaps that's what makes it harder for you to put yourself in other people's shoes. Having to learn a new language/stack/... for a contribution (especially a "casual" contribution) objectively raises the barrier to entry. That's not the same as to say "I can't do it because it's too hard". That's more like "I have better uses of my time and I choose to contribute to something else". The net effect of many many people doing this little choice over time is that projects use too unfamiliar stacks will receive less contributions. On the other hand, the contributions they will get will be made by people who truly care and possibly that will mean the contributions will be higher quality.
  - whateveracct 3y ago
    Put yourself in an author's shoes. git annex doesn't exist in the world - but you have the idea for it. Haskell is your strongest language, and honestly you don't really like programming in other languages. So you write it in Haskell and it works and becomes a somewhat popular piece of plumbing. For projects I write at least, I'm that author. My projects wouldn't even exist if I didn't use Haskell. So thinking about optimizing for long tail contributions by anonymous users is illogical on the face of it. It sounds to me you wouldn't think any open source project should be written in Haskell for these reasons. That seems silly to me. Also - I've contributed, debugged, etc projects in many languages I barely know. It's not that hard if you're actually talented and experienced. I actually fixed a bug in a small, single-author Haskell library for my first ever project. I hadn't even finished LYAH let alone fully understand what I had read. So when a supposedly "very experienced" software engineer "cannot learn Haskell" enough to make small changes, I raise my eyebrow. You can't do better than 21 year old me with 0 professional experience and 0 formal FP education?

ErikBjare 3y ago

I love git annex so much. Solves all of the difficult parts of data archival for me (but boy, what a learning curve, even if decent at git). Joey Hess is a legend for building it.

znxster 3y ago

Internet Archive (IABAK) offer this as well. https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK/git-annex_implementation https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK/g...