9 ms·

Show HN: Effort to clone unmaintained SourceForge projects to GitHub

146 points by hydragit 11y ago

lcswi 11y ago
Nice! But in my opinion better help archiveteam with their efforts!
- hydragit 11y ago
  I confess my ignorance regarding archive.org's various collections. There seem to be a lot of them, which one are you referring to?
  - danieloaks 11y ago
    They're not a part of archive.org (just a totally separate group with similar interests, lead by Jason Scott). The specific project page is here: http://archiveteam.org/index.php?title=SourceForge http://archiveteam.org/index.php?title=SourceForge The best place to pop in is probably on IRC. For the Sourceforge project it's #coldstorage on EFnet, http://chat.efnet.org:9090/?nick=&channels=%23coldstorage&Login=Login http://chat.efnet.org:9090/?nick=&channels=%23coldstorage&Lo... for the web client. Though note, the ArchiveTeam project seems to be paused right now.
    - bentpins 11y ago
      To add to this - ArchiveTeam often works with archive.org who have arranged long term storage for a lot of retrieved content.
  - voltagex_ 11y ago
    See the response just above yours - ArchiveTeam is separate from archive.org
- josso 11y ago
  Definitely. And if you want to help out visit [1]. There’s options to download binaries[2] as well as source code[3] from SourceForge. [1]: http://archiveteam.org/index.php?title=SourceForge http://archiveteam.org/index.php?title=SourceForge [2]: https://github.com/ArchiveTeam/sourceforge-grab https://github.com/ArchiveTeam/sourceforge-grab [3]: https://github.com/ArchiveTeam/sourceforge-grab-rsync https://github.com/ArchiveTeam/sourceforge-grab-rsync
- coliveira 11y ago
  Agreed. This article sounds more like an advertisement for GH. Also, why not using other platforms like bitbucket? Centralizing everything at GH is the worst scenario for open source.
  - shiggerino 11y ago
    Bitbucket is exactly the same. Gitlab has a community edition, and then there's Gogs, Kallithea and a few others.
pavlov 11y ago
Why Github? Copying from one commercial provider to another doesn't solve the fundamental problem. Using git helps, but most of those old repos will never get cloned. In 10 years time, Github may be the tired old service that gets acquired by a hedge fund that decides to monetize their repos. Such things are part of the corporate lifecycle.
- Klathmon 11y ago
  Do you have any other suggestions? Hosting these repos on donated/personal machines is (IMO) significantly less likely to stand the test of time. At least with a commercial entity there is a bit more "trust" involved that they won't disappear out of the blue one day. And if the time comes that Github starts to collapse, the process can be repeated. Just because something isn't permanent doesn't mean it's pointless.
  - wslh 11y ago
    > Do you have any other suggestions? Archive.org ? And weird that nobody suggested the Bitcoin block chain. I don't think binaries are a good fit but source code doesn't require a lot of space. With the current and future block size it will take sometime to make it happen.
    - joshstrange 11y ago
      Archive.org (while an amazing resource created lovingly by amaing people) is not a great front-end for stuff like this. Github is very easy to get started with and excels at code hosting. As for the blockchain that's a terrible idea, there are so many things wrong with it including the cost to push all of that data into the blockchain and the fact that while source code can be small it isn't always and it's magnitudes of times larger. Right now each block is 1MB and blocks take some 10 minutes for just 1 confirmation so you are looking at < 1.7KBpss (13.6Kbps) "upload" speeds. IF you actually attempted this you would have to have some sort of header on each transaction to tie it all together which lowers the speed even further. I'd bet money that if you started uploading nodes would either ban you or the core devs would do something to stop the chain from being filled with shit that now has to get replicated to 10's of thousands of machines across the globe.
Osmium 11y ago
Honestly, this is a serious issue for my field. There are so many obscure academic binaries hosted on SF... I hope someone manages to mirror them. [The fact that a lot of the scientific community is so backwards in adopting modern coding standards is another conversation for another day.]
ksherlock 11y ago
That's great. I started doing that myself (my own git server, not github) for some projects I care about. This effort seems include a very narrow list, though. For CVS, though, I suspect cvs-fast-export [1] will do a better job than git-cvsimport. 1 http://www.catb.org/esr/cvs-fast-export/ http://www.catb.org/esr/cvs-fast-export/
- hydragit 11y ago
  Thanks, I'll have a look
TazeTSchnitzel 11y ago
Why aren't you mirroring the binaries? These are vital for people in the future who do not have the time to set up a build environment for software from a decade ago. I'd also echo the concerns of others about GitHub. Proper archivists should do for SourceForge what they did for other projects. Archive Team, maybe? Looks like they have a wiki page: http://www.archiveteam.org/index.php?title=SourceForge http://www.archiveteam.org/index.php?title=SourceForge
- hydragit 11y ago
  Regarding binaries, I know these could be useful and I'd like to provide them, but I'm afraid some "not (yet?) very popular mirroring project" can't show how we can trust it regarding binaries. After all, a known site like SF is untrustable, so why would an unknown site would be more?
  - caseysoftware 11y ago
    Yes, this is a more challenging and potentially risky one. I think you're taking the right approach by capturing the code and the history. In fact, I think you're going above and beyond what most people should ask for or expect.
    - _wmd 11y ago
      Seems you could just side-step SF directly in this case and contact one of their mirrors: http://sourceforge.net/p/forge/documentation/Join%20as%20a%20Mirror/ http://sourceforge.net/p/forge/documentation/Join%20as%20a%2... http://sourceforge.net/p/forge/documentation/Mirrors/ http://sourceforge.net/p/forge/documentation/Mirrors/ I bet at least one techie working at one of those organisations would lend a sympathetic ear to the effort, if you could find them edit: running 'rsync -r' on a local mirror shows 512,000 directories from a..ju, but only 43k files. Mirroring all the downloads should be easy
- bentpins 11y ago
  This was in progress, 830GB was downloaded before a Sourceforge guy popped onto the IRC and said he's ok with the archiving, but that the robots.txt should be respected. This would put things at a practical standstill. So the downloading was paused, I'm not really sure what's happened in the week since. Right now Xfire's videos, several URL shortners' links, and Toshiba Support material are being archived. If you have spare cycles and bandwidth, and want to contribute, running an instance of the "ArchiveTeam Warrior" is pretty easy through docker or a VM. http://archiveteam.org/index.php?title=Warrior http://archiveteam.org/index.php?title=Warrior
coliveira 11y ago
This seems much more like a temporary fix, not really a solution. A few years from now GitHub can do the same thing that sf did. This after all seems to be the fate of commercial companies that explore open source, once they start to lose users to new competitors.
- deleted 11y ago
  [deleted]
estrabd 11y ago
Sourceforge is on the radar here, but maybe it's time to step it up. http://www.archiveteam.org/index.php?title=Fire_Drill http://www.archiveteam.org/index.php?title=Fire_Drill Update: seems others have linked to archiveteam.org, so maybe that's the best route. Is the OP part of the AT effort or do they know about each other? Maybe they should.
jmkni 11y ago
Nice. I agree with what the others are saying, there's a lot of source code for solving obscure programs that is only on Sourceforge. One example I found recently is a program called QLumEdit. I recently had to figure out how to work with EuLumdat files, and if it wasn't for the source code for this program on Sourceforge I would have been completely stumped (well not quite, but it would have taken me ages). If SF goes down the toilet, a lot of knowledge goes with it so this is awesome to see! If anybody is interested, I was converting this code from C++ to .net, my horrible hacky unrefactored effort is here - https://github.com/bumblebeeman/eulum.net https://github.com/bumblebeeman/eulum.net I am planning to make this code nicer, and develop it into a WPF app when I have time! I am getting pretty close too, here is my .net generated version of the images this program produces: http://imgur.com/PCmpnJ2 http://imgur.com/PCmpnJ2
jaytaylor 11y ago
What about creating a torrent containing all these unmaintained SF projects (with binary downloads included)? This would dramatically increase the odds that the content is never lost.
- OBAMA_THE_NOOB 11y ago
  +1
- nextweek2 11y ago
  The problem with torrents is the lack of incremental update support. If the base torrent gets updated it gets a new hash identifier. How do you know its been changed to ensure you get the latest version. When you do the swarm effectively gets diluted because some are on the new architecture and some are on older versions.
  - jaytaylor 11y ago
    If the SF projects aren't being updated, then what's the issue? The information is, by definition, static.
egsec 11y ago
Note 1: Moving things to GitHub or elsewhere does not remove them from SourceForge. So SF can continue to host and enjoy links on unmaintained websites, search engines etc. Note 2: If their business model is offering popular binaries and source, they can just copy these from other sites and repackage them. Open source software allows you to do this. If no one else is interesting in bundling and monetizing, then they can buy traffic and still succeed. Note 3: Remember that academy award winning movie from 1943? Not so great it today's light. While perhaps one of the goals of the Internet and cheap storage is to keep a copy of everything, and its often better to not re-invent the wheel, if something fall by the wayside, and its needed, it will be created. Note 4: There are plenty of websites which catalog useful abandonware, that someone had to find a physical disk drive from. If the software has value, chances are someone will eventually repost it somewhere without a massive organized effort. ---- There is clearly value in moving over some project to GitHub or elsewhere, but if some things are not migrated or moved life will go on.
- GFK_of_xmaspast 11y ago
  Per the historical record, that "academy award winning movie from 1943" was 'Casablanca'.
frik 11y ago
> Currently, for each cloned project, we mirror its CVS repository and its website. Please add "SVN" (Subversion)

146 points by hydragit 11y ago

lcswi 11y ago

Nice! But in my opinion better help archiveteam with their efforts!

hydragit 11y ago
I confess my ignorance regarding archive.org's various collections. There seem to be a lot of them, which one are you referring to?
- danieloaks 11y ago
  They're not a part of archive.org (just a totally separate group with similar interests, lead by Jason Scott). The specific project page is here: http://archiveteam.org/index.php?title=SourceForge http://archiveteam.org/index.php?title=SourceForge The best place to pop in is probably on IRC. For the Sourceforge project it's #coldstorage on EFnet, http://chat.efnet.org:9090/?nick=&channels=%23coldstorage&Login=Login http://chat.efnet.org:9090/?nick=&channels=%23coldstorage&Lo... for the web client. Though note, the ArchiveTeam project seems to be paused right now.
  - bentpins 11y ago
    To add to this - ArchiveTeam often works with archive.org who have arranged long term storage for a lot of retrieved content.
- voltagex_ 11y ago
  See the response just above yours - ArchiveTeam is separate from archive.org
josso 11y ago
Definitely. And if you want to help out visit [1]. There’s options to download binaries[2] as well as source code[3] from SourceForge. [1]: http://archiveteam.org/index.php?title=SourceForge http://archiveteam.org/index.php?title=SourceForge [2]: https://github.com/ArchiveTeam/sourceforge-grab https://github.com/ArchiveTeam/sourceforge-grab [3]: https://github.com/ArchiveTeam/sourceforge-grab-rsync https://github.com/ArchiveTeam/sourceforge-grab-rsync
coliveira 11y ago
Agreed. This article sounds more like an advertisement for GH. Also, why not using other platforms like bitbucket? Centralizing everything at GH is the worst scenario for open source.
- shiggerino 11y ago
  Bitbucket is exactly the same. Gitlab has a community edition, and then there's Gogs, Kallithea and a few others.

pavlov 11y ago

Why Github? Copying from one commercial provider to another doesn't solve the fundamental problem. Using git helps, but most of those old repos will never get cloned. In 10 years time, Github may be the tired old service that gets acquired by a hedge fund that decides to monetize their repos. Such things are part of the corporate lifecycle.

Klathmon 11y ago
Do you have any other suggestions? Hosting these repos on donated/personal machines is (IMO) significantly less likely to stand the test of time. At least with a commercial entity there is a bit more "trust" involved that they won't disappear out of the blue one day. And if the time comes that Github starts to collapse, the process can be repeated. Just because something isn't permanent doesn't mean it's pointless.
- wslh 11y ago
  > Do you have any other suggestions? Archive.org ? And weird that nobody suggested the Bitcoin block chain. I don't think binaries are a good fit but source code doesn't require a lot of space. With the current and future block size it will take sometime to make it happen.
  - joshstrange 11y ago
    Archive.org (while an amazing resource created lovingly by amaing people) is not a great front-end for stuff like this. Github is very easy to get started with and excels at code hosting. As for the blockchain that's a terrible idea, there are so many things wrong with it including the cost to push all of that data into the blockchain and the fact that while source code can be small it isn't always and it's magnitudes of times larger. Right now each block is 1MB and blocks take some 10 minutes for just 1 confirmation so you are looking at < 1.7KBpss (13.6Kbps) "upload" speeds. IF you actually attempted this you would have to have some sort of header on each transaction to tie it all together which lowers the speed even further. I'd bet money that if you started uploading nodes would either ban you or the core devs would do something to stop the chain from being filled with shit that now has to get replicated to 10's of thousands of machines across the globe.

Osmium 11y ago

Honestly, this is a serious issue for my field. There are so many obscure academic binaries hosted on SF... I hope someone manages to mirror them. [The fact that a lot of the scientific community is so backwards in adopting modern coding standards is another conversation for another day.]

ksherlock 11y ago

That's great. I started doing that myself (my own git server, not github) for some projects I care about. This effort seems include a very narrow list, though. For CVS, though, I suspect cvs-fast-export [1] will do a better job than git-cvsimport. 1 http://www.catb.org/esr/cvs-fast-export/ http://www.catb.org/esr/cvs-fast-export/

hydragit 11y ago
Thanks, I'll have a look

TazeTSchnitzel 11y ago

Why aren't you mirroring the binaries? These are vital for people in the future who do not have the time to set up a build environment for software from a decade ago. I'd also echo the concerns of others about GitHub. Proper archivists should do for SourceForge what they did for other projects. Archive Team, maybe? Looks like they have a wiki page: http://www.archiveteam.org/index.php?title=SourceForge http://www.archiveteam.org/index.php?title=SourceForge

hydragit 11y ago
Regarding binaries, I know these could be useful and I'd like to provide them, but I'm afraid some "not (yet?) very popular mirroring project" can't show how we can trust it regarding binaries. After all, a known site like SF is untrustable, so why would an unknown site would be more?
- caseysoftware 11y ago
  Yes, this is a more challenging and potentially risky one. I think you're taking the right approach by capturing the code and the history. In fact, I think you're going above and beyond what most people should ask for or expect.
  - _wmd 11y ago
    Seems you could just side-step SF directly in this case and contact one of their mirrors: http://sourceforge.net/p/forge/documentation/Join%20as%20a%20Mirror/ http://sourceforge.net/p/forge/documentation/Join%20as%20a%2... http://sourceforge.net/p/forge/documentation/Mirrors/ http://sourceforge.net/p/forge/documentation/Mirrors/ I bet at least one techie working at one of those organisations would lend a sympathetic ear to the effort, if you could find them edit: running 'rsync -r' on a local mirror shows 512,000 directories from a..ju, but only 43k files. Mirroring all the downloads should be easy
bentpins 11y ago
This was in progress, 830GB was downloaded before a Sourceforge guy popped onto the IRC and said he's ok with the archiving, but that the robots.txt should be respected. This would put things at a practical standstill. So the downloading was paused, I'm not really sure what's happened in the week since. Right now Xfire's videos, several URL shortners' links, and Toshiba Support material are being archived. If you have spare cycles and bandwidth, and want to contribute, running an instance of the "ArchiveTeam Warrior" is pretty easy through docker or a VM. http://archiveteam.org/index.php?title=Warrior http://archiveteam.org/index.php?title=Warrior

coliveira 11y ago

This seems much more like a temporary fix, not really a solution. A few years from now GitHub can do the same thing that sf did. This after all seems to be the fate of commercial companies that explore open source, once they start to lose users to new competitors.

deleted 11y ago
[deleted]

estrabd 11y ago

Sourceforge is on the radar here, but maybe it's time to step it up. http://www.archiveteam.org/index.php?title=Fire_Drill http://www.archiveteam.org/index.php?title=Fire_Drill Update: seems others have linked to archiveteam.org, so maybe that's the best route. Is the OP part of the AT effort or do they know about each other? Maybe they should.

jmkni 11y ago

Nice. I agree with what the others are saying, there's a lot of source code for solving obscure programs that is only on Sourceforge. One example I found recently is a program called QLumEdit. I recently had to figure out how to work with EuLumdat files, and if it wasn't for the source code for this program on Sourceforge I would have been completely stumped (well not quite, but it would have taken me ages). If SF goes down the toilet, a lot of knowledge goes with it so this is awesome to see! If anybody is interested, I was converting this code from C++ to .net, my horrible hacky unrefactored effort is here - https://github.com/bumblebeeman/eulum.net https://github.com/bumblebeeman/eulum.net I am planning to make this code nicer, and develop it into a WPF app when I have time! I am getting pretty close too, here is my .net generated version of the images this program produces: http://imgur.com/PCmpnJ2 http://imgur.com/PCmpnJ2

jaytaylor 11y ago

What about creating a torrent containing all these unmaintained SF projects (with binary downloads included)? This would dramatically increase the odds that the content is never lost.

OBAMA_THE_NOOB 11y ago
+1
nextweek2 11y ago
The problem with torrents is the lack of incremental update support. If the base torrent gets updated it gets a new hash identifier. How do you know its been changed to ensure you get the latest version. When you do the swarm effectively gets diluted because some are on the new architecture and some are on older versions.
- jaytaylor 11y ago
  If the SF projects aren't being updated, then what's the issue? The information is, by definition, static.

egsec 11y ago

Note 1: Moving things to GitHub or elsewhere does not remove them from SourceForge. So SF can continue to host and enjoy links on unmaintained websites, search engines etc. Note 2: If their business model is offering popular binaries and source, they can just copy these from other sites and repackage them. Open source software allows you to do this. If no one else is interesting in bundling and monetizing, then they can buy traffic and still succeed. Note 3: Remember that academy award winning movie from 1943? Not so great it today's light. While perhaps one of the goals of the Internet and cheap storage is to keep a copy of everything, and its often better to not re-invent the wheel, if something fall by the wayside, and its needed, it will be created. Note 4: There are plenty of websites which catalog useful abandonware, that someone had to find a physical disk drive from. If the software has value, chances are someone will eventually repost it somewhere without a massive organized effort. ---- There is clearly value in moving over some project to GitHub or elsewhere, but if some things are not migrated or moved life will go on.

GFK_of_xmaspast 11y ago
Per the historical record, that "academy award winning movie from 1943" was 'Casablanca'.

frik 11y ago

> Currently, for each cloned project, we mirror its CVS repository and its website. Please add "SVN" (Subversion)