5 ms·

Does Sci-Hub practice what it preaches? Do they provide a means of downloading their entire repository of research papers either via torrent or SFTP? Because if

Does Sci-Hub practice what it preaches? Do they provide a means of downloading their entire repository of research papers either via torrent or SFTP? Because if the entire repository isn't available to everyone, then how can we be sure that once they get raided those papers aren't lost forever?

erikpukinskis 11y ago
Why is that important? I would be just as happy with an API that allows downloading articles one at a time.
- edraferi 11y ago
  The parent comment wants to efficiently mirror the content to prevent it from disappearing in the event of a successful take down operation. This can be done by scraping an API, but bulk methods are faster and less likely to degrade service for other users.
  - xvilka 11y ago
    May be even using something like DHT on top of WebRTC could allow to upload/download those papers easily.
    - femto 11y ago
      There's a thumping amount of data there. 485 torrents, each torrent containing 100 zips of around 500MB each, each zip containing 1000 papers. Total = 500MB * 100 * 485 ~ 25TB. If your have a 10Mbit/s link into your house, it's 280 days of continuous downloading. New torrents seem to be appearing at the rate of 14 per month = 700GB/month = 2.7Mbit/s. For the majority of people, I'd say it is beyond any one person being able to mirror the lot. It seems like an application for a Freenet type system, but one where the files are guaranteed to never be deleted, even if they are rarely accessed.
      - harperlee 11y ago
        I would assume that most of that is due to PDFs weigthing a lot more than their content. If someone wants to donate IT skills and coding time (this is a thing that I've read several times in HN), perhaps it would be great that someone puts together a system that takes a PDF, OCRs it when necessary, detects images and graphs, and uses all of that to create a light version of the document (html/latex/markdown/whatever); and then puts all of those TB through that tool. The reduction in storage and network costs might be a lot more than a standard donation!
      - kwhitefoot 11y ago
        If enough people keep the torrents alive no one person would need to mirror all of it, certainly not all at once.
DarkLinkXXXX 11y ago
What you describe is available through libgens scimag section, and I believe they're affiliated.
- kanzure 11y ago
  libgen scimag collection is not included in their torrents, nor usenet downloads, etc. Could someone ask them to fix this? thanks edit: oh http://libgen.io/scimag/repository_torrent_notforall/ http://libgen.io/scimag/repository_torrent_notforall/
acveilleux 11y ago
SCi-Hub used hacked or shared university account to abuse the university proxies to get content. For big research institutions, it's an annoyance. To small colleges and university library it is a major headache that can get the whole university locked out from resources the library is paying dearly for. Source: My partner is a librarian at a smaller university, this happens every few months and they lose access to major online platforms for days if not longer every time.
- anewhnaccount 11y ago
  I feel like you're blaming the wrong party here.
  - _Codemonkeyism 11y ago
    The "blaming" is in your head. People who explain how things happen do not "blame".
    - conceit 11y ago
      Don't talk bullshit. There clearly is blame put on the proxies > it [the proxy] is a major headache that can get the whole university locked out Whether either the practice of lock out should be blamed for immoral licensing or the proxy servers blamed for immoral illicit access, is not properly explained here.
      - _Codemonkeyism 11y ago
        Don't talk bullshit. Just because you think in categories of 'blame' doesn't mean everyone else does. For some people there is a difference between causality and guilt.
ojii 11y ago
They apparently mirror everything to LibGen. I agree it would be nice to be able to easily mirror it though.
healer 11y ago
This is an important point that is not getting adequate discussion right now. A couple of days ago, the sight was unavailable for days both on the clear-net and in the dark-web. The sight resurfaced two days ago with two levels of captcha before anyone can search the site. What does this imply? My guess is that the site went down under an automated DDOS. A distributed archive will ameliorate this sort of situation.

erikpukinskis 11y ago

Why is that important? I would be just as happy with an API that allows downloading articles one at a time.

edraferi 11y ago
The parent comment wants to efficiently mirror the content to prevent it from disappearing in the event of a successful take down operation. This can be done by scraping an API, but bulk methods are faster and less likely to degrade service for other users.
- xvilka 11y ago
  May be even using something like DHT on top of WebRTC could allow to upload/download those papers easily.
  - femto 11y ago
    There's a thumping amount of data there. 485 torrents, each torrent containing 100 zips of around 500MB each, each zip containing 1000 papers. Total = 500MB * 100 * 485 ~ 25TB. If your have a 10Mbit/s link into your house, it's 280 days of continuous downloading. New torrents seem to be appearing at the rate of 14 per month = 700GB/month = 2.7Mbit/s. For the majority of people, I'd say it is beyond any one person being able to mirror the lot. It seems like an application for a Freenet type system, but one where the files are guaranteed to never be deleted, even if they are rarely accessed.
    - harperlee 11y ago
      I would assume that most of that is due to PDFs weigthing a lot more than their content. If someone wants to donate IT skills and coding time (this is a thing that I've read several times in HN), perhaps it would be great that someone puts together a system that takes a PDF, OCRs it when necessary, detects images and graphs, and uses all of that to create a light version of the document (html/latex/markdown/whatever); and then puts all of those TB through that tool. The reduction in storage and network costs might be a lot more than a standard donation!
    - kwhitefoot 11y ago
      If enough people keep the torrents alive no one person would need to mirror all of it, certainly not all at once.

DarkLinkXXXX 11y ago

What you describe is available through libgens scimag section, and I believe they're affiliated.

kanzure 11y ago
libgen scimag collection is not included in their torrents, nor usenet downloads, etc. Could someone ask them to fix this? thanks edit: oh http://libgen.io/scimag/repository_torrent_notforall/ http://libgen.io/scimag/repository_torrent_notforall/

acveilleux 11y ago

SCi-Hub used hacked or shared university account to abuse the university proxies to get content. For big research institutions, it's an annoyance. To small colleges and university library it is a major headache that can get the whole university locked out from resources the library is paying dearly for. Source: My partner is a librarian at a smaller university, this happens every few months and they lose access to major online platforms for days if not longer every time.

anewhnaccount 11y ago
I feel like you're blaming the wrong party here.
- _Codemonkeyism 11y ago
  The "blaming" is in your head. People who explain how things happen do not "blame".
  - conceit 11y ago
    Don't talk bullshit. There clearly is blame put on the proxies > it [the proxy] is a major headache that can get the whole university locked out Whether either the practice of lock out should be blamed for immoral licensing or the proxy servers blamed for immoral illicit access, is not properly explained here.
    - _Codemonkeyism 11y ago
      Don't talk bullshit. Just because you think in categories of 'blame' doesn't mean everyone else does. For some people there is a difference between causality and guilt.

ojii 11y ago

They apparently mirror everything to LibGen. I agree it would be nice to be able to easily mirror it though.

healer 11y ago

This is an important point that is not getting adequate discussion right now. A couple of days ago, the sight was unavailable for days both on the clear-net and in the dark-web. The sight resurfaced two days ago with two levels of captcha before anyone can search the site. What does this imply? My guess is that the site went down under an automated DDOS. A distributed archive will ameliorate this sort of situation.