4 ms·

Out of interest, is sci-hub generally available to get a copy of for...research purposes?

by bane 5y ago

cinntaile 5y ago
There are torrents available. I'm unsure if I can link to the site directly without getting banned so you'll have to check the Gizmodo article for the link to the site containing the torrents. [0] It's missing some of the 2021 papers though, judging by the dates. [0] https://gizmodo.com/archivists-want-to-make-sci-hub-un-censorable-1846898276 https://gizmodo.com/archivists-want-to-make-sci-hub-un-censo...
AlexCoventry 5y ago
Taking you at your word that you're just asking out of interest, I hear (but haven't verified) that libgen mirrors scihub. http://libgen.rs/dbdumps/ http://libgen.rs/dbdumps/
throwaway879080 5y ago
see this reddit thread on Scihub rescue mission https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_mission_for_scihub_and_open_science_we_are/ https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_... there is a link there to 851 torrents , ~77TB of data (compressed zip files), and also an index (sql db dump)
- Frost1x 5y ago
  77 TB of compressed publications. That's pretty crazy to think about, even if you consider overhead and multimedia assets like images often included in PDFs. I remember the old "the library of congress fits on this CDROM" analogies (which wasn't entirely true) but this takes it to a whole new level. At some point, it seems like in research, it will be far easier to skip the lit review and just do the work then later if you do the same work someone else did, compare results for consistency. We may yet get through the hurdle of the reproducibility crisis due to the deluge of information. The underlying issue though is, because you couldn't find the related effort that caused you to independently attempt to replicate the effort, you may also never find all the duplicate efforts to compare for consistency.
  - throwaway879080 5y ago
    about half a petabyte uncompressed, i'm pondering training a language model/embeddings on this treasure trove, immediate use cases - search, related/similar papers recommendations, medical informatics. it does require OCRing the entire collection first though
  - musicale 5y ago
    > 77 TB of compressed publications. PDF is fairly inefficient compared to formats like DVI (and consider that so many papers are produced using TeX anyway, though figures may be in various image formats.) But 77TB? You could host the whole thing in a shoebox with nine 8TB flash drives, a 10-socket powered USB hub, and a Raspberry Pi. Someone really needs to build Shoe-Hub.
vmception 5y ago
also, I want a version that lets me browse papers. Unlike people in academia or other research fields, I don't know the paper that I want to pirate yet. from my lived experience, this request has been seen as absurd and invalidated by many because their lived experience is always having feeds of papers and abstracts they can start from. or how they see a browser extension that doesn’t do what I’m asking at all to be good enough (there’s some that automatically give you the scihub link to any paper you are looking at, which still puts me at square one, which paper?). its a pretty basic feature though.
- abcc8 5y ago
  Unpaywall is a browser extension you might find useful. It provides links to open access papers, with the links appearing on pubmed pages, publisher pages, etc. A significant proportion of papers are now open access so this tool is very often useful.

Out of interest, is sci-hub generally available to get a copy of for...research purposes?

by bane 5y ago

Out of interest, is sci-hub generally available to get a copy of for...research purposes?

There are torrents available. I'm unsure if I can link to the site directly without getting banned so you'll have to check the Gizmodo article for the link to the site containing the torrents. [0] It's missing some of the 2021 papers though, judging by the dates. [0] https://gizmodo.com/archivists-want-to-make-sci-hub-un-censorable-1846898276 https://gizmodo.com/archivists-want-to-make-sci-hub-un-censo...

AlexCoventry 5y ago

Taking you at your word that you're just asking out of interest, I hear (but haven't verified) that libgen mirrors scihub. http://libgen.rs/dbdumps/ http://libgen.rs/dbdumps/

throwaway879080 5y ago

see this reddit thread on Scihub rescue mission https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_mission_for_scihub_and_open_science_we_are/ https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_... there is a link there to 851 torrents , ~77TB of data (compressed zip files), and also an index (sql db dump)

Frost1x 5y ago
77 TB of compressed publications. That's pretty crazy to think about, even if you consider overhead and multimedia assets like images often included in PDFs. I remember the old "the library of congress fits on this CDROM" analogies (which wasn't entirely true) but this takes it to a whole new level. At some point, it seems like in research, it will be far easier to skip the lit review and just do the work then later if you do the same work someone else did, compare results for consistency. We may yet get through the hurdle of the reproducibility crisis due to the deluge of information. The underlying issue though is, because you couldn't find the related effort that caused you to independently attempt to replicate the effort, you may also never find all the duplicate efforts to compare for consistency.
- throwaway879080 5y ago
  about half a petabyte uncompressed, i'm pondering training a language model/embeddings on this treasure trove, immediate use cases - search, related/similar papers recommendations, medical informatics. it does require OCRing the entire collection first though
- musicale 5y ago
  > 77 TB of compressed publications. PDF is fairly inefficient compared to formats like DVI (and consider that so many papers are produced using TeX anyway, though figures may be in various image formats.) But 77TB? You could host the whole thing in a shoebox with nine 8TB flash drives, a 10-socket powered USB hub, and a Raspberry Pi. Someone really needs to build Shoe-Hub.

vmception 5y ago

also, I want a version that lets me browse papers. Unlike people in academia or other research fields, I don't know the paper that I want to pirate yet. from my lived experience, this request has been seen as absurd and invalidated by many because their lived experience is always having feeds of papers and abstracts they can start from. or how they see a browser extension that doesn’t do what I’m asking at all to be good enough (there’s some that automatically give you the scihub link to any paper you are looking at, which still puts me at square one, which paper?). its a pretty basic feature though.

abcc8 5y ago
Unpaywall is a browser extension you might find useful. It provides links to open access papers, with the links appearing on pubmed pages, publisher pages, etc. A significant proportion of papers are now open access so this tool is very often useful.