4 ms·

The total collection is 54.54 TiB, with 690 torrents as of writing. For preservation purposes, I put magnet links to all the torrents here: https://pastebin.com

The total collection is 54.54 TiB, with 690 torrents as of writing. For preservation purposes, I put magnet links to all the torrents here: https://pastebin.com/zTAqS7wz https://pastebin.com/zTAqS7wz So, even if the torrents go away, the magnet links should still be usable. (Edited: fixed link)

userbinator 8y ago
What I find more amazing than the fact that some people have been so neighbourly as to share such a volume of information, is that this is still just a tiny fraction of all human knowledge. Scientific journals are certainly an important collection, but still miniscule in comparison to all the other books out there. I have collected ~20GB of automotive service manuals and related data, and that's a tiny amount too. To speak nothing of the many terabytes of entertainment others have...
- m-p-3 8y ago
  I've seen some Plex servers hosting in the amount of petabytes of content, it's crazy.
syedkarim 8y ago
Even if an entire satellite transponder (36 MHz) was used to broadcast these files (assuming 100Mbps broadcast/downlink), it would still take two months to download the whole thing.
- tim-- 8y ago
  Well, that depends on the code rate and modulation. DVB-S2X 32APSK 32/45 could do 150Mbps with the same bandwidth (as in, the same amount of radio space, not internet bandwidth).
phyzome 8y ago
Downloading all the torrent files: mkdir torrents for i in {000..689}; do curl -sS "http://gen.lib.rus.ec/scimag/repository_torrent/sm_${i}00000-${i}99999.torrent" -o "torrents/sm_${i}00000-${i}99999.torrent" -m 30; done ...which themselves take up about 75 MB, so here is the full collection of torrent files for your convenience: https://lab.brainonfire.net/drop/delete-after/20180630/torrents-20180520.tar.bz https://lab.brainonfire.net/drop/delete-after/20180630/torre... (please use the next link if possible, though!) And a torrent that is basically equivalent to that file, thanks to the below commenter: https://lab.brainonfire.net/drop/delete-after/20180630/scihub-torrents-20180520.torrent https://lab.brainonfire.net/drop/delete-after/20180630/scihu... Each torrent appears to index 100 zip files (about 800 MB each), each of which presumably contains 1000... journal articles? I don't know. The seed isn't blazing fast, so it will be a bit before I can pull out a random zip file from this randomly selected torrent and inspect it.
- nneonneo 8y ago
  I base64'd a torrent file containing the HTML index and all 690 current torrent files and stuck it here: https://pastebin.com/p1nN6veK https://pastebin.com/p1nN6veK. Would appreciate if you could upload that in a binary format somewhere. Magnet link for simplicity: magnet:?xt=urn%3Abtih%3A97fc8218775e3a1e5b90607435c0581f839e0f1d&dn=repository_torrent&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.open-internet.nl%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.skyts.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.piratepublic.com%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker1.wasabii.com.tw%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.zer0day.to%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.xku.tv%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.mg64.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2Foscar.reyesleon.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fopen.facedatabg.net%3A6969%2Fannounce&tr=udp%3A%2F%2Fmgtracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Finferno.demonoid.pw%3A3418%2Fannounce I am currently seeding this small file and will do so for as long as I am able. (Edit: updated magnet link and torrent to include trackers)
  - phyzome 8y ago
    Thanks! Uploaded as binary to https://lab.brainonfire.net/drop/delete-after/20180630/scihub-torrents-20180520.torrent https://lab.brainonfire.net/drop/delete-after/20180630/scihu... and currently seeding.
  - Faaak 8y ago
    I can't manage to load this magnet link in transmission: Error adding "magnet:?xt=urn%3Abtih%3A97fc8218775e3a1e5b90607435c0581f839e0f1d&dn=repository_torrent&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.open-internet.nl%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.skyts.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.piratepublic.com%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker1.wasabii.com.tw%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.zer0day.to%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.xku.tv%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.mg64.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2Foscar.reyesleon.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fopen.facedatabg.net%3A6969%2Fannounce&tr=udp%3A%2F%2Fmgtracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Finferno.demonoid.pw%3A3418%2Fannounce": invalid or corrupt torrent file
Iv 8y ago
Is there a way to see which are in the direst need of seeders? I am currently participating in some of them at random, as I have nowhere near 50 TiB of free space but who love to do it more efficiently.
- nneonneo 8y ago
  Possible in theory, just ping the trackers with each of the torrents and see which ones have the fewest peers.
logicallee 8y ago
???? This is obviously equal to 54,000 datasets of 1 gigabytes each, which I would say is a pretty large dataset to publish along with a paper, so it's hard for me to even imagine 54 thousand such 1-gigabyte datasets. A one gigabyte PDF is astronomical in size, PDF's are usually far far shorter. So why is this so large? Aren't they just PDF's, basically? And usually just a few pages? Excuse my ignorance. I'm very surprised at the size that you quote (54 TiB).
- aldanor 8y ago
  There’s dozens of millions of articles total, not thousands.
  - logicallee 8y ago
    wow. I didn't know there were a grand total of dozens of millions of journal articles published total worldwide. My guess would have been hundreds of thousands at most. These are journal articles, not high school book reports, right? I'm astounded at the number you quote. For example there are about 10,000 papers submitted per month to all of arxiv[1] - which is a huge number. That totals 132k per year or so. For there to be "dozens of millions" of journal articles, that would mean arxiv has just 0.83% of them. Given the very low bar to publishing on arXiv I would think if you compared the number of arxiv articles as a percentage of all pubished journal articles, it would be a lot more than 0.83%! So the "dozens of millions" of journal articles is extremely astounding to me. https://arxiv.org/stats/monthly_submissions https://arxiv.org/stats/monthly_submissions
    - nneonneo 8y ago
      arXiv only handles a small corner of scientific publications, so 0.83% doesn't sound out of whack. Most folks in biology or chemistry aren't on there, for example. A Nature study from 2014 (http://blogs.nature.com/news/2014/05/global-scientific-output-doubles-every-nine-years.html http://blogs.nature.com/news/2014/05/global-scientific-outpu...) pegs the number of papers published between 1980 and 2012, with at least one citation, at 38 million. So, 69 million publications including uncited ones and ones from outside this time range actually sounds like an underestimate - that is, the 69 million paper archive is probably missing a fair number of articles.
agumonkey 8y ago
54TiB for publications only ? do you know the total for the books too (which is more libgen than scihub). I remember a 200PiB estimate but I'd love to know more.
teamNOOB 8y ago
Do you recommend a seedbox for this? I'm currently with Ultraseedbox which is only 3TB. I am thinking of getting a new service specifically to download then provide another source/host of these files.

userbinator 8y ago

What I find more amazing than the fact that some people have been so neighbourly as to share such a volume of information, is that this is still just a tiny fraction of all human knowledge. Scientific journals are certainly an important collection, but still miniscule in comparison to all the other books out there. I have collected ~20GB of automotive service manuals and related data, and that's a tiny amount too. To speak nothing of the many terabytes of entertainment others have...

m-p-3 8y ago
I've seen some Plex servers hosting in the amount of petabytes of content, it's crazy.

syedkarim 8y ago

Even if an entire satellite transponder (36 MHz) was used to broadcast these files (assuming 100Mbps broadcast/downlink), it would still take two months to download the whole thing.

tim-- 8y ago
Well, that depends on the code rate and modulation. DVB-S2X 32APSK 32/45 could do 150Mbps with the same bandwidth (as in, the same amount of radio space, not internet bandwidth).

phyzome 8y ago

Downloading all the torrent files: mkdir torrents for i in {000..689}; do curl -sS "http://gen.lib.rus.ec/scimag/repository_torrent/sm_${i}00000-${i}99999.torrent" -o "torrents/sm_${i}00000-${i}99999.torrent" -m 30; done ...which themselves take up about 75 MB, so here is the full collection of torrent files for your convenience: https://lab.brainonfire.net/drop/delete-after/20180630/torrents-20180520.tar.bz https://lab.brainonfire.net/drop/delete-after/20180630/torre... (please use the next link if possible, though!) And a torrent that is basically equivalent to that file, thanks to the below commenter: https://lab.brainonfire.net/drop/delete-after/20180630/scihub-torrents-20180520.torrent https://lab.brainonfire.net/drop/delete-after/20180630/scihu... Each torrent appears to index 100 zip files (about 800 MB each), each of which presumably contains 1000... journal articles? I don't know. The seed isn't blazing fast, so it will be a bit before I can pull out a random zip file from this randomly selected torrent and inspect it.

nneonneo 8y ago
I base64'd a torrent file containing the HTML index and all 690 current torrent files and stuck it here: https://pastebin.com/p1nN6veK https://pastebin.com/p1nN6veK. Would appreciate if you could upload that in a binary format somewhere. Magnet link for simplicity: magnet:?xt=urn%3Abtih%3A97fc8218775e3a1e5b90607435c0581f839e0f1d&dn=repository_torrent&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.open-internet.nl%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.skyts.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.piratepublic.com%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker1.wasabii.com.tw%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.zer0day.to%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.xku.tv%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.mg64.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2Foscar.reyesleon.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fopen.facedatabg.net%3A6969%2Fannounce&tr=udp%3A%2F%2Fmgtracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Finferno.demonoid.pw%3A3418%2Fannounce I am currently seeding this small file and will do so for as long as I am able. (Edit: updated magnet link and torrent to include trackers)
- phyzome 8y ago
  Thanks! Uploaded as binary to https://lab.brainonfire.net/drop/delete-after/20180630/scihub-torrents-20180520.torrent https://lab.brainonfire.net/drop/delete-after/20180630/scihu... and currently seeding.
- Faaak 8y ago
  I can't manage to load this magnet link in transmission: Error adding "magnet:?xt=urn%3Abtih%3A97fc8218775e3a1e5b90607435c0581f839e0f1d&dn=repository_torrent&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.open-internet.nl%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.skyts.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.piratepublic.com%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker1.wasabii.com.tw%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.zer0day.to%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.xku.tv%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.mg64.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2Foscar.reyesleon.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fopen.facedatabg.net%3A6969%2Fannounce&tr=udp%3A%2F%2Fmgtracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Finferno.demonoid.pw%3A3418%2Fannounce": invalid or corrupt torrent file

Iv 8y ago

Is there a way to see which are in the direst need of seeders? I am currently participating in some of them at random, as I have nowhere near 50 TiB of free space but who love to do it more efficiently.

nneonneo 8y ago
Possible in theory, just ping the trackers with each of the torrents and see which ones have the fewest peers.

logicallee 8y ago

???? This is obviously equal to 54,000 datasets of 1 gigabytes each, which I would say is a pretty large dataset to publish along with a paper, so it's hard for me to even imagine 54 thousand such 1-gigabyte datasets. A one gigabyte PDF is astronomical in size, PDF's are usually far far shorter. So why is this so large? Aren't they just PDF's, basically? And usually just a few pages? Excuse my ignorance. I'm very surprised at the size that you quote (54 TiB).

aldanor 8y ago
There’s dozens of millions of articles total, not thousands.
- logicallee 8y ago
  wow. I didn't know there were a grand total of dozens of millions of journal articles published total worldwide. My guess would have been hundreds of thousands at most. These are journal articles, not high school book reports, right? I'm astounded at the number you quote. For example there are about 10,000 papers submitted per month to all of arxiv[1] - which is a huge number. That totals 132k per year or so. For there to be "dozens of millions" of journal articles, that would mean arxiv has just 0.83% of them. Given the very low bar to publishing on arXiv I would think if you compared the number of arxiv articles as a percentage of all pubished journal articles, it would be a lot more than 0.83%! So the "dozens of millions" of journal articles is extremely astounding to me. https://arxiv.org/stats/monthly_submissions https://arxiv.org/stats/monthly_submissions
  - nneonneo 8y ago
    arXiv only handles a small corner of scientific publications, so 0.83% doesn't sound out of whack. Most folks in biology or chemistry aren't on there, for example. A Nature study from 2014 (http://blogs.nature.com/news/2014/05/global-scientific-output-doubles-every-nine-years.html http://blogs.nature.com/news/2014/05/global-scientific-outpu...) pegs the number of papers published between 1980 and 2012, with at least one citation, at 38 million. So, 69 million publications including uncited ones and ones from outside this time range actually sounds like an underestimate - that is, the 69 million paper archive is probably missing a fair number of articles.

agumonkey 8y ago

54TiB for publications only ? do you know the total for the books too (which is more libgen than scihub). I remember a 200PiB estimate but I'd love to know more.

teamNOOB 8y ago

Do you recommend a seedbox for this? I'm currently with Ultraseedbox which is only 3TB. I am thinking of getting a new service specifically to download then provide another source/host of these files.