5 ms·

For those of us datahoarders, how large is sci hub? Surely it compresses extremely well considering it's all human readable text? I would love to mirror it via

by hak8or 8y ago

For those of us datahoarders, how large is sci hub? Surely it compresses extremely well considering it's all human readable text? I would love to mirror it via a torrent or on ipfs and whatnot, and I have a gigabit connection with roughly 80TB of space I can throw at this.

radus 8y ago
It's mostly composed of PDFs actually. Judging from my own library the average publication is about 2 MB, and I believe about 1e8 publications are currently indexed, so it's on the order of hundreds of TB.
- emilfihlman 8y ago
  Either a) those pdfs contain images or other graphics that don't compress easily or b) the text is not compressed.
  - radus 8y ago
    There are definitely many images and graphics. I don't know if my sample is representative of most literature, but I don't think it's too far off..
  - gpm 8y ago
    Or c) The text is terribly compressed. Probably a lot of the "text" is also a scan of text as well. Could be an interesting project to try and make a sci-hub optimized compression system.
    - Quiark 8y ago
      DejaVu format
- Reelin 8y ago
  This got me curious. For reference, my Zotero library is predominantly biomedical in nature. This device currently has 810 of the pdf files pulled down to the local drive, of which the vast majority are journal articles (a very few of the largest ones are presentation slide decks). Size Range Count [0, 0.5) 255 (31%) [0.5, 1) 179 (22%) [1, 1.5) 108 (13%) [1.5, 2) 69 (9%) [2, 3) 80 (10%) [3, 10) 95 (12%) > 10 24 (3%) That's ~50% < 1 MB and ~25% > 2 MB. I'm actually a bit surprised they're that small.
- agumonkey 8y ago
  btw anyone knows if there are PDF shrinkers ? some journals can weight a lot (scanned pages) even though there's potential to recompress images while keeping it readable
  - harryposner 8y ago
    I've used Ghostscript to shrink scanned PDFs for emailing, although I don't know how lossy the compression is. [0] https://askubuntu.com/a/256449 https://askubuntu.com/a/256449
    - agumonkey 8y ago
      I tried it but it was cpu intensive for very small improvements
frgtpsswrdlame 8y ago
Much (but I believe not all? I'd have to look more into it) is backed up on libgen: http://gen.lib.rus.ec/repository_torrent/ http://gen.lib.rus.ec/repository_torrent/ There's also the very bottom file here: http://download.library1.org/dbdumps/ http://download.library1.org/dbdumps/
- votepaunchy 8y ago
  Are SciHub torrents not here: http://gen.lib.rus.ec/scimag/repository_torrent/ http://gen.lib.rus.ec/scimag/repository_torrent/ I think your link may point to the LibGen torrents (books, mainly, I believe).
  - frgtpsswrdlame 8y ago
    Yep, you're right!
  - mirimir 8y ago
    I looked at a few of the torrents, and they're very roughly 150 GB each. So that's about 50 TB overall.
    - jacquesm 8y ago
      More like 70T.
      - mirimir 8y ago
        Thanks. I was too lazy to look at them all.
deleted 8y ago
[deleted]
jacquesm 8y ago
Currently 70T growing at about 150G every two weeks or so.
- Tepix 8y ago
  That means if you have access to a LTO-6 tape drive (best price per TB right now at around 8,88€ per TB) it only costs around 622€ ($700) to create a backup.

by hak8or 8y ago

radus 8y ago

It's mostly composed of PDFs actually. Judging from my own library the average publication is about 2 MB, and I believe about 1e8 publications are currently indexed, so it's on the order of hundreds of TB.

emilfihlman 8y ago
Either a) those pdfs contain images or other graphics that don't compress easily or b) the text is not compressed.
- radus 8y ago
  There are definitely many images and graphics. I don't know if my sample is representative of most literature, but I don't think it's too far off..
- gpm 8y ago
  Or c) The text is terribly compressed. Probably a lot of the "text" is also a scan of text as well. Could be an interesting project to try and make a sci-hub optimized compression system.
  - Quiark 8y ago
    DejaVu format
Reelin 8y ago
This got me curious. For reference, my Zotero library is predominantly biomedical in nature. This device currently has 810 of the pdf files pulled down to the local drive, of which the vast majority are journal articles (a very few of the largest ones are presentation slide decks). Size Range Count [0, 0.5) 255 (31%) [0.5, 1) 179 (22%) [1, 1.5) 108 (13%) [1.5, 2) 69 (9%) [2, 3) 80 (10%) [3, 10) 95 (12%) > 10 24 (3%) That's ~50% < 1 MB and ~25% > 2 MB. I'm actually a bit surprised they're that small.
agumonkey 8y ago
btw anyone knows if there are PDF shrinkers ? some journals can weight a lot (scanned pages) even though there's potential to recompress images while keeping it readable
- harryposner 8y ago
  I've used Ghostscript to shrink scanned PDFs for emailing, although I don't know how lossy the compression is. [0] https://askubuntu.com/a/256449 https://askubuntu.com/a/256449
  - agumonkey 8y ago
    I tried it but it was cpu intensive for very small improvements

frgtpsswrdlame 8y ago

Much (but I believe not all? I'd have to look more into it) is backed up on libgen: http://gen.lib.rus.ec/repository_torrent/ http://gen.lib.rus.ec/repository_torrent/ There's also the very bottom file here: http://download.library1.org/dbdumps/ http://download.library1.org/dbdumps/

votepaunchy 8y ago
Are SciHub torrents not here: http://gen.lib.rus.ec/scimag/repository_torrent/ http://gen.lib.rus.ec/scimag/repository_torrent/ I think your link may point to the LibGen torrents (books, mainly, I believe).
- frgtpsswrdlame 8y ago
  Yep, you're right!
- mirimir 8y ago
  I looked at a few of the torrents, and they're very roughly 150 GB each. So that's about 50 TB overall.
  - jacquesm 8y ago
    More like 70T.
    - mirimir 8y ago
      Thanks. I was too lazy to look at them all.

deleted 8y ago

[deleted]

jacquesm 8y ago

Currently 70T growing at about 150G every two weeks or so.

Tepix 8y ago
That means if you have access to a LTO-6 tape drive (best price per TB right now at around 8,88€ per TB) it only costs around 622€ ($700) to create a backup.