15 ms·

I've seen estimates of the total sci-hub/libgen size at anywhere from 20TB to 75+TB. Probably there are lots of dupes in there, and a dedicated dedup effort co

by na85 5y ago

I've seen estimates of the total sci-hub/libgen size at anywhere from 20TB to 75+TB. Probably there are lots of dupes in there, and a dedicated dedup effort could reduce that size by a bit. But taking the upper end: A WD Black 10 TB platter drive[0] costs $329, which means it would take only $2700 or so to have enough storage space to archive what's basically the sum total of human scientific knowledge. [0] - https://shop.westerndigital.com/products/internal-drives/wd-black-desktop-sata-hdd#WD101FZBX https://shop.westerndigital.com/products/internal-drives/wd-...

CheezeIt 5y ago
You can pick up an 8TB WD black external for $200, and I saw cheaper WD EasyStore 8TB’s at retail in Best Buy today at $180, and that’s just what I saw offhand earlier today.
- vzkq 5y ago
  Chia Coin is driving up storage prices for the reasons other currencies drove up GPU prices
- walrus01 5y ago
  Something is starting to go really wrong in HDD pricing, following GPUs and other things, because I bought that same 8TB drive two years ago for $147. Hard drive prices are supposed to decrease in $ per TB over a multi year period, with advances in areal density.
  - tmabraham 5y ago
    overall electronics supply chain issues, and I heard something about cryptocurrency mining also being involved with supply issues for hard drives.
    - IncRnd 5y ago
      Yep. Chia coin has made many of these hard drives get purchased.
  - sneak 5y ago
    There are currently supply chain issues in many different markets that will hopefully mostly resolve by the end of this year.
  - lamontcg 5y ago
    The "proof of space" Chia cryptocurrency: https://news.ycombinator.com/item?id=27009843 https://news.ycombinator.com/item?id=27009843
    - alpaca128 5y ago
      I'm starting to wonder what'll occur first: the bubble popping or the whole hardware market coming to a standstill because of 16 different "proof of xyz" schemes. I cannot wait for internet connections to become unusable after a currency introduces "proof of bandwidth".
      - kylegordon 5y ago
        https://pkt.cash/ https://pkt.cash/
      - bserge 5y ago
        Add them all together for the ultimate, proof of destruction: cryptocurrency backed by atmospheric pollution, fauna and flora extinction and natural resource exhaustion. Buy it now, it definitely won't last forever! :D Bu seriously, I still don't understand cryptocurrencies. So, your X discoin is worth Y million real money. But that's only while it's in cryptocurrency form. If people start cashing out, the value will plummet, right? Only the first few will get close to the high value? That seems very similar to a pyramid/ponzi scheme to me. On that note, would it survive some catastrophic event? Like the Internet going down. A bunch of gold will. Hell, invest in power generators and batteries, those will survive and be worth a lot. Can Bitcoin be restored and restarted from one computer with the whole ledger stored on it?
        someguyorother 5y ago
        > Add them all together for the ultimate, proof of destruction: cryptocurrency backed by atmospheric pollution, fauna and flora extinction and natural resource exhaustion. Make a DoomCoin! It would keep 12 difficulty counters, one for each proof of whatever, and getting farthest ahead in one of them lets you mine a new block. That way it can distort the prices of (sorry, "effectively allocate") CPUs, GPUs, disks, ASICs, RAM... anything else you can come up with. Sounds like a good project for a comic book villain. >Can Bitcoin be restored and restarted from one computer with the whole ledger stored on it? Yes, but it may be very slow the first period, as it would be used to the pre-collapse difficulty level.
  - CheezeIt 5y ago
    I’ll add, the 14TB WD externals disappeared off the shelves at the local Best Buy in the past month. The data hoarding subreddit was calling those the best price per TB. There were some $140 listings for an 8TB drive on Amazon a week or two ago, but now they’re $165 and out of stock. I think crypto and hoarding are the reasons; of course inflation is a multiplier.
- saagarjha 5y ago
  Costco has a Seagate 8TB drive on sale for $120 until tomorrow: https://www.costco.com/seagate-backup-plus-hub-8tb-desktop-hard-drive-with-rescue-data-recovery-services.product.100458004.html https://www.costco.com/seagate-backup-plus-hub-8tb-desktop-h...
tasogare 5y ago
> what's basically the sum total of human scientific knowledge That’s an exaggeration. First, a lot of scientific reference books are not a Sci-Hub. Second, decades of published materials have not been digitized and are not available there. Third, very recent work (2020-2021) are often missing as well. And finally, there is a biais towards English and articles in other languages (for instance Korean, Chinese) are often not present despite being indexed or available on other networks. Sci-hub is awesome and I use it almost daily but it’s not perfect yet and it’s not near "total sum of scientific knowledge" yet either.
Causality1 5y ago
Wow the storage market has gotten distorted. Only a month ago I bought a 12TB drive for $235.
- mfkp 5y ago
  I bought 4x 12tb Western Digital drives for $190 each back in October 2020.
- Black101 5y ago
  I bought an 8TB for $125 in November 2018 ... The same drive is $180 today: https://camelcamelcamel.com/product/B07D5V2ZXD https://camelcamelcamel.com/product/B07D5V2ZXD
andrewjl 5y ago
Obvious copyright & TOS issues aside, how does this compare to the S3 bill?
- na85 5y ago
  Good question. It's about $0.023 per GB per month for a US East Coast instance, so some $1700/mo for 75TB. Plus you don't really control the data if it's on someone else's computer so there's a potential opportunity cost there as well.
  - OkGoDoIt 5y ago
    On the other hand the data would be much more reliable in that scenario. A 10 GB consumer hard drive is likely to fail or at least corrupt some data at some point, and if you’re going to go with multiple redundancies then your cost goes up. But yeah, it’s all kind of moot. I had AWS suspend my account once because someone complained they found a copyrighted image on one of my servers. If it’s in the cloud, you don’t really control it.
    - na85 5y ago
      You can buy "datacenter-grade" drives from WD on their website, which are presumably of higher quality. Either way, though, this is going to cost five figures, annually, to store in S3. For half of one year of S3 you can have a fairly robust RAID setup, maybe replace 1 drive every two years, and enjoy the data any time you want it without fees-as-a-service. High five for avoiding AWS!
    - justinclift 5y ago
      > I had AWS suspend my account once because someone complained they found a copyrighted image on one of my servers. If it’s in the cloud, you don’t really control it. Considering the material here, that's more likely to happen than not. So, unlikely to be "much more reliable" for this particular scenario. ;)
  - kylegordon 5y ago
    Half that if you use Infrequent Access, reduce it more if you do something sensible with Glacier policies - it is an archive after all. You'd need to consider method costs, egress, etc on top of all that, to be fair
quixoticelixer- 5y ago
I'm really sorry for you if you think that all human knowledge is in scientific papers
- input_sh 5y ago
  You're confusing it with Sci-Hub. LibGen is for books what Sci-Hub is for papers, though you can also find papers, comics, and magazines on there.
washadjeffmad 5y ago
The non-Chia rate for HDDs is $0.15/GB. I'll add higher densities up to $0.18/GB, but not more.

by na85 5y ago

CheezeIt 5y ago

You can pick up an 8TB WD black external for $200, and I saw cheaper WD EasyStore 8TB’s at retail in Best Buy today at $180, and that’s just what I saw offhand earlier today.

vzkq 5y ago
Chia Coin is driving up storage prices for the reasons other currencies drove up GPU prices
walrus01 5y ago
Something is starting to go really wrong in HDD pricing, following GPUs and other things, because I bought that same 8TB drive two years ago for $147. Hard drive prices are supposed to decrease in $ per TB over a multi year period, with advances in areal density.
- tmabraham 5y ago
  overall electronics supply chain issues, and I heard something about cryptocurrency mining also being involved with supply issues for hard drives.
  - IncRnd 5y ago
    Yep. Chia coin has made many of these hard drives get purchased.
- sneak 5y ago
  There are currently supply chain issues in many different markets that will hopefully mostly resolve by the end of this year.
- lamontcg 5y ago
  The "proof of space" Chia cryptocurrency: https://news.ycombinator.com/item?id=27009843 https://news.ycombinator.com/item?id=27009843
  - alpaca128 5y ago
    I'm starting to wonder what'll occur first: the bubble popping or the whole hardware market coming to a standstill because of 16 different "proof of xyz" schemes. I cannot wait for internet connections to become unusable after a currency introduces "proof of bandwidth".
    - kylegordon 5y ago
      https://pkt.cash/ https://pkt.cash/
    - bserge 5y ago
      Add them all together for the ultimate, proof of destruction: cryptocurrency backed by atmospheric pollution, fauna and flora extinction and natural resource exhaustion. Buy it now, it definitely won't last forever! :D Bu seriously, I still don't understand cryptocurrencies. So, your X discoin is worth Y million real money. But that's only while it's in cryptocurrency form. If people start cashing out, the value will plummet, right? Only the first few will get close to the high value? That seems very similar to a pyramid/ponzi scheme to me. On that note, would it survive some catastrophic event? Like the Internet going down. A bunch of gold will. Hell, invest in power generators and batteries, those will survive and be worth a lot. Can Bitcoin be restored and restarted from one computer with the whole ledger stored on it?
      - someguyorother 5y ago
        > Add them all together for the ultimate, proof of destruction: cryptocurrency backed by atmospheric pollution, fauna and flora extinction and natural resource exhaustion. Make a DoomCoin! It would keep 12 difficulty counters, one for each proof of whatever, and getting farthest ahead in one of them lets you mine a new block. That way it can distort the prices of (sorry, "effectively allocate") CPUs, GPUs, disks, ASICs, RAM... anything else you can come up with. Sounds like a good project for a comic book villain. >Can Bitcoin be restored and restarted from one computer with the whole ledger stored on it? Yes, but it may be very slow the first period, as it would be used to the pre-collapse difficulty level.
- CheezeIt 5y ago
  I’ll add, the 14TB WD externals disappeared off the shelves at the local Best Buy in the past month. The data hoarding subreddit was calling those the best price per TB. There were some $140 listings for an 8TB drive on Amazon a week or two ago, but now they’re $165 and out of stock. I think crypto and hoarding are the reasons; of course inflation is a multiplier.
saagarjha 5y ago
Costco has a Seagate 8TB drive on sale for $120 until tomorrow: https://www.costco.com/seagate-backup-plus-hub-8tb-desktop-hard-drive-with-rescue-data-recovery-services.product.100458004.html https://www.costco.com/seagate-backup-plus-hub-8tb-desktop-h...

tasogare 5y ago

> what's basically the sum total of human scientific knowledge That’s an exaggeration. First, a lot of scientific reference books are not a Sci-Hub. Second, decades of published materials have not been digitized and are not available there. Third, very recent work (2020-2021) are often missing as well. And finally, there is a biais towards English and articles in other languages (for instance Korean, Chinese) are often not present despite being indexed or available on other networks. Sci-hub is awesome and I use it almost daily but it’s not perfect yet and it’s not near "total sum of scientific knowledge" yet either.

Causality1 5y ago

Wow the storage market has gotten distorted. Only a month ago I bought a 12TB drive for $235.

mfkp 5y ago
I bought 4x 12tb Western Digital drives for $190 each back in October 2020.
Black101 5y ago
I bought an 8TB for $125 in November 2018 ... The same drive is $180 today: https://camelcamelcamel.com/product/B07D5V2ZXD https://camelcamelcamel.com/product/B07D5V2ZXD

andrewjl 5y ago

Obvious copyright & TOS issues aside, how does this compare to the S3 bill?

na85 5y ago
Good question. It's about $0.023 per GB per month for a US East Coast instance, so some $1700/mo for 75TB. Plus you don't really control the data if it's on someone else's computer so there's a potential opportunity cost there as well.
- OkGoDoIt 5y ago
  On the other hand the data would be much more reliable in that scenario. A 10 GB consumer hard drive is likely to fail or at least corrupt some data at some point, and if you’re going to go with multiple redundancies then your cost goes up. But yeah, it’s all kind of moot. I had AWS suspend my account once because someone complained they found a copyrighted image on one of my servers. If it’s in the cloud, you don’t really control it.
  - na85 5y ago
    You can buy "datacenter-grade" drives from WD on their website, which are presumably of higher quality. Either way, though, this is going to cost five figures, annually, to store in S3. For half of one year of S3 you can have a fairly robust RAID setup, maybe replace 1 drive every two years, and enjoy the data any time you want it without fees-as-a-service. High five for avoiding AWS!
  - justinclift 5y ago
    > I had AWS suspend my account once because someone complained they found a copyrighted image on one of my servers. If it’s in the cloud, you don’t really control it. Considering the material here, that's more likely to happen than not. So, unlikely to be "much more reliable" for this particular scenario. ;)
- kylegordon 5y ago
  Half that if you use Infrequent Access, reduce it more if you do something sensible with Glacier policies - it is an archive after all. You'd need to consider method costs, egress, etc on top of all that, to be fair

quixoticelixer- 5y ago

I'm really sorry for you if you think that all human knowledge is in scientific papers

input_sh 5y ago
You're confusing it with Sci-Hub. LibGen is for books what Sci-Hub is for papers, though you can also find papers, comics, and magazines on there.

washadjeffmad 5y ago

The non-Chia rate for HDDs is $0.15/GB. I'll add higher densities up to $0.18/GB, but not more.