6 ms·

a plain old website or a publishing house with distribution services and syndication attached, but for a sane price. "a whole lot of tiny files" severely under

a plain old website or a publishing house with distribution services and syndication attached, but for a sane price. "a whole lot of tiny files" severely underestimates the scale at work. Libgen's coverage is relatively shallow, and pdf books tend to be huge, at least for older material. Scihub piggy backs on the publishers, so that's your reference. syndication, syndicate, quite apt don't you think? Libraries that coluded with the publishers and accepted the pricing must have been a huge part of the problem, at least historically. Now you know there's only one way out of a mafia.

einpoklum 5y ago
If the sane price is an optional "Donate to keep this site going" link, then ok. But only free access, without authentication or payment, to scientific papers, is sane. IMHO.
- munk-a 5y ago
  Might this be a case where the best resolution would be to have the government (which is at least partially funding nearly all of these papers) step in and add a ledger of papers as a proof of investment? The cost of maintaining a free and open DB of scientific advances and publications would be so incredibly insignificant compared to both the value and the continued investment in those advancements.
  - einpoklum 5y ago
    Well, some research venues (and publication venues) are not government-funded, and even if they are indirectly government funded, it's more of a sophistry than something which would make publishers hand over copies of the papers. Also, a per-government ledger would not be super-practicable. But if, say, the US, the EU and China would agree on something like this, and implement it, and have a common ledger, then it would not be some a big leap to make it properly international. Maybe even UN-based. That's a pretty big "if" though.
  - jpeloquin 5y ago
    > Might this be a case where the best resolution would be to have the government (which is at least partially funding nearly all of these papers) step in and add a ledger of papers as a proof of investment? I feel that we're halfway there already and are gaining ground. Does Pubmed Central [0] (a government-hosted open access repository for NIH-funded work) count as a "ledger" like you're referring to? The NSF's site does a good job of explaining current US open access policy [1]. There are occasional attempts to expand the open access mandate by legislation, such as FASTR [2]. A hypothetical expansion of the open access mandate to apply to all works from /institutions/ that receive indirect costs, not just individual projects that receive direct costs, would open things up even more. [0] https://www.ncbi.nlm.nih.gov/pmc/ https://www.ncbi.nlm.nih.gov/pmc/ [1] https://www.nsf.gov/pubs/2016/nsf16009/nsf16009.jsp#q1 https://www.nsf.gov/pubs/2016/nsf16009/nsf16009.jsp#q1 [2] https://sparcopen.org/our-work/fastr/ https://sparcopen.org/our-work/fastr/
- posterboy 5y ago
  I share the sentiment insofar as free access would benefit my own sanity, except when it is about hording. On the other hand, there is a slippery sloap to decide what isn't scientific so much as to not be required open knowledge. By the way, specialist knowledge and open knowledge is kind of a dichotomy. You would need to define the intersection of both. Suddenly you are looking at a patent system. Pay to Quote, citation fees, news websites already are demanding this from google, here in Germany, inuding Springer Press
jandrese 5y ago
In Internet scale it's not a lot of data. Most people who think they have big data don't. Estimates I've seen put the total Scihub cache at 85 million articles totaling 77TB. That's a single 2U server with room to spare. The hardest part is indexing and search, but it's a pretty small search space by Internet standards.
- HWR_14 5y ago
  It still amazes me that 77TB is considered "small". Isn't that still in the $500-$1,000 range of non-redundant storage? Or if hosted on AWS, isn't that almost $1,900 a month if no one accesses it? I know it's not Big Data(tm) big data, but it is a lot of data for something that can generate no revenue.
  - dredmorbius 5y ago
    For an institution, it's a rounding error. AWS is not the cheapest bulk-storage hosting possible.
  - pbhjpbhj 5y ago
    I'm prepared to accept "does generate no revenue" but "can generate no revenue" ...? Perhaps some sort of MTurk or captcha-like tasks per access? Patr[e]ons? Donation drives? Micro-payments? Something else??
    - HWR_14 5y ago
      Oh, it could generate revenue if it was legal. But it is not, so it seems difficult.
  - smichel17 5y ago
    > Isn't that still in the $500-$1,000 range of non-redundant storage? Sure. Let's add redundancy and bump by an order of magnitude to give some headroom -- $5-10k is a totally reasonable amount to fundraise for this sort of application. If it were legal, I'm sure any number of universities would happily shoulder that cost. It's miniscule compared to what they're paying Elsevier each year.
whimsicalism 5y ago
Libgen's coverage is definitely more shallow than scihub, but it is still pretty good.

einpoklum 5y ago

If the sane price is an optional "Donate to keep this site going" link, then ok. But only free access, without authentication or payment, to scientific papers, is sane. IMHO.

munk-a 5y ago
Might this be a case where the best resolution would be to have the government (which is at least partially funding nearly all of these papers) step in and add a ledger of papers as a proof of investment? The cost of maintaining a free and open DB of scientific advances and publications would be so incredibly insignificant compared to both the value and the continued investment in those advancements.
- einpoklum 5y ago
  Well, some research venues (and publication venues) are not government-funded, and even if they are indirectly government funded, it's more of a sophistry than something which would make publishers hand over copies of the papers. Also, a per-government ledger would not be super-practicable. But if, say, the US, the EU and China would agree on something like this, and implement it, and have a common ledger, then it would not be some a big leap to make it properly international. Maybe even UN-based. That's a pretty big "if" though.
- jpeloquin 5y ago
  > Might this be a case where the best resolution would be to have the government (which is at least partially funding nearly all of these papers) step in and add a ledger of papers as a proof of investment? I feel that we're halfway there already and are gaining ground. Does Pubmed Central [0] (a government-hosted open access repository for NIH-funded work) count as a "ledger" like you're referring to? The NSF's site does a good job of explaining current US open access policy [1]. There are occasional attempts to expand the open access mandate by legislation, such as FASTR [2]. A hypothetical expansion of the open access mandate to apply to all works from /institutions/ that receive indirect costs, not just individual projects that receive direct costs, would open things up even more. [0] https://www.ncbi.nlm.nih.gov/pmc/ https://www.ncbi.nlm.nih.gov/pmc/ [1] https://www.nsf.gov/pubs/2016/nsf16009/nsf16009.jsp#q1 https://www.nsf.gov/pubs/2016/nsf16009/nsf16009.jsp#q1 [2] https://sparcopen.org/our-work/fastr/ https://sparcopen.org/our-work/fastr/
posterboy 5y ago
I share the sentiment insofar as free access would benefit my own sanity, except when it is about hording. On the other hand, there is a slippery sloap to decide what isn't scientific so much as to not be required open knowledge. By the way, specialist knowledge and open knowledge is kind of a dichotomy. You would need to define the intersection of both. Suddenly you are looking at a patent system. Pay to Quote, citation fees, news websites already are demanding this from google, here in Germany, inuding Springer Press

jandrese 5y ago

In Internet scale it's not a lot of data. Most people who think they have big data don't. Estimates I've seen put the total Scihub cache at 85 million articles totaling 77TB. That's a single 2U server with room to spare. The hardest part is indexing and search, but it's a pretty small search space by Internet standards.

HWR_14 5y ago
It still amazes me that 77TB is considered "small". Isn't that still in the $500-$1,000 range of non-redundant storage? Or if hosted on AWS, isn't that almost $1,900 a month if no one accesses it? I know it's not Big Data(tm) big data, but it is a lot of data for something that can generate no revenue.
- dredmorbius 5y ago
  For an institution, it's a rounding error. AWS is not the cheapest bulk-storage hosting possible.
- pbhjpbhj 5y ago
  I'm prepared to accept "does generate no revenue" but "can generate no revenue" ...? Perhaps some sort of MTurk or captcha-like tasks per access? Patr[e]ons? Donation drives? Micro-payments? Something else??
  - HWR_14 5y ago
    Oh, it could generate revenue if it was legal. But it is not, so it seems difficult.
- smichel17 5y ago
  > Isn't that still in the $500-$1,000 range of non-redundant storage? Sure. Let's add redundancy and bump by an order of magnitude to give some headroom -- $5-10k is a totally reasonable amount to fundraise for this sort of application. If it were legal, I'm sure any number of universities would happily shoulder that cost. It's miniscule compared to what they're paying Elsevier each year.

whimsicalism 5y ago

Libgen's coverage is definitely more shallow than scihub, but it is still pretty good.