13 ms·

The Rise of Pirate Libraries (2016)

143 points by yitchelle 9y ago

DonbunEf7 9y ago
It's only a matter of time before somebody delivers a too-cheap-to-meter DOI-to-PDF service.
- JetSpiegel 9y ago
  scihub.cc?
- p4bl0 9y ago
  That is what https://oadoi.org/ https://oadoi.org/ and http://doai.io/ http://doai.io/ do. Sci-Hub can also ingest DOIs.
tomohawk 9y ago
The US government is not allowed to copyright anything, and is surely a funder of many of these works. Contractors for the US government should similarly not be able to copyright work done under contract. What would happen to these copyright arguments if this became the case?
- dawnerd 9y ago
  I tend to agree with you. Universities/Colleges that receive public money also should be able to copyright or patent findings. If public money funds it, it should be public domain.
  - lordnacho 9y ago
    What about some authority like a professor who writes a book about his subject? Suppose the guy's been paid or heavily subsidised through his career. Seems a bit unobvious to me. On the one hand, the guy's gotta bother to sit and write a book. OTOH, his book would be worthless if he didn't have the skills paid for by other people. I guess what's normal is that we disregard how his expertise was funded and just let him keep whatever he makes. After all, he doesn't have to write a textbook.
    - delinka 9y ago
      The difference is writing as part of one's career (at the direction of supervisors) and writing at one's own discretion. If your school says "yo, prof! Write a book on $subject," then it's a work-for-hire and, if public money is involved, should be public domain. If the professor takes it upon himself to write a book, presumably on hus own time, he then holds the copyright.
    - dawnerd 9y ago
      No different than programmers not owning the rights to what they develop while at work.
resf 9y ago
Although descriptive of the legal situation, I object to the term "pirate library". There is no practical difference between a pirate library and a brick and mortar library, except the larger stock of books online. My local library even has a photocopier. A library is a place where you can choose a book, and read it for free. That's how it's been for thousands of years. Copyright is a modern intruder and has no right to brand bone fide libraries as "pirate".
- DanBC 9y ago
  You know authors get payments from loans that libraries make, right? (At least, in UK and Ireland) https://www.plr.uk.com/ https://www.plr.uk.com/
  - delinka 9y ago
    I've never heard of such in the US. I'm now reading your link, but I'd like to hear if anyone know of similar arrangements in the US.
    - kasey_junk 9y ago
      In the US libraries pay for their physical books, which pay royalties per usual. For digital media they usually have a negotiated rate where they pay royalties and limit the number of copies "out on loan".
      - stinkytaco 9y ago
        Negotiated might be strong term. Most libraries work through middlemen who do that negotiation and are then given a rate that they can take or leave. There's no "shopping around" for digital media since publishers control the pricing and availability. This is proving to be a budgeting challenge since digital media is more expensive that physical. There are no discounts from wholesalers for bulk purchasing. Many libraries get physical media at up to a 40% discount from retail because they buy so much. Of course, digital media does not need to be maintained. It does not need to be cataloged and reshelved; it does not wear out or get damaged. But libraries are still adapting to this shift since there still is a great deal of physical media checking out and that staff is still needed.
krylon 9y ago
I really hate to be that person, and content-wise I find the article very interesting, but: “The text collections were far too valuable to simply delete,” he writes, and instead migrated to “closed, membership-only FTP servers.” More recently, though, those collections have moved online So an FTP server is not "online"? I know, I know, it is a minuscule nitpick. But for some reason this kind of thing takes my attention hostage. And let me repeat, content-wise this is a fine and fascinating article.
- salesguy222 9y ago
  Ehhhh I would give them a pass on this one. The word "online" is certainly not precise enough, but I do think it conveys the point that "FTP that requires credentials" is not equal to "freely and conveniently available by URL/google etc" and the reason for that of course is that the median internet user probably doesn't know what FTP is or how to use it
- peatmoss 9y ago
  > So an FTP server is not "online"? I always have to stop and consider whether I'm talking to someone who equates the web with the internet. If your first and only exposure to the internet was on the web, I suspect it's a very hard notion to shake. For those of us who routinely use non-http protocols (even if that's only ssh), or are old enough to remember using gopher, telnet, UUCP, NNTP, SMTP, IMAP, POP3, etc., then ~"moved online and off ftp" is just weird...
- PeterisP 9y ago
  It seems that the "moved online" part is less about FTP and more about the "closed, membership-only" part. Colloquially, "online" pretty much means "available online", and data that is on systems technically connected to internet but accessible only to a few people isn't considered available online.
  - krylon 9y ago
    I get that. The wording was not misleading or ambiguous. In my job, I have to deal with network problems of various kinds from time to time, and at the network layer, "online" has a specific meaning (Okay, actually several, depending on context), and that was what my brain came up with first. Like I said, I was nitpicking.
unhammer 9y ago
For those times when you click a link to a paywalled article, http://unpaywall.org http://unpaywall.org is a pretty handy Firefox extension that will redirect to a (legal) unpaywalled alternative site, if it exists.
jackvalentine 9y ago
The thing about these 'pirate libraries' that makes them seem so vital to me is they're made up of people far more motivated to preserve and keep available the documents than the actual copyright holders ever will be. I draw comparisons to the vast value that was lost when Oink and What.cd were shut down.
- cormacrelf 9y ago
  Those two should really be mentioned in an article about Pirate Libraries. They are exactly what the author describes, minus the 'rise' part.
- chx 9y ago
  I am already wondering whether a fundraiser to download KG would make sense. If that would disappear... there is absolutely no collection like that, anywhere. 100 000 non mainstream movies! If you all you want is a cold copy and not keep it online, the costs are not enormous perhaps 5K EUR or so.
- peatmoss 9y ago
  Sadly, I never even knew about What.cd when it was alive, not that I'd have had the time and wherewithal to join. But this is the article that made me lament its passing: https://qz.com/840661/what-cd-is-gone-a-eulogy-for-the-greatest-music-collection-in-the-world/ https://qz.com/840661/what-cd-is-gone-a-eulogy-for-the-great...
  - jackvalentine 9y ago
    Thanks for the link - the comparison to the burned Library of Alexandria was one I hadn't thought of before.
  - tomduncalf 9y ago
    Wow, likewise never knew of it until now! Sounds incredible although I also would never have had the time to take part. That is one thing that makes me a bit sad about Spotify (and there's not much, I think it's a great product and I'd happily pay more for it if I could be sure a reasonable amount went to the artists I listen to) - I worry that people will get so used to streaming from large, but limited catalogs, that obscure recordings (e.g. 90s dance tracks only released on vinyl and the label is now gone) will be forgotten. Hopefully there's enough hardcore fans to prevent that happening - but in an ideal world Spotify would somehow have access to all this kind of stuff you currently only find on YouTube!
contingencies 9y ago
I have a large physical book library. My 3 year old has a large pirated physical book library. I also have a huge pirated ebook library. In the future, my child will have a copy. Times, they are a-changin'.
- icantdrive55 9y ago
  There was a time, when The Pirate Bay was alive and well, I could find any book. I used to find obscure books on hydronic heating to watch making. It was around 2008, and most of us didn't think it was that bad of a crime, or if it was a crime in certain countries. Those days are gone forever. I wish I copied. A person could say true transfer of information was best in the past. Crazy? But I kinda understand the frustration of certain creators. Then again, it might have been the best free advertising some authors ever had?
- stinkytaco 9y ago
  > My 3 year old has a large pirated physical book library. Could you explain this? Did you steal these from someone?
  - Darkenetor 9y ago
    He read his father's without buying a second "license" himself for each book.
  - contingencies 9y ago
    They are books produced by pirate printing factories whose major business is making duplicates of famous children's books. Quality is great, they are basically indistinguishable from the real thing.
    - stinkytaco 9y ago
      I don't know if I realized such a thing existed. I've seen a few print-on-demand machines in libraries, so I guess I should have guessed, but I have never seen such a thing before.
deleted 9y ago
[deleted]
diego_moita 9y ago
As long as textbooks in North America keep being so ridiculously expensive[0] I wont take "pirate" as a valid moral argument. [0]https://www.amazon.com/Calculus-Early-Transcendentals-James-Stewart/dp/1285741552/ref=sr_1_1?ie=UTF8&qid=1500733012&sr=8-1&keywords=calculus+stewart+8th+edition https://www.amazon.com/Calculus-Early-Transcendentals-James-...
joshvm 9y ago
Anyone who is a student should know that most of the big publishers e.g. Springer, Cambridge Uni. Press, etc have extensive online libraries that your university probably subscribes to. You can go there and download complete books, many of which would be extremely expensive to buy otherwise. A lot of it is academic and dry, but occasionally there are some gems. It was fun to sift through random topics that I had no idea about (like forensic pathology) and build up a small eBook library. Best of all it's within your right as a student to do it (your uni pays thousands for these subscriptions), so exploit it while you can. Most of the time you have to download chapters separately, but I did write a little script that would authenticate with Shibboleth, pull the whole book and combine the PDFs. There was no DRM either, just a watermark with your IP or university details to discourage sharing.
- JadeNB 9y ago
  > You can go there and download complete books, many of which would be extremely expensive to buy otherwise. As much as I hate to say something nice about one of the big publishers, Springer even has a nice little filip: if you already have the eBook, you can buy a physical copy for very cheap ($25, I think). I've never tried this, so I don't know if having access through your university counts. > Most of the time you have to download chapters separately, but I did write a little script that would authenticate with Shibboleth, pull the whole book and combine the PDFs. There was no DRM either, just a watermark with your IP or university details to discourage sharing. Be careful with this, though; Cambridge, for example, bans you for a while if you download too many files. (Obviously "too many" isn't rigorously defined, but I hit it while downloading two many-chapter books.) I can imagine triggering a more aggressive ban if they think you look like a robot downloader.
  - joshvm 9y ago
    I made a list of books I liked the look of, saved it to a text and then setup the script to run every hour ± a bit. Wasn't going to be any faster than I could read them! I don't recall ever being rate limited so I guess the strategy worked.
- trobertson 9y ago
xtracto 9y ago
Reminds me of Gigapedia (later named library.nu), which started as soon as 2007 and in 2012 had a huge amount of books of all kinds Nowadays Library Genesis has a good collections, but it is mostly technical. Gigapedia was amazing in the diversity of the topics it covered. It definitely was a loss.
- JadeNB 9y ago
  What is the current state of such book libraries (in the sense of, "these are some that are alive")? I can never keep track of which among LibGen, BookFi, etc., are still alive, which are alive but not loading, which are the same under different names, and so on.
Xoros 9y ago
It's really complicated those copyright vs moral stuff. Remember Google have somewhere hundreds of thousands book ready to be shipped who probably never will be https://www.theatlantic.com/technology/archive/2017/04/the-tragedy-of-google-books/523320/#?single_page=true https://www.theatlantic.com/technology/archive/2017/04/the-t... Previously discussed here https://news.ycombinator.com/item?id=14172791 https://news.ycombinator.com/item?id=14172791
sl4i6j3o4i98g 9y ago
Alexandra Elbakyan is a modern day hero. Her work has put more knowledge in the hands of those who could otherwise not afford it than Sallie Mae. Cheers to you Alex!
supernumerary 9y ago
Extant libraries I am aware of, please add you're own in replies: https://www.cgpeers.to/ https://www.cgpeers.to/ Computer Graphics software, plugins, models and tutorials.
leed25d 9y ago
Is there some reason why such libraries have not established themselves on the dark web?
- zrth 9y ago
  I2P: http://dc-poisk.no-ip.org http://dc-poisk.no-ip.org
dhimmel 9y ago
Nice article on the history of Shadow Libraries. They've come a long way since their Russian roots in the 90s. We just released a study titled "Sci-Hub provides access to nearly all scholarly literature". Preprint at https://doi.org/10.7287/peerj.preprints.3100 https://doi.org/10.7287/peerj.preprints.3100. There's an accompanying interactive browser at https://greenelab.github.io/scihub https://greenelab.github.io/scihub. Some highlights: > As of March 2017, we find that Sci-Hub's database contains 68.9% of all 81.6 million scholarly articles, which rises to 85.2% for those published in closed access journals. > Coverage also varies by publisher, with the coverage of the largest publisher, Elsevier, at 97.3%. > we estimate that over a six-month period in 2015–2016, Sci-Hub provided access for 99.3% of valid incoming requests.

143 points by yitchelle 9y ago

DonbunEf7 9y ago

It's only a matter of time before somebody delivers a too-cheap-to-meter DOI-to-PDF service.

JetSpiegel 9y ago
scihub.cc?
p4bl0 9y ago
That is what https://oadoi.org/ https://oadoi.org/ and http://doai.io/ http://doai.io/ do. Sci-Hub can also ingest DOIs.

tomohawk 9y ago

The US government is not allowed to copyright anything, and is surely a funder of many of these works. Contractors for the US government should similarly not be able to copyright work done under contract. What would happen to these copyright arguments if this became the case?

dawnerd 9y ago
I tend to agree with you. Universities/Colleges that receive public money also should be able to copyright or patent findings. If public money funds it, it should be public domain.
- lordnacho 9y ago
  What about some authority like a professor who writes a book about his subject? Suppose the guy's been paid or heavily subsidised through his career. Seems a bit unobvious to me. On the one hand, the guy's gotta bother to sit and write a book. OTOH, his book would be worthless if he didn't have the skills paid for by other people. I guess what's normal is that we disregard how his expertise was funded and just let him keep whatever he makes. After all, he doesn't have to write a textbook.
  - delinka 9y ago
    The difference is writing as part of one's career (at the direction of supervisors) and writing at one's own discretion. If your school says "yo, prof! Write a book on $subject," then it's a work-for-hire and, if public money is involved, should be public domain. If the professor takes it upon himself to write a book, presumably on hus own time, he then holds the copyright.
  - dawnerd 9y ago
    No different than programmers not owning the rights to what they develop while at work.

resf 9y ago

Although descriptive of the legal situation, I object to the term "pirate library". There is no practical difference between a pirate library and a brick and mortar library, except the larger stock of books online. My local library even has a photocopier. A library is a place where you can choose a book, and read it for free. That's how it's been for thousands of years. Copyright is a modern intruder and has no right to brand bone fide libraries as "pirate".

DanBC 9y ago
You know authors get payments from loans that libraries make, right? (At least, in UK and Ireland) https://www.plr.uk.com/ https://www.plr.uk.com/
- delinka 9y ago
  I've never heard of such in the US. I'm now reading your link, but I'd like to hear if anyone know of similar arrangements in the US.
  - kasey_junk 9y ago
    In the US libraries pay for their physical books, which pay royalties per usual. For digital media they usually have a negotiated rate where they pay royalties and limit the number of copies "out on loan".
    - stinkytaco 9y ago
      Negotiated might be strong term. Most libraries work through middlemen who do that negotiation and are then given a rate that they can take or leave. There's no "shopping around" for digital media since publishers control the pricing and availability. This is proving to be a budgeting challenge since digital media is more expensive that physical. There are no discounts from wholesalers for bulk purchasing. Many libraries get physical media at up to a 40% discount from retail because they buy so much. Of course, digital media does not need to be maintained. It does not need to be cataloged and reshelved; it does not wear out or get damaged. But libraries are still adapting to this shift since there still is a great deal of physical media checking out and that staff is still needed.

krylon 9y ago

I really hate to be that person, and content-wise I find the article very interesting, but: “The text collections were far too valuable to simply delete,” he writes, and instead migrated to “closed, membership-only FTP servers.” More recently, though, those collections have moved online So an FTP server is not "online"? I know, I know, it is a minuscule nitpick. But for some reason this kind of thing takes my attention hostage. And let me repeat, content-wise this is a fine and fascinating article.

salesguy222 9y ago
Ehhhh I would give them a pass on this one. The word "online" is certainly not precise enough, but I do think it conveys the point that "FTP that requires credentials" is not equal to "freely and conveniently available by URL/google etc" and the reason for that of course is that the median internet user probably doesn't know what FTP is or how to use it
peatmoss 9y ago
> So an FTP server is not "online"? I always have to stop and consider whether I'm talking to someone who equates the web with the internet. If your first and only exposure to the internet was on the web, I suspect it's a very hard notion to shake. For those of us who routinely use non-http protocols (even if that's only ssh), or are old enough to remember using gopher, telnet, UUCP, NNTP, SMTP, IMAP, POP3, etc., then ~"moved online and off ftp" is just weird...
PeterisP 9y ago
It seems that the "moved online" part is less about FTP and more about the "closed, membership-only" part. Colloquially, "online" pretty much means "available online", and data that is on systems technically connected to internet but accessible only to a few people isn't considered available online.
- krylon 9y ago
  I get that. The wording was not misleading or ambiguous. In my job, I have to deal with network problems of various kinds from time to time, and at the network layer, "online" has a specific meaning (Okay, actually several, depending on context), and that was what my brain came up with first. Like I said, I was nitpicking.

unhammer 9y ago

For those times when you click a link to a paywalled article, http://unpaywall.org http://unpaywall.org is a pretty handy Firefox extension that will redirect to a (legal) unpaywalled alternative site, if it exists.

jackvalentine 9y ago

The thing about these 'pirate libraries' that makes them seem so vital to me is they're made up of people far more motivated to preserve and keep available the documents than the actual copyright holders ever will be. I draw comparisons to the vast value that was lost when Oink and What.cd were shut down.

cormacrelf 9y ago
Those two should really be mentioned in an article about Pirate Libraries. They are exactly what the author describes, minus the 'rise' part.
chx 9y ago
I am already wondering whether a fundraiser to download KG would make sense. If that would disappear... there is absolutely no collection like that, anywhere. 100 000 non mainstream movies! If you all you want is a cold copy and not keep it online, the costs are not enormous perhaps 5K EUR or so.
peatmoss 9y ago
Sadly, I never even knew about What.cd when it was alive, not that I'd have had the time and wherewithal to join. But this is the article that made me lament its passing: https://qz.com/840661/what-cd-is-gone-a-eulogy-for-the-greatest-music-collection-in-the-world/ https://qz.com/840661/what-cd-is-gone-a-eulogy-for-the-great...
- jackvalentine 9y ago
  Thanks for the link - the comparison to the burned Library of Alexandria was one I hadn't thought of before.
- tomduncalf 9y ago
  Wow, likewise never knew of it until now! Sounds incredible although I also would never have had the time to take part. That is one thing that makes me a bit sad about Spotify (and there's not much, I think it's a great product and I'd happily pay more for it if I could be sure a reasonable amount went to the artists I listen to) - I worry that people will get so used to streaming from large, but limited catalogs, that obscure recordings (e.g. 90s dance tracks only released on vinyl and the label is now gone) will be forgotten. Hopefully there's enough hardcore fans to prevent that happening - but in an ideal world Spotify would somehow have access to all this kind of stuff you currently only find on YouTube!

contingencies 9y ago

I have a large physical book library. My 3 year old has a large pirated physical book library. I also have a huge pirated ebook library. In the future, my child will have a copy. Times, they are a-changin'.

icantdrive55 9y ago
There was a time, when The Pirate Bay was alive and well, I could find any book. I used to find obscure books on hydronic heating to watch making. It was around 2008, and most of us didn't think it was that bad of a crime, or if it was a crime in certain countries. Those days are gone forever. I wish I copied. A person could say true transfer of information was best in the past. Crazy? But I kinda understand the frustration of certain creators. Then again, it might have been the best free advertising some authors ever had?
stinkytaco 9y ago
> My 3 year old has a large pirated physical book library. Could you explain this? Did you steal these from someone?
- Darkenetor 9y ago
  He read his father's without buying a second "license" himself for each book.
- contingencies 9y ago
  They are books produced by pirate printing factories whose major business is making duplicates of famous children's books. Quality is great, they are basically indistinguishable from the real thing.
  - stinkytaco 9y ago
    I don't know if I realized such a thing existed. I've seen a few print-on-demand machines in libraries, so I guess I should have guessed, but I have never seen such a thing before.

deleted 9y ago

[deleted]

diego_moita 9y ago

As long as textbooks in North America keep being so ridiculously expensive[0] I wont take "pirate" as a valid moral argument. [0]https://www.amazon.com/Calculus-Early-Transcendentals-James-Stewart/dp/1285741552/ref=sr_1_1?ie=UTF8&qid=1500733012&sr=8-1&keywords=calculus+stewart+8th+edition https://www.amazon.com/Calculus-Early-Transcendentals-James-...

joshvm 9y ago

Anyone who is a student should know that most of the big publishers e.g. Springer, Cambridge Uni. Press, etc have extensive online libraries that your university probably subscribes to. You can go there and download complete books, many of which would be extremely expensive to buy otherwise. A lot of it is academic and dry, but occasionally there are some gems. It was fun to sift through random topics that I had no idea about (like forensic pathology) and build up a small eBook library. Best of all it's within your right as a student to do it (your uni pays thousands for these subscriptions), so exploit it while you can. Most of the time you have to download chapters separately, but I did write a little script that would authenticate with Shibboleth, pull the whole book and combine the PDFs. There was no DRM either, just a watermark with your IP or university details to discourage sharing.

JadeNB 9y ago
> You can go there and download complete books, many of which would be extremely expensive to buy otherwise. As much as I hate to say something nice about one of the big publishers, Springer even has a nice little filip: if you already have the eBook, you can buy a physical copy for very cheap ($25, I think). I've never tried this, so I don't know if having access through your university counts. > Most of the time you have to download chapters separately, but I did write a little script that would authenticate with Shibboleth, pull the whole book and combine the PDFs. There was no DRM either, just a watermark with your IP or university details to discourage sharing. Be careful with this, though; Cambridge, for example, bans you for a while if you download too many files. (Obviously "too many" isn't rigorously defined, but I hit it while downloading two many-chapter books.) I can imagine triggering a more aggressive ban if they think you look like a robot downloader.
- joshvm 9y ago
  I made a list of books I liked the look of, saved it to a text and then setup the script to run every hour ± a bit. Wasn't going to be any faster than I could read them! I don't recall ever being rate limited so I guess the strategy worked.
trobertson 9y ago

xtracto 9y ago

Reminds me of Gigapedia (later named library.nu), which started as soon as 2007 and in 2012 had a huge amount of books of all kinds Nowadays Library Genesis has a good collections, but it is mostly technical. Gigapedia was amazing in the diversity of the topics it covered. It definitely was a loss.

JadeNB 9y ago
What is the current state of such book libraries (in the sense of, "these are some that are alive")? I can never keep track of which among LibGen, BookFi, etc., are still alive, which are alive but not loading, which are the same under different names, and so on.

Xoros 9y ago

It's really complicated those copyright vs moral stuff. Remember Google have somewhere hundreds of thousands book ready to be shipped who probably never will be https://www.theatlantic.com/technology/archive/2017/04/the-tragedy-of-google-books/523320/#?single_page=true https://www.theatlantic.com/technology/archive/2017/04/the-t... Previously discussed here https://news.ycombinator.com/item?id=14172791 https://news.ycombinator.com/item?id=14172791

sl4i6j3o4i98g 9y ago

Alexandra Elbakyan is a modern day hero. Her work has put more knowledge in the hands of those who could otherwise not afford it than Sallie Mae. Cheers to you Alex!

supernumerary 9y ago

Extant libraries I am aware of, please add you're own in replies: https://www.cgpeers.to/ https://www.cgpeers.to/ Computer Graphics software, plugins, models and tutorials.

leed25d 9y ago

Is there some reason why such libraries have not established themselves on the dark web?

zrth 9y ago
I2P: http://dc-poisk.no-ip.org http://dc-poisk.no-ip.org

dhimmel 9y ago

Nice article on the history of Shadow Libraries. They've come a long way since their Russian roots in the 90s. We just released a study titled "Sci-Hub provides access to nearly all scholarly literature". Preprint at https://doi.org/10.7287/peerj.preprints.3100 https://doi.org/10.7287/peerj.preprints.3100. There's an accompanying interactive browser at https://greenelab.github.io/scihub https://greenelab.github.io/scihub. Some highlights: > As of March 2017, we find that Sci-Hub's database contains 68.9% of all 81.6 million scholarly articles, which rises to 85.2% for those published in closed access journals. > Coverage also varies by publisher, with the coverage of the largest publisher, Elsevier, at 97.3%. > we estimate that over a six-month period in 2015–2016, Sci-Hub provided access for 99.3% of valid incoming requests.