6 ms·

What happened to Google's effort to scan university library books? (2017)

• fierarul 7y ago
  I heard of HathiTrust Research Center but didn't know almost all the books were scanned by Google. Google was nice to want to share. But I think the original and ongoing concern was improved image recognition and NLP. Ie. Machine Learning the whole world knowledge.
  • extra88 7y ago
    > Google was nice to want to share. Niceness had nothing to do with it. From the article, "As part of the deal, Google’s partner libraries made sure they got to keep digital copies of their scanned works for research and preservation use."
• GlenTheMachine 7y ago
  FWIW: you have to be affiliated with an academic institution to get access to most full texts on Hathi. However, one such academic institution is the Library of Congress. I would think anyone reading Hacker News in the US would qualify as a researcher at the Library of Congress; the catch (admittedly a big one) is that you have to present yourself in person to get a library card ("reader registration card" in LoC parlance) -- meaning that unless you live in the immediate DC area you have to travel there. And they're only valid for two years. Link to how to get a library card at the LoC: https://www.loc.gov/rr/readerregistration.html https://www.loc.gov/rr/readerregistration.html
  • toomuchtodo 7y ago
    I swung by the LoC on a biz trip to DC and obtaining the reader registration card was a quick and painless process; highly recommend to anyone on this forum who finds themselves in the area and can spare an hour.
    • saalweachter 7y ago
      Realizing that Congresspeople can check out books from the Library was the only thing that ever made me want to run for Congress.
  • TheAceOfHearts 7y ago
    I've always wanted to visit the Library of Congress and now I have even more of a reason to do so. Any other advice or suggestions for my visit?
    • joshklein 7y ago
      It may be the only way to find some of thing things preserved via citizen archives. I enjoyed the talk Ian MacKaye gave there about archiving the live performances of Minor Threat & Fugazi. Here [1] is a post on the subject. [1] https://blogs.loc.gov/thesignal/2013/05/ian-mackaye-and-citizen-archiving/ https://blogs.loc.gov/thesignal/2013/05/ian-mackaye-and-citi...
      • Sherl 7y ago
        Open only to citizens?
        • joshklein 7y ago
          Hmm, that seems unlikely but I honestly don’t know. Possibly free for US citizens but a small fee for visitors? I can’t imagine it would be closed to visitors, though you should be able to contact them to find out. The talk I so enjoyed is on YouTube with many others: https://youtu.be/AvqtY_7Q7hI?list=PLEA69BE43AA9F7E68 https://youtu.be/AvqtY_7Q7hI?list=PLEA69BE43AA9F7E68 (video 30 is MacKaye)
    • kd5bjo 7y ago
      It’s a closed-stack library and their catalog is online. Figure out the items you want to start with before you arrive so that you can put in the order; it takes a little while for the staff to fetch it from the stacks and deliver it to the reading room. Also, pick your reading room carefully: the general collection can be delivered anywhere, but there are are some items that you can only get in a specific one.
    • mobilefriendly 7y ago
      Be sure to visit the Jefferson Building, its a "temple to learning" with wonderful paintings and tiles.
  • est31 7y ago
    As you seem to know stuff about the LoC: I've read that of each newly published book in the US you have to send samples to the LoC. However, I've read that they don't keep all samples. Do you know what happens with the ones they discard with? How many are discarded and what are the criteria?
    • saalweachter 7y ago
      https://www.loc.gov/about/fascinating-facts/ https://www.loc.gov/about/fascinating-facts/ "Each working day the Library receives some 15,000 items and adds more than 10,000 items to its collections. Materials are acquired as Copyright deposits and through gift, purchase, other government agencies (state, local and federal), Cataloging in Publication (a pre-publication arrangement with publishers) and exchange with libraries in the United States and abroad. Items not selected for the collections or other internal purposes are used in the Library’s national and international exchange programs. Through these exchanges the Library acquires material that would not be available otherwise. The remaining items are made available to other federal agencies and are then available for donation to educational institutions, public bodies and nonprofit tax-exempt organizations in the United States."
  • mimixco 7y ago
    Another way to access Hathi is by physically visiting an affiliated university campus and doing it from their computers. Most university libraries allow the public this access.
  • btrettel 7y ago
    As I recall the Library of Congress also has some digital subscriptions on-site, though not Elsevier. One subscription the Library of Congress has that Libgen and Scihub don't have is ProQuest, so you can download theses and dissertations from ProQuest there. I'm not sure that you even need to be registered to use the building wifi, actually, though I'd recommend that too.
  • idclip 7y ago
    Can non americans get one?
    • GlenTheMachine 7y ago
      I would guess so but the website doesn’t specify.
      • idclip 7y ago
        I will find out someday, its on my bucket list now
  • JohnFen 7y ago
    I wish I had known that when I last visited DC. I'll make sure to get a registration card on my next visit.
  • deleted 7y ago
    [deleted]
• oefrha 7y ago
  As an academic library user, it would be especially helpful if one could read obscure/old titles stored offsite in digital form; currently one has to have it shipped to campus, pick it up, and maybe realize it doesn’t actually contain the needed info — a waste of energy on multiple parties.
  • jdshaffer 7y ago
    I find archive.org fairly useful for a lot of older books. If they are past the copyright date, then many books can be found and downloaded as PDFs. I've found lots of old Language & Literature books from 80-140 years ago. :-)
• at_a_remove 7y ago
  Speaking from experience, the Hathi metadata is a hot mess. If I had to guess, I would say that it was largely sourced from whatever institutions catalog without a great deal of regularization. There are levels of dirtiness in data. Hathi's metadata reached a level of dirtiness where the filthier fields could actually be mined for other data, given enough labor. While I was able to extract some benefits from this at some cost, it was no substitute for having things done well. Many of the scans are described as "unusable" by the faculty and having looked at many of their complaints I would agree.
  • claudeganon 7y ago
    I worked on this data in a former life and it is indeed a mess and derived from institutional records. But it’s not any more a mess than most other forms of library-derived metadata because MARC and related formats are essentially pseudo-schemas that are at best haphazardly applied. From what I recall, there were attempts at regularization by the big universities involved with project, but they’re both not very good at tech projects and especially not good at finding and keeping the high-level talent that would be needed for something of this scale. But if you think bikeshedding is bad in tech, take a look into some disputes between library cataloguers over their arcana. Much of what they produce is valuable at level of a trained cataloguer who can navigate its inconsistencies and vagaries, but rarely so at the machine level.
• Metacelsus 7y ago
  See also: https://www.theatlantic.com/technology/archive/2017/04/the-tragedy-of-google-books/523320/ https://www.theatlantic.com/technology/archive/2017/04/the-t...
• softfalcon 7y ago
  What happened? tldr; lawyers...
  • readhn 7y ago
    like it was a surprise
• williesleg 7y ago
  They're worthless now?
• rwhaling 7y ago
  I used to work with HathiTrust and attended a few workshops; my experience was that the notion of a "data capsule" that ensures "non-consumptive use" is quite onerous, in practice, and that getting that access can acquire prior approval and supervision of research activities: https://wiki.htrc.illinois.edu/display/COM/HTRC+Data+Capsule+Environment https://wiki.htrc.illinois.edu/display/COM/HTRC+Data+Capsule... I think it's actually a clever technical solution to a hideous legal problem, and it makes a lot of research possible that would be totally illegal otherwise, but it 100% gets in the way of totally legitimate research as well.
  • mcguire 7y ago
    Does it make you feel any better that most of the texts over a decade old are permanently out of print and thus valueless, and that the hideous legal problem only applies to a tiny fraction of the works in question?
    • marvindanig 7y ago
      Not to mention how much of Amazon's rainforest have been/are being razed to ground to make way for pulp farming for the dead trees. It's scary and stupid both.
      • ars 7y ago
        Are you being sarcastic? Amazon rainforest wood is not used for paper, the trees are cut down for farmland, not for the wood. There are far better sources of wood, but land is more scarce.
        • marvindanig 7y ago
          Who said anything about using old trees for book pulp? Pulp farming requires arable land which is taken away from rainforests.
  • jccalhoun 7y ago
    My experience with HathiTrust is that I search for something through my university library's site, I click on a link to something that takes me to Hathi and it says I can't access it. Thanks for nothing HathiTrust. I guess it might be useful for people that want to do some quantitative work but for those of us who want to actually read the works it is just digital blue balls.
• mcguire 7y ago
  "A Data Capsule is a secure, virtual computer that allows what’s known as “non-consumptive” research, meaning that a scholar can do computational analysis of texts without downloading or reading them. The process respects copyright while enabling work based on copyrighted materials." And that is completely ridiculous; a technical solution to an invented problem.
  • heartbeats 7y ago
    There's an easier solution, we call it "my friend Ivan". I have a friend, Ivan, who lives somewhere in the world - he's notoriously reticent about his location, and only communicates over OTR at strange times. Whenever I want to get some research done, I ask him if he happens to know this or that fact about this or that copyrighted database. I then cite him as a source if anyone has any questions.
  • claudeganon 7y ago
    > And that is completely ridiculous; a technical solution to an invented problem. Congratulations, you’ve just described most software projects by libraries and archives. It’s funny, because back in the 60s-80s, libraries were leaders in building shared data systems and networked infrastructure. The history of OCLC describes this well. But once the web came around, they had an identity crisis, were unable to react to technology trends, and largely got conned into predatory and restrictive arrangements by service providers (Elsevier, ProQuest, etc). The same thing actually happened in the past with microfiche and led to libraries destroying huge, valuable portions of their collections, that could have been better preserved with the advent of scanning technologies.
    • orpheansodality 7y ago
      Most librarians would agree with you that the current situation is terrible, and if you look at library literature they've generally felt that way from the beginning. They are highly constrained by copyright law. re: microfiche, it lasts much longer than digital scans -- if kept in good conditions it has a usable half-life on the scale of >100 years, while digital files need much more active maintenance to both prevent bit-rot and to ensure the file-type is still readable (eg countless file formats have been abandoned and are only currently accessible via emulators of older machines). And if you want to bring the cloud into this, most libraries don't have the funding to bring in the technical know-how to manage a private S3 instance only accessible from that building.
• Haga 7y ago
  Should have leaked them to Sci hub
• Anon84 7y ago
  Another story, with a lot more detail: https://www.wired.com/2017/04/how-google-book-search-got-lost/ https://www.wired.com/2017/04/how-google-book-search-got-los...
• musicale 7y ago
  > “Somewhere at Google there is a database containing 25 million books and nobody is allowed to read them.” That is the key result: the greatest library in human history is locked shut, primarily because it was perceived as a threat to the legacy business model of paper book publishing (and royalties based on sales of paper books.) Restricting libraries to "non-consumptive use" betrays the fundamental purpose of a library: to enable people to read the books in the collection; it makes little sense with physical books and even less with digital texts which don't wear out.
• yaroslavvb 7y ago
  I was an engineer in Google Books in 2011 and got an impression that this effort turned out to be not as useful as expected. This took away energy from the legal fight. Having access to every printed book seemed amazing in principle, but it didn't translate to amazing demand for the service. My theory is that it didn't lower the barriers to knowledge much -- there's not that much useful information left in offline-only storage.
  • mnl 7y ago
    I think there's a lot of information sleeping in depositories only on paper. The thing is people are busy with the urgent stuff and stay away from the important if they can't get something immediate from it. I've used extensively Google Books for research of 17th century Mathematics (not just me, also the whole department were I was at that point). I couldn't have afforded to visit all the libraries to look up the originals but thanks to this repository I could download several copies of obscure works, which gives your very interesting insights: sometimes you hit a personal copy of a previous researcher, or there's an ex-libris from an old institution that you know your author was connected to, or you just discover something no one had noticed before simply with a clever string search. You can even backtrack to the forgotten true primary source of a mistakenly repeated fact for instance. It's been a blessing, honestly. If you are into whatever not too transited alley of history, you realise soon that your (former?) company has produced a treasure trove for future generations that just shouldn't be shut down ever: there are many books there not digitized anywhere else. So thank you very, very much for your work there. It definitely belongs to the important.
• selfishgene 7y ago
  Libgen happened.
• lawrenceyan 7y ago
  It sucks that we could have basically had a virtual Library of Alexandria thanks to Google, but because of book publishers lobbying in order to try and conserve their monopolies, it's now locked away to the general public.