10 ms·

How Badly Is Google Books Search Broken, and Why?

134 points by longdefeat 8y ago

mimixco 8y ago
Based on my experience with research using Google Books and Hathi Trust, this author is correct. Google has purposely broken Google Books so that it doesn't compete with any paid sources for these materials -- even if there aren't any paid sources and the book is out of print! The TL;DR is that Google Books started out with the goal of digitizing every book ever written. Publishers sued, so they crippled their search and display functions and handed over the full texts they already had to a group called Hathi Trust. Hathi Trust is seriously crippled on purpose. It only allows access to full texts when you are sitting in a physical library of one its partner universities. That's right... I can drive to a big university, sit in their library, and download a full PDF of any book I like. But if I'm at my house, I can only read one page at a time in a browser. This is ridiculous. Hathi Trust is helping the oil business more than they're helping researchers. The marriage of Google Books content and Hathi Trust as a distribution platform is a joke. In some cases, you will even have to order an old book from interlibrary loan (see worldcat.org) if you can even get it -- when all the while Google has a scanned copy!
- kevin_thibedeau 8y ago
  If you were an author of a book with active copyright you might want to get paid for it.
  - mimixco 8y ago
    As a matter of fact, I am. I'm speaking here of old, out-of-print books, where this is very much a problem with Google Books and Hathi Trust.
    - jandrese 8y ago
      It's also a fundamental problem with our current copyright law where material remains in copyright for many decades after it has stopped being printed and sold through primary channels. Copyright should automatically expire 10 or 15 years after the last printing IMHO. If nobody cares enough to put it up for sale or even make a tiny print run just to renew the copyright, why should the government continue to enforce it? Of course this scheme falls apart a bit in the digital age, except that even ebooks get pulled from the shelves for no apparent reason. Maybe we should just go back to having to explicitly renew copyright after 15 years or so, with a fee just large enough to convince people to drop dormant works. Maybe a couple hundred bucks every 5 years. The best part would be having some easily accessed online system where you could check the copyright status of any work, including current contact information for the rightsholders if you want to arrange payment.
      - izacus 8y ago
        Expiring copyright after 15 years no matter the printing might even be a better solution. That would allow other people to build and extend works, movies and software. 15 years is plenty to make a mountain of cash.
        TeMPOraL 8y ago
        Yup. The way I see it, a lot of IP problems are caused by people who want to take the law intended for promoting new works, and turn it into a source of passive income.
        jandrese 8y ago
        But people wouldn't be encouraged to make original works if their great grandchildren would be robbed of the opportunity to squabble over the rights to it in the courts.
  - Spooky23 8y ago
    How? My grandfather wrote a book in the 1940s that’s been out of print since the mid 50s. Every entity associated with the book is dead, including the publisher, which merged into another in the late 50s and is probably an inactive imprint of some successor company. Grandpa died in 1985, and a cousin or I is likely the heir to his rights. I have a copy of the book, which I bought via Alibris from a bookstore in Wales 15 years ago. If you needed the book for research, you’d probably get it via inter-library loan from a university or a big city library. Whomever the publisher is, they don’t have it and aren’t selling it. In no scenario does anyone get paid for transacting, other than a reseller or the post office.
    - greglindahl 8y ago
      That's a classic "orphan copyright" situation.
      - ghaff 8y ago
        Right. And orphan works legislation has tended to be opposed by individual creators or at least the organizations that purport to represent them. The (not totally unreasonable) theory is that individuals or their estates--let's leave aside the fact that copyright terms are almost certainly too long--may well inadvertently not keep on top of what's needed to keep works non-orphaned. But "Disney" (or whoever) will most certainly be ready to pounce on anything they can acquire for free for whatever technical reason.
    - mimixco 8y ago
      Right... And since Google Books probably has a scan, who cares if you download the PDF? This problem is impeding real research without helping anyone.
    - dwyerm 8y ago
      Stay with me for a second; I'm going to go wildly afield... does intellectual property need an Adverse Possession law, too? [1] Your grandfather had a book, the rights of which should have been presumably passed down to you. My grandfather had a patented mining claim that has been passed down to me. Where the ownership of your grandfather's book is questionable, for me the physical corners of the property are questionable. A number of them are defined by things like a "4 foot spruce post" or a "12 inch diameter tree trunk" that haven't survived the ravages of time. But it is important that I patrol my property at least every couple of years because of Adverse Possession. If someone else were to use my property continuously and I don't say anything against it, one day their trespassing suddenly and magically would become ownership. For a land-owner, it is a scary idea that someone can steal my property from me, as has actually happened. [2] But I can acknowledge that it makes some sense. It comes from the idea that land is meant to be used, and if you aren't using it, maybe the person who is using it should get the rights. If nobody can stand up for an intellectual property claim, perhaps some kind of adverse possession is in order. [1] https://en.wikipedia.org/wiki/Adverse_possession https://en.wikipedia.org/wiki/Adverse_possession [2] http://articles.latimes.com/2007/dec/03/nation/na-land3 http://articles.latimes.com/2007/dec/03/nation/na-land3
  - dredmorbius 8y ago
    Hathi will not provide access to even out-of-copyright works. LibGen has no such problems.
- deleted 8y ago
  [deleted]
drallison 8y ago
Google Books seems to have major problems. An alternative interested parties should explore is Archive.org. The Internet Archive has a significant collection of scanned books and other materials.
oasisbob 8y ago
This breakage reminds me of similar constrained-search problems with the Google Groups/ Dejanews USENET archive. Once upon a time it was nice to research with.
- darkpuma 8y ago
  It reminds me of the demise of google's code search and github's woefully inadequate code search. Debian's Code Search is okay, but github allowing it would be great.
greglindahl 8y ago
Here's an example of an exploration tool built using the content of books: https://books.archivelab.org/dateviz/ https://books.archivelab.org/dateviz/ It would be nice if anyone could build such tools, but all of that data is locked up inside of places like Google Books and Hathi Trust. Google isn't even interested in making their metadata available, other than by running searches.
ccleve 8y ago
I wonder if the date problem is a bug, not a feature? Is it possible to dump the metadata of a book and check if they have the right date? There should probably be multiple dates for a book -- date written, date copyrighted, date published, date of latest edition, etc. My guess is that Google does not have a publicly-available issue tracker for Google Books so you can't easily report this problem. Hacker News is a good way to get their attention, though...
phonon 8y ago
Using https://babel.hathitrust.org/cgi/ls?a=srchls&anyall1=all&q1=%22set+in++stone%22&field1=ocr&op3=AND&yop=before&pdate_end=2002&facet_lang=&facet_format= https://babel.hathitrust.org/cgi/ls?a=srchls&anyall1=all&q1=... I get 14,515 results, with 3,115 of them full view. There is also https://analytics.hathitrust.org/ https://analytics.hathitrust.org/ which seems interesting!
- dtagames 8y ago
  And all of them are full view if you are physically sitting in a Hathi Trust "partner" university library. These libraries are open to the public and allow downloading and saving of the materials you browse, making the whole point of locking them up completely pointless.
  - elektor 8y ago
    Is it then possible for an Aaron Schwartz/ Sci Hub character to then download all of the available books and make them available on the internet?
    - sodosopa 8y ago
      yes and that’s theft.
  - toomuchtodo 8y ago
    Mind emailing me? You don’t have contact info in your profile. I’d be interested in hanging at a library for a few hours with my laptop and have some questions before going.
    - mimixco 8y ago
      To whom is this directed? If it's to me, just write your question here. I've pretty much already explained how it works. Go to the Hathi Trust login screen. Pull down the list of partner institutions. Go to one of them in person. (or get a Library of Congress card, also in person.) After the institution logs in, you can download anything you want. Bring a flash drive or portable hard disk and take home your PDFs. I'd recommend calling the partner library first to make sure someone there knows the Hathi login. It's not that popular a resource, sadly, and many people have no idea what it is. You may also find a friendly librarian who is willing to do the download for you and email you the PDF, saving an in-person visit. One thing to note: You don't have to be a student to use the resources of university libraries. They're open to everyone.
jetrink 8y ago
Searching within books is broken too. If you search for 'cat', any instance of the word 'cats' will also be returned (and any other word beginning with 'cat'), but with the message, "No preview is available." No link to the page of the result is provided. It seems like the kind of bug that should be straightforward to locate and fix, but it has been this way for years. (My guess is that the tool that builds previews doesn't recognize partial matches.)
- mimixco 8y ago
  It's not a bug. It's on purpose. They actively limit previews, full page views, and downloads -- even though you can go to a university and download the same book from Hathi Trust. Hathi Trust isn't paying the copyright holders, either, so who cares?
mimixco 8y ago
Here's an easy example... Google Books has many old issues of Popular Mechanics which exist in full downloadable form on the Internet Archive -- yet you can only see individual pages on Google Books and can't download the magazine. This is because Google Books is acting like they own the copyright (or, at least, they feel the need to police it.) There are many cases where you can download the entire book from Hathi Trust when you are sitting at a university library, giving you a PDF you can use anywhere. But you cannot even see the entire book or download it from Google Books (which has given its scan to Hathi). This is just stupid.
ppod 8y ago
Google tried to make this work, but they were sued; and then they made a deal, and then many, many people objected to the resulting deal. This is the usual process whereby a corporation is first criticised for having too much power, and then when they relinquish power they are criticised for not doing enough. https://www.theatlantic.com/technology/archive/2017/04/the-tragedy-of-google-books/523320/ https://www.theatlantic.com/technology/archive/2017/04/the-t... The end of that article is a not-so-subtle plea for someone within google to perhaps accidentally anonymously place this material in public.
- mimixco 8y ago
  Exactly my point. The "deal" resulted in anyone being able to download the full books if they go to a university library which is a partner of Hathi Trust. Of course, after that download, you have the full PDF and you can do whatever you want! This isn't protection for copyright holders. (Hathi Trust got its books from Google Books scans and doesn't pay copyright holders.) This "deal" isn't helping anyone and it hurts researchers.
  - ahi 8y ago
    HathiTrust does not provide access to in copyright materials to anyone. They provide research access for "non-consumptive" use through the HathiTrust Research Center.
    - mimixco 8y ago
      That's not correct. First of all, all books are copyright by their authors. Anyone can go to a partner library (there's a list on the login page of the Hathi site) and download those books as PDFs. Try it! I've done it many times.
      - ahi 8y ago
        These pages provide a decent overview of access restrictions for HathiTrust materials: https://www.hathitrust.org/features_benefits https://www.hathitrust.org/features_benefits https://www.hathitrust.org/out-of-print-brittle https://www.hathitrust.org/out-of-print-brittle
mimixco 8y ago
Here's another little-known fact: Every American is entitled to a library card from the Library of Congress (you have to go there in person to get one). The Library of Congress is a Hathi Trust partner! So if you go get that card, you can download all of the out-of-print books that Google scanned on your own computer. No copyright holders are getting paid (and no one is being harmed), so why all these barriers in-between?
- sodosopa 8y ago
  No copyright holders being paid is theft. Authors write and should be paid for their efforts. This is glorified and legalized privacy.
  - mimixco 8y ago
    No, we're talking about out-of-print books. There is no one around to claim copyright or payment. No one is trying to download Harry Potter from Google Books, lol!
  - Fnoord 8y ago
    > No copyright holders being paid is theft No, it isn't. It is copyright infringement. Theft is something completely different. Different laws apply.
    - sodosopa 8y ago
      If you take money out of an author/musician/artist’s hand for a work they created. You can call it anything you desire but it’s theft and piracy.
      - Fnoord 8y ago
        So if I decide not to take the bus its theft as well? If I decide to steal a bicycle, its theft from the bus company?
        8y ago
ChuckMcM 8y ago
It isn't surprising, this bit is sad: Google Books has failed to live up to its promise as the company has moved away from its original mission of organizing information for people. Google was only about organizing all the worlds information while search ads was an unlimited fountain of money. As Google's ability to generate money with search ads has dwindled, their more grand (and not monetizable) projects have been either starved for resources or outright killed. Sure the lawsuit was a pain. And book publishers are turds for arguing that they still have rights to books that they won't publish ever again. But the courts found that there was nothing wrong with Google having the information[1]. That trove of text could be the worlds greatest source of knowledge but as we all know, people using internet search for work never click on ads and not enough of them are willing to pay a subscription service price to cover the cost of infrastructure. Google hoped that at one time they would make money by printing on demand those books that were out of print but people wanted, but that was shot down by short sighted publishers and agents. Perhaps it will be taken up by Amazon which has the resources to do it. [1] https://arstechnica.com/tech-policy/2015/10/appeals-court-rules-that-google-book-scanning-is-fair-use/ https://arstechnica.com/tech-policy/2015/10/appeals-court-ru...
- porpoisely 8y ago
  It is sad. But it's not just google books, it's google search, google news, youtube, etc. There was a time when all of google's properties catered to the users. Their search engine was the best. Google news was the best aggregate site. Youtube recommends used to be amazing to the point you could spend hours following their recommends. Now google search, google news, youtube, etc are all garbage. It doesn't serve the people. It serves corporate interests. You can thank media companies and the elites who pressured them for that.
  - ChuckMcM 8y ago
    No need to cast aspersions on 'elites' and 'corporate interests', what is pressuring them is that the ratio of the amount of money coming in to that going out, has to be maintained at a certain level for Google to remain Google. They really have only two choices there, either sell more ads (generally means putting ads on more things, or coming up with new ways to charge for new things like being in the 'shopping' box on product searches) or cut costs which means shutting down projects, reducing staff, Etc. Depending on how you look at it, Google gets something like one 500th of what they used to get for an ad on their web site.
babalulu 8y ago
Google Books has issues of Billboard magazine dating back to 1942. It used to be valuable for research, but it's become much less so over the years. Currently, search results that return actual magazine issues are limited to the first page. After that, it's just normal Google links. Even searching for something like Elvis or Glenn Miller, both of whom should have been in a crapload of issues, returns only one page of relevant results. Trying to search by date is very hard. Limiting search to "Glenn Miller October 1942" might return one or two relevant results, or it might not return any. Trying to search by issue date doesn't work at all. They have an index of Billboard issues which allows you to go to individual issues and read them, but the index stops at 50 pages, and for a weekly magazine, that limits the index to only a handful of years. Using the index, you can't go directly to issues before the 1980s, and with search by issue date useless, that means you're just out of luck if you want to see a particular issue in the 1970s.
- rasz 8y ago
  They do seem to be crippling book search on purpose. Just yesterday I was looking for "PC Mag 1997 january Pentium MMX" and Google refused to return PC Mag 7 Jan 1997 issue results, whats even more weird clicking "browse all issues" returns The requested URL /books/serial/ISSN:08888507?rview=1 was not found on this server but "About this magazine" will happily give you a list of all scanned issues :o and opening january one will let you search it and will return positive results.
gwern 8y ago
> If I worked at Google, I would have implemented a text-based date-prediction algorithm to flag erroneously classified books. (I have actually done this and sent a list to the HathiTurst of books they may have erroneously released into the public domain. It works). With friends like these, who needs enemies?
deleted 8y ago
[deleted]
philipkglass 8y ago
If you're using HathiTrust seriously and aren't affiliated with a partner library, consider Hathi Download Helper to get complete public domain books archived to local storage. I wrote an earlier command-line version of the tool. Someone else built a GUI and put in the work to keep up with the evolving API. I often use Google to locate a book, then check Internet Archive and HathiTrust if it's old enough that it should be public domain under US law. I really appreciate HathiTrust putting in the effort to check copyright renewals and make more of their materials fully visible. I don't appreciate the technical barriers to downloads that they erect, but that's out of the hands of the developers working there. As long as their web viewer shows individual pages you can be sure there will be a way to reassemble full books.
- robin_reala 8y ago
  Hey, thanks for this. The Hathi interface is slow to say the least, so it ends up being my fallback to archive.org more often that not.
- mimixco 8y ago
  Once I needed a copy of The Congressional Record from the 40's and Google had marked it non-public! I had to make several calls to Hathi to explain that the Record is public by law and then they had to contact Google to get the restriction lifted. It's a lot of rigamarole for information that researchers need.
- mimixco 8y ago
  Thanks for that tip and for writing the software! I had to play that game of manually assembling PDF pages for an old magazine article lately.
minikites 8y ago
It's fascinating to me that I see so many comments arguing for the freedom of information (not required to stay alive) where the same commenters argue against taxation or basic income.
- TeMPOraL 8y ago
  Do you have a spreadsheet mapping HN user handles to their positions on taxation and UBI? HN has a lot of users. There's often a pretty low overlap of commenters between different submissions.
  - minikites 8y ago
    My comments seem to strike a chord with a lot of HN users, I've had more than one person go through my comment history and reply to days or weeks old comments of mine.
EliRivers 8y ago
I completed a Masters in Mathematics back in 2014, involving a lot of historical research into the development of geometry. At the time, the ease with which I could open up books written over a century ago, search them and read them, was a fantastically useful tool. I have some of those sources still on my hard drive in their scanned PDF format. They've now effectively vanished from the open internet. So much, available for such a short window. Our children will never believe us when we tell them what was once right there at our fingertips, and those that do should never forgive us.
- toomuchtodo 8y ago
  Consider uploading them to the Internet Archive.
amelius 8y ago
What would be a scientific approach to compare search results? Let's say I do a search on DDG and on Google, how do I determine which engine provided the most accurate results?
- brownbat 8y ago
  I think if you have a population of people doing the same search, split randomly across the two sites, things like how long it takes to leave your site through a search result, how far down that result is, and how often people come back to rephrase their search are all good metrics. Not sure there's an answer for a single search for a single individual though.
nottorp 8y ago
I thought google search (without "books") is broken... why is the books search being broken a surprise?

134 points by longdefeat 8y ago

Based on my experience with research using Google Books and Hathi Trust, this author is correct. Google has purposely broken Google Books so that it doesn't compete with any paid sources for these materials -- even if there aren't any paid sources and the book is out of print! The TL;DR is that Google Books started out with the goal of digitizing every book ever written. Publishers sued, so they crippled their search and display functions and handed over the full texts they already had to a group called Hathi Trust. Hathi Trust is seriously crippled on purpose. It only allows access to full texts when you are sitting in a physical library of one its partner universities. That's right... I can drive to a big university, sit in their library, and download a full PDF of any book I like. But if I'm at my house, I can only read one page at a time in a browser. This is ridiculous. Hathi Trust is helping the oil business more than they're helping researchers. The marriage of Google Books content and Hathi Trust as a distribution platform is a joke. In some cases, you will even have to order an old book from interlibrary loan (see worldcat.org) if you can even get it -- when all the while Google has a scanned copy!

kevin_thibedeau 8y ago
If you were an author of a book with active copyright you might want to get paid for it.
- mimixco 8y ago
  As a matter of fact, I am. I'm speaking here of old, out-of-print books, where this is very much a problem with Google Books and Hathi Trust.
  - jandrese 8y ago
    It's also a fundamental problem with our current copyright law where material remains in copyright for many decades after it has stopped being printed and sold through primary channels. Copyright should automatically expire 10 or 15 years after the last printing IMHO. If nobody cares enough to put it up for sale or even make a tiny print run just to renew the copyright, why should the government continue to enforce it? Of course this scheme falls apart a bit in the digital age, except that even ebooks get pulled from the shelves for no apparent reason. Maybe we should just go back to having to explicitly renew copyright after 15 years or so, with a fee just large enough to convince people to drop dormant works. Maybe a couple hundred bucks every 5 years. The best part would be having some easily accessed online system where you could check the copyright status of any work, including current contact information for the rightsholders if you want to arrange payment.
    - izacus 8y ago
      Expiring copyright after 15 years no matter the printing might even be a better solution. That would allow other people to build and extend works, movies and software. 15 years is plenty to make a mountain of cash.
      - TeMPOraL 8y ago
        Yup. The way I see it, a lot of IP problems are caused by people who want to take the law intended for promoting new works, and turn it into a source of passive income.
        jandrese 8y ago
        But people wouldn't be encouraged to make original works if their great grandchildren would be robbed of the opportunity to squabble over the rights to it in the courts.
- Spooky23 8y ago
  How? My grandfather wrote a book in the 1940s that’s been out of print since the mid 50s. Every entity associated with the book is dead, including the publisher, which merged into another in the late 50s and is probably an inactive imprint of some successor company. Grandpa died in 1985, and a cousin or I is likely the heir to his rights. I have a copy of the book, which I bought via Alibris from a bookstore in Wales 15 years ago. If you needed the book for research, you’d probably get it via inter-library loan from a university or a big city library. Whomever the publisher is, they don’t have it and aren’t selling it. In no scenario does anyone get paid for transacting, other than a reseller or the post office.
  - greglindahl 8y ago
    That's a classic "orphan copyright" situation.
    - ghaff 8y ago
      Right. And orphan works legislation has tended to be opposed by individual creators or at least the organizations that purport to represent them. The (not totally unreasonable) theory is that individuals or their estates--let's leave aside the fact that copyright terms are almost certainly too long--may well inadvertently not keep on top of what's needed to keep works non-orphaned. But "Disney" (or whoever) will most certainly be ready to pounce on anything they can acquire for free for whatever technical reason.
  - mimixco 8y ago
    Right... And since Google Books probably has a scan, who cares if you download the PDF? This problem is impeding real research without helping anyone.
  - dwyerm 8y ago
    Stay with me for a second; I'm going to go wildly afield... does intellectual property need an Adverse Possession law, too? [1] Your grandfather had a book, the rights of which should have been presumably passed down to you. My grandfather had a patented mining claim that has been passed down to me. Where the ownership of your grandfather's book is questionable, for me the physical corners of the property are questionable. A number of them are defined by things like a "4 foot spruce post" or a "12 inch diameter tree trunk" that haven't survived the ravages of time. But it is important that I patrol my property at least every couple of years because of Adverse Possession. If someone else were to use my property continuously and I don't say anything against it, one day their trespassing suddenly and magically would become ownership. For a land-owner, it is a scary idea that someone can steal my property from me, as has actually happened. [2] But I can acknowledge that it makes some sense. It comes from the idea that land is meant to be used, and if you aren't using it, maybe the person who is using it should get the rights. If nobody can stand up for an intellectual property claim, perhaps some kind of adverse possession is in order. [1] https://en.wikipedia.org/wiki/Adverse_possession https://en.wikipedia.org/wiki/Adverse_possession [2] http://articles.latimes.com/2007/dec/03/nation/na-land3 http://articles.latimes.com/2007/dec/03/nation/na-land3
- dredmorbius 8y ago
  Hathi will not provide access to even out-of-copyright works. LibGen has no such problems.
deleted 8y ago
[deleted]

drallison 8y ago

Google Books seems to have major problems. An alternative interested parties should explore is Archive.org. The Internet Archive has a significant collection of scanned books and other materials.

oasisbob 8y ago

This breakage reminds me of similar constrained-search problems with the Google Groups/ Dejanews USENET archive. Once upon a time it was nice to research with.

darkpuma 8y ago
It reminds me of the demise of google's code search and github's woefully inadequate code search. Debian's Code Search is okay, but github allowing it would be great.

greglindahl 8y ago

Here's an example of an exploration tool built using the content of books: https://books.archivelab.org/dateviz/ https://books.archivelab.org/dateviz/ It would be nice if anyone could build such tools, but all of that data is locked up inside of places like Google Books and Hathi Trust. Google isn't even interested in making their metadata available, other than by running searches.

ccleve 8y ago

I wonder if the date problem is a bug, not a feature? Is it possible to dump the metadata of a book and check if they have the right date? There should probably be multiple dates for a book -- date written, date copyrighted, date published, date of latest edition, etc. My guess is that Google does not have a publicly-available issue tracker for Google Books so you can't easily report this problem. Hacker News is a good way to get their attention, though...

phonon 8y ago

Using https://babel.hathitrust.org/cgi/ls?a=srchls&anyall1=all&q1=%22set+in++stone%22&field1=ocr&op3=AND&yop=before&pdate_end=2002&facet_lang=&facet_format= https://babel.hathitrust.org/cgi/ls?a=srchls&anyall1=all&q1=... I get 14,515 results, with 3,115 of them full view. There is also https://analytics.hathitrust.org/ https://analytics.hathitrust.org/ which seems interesting!

dtagames 8y ago
And all of them are full view if you are physically sitting in a Hathi Trust "partner" university library. These libraries are open to the public and allow downloading and saving of the materials you browse, making the whole point of locking them up completely pointless.
- elektor 8y ago
  Is it then possible for an Aaron Schwartz/ Sci Hub character to then download all of the available books and make them available on the internet?
  - sodosopa 8y ago
    yes and that’s theft.
- toomuchtodo 8y ago
  Mind emailing me? You don’t have contact info in your profile. I’d be interested in hanging at a library for a few hours with my laptop and have some questions before going.
  - mimixco 8y ago
    To whom is this directed? If it's to me, just write your question here. I've pretty much already explained how it works. Go to the Hathi Trust login screen. Pull down the list of partner institutions. Go to one of them in person. (or get a Library of Congress card, also in person.) After the institution logs in, you can download anything you want. Bring a flash drive or portable hard disk and take home your PDFs. I'd recommend calling the partner library first to make sure someone there knows the Hathi login. It's not that popular a resource, sadly, and many people have no idea what it is. You may also find a friendly librarian who is willing to do the download for you and email you the PDF, saving an in-person visit. One thing to note: You don't have to be a student to use the resources of university libraries. They're open to everyone.

jetrink 8y ago

Searching within books is broken too. If you search for 'cat', any instance of the word 'cats' will also be returned (and any other word beginning with 'cat'), but with the message, "No preview is available." No link to the page of the result is provided. It seems like the kind of bug that should be straightforward to locate and fix, but it has been this way for years. (My guess is that the tool that builds previews doesn't recognize partial matches.)

mimixco 8y ago
It's not a bug. It's on purpose. They actively limit previews, full page views, and downloads -- even though you can go to a university and download the same book from Hathi Trust. Hathi Trust isn't paying the copyright holders, either, so who cares?

mimixco 8y ago

Here's an easy example... Google Books has many old issues of Popular Mechanics which exist in full downloadable form on the Internet Archive -- yet you can only see individual pages on Google Books and can't download the magazine. This is because Google Books is acting like they own the copyright (or, at least, they feel the need to police it.) There are many cases where you can download the entire book from Hathi Trust when you are sitting at a university library, giving you a PDF you can use anywhere. But you cannot even see the entire book or download it from Google Books (which has given its scan to Hathi). This is just stupid.

ppod 8y ago

Google tried to make this work, but they were sued; and then they made a deal, and then many, many people objected to the resulting deal. This is the usual process whereby a corporation is first criticised for having too much power, and then when they relinquish power they are criticised for not doing enough. https://www.theatlantic.com/technology/archive/2017/04/the-tragedy-of-google-books/523320/ https://www.theatlantic.com/technology/archive/2017/04/the-t... The end of that article is a not-so-subtle plea for someone within google to perhaps accidentally anonymously place this material in public.

mimixco 8y ago
Exactly my point. The "deal" resulted in anyone being able to download the full books if they go to a university library which is a partner of Hathi Trust. Of course, after that download, you have the full PDF and you can do whatever you want! This isn't protection for copyright holders. (Hathi Trust got its books from Google Books scans and doesn't pay copyright holders.) This "deal" isn't helping anyone and it hurts researchers.
- ahi 8y ago
  HathiTrust does not provide access to in copyright materials to anyone. They provide research access for "non-consumptive" use through the HathiTrust Research Center.
  - mimixco 8y ago
    That's not correct. First of all, all books are copyright by their authors. Anyone can go to a partner library (there's a list on the login page of the Hathi site) and download those books as PDFs. Try it! I've done it many times.
    - ahi 8y ago
      These pages provide a decent overview of access restrictions for HathiTrust materials: https://www.hathitrust.org/features_benefits https://www.hathitrust.org/features_benefits https://www.hathitrust.org/out-of-print-brittle https://www.hathitrust.org/out-of-print-brittle

mimixco 8y ago

Here's another little-known fact: Every American is entitled to a library card from the Library of Congress (you have to go there in person to get one). The Library of Congress is a Hathi Trust partner! So if you go get that card, you can download all of the out-of-print books that Google scanned on your own computer. No copyright holders are getting paid (and no one is being harmed), so why all these barriers in-between?

sodosopa 8y ago
No copyright holders being paid is theft. Authors write and should be paid for their efforts. This is glorified and legalized privacy.
- mimixco 8y ago
  No, we're talking about out-of-print books. There is no one around to claim copyright or payment. No one is trying to download Harry Potter from Google Books, lol!
- Fnoord 8y ago
  > No copyright holders being paid is theft No, it isn't. It is copyright infringement. Theft is something completely different. Different laws apply.
  - sodosopa 8y ago
    If you take money out of an author/musician/artist’s hand for a work they created. You can call it anything you desire but it’s theft and piracy.
    - Fnoord 8y ago
      So if I decide not to take the bus its theft as well? If I decide to steal a bicycle, its theft from the bus company?
      - 8y ago

ChuckMcM 8y ago

It isn't surprising, this bit is sad: Google Books has failed to live up to its promise as the company has moved away from its original mission of organizing information for people. Google was only about organizing all the worlds information while search ads was an unlimited fountain of money. As Google's ability to generate money with search ads has dwindled, their more grand (and not monetizable) projects have been either starved for resources or outright killed. Sure the lawsuit was a pain. And book publishers are turds for arguing that they still have rights to books that they won't publish ever again. But the courts found that there was nothing wrong with Google having the information[1]. That trove of text could be the worlds greatest source of knowledge but as we all know, people using internet search for work never click on ads and not enough of them are willing to pay a subscription service price to cover the cost of infrastructure. Google hoped that at one time they would make money by printing on demand those books that were out of print but people wanted, but that was shot down by short sighted publishers and agents. Perhaps it will be taken up by Amazon which has the resources to do it. [1] https://arstechnica.com/tech-policy/2015/10/appeals-court-rules-that-google-book-scanning-is-fair-use/ https://arstechnica.com/tech-policy/2015/10/appeals-court-ru...

porpoisely 8y ago
It is sad. But it's not just google books, it's google search, google news, youtube, etc. There was a time when all of google's properties catered to the users. Their search engine was the best. Google news was the best aggregate site. Youtube recommends used to be amazing to the point you could spend hours following their recommends. Now google search, google news, youtube, etc are all garbage. It doesn't serve the people. It serves corporate interests. You can thank media companies and the elites who pressured them for that.
- ChuckMcM 8y ago
  No need to cast aspersions on 'elites' and 'corporate interests', what is pressuring them is that the ratio of the amount of money coming in to that going out, has to be maintained at a certain level for Google to remain Google. They really have only two choices there, either sell more ads (generally means putting ads on more things, or coming up with new ways to charge for new things like being in the 'shopping' box on product searches) or cut costs which means shutting down projects, reducing staff, Etc. Depending on how you look at it, Google gets something like one 500th of what they used to get for an ad on their web site.

babalulu 8y ago

Google Books has issues of Billboard magazine dating back to 1942. It used to be valuable for research, but it's become much less so over the years. Currently, search results that return actual magazine issues are limited to the first page. After that, it's just normal Google links. Even searching for something like Elvis or Glenn Miller, both of whom should have been in a crapload of issues, returns only one page of relevant results. Trying to search by date is very hard. Limiting search to "Glenn Miller October 1942" might return one or two relevant results, or it might not return any. Trying to search by issue date doesn't work at all. They have an index of Billboard issues which allows you to go to individual issues and read them, but the index stops at 50 pages, and for a weekly magazine, that limits the index to only a handful of years. Using the index, you can't go directly to issues before the 1980s, and with search by issue date useless, that means you're just out of luck if you want to see a particular issue in the 1970s.

rasz 8y ago
They do seem to be crippling book search on purpose. Just yesterday I was looking for "PC Mag 1997 january Pentium MMX" and Google refused to return PC Mag 7 Jan 1997 issue results, whats even more weird clicking "browse all issues" returns The requested URL /books/serial/ISSN:08888507?rview=1 was not found on this server but "About this magazine" will happily give you a list of all scanned issues :o and opening january one will let you search it and will return positive results.

gwern 8y ago

> If I worked at Google, I would have implemented a text-based date-prediction algorithm to flag erroneously classified books. (I have actually done this and sent a list to the HathiTurst of books they may have erroneously released into the public domain. It works). With friends like these, who needs enemies?

deleted 8y ago

[deleted]

philipkglass 8y ago

If you're using HathiTrust seriously and aren't affiliated with a partner library, consider Hathi Download Helper to get complete public domain books archived to local storage. I wrote an earlier command-line version of the tool. Someone else built a GUI and put in the work to keep up with the evolving API. I often use Google to locate a book, then check Internet Archive and HathiTrust if it's old enough that it should be public domain under US law. I really appreciate HathiTrust putting in the effort to check copyright renewals and make more of their materials fully visible. I don't appreciate the technical barriers to downloads that they erect, but that's out of the hands of the developers working there. As long as their web viewer shows individual pages you can be sure there will be a way to reassemble full books.

robin_reala 8y ago
Hey, thanks for this. The Hathi interface is slow to say the least, so it ends up being my fallback to archive.org more often that not.
mimixco 8y ago
Once I needed a copy of The Congressional Record from the 40's and Google had marked it non-public! I had to make several calls to Hathi to explain that the Record is public by law and then they had to contact Google to get the restriction lifted. It's a lot of rigamarole for information that researchers need.
mimixco 8y ago
Thanks for that tip and for writing the software! I had to play that game of manually assembling PDF pages for an old magazine article lately.

minikites 8y ago

It's fascinating to me that I see so many comments arguing for the freedom of information (not required to stay alive) where the same commenters argue against taxation or basic income.

TeMPOraL 8y ago
Do you have a spreadsheet mapping HN user handles to their positions on taxation and UBI? HN has a lot of users. There's often a pretty low overlap of commenters between different submissions.
- minikites 8y ago
  My comments seem to strike a chord with a lot of HN users, I've had more than one person go through my comment history and reply to days or weeks old comments of mine.

EliRivers 8y ago

I completed a Masters in Mathematics back in 2014, involving a lot of historical research into the development of geometry. At the time, the ease with which I could open up books written over a century ago, search them and read them, was a fantastically useful tool. I have some of those sources still on my hard drive in their scanned PDF format. They've now effectively vanished from the open internet. So much, available for such a short window. Our children will never believe us when we tell them what was once right there at our fingertips, and those that do should never forgive us.

toomuchtodo 8y ago
Consider uploading them to the Internet Archive.

amelius 8y ago

What would be a scientific approach to compare search results? Let's say I do a search on DDG and on Google, how do I determine which engine provided the most accurate results?

brownbat 8y ago
I think if you have a population of people doing the same search, split randomly across the two sites, things like how long it takes to leave your site through a search result, how far down that result is, and how often people come back to rephrase their search are all good metrics. Not sure there's an answer for a single search for a single individual though.

nottorp 8y ago

I thought google search (without "books") is broken... why is the books search being broken a surprise?