9 ms·

How Google Book Search Got Lost

155 points by chrismealy 10y ago

deleted 10y ago
[deleted]
sebisebi 10y ago
The articles alludes to it at the end: the "corpus" of scanned books is incredibly valuable to a big data company like google and gives them a real edge against other companies.
- WalterBright 10y ago
  If it is incredibly valuable, other companies can do it too, and sell access to the corpus to other companies.
  - _rpd 10y ago
    Seems a natural investment for Amazon.
    - greglindahl 10y ago
      Here's Amazon's "search inside" program, which started in 2001: http://newsbreaks.infotoday.com/NewsBreaks/Search-Inside-the-Book-FullText-on-Amazon-16587.asp http://newsbreaks.infotoday.com/NewsBreaks/Search-Inside-the...
  - ot 10y ago
    Wouldn't they also run into the licensing problem, even more so if they try to sell access?
dzdt 10y ago
This is the value-destruction side of copyright protection. With more reasonable length and more easily determinable copyright status, efforts like google books would be able to do so many more things. But without a permissive legal framework that innovation is shut down.
- baldfat 10y ago
  Blame it 100% on Micky Mouse and Disney that is where all the blame belongs.
  - smnrchrds 10y ago
    Here is what I don't understand: how come copyright gets expanded left and right but patent does not. Does Disney the entertainment industry have more power than the entirety of traditional industries which depend on patents to make a profit (pharmeceutical, manufacturing, electronics, etc.) combined?
    - unabridged 10y ago
      For patents you have companies fighting on the other side to limit them, so they can use the technology eventually. For copyrights there is no resistance with deep pockets.
    - Houshalter 10y ago
      A lot of businesses benefit from expired patents. Any business that uses technology would be impossible if patents never expired. No business really benefits that much from expired copyright. When Disney lobbies to extend copyright, who is going to spend millions to fight them?
      - baldfat 10y ago
        > who is going to spend millions to fight them? I am hopping one day everyone will be feed up and there will be millions in funding. Personally I don't think this gets fixed till we can give special copyright to Disney and other corporations and free up the other 99.5% of copyright.
        ImSkeptical 10y ago
        That seems like a great compromise. Pay a ten million dollar fee to extend your copyright by ten years. Most material becomes public domain, some exceptionally valuable properties remain privately held and generate value for the holder. Plus, we get some additional tax revenue.
WalterBright 10y ago
> "They should have just licensed the books instead.” The problem with orphaned works is that can't be done, as nobody knows who owns them.
- CobrastanJorji 10y ago
  One presumes that, as the speaker is the Author's Guild, they would probably be happy to fix that problem by accepting the licensing fees for orphaned works themselves.
  - greglindahl 10y ago
    Indeed, that was attempted, but it was vetoed by the judge: https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,_Inc.#Proposed_settlement_agreement https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,...
  - ghaff 10y ago
    Furthermore, there's a fair bit of pushback by many content creators around orphan works legislation. This has been perhaps most pronounced with photography. There seems to be a concern that big corporations won't try very hard to reach people who have neglected to renew copyrights and will snap up their works. (Not saying I agree with this POV but it's out there.)
    - pbhjpbhj 10y ago
      >people who have neglected to renew copyrights // Renewal, indeed registration, is largely a USA thing since the Berne Convention (adopted nearly everywhere before USA signed in 1988) did away with the need for registration in the 19th Century. Worth bearing in mind, the USA as with "English" measurement is a good step out of line with the practice in the rest of the world. Any argument based on someone not registering is going to need completely rethinking outside USA.
      - ghaff 10y ago
gridit 10y ago
"If Google could find a way to take that corpus, sliced and diced by genre, topic, time period, all the ways you can divide it, and make that available to machine-learning researchers and hobbyists at universities and out in the wild, I’ll bet there’s some really interesting work that could come out of that. Nobody knows what,” Sloan says. He assumes Google is already doing this internally. Jaskiewicz and others at Google would not say." For books that are scanned, but with no extra licensing, would Google be allowed to do anything with the data? Create a very delocalized n-gram set? Use it as the "test" set (not even cross-validation, where it might influence hyperparams) for a ML algorithm? Edit: would love to know where google's authorization derives from, with the ngram set. Somewhere in the Judge's orders? A negotiated fee with the Authors Guild?
- SomewhatLikely 10y ago
  They already have. http://commondatastorage.googleapis.com/books/syntactic-ngrams/index.html http://commondatastorage.googleapis.com/books/syntactic-ngra...
  - yohui 10y ago
    See also Google Ngram Viewer: https://books.google.com/ngrams https://books.google.com/ngrams Wikipedia entry: https://en.wikipedia.org/wiki/Google_Ngram_Viewer https://en.wikipedia.org/wiki/Google_Ngram_Viewer
- deleted 10y ago
  [deleted]
- gridit 10y ago
  Ok, here is one of the important opinions in the Google Books settlement, by Judge Chin in 2013 [0]. He basically says (paraphrasing), "I'm going to assume Google has violated copyright by creating digital copies and serving them. But it's fair use, because the new products are transformative". For example, re:ngrams """ Similarly, Google Books is also transformative in the sense that it has transformed book text into data for purposes of substantive research, including data mining and text mining in new areas, thereby opening up new fields of research. Words in books are being used in a way they have not been used before. Google Books has created something new in the use of book text-the frequency of words and trends in their usage provide substantive information. [...] On the other hand, fair use has been found even where a defendant benefitted commercially from the unlicensed use of copyrighted works """ Oh man, this is mind-blowing. [0] https://copyright-casebook.com/about/recent-cases-edited/authors-guild-v-google-inc-954-f-supp-2d-282-s-d-n-y-2013/ https://copyright-casebook.com/about/recent-cases-edited/aut...
greglindahl 10y ago
If you'd like to see an example of what innovative things you can do with book contents: https://blog.archive.org/2016/02/09/how-will-we-explore-books-in-the-21st-century/ https://blog.archive.org/2016/02/09/how-will-we-explore-book... https://books.archivelab.org/dateviz/ https://books.archivelab.org/dateviz/
- pbhjpbhj 10y ago
  I'm not that impressed, am i missing something? Looks like pre-indexed search matched with full-text search??
  - greglindahl 10y ago
    It's an index of concepts that appear in a sentence with dates. The blog post shows an example of how this type of index surfaces important dates for the concept 'Gregorian Calendar'.
johansch 10y ago
(Is there some protest somewhere I can join to fight this idiotic image-fade in effect? You know, the one made universally hated by medium.com.)
- analogmemory 10y ago
  You want to protest lazy loading images? Cool.
  - johansch 10y ago
    Yup.
- vultour 10y ago
  That idiotic effect saves bandwidth and allows the page to load more quickly.
  - johansch 10y ago
    I have plenty of bandwidth. (Also: exactly what bandwidth does it save, even for those that have little bandwidth? Aren't browsers better equipped to decide what images to load or not?) The page doesn't load any faster; the only effect is that I see a stupid fade-in instead of seeing the page with the image more or less immediately. The Cloudflare host serving the image is a 4 ms roundtrip away. The bandwidth is 250/100 Mbit/s. #ab -n 100 -c 10 "https://cdn-images-1.medium.com/max/1500/1*UW2Mz4yMD65VYgFZILiL_w.png" .. Requests per second: 113.23 [#/sec] (mean) Time per request: 88.314 [ms] (mean) Time per request: 8.831 [ms] (mean, across all concurrent requests) Transfer rate: 25649.12 [Kbytes/sec] received So to summarize my points: - The javascript-driven delayed loading + fade-in is completely unnecessary and primarily distracting for people with high bandwidth connnections. - The browser is in a better spot to figure out how to handle the loading of these images. Just make sure the dimensions are set so that the layout isn't modified after the images have loaded. - The fade-in effect was novel the first week, a couple of years ago, but is now just annoying. If any of the client-side developers who no doubt downvoted this sub-thread would step up to actually talk about it, it would be quite welcome. :)
amelius 10y ago
Perhaps they can dump it as a torrent, so any hacker could build a valuable service with it. See e.g. SciHub.
theparanoid 10y ago
[deleted]
- draw_down 10y ago
  Occam's Razor: Copyright holders prevented it from being useful
- pc2g4d 10y ago
  > "the ranger account of a bear my dad fought in Yellowstone" Link?
petra 10y ago
It's not terribly hard to think of good uses for Google Books[1]. It's just, the legality was a bit murky, and what are the incentives ? One idea(among many, surely): many people prefer visual explanations. In many subject areas, books offer better visual explanations. If when searching for something, Google would have also linked to some visual explanations from books(in the main search), and used machine learning to find the best - that may be really great, may really improve the experience and value of the web. But is it legal ? it's unclear. Can Google monetize that ? probably no. Or am i wrong, is there some way Google can monetize that?
- JTon 10y ago
  > Can Google monetize that ? probably no. I'm sure Google could work out some affiliate fee/scheme for directing the user to a marketplace to buy the book whose image is shown. But if Google directs the user to it's own marketplace, it would surely get a cut, no?
  - petra 10y ago
    That's a good idea and Google is already putting Amazon links in Google Books. The US publishing industry is $28B. technical/educational books are 50% of that, And Google could only get 10% for that market(many other channels), and let's say they'll get 10% affiliate fee. that equals 28B * 0.5 * 0.1 * 0.1 = $140 million/year , for what they do currently. Books are quite expensive, so adding visual links to the main search engine, would say double that - $280 million/yr. Not much for Google. But on the negative side, adding visual links will distract people from surfing to sites filled with ads,clicking search ads, etc - so maybe it will cost Google money. Maybe a lot of money. That could be the logic behind why Google Forums, clearly a valuable service, was dumped[1]. So Google probably needs a more serious way to monetize it to justify the effort.
- seanp2k2 10y ago
  "Google's mission is to organize the world's information and make it universally accessible and useful." Literally why Google exists: https://www.google.com/intl/en/about/ https://www.google.com/intl/en/about/
The_ed17 10y ago
This. Relatedly, losing an easy Google News Archive was killer for some of the research I'd like to do. Several papers/articles I wrote in c. 2010 would not be possible to do today.
- greglindahl 10y ago
  Common Crawl has a new news archive started a few months ago (http://commoncrawl.org/2016/10/news-dataset-available/ http://commoncrawl.org/2016/10/news-dataset-available/) and the Internet Archive has had one going for quite a while.
  - The_ed17 10y ago
    Thanks for this! I'm talking about old scanned newspapers. :-) The Internet Archive has a good start, but it's pretty heavy on Kentucky, and few have in-text search available, which is killer if you're researching an event with few/no specific dates. (That's not to knock them—IA is pretty amazing, and OCRing newspapers is notoriously difficult.)
- cdibona 10y ago
  Do you mean the old deja thing? We (google) got a copy to the archive years ago.
  - The_ed17 10y ago
    This reply got a bit convoluted. My apologies. First, I'm referring to this: https://www.theatlantic.com/technology/archive/2011/05/google-shuts-down-newspaper-archive-project/239239/ https://www.theatlantic.com/technology/archive/2011/05/googl... It's still /technically/ possible to search what's there via https://news.google.com/newspapers https://news.google.com/newspapers. Still, it's not exactly user-intuitive, and not being able to sort/search by date can make historical research very difficult (especially when the OCR isn't perfect—that's common, but trying several different phrases to make sure you've found everything is way easier when searching range of years). Some related thoughts can be found in an old Hacker News post: https://news.ycombinator.com/item?id=7408034 https://news.ycombinator.com/item?id=7408034 Online newspaper archives are a ridiculously awesome boon for the humanities. Chronicling America from the Library of Congress, for instance, is great. It's the de facto successor to Google News Archive in the US. I just wish that Google News Archive could get a couple of the old search features back to aid researchers. :-) Second, on a quick tangent I just discovered: when you select "archives" at news.google.com, it says "looking for scanned newspapers?" with a link to: https://support.google.com/news/answer/3334 https://support.google.com/news/answer/3334. But there's nothing there anymore about scanned news. :-)
ChuckMcM 10y ago
I think Book Search is entirely a red herring, one of Google's many 'say one thing, do something entirely different.' Kind of herring. A great example I know of personally was the Google 411 project. Google 411 was a project which offered 'directory services' by voice over your phone. It was pretty cool, you could call its 800 number, ask for a listing, and it would use Google search to find it and then read it back to you. Then you got to tell it how well it did. People started using it, adoption spread, and then they cut it off. So why did Google run this service in the first place? Was it to see if they could make a business out of directory services? No, it was to collect a data set where a spoken phrase could be matched to its exact translation (the thing they were looking for) which could then be processed and re-processed to train algorithms for independent speaker recognition. On the one hand they could try to pay a million people to come in and say something and then confirm or deny they understood what they said, or they could use a bit of spare hardware and collect that information with out paying anyone a dime. They own that data set, it is extremely valuable for testing improvements in voice recognition, and it is yet another barrier to a new company trying to get into that space. Now let's look at Google Books. The 'story' was digitize all the books and make a great library available, and maybe even offer up PDF copies of out of print books for people. It set off a legal firestorm (as mentioned in the article) it got all of the archivists on board and libraries contributed millions of volumes, and Google got a ruling out of the Supreme Court that said that digitizing a book was not an act of copyright infringement. But the subtle part is that while folks say 'everything is on the web' (and that may be true) literally 99.99% of everything on the web is complete and utter crap, written by people who are seeking to game advertising selection engines not digitize information. Most of the stuff in books is not crap, because it cost someone significant time and labor to take that information and publish it. (romance novels excluded :-)). Google had digitized the single largest collection of human knowledge ever, and put it in a form where 100's of thousands of machines can process and re-process it to derive ontological accurate facts into the largest knowledge base in history. If you want to test whether your algorithm can identify credible information, there is no better way to do it than to prime it with a ground truth which has much higher credibility than most of the accessible data out there. That data set exists, they own it, and they don't have any obligation to share it with anyone. And it will allow for the creation of trained models for differentiating fact from fiction, jest for insult, and command from comment. It is my opinion that any company that wants to be 'serious' about AI better have access to an equivalent data set or they will lose.
ma2rten 10y ago
And also: the dawning realization that Scanning All The Books, however useful, might not change the world in any fundamental way. I think this is actually the main takeaway.
- douche 10y ago
  Remember when ReCaptcha used Google Books snippets? You may not, it's been swapped over to Street View images for a while. But Google Books has been a big deal for automated OCR.
douche 10y ago
I'm sad that outfits like Project Gutenberg are stonewalled at 1923 or whatever year Steamboat Willie came out. There are so many good books whose authors' grandchildren are dead, that can't legally be reproduced.
- waqf 10y ago
  Library Genesis is your friend. (http://libgen.io http://libgen.io or http://gen.lib.rus.ec http://gen.lib.rus.ec, among other mirrors.)
known 10y ago
I believe PageRank is irrelevant to Google Books

155 points by chrismealy 10y ago

deleted 10y ago

[deleted]

The articles alludes to it at the end: the "corpus" of scanned books is incredibly valuable to a big data company like google and gives them a real edge against other companies.

WalterBright 10y ago
If it is incredibly valuable, other companies can do it too, and sell access to the corpus to other companies.
- _rpd 10y ago
  Seems a natural investment for Amazon.
  - greglindahl 10y ago
    Here's Amazon's "search inside" program, which started in 2001: http://newsbreaks.infotoday.com/NewsBreaks/Search-Inside-the-Book-FullText-on-Amazon-16587.asp http://newsbreaks.infotoday.com/NewsBreaks/Search-Inside-the...
- ot 10y ago
  Wouldn't they also run into the licensing problem, even more so if they try to sell access?

dzdt 10y ago

This is the value-destruction side of copyright protection. With more reasonable length and more easily determinable copyright status, efforts like google books would be able to do so many more things. But without a permissive legal framework that innovation is shut down.

baldfat 10y ago
Blame it 100% on Micky Mouse and Disney that is where all the blame belongs.
- smnrchrds 10y ago
  Here is what I don't understand: how come copyright gets expanded left and right but patent does not. Does Disney the entertainment industry have more power than the entirety of traditional industries which depend on patents to make a profit (pharmeceutical, manufacturing, electronics, etc.) combined?
  - unabridged 10y ago
    For patents you have companies fighting on the other side to limit them, so they can use the technology eventually. For copyrights there is no resistance with deep pockets.
  - Houshalter 10y ago
    A lot of businesses benefit from expired patents. Any business that uses technology would be impossible if patents never expired. No business really benefits that much from expired copyright. When Disney lobbies to extend copyright, who is going to spend millions to fight them?
    - baldfat 10y ago
      > who is going to spend millions to fight them? I am hopping one day everyone will be feed up and there will be millions in funding. Personally I don't think this gets fixed till we can give special copyright to Disney and other corporations and free up the other 99.5% of copyright.
      - ImSkeptical 10y ago
        That seems like a great compromise. Pay a ten million dollar fee to extend your copyright by ten years. Most material becomes public domain, some exceptionally valuable properties remain privately held and generate value for the holder. Plus, we get some additional tax revenue.

WalterBright 10y ago

> "They should have just licensed the books instead.” The problem with orphaned works is that can't be done, as nobody knows who owns them.

CobrastanJorji 10y ago
One presumes that, as the speaker is the Author's Guild, they would probably be happy to fix that problem by accepting the licensing fees for orphaned works themselves.
- greglindahl 10y ago
  Indeed, that was attempted, but it was vetoed by the judge: https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,_Inc.#Proposed_settlement_agreement https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,...
- ghaff 10y ago
  Furthermore, there's a fair bit of pushback by many content creators around orphan works legislation. This has been perhaps most pronounced with photography. There seems to be a concern that big corporations won't try very hard to reach people who have neglected to renew copyrights and will snap up their works. (Not saying I agree with this POV but it's out there.)
  - pbhjpbhj 10y ago
    >people who have neglected to renew copyrights // Renewal, indeed registration, is largely a USA thing since the Berne Convention (adopted nearly everywhere before USA signed in 1988) did away with the need for registration in the 19th Century. Worth bearing in mind, the USA as with "English" measurement is a good step out of line with the practice in the rest of the world. Any argument based on someone not registering is going to need completely rethinking outside USA.
    - ghaff 10y ago

gridit 10y ago

"If Google could find a way to take that corpus, sliced and diced by genre, topic, time period, all the ways you can divide it, and make that available to machine-learning researchers and hobbyists at universities and out in the wild, I’ll bet there’s some really interesting work that could come out of that. Nobody knows what,” Sloan says. He assumes Google is already doing this internally. Jaskiewicz and others at Google would not say." For books that are scanned, but with no extra licensing, would Google be allowed to do anything with the data? Create a very delocalized n-gram set? Use it as the "test" set (not even cross-validation, where it might influence hyperparams) for a ML algorithm? Edit: would love to know where google's authorization derives from, with the ngram set. Somewhere in the Judge's orders? A negotiated fee with the Authors Guild?

SomewhatLikely 10y ago
They already have. http://commondatastorage.googleapis.com/books/syntactic-ngrams/index.html http://commondatastorage.googleapis.com/books/syntactic-ngra...
- yohui 10y ago
  See also Google Ngram Viewer: https://books.google.com/ngrams https://books.google.com/ngrams Wikipedia entry: https://en.wikipedia.org/wiki/Google_Ngram_Viewer https://en.wikipedia.org/wiki/Google_Ngram_Viewer
deleted 10y ago
[deleted]
gridit 10y ago
Ok, here is one of the important opinions in the Google Books settlement, by Judge Chin in 2013 [0]. He basically says (paraphrasing), "I'm going to assume Google has violated copyright by creating digital copies and serving them. But it's fair use, because the new products are transformative". For example, re:ngrams """ Similarly, Google Books is also transformative in the sense that it has transformed book text into data for purposes of substantive research, including data mining and text mining in new areas, thereby opening up new fields of research. Words in books are being used in a way they have not been used before. Google Books has created something new in the use of book text-the frequency of words and trends in their usage provide substantive information. [...] On the other hand, fair use has been found even where a defendant benefitted commercially from the unlicensed use of copyrighted works """ Oh man, this is mind-blowing. [0] https://copyright-casebook.com/about/recent-cases-edited/authors-guild-v-google-inc-954-f-supp-2d-282-s-d-n-y-2013/ https://copyright-casebook.com/about/recent-cases-edited/aut...

greglindahl 10y ago

If you'd like to see an example of what innovative things you can do with book contents: https://blog.archive.org/2016/02/09/how-will-we-explore-books-in-the-21st-century/ https://blog.archive.org/2016/02/09/how-will-we-explore-book... https://books.archivelab.org/dateviz/ https://books.archivelab.org/dateviz/

pbhjpbhj 10y ago
I'm not that impressed, am i missing something? Looks like pre-indexed search matched with full-text search??
- greglindahl 10y ago
  It's an index of concepts that appear in a sentence with dates. The blog post shows an example of how this type of index surfaces important dates for the concept 'Gregorian Calendar'.

johansch 10y ago

(Is there some protest somewhere I can join to fight this idiotic image-fade in effect? You know, the one made universally hated by medium.com.)

analogmemory 10y ago
You want to protest lazy loading images? Cool.
- johansch 10y ago
  Yup.
vultour 10y ago
That idiotic effect saves bandwidth and allows the page to load more quickly.
- johansch 10y ago
  I have plenty of bandwidth. (Also: exactly what bandwidth does it save, even for those that have little bandwidth? Aren't browsers better equipped to decide what images to load or not?) The page doesn't load any faster; the only effect is that I see a stupid fade-in instead of seeing the page with the image more or less immediately. The Cloudflare host serving the image is a 4 ms roundtrip away. The bandwidth is 250/100 Mbit/s. #ab -n 100 -c 10 "https://cdn-images-1.medium.com/max/1500/1*UW2Mz4yMD65VYgFZILiL_w.png" .. Requests per second: 113.23 [#/sec] (mean) Time per request: 88.314 [ms] (mean) Time per request: 8.831 [ms] (mean, across all concurrent requests) Transfer rate: 25649.12 [Kbytes/sec] received So to summarize my points: - The javascript-driven delayed loading + fade-in is completely unnecessary and primarily distracting for people with high bandwidth connnections. - The browser is in a better spot to figure out how to handle the loading of these images. Just make sure the dimensions are set so that the layout isn't modified after the images have loaded. - The fade-in effect was novel the first week, a couple of years ago, but is now just annoying. If any of the client-side developers who no doubt downvoted this sub-thread would step up to actually talk about it, it would be quite welcome. :)

amelius 10y ago

Perhaps they can dump it as a torrent, so any hacker could build a valuable service with it. See e.g. SciHub.

theparanoid 10y ago

[deleted]

draw_down 10y ago
Occam's Razor: Copyright holders prevented it from being useful
pc2g4d 10y ago
> "the ranger account of a bear my dad fought in Yellowstone" Link?

petra 10y ago

It's not terribly hard to think of good uses for Google Books[1]. It's just, the legality was a bit murky, and what are the incentives ? One idea(among many, surely): many people prefer visual explanations. In many subject areas, books offer better visual explanations. If when searching for something, Google would have also linked to some visual explanations from books(in the main search), and used machine learning to find the best - that may be really great, may really improve the experience and value of the web. But is it legal ? it's unclear. Can Google monetize that ? probably no. Or am i wrong, is there some way Google can monetize that?

JTon 10y ago
> Can Google monetize that ? probably no. I'm sure Google could work out some affiliate fee/scheme for directing the user to a marketplace to buy the book whose image is shown. But if Google directs the user to it's own marketplace, it would surely get a cut, no?
- petra 10y ago
  That's a good idea and Google is already putting Amazon links in Google Books. The US publishing industry is $28B. technical/educational books are 50% of that, And Google could only get 10% for that market(many other channels), and let's say they'll get 10% affiliate fee. that equals 28B * 0.5 * 0.1 * 0.1 = $140 million/year , for what they do currently. Books are quite expensive, so adding visual links to the main search engine, would say double that - $280 million/yr. Not much for Google. But on the negative side, adding visual links will distract people from surfing to sites filled with ads,clicking search ads, etc - so maybe it will cost Google money. Maybe a lot of money. That could be the logic behind why Google Forums, clearly a valuable service, was dumped[1]. So Google probably needs a more serious way to monetize it to justify the effort.
seanp2k2 10y ago
"Google's mission is to organize the world's information and make it universally accessible and useful." Literally why Google exists: https://www.google.com/intl/en/about/ https://www.google.com/intl/en/about/

The_ed17 10y ago

This. Relatedly, losing an easy Google News Archive was killer for some of the research I'd like to do. Several papers/articles I wrote in c. 2010 would not be possible to do today.

greglindahl 10y ago
Common Crawl has a new news archive started a few months ago (http://commoncrawl.org/2016/10/news-dataset-available/ http://commoncrawl.org/2016/10/news-dataset-available/) and the Internet Archive has had one going for quite a while.
- The_ed17 10y ago
  Thanks for this! I'm talking about old scanned newspapers. :-) The Internet Archive has a good start, but it's pretty heavy on Kentucky, and few have in-text search available, which is killer if you're researching an event with few/no specific dates. (That's not to knock them—IA is pretty amazing, and OCRing newspapers is notoriously difficult.)
cdibona 10y ago
Do you mean the old deja thing? We (google) got a copy to the archive years ago.
- The_ed17 10y ago
  This reply got a bit convoluted. My apologies. First, I'm referring to this: https://www.theatlantic.com/technology/archive/2011/05/google-shuts-down-newspaper-archive-project/239239/ https://www.theatlantic.com/technology/archive/2011/05/googl... It's still /technically/ possible to search what's there via https://news.google.com/newspapers https://news.google.com/newspapers. Still, it's not exactly user-intuitive, and not being able to sort/search by date can make historical research very difficult (especially when the OCR isn't perfect—that's common, but trying several different phrases to make sure you've found everything is way easier when searching range of years). Some related thoughts can be found in an old Hacker News post: https://news.ycombinator.com/item?id=7408034 https://news.ycombinator.com/item?id=7408034 Online newspaper archives are a ridiculously awesome boon for the humanities. Chronicling America from the Library of Congress, for instance, is great. It's the de facto successor to Google News Archive in the US. I just wish that Google News Archive could get a couple of the old search features back to aid researchers. :-) Second, on a quick tangent I just discovered: when you select "archives" at news.google.com, it says "looking for scanned newspapers?" with a link to: https://support.google.com/news/answer/3334 https://support.google.com/news/answer/3334. But there's nothing there anymore about scanned news. :-)

ChuckMcM 10y ago

I think Book Search is entirely a red herring, one of Google's many 'say one thing, do something entirely different.' Kind of herring. A great example I know of personally was the Google 411 project. Google 411 was a project which offered 'directory services' by voice over your phone. It was pretty cool, you could call its 800 number, ask for a listing, and it would use Google search to find it and then read it back to you. Then you got to tell it how well it did. People started using it, adoption spread, and then they cut it off. So why did Google run this service in the first place? Was it to see if they could make a business out of directory services? No, it was to collect a data set where a spoken phrase could be matched to its exact translation (the thing they were looking for) which could then be processed and re-processed to train algorithms for independent speaker recognition. On the one hand they could try to pay a million people to come in and say something and then confirm or deny they understood what they said, or they could use a bit of spare hardware and collect that information with out paying anyone a dime. They own that data set, it is extremely valuable for testing improvements in voice recognition, and it is yet another barrier to a new company trying to get into that space. Now let's look at Google Books. The 'story' was digitize all the books and make a great library available, and maybe even offer up PDF copies of out of print books for people. It set off a legal firestorm (as mentioned in the article) it got all of the archivists on board and libraries contributed millions of volumes, and Google got a ruling out of the Supreme Court that said that digitizing a book was not an act of copyright infringement. But the subtle part is that while folks say 'everything is on the web' (and that may be true) literally 99.99% of everything on the web is complete and utter crap, written by people who are seeking to game advertising selection engines not digitize information. Most of the stuff in books is not crap, because it cost someone significant time and labor to take that information and publish it. (romance novels excluded :-)). Google had digitized the single largest collection of human knowledge ever, and put it in a form where 100's of thousands of machines can process and re-process it to derive ontological accurate facts into the largest knowledge base in history. If you want to test whether your algorithm can identify credible information, there is no better way to do it than to prime it with a ground truth which has much higher credibility than most of the accessible data out there. That data set exists, they own it, and they don't have any obligation to share it with anyone. And it will allow for the creation of trained models for differentiating fact from fiction, jest for insult, and command from comment. It is my opinion that any company that wants to be 'serious' about AI better have access to an equivalent data set or they will lose.

ma2rten 10y ago

And also: the dawning realization that Scanning All The Books, however useful, might not change the world in any fundamental way. I think this is actually the main takeaway.

douche 10y ago
Remember when ReCaptcha used Google Books snippets? You may not, it's been swapped over to Street View images for a while. But Google Books has been a big deal for automated OCR.

douche 10y ago

I'm sad that outfits like Project Gutenberg are stonewalled at 1923 or whatever year Steamboat Willie came out. There are so many good books whose authors' grandchildren are dead, that can't legally be reproduced.

waqf 10y ago
Library Genesis is your friend. (http://libgen.io http://libgen.io or http://gen.lib.rus.ec http://gen.lib.rus.ec, among other mirrors.)

known 10y ago

I believe PageRank is irrelevant to Google Books