18 ms·
The Tragedy of Google Books (2017)
- pluc 2y ago[flagged]
- deleted 2y ago[deleted]
- pvg 2y agohttps://archive.is/rQ7Zb https://archive.is/rQ7Zb
- neonate 2y agohttps://web.archive.org/web/20170719004247/https://www.theatlantic.com/technology/archive/2017/04/the-tragedy-of-google-books/523320/ https://web.archive.org/web/20170719004247/https://www.theat...
- LisaDziuba 2y ago[dead]
- datadrivenangel 2y agoThanks Paul!
- senkora 2y agoI’m sure the lawyers will eventually figure out a way to train an LLM on them.
- datadrivenangel 2y agoThey probably already have! It seems like an amazing training dataset even if you can't share source data.
- amelius 2y agoHow do you train an LLM such that it is guaranteed to never regurgitate its training data?
- ASalazarMX 2y agoYou punish it if parts of the answer can be found in its training data, and reward it otherwise.
- amelius 2y agoBut the whole point of the training is that you reward it if it correctly reproduces the next token.
- zeroxfe 2y agoThat's not the whole point of the training. It's just (very loosely) a measure of loss used during pre-training. There are many post-training and alignment stages in a typical model that are designed to reward high-quality responses. Technically, yes, it's impossible to guarantee that it won't just regurgitate source material (which is mostly around the tails of the data distribution), but the whole point of training is to build generalized intelligence.
- amelius 2y agoI guess I used the wrong wording but it doesn't change the argument. Yes, the whole point of training is to build generalized intelligence (or at least that's what we __hope__ for). But as far as I understand, we do it __mainly__ by training for the next word in the sequence. PS: you speak of "pre-training" and "post-training", so I'm curious what you think is the main part of the training (?)
- thayne 2y agoIMO if a work is out of print (or equivalent depending on the medium) for more than a few years, it should be released into the public domain. Or maybe something like the public domain, but requires attribution.
- kps 2y agoLike trademark: Use it or lose it. (The reality is that publishers would put lazy photocopies up for sale at ten zillion dollars a piece.)
- giraffe_lady 2y agoThen every book will be immediately out of print after its initial run, while the not-quite-a-cartel of publishers all decline to print it until it hits the point where they no longer have to pay the author.
- deleted 2y ago[deleted]
- Jtsummers 2y agoThen the publisher loses out on exclusive publishing rights and also loses money. It's in their interests to keep it in print so long as it's a profitable book, even if they have to pay some percentage to the author. Once it goes into public domain every publisher can reprint it and the original publisher has to compete with them on price.
- deleted 2y ago[deleted]
- giraffe_lady 2y agook
- jamiek88 2y ago> so long as it's a profitable book And here is the rub. You’ll end up with three or four super authors with the rest being ripped off. Much better for it to revert to the author in that situation IMO.
- andrewstuart 2y agoGoogle must be tempted to put them in an LLM.
- bborud 2y agoIt would surprise me greatly if they haven't already.
- johnobrien1010 2y agoAnother reason that they should never have been allowed to ingest all the books in the first place. Without paying for the rights to use the digital form of the book, a use which is explicitly prohibited by the publisher, they digitized the books anyway. If they used it to train an LLM, and the LLM regurgitates near facsimiles of all the copyrighted works without compensation to the original rights holders, that seems like something that should be illegal.
- anoncow 2y agoSad and criminal.
- 2OEH8eoCRo0 2y agoThe tragedy is that Google is tasked with this at all. It would be cool if public libraries could work together on a massive public digital library. This shouldn't be Google's responsibility.
- Jtsummers 2y agoGoogle wasn't tasked (by a third party) with this, they chose to do it.
- ants_everywhere 2y agoarguably Google was invented to fund this project. The books project predates the search engine and the search engine grew out of the project of creating a universal digital library. The PageRank algorithm is one of a class of algorithms used to score citations in books and papers.
- NoMoreNicksLeft 2y agoAll humans everywhere have a responsibility to preserve culture and knowledge to the best of their ability. I think what you meant to say is that none of us can trust Google with this important task.
- renewiltord 2y agoOne of the great tragedies of civilization is that we leave things in the hands of those who do them rather than in the hands of those who tell us about our responsibility to do them.
- NoMoreNicksLeft 2y agoI personally do what I can. I've been trying to find old phone books and catalogs at garage sales, scanning them when I can get them. I teach my children that this is a responsibility of theirs. But if you're offering me even a fraction of Google's budget, I think I might manage to scale things up.
- xipho 2y agoA huge proportion of this corpus is found in the Hathi Trust (see https://www.hathitrust.org/the-collection/ https://www.hathitrust.org/the-collection/). We have had a grant to crawl and derive an index on it via their supercomputing resources. I'm sure they are looking to LLM proposals, though they are exceedingly careful about the copyright issues. https://www.hathitrust.org/ https://www.hathitrust.org/
- fredgrott 2y agothank you as some of us were looking for something to replace the archive.org digital book library part....
- jsemrau 2y ago>I'm sure they are looking to LLM proposals Well, it is a use case for this challenge https://www.kaggle.com/competitions/gemini-long-context https://www.kaggle.com/competitions/gemini-long-context
- philipkglass 2y agoThese Google scans are also available in the HathiTrust [1], an organization built from the big academic libraries that participated in early book digitization efforts. The HathiTrust is better about letting the public read books that have actually fallen into the public domain. I have found many books that are "snippet view" only on Google Books but freely visible on HathiTrust. If you are a student or researcher at one of the participating HathiTrust institutions, you can also get access to scans of books that are still in copyright. The one advantage Google Books still has is that its search tools are much faster and sometimes better, so it can be useful to search for phrases or topics on Google Books and then jump over to HathiTrust to read specific books surfaced by the search. [1] https://www.hathitrust.org/ https://www.hathitrust.org/
- acidburnNSA 2y agoHathitrust has been absolutely transformative for me, as an amateur nuclear enterprise historian.
- germinalphrase 2y ago“…nuclear enterprise…” As in, the business of running a nuclear energy plant?
- acidburnNSA 2y agoYes. Electricity, propulsion, desalination, space heating, etc.
- dghlsakjg 2y agoThe old Lia pocket warmer?
- acidburnNSA 2y agoYes!
- 2y ago
- Zigurd 2y agoO'Reilly, for whom I've been a lead author and co-author, did this: https://www.oreilly.com/pub/pr/1042 https://www.oreilly.com/pub/pr/1042 They call it Founder's Copyright. The also use Creative Commons. The goal is to make out of print books available at no cost.
- card_zero 2y ago> A complete list of available titles is at www.oreilly.com/openbook Exciting! Follows link Link no longer exists, gets O'Reilly front page instead "Introducing the AI Academy, Help your entire org put GenAI to work" Thanks O'Reilly.
- deleted 2y ago[deleted]
- MollyRealized 2y agoIt's okay, I'll just check the Wayb--shit
- tourmalinetaco 2y agoWayback Machine has been working for the past few days, look: https://web.archive.org/web/20240607220047/http://www.oreilly.com/openbook/ https://web.archive.org/web/20240607220047/http://www.oreill...
- Zigurd 2y agoIt looks like the pdfs of at least some of the books are there. But it's a pretty small subset of their out of print books.
- ToucanLoucan 2y agoThe original dream of the internet: Information, freely available to any who want it. The new dream of the internet: Some information, that aligns with the values of our advertisers, delivered via an LLM that sometimes makes shit up.
- svilen_dobrev 2y agoThis seems to be the fate of knowledge/content that stays in institutions which have been built with the idea of collecting it and growing it.. but have turned into walled gardens/crypts of sort. Rot/Rust and be forgotten. A very cynical and dark view is that the New things/people need that oblivion in order to feel great, for not haveing to compare with old great-er ones. Rewriting history as it seems fit the current powers-that-be, is easier this way. Or may be it's just collective stupidity? or societal immaturity ? (i am coming from completely different killed project on a different continent, but the idea is the same)
- kyleee 2y agoI think you are on to something, people frequently don’t want to grapple with and understand what has been done before, they prefer to just wing it and move forward on their own.
- SapporoChris 2y agoI am fairly certain there is more knowledge/content available to anyone in this century than last century or any century before it. But perhaps I have misread your comment.
- shiroiushi 2y agoI seriously doubt there's very much highly relevant old knowledge locked away somewhere. Is there interesting stuff we don't have good access to? Sure, but mainly of interest to historians (pro or amateur). You're not likely to find the cure for cancer written down in some 1000-year-old book somewhere. And while a few people might really be interested in reading decades or centuries-old novels that weren't popular enough to be called "classics" now, the vast majority of people aren't going to find such stories about people in the distant past all that interesting. Of course, it's best to preserve past knowledge, but I think the idea that this is part of some kind of conspiracy to keep people buying new stuff is pretty silly. People are always going to want new stuff, as society grows and changes.
- sersi 2y ago> find the cure for cancer written down in some 1000-year-old book somewhere. While you're most likely right about the cure for cancer, I did want to note that this is kind of how the cure for malaria was found (artemisinin). Tu Youyou who won the nobel prize systematically investigated Traditional Chinese Medecine remedies until she came across one that was effective. That particular remedy was described in a 1600 years old text The Handbook of Prescriptions for Emergency Treatments, written in 340 by Ge Hong. Note: before anyone think that the fact that a remedy described in traditional Chinese texts mean that TCM is reliable and a viable alternative in all text, she screened over 2,000 traditional Chinese recipes and made 380 herbal extracts, from some 200 herbs which were tested on mice. So, yes one of the remedy were successful but the success rate of TCM was not particularly high :)
- Animats 2y agoWe need a Copyright Term Reduction Act. It's time. 50 years, renewal is possible but expensive.
- ASalazarMX 2y agoEven 50 is a lot, because it starts at the death of the author. Popular culture shouldn't remain locked out for generations. 50 maximum would be ideal, two generations from the one who experienced it in the original cultural context.
- Animats 2y ago50 years from first publication. That's all the TRIPS agreement requires.[1] [1] https://en.wikipedia.org/wiki/TRIPS_Agreement https://en.wikipedia.org/wiki/TRIPS_Agreement
- ASalazarMX 2y agoAnd that's still a lot, since fifty years from publication is the minimum to abide to TRIPS, but we're used to much worse, so it doesn't sound as bad now. It could be shorter, things move a lot faster nowadays, a single generation of monopoly means more today than a hundred years ago.
- mjevans 2y agoJust my opinion but as a starting point for the argument... * 20 years from date of first publish (renewable up to CAP? 50 years) * Must remain available every year * 10 year renewal blocks with massive registration fee increases * Compulsory maximum license fee cap (can offer for less) in the laws Note this is not TRADE MARK; trade marks are _consumer protection_ related to 'brand ownership'.
- gosub100 2y agoAnd reign in the damages for infringement to some amount closer to what was actually lost. For instance, if someone has a million books on a drive they haven't deprived the publisher of a million sales for chrissakes
- ErikAugust 2y ago“Page had always wanted to digitize books. Way back in 1996, the student project that eventually became Google—a “crawler” that would ingest documents and rank them for relevance against a user’s query—was actually conceived as part of an effort “to develop the enabling technologies for a single, integrated and universal digital library.” The idea was that in the future, once all books were digitized, you’d be able to map the citations among them, see which books got cited the most, and use that data to give better search results to library patrons. But books still lived mostly on paper. Page and his research partner, Sergey Brin, developed their popularity-contest-by-citation idea using pages from the World Wide Web.“ Larry Page had some cool ideas… can’t imagine Books will ever be resurrected, unfortunately.
- dekhn 2y agoHe really wanted to digitize all of them to provide reference and training data for early language models (well before LLMs, transformers, etc). He also had a plan (with George Church) to build enormous warehouses holding large-scale biology research infrastructure right next to google data centers. Because most biology research is done at locations that have reached their limit on computational/storage capacity. Larry had many good ideas but he struggled to get the majority of them off the ground. For example, when Trump was president and invited all the major tech leaders, Larry came with a plan to upgrade the US electrical system with long-range DC.
- shiroiushi 2y ago>Larry came with a plan to upgrade the US electrical system with long-range DC. I feel like some crucial detail is missing here. They already use HVDC for long-distance transmission lines, inside and outside of the US. Texas could benefit from it I suppose, but the US in general already uses it where appropriate AFAIK.
- pyrale 2y ago> Larry had many good ideas but he struggled to get the majority of them off the ground. For example, when Trump was president and invited all the major tech leaders, Larry came with a plan to upgrade the US electrical system with long-range DC. I fail to see how that would be a good idea. To me, it looks like some magnate in a completely unrelated industry, who is megalomaniac enough to believe that they can enter a completely unrelated industry and explain to experts how things ought to get done.
- submeta 2y agoWith library genesis, who needs Google Books anymore? I buy books physically to support the author/s and download an epub version from said site to my kindle. The physical books I hardly read, they are for my shelf. Although I love the feeling of printed books, but I read in bed, and it‘s easier to hold an ebook. Also I read when I commute. It’s lighter to have my Kindle Oasis with me with tons of books on it.
- layer8 2y agoMany books aren’t on libgen. It’s been rather hit and miss for me.
- consf 2y ago[dead]
- ghaff 2y agoThere’s the everything available online for free mindset. But, yes, I’ve basically donated all my books that were in the public domain. And, in general, have been massively purging my book collection of stuff I won’t realistically read again.
- submeta 2y agoI do buy books, to support the authors. And I would encourage anyone to support the authors they like to read.
- ASalazarMX 2y agoI agree, but also wouldn't lose sleep for pirating a book of an author that died more than 20 years ago, in most contexts.
- kccqzy 2y agoSomeone needs to scan the book and upload it to library genesis. The article said Google had developed this massively efficient apparatus for scanning (or taking photographs of) books, and most of the article was about out-of-print books. I personally have actually tried to contribute to libgen a particular difficult-to-find-online book by buying it, scanning it, and uploading it. There need to be more people doing this.
- yonran 2y ago> Dan Clancy, the Google engineering lead on the project who helped design the settlement, thinks that it was a particular brand of objector—not Google’s competitors but “sympathetic entities” you’d think would be in favor of it, like library enthusiasts, academic authors, and so on—that ultimately flipped the DOJ. I was at Google in 2009 on a team adjacent to Dan Clancy when he was most excited about the Authors’ Guild negotiations to publish orphan works and create a portal to pay copyright holders who signed up, and I recall that one opponent that he was frustrated at was Brewster Kahle of the Internet Archive, who filed a jealous amicus brief (https://docs.justia.com/cases/federal/district-courts/new-york/nysdce/1:2005cv08136/273913/291 https://docs.justia.com/cases/federal/district-courts/new-yo...) complaining that the Authors’ Guild settlement would not grant him access to publishing orphan works too. In my opinion Kahle was wrong; the existence of one orphan works clearinghouse would have encouraged Congress to grant more libraries access instead of doing nothing which is what actually happened in the 15 year since then. Instead of one company selling out-of-print but in-copyright books, or multiple organizations, no one is allowed to sell them today. Since then, of course, Brewster Kahle launched an e-library of copyrighted books without legal authorization anyway which will probably be the death of the current organization that runs the Internet Archive. Tragic all around.
- jamiek88 2y agoThat pandemic library was a huge, obvious over step by him. It will have consequences far beyond the immediate lawsuit too. The very concept has basically been iced for a generation and the net is only getting more locked down not less.
- jmb99 2y agoFortunately (by some definition of fortunately), most countries don’t agree on exactly how the web should be “locked down.” This benefits at least some people (like me) who live in countries who make also no effort to restrict what can be shoved down the internet tube, including from countries that don’t particularly care about western copyright law. Would it be nice to have a fully sanctioned pandemic library-style service? Absolutely. But I have never once looked for a textbook, paper, regular book, etc online and not found a copy for free. Usually takes the same amount of time or less compared to finding a copy on Amazon (if it’s currently in print), and almost always less time using my library’s clunky online ebook platform[1]. Is that legal? Technically yes, in my country. Is it ethical? Debatable, depending on who you’re asking. But for me personally, I have found it to be getting substantially easier to find high quality copies of copyrighted anything in the past 3-5 years compared to 10-15 years ago, so I don’t necessarily agree with the blanket statement that “the net is only getting more locked down.” [1] I like to use the library as much as possible, if for nothing else than to increase usage numbers to marginally positively decrease the likelihood of finding cuts.
- caseysoftware 2y agoI worked at the Library of Congress on their Digital Preservation Project, circa 2001-2003. The stated goal was to "digitize all of the Library's collections" and while most people think of books, I was in the Motion Picture Broadcast and Recorded Sound Division. In our collection were Thomas Edison's first motion pictures, wire spool recordings from reporters at D-Day, and LPs of some of the greatest musicians of all time. And that was just our Division. Others - like American Heritage - had photos from the US Civil War and more. Anyway, while the Rights information is one big, ugly tangled web, the other side is the hardware to read the formats. Much of the media is fragile and/or dangerous to use so you have to be exceptionally careful. Then you have to document all the settings you used because imagine that three months from now, you learn some filter you used was wrong or the hardware was misconfigured.. you need to go back and understand what was affected how. Cool space. I wish I'd worked there longer.
- caseysoftware 2y agoAlso.. it was fun learning the answer to "what is the work?" If you have an LP or wire spool recording, the audio is the key, obvious work. But then you have the album cover, the spool case, and the physical condition of the media. Being able to see an album cover or read a reporter's notes/labeling is almost as important as the audio.
- deleted 2y ago[deleted]
- ForHackernews 2y agoIs the Library of Congress really beholden to copyright laws? I guess I assumed as the national deposit library they had a special exemption to copy any damn thing they pleased for archival purposes. If they don't have that prerogative, they probably should, and Congress should legislate that to be the case.
- aspenmayer 2y agoThe Library of Congress and its staff determine fair use exceptions in certain contexts so I’m not sure who could find fault with them, as they could simply authorize it before or after the fact, from what I understand.
- carlosjobim 2y agoFor Kagi users, I recommend putting books.google.com as a pinned domain. This way, you'll many times be presented with some of the best sources for any search query. Then it's a matter of finding the ePub file of that book. To read on MacOS, FBReader is a high quality app.
- emmelaich 2y agoThanks. Looks like it's available for Windows/Linux too. At last as of FBReader 2.1.2 30th September 2024.
- DrNosferatu 2y agoI never seen an explicit mention if the Google Books corpus was indeed or not used for training LLMs… Anyone knows more about it?
- boramalper 2y agoOf course someone needs to scan/digitise those books but for those which already are, there is Anna’s Archive. https://en.wikipedia.org/wiki/Anna%27s_Archive https://en.wikipedia.org/wiki/Anna%27s_Archive
- fx1994 2y agoit's a shame you have to pirate your way to find a book that is practically unavailable, but I support pirating old unavailable stuff
- einpoklum 2y agoWritten from a capitalist perspective, extolling "market forces" and legitimizing corporate and government limitations on copying. "between 1923 and 1963 ... copyrights back then had to be renewed, and often the rightsholder wouldn’t bother filing the paperwork" - oh no, how terrible. How lucky we are that in these modern times one doesn't even have to file paperwork in order to prevent you from copying information. and they go on to suck to Google and decry how they didn't get to legitimize their control over a large swath of human knowledge and cultural heritage. "It certainly seems unlikely that someone is going to spend political capital—especially today—trying to change the licensing regime for books, let alone old ones." <- copyright regime, licensing regime - all of this stuff is illegitimate apriori. Poetry, literature, music, software, papers and books - we cannot and must not tolerate restrictions on their dissemination. What arrangements the commercial and governmental entities come to, our "arrangement" should be that everything gets disseminated widely and without restriction, so that curtailment, censorship, commercial control etc. just fail.
- afh1 2y agoIronically behind a paywall (and below a political ad)
- tempfile 2y ago> what happened with piano rolls, with records, with radio, and with cable—isn’t that copyright holders squash the new technology. Instead, they cut a deal and start making money from it. > “History has shown that time and market forces often provide equilibrium in balancing interests,” Wu writes. It is completely braindead to argue that market forces had anything to do with compulsory licensing. It is a matter determined by courts in the public interest.
- kbbgl87 2y ago> “Somewhere at Google there is a database containing 25 million books and nobody is allowed to read them.” Greeted with a paywall on the source. Hypocracy...
- shadytrees 2y agoJames Somers writes beautifully; https://www.newyorker.com/contributors/james-somers https://www.newyorker.com/contributors/james-somers has some of his other writing
- mcepl 2y ago> Copyright terms have been radically extended in this country largely to keep pace with Europe, where the standard has long been that copyrights last for the life of the author plus 50 years. But the European idea, “It’s based on natural law as opposed to positive law,” Lateef Mtima, a copyright scholar at Howard University Law School, said. “Their whole thought process is coming out of France and Hugo and those guys that like, you know, ‘My work is my enfant,’” he said, “and the state has absolutely no right to do anything with it—kind of a Lockean point of view.” As the world has flattened, copyright laws have converged, lest one country be at a disadvantage by freeing its intellectual products for exploitation by the others. And so the American idea of using copyright primarily as a vehicle, per the constitution, “to promote the Progress of Science and useful Arts,” not to protect authors, has eroded to the point where today we’ve locked up nearly every book published after 1923. This is disingenuous: the article doesn’t mention that the biggest proponent of the prolonging of the copyright terms were Americans (e.g., Walt Disney Corp and Jack Valenti, see “Mickey Mouse Protection Act” for more) not Europeans.
- geniium 2y agoTL;DR: bye bye Google
- theendisney4 2y agoProgrammers not law makers really control what goes and doesnt online. Bittorent and ipfs etc are nice but things would be better if there was a large static archive with desktop clients exchanging chunks in a complex modular way. Say: I have pages 1-15 of file 123456, you have page 16 but are looking for page 1 of doc 2345, if i can obtain that page a fast exchange is possible. If not a different module can issue an iou that either means i owe something, you are owed something or both. Other modules could create groups that aim to store part of the archive without duplication amoung members. Spam driven modules could also be interesting. The archive can be organized by how dubious the copyright is so that one can limit participation to 50 or 100+ year old publications and/or living or dead authors. Its not unlike living on a far away island with the british empire seeking to control every aspect of your life without sufficient means of force.
- renewiltord 2y agoGood. It’s important that free access not be permitted. We don’t know what personal data might be contained within. We should only allow those works after a human (appropriately certified) has verified that no personal data exists within. If it exists within the book must be destroyed in its entirety. Too many works of so-called scholarship have relied on the personal letters of dead people. We should not reward grave robbing. The most important thing is the personal data. We must protect the personal data.
- mparnisari 2y ago> We don’t know what personal data might be contained within You really think that a book author doesn't know that whatever they put on a book will stay there forever? What a weird take :p
- renewiltord 2y agoThey could have put other people’s personal data in. The only thing that matters is personal data. We have a right to making sure it’s removed. Once these massive privacy risks are dealt with we can start again. But only after we have a comprehensive report of the environmental consequences of this. We’ve all had enough of people just stomping over our rights in this move fast and break things / ask forgiveness instead of permission crap. Do it right. And respectfully instead of abusing people’s data.
- mparnisari 2y agoWould it not be a viable solution to let Google scan and sell books, but force them to give the profit from the sales to the government?
- rekabis 2y agoLet’s rewrite copyright law: 1. The author gets to say, “I produced this”, and to control if it gets published. 2. Exclusive copyright for 15 year terms. 3. Renewal possible if author still alive. Non-human rights holders (corporations, etc.) limited to 30 years total (one renewal) from date of first publication, regardless of item ownership. Failure to renew automatically opens up the product. 4. Existing copyright can be overridden if demand isn’t being adequately serviced (sliding scale, challenger must capture minimum % of existing market demand to prove). Pricing of overriding attempts must be reasonable, only cost of production can be directly paid for, everything else goes into an escrow account until the attempt is concluded. This is where anti-abuse rules for both sides are most extensive. Information and knowledge must be free. Our civilization depends vitally upon that freedom.