11 ms·

I recently discovered the https://openlibrary.org/ https://openlibrary.org/ by The Internet Archive. On the face of it, their "about" page[1] sounds appealing

I recently discovered the https://openlibrary.org/ https://openlibrary.org/ by The Internet Archive. On the face of it, their "about" page[1] sounds appealing (not least because it resonates with my open source values): One web page for every book ever published. It's a lofty but achievable goal. To build Open Library, we need hundreds of millions of book records, a wiki interface, and lots of people who are willing to contribute their time and effort to building the site. To date, we have gathered over 20 million records from a variety of large catalogs as well as single contributions, with more on the way. Open Library is an open project: the software is open, the data are open, the documentation is open, and we welcome your contribution. Whether you fix a typo, add a book, or write a widget--it's all welcome. We have a small team of fantastic programmers who have accomplished a lot, but we can't do it alone! --- They also seem to provide an API[2]. [1] https://openlibrary.org/about https://openlibrary.org/about [2] https://openlibrary.org/developers/api https://openlibrary.org/developers/api

deleted 6y ago
[deleted]
philipn 6y ago
Aaron Swartz actually built the original version of the Open Library site.
- neojumi 6y ago
  The regime responsible for his prosecution is back. Maybe they will pick up where they left off too.
gravitas 6y ago
Sadly, the Goodreads importer appears broken - a fresh export just now of my Goodreads data (<100k) is failing to import with a generic "oops it failed" error almost immediately. :( [1] https://www.goodreads.com/review/import https://www.goodreads.com/review/import (export) [2] https://openlibrary.org/account/import/goodreads https://openlibrary.org/account/import/goodreads
- simonklitj 6y ago
  Not just you, gives me the same error :(
  - cdrini 6y ago
    Hi! I work on Open Library; sorry about that! We had a Python 3 migration that stirred the pot a little. We deployed a fix, so it should be working now! https://github.com/internetarchive/openlibrary/pull/4259 https://github.com/internetarchive/openlibrary/pull/4259
- gwern 6y ago
  The GR CSV export worked for me just now. Possibly an overload problem. Regardless, the writing is now on the wall for GR - get out while you still can.
- cdrini 6y ago
  Hello! I work on Open Library; sorry for the bug! We recently deployed a big Python 3 migration that stirred the pot a little. The import issue should now be fixed: https://github.com/internetarchive/openlibrary/pull/4259 https://github.com/internetarchive/openlibrary/pull/4259
  - kradeelav 6y ago
    Thank you so much for work you do with OL! I'm a goodreads librarian and would love to import my data/lists/etc over, but whenever I try to access (https://openlibrary.org/account/import/goodreads https://openlibrary.org/account/import/goodreads), it gives me a 'method not allowed' error. (granted, running on older versions of firefox, but still thought it might be a bug you would like to be aware of.)
    - mekarpeles 6y ago
      <3 Also, Open Library has a librarian program. If you use the https://openlibrary.org/volunteer https://openlibrary.org/volunteer page and click the Librarian link, this will send an email to Lisa, our head community librarian and we can help you access our slack channel and request access to our librarian features. We also have an optional weekly call for folks to raise issues and questions (e.g. "I want book series!") More about librarianship @ Open Library: https://openlibrary.org/about/lib https://openlibrary.org/about/lib If you hit any issues with this process, feel invited to send me an email to mek@archive.org.
fm4d 6y ago
Its nice that its opensource, backed by Internet Archive and the Controled Digital Lending program is cool too, but how is it possible that a project 14 years in development is such a mess? Just try and search for some popular books and see for yourself, the most important feature - search for books well, is not present. Basic features are missing, book data is often wrong, etc... honestly why would I join such a project instead of starting a new one?
- traverseda 6y ago
  This book is actually a candle, I'm pretty sure: https://openlibrary.org/books/OL28314296M/Harry_Potter https://openlibrary.org/books/OL28314296M/Harry_Potter How do you even get an ISBN for a candle?
  - ORioN63 6y ago
    History section is clear: ImportBot scrapes Amazon.com, matches product id with ISBN10 (which can be converted into ISBN13 without anything else IIRC), imports product as book. Candle: https://www.amazon.com/gp/product/168298494X https://www.amazon.com/gp/product/168298494X
    - tialaramex 6y ago
      > ISBN10 (which can be converted into ISBN13 without anything else IIRC) Yes. There clearly only needs to be one way to identify things, so commonly larger systems just incorporate the smaller ones wholesale. The 13-digit system just incorporates the prior 10-digit system for International Standard Book Numbers with a 978 prefix called "bookland" (most prefixes in this system are geographic, but of course books aren't really from one single geographic region, so they're from "bookland") and adds a new set of possible codes. It also incorporates the entire 12-digit American "UPC" system. Several other (less well known) systems were gobbled up the same way as "bookland", just allocating them imaginary geographic regions in the 13-digit system. There's actually a fourteen digit system, but the lead digit tells you about how many of something are bundled e.g. so a distributor can distinguish a truck full of Pepsi cans from just one case or a single can in terms of things you can order. Lead digit 0 means "single" so if you know the 10 digit ISBN you can not only make a 13-digit EAN for that, you can make the 14-digit GTIN that means "just one of this book" which in most cases would be what you want.
acomjean 6y ago
Doesn’t the library of Congress in the US issue ISBN numbers for each book published. there must be a public listing of those. After some looking, there are Some private databases with millions of # but no official site. Eg https://isbndb.com/isbn-database https://isbndb.com/isbn-database
- deleted 6y ago
  [deleted]
- pmyteh 6y ago
  No. ISBNs are issued by publishers, from delegated blocks, and there's no unified listing. For books in the collections of large libraries (like the LoC) there will be a public catalogue entry with the ISBN attached, but they don't assign it. There were also a lot of books published before ISBNs were created, and not every book has an ISBN attached even to this day.
  - bloak 6y ago
    That is all correct. Another consideration: occasionally an incompetent publisher puts the same ISBN on two unrelated editions. A bibliographer might just carefully record what's printed on the book, but many users of ISBNs won't be at all happy with that "solution".
    - efreak 6y ago
      I've seen a number of books, mostly children's books from scholastic, with different editions (different art, copyright, etc) that have the same ISBN number. I've since removed most duplicate books from my collection, but I had found a bunch when I last sorted my books; Maniac Magee is the only one I can remember by title. More annoying was the book with a CD attached to the front, that caused the finish to peel off when the CD was removed (the CD contained a non-working installer for a screensaver that, when extracted using other tools, also didn't work).
  - dmitriid 6y ago
    Also: ISBNs often change when publishers are bought, split, sold, merged etc.
activatedgeek 6y ago
Since the past few months I have been searching for a Goodreads alternative. Something that only keeps my books. I don't care about the social features that much. And I think this is it. I am going to donate a tiny bit right away! Although, I just tried importing my Goodreads export into Open Library and I get the following "Internal Error": > Hmm... > Sorry. There seems to be a problem with what you were just looking at. > We've noted the error xxxx-xx-xx/yyyyyy and will look into it as soon as possible. Head for home? Anyone else facing this issue?
- AdmiralAsshat 6y ago
  Have you tried Library Thing? https://www.librarything.com/ https://www.librarything.com/
  - jiofih 6y ago
    Adding your data to yet another proprietary archive doesn’t sound like much of a solution..
    - activatedgeek 6y ago
      I don't think I am particularly worried about a proprietary solution. If I am putting my preferences in the public, pretty sure I am giving up on some dimension of privacy. With books at least, it doesn't seem so concerning as long as the recommendations don't suck too much (I don't particularly rely on recommendations that much though). But of course, you never know!
      - jiofih 6y ago
        You do realize this entire post here is about GoodReads discontinuing their API? And as a reaction to that, you’re suggesting moving to another provider that doesn’t allow exporting your data either.
        WorldMaker 6y ago
throw0101a 6y ago
See also WorldCat: > WorldCat is a union catalog that itemizes the collections of 17,900 libraries in 123 countries and territories[4] that participate in the OCLC global cooperative. It is operated by OCLC, Inc.[5] The subscribing member libraries collectively maintain WorldCat's database, the world's largest bibliographic database.[6] * https://en.wikipedia.org/wiki/WorldCat https://en.wikipedia.org/wiki/WorldCat * https://www.worldcat.org https://www.worldcat.org
khalilravanna 6y ago
Anyone know of a project like this for video games? I got a spreadsheet I use to organize games I’ve played/am going to play and always looking for an easier way to get metadata. I was also looking at building something like Goodreads for video games and similarly that data would have been great.
- corobo 6y ago
  Depends on your use case but yes: https://www.giantbomb.com/api/ https://www.giantbomb.com/api/
- Pet_Ant 6y ago
  What about https://videogamegeek.com/ https://videogamegeek.com/ ?
  - Cyph0n 6y ago
    I’m surprised I’ve never heard of this. Thanks for sharing.
    - jacobobryant 6y ago
      also rawg.io's api. I've used them in the past.
- Xavdidtheshadow 6y ago
  https://www.igdb.com/ https://www.igdb.com/ is the one! I do the same thing. I've got a great process built on Airtable + IGDB.
  - khalilravanna 6y ago
    Operated by Twitch nice. Interesting they have a “Time To Beat” field but it’s not really populated. I also use Airtable and was thinking of writing a Zapier hook to scrape HowLongToBeat.com for this info. It’s one of my favorite stats to help prioritize what game to play next. I can’t be dumping 100 hours into JRPGs back to back lol.
cdrini 6y ago
Hi! I work on Open Library. Yep, Open Library has public APIs, and data dumps (updated monthly) of all our books/authors if anyone needs them. https://openlibrary.org/developers/dumps https://openlibrary.org/developers/dumps The project is also open source, and you can find the code (and contribute!) on GitHub: https://github.com/internetarchive/openlibrary https://github.com/internetarchive/openlibrary
jabo 6y ago
Openlibrary looks pretty awesome. Thank you for sharing! Would anyone be interested in having an instant search experience for this books dataset, like the one I built for the 2M recipes database posted on HN earlier this week: https://news.ycombinator.com/item?id=25365397 https://news.ycombinator.com/item?id=25365397
- obviyus 6y ago
  That would be amazing!
  - jabo 6y ago
    Alright! A handful of upvotes in addition to a comment. I’m on it! I’ll whip something up and report back in a few hours.
    - jabo 6y ago
      Looks like it's going to be a while before I can download the dataset. Seems to be pretty slow even when downloading on EC2 with a 10Gpbs connection. ol_dump_latest.txt.gz 0%[ ] 28.73M 115KB/s eta 21h 54m
      - dmje 6y ago
        Be great to see your search on there. I was admiring the speed and accuracy on the post you put on the other day...
        jabo 6y ago
        Thank you! Working on it as we speak.
- mekarpeles 6y ago
  Jabo, please also help us @ openlibrary improve our search. @cdrini is the lead on our solr efforts and we could really benefit from teaming with someone who is really passionate about search. If you have questions about using our data, please send over a message and I'm happy to help: mek@archive.org
mekarpeles 6y ago
For anyone who wishes Open Library was even better, please join one of our weekly community calls @ 11:30am Pacific. For an invite, please send me an email at mek@archive.org or go to: https://openlibrary.org/volunteer https://openlibrary.org/volunteer # APIs & Data Dumps - https://openlibrary.org/developers/api https://openlibrary.org/developers/api - https://openlibrary.org/dev/docs/api/books https://openlibrary.org/dev/docs/api/books - https://openlibrary.org/developers/dumps https://openlibrary.org/developers/dumps monthly data dumps for if you need bulk access and the APIs are not enough. # Spread the word Also, if you want to help raise awareness of this resource, please help us get the word out on twitter! 1. https://twitter.com/openlibrary/status/1338185940469051392 https://twitter.com/openlibrary/status/1338185940469051392 2. https://twitter.com/openlibrary/status/1338186553915367425 https://twitter.com/openlibrary/status/1338186553915367425 # Issues Thank you all for helping us discover some issues with our goodreads importer and search (recently migrated to Python3 + thanks @cdrini et al for these fast bug fixes! If you notice an problem, please help open an issue here: https://github.com/internetarchive/openlibrary/issues/new/choose https://github.com/internetarchive/openlibrary/issues/new/ch... # Learn More - https://archive.org/details/openlibrary-tour-2020/openlibrary.ogv https://archive.org/details/openlibrary-tour-2020/openlibrar... if you want to learn more about Open Library, here's a short intro vid. - https://github.com/internetarchive/openlibrary https://github.com/internetarchive/openlibrary if you want to follow on github.
- rikroots 6y ago
  Hi, Mek. Awesome project! How do I go about claiming my author page? The current book listed there has been officially "retired" for over a decade now, and I have plenty of other books that I'd be happy to add. https://openlibrary.org/authors/OL2965893A/Rik_Roots https://openlibrary.org/authors/OL2965893A/Rik_Roots
  - mekarpeles 6y ago
    Howdy rikroots! Thanks for this kind message, we're really excited about designing a way to help authors claim their pages. It's not a feature our current author pages have yet. As a project of the non-profit Internet Archive, having a trusted catalog is pretty paramount to what we're trying to accomplish and Aaron Swartz's original dream for an Open Library. I've created an issue for this feature (as it's something we've discussed doing for a while). I'll also be adding this to our upcoming Tuesday 2021 roadmap discussion. https://github.com/internetarchive/openlibrary/issues/4263 https://github.com/internetarchive/openlibrary/issues/4263 Thanks for raising this
bborud 6y ago
This is actually very cool. Dissatisfied with how slow and clunky Goodreads is I actually thought about making my own (albeit much simpler) version of Goodreads to keep track of my reading habits. I often dig through Goodreads to find books or authors I can't remember the names of -- and Goodreads isn't great for that. Open Library actually provides the missing piece. The fact that they offer bulk downloads also makes it easier to be a good internet citizen and not send tons of API traffic their way. Looks like I'll have to set up a monthly donation. I'd really like see openlibrary succeed.
bbkane 6y ago
I just signed up for this and imported my GoodReads csv export. the csv has 90ish rows and I was only able to import 60ish rows. I get that Open Library doesn't have as much data as GoodReads, but I wish it would show me the data it couldn't import so I could add it manually to Open Library's data store. Nevertheless, I love the idea and I'll be opening bug reports and maybe code contributions if something looks easy enough.
- mekarpeles 6y ago
  bbkane, this is a great idea (identifying which books didn't import). If you'd be so kind as to help, please open a feature request for this! https://github.com/internetarchive/openlibrary/issues/new/choose https://github.com/internetarchive/openlibrary/issues/new/ch... If you tag @tabshaikh who helped implement the importer and me @mekarpeles we can make sure it gets triaged and tagged correctly this week :)
  - bbkane 6y ago
    I just created https://github.com/internetarchive/openlibrary/issues/4275 https://github.com/internetarchive/openlibrary/issues/4275 - thanks!!
Funes- 6y ago
Upon visiting the Open Library, I'm greeted by a banner covering the top half of the screen, asking me for a donation to keep up with bandwidth costs. Isn't this platform, as well as the Internet Archive or Wikipedia, exactly of the kind that would benefit from being built on top of some kind of P2P network? Content is generated and maintained collectively; why isn't infrastructure treated the same way?
- mekarpeles 6y ago
  Hi Funes, Great points here 1. The banner happens last month of the year (Wikipedia being the perfect analog). Yes, there are mixed feelings and it's not the world's best experience :P 2. Our entire data set is available to download as in bulk https://openlibrary.org/developers/dumps https://openlibrary.org/developers/dumps because we'd love to see a decentralized p2p version 3. https://github.com/mouse-reeve/bookwyrm https://github.com/mouse-reeve/bookwyrm Mouse who used to work @ Internet Archive has a decentralized version of Open Library (Bookwyrm) and it's worth checking out. 4. For the last 5 or so years the Internet Archive has been cultivating a dweb/dapp community and integrating with IIIF, Dat, IPFS, gun, bittorent, webtorrents, and others and hosting regular summits and meetups https://blog.archive.org/2018/07/21/decentralized-web-faq/ https://blog.archive.org/2018/07/21/decentralized-web-faq/ 5. The wayback machine is an interesting case study: it turns out, incentive structures (even things like FIL/filecoin) haven't been able to perfectly crack the nut on getting folks interested enough to preserve the whole wayback machine. There's petabytes of material and there's a powerlaw about what people care about today. Internet Archive realized what we care about today may not be the same as tomorrow, and so there's a cost eaten (the incentive comes from economies of scale generated by intrinsic desire rather than $). And in a way, this centralized solution (economies of scale) IS the solution a community came up with. It has flaws and advantages (tradeoffs), such as centralized points of failure, and I think the archive would be (and has been) ecstatic to explore improving these opportunities.
  - neojumi 6y ago
    Sounds nightmarish. IPFS would work if the project took aim at becoming the default filesystem for linux. There's no storage issue if every new server, terminal and handset was an ipfs node by default and local storage volumes and proprietary cloud storage platforms were the fragile fringe use filesystems. Perhaps the Open Library and the Wayback Machine only work in a more open internet than we have today.
drusepth 6y ago
I built a private Goodreads "competitor" for friends (with groups and book clubs and whatnot) using the Open Library data dumps for book/author/publisher data (since GR APIs were too restrictive in how they could be used). They're great, easy to use, and the site behind them looks like it's run well and stable (edit: didn't realize they're under Internet Archive!). Would definitely recommend them as an alternative.
einpoklum 6y ago
Should this be a separate HN post?
mekarpeles 6y ago
Thank you buttscicles (hard saying that with a straight face) for OP'ing this thread and to Joe Alcorn for the amazing original article. I haven't shared this yet -- it's more for the community, but I've tried to address various questions from the community and distill answers + resources for Open Library here: https://blog.openlibrary.org/2020/12/13/importing-your-goodreads-accessing-them-with-open-librarys-apis/ https://blog.openlibrary.org/2020/12/13/importing-your-goodr... Tried my best to include others players in the space (wikidata, inventaire, bookbrainz, worldcat, bookwyrm) who are doing great work and pay respects to readng, storygraph and other innovative services which are breaking onto the scene.
sohkamyung 6y ago
I'm looking for a site that supports not only books but short stories. My primary interest is in recording my thoughts on books and stories I've read, so a review site is what I'm looking for. This is more challenging for short stories as they don't have a ISBN, may appear online, or in a fiction magazine or in an anthology. So, I may put down my thoughts on short story X that I read in book Y, but if I look up book Z that also features that short story, my thoughts on the story would also appear there. In short, I'm looking for a site that also records short stories like the ISFDB [1], but allows users to add reviews. So far, I haven't found one. I'm now putting down my notes on short stories in a Zotero database. [1] http://isfdb.org/ http://isfdb.org/
markdown 6y ago
Do you know how to filter a search on openlibrary by books that are in the library? It's annoying to search and get hundreds of "Not in Library" results.
dredmorbius 6y ago
The "one page for every book" goal seems to position Open Library as a rival to OCLC (Worldcat), as well as the almost perfectly useless Hathi Trust.

deleted 6y ago

[deleted]

philipn 6y ago

Aaron Swartz actually built the original version of the Open Library site.

neojumi 6y ago
The regime responsible for his prosecution is back. Maybe they will pick up where they left off too.

gravitas 6y ago

Sadly, the Goodreads importer appears broken - a fresh export just now of my Goodreads data (<100k) is failing to import with a generic "oops it failed" error almost immediately. :( [1] https://www.goodreads.com/review/import https://www.goodreads.com/review/import (export) [2] https://openlibrary.org/account/import/goodreads https://openlibrary.org/account/import/goodreads

simonklitj 6y ago
Not just you, gives me the same error :(
- cdrini 6y ago
  Hi! I work on Open Library; sorry about that! We had a Python 3 migration that stirred the pot a little. We deployed a fix, so it should be working now! https://github.com/internetarchive/openlibrary/pull/4259 https://github.com/internetarchive/openlibrary/pull/4259
gwern 6y ago
The GR CSV export worked for me just now. Possibly an overload problem. Regardless, the writing is now on the wall for GR - get out while you still can.
cdrini 6y ago
Hello! I work on Open Library; sorry for the bug! We recently deployed a big Python 3 migration that stirred the pot a little. The import issue should now be fixed: https://github.com/internetarchive/openlibrary/pull/4259 https://github.com/internetarchive/openlibrary/pull/4259
- kradeelav 6y ago
  Thank you so much for work you do with OL! I'm a goodreads librarian and would love to import my data/lists/etc over, but whenever I try to access (https://openlibrary.org/account/import/goodreads https://openlibrary.org/account/import/goodreads), it gives me a 'method not allowed' error. (granted, running on older versions of firefox, but still thought it might be a bug you would like to be aware of.)
  - mekarpeles 6y ago
    <3 Also, Open Library has a librarian program. If you use the https://openlibrary.org/volunteer https://openlibrary.org/volunteer page and click the Librarian link, this will send an email to Lisa, our head community librarian and we can help you access our slack channel and request access to our librarian features. We also have an optional weekly call for folks to raise issues and questions (e.g. "I want book series!") More about librarianship @ Open Library: https://openlibrary.org/about/lib https://openlibrary.org/about/lib If you hit any issues with this process, feel invited to send me an email to mek@archive.org.

fm4d 6y ago

Its nice that its opensource, backed by Internet Archive and the Controled Digital Lending program is cool too, but how is it possible that a project 14 years in development is such a mess? Just try and search for some popular books and see for yourself, the most important feature - search for books well, is not present. Basic features are missing, book data is often wrong, etc... honestly why would I join such a project instead of starting a new one?

traverseda 6y ago
This book is actually a candle, I'm pretty sure: https://openlibrary.org/books/OL28314296M/Harry_Potter https://openlibrary.org/books/OL28314296M/Harry_Potter How do you even get an ISBN for a candle?
- ORioN63 6y ago
  History section is clear: ImportBot scrapes Amazon.com, matches product id with ISBN10 (which can be converted into ISBN13 without anything else IIRC), imports product as book. Candle: https://www.amazon.com/gp/product/168298494X https://www.amazon.com/gp/product/168298494X
  - tialaramex 6y ago
    > ISBN10 (which can be converted into ISBN13 without anything else IIRC) Yes. There clearly only needs to be one way to identify things, so commonly larger systems just incorporate the smaller ones wholesale. The 13-digit system just incorporates the prior 10-digit system for International Standard Book Numbers with a 978 prefix called "bookland" (most prefixes in this system are geographic, but of course books aren't really from one single geographic region, so they're from "bookland") and adds a new set of possible codes. It also incorporates the entire 12-digit American "UPC" system. Several other (less well known) systems were gobbled up the same way as "bookland", just allocating them imaginary geographic regions in the 13-digit system. There's actually a fourteen digit system, but the lead digit tells you about how many of something are bundled e.g. so a distributor can distinguish a truck full of Pepsi cans from just one case or a single can in terms of things you can order. Lead digit 0 means "single" so if you know the 10 digit ISBN you can not only make a 13-digit EAN for that, you can make the 14-digit GTIN that means "just one of this book" which in most cases would be what you want.

acomjean 6y ago

Doesn’t the library of Congress in the US issue ISBN numbers for each book published. there must be a public listing of those. After some looking, there are Some private databases with millions of # but no official site. Eg https://isbndb.com/isbn-database https://isbndb.com/isbn-database

deleted 6y ago
[deleted]
pmyteh 6y ago
No. ISBNs are issued by publishers, from delegated blocks, and there's no unified listing. For books in the collections of large libraries (like the LoC) there will be a public catalogue entry with the ISBN attached, but they don't assign it. There were also a lot of books published before ISBNs were created, and not every book has an ISBN attached even to this day.
- bloak 6y ago
  That is all correct. Another consideration: occasionally an incompetent publisher puts the same ISBN on two unrelated editions. A bibliographer might just carefully record what's printed on the book, but many users of ISBNs won't be at all happy with that "solution".
  - efreak 6y ago
    I've seen a number of books, mostly children's books from scholastic, with different editions (different art, copyright, etc) that have the same ISBN number. I've since removed most duplicate books from my collection, but I had found a bunch when I last sorted my books; Maniac Magee is the only one I can remember by title. More annoying was the book with a CD attached to the front, that caused the finish to peel off when the CD was removed (the CD contained a non-working installer for a screensaver that, when extracted using other tools, also didn't work).
- dmitriid 6y ago
  Also: ISBNs often change when publishers are bought, split, sold, merged etc.

activatedgeek 6y ago

Since the past few months I have been searching for a Goodreads alternative. Something that only keeps my books. I don't care about the social features that much. And I think this is it. I am going to donate a tiny bit right away! Although, I just tried importing my Goodreads export into Open Library and I get the following "Internal Error": > Hmm... > Sorry. There seems to be a problem with what you were just looking at. > We've noted the error xxxx-xx-xx/yyyyyy and will look into it as soon as possible. Head for home? Anyone else facing this issue?

AdmiralAsshat 6y ago
Have you tried Library Thing? https://www.librarything.com/ https://www.librarything.com/
- jiofih 6y ago
  Adding your data to yet another proprietary archive doesn’t sound like much of a solution..
  - activatedgeek 6y ago
    I don't think I am particularly worried about a proprietary solution. If I am putting my preferences in the public, pretty sure I am giving up on some dimension of privacy. With books at least, it doesn't seem so concerning as long as the recommendations don't suck too much (I don't particularly rely on recommendations that much though). But of course, you never know!
    - jiofih 6y ago
      You do realize this entire post here is about GoodReads discontinuing their API? And as a reaction to that, you’re suggesting moving to another provider that doesn’t allow exporting your data either.
      - WorldMaker 6y ago

throw0101a 6y ago

See also WorldCat: > WorldCat is a union catalog that itemizes the collections of 17,900 libraries in 123 countries and territories[4] that participate in the OCLC global cooperative. It is operated by OCLC, Inc.[5] The subscribing member libraries collectively maintain WorldCat's database, the world's largest bibliographic database.[6] * https://en.wikipedia.org/wiki/WorldCat https://en.wikipedia.org/wiki/WorldCat * https://www.worldcat.org https://www.worldcat.org

khalilravanna 6y ago

Anyone know of a project like this for video games? I got a spreadsheet I use to organize games I’ve played/am going to play and always looking for an easier way to get metadata. I was also looking at building something like Goodreads for video games and similarly that data would have been great.

corobo 6y ago
Depends on your use case but yes: https://www.giantbomb.com/api/ https://www.giantbomb.com/api/
Pet_Ant 6y ago
What about https://videogamegeek.com/ https://videogamegeek.com/ ?
- Cyph0n 6y ago
  I’m surprised I’ve never heard of this. Thanks for sharing.
  - jacobobryant 6y ago
    also rawg.io's api. I've used them in the past.
Xavdidtheshadow 6y ago
https://www.igdb.com/ https://www.igdb.com/ is the one! I do the same thing. I've got a great process built on Airtable + IGDB.
- khalilravanna 6y ago
  Operated by Twitch nice. Interesting they have a “Time To Beat” field but it’s not really populated. I also use Airtable and was thinking of writing a Zapier hook to scrape HowLongToBeat.com for this info. It’s one of my favorite stats to help prioritize what game to play next. I can’t be dumping 100 hours into JRPGs back to back lol.

cdrini 6y ago

Hi! I work on Open Library. Yep, Open Library has public APIs, and data dumps (updated monthly) of all our books/authors if anyone needs them. https://openlibrary.org/developers/dumps https://openlibrary.org/developers/dumps The project is also open source, and you can find the code (and contribute!) on GitHub: https://github.com/internetarchive/openlibrary https://github.com/internetarchive/openlibrary

jabo 6y ago

Openlibrary looks pretty awesome. Thank you for sharing! Would anyone be interested in having an instant search experience for this books dataset, like the one I built for the 2M recipes database posted on HN earlier this week: https://news.ycombinator.com/item?id=25365397 https://news.ycombinator.com/item?id=25365397

obviyus 6y ago
That would be amazing!
- jabo 6y ago
  Alright! A handful of upvotes in addition to a comment. I’m on it! I’ll whip something up and report back in a few hours.
  - jabo 6y ago
    Looks like it's going to be a while before I can download the dataset. Seems to be pretty slow even when downloading on EC2 with a 10Gpbs connection. ol_dump_latest.txt.gz 0%[ ] 28.73M 115KB/s eta 21h 54m
    - dmje 6y ago
      Be great to see your search on there. I was admiring the speed and accuracy on the post you put on the other day...
      - jabo 6y ago
        Thank you! Working on it as we speak.
mekarpeles 6y ago
Jabo, please also help us @ openlibrary improve our search. @cdrini is the lead on our solr efforts and we could really benefit from teaming with someone who is really passionate about search. If you have questions about using our data, please send over a message and I'm happy to help: mek@archive.org

mekarpeles 6y ago

For anyone who wishes Open Library was even better, please join one of our weekly community calls @ 11:30am Pacific. For an invite, please send me an email at mek@archive.org or go to: https://openlibrary.org/volunteer https://openlibrary.org/volunteer # APIs & Data Dumps - https://openlibrary.org/developers/api https://openlibrary.org/developers/api - https://openlibrary.org/dev/docs/api/books https://openlibrary.org/dev/docs/api/books - https://openlibrary.org/developers/dumps https://openlibrary.org/developers/dumps monthly data dumps for if you need bulk access and the APIs are not enough. # Spread the word Also, if you want to help raise awareness of this resource, please help us get the word out on twitter! 1. https://twitter.com/openlibrary/status/1338185940469051392 https://twitter.com/openlibrary/status/1338185940469051392 2. https://twitter.com/openlibrary/status/1338186553915367425 https://twitter.com/openlibrary/status/1338186553915367425 # Issues Thank you all for helping us discover some issues with our goodreads importer and search (recently migrated to Python3 + thanks @cdrini et al for these fast bug fixes! If you notice an problem, please help open an issue here: https://github.com/internetarchive/openlibrary/issues/new/choose https://github.com/internetarchive/openlibrary/issues/new/ch... # Learn More - https://archive.org/details/openlibrary-tour-2020/openlibrary.ogv https://archive.org/details/openlibrary-tour-2020/openlibrar... if you want to learn more about Open Library, here's a short intro vid. - https://github.com/internetarchive/openlibrary https://github.com/internetarchive/openlibrary if you want to follow on github.

rikroots 6y ago
Hi, Mek. Awesome project! How do I go about claiming my author page? The current book listed there has been officially "retired" for over a decade now, and I have plenty of other books that I'd be happy to add. https://openlibrary.org/authors/OL2965893A/Rik_Roots https://openlibrary.org/authors/OL2965893A/Rik_Roots
- mekarpeles 6y ago
  Howdy rikroots! Thanks for this kind message, we're really excited about designing a way to help authors claim their pages. It's not a feature our current author pages have yet. As a project of the non-profit Internet Archive, having a trusted catalog is pretty paramount to what we're trying to accomplish and Aaron Swartz's original dream for an Open Library. I've created an issue for this feature (as it's something we've discussed doing for a while). I'll also be adding this to our upcoming Tuesday 2021 roadmap discussion. https://github.com/internetarchive/openlibrary/issues/4263 https://github.com/internetarchive/openlibrary/issues/4263 Thanks for raising this

bborud 6y ago

This is actually very cool. Dissatisfied with how slow and clunky Goodreads is I actually thought about making my own (albeit much simpler) version of Goodreads to keep track of my reading habits. I often dig through Goodreads to find books or authors I can't remember the names of -- and Goodreads isn't great for that. Open Library actually provides the missing piece. The fact that they offer bulk downloads also makes it easier to be a good internet citizen and not send tons of API traffic their way. Looks like I'll have to set up a monthly donation. I'd really like see openlibrary succeed.

bbkane 6y ago

I just signed up for this and imported my GoodReads csv export. the csv has 90ish rows and I was only able to import 60ish rows. I get that Open Library doesn't have as much data as GoodReads, but I wish it would show me the data it couldn't import so I could add it manually to Open Library's data store. Nevertheless, I love the idea and I'll be opening bug reports and maybe code contributions if something looks easy enough.

mekarpeles 6y ago
bbkane, this is a great idea (identifying which books didn't import). If you'd be so kind as to help, please open a feature request for this! https://github.com/internetarchive/openlibrary/issues/new/choose https://github.com/internetarchive/openlibrary/issues/new/ch... If you tag @tabshaikh who helped implement the importer and me @mekarpeles we can make sure it gets triaged and tagged correctly this week :)
- bbkane 6y ago
  I just created https://github.com/internetarchive/openlibrary/issues/4275 https://github.com/internetarchive/openlibrary/issues/4275 - thanks!!

Funes- 6y ago

Upon visiting the Open Library, I'm greeted by a banner covering the top half of the screen, asking me for a donation to keep up with bandwidth costs. Isn't this platform, as well as the Internet Archive or Wikipedia, exactly of the kind that would benefit from being built on top of some kind of P2P network? Content is generated and maintained collectively; why isn't infrastructure treated the same way?

mekarpeles 6y ago
Hi Funes, Great points here 1. The banner happens last month of the year (Wikipedia being the perfect analog). Yes, there are mixed feelings and it's not the world's best experience :P 2. Our entire data set is available to download as in bulk https://openlibrary.org/developers/dumps https://openlibrary.org/developers/dumps because we'd love to see a decentralized p2p version 3. https://github.com/mouse-reeve/bookwyrm https://github.com/mouse-reeve/bookwyrm Mouse who used to work @ Internet Archive has a decentralized version of Open Library (Bookwyrm) and it's worth checking out. 4. For the last 5 or so years the Internet Archive has been cultivating a dweb/dapp community and integrating with IIIF, Dat, IPFS, gun, bittorent, webtorrents, and others and hosting regular summits and meetups https://blog.archive.org/2018/07/21/decentralized-web-faq/ https://blog.archive.org/2018/07/21/decentralized-web-faq/ 5. The wayback machine is an interesting case study: it turns out, incentive structures (even things like FIL/filecoin) haven't been able to perfectly crack the nut on getting folks interested enough to preserve the whole wayback machine. There's petabytes of material and there's a powerlaw about what people care about today. Internet Archive realized what we care about today may not be the same as tomorrow, and so there's a cost eaten (the incentive comes from economies of scale generated by intrinsic desire rather than $). And in a way, this centralized solution (economies of scale) IS the solution a community came up with. It has flaws and advantages (tradeoffs), such as centralized points of failure, and I think the archive would be (and has been) ecstatic to explore improving these opportunities.
- neojumi 6y ago
  Sounds nightmarish. IPFS would work if the project took aim at becoming the default filesystem for linux. There's no storage issue if every new server, terminal and handset was an ipfs node by default and local storage volumes and proprietary cloud storage platforms were the fragile fringe use filesystems. Perhaps the Open Library and the Wayback Machine only work in a more open internet than we have today.

drusepth 6y ago

I built a private Goodreads "competitor" for friends (with groups and book clubs and whatnot) using the Open Library data dumps for book/author/publisher data (since GR APIs were too restrictive in how they could be used). They're great, easy to use, and the site behind them looks like it's run well and stable (edit: didn't realize they're under Internet Archive!). Would definitely recommend them as an alternative.

einpoklum 6y ago

Should this be a separate HN post?

mekarpeles 6y ago

Thank you buttscicles (hard saying that with a straight face) for OP'ing this thread and to Joe Alcorn for the amazing original article. I haven't shared this yet -- it's more for the community, but I've tried to address various questions from the community and distill answers + resources for Open Library here: https://blog.openlibrary.org/2020/12/13/importing-your-goodreads-accessing-them-with-open-librarys-apis/ https://blog.openlibrary.org/2020/12/13/importing-your-goodr... Tried my best to include others players in the space (wikidata, inventaire, bookbrainz, worldcat, bookwyrm) who are doing great work and pay respects to readng, storygraph and other innovative services which are breaking onto the scene.

sohkamyung 6y ago

I'm looking for a site that supports not only books but short stories. My primary interest is in recording my thoughts on books and stories I've read, so a review site is what I'm looking for. This is more challenging for short stories as they don't have a ISBN, may appear online, or in a fiction magazine or in an anthology. So, I may put down my thoughts on short story X that I read in book Y, but if I look up book Z that also features that short story, my thoughts on the story would also appear there. In short, I'm looking for a site that also records short stories like the ISFDB [1], but allows users to add reviews. So far, I haven't found one. I'm now putting down my notes on short stories in a Zotero database. [1] http://isfdb.org/ http://isfdb.org/

markdown 6y ago

Do you know how to filter a search on openlibrary by books that are in the library? It's annoying to search and get hundreds of "Not in Library" results.

dredmorbius 6y ago

The "one page for every book" goal seems to position Open Library as a rival to OCLC (Worldcat), as well as the almost perfectly useless Hathi Trust.