3 ms·

I've been working on a competitor for a while now, and the hardest part of replicating functionality is the data. OpenLibrary is probably the best source for bo

I've been working on a competitor for a while now, and the hardest part of replicating functionality is the data. OpenLibrary is probably the best source for book metadata online, but even their library dumps are riddled with mistakes that manifest in weird ways as you start building your own library. The Goodreads site sucks, but they've got surprising data quality that I don't think anyone else has; and they have a super restrictive data policy so you can't repurpose book data, reviews, shelf data, etc, even when users auth with a Goodreads account. It's a small moat, but definitely penetrable with more than a little effort.

trollied 7y ago
You could nick the scraping code from Calibre...
- dewey 7y ago
  The scraping part is probably not the complicated portion of the endeavor.
hombre_fatal 7y ago
Demanding perfect data is a waste of time that will let you procrastinate your product indefinitely.
- maw 7y ago
  If you're aiming for "like X, but for people who are actually interested in the market X supposedly serves" then maybe this isn't true.
barkingcat 7y ago
The good data quality is actually an artifact of humans being involved in every step of the cataloging process. There's a large group in goodreads called the GoodReads Librarians, and that group has around a hundred thousand dedicated people who go through and flag anomalies, correct titles and indexes etc Book publishers or people who've worked in book publishing will know that the book database is one area you don't want to mess with unless you know what you are doing. ISBN's are not the be all and end all of the story, and when you start taking into account special editions, covers, ebook editions, language translations, you'll start to realize that the Book Catalog system going back in history, including Dewey decimal system is a marvel of human achievement. Of course establishing a good quality index is going to take work. People often forget that quality take human work and effort. EDIT: I lied. I changed the number from my original estimate of a "few hundred" to "hundred thousand". The Goodreads Librarians group has 103718 members as of when I just peeked now - so it's actually a large number of humans submitting fixes to their catalog. https://www.goodreads.com/group/show/220-goodreads-librarians-group https://www.goodreads.com/group/show/220-goodreads-librarian... If you take a look at the kind of discussions taking place, those are the kinds of things any competitor to Goodreads needs to know about.
spullara 7y ago
Facts are not copyrightable and scraping has been determined to be legal. IANAL, but I'm not sure law would protect them from the factual metadata about books being repurposed.
- jjeaff 7y ago
  Not being illegal wouldn't protect you from a crushing lawsuit though. Especially since the details likely vary (linked in data was publicly accessible, not sure if Goodreads requires a login).
johnmaguire2013 7y ago
I wonder if a partnership with Kobo, or even better Nook (Barnes & Noble) could help solve both the data problem and the issue with Kindle integration - while potentially bolstering the e-reader that integrates with it as well.
- Kihashi 7y ago
  I for sure would take that sort of integration into account when looking at a new ereader. Although the primary thing I'd be looking for is library integration a la OverDrive.

trollied 7y ago

You could nick the scraping code from Calibre...

dewey 7y ago
The scraping part is probably not the complicated portion of the endeavor.

hombre_fatal 7y ago

Demanding perfect data is a waste of time that will let you procrastinate your product indefinitely.

maw 7y ago
If you're aiming for "like X, but for people who are actually interested in the market X supposedly serves" then maybe this isn't true.

barkingcat 7y ago

The good data quality is actually an artifact of humans being involved in every step of the cataloging process. There's a large group in goodreads called the GoodReads Librarians, and that group has around a hundred thousand dedicated people who go through and flag anomalies, correct titles and indexes etc Book publishers or people who've worked in book publishing will know that the book database is one area you don't want to mess with unless you know what you are doing. ISBN's are not the be all and end all of the story, and when you start taking into account special editions, covers, ebook editions, language translations, you'll start to realize that the Book Catalog system going back in history, including Dewey decimal system is a marvel of human achievement. Of course establishing a good quality index is going to take work. People often forget that quality take human work and effort. EDIT: I lied. I changed the number from my original estimate of a "few hundred" to "hundred thousand". The Goodreads Librarians group has 103718 members as of when I just peeked now - so it's actually a large number of humans submitting fixes to their catalog. https://www.goodreads.com/group/show/220-goodreads-librarians-group https://www.goodreads.com/group/show/220-goodreads-librarian... If you take a look at the kind of discussions taking place, those are the kinds of things any competitor to Goodreads needs to know about.

spullara 7y ago

Facts are not copyrightable and scraping has been determined to be legal. IANAL, but I'm not sure law would protect them from the factual metadata about books being repurposed.

jjeaff 7y ago
Not being illegal wouldn't protect you from a crushing lawsuit though. Especially since the details likely vary (linked in data was publicly accessible, not sure if Goodreads requires a login).

johnmaguire2013 7y ago

I wonder if a partnership with Kobo, or even better Nook (Barnes & Noble) could help solve both the data problem and the issue with Kindle integration - while potentially bolstering the e-reader that integrates with it as well.

Kihashi 7y ago
I for sure would take that sort of integration into account when looking at a new ereader. Although the primary thing I'd be looking for is library integration a la OverDrive.