4 ms·

Book metadata quality is extremely important and it's hard to get right in a short timeframe (e.g., < 3 years). Over the past decade, Goodreads builds a huge a

by wenbin 5y ago

Book metadata quality is extremely important and it's hard to get right in a short timeframe (e.g., < 3 years). Over the past decade, Goodreads builds a huge army of volunteer members (120,000+) to help correct book metadata [1][2]. But to compete with Goodreads, a new service can start from a vertical, instead of ALL BOOKS IN THE WORLD. A subreddit could be a good MVP. - [1] https://help.goodreads.com/s/article/What-is-a-Goodreads-Librarian-1553870935316 https://help.goodreads.com/s/article/What-is-a-Goodreads-Lib... - [2] https://www.goodreads.com/group/show/220-goodreads-librarians-group https://www.goodreads.com/group/show/220-goodreads-librarian...

villasv 5y ago
In other words, good quality book metadata is expensive. Book recommendation/reviews aren't haven't been so profitable as to justify such investment. If I was a publisher, I'd put my money on the book influencers, not on Goodreads and alternatives anyway.
- bwb 5y ago
  It isn't expensive... it doesn't exist. Ingram's API is probably the best and it is messy as hell. I've been playing with it and it will help augment data, but you have to have humans to really fine tune things. I just entered 2,500 books manually to launch this on Monday (https://shepherd.com/ https://shepherd.com/) and I still am going to need to go back to build relationships between the types of authors (lead, translator, illustrator, etc). It is a hard problem. Ingram's is like $1k a month, maybe $2k for the full flat files. I can't find anything that is high quality and with a bigger data set.
  - villasv 5y ago
    > It isn't expensive... it doesn't exist. It's the same thing, but I get what you mean. By expensive, I mean too expensive to get to invest in building it, not as in expensive to buy off-the-shelf.
    - bwb 5y ago
      Ah gotcha. I have a dream of one day combining the data I am putting together with Storygraph or others and trying to do an at-cost API for book data to encourage more reading/book apps. There is so much free time being put into Goodreads that doesn't seem to do anyone any good.
      - villasv 5y ago
        I did some data management a while back for higher education institutions from Brazil and took that opportunity to contribute to WikiData. Not a recommendation, just a shoutout to them. Also, OpenRefine helped so much, kudos to all involved.
  - scandox 5y ago
    I like what you're doing with Shepherd.com. I think the idea of curated reading lists is a good one and I've often contemplated it. The most important thing I was given at University were reading lists. I often wondered if getting students from courses all over the world to send in copies of their University course reading lists would be possible/useful. To give a concrete example my Eng lit course in the 90s, each year we would get 6 reading lists covering different topics/historical periods and with a thematic direction: like (made up) Social Mobility in Renaissance Literature. These lists are produced by people with significant expertise and strong opinions, but also had to ensure certain ground was covered. It seems trivial in a sense, but actually when you come fresh to a topic it's hard to get a great list! Anyway I think what you're doing is excellent. A question and an observation: - Why not use openlibrary as referenced in the top comment here (https://news.ycombinator.com/item?id=26837057 https://news.ycombinator.com/item?id=26837057)? - I like the author led lists and I get why the page is strongly focused on who the recommendation is coming from / why I should care...but there was also a part of me that wanted quickly to see the general tenor of the recommendations. Not sure how one could possibly satisfy both desires above the fold! But anyway it was a "feeling".
    - bwb 5y ago
      Thanks! I don't want to get yelled at... but in my testing OpenLibrary data was some of the worst out of all the datasets/APIs I tested. I would love to see that project succeed. I am not sure how they are going to achieve what they are trying to do within the current labor/"business" model. I hear you on the general tenor of the recommendations, if you have a second shoot me an email at ben@shepehrd.com as I'd love to get your thoughts on something.
Mediterraneo10 5y ago
The volunteer army is pretty shackled. There are certain problems with book metadata that cannot be fixed by ordinary members with librarian status. They can only be fixed by Goodreads staff. However, Goodreads Staff (apparently a skeleton crew that are barely keeping the lights on) no longer respond to reports about metadata problems.
throwawayjcvbs 5y ago
this is the real reason goodreads wont be dethroned. book data is very very messy and requires lots of manual cleaning by humans and goodreads is where the humans are. its like trying to make a wikipedia competitor. it just wont happen.

by wenbin 5y ago

villasv 5y ago

In other words, good quality book metadata is expensive. Book recommendation/reviews aren't haven't been so profitable as to justify such investment. If I was a publisher, I'd put my money on the book influencers, not on Goodreads and alternatives anyway.

bwb 5y ago
It isn't expensive... it doesn't exist. Ingram's API is probably the best and it is messy as hell. I've been playing with it and it will help augment data, but you have to have humans to really fine tune things. I just entered 2,500 books manually to launch this on Monday (https://shepherd.com/ https://shepherd.com/) and I still am going to need to go back to build relationships between the types of authors (lead, translator, illustrator, etc). It is a hard problem. Ingram's is like $1k a month, maybe $2k for the full flat files. I can't find anything that is high quality and with a bigger data set.
- villasv 5y ago
  > It isn't expensive... it doesn't exist. It's the same thing, but I get what you mean. By expensive, I mean too expensive to get to invest in building it, not as in expensive to buy off-the-shelf.
  - bwb 5y ago
    Ah gotcha. I have a dream of one day combining the data I am putting together with Storygraph or others and trying to do an at-cost API for book data to encourage more reading/book apps. There is so much free time being put into Goodreads that doesn't seem to do anyone any good.
    - villasv 5y ago
      I did some data management a while back for higher education institutions from Brazil and took that opportunity to contribute to WikiData. Not a recommendation, just a shoutout to them. Also, OpenRefine helped so much, kudos to all involved.
- scandox 5y ago
  I like what you're doing with Shepherd.com. I think the idea of curated reading lists is a good one and I've often contemplated it. The most important thing I was given at University were reading lists. I often wondered if getting students from courses all over the world to send in copies of their University course reading lists would be possible/useful. To give a concrete example my Eng lit course in the 90s, each year we would get 6 reading lists covering different topics/historical periods and with a thematic direction: like (made up) Social Mobility in Renaissance Literature. These lists are produced by people with significant expertise and strong opinions, but also had to ensure certain ground was covered. It seems trivial in a sense, but actually when you come fresh to a topic it's hard to get a great list! Anyway I think what you're doing is excellent. A question and an observation: - Why not use openlibrary as referenced in the top comment here (https://news.ycombinator.com/item?id=26837057 https://news.ycombinator.com/item?id=26837057)? - I like the author led lists and I get why the page is strongly focused on who the recommendation is coming from / why I should care...but there was also a part of me that wanted quickly to see the general tenor of the recommendations. Not sure how one could possibly satisfy both desires above the fold! But anyway it was a "feeling".
  - bwb 5y ago
    Thanks! I don't want to get yelled at... but in my testing OpenLibrary data was some of the worst out of all the datasets/APIs I tested. I would love to see that project succeed. I am not sure how they are going to achieve what they are trying to do within the current labor/"business" model. I hear you on the general tenor of the recommendations, if you have a second shoot me an email at ben@shepehrd.com as I'd love to get your thoughts on something.

Mediterraneo10 5y ago

The volunteer army is pretty shackled. There are certain problems with book metadata that cannot be fixed by ordinary members with librarian status. They can only be fixed by Goodreads staff. However, Goodreads Staff (apparently a skeleton crew that are barely keeping the lights on) no longer respond to reports about metadata problems.

throwawayjcvbs 5y ago

this is the real reason goodreads wont be dethroned. book data is very very messy and requires lots of manual cleaning by humans and goodreads is where the humans are. its like trying to make a wikipedia competitor. it just wont happen.