3 ms·

I always wonder, how do sites like this get their list of books and book metadata? Do publishers have an API? What about Amazon?

by skizm 2y ago

atrus 2y ago
https://openlibrary.org/ https://openlibrary.org/ has a pretty good set of data and a decent API. You can mix and match too, since openlibraries covers kinda suck half the time.
- rafram 2y ago
  It's alright. I would use it for a tool where the most important part is having a more-or-less accurate author <-> title <-> ISBN mapping, but not for anything where I need precise bibliographic metadata.
bwb 2y ago
It is so bad!!! I use Nielsen's API, but the data is pretty rough, and you have to spend a lot of time cleaning it. Plus, the archaic industry standards around genre are hard to translate to what readers use - https://www.bisg.org/complete-bisac-subject-headings-list https://www.bisg.org/complete-bisac-subject-headings-list. Ingrams and Bowker are the other big metadata providers. Ingram's is good but expensive, but the data faces the same issues.
- matwood 2y ago
  If you have an ISBN you can also check Worldcat, but at scale it's also probably not free. And if you're working with anything that might have an academic slant, Crossref can be useful. Book metadata is very challenging. Even the publishers of said books are pretty bad at delivering good metadata.
  - rafram 2y ago
    WorldCat doesn't have an official API for anyone but member libraries, scraping the site is not fun, and the quality of the data isn't great (it's essentially bulk-imported from member libraries' catalogs). National databases like the Library of Congress are significantly better - WorldCat is best used as a fallback for books that aren't included in the high-quality databases.
    - bwb 2y ago
      Ya I do pull in LOC data, but its limited to a very small number of books. Good quality though.
- compootr 2y ago
  I forgot if Anna's archive contains metadata but that'd be my free-ish solution
  - jamiek88 2y ago
    I thought Anna’s was dead?
    - tux3 2y ago
      The last blog post is from just a couple days ago. They're apparently setting their sights on a map of all known ISBNs, and hoping to turn the map from red to green
- wood_spirit 2y ago
  Do any of the sites use collaborative filtering for “uses who like x also like y” for a nice filter bubble? I imagine it complements or my even supersede tags. (Admit I haven’t looked at all the sites people are mentioning in the comments yet- lots of good leads!)
  - bwb 2y ago
    ya, I do this; I am not sure what the other websites are doing, though. If you email them I bet they will tell you.
- everybodyknows 2y ago
  Interesting that Wikipedia considers "campus" and "varsity" to be novel genres, but neither of those words appears here on BISG: https://www.bisg.org/fiction https://www.bisg.org/fiction
PaulHoule 2y ago
Freebase got about 50% of the way to a good book database but Google killed it.
deleted 2y ago
[deleted]
greenie_beans 2y ago
You want to read a screed about this? I just published bc I couldn't figure out how to share unpublished substack post, but it needs to be edited: https://bookheadlife.substack.com/p/the-mess-of-bibliographic-metadata https://bookheadlife.substack.com/p/the-mess-of-bibliographi...

I always wonder, how do sites like this get their list of books and book metadata? Do publishers have an API? What about Amazon?

by skizm 2y ago

I always wonder, how do sites like this get their list of books and book metadata? Do publishers have an API? What about Amazon?

atrus 2y ago

https://openlibrary.org/ https://openlibrary.org/ has a pretty good set of data and a decent API. You can mix and match too, since openlibraries covers kinda suck half the time.

rafram 2y ago
It's alright. I would use it for a tool where the most important part is having a more-or-less accurate author <-> title <-> ISBN mapping, but not for anything where I need precise bibliographic metadata.

bwb 2y ago

It is so bad!!! I use Nielsen's API, but the data is pretty rough, and you have to spend a lot of time cleaning it. Plus, the archaic industry standards around genre are hard to translate to what readers use - https://www.bisg.org/complete-bisac-subject-headings-list https://www.bisg.org/complete-bisac-subject-headings-list. Ingrams and Bowker are the other big metadata providers. Ingram's is good but expensive, but the data faces the same issues.

matwood 2y ago
If you have an ISBN you can also check Worldcat, but at scale it's also probably not free. And if you're working with anything that might have an academic slant, Crossref can be useful. Book metadata is very challenging. Even the publishers of said books are pretty bad at delivering good metadata.
- rafram 2y ago
  WorldCat doesn't have an official API for anyone but member libraries, scraping the site is not fun, and the quality of the data isn't great (it's essentially bulk-imported from member libraries' catalogs). National databases like the Library of Congress are significantly better - WorldCat is best used as a fallback for books that aren't included in the high-quality databases.
  - bwb 2y ago
    Ya I do pull in LOC data, but its limited to a very small number of books. Good quality though.
compootr 2y ago
I forgot if Anna's archive contains metadata but that'd be my free-ish solution
- jamiek88 2y ago
  I thought Anna’s was dead?
  - tux3 2y ago
    The last blog post is from just a couple days ago. They're apparently setting their sights on a map of all known ISBNs, and hoping to turn the map from red to green
wood_spirit 2y ago
Do any of the sites use collaborative filtering for “uses who like x also like y” for a nice filter bubble? I imagine it complements or my even supersede tags. (Admit I haven’t looked at all the sites people are mentioning in the comments yet- lots of good leads!)
- bwb 2y ago
  ya, I do this; I am not sure what the other websites are doing, though. If you email them I bet they will tell you.
everybodyknows 2y ago
Interesting that Wikipedia considers "campus" and "varsity" to be novel genres, but neither of those words appears here on BISG: https://www.bisg.org/fiction https://www.bisg.org/fiction

PaulHoule 2y ago

Freebase got about 50% of the way to a good book database but Google killed it.

deleted 2y ago

[deleted]

greenie_beans 2y ago

You want to read a screed about this? I just published bc I couldn't figure out how to share unpublished substack post, but it needs to be edited: https://bookheadlife.substack.com/p/the-mess-of-bibliographic-metadata https://bookheadlife.substack.com/p/the-mess-of-bibliographi...