3 ms·

> 1. Use links, when there’s a list Yep, this is on my list to do for next year. I just ran out of time to do that this year (this is the first year I've done

by bwb 3y ago

> 1. Use links, when there’s a list Yep, this is on my list to do for next year. I just ran out of time to do that this year (this is the first year I've done this). I will also rethink what data I show for 2024 as I want to better show which authors picked which books visually. > 2. Get a better source for the book data: I wish there were a better source; book data is a nightmare. There is only one source of book data and that is the publishers through a format called ONIX. The publishers have no idea what they are doing, and so they stuff the book's "description" full of marketing crap as well as abuse the BISAC-defined genres :(. Goodreads has an army of unpaid people who clean this all up. I am looking at rewriting the book's description using AI as it seems like it is the only choice to get rid of that nonsense without a ton of manual work (since I don’t have user accounts to crowdsource that). Genre miscategorization is harder, I am playing with AI to help me clean that up, but also thinking about using crowdsourcing some of that. Just one example: I found several editions of Dune that report they were published in the 1780s and 1880s... and this is a book selling millions of copies a year.

Semaphor 3y ago
Have you tried archive.org’s OpenLibrary [0]? They have an API and at least sometimes decent data. Great news for 1. ;) [0]: https://openlibrary.org https://openlibrary.org
- bwb 3y ago
  Ya, I tested it in the first year and it was worse than publisher data by a wide margin. I will take another look soon to see if it improved. And maybe I can use their data to augment publisher data and improve the overall quality.
  - Semaphor 3y ago
    Ouch, okay, that sucks. At least the descriptions I checked were always decent.
    - bwb 3y ago
      Ya I didn't test that at the time (more book meta data), that I will look at. I also might be pulling the wrong ONIX data and need to up my license and take on the full HUGE database of books rather than the API I am using.
michaelt 3y ago
> Genre miscategorization is harder, I am playing with AI to help me clean that up, but also thinking about using crowdsourcing some of that. AI could indeed help with that. But if you're publishing a books-of-the-year list, you could also check the categorisation manually - especially if you adjust the inclusion criteria to give you a tractable number of books. For some datasets, 4 hours of manual classification will do a better job than 40 hours of trying-to-get-the-AI-to-work :)

by bwb 3y ago

Semaphor 3y ago

Have you tried archive.org’s OpenLibrary [0]? They have an API and at least sometimes decent data. Great news for 1. ;) [0]: https://openlibrary.org https://openlibrary.org

bwb 3y ago
Ya, I tested it in the first year and it was worse than publisher data by a wide margin. I will take another look soon to see if it improved. And maybe I can use their data to augment publisher data and improve the overall quality.
- Semaphor 3y ago
  Ouch, okay, that sucks. At least the descriptions I checked were always decent.
  - bwb 3y ago
    Ya I didn't test that at the time (more book meta data), that I will look at. I also might be pulling the wrong ONIX data and need to up my license and take on the full HUGE database of books rather than the API I am using.

michaelt 3y ago

> Genre miscategorization is harder, I am playing with AI to help me clean that up, but also thinking about using crowdsourcing some of that. AI could indeed help with that. But if you're publishing a books-of-the-year list, you could also check the categorisation manually - especially if you adjust the inclusion criteria to give you a tractable number of books. For some datasets, 4 hours of manual classification will do a better job than 40 hours of trying-to-get-the-AI-to-work :)