4 ms·
Good lord what I wouldn’t give for a better goodreads. Never has so much useful and actionable data been squandered. For example, I could write pages about how
by johnfn 6y ago
Good lord what I wouldn’t give for a better goodreads. Never has so much useful and actionable data been squandered. For example, I could write pages about how bad the “top books” lists are. The Goodreads Choice Awards are purely and literally popularity contests, for example. Why would this be the case - you have millions of user ratings, you should be using those to surface exciting and unknown books rather than throwing people back at the same few authors they already know. The lists work exactly the same way - they discard ratings entirely and only take into account the quantity of books. It would be as if you said that the top 40 is the best music because everyone listens to it.
So you don’t think I’m all talk, a while back I got so fed up with this that I wrote my own script to scrape goodreads and find the actual best books, not just the most popular, and I found a wealth of really good and unknown books, including two books that are now my favorite books of all time. It was a side project that took me an hour. Why goodreads can’t do this is utterly beyond me.
Aaaargh!!!
- PaulStatezny 6y agoWould you share the titles and authors of those two books?
- johnfn 6y agoInfinite jest by David foster wallace and only forwards - forget the author name off the top of my head. Infinite jest is pretty popular these days. The other I have yet to hear anyone say they have read.
- tomstuart 6y agoI’ve read Only Forward by Michael Marshall Smith. It stuck in my mind the same way that Vurt did. I still think about the man with no head sometimes, and the idea of a road being different when you’re going the other way on it.
- defen 6y agoLiking Infinite Jest suggests an interest in American literary maximalism and / or comic novels. Suggestions: The Sot-Weed Factor, Gravity’s Rainbow, Mason & Dixon, Underworld, and if you’re feeling particularly masochistic The Tunnel
- SamBam 6y agoOk, but was Infinite Jest ever an "unknown" book? I'm pretty sure it was one of the most hyped books in the world the year it was written. And I'm pretty sure it's always been in the 4.x's on Goodreads.
- johnfn 6y agoI’d never heard of it nor seen it on goodreads till I made my list. I’m sure it’s popular in some circles, but not mine.
- taeric 6y agoI'm confused. What you describe is really just another form of popularity contest, no? Unless you have a set of reviewers that you value over others. One might call them the critical reviewers. (No, I'm not trying to be subtle) So, what books did you find? Why not make a post on your method?
- johnfn 6y agoNot exactly. Goodreads will value a million people giving a book 1 star higher than 20k giving it 5 stars. I just sorted by average score. I posted the books in this thread (sorry, on mobile right now).
- taeric 6y agoBut that is still a popularity contest, of sorts. In general, any ranking system is a popularity system. Which is logical, as there is no objective ranking of books. That is, it isn't an ordered set. That said, I can see value in providing a score not just in the items, but on the ranking system used. Could even be adaptive based on genre. (Indeed, I would love a system to see if any books have factual errors and such. But, that is just trying to reinvent the citation systems of academia.) And understood on the mobile. Same boat for me.
- johnfn 6y agoI don’t understand. A book with a single review could place higher than a book with millions. How is that a popularity contest?
- taeric 6y agoDepends on who reviewed it. Easier to think of this from the other side. I have a few friends that, if they tell me to avoid something, it can typically override however popular something may be.
- xuhu 6y agoHow did you define the actual "best books" ? I usually pick a group (Goodreads users in country x), scrape the books of all its users, and sort them by frequency: https://github.com/harjoc/goodreads-group-books https://github.com/harjoc/goodreads-group-books
- bbarnett 6y agoThinking on johnfn's response, and yours, these are all simple SQL queries. You could easily delineate to others what each query does, and let people choose from a list. EG "best average" "most reviewed" "users in country + frequency" and such. This doesn't even need to hit performance, as these queries could all run on a secondary SQL server, be cached in redis/whatever once an hour, or even day. After all, this sort of list output doesn't need to be second by second changing. So simple to implement, non-performance impacting, which makes it even more curious as to 'why not'. Likely, a UI/UX person thinks it will 'confuse' people. (I know that may not be the case, but it seems to kill more sites, and more flexible functionality, than anything...)
- iamkroot 6y agoCan you share this script?
- johnfn 6y agoWow, turns out I still have it on GitHub, including the results I ground out 8 years ago. The script probably doesn't work, but the results are still good: https://github.com/johnfn/GoodReadsScraper https://github.com/johnfn/GoodReadsScraper Particularly, load bigdata.js into nodejs and then run a command sort of like this to parse out the results, filtering out young adult/romance/religion stuff/comics: bigdata.filter(f => f.ratings > 5000 && !f.genres.includes('Young Adult') && !f.genres.includes('Religion') && !f.genres.includes('Romance') && !f.genres.includes('Sequential Art')).map(f => f.title) I get some pretty interesting stuff. First result is the Constitution... OK, fair enough... but the next 10 or so are: 'A Song of Ice and Fire', 'Collected Fictions', (by Borges) 'The Name of the Wind', 'Infinite Jest', 'The Complete Works', 'The Way of Kings', 'The Wise Man\'s Fear', 'Ficciones', 'A Storm of Swords: Blood and Gold', 'The Complete Stories', 'Labyrinths', 'Don\'t Let the Pigeon Drive the Bus!', 'The Hiding Place', Is it an amazing list? Eh. Is Name of the Wind better than Infinite Jest? Probably not. They're both fantastic books, though! And still, it's way better than Listopia. Also, this data is 8 years old. I bet it would be way better if I were to clean it up and run it in 2020.
- jordiburgos 6y agoPlease share the results of your script !! :)
- johnfn 6y agoPosted it elsewhere in the thread - go check it out.