3 ms·

This is great, but an important caveat: this is actually “books for which the Amazon links are most frequently given” (maybe other sites too; the About page doe

by svat 8y ago

This is great, but an important caveat: this is actually “books for which the Amazon links are most frequently given” (maybe other sites too; the About page doesn't clarify), i.e. “most-linked books”, rather than “most-discussed books”. For example, it says that on the TeX/LaTeX StackExchange, The TeXbook is mentioned only 6 times (https://bookinsider.gitlab.io/2018/12/01/top-20-books-on-tex/ https://bookinsider.gitlab.io/2018/12/01/top-20-books-on-tex...), while in fact it's mentioned closer to 1357 times (https://tex.stackexchange.com/search?q=texbook https://tex.stackexchange.com/search?q=texbook). What appears to be true is that only 6 times someone bothered to link to Amazon when mentioning the book. Worse, for the English Language & Usage site, the Oxford English Dictionary does not even show up in the results (https://bookinsider.gitlab.io/2018/12/01/top-20-books-on-english/ https://bookinsider.gitlab.io/2018/12/01/top-20-books-on-eng...) while in reality it's mentioned about 7000 times by acronym (https://english.stackexchange.com/search?q=oed https://english.stackexchange.com/search?q=oed) and about 1600 times by full name (https://english.stackexchange.com/search?q=%22oxford%20english%20dictionary%22 https://english.stackexchange.com/search?q=%22oxford%20engli...). Under what circumstances will someone add a link when mentioning a book? I can think of two: - The user thinks the book is not sufficiently well-known, so they add a link to Amazon or some other such site, for the readers to learn more. - The user is trying to make money off affiliate links, or whatever. For books that are well-known, or for the typical (lazy) user like me, books are going to mentioned and discussed without any link to anywhere being added (and often by acronym or nickname). So in that sense this site is actually likely to miss all the most frequently discussed books — the ones so well-known that no one bothers to link to anything when mentioning them (as in the examples above). Shows that the hard part of data analysis is usually data cleanup (eliminating false positives and false negatives) and normalization (this one seems to treat links to different editions on Amazon as different books). All that said, this site is useful nevertheless; thanks for making it!

BookInsider 8y ago
Thank you for your input! I will keep improving the site based on your suggestions and advice from others. I add a patreon link to my site so hopefully that can become a progress log. https://www.patreon.com/BookInsider https://www.patreon.com/BookInsider
dorkwood 8y ago
I tried to do something like this years ago by scraping websites in a particular niche. I decided that I wanted to count a book any time it was mentioned -- not just when it was linked to. As a junior-level programmer, however, it was a problem I couldn't solve. How do you determine whether someone using the word "mindset" is talking about the book by the same name, or just using the word?
- svat 8y ago
  It's a hard problem; one you can't ever fully solve. You need to accept some degree of error, as pretty soon you'll hit the point where every incremental improvement requires you to basically double the amount of effort. (You can start by incorporating some signals like whether the word/title occurs in uppercase, whether it has any special formatting like italics, whether the author's name occurs nearby, and so on.) But what you can do is be very up front about the error: you can be explicit about your methods and describe their modes of failure, give examples of some of the things you might miss, try to analyze your error and how bad it is, and so on, and finally leave it to the reader to decide how seriously to take your results. (If you see some recent papers they include a “Threats to Validity” section, e.g. Section 3.4 here: https://people.engr.ncsu.edu/ermurph3/papers/seip18.pdf https://people.engr.ncsu.edu/ermurph3/papers/seip18.pdf)

by svat 8y ago

BookInsider 8y ago

Thank you for your input! I will keep improving the site based on your suggestions and advice from others. I add a patreon link to my site so hopefully that can become a progress log. https://www.patreon.com/BookInsider https://www.patreon.com/BookInsider

dorkwood 8y ago

I tried to do something like this years ago by scraping websites in a particular niche. I decided that I wanted to count a book any time it was mentioned -- not just when it was linked to. As a junior-level programmer, however, it was a problem I couldn't solve. How do you determine whether someone using the word "mindset" is talking about the book by the same name, or just using the word?

svat 8y ago
It's a hard problem; one you can't ever fully solve. You need to accept some degree of error, as pretty soon you'll hit the point where every incremental improvement requires you to basically double the amount of effort. (You can start by incorporating some signals like whether the word/title occurs in uppercase, whether it has any special formatting like italics, whether the author's name occurs nearby, and so on.) But what you can do is be very up front about the error: you can be explicit about your methods and describe their modes of failure, give examples of some of the things you might miss, try to analyze your error and how bad it is, and so on, and finally leave it to the reader to decide how seriously to take your results. (If you see some recent papers they include a “Threats to Validity” section, e.g. Section 3.4 here: https://people.engr.ncsu.edu/ermurph3/papers/seip18.pdf https://people.engr.ncsu.edu/ermurph3/papers/seip18.pdf)