5 ms·
I did something pretty similar over christmas, though I used named entity recognition to extract book titles rather than looking for amazon links, and (so far)
by _lpa_ 11y ago
I did something pretty similar over christmas, though I used named entity recognition to extract book titles rather than looking for amazon links, and (so far) also limited it to specific "Ask HN" threads about books. You can find it here: http://www.hnreads.com/ http://www.hnreads.com/. It is interesting to see how little overlap there is between the two, though that may be due to my using far fewer (and also newer) threads!
- gkst 11y agoCool project! Do you mind telling which library or service you used for the named entity recognition?
- _lpa_ 11y agoI actually wrote it myself, based on conditional random fields! I would not recommend doing this - it was mostly an exercise to learn how they work. I should probably have mentioned that I manually cleaned up the results a bit. I understand nltk has NER capabilities and python bindings, though.
- gkst 11y agoThanks for your reply! Going to look into the NER capabilities of NLTK.
- jccalhoun 11y agoAs a non-programmer I find your list more interesting.
- erispoe 11y agoInteresting that your list yields mainly fiction, while OP's list is mainly non-fiction.
- temo4ka 11y agoMany thanks for sharing this. One usability nitpick though: please consider fixing links so that they could be opened in new tabs/windows.
- tedmiston 11y agoOpened 7 new tabs from the interesting books to me before I realized this didn't work too. It seems the URLs are not quite "pages" -- opening one and refreshing it redirects home as well. Nonetheless, this is an awesome project.
- _lpa_ 11y agoYeah this is my mistake - sorry :(. I'll fix it as soon as I have time.
- chris_wot 11y agoIf it's not too much bother, any chance you'd be willing to put this on a git repository, like GitHub or Gitlab? It would be cool to be able to contribute to: for instance, being able to filter the list to fiction and non-fiction would be pretty great :-)
- _lpa_ 11y agoI might be a bit embarrassed with the code quality - this was one of my first attempts at front end development. I was planning to expand the features of the site a bit, including genre filters, so perhaps I could clean it up a bit and release it when I've done that.
- chris_wot 11y agoOh, I wouldn't worry about that :-) if you are concerned, just say it's rough and ready in the readme. I doubt anyone will be critical, and if they are then tell them to piss off! The beauty of open code is others can help improve it
- Pietertje 11y agoThanks for sharing! This list is a list I actually expected the OP's list to be. Probably because I'm also more likely to view the Ask HN threads about books.
- bitcointicker 11y agoSurprised to see Permutation City in that list. Given that the book is written in 1994, Gregg displays admirable prescience about how computing would develop. Honestly you would think it was written in the last 5 years or so. His vision of cloud computing is absolutely outstanding. It blew me away when I checked when the book was written after reading the first few chapters. I'd read Schild's Ladder prior to reading Permutation city, which is also a good read. It does seem to get bogged down in the technical and descriptive side of things at times, however, it's a fantastic idea for a story. The main premise of the film would make a great movie. Whilst I'm on the subject of good "Hard sci-fi" novels, Tau Zero is also worth reading. Edit - I'll also throw this in: http://www.amazon.co.uk/gp/product/0814703259 http://www.amazon.co.uk/gp/product/0814703259 Magic :-)
- jcrites 11y agoIf you like Permutation City, you might also enjoy another of Egan's works, Diaspora. http://www.amazon.com/Diaspora-Novel-Greg-Egan/dp/1597805424/ http://www.amazon.com/Diaspora-Novel-Greg-Egan/dp/1597805424... "Since the Introdus in the twenty-first century, humanity has reconfigured itself drastically. Most chose immortality, joining the polises to become conscious software. Others opted for gleisners: disposable, renewable robotic bodies that remain in contact with the physical world of force and friction. Many of these have left the solar system forever in fusion-drive starships. And there are the holdouts: the fleshers left behind in the muck and jungle of Earth—some devolved into dream apes, others cavorting in the seas or the air—while the statics and bridgers try to shape out a roughly human destiny." Egan's books have been some of the most thought-provoking I've ever read as far as science fiction technology. A lot of the works were out of print until recently; I'm glad to see there's been a resurgence of interest in his writing, and the availability of his works.
- ngoldbaum 11y agoAnother Egan book well worth reading is Schild's Ladder, which reminds me a lot of Diaspora and was recently reissued in the US: http://www.amazon.com/Schilds-Ladder-Novel-Greg-Egan/dp/1597805440 http://www.amazon.com/Schilds-Ladder-Novel-Greg-Egan/dp/1597...
- edpichler 11y agoGood result. Like a curated list, but automated. I saved here for my reading list.
- wtracy 11y agoOn the system browser on my phone (a Kyocera Rise running Android 4.0.4) all I see on that page is the header and footer, no content. :-( I get the same result if I hit it with Firefox with Javascript disabled, but Javascript is very much enabled on my phone browser.
- rahimnathwani 11y agoThis is really cool. I started reading HPMoR after finding it. I'm curious: after performing NET on the corpus, how did you filter to find books only? Did you just search on Amazon's catalogue for exact matches, or was more tweaking required?
- _lpa_ 11y agoI built a little tool to help with that, but pretty much the first result on amazon was always right - for books with more than one match, at least.
- rahimnathwani 11y agoThanks. How did you deal with, e.g. 'Steve Jobs'? I guess a generic NER tool would extract that as a single entity. Then you'd search Amazon and find an exact title match (the biography by Isaacson). But you probably don't want this in your list, as the references were to Steve Jobs the person, not Steve Jobs the book. So did you use some manual review step to remove these?
- _lpa_ 11y agoWell, the nature of NER is that it can extract 'Steve Jobs' as either a PERSON or a BOOK entity (in my case). You can in principal extract whatever kind of entity you want, provided you have appropriate training data. The idea (loosely) is that it learns the structure of text around, say, a book. You can then label a piece of text as a book, despite not previously knowing that title. It doesn't solve the problem of two books having identical titles, however :). (Note: I'm not an expert in NER/NLP, so please correct me if I have got something wrong!)
- rahimnathwani 11y agoAha, I see. Thanks. I thought you were using some generic pre-trained NER, but I understand now that you used either one which was pre-trained to find books, or you trained your own.
- fratlas 11y agoOne thing that struct me about your site (apart from being a good list, well done!) is how blazingly fast (close to HN, which I find funny) the page loads. Could you fill us mere mortals in on how the fuck you got it so fast?
- res0nat0r 11y agoLooks like fastly: $ host www.hnreads.com www.hnreads.com is an alias for global-nossl.fastly.net. global-nossl.fastly.net has address 23.235.44.204
- cwh 11y agoit's a simple js view for a 1.4mb json file, so "you got it so fast" really just depends on how quickly you will download that 1.4mb file.
- jonahx 11y agoIt looks like it loads all the data on the initial page load. After that, it's a SPA app and all the links are just drawing local data.
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- ALee 11y agoIt would be great if you or OP could do the analysis again on HN Reads, but with the larger dataset.