6 ms·
Show HN: Realtime Hacker News Summarization Using Machine Learning
- alistproducer2 10y agoThis would really benifit from an explanation of what I'm seeing. can't really upvote it without knowing what's going on....
- bexp 10y agothere are couple things here: 1. scrapping web page to get text content 2. use NLTK to proccess text and get summary and keywords 3. wrap it into REST API and serve as web service
- jeffehobbs 10y agoYou could probably get cleaner input for step 1 via the Mercury API [https://mercury.postlight.com/web-parser/ https://mercury.postlight.com/web-parser/] — it has a lot of affordances for different kinds of HTML formatting.
- bexp 10y agoThanks will try at some point, my biggest concern was that those kind of API's are almost all paid and rarely open sourced.
- brunopedro 10y agoWhat summarization algorithm are you using?
- bexp 10y agoI'm using variation of TextRank
- acd 10y agoGood summary, how is h 10 built? It would be cool to summarize other sites feed with the same tech.
- Raphmedia 10y agoDoes it summarize our comment threads or the pages it links to?
- ostegm 10y agoCan you describe a little about how you go from raw text to a summary? Is this supervised learning? If so, what is the training data?
- bexp 10y agoIt's a variation of TextRank algorithm
- singularity2001 10y agojs error: "get top stories error index.html:56:4"
- bexp 10y agoshould be fixed by now, thanks for pointing this out
- xiamx 10y agoGot that this is using nltk. Can you comment on which summarization algorithm you are using (TextRank, SumBasic and etc.) What's the motivation towards choosing one algorithm over another and how's its performance on average HN articles?
- mendeza 10y agoThe summary seems to pick the most unique sentences, so maybe LexRank? Amazing though, I've been really wanting a way to skim HN articles quicker, since there is so much volume and you don't want to miss anything Great work!
- a_c 10y agoThis. It would be great if we can have the source code alongside with OP's comment on his/her summarization
- apapli 10y agoAgree. It would be great to see your source code if at all possible, would be greatly appreciated.
- spraak 10y agoI'm glad that someone made this! Ever since I've known about ML and HN I've wanted to make something like this
- ecesena 10y agoNot sure what do you use to scrape, but I can recommend newspaper to extract text. I contributed a while ago and it was pretty solid. https://github.com/codelucas/newspaper https://github.com/codelucas/newspaper
- bexp 10y agothanks will try that
- sohkamyung 10y agoThanks for that. The Newspaper demo site seems to show it working quite well on the news sites I usually browse.
- s_ngularity 10y agoIt would be good if it still included a link to the HN comments for each post
- bexp 10y agodone !
- alexilliamson 10y agoThere are still posts with no link to the comments. Seems like it is happening whenever the summary is also missing below the title (currently "5. Ask HN: Leave job right before app goes to production?" is like this)
- earthly10x 10y agoWould be nice to connect other data to it like stocks e.g. https://cymetica.slack.com/apps/A26G72726-quantbot https://cymetica.slack.com/apps/A26G72726-quantbot
- usgroup 10y agoConsider just using the first paragraph as the summary. For the sort of thing on HN, it'll be spot on more times than not.
- aaronhoffman 10y agoYes, could probably just grab the open graph tag data for most articles and display that. But this is a cool idea.
- mulrian 10y agoIs this open source?
- bexp 10y agonot yet but maybe in future.. There is bunch of summarizing project on github though.
- travelinreid 10y agohttps://thepul.se/ https://thepul.se/ also uses an algorithm based on nltk to summarize general news. (It also includes a nice speed-reader.)