13 ms·
Algorithmic tagging of Hacker News or any other site
- platypii 12y agoDirect link to HN with tags: http://algorithmia.com/demo/hn http://algorithmia.com/demo/hn
- EGreg 12y agoI always wondered -- how are some sites able to get an up-to-date mirror of HN, when HN blocks usage of its API after a while? Are they using some alternative API that was blessed by HN?
- doppenhe 12y agowe just use their RSS
- EGreg 12y agoAnd they don't block that after a while? How does iHackerNews show all the comments and everything? Where are these RSS feeds? I doubt that's how "the pros" so it.
- doppenhe 12y agohttps://news.ycombinator.com/rss https://news.ycombinator.com/rss
- GFK_of_xmaspast 12y agoWhat good is a blocked RSS to anybody?
- maxerickson 12y agoThere is also an api: https://hn.algolia.com/api https://hn.algolia.com/api I would think lots of apps are still scraping pages.
- snippyhollow 12y agoI did that in 2012 for a pet project with a friend https://github.com/SnippyHolloW/HN_stats https://github.com/SnippyHolloW/HN_stats Here is the trained topic model (Nov. 30, 2012) with only 40 topics (for file-size mainly) https://dl.dropboxusercontent.com/u/14035465/hn40_lemmatized.ldamodel https://dl.dropboxusercontent.com/u/14035465/hn40_lemmatized... You can load it with Python: from gensim.models import ldamodel lda = ldamodel.LdaModel.load("hn40_lemmatized.ldamodel") lda.alpha = [lda.alpha for _ in range(40)] # because there was a change since 2012 lda.show_topics() Now if you can figure out what is this file: https://dl.dropboxusercontent.com/u/14035465/pg40.params https://dl.dropboxusercontent.com/u/14035465/pg40.params I'll pay you a beer next time you're in Paris or I'm in the Valley. ;-)
- doppenhe 12y agoI'll take a look, thanks!
- sitkack 12y ago[0.006791154078718692, 0.004654721825624361, 0.004632114646875695, 0.011976800134937546, 0.01799155954072435, 0.009181094647452455, 0.0345230793213232, 0.005232498042562552, 0.011402661654834138, 0.009024477282685779, 0.007034922780349653, 0.0031922239118904504, 0.007097905854058182, 0.004999249488505551, 0.016499595508879424, 0.024729974527642036, 0.004985711413178751, 0.03119529793092641, 0.015437847520669401, 0.2948424084650949, 0.06912364384956156, 0.004776347484051836, 0.0893067258386264, 0.018226129712679208, 0.0315656235097838, 0.006267920316323028, 0.01240414536928756, 0.005343403072840281, 0.006566103139036195, 0.009403510178615212, 0.009875448474490003, 0.0038449507757111973, 0.007531241580033292, 0.0014680865916836157, 0.00767397135040071, 0.002118254148078463, 0.02605710351719099, 0.034716581721697254, 0.002314474872742398, 0.12599103592023264]
- snippyhollow 12y agoYes but what does it mean?
- Goosey 12y agoLooking at the hn demo, I'm impressed. There are definitely relevant tags being generated. Unfortunately there also some noisy tags which clutter the results. Taking one example, the post "DevOps? Join us in the fight against the Big Telcos" given the tags "phone tools sendhub we're news experience customers comfortable", I would say that "we're" is unarguably noise. Another example, "Questions for Donald Knuth" with tags "computer programming don i've knuth taocp algorithms i'm" I would call out "i've" and "i'm". There are other words in both examples that I personally would not use as tags, but I can't really say they would be universally not-useful. I think a vast improvement could be made just by having a dictionary blacklist filled with things like these - from this tiny sampling contractions seem to be a big loser.
- doppenhe 12y agoAgreed. Actually we could turn up the number of times it runs inside LDA algorithm and those would fix up but it affects performance. This was just a quick and dirty example (with an expectation of high traffic). You can also seed LDA with a whitelist of words which we didn't do either - again all in the name of a quick and dirty solution to show. Glad you liked it!
- ppod 12y agoTry using tf-idf instead of raw word frequencies.
- doppenhe 12y agonot using raw word frequencies but http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation. Didnt know about tf-idf thanks for the tip.
- hnriot 12y agoit's hard to imagine someone knowing an LDA without also knowing about TF.IDF (it's a dot product, not a hyphen)
- shawabawa3 12y agoDoesn't seem to handle pdfs properly. For the mtgox link it comes up with > stream rotate type/page font structparents endobj obj endstream
- doppenhe 12y agothe demo version if we cant scrape the text from the HTML we cant really run the topic analysis against it. pdfs, images, etc wont work.
- NicoJuicy 12y agoLol, this seriously took me by suprise. I'm currently developing a HackerNews with tags (you can self host it). I quickly generated this Google Form, if you are interested for being a beta user in the nearby future https://docs.google.com/forms/d/1UeSD11hrjwhsVbbPiv63VZBrEczzG5Tr4lwkuKAzY8A/viewform?usp=send_form https://docs.google.com/forms/d/1UeSD11hrjwhsVbbPiv63VZBrEcz... PS. Screenshot included + it's already in alpha in a company with 100 users.
- zokier 12y agoDo you know https://lobste.rs/ https://lobste.rs/ which is essentially HN with tags.
- NicoJuicy 12y agoYeah, i know lobste.rs. But i go much further then HN or lobste.rs.. (not limiting with only url's or texts is just one feature). It's more a "Document Management System" with HN influence for larger Businesses (or public websites) with ( > 30 users) then a HN copy. Call it lobste.rs 2.0
- NKCSS 12y agoNot too impressed to be honest; singular/plural forms are not treated equal; not familiar with LDA, but I've written and LSA implementation in the past, and it did a lot better than what is shown here.
- lugg 12y agoI'm sure there is an amusing LSD joke in there somewhere.
- Theodores 12y agoThis does well on the 'T Shirt test' on some sites, e.g. http://www.riverisland.com/men/t-shirts--vests http://www.riverisland.com/men/t-shirts--vests This could be really useful in ecommerce for creating search keywords for category pages. The noise in the results matters not, so long as it gets 'T-Shirt' and someone searches for 'T-shirt' then all is well and good. Are you looking to plug what you have into something such as the Magento e-commerce platform? The right clients could pay proper money for this functionality. It is something I would quite like to speak to you about.
- doppenhe 12y agodefinitely interested please email me at diego at algorithmia dot com
- gibrown 12y agoLDA/Topic Modeling is interesting stuff. I always feel like the way this data gets surfaced as "tags" is very ineffective. Any non-tech person would look at this and generally be confused. So this item is triggering my rants against tagging: - Tagging is like trying to predict the future. What word will help some future person to get to this content? - Tagging often tries to fill the hole left by bad search - There is no evaluation method to measure how good a set of tags are - Tags make very bad UI clutter. Some of these points are related to encouraging users to tag content, but auto-tagging also seems problematic. To me something more along the lines of entity extraction is more useful because it is a well defined problem, and can be used to improve a lot of other applications.
- sitkack 12y agoIt seems like you would want to run k-means over the comments and the tags to pull out semantically meaningful words for tags and then reduce the total number of tags over the corpus. Then say, use wikipedia to generate an automatic taxonomy where those extract words occur.
- doppenhe 12y agowe do have k-means in our API as well this wouldn't be hard to do.
- sytelus 12y agoTagging is useful to summarize the content. It's like saying describe this article in 3 nouns. Lot of HN articles are cryptic and if you can pull out good tags it can be helpful to prioritizing its reading. It's even more helpful when there are 100s of comments and you want to know key topics of discussion. The problem is often that generated tags are often very poor quality. To understand the utility of tagging, look at some article, read it and then put 3 words that best describes the topics. I bet most others would find human generated tags very useful. Machine generated tags are usually no where close to what humans would generate.
- lugg 12y agoI would like to see this kind of tagging being used to improve search results and simply hiding them. Could increase rank when there are more tag hits. Although I guess that is essentially what good search is. Tags while cluttering the ui do help you find similar content. Still not as good as a good reccomendation system but a decent stop gap measure in some instances.
- EGreg 12y agoLDA is very impressive. But it might be better to have an iterative algorithm that forms a linear-algebraic basis from several tags (and let people add more tags as vectors into the mix) and then every time people upvote something, you update their interests (points in the linear algebraic space) and then every time an article gets upvoted you update ITS tags ... after a while the system converges to a very useful structure and new members can see correctly tagged articles and the system learns their interests by itself do you know anything like this already existing?
- frik 12y agoslashdot.org tag system ?
- dlsym 12y agoThis has real poetic potential: "Erlang and code style" process erlang undefined file write data true code
- NicoJuicy 12y agoI like this project (i am creating something like this, so i'm pretty serious). But doesn't the auto-tagging feature make to much noise for a business use-case? For example, it tags a article of Amazon and includes Google in the tags. White-listing words wouldn't fix this (Google is a whitelisted word if Amazon is). I don't know about LDA though. Perhaps a proper tag administration would fix this, but then you'd have to remove tags on the go.
- doppenhe 12y agowould love to chat more diego at algorithmia dot com.
- zokier 12y agoAfter watching "Enough Machine Learning to Make Hacker News Readable Again"[1] I thought of recommendation engine/machine learning based linkshare/discussion system (eg HN/reddit style). Your frontpage would be continuously formed by your up/down-votes. I'm not sure if the same could be applied to comment threads too, essentially creating automatic moderation. Algorithmic tagging would certainly be useful for that kind of site. [1] https://news.ycombinator.com/item?id=7712297 https://news.ycombinator.com/item?id=7712297
- superduper33 12y agoCareface.
- doczoidberg 12y agoWith german sites it does not work so well. There is no blacklist for to generic terms for other languages than english?
- vhf 12y agoVery interesting. I have been doing some research towards automatic tagging lately, and I found several Python project coming close to this goal : https://pypi.python.org/pypi/topia.termextract/ https://pypi.python.org/pypi/topia.termextract/ , https://github.com/aneesha/RAKE https://github.com/aneesha/RAKE , https://github.com/ednapiranha/auto-tagify https://github.com/ednapiranha/auto-tagify but none of them is satisfying, whereas Algorithmic Tagging of HN looks pretty good. I have been trying to implement a similar feature for http://reSRC.io http://reSRC.io, to automagically tag articles for easy retrieval through the tag search engine.
- draz 12y ago@doppenhe - any hunch as to how well it would work on transcripts?
- doppenhe 12y agoyou can try it yourself at the bottom of the blog post or you can send me a url and I can try a bunch for you diego at algorithmia dot com or @doppenhe.
- hnriot 12y agoMaybe also take a look at AlchemyAPI
- justplay 12y agolooks cool.
- andrew_gardener 12y agoAfter receiving 42 comments, I've ran their tagging algorithm on this page and got: tags tagging hours link doppenhe reply ago lda looks pretty promising!
- vincentbarr 12y agoerror: failed to find worker for algorithm
- doppenhe 12y agoHN took us down for a second. Back up and running. Thanks for reporting.
- pjbrunet 12y agoI signed up. Not sure if i would use it, but the Algorithmia concept is pretty interesting.
- sytelus 12y agoWell, it's not that easy. The algorithms are very primitive and too full of noise to be useful. For example, try this on restaurant reviews like http://www.yelp.com/biz/el-gaucho-seattle http://www.yelp.com/biz/el-gaucho-seattle. I get these tags: steak reviews seattle food service gaucho restaurant review Not useful, right? The current state of the art would use much more sophisticated NLP for generating POS tags and use sentiment analysis. For example, check out MSR Splat at http://research.microsoft.com/en-us/projects/msrsplat/default.aspx http://research.microsoft.com/en-us/projects/msrsplat/defaul....
- nopal 12y agoHas anyone seen Open Calais [1]? It does tagging and categorization. It's been around for years and seems pretty powerful. It's a bit lower-level than Algorithmia (not href aware), but it seems more powerful, and a system like Algorithmia could be built on it. [1] http://www.opencalais.com/about http://www.opencalais.com/about