4 ms·
I always wanted to generate a simple service that classifies websites. Something where you dump in the HTML/URL and it returns something like "agriculture"/"gov
by rb2k_ 16y ago
I always wanted to generate a simple service that classifies websites. Something where you dump in the HTML/URL and it returns something like "agriculture"/"government"/"retail"/"education".
I already have a set of a few thousand classifications at hand. What would probably be a good algorithm to run it through? I assume I'd use something like webstemmer/boilerpipe/... to extract just the main text first.
What I am a bit uncertain is what I should do after that. My guess would be that I isolate the nouns/adjectives with the highest frequency and do a clustering with my already categorized dataset as training data.
Does that somewhat makes sense?
If yes: any recommendations or alternatives for libraries (preferably ruby) or just algorithms themselves (k-means, svm, neural network...)
- buss 16y agoI think you'd want something like LDA. See https://cwiki.apache.org/MAHOUT/latent-dirichlet-allocation.html https://cwiki.apache.org/MAHOUT/latent-dirichlet-allocation....
- rb2k_ 16y agoThanks for the recommendation! Even found a ruby wrapper arround the C lib: https://github.com/ealdent/lda-ruby https://github.com/ealdent/lda-ruby
- hooande 16y agoFor simple text classification, you should just use something like Wekka, rainbow or the Google Prediction API. The hardest part is always labeling and verifying your category training data. There are many open source algorithms that can do the classification and any form of naive bayes will probably be good enough.
- flavy 16y agoHi,we are launching a service exactly like the one you were referring to. You can try the API already. Check out the docs https://sites.google.com/site/thinkersrus/products-1/scienceapialpha https://sites.google.com/site/thinkersrus/products-1/science... and the demo http://www.thinkersr.us/demo.html http://www.thinkersr.us/demo.html
- rb2k_ 16y agoOh nice, looks interesting and worked pretty well from my few tests. Sadly, with the amount of requests I'd be doing (10000+ just for training), I guess I should build myself something on my own