2 ms·
Remember "folksonomy" ...? So I would like to see some kind of automated keyword extraction for each blog to allow selection of topical blogs. OK, keywords wou
by jcynix 3y ago
Remember "folksonomy" ...? So I would like to see some kind of automated keyword extraction for each blog to allow selection of topical blogs.
OK, keywords would need to normalied (e.g.plural to singular, plus maybe a bit of abstraction) and this would be a first (tiny) step to make it a blog-centered search engine ... ;-0
- marginalia_nu 3y agoI've been experimenting with techniques for figuring out the topic of a blog, and never quite gotten anything to work well that didn't involve extreme computational costs that just do not work on a search-engine scale without a serious hardware budget (zero shot classifiers did a decent job IIRC). An unga bunga technique like TF-IDF works well for a given document, but less well for a website. Had some half-decent success finding similar blogs given a reference though.
- jcynix 3y agoYes, I expect full text extraction and normalization to be computationally expensive. I did some simple minded experiments on Usenet feeds back in the 90ies and it was quite some work for the VAX it ran on.
- marginalia_nu 3y agoYou can get it to run fairly fast on modern hardware. Like run a text extraction, tokenization and POS-tagging workflow on a quarter billion documents on PC hardware, takes like 24-36 hours. That's doable and affordable. But ML-adjacent methods are not. Requires far too much GPU compute, have no A100s :-/