8 ms·
Show HN: Using word vectors to classify spam messages
- amelius 9y agoSuggestion for better title: "Collaboratively Filtering Spam with Word Vectors while Respecting Privacy"
- chrbarrol 9y agoI was hoping to learn about word2vec by reading the sourcecode but am I right when I say this has nothing to do with word2vec?
- drwl 9y agoLooks like it uses GloVe and not word2vec. They're both algorithms for generating word vectors but they are different.
- RHSman2 9y agoNot by much
- doody_parizada 9y agoWe started out with word2vec, but discovered it was easier to work with GloVe for our purpose. There are ways to convert glove vectors to word2vec format such as: https://radimrehurek.com/gensim/scripts/glove2word2vec.html https://radimrehurek.com/gensim/scripts/glove2word2vec.html
- chasing 9y agoWhy is Silicon Valley so interested in censoring certain kinds of speech?
- dang 9y agoThe HN community is international and overwhelmingly not based in Silicon Valley. From his GitHub profile it looks like the author of this project isn't either. So what you said is considerably off the mark. Either way, though, please don't post flamebait here. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- chasing 9y ago'Twas a joke based on a series of other highly active threads on this site. I assumed it would be taken as such. My error!
- erik_seaberg 9y agoAs a recipient of email, choosing not to read a message is not an offense against free speech, especially when email has become unusable to the individual without doing so.
- Arnt 9y agoThis sounds like an early version of DCC: https://www.rhyolite.com/dcc/ https://www.rhyolite.com/dcc/ At first glance, I don't see anything that DCC didn't do, what did I miss?
- EmilStenstrom 9y agoIt seems DCC isn't using word vectors at all? Using word vectors you can know that viagra and v14gr4 is the same word, because it is used in the same way in messages. That in turn means you don't need word lists, and can instead build from huge knowledge bases like GloVe.
- massaman_yams 9y agoThat, and the fact that a message is sent in bulk isn't actually a very strong indicator that the message is spam, at least in the email world. As one input to a filtering system, it can be useful, but not as a rule applied on its own without consideration for other factors.
- mci 9y agoSounds like a fun project. However, I doubt if word vectors buy you anything more than, say, old good Nilsimsa from 2001 (https://en.wikipedia.org/wiki/Nilsimsa_Hash https://en.wikipedia.org/wiki/Nilsimsa_Hash). Side note: py-nilsimsa should iterate over Unicode points instead of UTF-8 bytes. As it stands now, the similarity of any texts in the same language using a non-Latin script is ~80 rather than ~0.
- laretluval 9y agoword2vec has the advantage that you could potentially identify spam messages that are paraphrases rather than exact copies of the ones in the training set.
- mci 9y ago1. Pedantically: it's GloVe, not word2vec. 2. Nilsimsa or any locality-sensitive hash detect changed messages, too, be the changes synonyms or not. 3. I don't think OP's GloVe contains words like v1agra.
- doody_parizada 9y agoWe don't have words like v1agra. As I mentioned in the README, we took vectors pretrained on wikipedia. One of the possible improvements can be to train the vectors on our own dataset.
- programmarchy 9y agoSlightly tangential, but does anyone know if word2vec can be used in a compound form to build up "concepts"? I'm interested to know if it could be used to identify parallelism in works of literature e.g. identifying plagiarism, parallels between the old and new testament, or intertextual works like Ulysses by Joyce and the Odyssey.
- physicsyogi 9y agoMaybe look into ConceptNet Numberbatch: https://github.com/commonsense/conceptnet-numberbatch/blob/master/README.md https://github.com/commonsense/conceptnet-numberbatch/blob/m...
- doody_parizada 9y agoOne of the things we found out working on this project, is the problem of converting a word vector to a paragraph vector. Apparently there are many ways to do so and each one yields different results based on the length of the text and content. We used a weighted average of the words based on their frequency in a corpus.
- abc-xyz 9y agoThis may be off topic, but could this be used for classifying the trustworthiness or Amazon/App Store/etc reviews? Or does anyone perhaps know about an open source project that can be used to achieve this by someone who doesn't know anything about machine learning?
- codegladiator 9y ago> https://thereviewindex.com/blog/hello-world https://thereviewindex.com/blog/hello-world