5 ms·
A related thing that amuses me: One of the (less popular†) benchmarks for embedding models is their performance on a TEFL word analogy test. At least as of the
by mr_crankypants 7y ago
A related thing that amuses me: One of the (less popular†) benchmarks for embedding models is their performance on a TEFL word analogy test. At least as of the last time I perused the literature, which was a few years ago, there was one algorithm with which I've seen anyone report really high scores on this test. For all the crowing about how great Word2Vec is at finding analogies, it wasn't SGNS or CBOW. Not GloVe, either. It was boring, unsexy, 30-year-old, doesn't-even-have-a-strong-theoretical-foundation latent semantic analysis.
† Because it doesn't measure performance at something people typically want to do with these models in real life - there's a strong "party trick" component to the word analogy stuff.
- Der_Einzige 7y agoTopic models are going to get blown the hell out by UMAP and other manifold based dimensionality reduction algorithms
- mr_crankypants 7y agoI haven't been following the literature too closely lately - have there been promising results in using it for topic modeling or semantic vector modeling? My own sense is, the real problem that the space needs to contend with isn't that the math isn't elaborate enough. It's that we're still waiting for someone to come up with a really good way to deal with polysemy, and with multi-word phrases that form a single semantic unit.
- Der_Einzige 7y agoRe: promising results Yes, there have been. https://github.com/lmcinnes/umap_paper_notebooks https://github.com/lmcinnes/umap_paper_notebooks The author of UMAP shows impressive results with 3 million word vectors For dealing with multiple meanings, It's already been done. Sense2vec resolves most of these issues and a wordnet integrated version of word2vec or the newer "XLNet" would be state of the art by a long shot but no one seems to want to implement it so the world waits longer for good NLP models I guess...
- mr_crankypants 7y agoSense2vec isn't really what I'm waiting for. It relies on word sense disambiguation, which tends to be one of those very language-specific things, and so I'd expect (but haven't verified) that, like other techniques that rely on language-specific bits, it wouldn't work as well on most non-English text. And the most interesting polysemy problems aren't to do with part of speech. They're things like "apple-as-in-food" vs "apple-as-in-computer", or figuring out that "The Big Apple" doesn't have anything to do with either of those. What would be really interesting is dealing well with jargon, slang, and terms of art. As far as those notebooks, is there one in particular I should be looking at? I might have missed something, but the stuff I saw basically just demonstrated, "Hey, we can handle a lot of training data really fast." What I'd be more interested in seeing is, "Hey, plug us into your document classification pipeline and your performance (as in accuracy) metrics won't know what hit them." edit: For a more concrete example of what I'd like to see, and going back to the analogy task: The holy grail I'm looking for isn't "king - man + woman = queen". It's more like "software engineer = programmer", and also "software engineer != software + engineer".
- Der_Einzige 7y agoYou'll find that if you run UMAP on a large corpus (the same size as your original word embeddings), the ones it'll generate (especially if you feed it any labels as UMAP supports semi-supervised and supervised dimensionality reduction) should outperform those generated I'd even wager by modern transformers. if they don't, than they'll be like 2% worse for a lot of speed improvement on the currently single threaded implementation of UMAP Oh and you can use UMAP to concat tons of vector models together and all other side data for super-loaded embeddings
- visarga 7y ago> we're still waiting for someone to come up with a really good way to deal with polysemy It's been done: contextual embeddings - ELMO, BERT.