3 ms·

AI can infer information about data that does not appear in the training set, depending on the actual tech being used and what you are calling "new data". An e

AI can infer information about data that does not appear in the training set, depending on the actual tech being used and what you are calling "new data". An easy to grasp example from a while back in NLP is word embeddings. We started out with vectors for each word in the training corpus, which meant we had no information for words outside of the training set, like you say. Then people came up with "character n-grams", which built vectors for substrings, which allowed us to encode information for words outside of the training corpus based on the vectors of the constituent substrings. You could imagine a system that could deliberate across levels of abstraction to find lines of analogical reasoning to connect things together in novel ways that people would find surprising and interesting in similar ways to human creative output, since it's all computation either way, but who knows how far away we could be from making such a system that could actually generate reasonable, surprising outputs.

beiller 8y ago
I think we've all seen the state-of-the-art with words the system have never seen before, and it's not very impressive. I think you confused some terms in your comment. Word embeddings like word2vec, is a vector that considers surrounding words typically, and is really just a way to compress one-hot vectors into a smaller dimensional space. So it can infer from the surrounding words that it is similar in position to another word in a given word sequence but that is pretty much it. Generally you cannot vectorize a word it has never seen in the fitting stage of word2vec. I think once you really start hacking away at this stuff, you will someday come to the thought: is my "creative" model over-fitting (repeating its trained inputs), or is this something completely new generated? Where do you draw that line? A word embedding AI cannot invent new words. It can make new word combinations, and even pick up grammar rules, but the results suck. I have tried my hand at text generation using LSTMs, and much smarter people than me have tried, and it's not looking great. I will continue hacking away at it however :)

beiller 8y ago

I think we've all seen the state-of-the-art with words the system have never seen before, and it's not very impressive. I think you confused some terms in your comment. Word embeddings like word2vec, is a vector that considers surrounding words typically, and is really just a way to compress one-hot vectors into a smaller dimensional space. So it can infer from the surrounding words that it is similar in position to another word in a given word sequence but that is pretty much it. Generally you cannot vectorize a word it has never seen in the fitting stage of word2vec. I think once you really start hacking away at this stuff, you will someday come to the thought: is my "creative" model over-fitting (repeating its trained inputs), or is this something completely new generated? Where do you draw that line? A word embedding AI cannot invent new words. It can make new word combinations, and even pick up grammar rules, but the results suck. I have tried my hand at text generation using LSTMs, and much smarter people than me have tried, and it's not looking great. I will continue hacking away at it however :)