I think we've all seen the state-of-the-art with words the system have never seen before, and it's not very impressive. I think you confused some terms in your

I think we've all seen the state-of-the-art with words the system have never seen before, and it's not very impressive. I think you confused some terms in your comment. Word embeddings like word2vec, is a vector that considers surrounding words typically, and is really just a way to compress one-hot vectors into a smaller dimensional space. So it can infer from the surrounding words that it is similar in position to another word in a given word sequence but that is pretty much it. Generally you cannot vectorize a word it has never seen in the fitting stage of word2vec. I think once you really start hacking away at this stuff, you will someday come to the thought: is my "creative" model over-fitting (repeating its trained inputs), or is this something completely new generated? Where do you draw that line? A word embedding AI cannot invent new words. It can make new word combinations, and even pick up grammar rules, but the results suck. I have tried my hand at text generation using LSTMs, and much smarter people than me have tried, and it's not looking great. I will continue hacking away at it however :)