4 ms·
He makes a good point about continuous functions. But actually neural nets are quite good at handling discrete elements like words, likes, atoms in a molecule,
by vonnik 10y ago
He makes a good point about continuous functions. But actually neural nets are quite good at handling discrete elements like words, likes, atoms in a molecule, etc.
Deep learning is part of enormous advances in NLP[0], just as it has set records to accuracy in almost every field of machine perception, from vision to audio.
Neural word embeddings like those produced by word2vec[1] make for very useful feature vectors when fed into other neural nets.
The headline of this post should be that NLP is harder than, say, image processing. In fact, for non-specialists, none of it is easy, because tuning hyperparameters is hard.
The kind of NLP that tries to reproduce human-level sentences and understanding is simply a more complex problem, given the plasticity of language.
[0] https://arxiv.org/abs/1611.04558 https://arxiv.org/abs/1611.04558
[1] https://deeplearning4j.org/word2vec https://deeplearning4j.org/word2vec
- jmmcd 10y agoThere's a good point to be made about continuous functions, but the author hasn't made it, because he/she doesn't know the difference between continuous functions and time-series. > What is a continuous function (or continuous data)? It is a sequence where each item is related to the one before and one after determined by a process. So, to paraphrase the article: My reply should actually stop here with one sentence.
- fnl 10y agoI agree that the author's interpretation of a continuous function is -umm- lacking, but I think it is thought provoking if you try to read it as what he probably meant, a continuous topological mapping from the character sequence to some function of the semantics of the text. Assuming that mapping does not exist would imply that we'd need more than a LSTM to build a proper language model some day. Yet, LSTMs are probably as close to a perfect model as we can currently get... So maybe this line of thought can help somebody imagine the "missing piece"?
- fnl 10y agoGood point. I think the author quite clearly meant continuous wrt. a whole sentence or even text, not single (sparse) words, though. Take German, for example: The last word in a sentence that ends in a verb defines the whole structure of the sentence. Like, "Wir haben heute etwas über Neuronale Netzwerke gelernt.", which is in part also reflected in particularly problematic languages not being "projective". This gets even worse with "long-range" semantic cross-references, like anaphora. And, the latest Google Translate is still notoriously bad at the German language, at least [1]. Therefore, I think the article has a valid point, and I'd really like to hear some actual points where he is wrong before dismissing it (though, yes, I agree it is rather shallow, and best a good thought starter). [1] Google Translate on the German Wikipedia entry for Weihnachten (X-mas): https://translate.googleusercontent.com/translate_c?depth=1&nv=1&rurl=translate.google.com&tl=en&u=https://de.wikipedia.org/wiki/Weihnachten https://translate.googleusercontent.com/translate_c?depth=1&...
- nl 10y agoWhy do you say German is notoriously bad? I don't speak German, but that linked page reads fine to me, and their neural translation system has state-of-the-art performance for both English->German and German->English at least[1]. I would have thought that the well-structured nature of German makes it reasonably easy to translate, assuming you have a NN architecture with sufficient memory range. [1] https://arxiv.org/abs/1611.04558 https://arxiv.org/abs/1611.04558
- fnl 10y agoThe translation isn't "only nonsense", and in many cases even grammatically correct, but mostly semantically wrong, or rather weird. In any case, a long shot from a human translation. But I wouldn't agree that it's "quite good". And, this is the easy case. EN->DE works even worse.
- Houshalter 10y agoDoesn't German have a ton more compound words though? That could be an issue because their system is word based.
- vonnik 10y agoIt does. In a way you face a parsing problem similar to Asian languages that don't use spaces to separate words.
- fnl 10y agoActually, the issue about continuity the article addresses is just the same for word embeddings, namely when there is an ambiguity. If the word has more than one meaning, the correct mapping is context dependent and the "one mapping fits all" approach that we are currently taking here is semantically incorrect. Yet another hard problem not easily resolved, and also attributable to "long-range dependencies".