4 ms·
Not really. Using a statistical approach to text modelling, specifically using Markov Chains, was proposed by Shannon in 1948. But yeah, there's a point in the
by Polygator 6y ago
Not really. Using a statistical approach to text modelling, specifically using Markov Chains, was proposed by Shannon in 1948. But yeah, there's a point in the 2000s where generative grammar/ symbolic approaches were pretty much left behind by NN methods.
When we discuss Google's input in NLP, the most important contribution is certainly the "Attention is all you Need" paper, which paved the way for BERT and GTP (Alphafold also uses Attention networks, btw)
- adw 6y ago> generative grammar/ symbolic approaches were pretty much left behind by NN methods Which is the same thing as hand-engineered feature stacks being left behind in vision problems, really. The story in every field is more or less "you're not clever enough to engineer good features"; "you might be clever enough to define good symmetries for the feature space in which the features live... maybe" (convolutional neural networks in image problems); "... but maybe not even that" (attention mechanisms).
- snovv_crash 6y agoThe hand generated features are still superior for SfM style problems, where the geometry is well defined but would need to be learned by the NN from scratch.