3 ms·

> The other major case in point being NLP Speaking of which, Google Translate was published in 2006, but when did the "learning from data" approach became an a

by sampo 6y ago

> The other major case in point being NLP Speaking of which, Google Translate was published in 2006, but when did the "learning from data" approach became an accepted idea in machine translation? I think the earlier attempts at machine translation were more about trying to codify grammar rules in software, than doing statistical learning from large text corpuses? I remember in 2002, the approach of leaning protein substructures from data was already the best performing approach in the protein folding problem.

xbmcuser 6y ago
I think Netflix model of simplifying then translating should work better with internet forums and blog posts. Looking at how google works I was hoping someone in google wold adopt it and release a competing product against google translate https://arxiv.org/abs/2005.11197 https://arxiv.org/abs/2005.11197
Polygator 6y ago
Not really. Using a statistical approach to text modelling, specifically using Markov Chains, was proposed by Shannon in 1948. But yeah, there's a point in the 2000s where generative grammar/ symbolic approaches were pretty much left behind by NN methods. When we discuss Google's input in NLP, the most important contribution is certainly the "Attention is all you Need" paper, which paved the way for BERT and GTP (Alphafold also uses Attention networks, btw)
- adw 6y ago
  > generative grammar/ symbolic approaches were pretty much left behind by NN methods Which is the same thing as hand-engineered feature stacks being left behind in vision problems, really. The story in every field is more or less "you're not clever enough to engineer good features"; "you might be clever enough to define good symmetries for the feature space in which the features live... maybe" (convolutional neural networks in image problems); "... but maybe not even that" (attention mechanisms).
  - snovv_crash 6y ago
    The hand generated features are still superior for SfM style problems, where the geometry is well defined but would need to be learned by the NN from scratch.
std_badalloc 6y ago
I think it was with the seq2seq paper of Sutskever, Vinyals and Le in 2014: https://arxiv.org/abs/1409.3215 https://arxiv.org/abs/1409.3215 People were doing a mix of learning from data and hand engineered solutions before this, but this was the first system learned end-to-end, afaik.

by sampo 6y ago

xbmcuser 6y ago

I think Netflix model of simplifying then translating should work better with internet forums and blog posts. Looking at how google works I was hoping someone in google wold adopt it and release a competing product against google translate https://arxiv.org/abs/2005.11197 https://arxiv.org/abs/2005.11197

Polygator 6y ago

Not really. Using a statistical approach to text modelling, specifically using Markov Chains, was proposed by Shannon in 1948. But yeah, there's a point in the 2000s where generative grammar/ symbolic approaches were pretty much left behind by NN methods. When we discuss Google's input in NLP, the most important contribution is certainly the "Attention is all you Need" paper, which paved the way for BERT and GTP (Alphafold also uses Attention networks, btw)

adw 6y ago
> generative grammar/ symbolic approaches were pretty much left behind by NN methods Which is the same thing as hand-engineered feature stacks being left behind in vision problems, really. The story in every field is more or less "you're not clever enough to engineer good features"; "you might be clever enough to define good symmetries for the feature space in which the features live... maybe" (convolutional neural networks in image problems); "... but maybe not even that" (attention mechanisms).
- snovv_crash 6y ago
  The hand generated features are still superior for SfM style problems, where the geometry is well defined but would need to be learned by the NN from scratch.

std_badalloc 6y ago

I think it was with the seq2seq paper of Sutskever, Vinyals and Le in 2014: https://arxiv.org/abs/1409.3215 https://arxiv.org/abs/1409.3215 People were doing a mix of learning from data and hand engineered solutions before this, but this was the first system learned end-to-end, afaik.