5 ms·
> Also were there any new real SOTA on any NLP tasks since last summer? I feel like accuracy progress has frozen.. Just in the last week there are two papers w
by huac 7y ago
> Also were there any new real SOTA on any NLP tasks since last summer? I feel like accuracy progress has frozen..
Just in the last week there are two papers which claim SOTA on different tasks.
Microsoft released Turing NLG (https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/ https://www.microsoft.com/en-us/research/blog/turing-nlg-a-1...) recently which claims SOTA on a couple tasks, it seems like the same transformer architecture, but with more layers and parameters, made feasible by training efficiency improvements. The biggest one seems to be partitioning how the model learns across different processes instead of replicating those states, which significantly improves communication, memory overhead, and training speed.
Deepmind released the Compressive Transformer (https://deepmind.com/blog/article/A_new_model_and_dataset_for_long-range_memory https://deepmind.com/blog/article/A_new_model_and_dataset_fo...) which claims SOTA on two other "long-range" benchmarks. My understanding of the improvement here is that instead of discarding older states, in the traditional attention layer, the compressive transformer learns which states to keep, and which states to remove.
I think these are two good examples of paper archetypes -- one where SOTA is achieved through more layers/training data/neurons (and the more interesting contribution is the improvements to model parallelism in training), and one where SOTA is achieved through a new/improved model architecture.
I wonder, for most industrial practitioners, how much either paper is useful though. The Microsoft paper helps for training billion parameter models, but most won't train a model that deep; the Deepmind paper helps for training models over very long sequences, but most people aren't using book-length sequences.*
* I remember reading somewhere that the attention mechanisms tend to only remember around 5 states (would love to see a source or study on this), which is pretty short, so would be interesting to try this model and see if the compressed transformer/attention mechanism can remember longer sequences.
- nl 7y agoNeither of these are Graph NNs though (although it was a weird question because Graph NNs solve a different problem)
- huac 7y agoyes, true. there has been progress on NLP SOTA, just not from graph NN architectures. I have thought about using graph NN's where I am currently using RNN's, but they seem fairly immature / difficult to train over very large datasets so have not made much progress. I was following the work of https://www.octavian.ai/ https://www.octavian.ai/, but have not seen much recently. http://web.stanford.edu/class/cs224w/info.html http://web.stanford.edu/class/cs224w/info.html also looks interesting. looking briefly over the student projects, there appear to be a couple nlp projects (e.g. http://web.stanford.edu/class/cs224w/project/26418192.pdf http://web.stanford.edu/class/cs224w/project/26418192.pdf) but most leverage knowledge graph concepts.
- nl 7y agoOn the contrary, Graph NN's I've found Graph NNs extraordinarily useful and very easy to train! I'm not sure what kind of problems you are trying to use them for, but generally they are really useful for node classification or recommendation type tasks. So for example in the knowledge graph context, you can do things like give it a tiger, jaguar and panther and it will find nodes like lions and leopards. https://arxiv.org/pdf/1901.00596.pdf https://arxiv.org/pdf/1901.00596.pdf is a survey paper which has a decent overview of the tasks they are useful for. http://research.baidu.com/Public/uploads/5c1c9a58317b3.pdf http://research.baidu.com/Public/uploads/5c1c9a58317b3.pdf is using them for QA, but I'm not familiar enough with the dataset to evaluate it fairly.
- huac 7y agoI spent some time experimenting with https://github.com/facebookresearch/PyTorch-BigGraph https://github.com/facebookresearch/PyTorch-BigGraph and a few other libraries, I ran into some challenges given that my dataset was very large (O(100M) edges) and very sparse; and didn't pursue it further. The papers you linked are very interesting, I will have to dig further! One more recent writeup: https://eng.uber.com/uber-eats-graph-learning/ https://eng.uber.com/uber-eats-graph-learning/ -- a real production use case, seems promising to explore more.