4 ms·

I would agree with you to a certain extent, but I still think there is a big missing component to make this a reality. Larger and more accurate general language

I would agree with you to a certain extent, but I still think there is a big missing component to make this a reality. Larger and more accurate general language models are great, but to enable use cases other than categorization, translation, summarization, etc., there will almost certainly have to a be contextualized knowledge graph layer. This is basically what I assume you mean when you say "the ability to create, synthesize, and understand content." The way I see it, transformer-based general language models will be the first and last step of a NLP system. I other words, they will do the raw processing of the input and will do the "take this output and put it in natural language based on this context" portion. Automating the part in the middle, the part that actually understands what the text means and can do logic with it, there's still no progress on that AFAIK. This is similar to how computer vision models work great in most cases, but to build a self-driving car you still need all the other components that do path planning, predicting what the cars around you will do based on the state of the environment, etc.

codingslave 7y ago
Replying from my laptop. I agree, there needs to be a way to represent relationships between information. I personally don't think knowledge graphs will be the ones to do it, not because they dont work, but because of how imperfect they are, in the data quality sense. See this paper here: "DIFFERENTIABLE REASONING OVER A VIRTUAL KNOWLEDGE BASE" https://openreview.net/pdf?id=SJxstlHFPH https://openreview.net/pdf?id=SJxstlHFPH Which is a recent effort, among many, by google research to build a model that can view a document as a knowledge graph, instead of explicitly tying pieces of the document to the graph, the idea to is to create a graph from the document. This is paper is a bit different from that, they do input a knowledge graph for training, but I think the idea and track of where they are headed has a ton of room to evolve. The trick is that transformer models have unlocked the ability to understand the text, so all of this "quasi knowledge graph extraction" that i was just explaining, is only recently possible! There's no research on it, because the baseline understanding of tokens has been too primitive. This is why there is so much room to grow, BERT has unlocked new methods, it can be used as a base for a ton of new NLP. Just to emphasize again, I'm not saying what I outlined above will be a good way to do it, just that ideas like this could only be tested recently. There's a million new ways to spin this problem.
- cfoster0 7y ago
  Related: a recent paper from Antoine Bosselut and Yejin Choi explores dynamically constructing a context-specific, common sense knowledge graph using a transformer, in the context of question answering. https://arxiv.org/abs/1911.03876 https://arxiv.org/abs/1911.03876

codingslave 7y ago

Replying from my laptop. I agree, there needs to be a way to represent relationships between information. I personally don't think knowledge graphs will be the ones to do it, not because they dont work, but because of how imperfect they are, in the data quality sense. See this paper here: "DIFFERENTIABLE REASONING OVER A VIRTUAL KNOWLEDGE BASE" https://openreview.net/pdf?id=SJxstlHFPH https://openreview.net/pdf?id=SJxstlHFPH Which is a recent effort, among many, by google research to build a model that can view a document as a knowledge graph, instead of explicitly tying pieces of the document to the graph, the idea to is to create a graph from the document. This is paper is a bit different from that, they do input a knowledge graph for training, but I think the idea and track of where they are headed has a ton of room to evolve. The trick is that transformer models have unlocked the ability to understand the text, so all of this "quasi knowledge graph extraction" that i was just explaining, is only recently possible! There's no research on it, because the baseline understanding of tokens has been too primitive. This is why there is so much room to grow, BERT has unlocked new methods, it can be used as a base for a ton of new NLP. Just to emphasize again, I'm not saying what I outlined above will be a good way to do it, just that ideas like this could only be tested recently. There's a million new ways to spin this problem.

cfoster0 7y ago
Related: a recent paper from Antoine Bosselut and Yejin Choi explores dynamically constructing a context-specific, common sense knowledge graph using a transformer, in the context of question answering. https://arxiv.org/abs/1911.03876 https://arxiv.org/abs/1911.03876