3 ms·
What exactly do you expect from these Transformer-based models? This particular one is underwhelming because it provides a tiny improvement over the previous la
by p1esk 7y ago
What exactly do you expect from these Transformer-based models? This particular one is underwhelming because it provides a tiny improvement over the previous largest one (from Nvidia) with more than double the size.
The fundamental limitation of these "optimization tools" as you call them - they don't have any common sense, and any way to query an external source of information (e.g. wikipedia), or ask a human to clarify.
Another big problem is we don't have any way to do quality filtering on the outputs. From my experiments with GPT-2, it produces one interesting paragraph of text out of 20 - if you squint at it really hard. And most of those 20 don't make much sense at all.
So no, the existing ideas are definitely not enough. Maybe some novel hybrid of symbolic AI with statistical optimization will lead to a breakthrough. This one does not strike me as anything other than "let's use moar weights!!"
- sanxiyn 7y ago> they don't have any common sense What do you mean by this? Of course they do, learned from their training data. For example, here is quote from conversation 38 of https://github.com/google-research/google-research/blob/master/meena/meena.txt https://github.com/google-research/google-research/blob/mast... Human: Do you like Korean food in general? Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi. It seems to me Meena "knows" bulgogi and kimchi are Korean foods. Isn't that common sense? If it isn't, what do you mean by "common sense"?
- p1esk 7y agoTry asking it a follow up question not commonly found in the training data: such as "do you think bulgogi would grow on Mars?", and see what kind of gibberish you will get in response. Moreover, the model has no way of self-diagnosing whenever it produces gibberish.
- amylene 7y agoWhat is the baseline comparison here? If you asked a random 100 people on the street this question?
- sgt101 7y agoI'd buy that if Meena could infer and reason about her own answers. Human: Do you like Korean food in general? Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi. Human: Ok what should I shop for ? Meena : You've got almost everything but you need a pear, the steak and some ginger. The problem with language models as commonsense is that they are collections of patterns and associations, and that they don't have inference models or solvers - unlike my dog for example!
- ctoth 7y agoSome papers which you might find interesting: Lin, B. Y., Chen, X., Chen, J., & Ren, X. (2019). KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning. 2822–2832. https://doi.org/10.18653/v1/d19-1282 https://doi.org/10.18653/v1/d19-1282 Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., & Wang, P. (2019). K-BERT: Enabling Language Representation with Knowledge Graph. Retrieved from http://arxiv.org/abs/1909.07606 http://arxiv.org/abs/1909.07606 Trinh, T. H., & Le, Q. V. (2019). Do Language Models Have Common Sense? Iclr, 1–12. Ostendorff, M., Bourgonje, P., Berger, M., Moreno-Schneider, J., Rehm, G., & Gipp, B. (2019). Enriching BERT with Knowledge Graph Embeddings for Document Classification. Retrieved from http://arxiv.org/abs/1909.08402 http://arxiv.org/abs/1909.08402