4 ms·
Separate comment because separate point. To my mind, all this benchmark-beating is just pointless faffing about and not a little frustrating to see that this i
by YeGoblynQueenne 6y ago
Separate comment because separate point.
To my mind, all this benchmark-beating is just pointless faffing about and not a little frustrating to see that this is where the current state of the art is at. NLP systems keep getting better and better at beating benchmarks and yet we still don't have artificial agents that can hold an intelligent conversation, even in a restricted context.
Or rather, we had such programs since the late 1960's, with no one else's than Terry Winograd's own doctoral thesis system, SHRDLU [1], but it seems that in the race to build powerful classifiers, we forgot how to build systems that can interact intelligently with a human.
Here's a small excerpt from SHRDLU (taken from the wikipedia page on SHRDLU, originally in an appendix in Terry Winograd's thesis):
Person: Is there a large block behind a pyramid?
Computer: YES, THREE OF THEM: A LARGE RED ONE, A LARGE GREEN CUBE, AND THE BLUE ONE.
Person: Put a small one onto the green cube which supports a pyramid.
Computer: OK.
Person: Put the littlest pyramid on top of it.
Computer: OK.
Person: Does the shortest thing the tallest pyramid's support supports support anything green?
Computer: YES, THE GREEN PYRAMID.
Person: What color is it?
Computer: BY "IT", I ASSUME YOU MEAN THE SHORTEST THING THE TALLEST PYRAMID'S SUPPORT SUPPORTS.
Computer: RED
Note that this is not just mindless text generation. The conversation is held in the context of a "blocks world" where a robot with a single hand and eye ("Computer" in the excerpt above) moves blocks of various shapes and colours around, as directed by a human user in free-form natural language. When the Computer says "OK" after it's directed to "put the littlest pyramid on top of it" it's because it really has grabbed the smallest pyramid in the blocks world and placed it on top of the small block in an earlier sentence, as the Person asked. The program has a memory module to keep track of what ellipses like "it", "one" etc refer to throughout the conversation.
SHRDLU was a traditional program hand-crafted by a single PhD student- no machine learning, no statistical techniques. It included, among other things, a context-free grammar (!) of natural English and a planner (to control the robot's hand) all written in Lisp and PLANNER. In its limited domain, it was smarter than anything ever created with statisical NLP methods.
______________________
[1] https://en.wikipedia.org/wiki/SHRDLU https://en.wikipedia.org/wiki/SHRDLU
- quotemstr 6y agoWhy is it surprising that a CFG can approximate a subset of English grammar?
- trott 6y ago> Why is it surprising that a CFG can approximate a subset of English grammar? "Colorless green ideas sleep furiously" is a famous example of a sentence that is grammatical, but meaningless. The goal of SHRDLU was far more ambitious than approximating English grammar.
- liuliu 6y agoWe knew hand-crafted program in limited domains can work for NLP, computer vision and voice recognition long time ago. The challenge is always, that the limited domain can be extremely limited, and to get anything practically interesting requires a lot of human involvement to encode the world (expert system). Statistical methods traded that. With data, some labelled, some unlabelled and some weakly-labelled, we can generate these models with much more efficient human involvement (refine the statistical models and labelling data). I honestly don't see the frustration. Yes, current NLP model may not be the "intelligent agent" everyone looking for yet to any extent. But claiming it is all faffing and no better than 1960s is quite a stretch.
- joe_the_user 6y agoThe thing that qualifies as "faffing" in my opinion isn't the statistical NLP programs, which "are what they are" but claims of progress based primarily on benchmarks, as YeGoblynQueene rightly states. And "limited domain" is relative. A program that gets many aspects of language right talking about a small world might be said to have a larger domain than a program that outputs stream of semi-plausible, semi-gibberish involving the whole of English language. Which again isn't saying modern NLP is nothing but rather we should a somewhat better to talk of it (and machine learning generally) than "hitting benchmarks".
- YeGoblynQueenne 6y ago>> We knew hand-crafted program in limited domains can work for NLP, computer vision and voice recognition long time ago. Yes, we did. So- where are the natural language interfaces by which we can communicate with artificial agents in such limited domains? Where are the applications, today, that exhibit behaviour as seemingly intelligent as SHRDLU in the '60s? I mean, have you personally seen and interacted with one? Can you show me an example of such a modern system? Edit: Note again that SHRDLU was created by a single PhD student with all the resources of ... a single PhD student. It's no stretch to imagine that an entity of the size of Google or Facebook could achieve something considerably more useful, still in a limited domain. But this has never even been attempted. Yes, it is faffing about. Basically NLP gave up on figuring out how language works and switched to a massive attempt to model large datasets evaluated by contrived benchmarks that serve no other purpose than to show how well modern techniques can model large datasets.