3 ms·
>> We knew hand-crafted program in limited domains can work for NLP, computer vision and voice recognition long time ago. Yes, we did. So- where are the natura
by YeGoblynQueenne 6y ago
>> We knew hand-crafted program in limited domains can work for NLP, computer vision and voice recognition long time ago.
Yes, we did. So- where are the natural language interfaces by which we can communicate with artificial agents in such limited domains? Where are the applications, today, that exhibit behaviour as seemingly intelligent as SHRDLU in the '60s? I mean, have you personally seen and interacted with one? Can you show me an example of such a modern system? Edit: Note again that SHRDLU was created by a single PhD student with all the resources of ... a single PhD student. It's no stretch to imagine that an entity of the size of Google or Facebook could achieve something considerably more useful, still in a limited domain. But this has never even been attempted.
Yes, it is faffing about. Basically NLP gave up on figuring out how language works and switched to a massive attempt to model large datasets evaluated by contrived benchmarks that serve no other purpose than to show how well modern techniques can model large datasets.
- rvense 6y agoMy impression is that the systems never progressed much after SHRDLU even though there were attempts at larger scale "expert systems". But adding more advanced rules and patterns proved extremely difficult and did not always have the expected effect of making the systems more general. There was the whole AI winter thing, of course, but that was as much a result of things not living up to the hype as a cause.
- YeGoblynQueenne 6y agoThis doesn't directly address your question, though it perhaps can give you some pointers if you want to read about the history of AI and the AI winter of the '80s, but in a way SHRDLU featured prominently in the AI winter, at least in Europe, particularly in the UK. So, in the UK at least the AI winter was precipitated by the Lighthill Report, a report published in 1973, compiled by a Sir James Lighthill and commissioned by the British Research Council, i.e. the people who held all the research money at the time in the UK. The report was furiously damning of AI research of the time, mostly because of grave misunderstandings e.g. with respect to combinatorial explosion and basically accused researchers of, well, faffing about and not doing anything useful with their grant money. The only exception to this was SHRDLU, that Lighthill praised as an example of how AI should be done. Anyway, if you have time, you can watch the televised debate between Lighthill and three luminaries of AI, John McCarthy (the man who named the field, created Lisp and did a few other notable things), Donald Michie (known for his MENACE reinforcement-learning program running on... matchboxes, and basically setting up AI research in the UK) and Richard Gregorie (a cognitive scientist from the US for whom I confess I don't know much). The (short) wikipedia article on the Lighthill Report has links to all the youtube videos: https://en.wikipedia.org/wiki/Lighthill_report https://en.wikipedia.org/wiki/Lighthill_report It's interesting to see in the videos the demonstration of the Freddy robot from Edinburgh, that was capable of constructing objects by detecting their components with early machine vision techniques. In the 1960's. Incidentally: Even with today's knowledge, methodology, software tools, and so on, getting a robot to do this kind of thing would be a fairly complex and ambitious project. http://www.aiai.ed.ac.uk/project/freddy/ http://www.aiai.ed.ac.uk/project/freddy/ The above was written sometime in the '90s, I reckon but it is still true today. Unfortunately, Lighthill's report killed the budding robotics research sector in the UK and it has literally never recovered since. This is typical of the AI winter of the '80s. Promising avenues of research were abandoned not because of any scientific reasons, as is sometimes assumed ("expert systems didn't scale" etc) but, rather, because pencil pushers in charge of disbursing public money didn't get the science. Edit: A couple more pointers. John McCarthy's review of the Lighthill Report: http://www-formal.stanford.edu/jmc/reviews/lighthill/lighthill.html http://www-formal.stanford.edu/jmc/reviews/lighthill/lighthi... An article on the AI winter of the '80s by the editor of IEEE Intelligent Systems: https://www.computer.org/csdl/magazine/ex/2008/02/mex2008020002/13rRUyeCkdP https://www.computer.org/csdl/magazine/ex/2008/02/mex2008020...
- rvense 6y agoInteresting, thank you for the clarifications.
- lacker 6y agoThe modern natural language interfaces with limited domains are Alexa and Siri. Yes, they’re limited. But they are far more impressive and useful than SHRDLU.
- YeGoblynQueenne 6y agoAlexa and Siri (and friends) are competely incapable of interacting with a user with the precision of SHRDLU. You can ask them to retrieve information from a Google search but e.g. they have no memory of the anaphora in earlier sentences in the same conversation. If you say "it" a few times to refer to different entities they completely lose the plot. They are also completely incapable of reasoning about their environment, not least because they don't have any concept of an "environment" - which was represented by the planner and the PROGRAMMAR language in SHRDLU. And of course, systems like Siri and Alexa can't do anything even remotely like correctly disambiguating the "support support supports" show-off sentence in the excerpt above. Not even close. Edit: Sorry, there's a misunderstanding about "limited domain" in your comment. Alexa and Siri don't operate in a limited domain. A "limited domain" would be something like being in charge of your music collection and nothing else. Alexa annd Siri etc are supposed to be general-use agents. I mean, they are, it's just that they suck at it... and would still suck in a limited domain also.
- randcraw 6y agoIt’s not meaningful to compare SHRDLU with today’s verbal search interfaces. The world SHRDLU manipulated were only stackable blocks and the only relations it knew were ‘above’ and ‘below’. The entire scope of its endeavor was describing the state of block stacks and basic ways to reorder stacks to satisfy the single relation of above-ness and below-ness. Time and causality and even basic probability were all absent from SHRDLU’s model. Not surprisingly the work was a dead end that even Winograd was quick to abandon, as he subsequently exited the field of experimental AI for the more conceptual models of cognitive science and HCI.
- YeGoblynQueenne 6y ago>> The world SHRDLU manipulated were only stackable blocks and the only relations it knew were ‘above’ and ‘below’. If that were true then SHRDLU would have operated in a one-dimensional world. Instead, it moved blocks around a 3-dimensional world. It could understand more spatial relations than "above" and "below", as in the example I quote above where it is asked "Is there a large block behind a pyramid?". It stacked blocks, but in doing so it also had to move others out of the way etc. That is no big mystery, like I say SHRDLU used a planner and even in the 1960s' there were planners capabe of solving block stacking problems in 3d environments. Time and causality were certainly not "absent" from SHRDLU! Like I say, it had a memory that allowed it to keep track of what it had done previously and it could explain the reasoning behind its actions, like in this excerpt from the demonstration quoted in wikipedia: Person: Had you touched any pyramid before you put the green one on the little cube? Computer: YES, THE GREEN ONE. Person: When did you pick it up? Computer: WHILE I WAS STACKING UP THE RED CUBE, A LARGE RED BLOCK AND A LARGE GREEN CUBE. Person: Why? Computer: TO GET RID OF IT. Person: Why did you do that? Computer: TO CLEAN OFF THE RED CUBE. Person: Why did you clear off that cube? Computer: TO PUT IT ON A LARGE GREEN CUBE. Person: Why did you do that? Computer: TO STACK UP THE RED CUBE , A LARGE RED BLOCK AND A LARGE GREEN CUBE. Person: Why did you do that? Computer: BECAUSE YOU ASKED ME TO. Person: How did you do it? Computer: BY PUTTING A LARGE RED BLOCK ON THE TABLE ; THEN LETTING GO OF IT ; THEN PUTTING A LARGE GREEN CUBE ON IT ; THEN LETTING GO OF THAT CUBE ; THEN PUTTING THE RED CUBE ON THAT CUBE; THEN LETTING GO OF THAT CUBE. This is a question about time and causality: the user asks why the program performed a set of operations in an earlier time. >> Not surprisingly the work was a dead end that even Winograd was quick to abandon, as he subsequently exited the field of experimental AI for the more conceptual models of cognitive science and HCI. Regarding Winograd's subsequent work direction, this is what he had to say about it himself: How would you say SHRDLU influenced your subsequent work and/or philosophy in AI? Having insight into the limitations I encountered in trying to extend SHRDLU beyond micro-worlds was the key opening to the philosophical views that I developed in the work with Flores. The closest thing I have online is the paper Thinking machines: Can there be? Are we? How would you characterize AI since SHRDLU? Why do you think no one took SHRDLU or SHRDLU-like applications to the next level? There are fundamental gulfs between the way that SHRDLU and its kin operate, and whatever it is that goes on in our brains. I don't think that current research has made much progress in crossing that gulf, and the relevant science may take decades or more to get to the point where the initial ambitions become realistic. In the meantime AI took on much more doable goals of working in less ambitious niches, or accepting less-than-human results (as in translation). What future do you see for natural language computing and/or general AI? Continued progress in limited domain and approximate approaches (including with speech). Very long term research is needed to get a handle on human-level natural language. http://maf.directory/misc/shrdlu.html http://maf.directory/misc/shrdlu.html My reading of this is he realised that natural language understanding is not an easy thing. I don't disagree one bit and I don't think for a moment that SHRDLU could "understand" anything at all. But it was certainly capable of much more intelligent-looking behaviour than modern statistical machine-learning based systems. Winograd's reply above says that it's hard to extend SHRDLU outside of its limited domain, but my point is that a program that can operate this well in a limited domain is still useful and much more useful than a program that can operate in arbitrary domains but is dumb as bricks, like modern conversational agents that have nothing like a context of their world that they can refer to, to choose appropriate contributions to a conversation. He also hints to the shift of AI research targets from figuring out how natural language works to "less ambitious niches" and "less than human results", which I also point out in my comments in this thread. This is condensed - I'm happy to elaborate if you wish. I have to say I was very surprised by your comment, particularly the certainty with which you asserted SHRDLU's limitations. May I ask- where did you come across information that SHRDLU could only understand "above and below" relations and that time and causality were absent from its "model"?
- joe_the_user 6y agoGood question! As you say SHRDLU was the best program of that time. I've read Winnograd's book and looked at the SHRDLU source code (not that I'm much of a lisp hacker but I got some time). It's built-on a parser and planner (logic program, pre-prolog). And it's built the old-fashioned-way, rewriting the source code with the parser rewriting input and then re-rerunning and other harry things. I think this achieves the parsing of idiomatic constructions on a high level. I believe the "raw lisp" of the day was both incredibly powerful since you could do anything and incredibly hard to scale ... you could do anything. Winnograd wrote it himself but I think that's because he had to write it himself. In a sense, a programmer is always most productive when they are writing something by themselves because they don't have to explain anything they are doing (until the fix-ups and complexity overwhelm them but the base fact remains). And in the case of SHRDLU, Winnograd would have had an especially hard time explaining what he was doing. I mean, there was a theory behind - I've read Winnograd's book. But there was lots and lots of brilliant spaghetti/glue-code to actually make it work, code that jumped module and function boundaries. And the final had a reputation of being very buggy, sometimes it worked and sometimes it didn't. And Winnograd was a brilliant programmer and widely read in linguistics and other fields. The software industry is an industry. No company wants to depend on the brilliance of it's workers. A company needs to produce based on some sort of average and a person working with just average skills isn't going to do SHRDLU. So, yeah, I think that's why actual commercial programs never reached the level of SHRDLU
- _emacsomancer_ 6y agoSo what you're saying is that we really shouldn't be relying on industry for good AI?
- joe_the_user 6y agoWell, an "industrial model" is a model of a factory and it seems unlikely you could product a fully-functional, from scratch GFAI program like SHRDLU in something like a factory. Perhaps one could create a different kind of enterprise for this but it's kind of an open problem.
- YeGoblynQueenne 6y ago
- zodiac 6y ago> It's no stretch to imagine that an entity of the size of Google or Facebook could achieve something considerably more useful, still in a limited domain. But this has never even been attempted. How do you know if no one has attempted it, or if all attempts so far have failed to achieve their goals? One of the claimed downsides of "hand-crafted systems in limited domains" (i.e., something like the SHRDLU approach) is that they would take too much effort to create when the domain is expanded to something even slightly bigger than SHRDLU's domain, so a lack of successful systems could be evidence of no one trying, or it could be evidence that the claimed downside is indeed true. The fact that a working system for the limited domains of e.g. customer support or medical diagnosis would be worth a lot of money suggests to me that they must have been tried, but that nothing useful could be built, and we didn't hear about the failed attempts, meaning that those domains (at least) are too big for hand-crafted systems to work. > Yes, it is faffing about. Basically NLP gave up on figuring out how language works and switched to a massive attempt to model large datasets evaluated by contrived benchmarks that serve no other purpose than to show how well modern techniques can model large datasets. It is inaccurate to say that all benchmarks are useless. For instance, language translation (as in Google Translate) is a benchmark NLP task, but it is also something I personally use at least every week, and deep learning based solutions beat handcrafted systems by a lot for this particular task (speaking as an end user who has used systems based on both approaches). The same comments apply to audio transcription (e.g. generating subtitles on YouTube) as well.
- deleted 6y ago[deleted]
- YeGoblynQueenne 6y ago>> The fact that a working system for the limited domains of e.g. customer support or medical diagnosis would be worth a lot of money suggests to me that they must have been tried, but that nothing useful could be built, and we didn't hear about the failed attempts, meaning that those domains (at least) are too big for hand-crafted systems to work. That's actually a good point and one I've made myself a few times in the past- negative results never see the light of day. So, yes, I agree we can't know for sure that such systems haven't been attempted, despite the certainty with which I assert this in my comment above. On the other hand, there are some strong hints. First of all, there was an AI winter in the 1980's after which most of the field turned very hard towards statistical techniques and away from the kind of work in SHRDLU. This kind of work became radioactive for many years and it would have been very hard to justify putting a PhD student, or ten, to work trying to even reproduce it, let alone extend it. That's in academia. In the industry, it's clear that nowadays at least companies like the FANGs strongly champion statistical machine learning and anyone proposing spending actual money on such a research program ("Hey, let's go back to the 1960's and start all over again!") would be laughed out of the building. That is, I believe there are strong hints that the political climate in academia and the culture in large companies, has suppressed any attempt to do work of this kind. But that's only my conjecture, so there you have it. >> It is inaccurate to say that all benchmarks are useless. Of course. My point is that current benchmarks are useless. The fact that you find Google translate useful is unrelated to how well Google translate scores in benchmarks, which are not designed to measure user satisfaction but instead are supposed to tell us something about the formal properties of the system. In any case, for translation in particular, it's not controversial that there are no good benchmarks and metrics and many people in NLP will tell you that is the case. In fact, I'm saying this myself because I was told during my Master's, by our NLP tutor who is a researcher in the field. Also see the following article which indcludes a discussion of commonly used metrics in machine translation and the difficulty of evaluating machine translation systems: https://www.skynettoday.com/editorials/state_of_nmt https://www.skynettoday.com/editorials/state_of_nmt