7 ms·

> Accumulated errors, such as the deduction that Internet cookies were a kind of baked good, led NELL to deduce from the phrases "I deleted my Internet cookies"

by 3rd3 13y ago

> Accumulated errors, such as the deduction that Internet cookies were a kind of baked good, led NELL to deduce from the phrases "I deleted my Internet cookies" and "I deleted my files" that "computer files" also belonged in the baked goods category. That is hilarious! By the way, there is another intersting never ending lerner which is based on random images from the internet: http://www.neil-kb.com/ http://www.neil-kb.com/

blueblob 13y ago
Thanks both of you for these, very interesting. I am looking at this problem on my own at the moment and didn't find either on my own.
adwf 13y ago
Yes 87% is quite impressive, but that remaining 13% is all important. The errors sadly propagate (as above); more importantly, the errors tend to strike randomly also. eg. A human might not be expected to know too much medical terminology, hence could easily get names/locations/functions of muscles and organs mixed up. Whereas a computer might be brilliant at that, but then suddenly not know what a door knob is (door handle? door knob of butter? etc...) So ultimately to make a computer seem intelligent the problem is two-fold: Getting a high percentage of right answers, whilst also getting the "right" wrong answers. Afterall, we don't criticize a human too much for not knowing something like the exact geographic location of a small African nation. But we definitely would if they thought that it was on Mars! It's exactly the same with computers, they're just far more likely to make wildly incorrect mistakes than understandably incorrect ones. So finding a way to mitigate computer mistakes, to make them milder, is every bit as important as eliminating them entirely - a potentially impossible task.
- evunveot 13y ago
  So finding a way to mitigate computer mistakes, to make them milder, is every bit as important as eliminating them entirely - a potentially impossible task. Seems like they should run multiple instances against different but overlapping corpuses and see what they agree on. For all we know, any single person would forget what a door knob is if they had all the information in the world in their brain. But maybe it already works that way internally, I don't know.
- blueblob 13y ago
  If you think about the way that humans learn, we get formal education that the computer doesn't get. We are getting corrected for our mistakes by other people. This work did not originally do that. The newer versions of it get corrected by people but that doesn't mean that it couldn't be corrected by more instances of itself that operate over different datasets via a consensus algorithm. The algorithm will also probably get flack that it is not unsupervised when human learning is actually partially supervised. One could argue that humans have a similar problem of errors propagating. Areas that we feel strongly about can bias us against learning in fields such as religion, politics, and, of course, programming language design. I think that it is possible that humans would make some similarly poor responses in areas that we don't talk about often.
  - cortesoft 13y ago
    I agree. In fact, errors like thinking an internet cookie is a baked good sounds EXACTLY like the type or mistake a small child would make.
    - 3rd3 13y ago
      That makes me remember that as a small kid there were certain spots in my environment beyond which, I was conviced, infinite wilderness would adjoin or the world would just end (for example behind forests or hedges). Growing older I was often sobered when I discrovered there was just bog-standard urban landscape. Yet, the machine seems to come from an entirely differernt direction. It has much more facts accumulated than a child and can articulate and process them to absolute precision.
      - Gracana 13y ago
        I remember visiting my childhood elementary school and seeing that the "forest" behind the building was just a scraggly patch of trees with a chain-link fence on the other side. It seemed so much bigger back then...
  - adwf 13y ago
    I agree absolutely. I believe that future AI's will be educated in much the same way we educate children. Perhaps less education will be needed - possibly we could start them off at a higher age bracket for example. But ultimately I'm certain the first "strong" AI's will be educated/supervised to some degree. My point was more that perhaps we're a bit too focused on the wrong metric for success. The current criteria set is {positives, negatives, false positives, false negatives}, and we try to optimise for high/low degrees of one or the other in order to determine whether a particular approach is successful or not. What is then overlooked, is that perhaps we don't need to have a near-perfect positive rate, but instead achieve an acceptably-incorrect false positive or false negative rate. Where the answer may be wrong, but it's not too far wrong. Much like a human might pin a country like India in the wrong place on the map, but wouldn't ever put it in the middle of the Indian ocean. In summation: Perhaps the key for computers to appear intelligent, is not to be perfectly correct, but to be not too disastrously incorrect.
- varelse 13y ago
  13% error rate in a self-driving car, hilarity ensues...
  - Raphael 13y ago
    Leave room for a margin of error, same as humans do.
  - adwf 13y ago
    Exactly. A human would usually make "acceptably incorrect" decisions, such as not indicating, or speeding. But these mistakes generally don't end in disaster. Only occasionally making a really bad mistake ending in an accident. So out of those 13%, it might only end up with <0.1% overall as accidents, the rest being recoverable. Whereas the uncontrolled randomness of computer mistakes could be horrendous. Hence why I'm saying an interesting avenue to look at in AI, might be to achieve acceptably incorrect answers (acceptable negatives), rather than just aiming for high true-positive and low false-positive rates.
- Houshalter 13y ago
  Well reading through their site it seems it gets some categories of facts right nearly all the time, and some categories it does poorly at. This should be expected because some things are hard to understand even for a human, especially if you don't already know a lot about the subject. However it does seem to assign a confidence estimate to every fact it learns. So it does get some facts wrong, but at least it knows what probability there is the fact is wrong and doesn't assume it is 100% true.
- waveman2 13y ago
  > they're just far more likely to make wildly incorrect mistakes than understandably incorrect ones Have you spoken to some members of the general public and asked what they think? People believe all sorts of wacky things. We are just a lot more tolerant of the mistakes people make. http://www.buzzfeed.com/mjs538/things-americans-believe-in http://www.buzzfeed.com/mjs538/things-americans-believe-in

by 3rd3 13y ago

blueblob 13y ago

Thanks both of you for these, very interesting. I am looking at this problem on my own at the moment and didn't find either on my own.

adwf 13y ago

Yes 87% is quite impressive, but that remaining 13% is all important. The errors sadly propagate (as above); more importantly, the errors tend to strike randomly also. eg. A human might not be expected to know too much medical terminology, hence could easily get names/locations/functions of muscles and organs mixed up. Whereas a computer might be brilliant at that, but then suddenly not know what a door knob is (door handle? door knob of butter? etc...) So ultimately to make a computer seem intelligent the problem is two-fold: Getting a high percentage of right answers, whilst also getting the "right" wrong answers. Afterall, we don't criticize a human too much for not knowing something like the exact geographic location of a small African nation. But we definitely would if they thought that it was on Mars! It's exactly the same with computers, they're just far more likely to make wildly incorrect mistakes than understandably incorrect ones. So finding a way to mitigate computer mistakes, to make them milder, is every bit as important as eliminating them entirely - a potentially impossible task.

evunveot 13y ago
So finding a way to mitigate computer mistakes, to make them milder, is every bit as important as eliminating them entirely - a potentially impossible task. Seems like they should run multiple instances against different but overlapping corpuses and see what they agree on. For all we know, any single person would forget what a door knob is if they had all the information in the world in their brain. But maybe it already works that way internally, I don't know.
blueblob 13y ago
If you think about the way that humans learn, we get formal education that the computer doesn't get. We are getting corrected for our mistakes by other people. This work did not originally do that. The newer versions of it get corrected by people but that doesn't mean that it couldn't be corrected by more instances of itself that operate over different datasets via a consensus algorithm. The algorithm will also probably get flack that it is not unsupervised when human learning is actually partially supervised. One could argue that humans have a similar problem of errors propagating. Areas that we feel strongly about can bias us against learning in fields such as religion, politics, and, of course, programming language design. I think that it is possible that humans would make some similarly poor responses in areas that we don't talk about often.
- cortesoft 13y ago
  I agree. In fact, errors like thinking an internet cookie is a baked good sounds EXACTLY like the type or mistake a small child would make.
  - 3rd3 13y ago
    That makes me remember that as a small kid there were certain spots in my environment beyond which, I was conviced, infinite wilderness would adjoin or the world would just end (for example behind forests or hedges). Growing older I was often sobered when I discrovered there was just bog-standard urban landscape. Yet, the machine seems to come from an entirely differernt direction. It has much more facts accumulated than a child and can articulate and process them to absolute precision.
    - Gracana 13y ago
      I remember visiting my childhood elementary school and seeing that the "forest" behind the building was just a scraggly patch of trees with a chain-link fence on the other side. It seemed so much bigger back then...
- adwf 13y ago
  I agree absolutely. I believe that future AI's will be educated in much the same way we educate children. Perhaps less education will be needed - possibly we could start them off at a higher age bracket for example. But ultimately I'm certain the first "strong" AI's will be educated/supervised to some degree. My point was more that perhaps we're a bit too focused on the wrong metric for success. The current criteria set is {positives, negatives, false positives, false negatives}, and we try to optimise for high/low degrees of one or the other in order to determine whether a particular approach is successful or not. What is then overlooked, is that perhaps we don't need to have a near-perfect positive rate, but instead achieve an acceptably-incorrect false positive or false negative rate. Where the answer may be wrong, but it's not too far wrong. Much like a human might pin a country like India in the wrong place on the map, but wouldn't ever put it in the middle of the Indian ocean. In summation: Perhaps the key for computers to appear intelligent, is not to be perfectly correct, but to be not too disastrously incorrect.
varelse 13y ago
13% error rate in a self-driving car, hilarity ensues...
- Raphael 13y ago
  Leave room for a margin of error, same as humans do.
- adwf 13y ago
  Exactly. A human would usually make "acceptably incorrect" decisions, such as not indicating, or speeding. But these mistakes generally don't end in disaster. Only occasionally making a really bad mistake ending in an accident. So out of those 13%, it might only end up with <0.1% overall as accidents, the rest being recoverable. Whereas the uncontrolled randomness of computer mistakes could be horrendous. Hence why I'm saying an interesting avenue to look at in AI, might be to achieve acceptably incorrect answers (acceptable negatives), rather than just aiming for high true-positive and low false-positive rates.
Houshalter 13y ago
Well reading through their site it seems it gets some categories of facts right nearly all the time, and some categories it does poorly at. This should be expected because some things are hard to understand even for a human, especially if you don't already know a lot about the subject. However it does seem to assign a confidence estimate to every fact it learns. So it does get some facts wrong, but at least it knows what probability there is the fact is wrong and doesn't assume it is 100% true.
waveman2 13y ago
> they're just far more likely to make wildly incorrect mistakes than understandably incorrect ones Have you spoken to some members of the general public and asked what they think? People believe all sorts of wacky things. We are just a lot more tolerant of the mistakes people make. http://www.buzzfeed.com/mjs538/things-americans-believe-in http://www.buzzfeed.com/mjs538/things-americans-believe-in