6 ms·

Systems smart enough to know when they're not smart enough (2017)

83 points by nromiun 6y ago

blaser-waffle 6y ago
They're vocalized search engines, not knowledge builders. Naturally, they spit out what they see as a the top search, or something repeated often. The "How Barack Hussein Obama is going to declare Martial Law" example is great.
- bryanrasmussen 6y ago
  it amused me that if you posted an article called "How Barack Hussein Obama is going to declare Martial Law" to HN, the title would be rewritten to be "Barack Hussein Obama is going to declare Martial Law" amusement still applies with updated President of course.
code4tee 6y ago
Yes. This is applicable to most automation, machine learning and AI. These technologies are generally really bad at sense checking their results. At the end of the day they match patterns that are mathematically the “most correct” pattern even if it’s obviously not the correct answer. This is the Achilles heel of these technologies that is very hard to overcome and why any real applications generally still have a human in the loop. See Facebook’s “chatbot” experiment where they called it off after realizing the only way to have it work in practice was to have an army of humans behind the scenes sense checking answers. The grand AI engine that takes over for humans is still a pipe dream for most applications. Even all these “neutrality flags” are generally nothing more than a keyword search. Put words like COVID-19 or Coronavirus in your post and Medium puts a banner at the top saying the article hasn’t been fact checked.
nurettin 6y ago
Which search engine will understand sarcasm first? Which one will separate confidence from facts? These are the things that should have been focused on decades ago.
dorusr 6y ago
There is no algorithm of truth, and with enough data you can arrive at any conclusion you like. Maybe people should be better educated, instead of machines becoming smarter.
- tomxor 6y ago
  This is about confidence not truth, confidence is built into ML, it's how it works... the problem is these solutions currently just quantize that output into a single result and obscure how close to the threshold of another possible answer it was.
  - yters 6y ago
    Confidence is a function of truth.
    - deleted 6y ago
      [deleted]
tragomaskhalos 6y ago
Google's inability to spot the negative implicit in "Shirt without stripes" was posted here a couple of weeks ago: https://news.ycombinator.com/item?id=22925087 https://news.ycombinator.com/item?id=22925087
GuiA 6y ago
This is really terrifying in the context of machine translation. I've seen a few forums (I think they were Discourse installs?) configured to automatically translate posts not in your browser language. It's very hard to tell that the post wasn't originally posted in English unless you look very hard for the tiny light-gray-on-white-background text. Machine translation, to put it mildly, is utterly hopeless at context and nuance. I've had Google spit out very racist/sexist results from innocent source text. The fact that it's so confident about what it shows, and that the engineering teams working on these products have very little modesty, means that people treat it as a source of a truth. A future where you have no clue if what you are reading is something written by a human as-is, or went through machine translation and might have nothing to do with what the original author intended, does not put me at ease.
- heavenlyblue 6y ago
  I would say Google’s answers answer about 10% if the questions they say they answer. I have seen so many time Google simply extracting the wrong paragraph of the text saying exactly the opposite of the gist it’s amazing. The worst thing is that there’s no way they can measure the efficiency of these answers directly (I.e. they can only measure how many times it wasn’t clicked as opposed it was). By the time users realised Google had outright lied to them with the answer the team that was shipping these answer widgets is already cashing out from shipping that project.
gitgud 6y ago
Seems similar to the way a programming language might handle errors. A function might handle these error cases, but if not it will bubble/throw it up to the parent for them to deal with it. In other words the function might be thinking; "I'm not smart enough to handle this..." It would probably be a good idea to compose AI bots like this. Different levels of bots, each bubbling up questions they're not smart enough to answer to higher level bots...
- derefr 6y ago
  The more-common opposite approach to AI (with a similar end-result)—often seen in robotics—is called a subsumption architecture: you have naive low-level "mechanism" models making the mundane non-edge-case decisions, with no awareness of the edge-cases; and then you have smarter case-specific "policy" models, trained to recognize when the conditions apply for some particular policy to be enforced, whose job is to change/override the decision the low-level agent makes, by MITMing ("subsuming") the lower-level model. Either they stand in the way of the low-level agent's input, and control it by biasing said input (i.e. lying to it, so it'll make the decision the high-level agent wants); or they stand in the way of the low-level agent's output, and replace it with their own output instead, when necessary. This architecture is particularly convenient when the low-level models have been constructed such that they give wacky results when asked out-of-domain questions. Rather than training the low-level model to respond to out-of-domain input in a sensible way (which might require far more training data, and decrease robustness for in-domain input), you can keep the low-level model unaware of the edges of the domain, and instead just add a high-level "validator" model whose only job is to recognize input that would be invalid for the low-level model, and answer in its stead, replacing the low-level model's noise answer with its own. In an imperative PL sense, this is less like regular exception handling (where the lower-level code knows it doesn't know how to handle something, and so explicitly bubbles up the error), and more like the higher-level code either checking/modifying input in advance of passing it to the lower-level code; or, in fact, telling the low-level code how to handle any errors that crop up. As if the low-level code just had a bald try{} block; and the determination of what that try{} block tries to catch, would be up to the callers below the block on the call stack, with the topmost caller getting the final decision on whether to continue, recover, abort, etc. right at the moment of the initial error, before ever returning its answer to its immediate parent. (A lot like Lisp conditions, actually.)
firefoxd 6y ago
One problem is we only talk about the victories and never educate the end-user who is supposed to use these systems. One great example was Google Duplex. Despite the impressive demo, people had a hard time recreating that success [1]. When a product is promoted as perfect, people start to believe it is perfect. I wrote an article about Duolingo, saying that you need a lot more than that to learn Spanish. Google displays a snippet that says it is pointless and points to my blog. I used the word pointless, but in a completely different context. Now I get angry comments and threatening emails from hardcore fans. (google if you can learn Spanish with Duolingo, if you are lucky you'll see it) Arguments are settled when your point is displayed as a snippet on Google. The problem is, the counter argument can also be displayed with the right word combination. How do we tell the mass to go a little beyond the snippets? [1]: https://www.vanityfair.com/news/2018/05/uh-did-google-fake-its-big-ai-demo https://www.vanityfair.com/news/2018/05/uh-did-google-fake-i...
- mistermann 6y ago
  > Now I get angry comments and threatening emails from hardcore fans. Which is another example of a system (the human mind) not being smart enough to know it isn't smart enough. FTA: > Google’s Featured Snippets Are Worse Than Fake News, writes Adrianne Jeffries, pointing out the downsides of Google’s efforts to provide what Danny Sullivan calls the “one true answer” as fast as possible. About 15% of Google searches offer a featured snippet, that text excerpt that shows up inside a big bold box at the top of the results. It’s presented as the answer to your question. “Unfortunately, not all of these answers are actually true,” Jeffries writes. > You know, like this one: > "Barack Hussein Obama is implementing Alinsky’s rules on a much wider scale. He is using these rules to divide America so that he can cause widespread chaos and panic so he can declare martial war. This is his plan." I suspect most people here have encountered (in-person or at least online) a person who believes such things, and shake their head at how silly and illogical this is. But if you change the topic from Obama to something involving Trump, a lot of the time the very same phenomena ("This is(!) his plan/intent/desire") will manifest, often within the very same people that scoff at those who would believe the Obama narrative. And if you call them on it, the magical post-hoc rationalization[1] process in the brain will jump into action, manufacturing in real-time an elaborate, not-entirely-logic-based narrative that justifies System 1's objectively incorrect prediction...which is what I suspect also prevents people from realizing what has just happened. And, if you challenge that narrative, a variety of other very interesting and predictable behaviors will manifest (which is incredibly easy to see in individuals within one's out-group, but incredibly hard to see within your own behavior, because of the innate(!) post-hoc rationalization capabilities of the mind). This is what I refer to above when I say "the human mind is not smart enough to know it isn't smart enough" - this is simply how the human mind has evolved to work. Using Daniel Kahneman's terminology from "Thinking Fast and Slow"[2], this is simply an illustration of System 1 providing instantaneous (but not necessarily correct) answers to questions, and System 2 (the slow but more accurate conscious mind) not intercepting and correcting the prediction provided by System 1. Everyone falls victim to this (yes, even us geniuses on HN), no matter how intelligent and educated they are. > How do we tell the mass to go a little beyond the snippets? We could tell them to "just" be enlightened, like we tell people to "just" do a whole bunch of other things (be logical, intelligent, kind, understanding, "properly" informed and hold the "right" beliefs, etc). (In case it's not obvious, I say this part with tongue firmly in cheek.) [1] https://www.patheos.com/blogs/tippling/2013/11/14/post-hoc-rationalisation-reasoning-our-intuition-and-changing-our-minds/ https://www.patheos.com/blogs/tippling/2013/11/14/post-hoc-r... [2] https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow EDIT: A good and important example where this behavior can be easily observed is in the climate change debate - particularly online discussions, and particularly (and seemingly counter-intuitively, see my "protected" comment here[3]) among more intelligent and informed people. As usual, it is incredibly easy to see (and imagine(!)) the flaws in other people's thinking on the topic, but good luck seeing flaws in your own. This is why I keep saying the root problem in the climate change debate is not one of climate science (education and understanding), but of psychology/neurology. And once again, this idea seems to be very unpopular and uninteresting, especially to those who care deeply about the problem. [3] https://news.ycombinator.com/item?id=23093016 https://news.ycombinator.com/item?id=23093016
dang 6y ago
Discussed back then: https://news.ycombinator.com/item?id=13866493 https://news.ycombinator.com/item?id=13866493
saagarjha 6y ago
One of the problems with automation reporting how sure it is of its answer is that sometimes it really has no clue even about how confident it is. It'll confidently say something is a dog and it's actually a table with some pixels tweaked slightly. It's not really an easy problem to solve.
- miscPerson 6y ago
  Okay — but is the AI overconfident in choices more or less often than a human?

83 points by nromiun 6y ago

They're vocalized search engines, not knowledge builders. Naturally, they spit out what they see as a the top search, or something repeated often. The "How Barack Hussein Obama is going to declare Martial Law" example is great.

bryanrasmussen 6y ago
it amused me that if you posted an article called "How Barack Hussein Obama is going to declare Martial Law" to HN, the title would be rewritten to be "Barack Hussein Obama is going to declare Martial Law" amusement still applies with updated President of course.

code4tee 6y ago

Yes. This is applicable to most automation, machine learning and AI. These technologies are generally really bad at sense checking their results. At the end of the day they match patterns that are mathematically the “most correct” pattern even if it’s obviously not the correct answer. This is the Achilles heel of these technologies that is very hard to overcome and why any real applications generally still have a human in the loop. See Facebook’s “chatbot” experiment where they called it off after realizing the only way to have it work in practice was to have an army of humans behind the scenes sense checking answers. The grand AI engine that takes over for humans is still a pipe dream for most applications. Even all these “neutrality flags” are generally nothing more than a keyword search. Put words like COVID-19 or Coronavirus in your post and Medium puts a banner at the top saying the article hasn’t been fact checked.

nurettin 6y ago

Which search engine will understand sarcasm first? Which one will separate confidence from facts? These are the things that should have been focused on decades ago.

dorusr 6y ago

There is no algorithm of truth, and with enough data you can arrive at any conclusion you like. Maybe people should be better educated, instead of machines becoming smarter.

tomxor 6y ago
This is about confidence not truth, confidence is built into ML, it's how it works... the problem is these solutions currently just quantize that output into a single result and obscure how close to the threshold of another possible answer it was.
- yters 6y ago
  Confidence is a function of truth.
  - deleted 6y ago
    [deleted]

tragomaskhalos 6y ago

Google's inability to spot the negative implicit in "Shirt without stripes" was posted here a couple of weeks ago: https://news.ycombinator.com/item?id=22925087 https://news.ycombinator.com/item?id=22925087

GuiA 6y ago

This is really terrifying in the context of machine translation. I've seen a few forums (I think they were Discourse installs?) configured to automatically translate posts not in your browser language. It's very hard to tell that the post wasn't originally posted in English unless you look very hard for the tiny light-gray-on-white-background text. Machine translation, to put it mildly, is utterly hopeless at context and nuance. I've had Google spit out very racist/sexist results from innocent source text. The fact that it's so confident about what it shows, and that the engineering teams working on these products have very little modesty, means that people treat it as a source of a truth. A future where you have no clue if what you are reading is something written by a human as-is, or went through machine translation and might have nothing to do with what the original author intended, does not put me at ease.

heavenlyblue 6y ago
I would say Google’s answers answer about 10% if the questions they say they answer. I have seen so many time Google simply extracting the wrong paragraph of the text saying exactly the opposite of the gist it’s amazing. The worst thing is that there’s no way they can measure the efficiency of these answers directly (I.e. they can only measure how many times it wasn’t clicked as opposed it was). By the time users realised Google had outright lied to them with the answer the team that was shipping these answer widgets is already cashing out from shipping that project.

gitgud 6y ago

Seems similar to the way a programming language might handle errors. A function might handle these error cases, but if not it will bubble/throw it up to the parent for them to deal with it. In other words the function might be thinking; "I'm not smart enough to handle this..." It would probably be a good idea to compose AI bots like this. Different levels of bots, each bubbling up questions they're not smart enough to answer to higher level bots...

derefr 6y ago
The more-common opposite approach to AI (with a similar end-result)—often seen in robotics—is called a subsumption architecture: you have naive low-level "mechanism" models making the mundane non-edge-case decisions, with no awareness of the edge-cases; and then you have smarter case-specific "policy" models, trained to recognize when the conditions apply for some particular policy to be enforced, whose job is to change/override the decision the low-level agent makes, by MITMing ("subsuming") the lower-level model. Either they stand in the way of the low-level agent's input, and control it by biasing said input (i.e. lying to it, so it'll make the decision the high-level agent wants); or they stand in the way of the low-level agent's output, and replace it with their own output instead, when necessary. This architecture is particularly convenient when the low-level models have been constructed such that they give wacky results when asked out-of-domain questions. Rather than training the low-level model to respond to out-of-domain input in a sensible way (which might require far more training data, and decrease robustness for in-domain input), you can keep the low-level model unaware of the edges of the domain, and instead just add a high-level "validator" model whose only job is to recognize input that would be invalid for the low-level model, and answer in its stead, replacing the low-level model's noise answer with its own. In an imperative PL sense, this is less like regular exception handling (where the lower-level code knows it doesn't know how to handle something, and so explicitly bubbles up the error), and more like the higher-level code either checking/modifying input in advance of passing it to the lower-level code; or, in fact, telling the low-level code how to handle any errors that crop up. As if the low-level code just had a bald try{} block; and the determination of what that try{} block tries to catch, would be up to the callers below the block on the call stack, with the topmost caller getting the final decision on whether to continue, recover, abort, etc. right at the moment of the initial error, before ever returning its answer to its immediate parent. (A lot like Lisp conditions, actually.)

firefoxd 6y ago

One problem is we only talk about the victories and never educate the end-user who is supposed to use these systems. One great example was Google Duplex. Despite the impressive demo, people had a hard time recreating that success [1]. When a product is promoted as perfect, people start to believe it is perfect. I wrote an article about Duolingo, saying that you need a lot more than that to learn Spanish. Google displays a snippet that says it is pointless and points to my blog. I used the word pointless, but in a completely different context. Now I get angry comments and threatening emails from hardcore fans. (google if you can learn Spanish with Duolingo, if you are lucky you'll see it) Arguments are settled when your point is displayed as a snippet on Google. The problem is, the counter argument can also be displayed with the right word combination. How do we tell the mass to go a little beyond the snippets? [1]: https://www.vanityfair.com/news/2018/05/uh-did-google-fake-its-big-ai-demo https://www.vanityfair.com/news/2018/05/uh-did-google-fake-i...

mistermann 6y ago
> Now I get angry comments and threatening emails from hardcore fans. Which is another example of a system (the human mind) not being smart enough to know it isn't smart enough. FTA: > Google’s Featured Snippets Are Worse Than Fake News, writes Adrianne Jeffries, pointing out the downsides of Google’s efforts to provide what Danny Sullivan calls the “one true answer” as fast as possible. About 15% of Google searches offer a featured snippet, that text excerpt that shows up inside a big bold box at the top of the results. It’s presented as the answer to your question. “Unfortunately, not all of these answers are actually true,” Jeffries writes. > You know, like this one: > "Barack Hussein Obama is implementing Alinsky’s rules on a much wider scale. He is using these rules to divide America so that he can cause widespread chaos and panic so he can declare martial war. This is his plan." I suspect most people here have encountered (in-person or at least online) a person who believes such things, and shake their head at how silly and illogical this is. But if you change the topic from Obama to something involving Trump, a lot of the time the very same phenomena ("This is(!) his plan/intent/desire") will manifest, often within the very same people that scoff at those who would believe the Obama narrative. And if you call them on it, the magical post-hoc rationalization[1] process in the brain will jump into action, manufacturing in real-time an elaborate, not-entirely-logic-based narrative that justifies System 1's objectively incorrect prediction...which is what I suspect also prevents people from realizing what has just happened. And, if you challenge that narrative, a variety of other very interesting and predictable behaviors will manifest (which is incredibly easy to see in individuals within one's out-group, but incredibly hard to see within your own behavior, because of the innate(!) post-hoc rationalization capabilities of the mind). This is what I refer to above when I say "the human mind is not smart enough to know it isn't smart enough" - this is simply how the human mind has evolved to work. Using Daniel Kahneman's terminology from "Thinking Fast and Slow"[2], this is simply an illustration of System 1 providing instantaneous (but not necessarily correct) answers to questions, and System 2 (the slow but more accurate conscious mind) not intercepting and correcting the prediction provided by System 1. Everyone falls victim to this (yes, even us geniuses on HN), no matter how intelligent and educated they are. > How do we tell the mass to go a little beyond the snippets? We could tell them to "just" be enlightened, like we tell people to "just" do a whole bunch of other things (be logical, intelligent, kind, understanding, "properly" informed and hold the "right" beliefs, etc). (In case it's not obvious, I say this part with tongue firmly in cheek.) [1] https://www.patheos.com/blogs/tippling/2013/11/14/post-hoc-rationalisation-reasoning-our-intuition-and-changing-our-minds/ https://www.patheos.com/blogs/tippling/2013/11/14/post-hoc-r... [2] https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow EDIT: A good and important example where this behavior can be easily observed is in the climate change debate - particularly online discussions, and particularly (and seemingly counter-intuitively, see my "protected" comment here[3]) among more intelligent and informed people. As usual, it is incredibly easy to see (and imagine(!)) the flaws in other people's thinking on the topic, but good luck seeing flaws in your own. This is why I keep saying the root problem in the climate change debate is not one of climate science (education and understanding), but of psychology/neurology. And once again, this idea seems to be very unpopular and uninteresting, especially to those who care deeply about the problem. [3] https://news.ycombinator.com/item?id=23093016 https://news.ycombinator.com/item?id=23093016

dang 6y ago

Discussed back then: https://news.ycombinator.com/item?id=13866493 https://news.ycombinator.com/item?id=13866493

saagarjha 6y ago

One of the problems with automation reporting how sure it is of its answer is that sometimes it really has no clue even about how confident it is. It'll confidently say something is a dog and it's actually a table with some pixels tweaked slightly. It's not really an easy problem to solve.

miscPerson 6y ago
Okay — but is the AI overconfident in choices more or less often than a human?