3 ms·

Counterexample: Convolutional Neural Networks (CNNs) (or any ML model, really) We still don't fully understand how or why they work so well. If you have softw

by umvi 7y ago

Counterexample: Convolutional Neural Networks (CNNs) (or any ML model, really) We still don't fully understand how or why they work so well. If you have software that queries a CNN and the CNN returns some prediction, there's absolutely no way of knowing or understanding the reasoning behind the decision. It's a black box of magical layers and weights that have been "trained" to make good predictions. If you have fraud detection software using ML and it flags something as fraud, that's the extent of the information you get. In this case the computer cannot be understood - it cannot explain to you how it arrived at the conclusion that something is fraud, nor can you explain how the computer arrived at the decision. It just has huge tables of weights and biases that magically "know" when something is fraud.

thisisbrians 7y ago
True, but you also aren't tasked with going in and manually adjusting weights to improve the predictions. And if you are curious enough, you can certainly look at the theory and mathematics that underpin how these systems do work, even if you don't understand exactly how the training data is making its way into your model: someone clearly figured this out and that's why these systems work. Put another way: you can treat an algorithm or routine as a black box if you know it works and understand its context in your system. You can even learn the right questions to ask if you do need to dig into that particular box at some point. Without some abstractions, we'd all be constantly paralyzed. (edit: formatting)
- yuliyp 7y ago
  The problem comes when you need to peer inside the black box because it's working counter to your expectations of it: With most systems you can do so by reading code / documentation etc. With ML you're kind of stuck. You can conduct experiments to see what changes about the behavior of the model, but even that proves tricky when the model has thousands of inputs.
JoshTriplett 7y ago
There are numerous research results about building understandable, analyzable neural network models. Most models don't bother, because that isn't a necessary step to get reasonable answers. But it's possible.
- shrimp_emoji 7y ago
  Really? That's encouraging. I thought we're heading into some ironic dark enlightenment horseshoe of blackboxiness where the only artificial intelligence that works is as inscrutable as the natural intelligence it was meant to compete with.
- criddell 7y ago
  When the model is applied to a regulated part of the market, it's becoming apparent that the institutions using the model have to be able to defend it. David Heinemeier Hansson (DHH) posted an interesting thread to Twitter after he was granted a credit limit 20x higher than his wife and the credit card company's initial response was it's just the algorithm. https://twitter.com/dhh/status/1192540900393705474 https://twitter.com/dhh/status/1192540900393705474
Veluga 7y ago
The author has written about systems that defy detailed understanding as a follow up to this article in which he mentions Deep Reinforcement Learning Models as an example. It's a good read, especially in combination with this one.
shadowgovt 7y ago
It's an open (and kind of fascinating) field of research on how to make these more discoverable. Example: https://finale.seas.harvard.edu/files/finale/files/regional_tree_regularization_for_interpretability_in_deep_neural_networks.pdf https://finale.seas.harvard.edu/files/finale/files/regional_...
rvz 7y ago
Another point is that it's more dangerous to use deep-learning methods like this in highly regulated and sensitive environments like the healthcare industry for medical diagnosis. Medical devices involved in patient diagnosis are required to be transparent in its decision process in order to understand how it came to its decision. Despite CNNs having highly accurate results, its pretty much as useful as a black box to a doctor or medical professional if it cannot explain itself, which is one part of the hype of the deep-learning approach and is an ongoing problem in the AI research. There are biases that can occur in CNNs which can skew some results and worst of all, debugging these CNNs or deep-learning systems around "what went wrong" will be harder given its black box nature and will be done by tweaking the hyper-parameters to get a "good enough" result which is unacceptable to professionals in mission critical industries or those dealing with human life.
hinkley 7y ago
I love my bank but hate their fraud detection (someone suggested last time I brought this up that it might be outsourced). I went way across town to save a couple hundred bucks on a new TV. By the time I got home they had flagged by card. Oh, yeah, TV. I should call them and tell them it was me. No. They flagged it because I spent <$10 at a carwash on the way home. My old bank would flag me for buying gas on a road trip I took 3 times, every time. When your bank thinks that going out of town is exceptional behavior, well, it feels a little judgemental.
- cellularmitosis 7y ago
  Back when I bought a 250cc motorcycle, my card was constantly getting flagged (lots of $1.50 gas stops ~50 miles from home). It took months for this to let up.
- zanderwohl 7y ago
  The road trip one is more common to flag for fraud; if you leave your card at a gas station accidentally, someone may else may pick it up and make a few minor purchases. It happened to me when I used my credit card to fill up two cars - mine and my friends'. According to the attendant, it happens a lot.
geofft 7y ago
The author has a followup post about that exact case and experiences trying to understand them: https://blog.nelhage.com/post/systems-that-defy-understanding/ https://blog.nelhage.com/post/systems-that-defy-understandin...

by umvi 7y ago

thisisbrians 7y ago

True, but you also aren't tasked with going in and manually adjusting weights to improve the predictions. And if you are curious enough, you can certainly look at the theory and mathematics that underpin how these systems do work, even if you don't understand exactly how the training data is making its way into your model: someone clearly figured this out and that's why these systems work. Put another way: you can treat an algorithm or routine as a black box if you know it works and understand its context in your system. You can even learn the right questions to ask if you do need to dig into that particular box at some point. Without some abstractions, we'd all be constantly paralyzed. (edit: formatting)

yuliyp 7y ago
The problem comes when you need to peer inside the black box because it's working counter to your expectations of it: With most systems you can do so by reading code / documentation etc. With ML you're kind of stuck. You can conduct experiments to see what changes about the behavior of the model, but even that proves tricky when the model has thousands of inputs.

JoshTriplett 7y ago

There are numerous research results about building understandable, analyzable neural network models. Most models don't bother, because that isn't a necessary step to get reasonable answers. But it's possible.

shrimp_emoji 7y ago
Really? That's encouraging. I thought we're heading into some ironic dark enlightenment horseshoe of blackboxiness where the only artificial intelligence that works is as inscrutable as the natural intelligence it was meant to compete with.
criddell 7y ago
When the model is applied to a regulated part of the market, it's becoming apparent that the institutions using the model have to be able to defend it. David Heinemeier Hansson (DHH) posted an interesting thread to Twitter after he was granted a credit limit 20x higher than his wife and the credit card company's initial response was it's just the algorithm. https://twitter.com/dhh/status/1192540900393705474 https://twitter.com/dhh/status/1192540900393705474

Veluga 7y ago

The author has written about systems that defy detailed understanding as a follow up to this article in which he mentions Deep Reinforcement Learning Models as an example. It's a good read, especially in combination with this one.

shadowgovt 7y ago

It's an open (and kind of fascinating) field of research on how to make these more discoverable. Example: https://finale.seas.harvard.edu/files/finale/files/regional_tree_regularization_for_interpretability_in_deep_neural_networks.pdf https://finale.seas.harvard.edu/files/finale/files/regional_...

rvz 7y ago

Another point is that it's more dangerous to use deep-learning methods like this in highly regulated and sensitive environments like the healthcare industry for medical diagnosis. Medical devices involved in patient diagnosis are required to be transparent in its decision process in order to understand how it came to its decision. Despite CNNs having highly accurate results, its pretty much as useful as a black box to a doctor or medical professional if it cannot explain itself, which is one part of the hype of the deep-learning approach and is an ongoing problem in the AI research. There are biases that can occur in CNNs which can skew some results and worst of all, debugging these CNNs or deep-learning systems around "what went wrong" will be harder given its black box nature and will be done by tweaking the hyper-parameters to get a "good enough" result which is unacceptable to professionals in mission critical industries or those dealing with human life.

hinkley 7y ago

I love my bank but hate their fraud detection (someone suggested last time I brought this up that it might be outsourced). I went way across town to save a couple hundred bucks on a new TV. By the time I got home they had flagged by card. Oh, yeah, TV. I should call them and tell them it was me. No. They flagged it because I spent <$10 at a carwash on the way home. My old bank would flag me for buying gas on a road trip I took 3 times, every time. When your bank thinks that going out of town is exceptional behavior, well, it feels a little judgemental.

cellularmitosis 7y ago
Back when I bought a 250cc motorcycle, my card was constantly getting flagged (lots of $1.50 gas stops ~50 miles from home). It took months for this to let up.
zanderwohl 7y ago
The road trip one is more common to flag for fraud; if you leave your card at a gas station accidentally, someone may else may pick it up and make a few minor purchases. It happened to me when I used my credit card to fill up two cars - mine and my friends'. According to the attendant, it happens a lot.

geofft 7y ago

The author has a followup post about that exact case and experiences trying to understand them: https://blog.nelhage.com/post/systems-that-defy-understanding/ https://blog.nelhage.com/post/systems-that-defy-understandin...