3 ms·
There has been a lot of research in understanding neural networks and making them less of a black box. If you classify cat from dog videos on YouTube, it doesn'
by ma2rten 7y ago
There has been a lot of research in understanding neural networks and making them less of a black box. If you classify cat from dog videos on YouTube, it doesn't matter if you make a mistake every now and again. But if you want to build a self-driving car or make a medical diagnosis, you better be able to explain which your network made a certain decision.
- erikpukinskis 7y agoDoes the FDA require you are able to explain how drugs work? Or do you just have to show their efficacy and safety in trials?
- bonoboTP 7y agoI'm hearing this in the last few years quite often but I'm not sure what kind of explanation you mean. The x-ray image was incorrectly misdiagnosed, because... What type of thing should come here? ... it didn't look like the other class. ... it didn't have this weird smudge thing on the top left in which case usually there should be a little hazier blob in the middle, except when the pointiness of the thing that is to the right of the brightest blabla... You get the idea, my description is even exaggerating the nameability and describability of these structures. You'd get a long and complex description at best, because simple models don't work for pattern recognition. But even if you made sure to just use well understood features like edge thickness, angles, sizes of connected components etc, how would a boolean formula of a hundred such terms be helpful in court or wherever you want to use these explanations?
- ma2rten 7y agoFor example there is a concept visual attention. You plot which areas of the image the model pays attention to when making it's decision.