4 ms·

Cynthia has a great paper where she distinguishes between "interpretable" and "explainable" https://arxiv.org/pdf/1811.10154.pdf https://arxiv.org/pdf/1811.101

Cynthia has a great paper where she distinguishes between "interpretable" and "explainable" https://arxiv.org/pdf/1811.10154.pdf https://arxiv.org/pdf/1811.10154.pdf

randcraw 5y ago
"Explainable" just means you didn't build your app/service using any technology that isn't interpretable. Her argument is that the strategy of reverse engineering a method to convert it from inexplicable to explicable is inherently less effective than maintaining explicability at all times in the app's genesis -- from the design phase through implementation. But Rudin's Premise is philosophical more than practical. If the problem at hand is better solved using a black box (in terms of accuracy, precision, robustness, etc), her premise says simply, don't do it. Unfortunately in the cutthroat world of capitalism, that strategy can't compete with the cutting edge. Where Rudin's Premise is more suitable is in writing regulations to address AI app problems where social unfairness is unchecked (like the COMPAS app that advises legal authorities on meting out parole decisions without explaining its reasoning). There are many such (ab)uses for AI today in social services or policing which merit rethinking since AI-based injustice so offer bedevils the proprietary lack of transparency in such apps. Another excellent discussion of problems like these is Cathy O'Neil's book "Weapons of Math Destruction". Too bad she couldn't share the Squirrel prize. https://www.amazon.com/Weapons-Math-Destruction-Increases-Inequality/dp/0553418815 https://www.amazon.com/Weapons-Math-Destruction-Increases-In...
- qPM9l3XJrF 5y ago
  >If the problem at hand is better solved using a black box (in terms of accuracy, precision, robustness, etc) It's been a while since I read her work, but IIRC one of the positions she argues for, which I find plausible, is that interpretable models can be performance competitive. For example, it could be that the only reason black box methods outperform is because they've been more heavily researched, and if we were to put more research into interpretable methods, we could achieve parity. I also mentioned a few reasons why we might expect interpretable models to perform better a priori in this comment https://news.ycombinator.com/item?id=28838321 https://news.ycombinator.com/item?id=28838321
  - randcraw 5y ago
    I'd find Rudin's argument a lot more convincing if she offered an existence proof, like using the same number of examples to train an equally discriminative SVM or random forest (or hybrid) that can equal the performance of AlexNet in 2012 on the ImageNet ILSVRC (or in another domain where DNNs are SOTA). Until that can be done, I think few outside academia will invest time or money in alternative non-DNN methods in the hope of competing with today's even superior DNN variants. There's a decade of evidence now that DNNs are incontestable discriminators in numerous domains, relative to pre-2012 ML technology anyway.
    - qPM9l3XJrF 5y ago
      >There's a decade of evidence now that DNNs are incontestable discriminators in numerous domains, relative to pre-2012 ML technology anyway. Do we know that this is due to inherent superiority of DNNs, vs just experiencing a virtuous cycle of success leading to increasing investment leading to more success?
- pjmorris 5y ago
  From the linked paper, linked again [0] below, I think this represents Rudins philosophical view, and why it could be practical: Here is the Rashomon set argument: Consider that the data permit a large set of reasonably accurate predictive models to exist. Because this set of accurate models is large, it often contains at least one model that is interpretable. This model is thus both interpretable and accurate. Unpacking this argument slightly, for a given data set, we define the Rashomon set as the set of reasonably accurate predictive models (say within a given accuracy from the best model accuracy of boosted decision trees). Because the data are finite, the data could admit many close-to-optimal models that predict differently from each other: a large Rashomon set. I suspect this happens often in practice because sometimes many different machine learning algorithms perform similarly on the same dataset, despite having different functional forms (e.g., random forests, neural networks, support vector machines). As long as the Rashomon set contains a large enough set of models with diverse predictions, it probably contains functions that can be approximated well by simpler functions, and so the Rashomon set can also contain these simpler functions. Said another way, uncertainty arising from the data leads to a Rashomon set, a larger Rashomon set probably contains interpretable models, thus interpretable accurate models often exist. If this theory holds, we should expect to see interpretable models exist across domains. These interpretable models may be hard to find through optimization, but at least there is a reason we might expect that such models exist. [0] https://arxiv.org/pdf/1811.10154.pdf https://arxiv.org/pdf/1811.10154.pdf
nothrowaways 5y ago
Oh thanks for the pointers

Cynthia has a great paper where she distinguishes between "interpretable" and "explainable" https://arxiv.org/pdf/1811.10154.pdf https://arxiv.org/pdf/1811.10154.pdf

randcraw 5y ago

"Explainable" just means you didn't build your app/service using any technology that isn't interpretable. Her argument is that the strategy of reverse engineering a method to convert it from inexplicable to explicable is inherently less effective than maintaining explicability at all times in the app's genesis -- from the design phase through implementation. But Rudin's Premise is philosophical more than practical. If the problem at hand is better solved using a black box (in terms of accuracy, precision, robustness, etc), her premise says simply, don't do it. Unfortunately in the cutthroat world of capitalism, that strategy can't compete with the cutting edge. Where Rudin's Premise is more suitable is in writing regulations to address AI app problems where social unfairness is unchecked (like the COMPAS app that advises legal authorities on meting out parole decisions without explaining its reasoning). There are many such (ab)uses for AI today in social services or policing which merit rethinking since AI-based injustice so offer bedevils the proprietary lack of transparency in such apps. Another excellent discussion of problems like these is Cathy O'Neil's book "Weapons of Math Destruction". Too bad she couldn't share the Squirrel prize. https://www.amazon.com/Weapons-Math-Destruction-Increases-Inequality/dp/0553418815 https://www.amazon.com/Weapons-Math-Destruction-Increases-In...

qPM9l3XJrF 5y ago
>If the problem at hand is better solved using a black box (in terms of accuracy, precision, robustness, etc) It's been a while since I read her work, but IIRC one of the positions she argues for, which I find plausible, is that interpretable models can be performance competitive. For example, it could be that the only reason black box methods outperform is because they've been more heavily researched, and if we were to put more research into interpretable methods, we could achieve parity. I also mentioned a few reasons why we might expect interpretable models to perform better a priori in this comment https://news.ycombinator.com/item?id=28838321 https://news.ycombinator.com/item?id=28838321
- randcraw 5y ago
  I'd find Rudin's argument a lot more convincing if she offered an existence proof, like using the same number of examples to train an equally discriminative SVM or random forest (or hybrid) that can equal the performance of AlexNet in 2012 on the ImageNet ILSVRC (or in another domain where DNNs are SOTA). Until that can be done, I think few outside academia will invest time or money in alternative non-DNN methods in the hope of competing with today's even superior DNN variants. There's a decade of evidence now that DNNs are incontestable discriminators in numerous domains, relative to pre-2012 ML technology anyway.
  - qPM9l3XJrF 5y ago
    >There's a decade of evidence now that DNNs are incontestable discriminators in numerous domains, relative to pre-2012 ML technology anyway. Do we know that this is due to inherent superiority of DNNs, vs just experiencing a virtuous cycle of success leading to increasing investment leading to more success?
pjmorris 5y ago
From the linked paper, linked again [0] below, I think this represents Rudins philosophical view, and why it could be practical: Here is the Rashomon set argument: Consider that the data permit a large set of reasonably accurate predictive models to exist. Because this set of accurate models is large, it often contains at least one model that is interpretable. This model is thus both interpretable and accurate. Unpacking this argument slightly, for a given data set, we define the Rashomon set as the set of reasonably accurate predictive models (say within a given accuracy from the best model accuracy of boosted decision trees). Because the data are finite, the data could admit many close-to-optimal models that predict differently from each other: a large Rashomon set. I suspect this happens often in practice because sometimes many different machine learning algorithms perform similarly on the same dataset, despite having different functional forms (e.g., random forests, neural networks, support vector machines). As long as the Rashomon set contains a large enough set of models with diverse predictions, it probably contains functions that can be approximated well by simpler functions, and so the Rashomon set can also contain these simpler functions. Said another way, uncertainty arising from the data leads to a Rashomon set, a larger Rashomon set probably contains interpretable models, thus interpretable accurate models often exist. If this theory holds, we should expect to see interpretable models exist across domains. These interpretable models may be hard to find through optimization, but at least there is a reason we might expect that such models exist. [0] https://arxiv.org/pdf/1811.10154.pdf https://arxiv.org/pdf/1811.10154.pdf

nothrowaways 5y ago

Oh thanks for the pointers