This paper is neat! More broadly, I am very concerned by how difficult it is to actually meaningfully run model explainability techniques on black-box models i

This paper is neat! More broadly, I am very concerned by how difficult it is to actually meaningfully run model explainability techniques on black-box models in the context of text/images in the real world. Most people are not AI researchers and do not want to deal with the hassle of trying random research code off of github (which usually is chalk full of bugs) the ELI5 library implements LIME really well for text, and I have not found an equivalent with SHAP that has all of the same functionality (highlighting most important words and exporting in pretty HTML being the big one). It's even worse when you start moving towards the neural network gradient explanations. There are libraries like Captum except in practice it fails on all the big-azz transformer language models I try to train. I'm still not sure as to how to run any of these gradient explanation methods on pytorch transformer models. Even if they do run, they don't display saliency maps with as much detail as the ELI5 library does... There's so much good research in explainable AI, and so much engineering work which remains to be done! FFS I had to hack the hell out of ELI5/LIME to get it to work on clustering (and I still don't know why explainable clustering is not really done, and why I couldn't find an off-the-shelf solution for this.)