7 ms·
How does this compare to or improve on applying something like SHAP[0][1] on a model? The idea in the first line that "we currently don't understand how to make
by adamiscool8 2y ago
How does this compare to or improve on applying something like SHAP[0][1] on a model? The idea in the first line that "we currently don't understand how to make sense of the neural activity within language models." is..straight up false?
[0] https://github.com/shap/shap https://github.com/shap/shap
[1] https://en.wikipedia.org/wiki/Shapley_value#In_machine_learning https://en.wikipedia.org/wiki/Shapley_value#In_machine_learn...
- szvsw 2y agoSHAP is pretty separate IMO. Shapley analysis is really a game theoretical methodology that is model agnostic and is only about determining how individual sections of the input contribute to a given prediction, not about how the model actually works internally to produce an output. As long as you have a callable black box, you can compute Shapley values (or approximations); it does not speak to how or why the model actually works internally.