The problem comes when you need to peer inside the black box because it's working counter to your expectations of it: With most systems you can do so by reading

The problem comes when you need to peer inside the black box because it's working counter to your expectations of it: With most systems you can do so by reading code / documentation etc. With ML you're kind of stuck. You can conduct experiments to see what changes about the behavior of the model, but even that proves tricky when the model has thousands of inputs.