Building a model on some training data is easy. Debugging it is hard. Th machine learned model is essentially equivalent of code that gives probabilistic answer

Building a model on some training data is easy. Debugging it is hard. Th machine learned model is essentially equivalent of code that gives probabilistic answers. But there are no breakpoints to put or watch to add. When model doesn't give the answer you expect, you need to do statistical debugging. Is the problem in training data, do you have correct sampling, are you missing a feature, is your feature selection optimal, does you featurizer have a bug somewhere? Possibilities are endless. Debugging ML model is black art. Most ML "users" would simply give up if model doesn't work on some cases or if they added a cool feature but results are not that good. In my experience less than 10% of people who claims ML expertise are actually good at doing statistical debugging and able to identify issue when model doesn't work as expected.