"Once these systems get complicated enough, it gets really hard to do that." That's why I qualified it. Obviously if you're working in neural nets, you're not

"Once these systems get complicated enough, it gets really hard to do that." That's why I qualified it. Obviously if you're working in neural nets, you're not going to be cracking open that hood. But one of the dirty secrets of ML is you don't always need those techniques. There's the space of problems where linear regression and similar techniques are all you need, there's the class of problems where it doesn't matter what you throw at it you're not getting anything, and the "in between" where deep learning and such helps certainly has some interesting problems in the space, but it isn't necessarily the majority of problems you encounter in the wild. The scary thing is that not being able to crack open the hood doesn't mean the model isn't complete nonsense. It just means you can't examine it.