TLDR: Many ML models in production are terrible, because they were trained on terrible data. These bad models are being used in high stakes situations, such as

TLDR: Many ML models in production are terrible, because they were trained on terrible data. These bad models are being used in high stakes situations, such as COVID-19 detection. ML engineers need professional ethos/regulation, analogous to how civic engineers seeking to build a bridge don't screw around. My take: Yep, if the model is used a high stakes situation, this is absolutely the case. The model should be required to undergo rigorous testing / peer review before it's released into the wild. In a high stakes situations, we have to ensure that a model is good before people get their hands on it, because people can be reliably depended on to treat the model as an oracle. The metaphor of a "crumbling foundation" is a bad one, though. It's just unregulated; models aren't leaning on one another, and there isn't a risk of wholesale collapse.