This seems like a very useful framework to consider ML systems in. The thing about an ML system as such is that such a system is intended to turn big mounds of

This seems like a very useful framework to consider ML systems in. The thing about an ML system as such is that such a system is intended to turn big mounds of data into a predictions/classification without a human having to directly considered the multitude of questions otherwise addressed in large scale software design. IE, a multitude of boundaries and criteria are replaced by one criteria - "it works". The thing is that this set of boundaries and criteria still exists even if they individual setting the system considers the situation solved. This manifests both as the world changing over time and as other people not being perhaps as satisfied with the results of the system as those who created it, this being just two potential gotchas.