I’ve worked for over a decade in industry machine learning, about half of that in quant finance. It is definitely much worse in finance than other fields. Even

I’ve worked for over a decade in industry machine learning, about half of that in quant finance. It is definitely much worse in finance than other fields. Even medical records do not present the same degree of esoteric data formatting and mismatching. It’s not really even a matter of data cleaning. It’s that there is _no_ way to clean the data, and the only useful approach is to pay 10s of thousands of dollars to data vendors whose products have intractable errors, and then build huge data vslidation and imputation systems around it. When it boils down to fiduciary duty to the client, and you have a contractual obligation regarding portfolio composition, then you can’t live with “good enough” data cleaning. Even one single asset with an incorrect identifier from your data vendor can cause you to e.g. invest in an Israeli company in a portfolio with a client obligation to invest in no Israeli companies (that is a real example I encountered before).