It's less about over/under-performance and more with incentive alignment and political perception, and organizational maturity. It's really, really hard to get

It's less about over/under-performance and more with incentive alignment and political perception, and organizational maturity. It's really, really hard to get executive budgetary approval for a foundational data audit/cleaning project (comprehensive data cataloging, data cleaning, source auditing and validation, etc). Doing so implicitly admits that you weren't doing that before, and now you have to pay gobs of money to fix it. The larger the company, the more infeasible it is to push this through because of the breadth of technical/analytical debt that has accrued and the price tag associated with the project, combined with the perception of incompetency (i.e. it's an expensive project that's fixing a problem you as the executive shouldn't have let happen to begin with). Whereas an ML/AI project/initiative/push is a net new capability that you're spearheading, and it's easier to get the political traction to spend money on net new things, especially buzzwordy net new things that the firm can use to be viewed as cutting edge. The fact that you're rolling up the cost of a complete data management audit to be able to even do the ML/AI project is a minor bullet-point that doesn't matter. Executive expectation is that anything ML/AI/new-age-techy is going to be astronomically expensive anyway, so it doesn't get noticed that they're paying a premium on labor to do it as a combined project rather than as two separate projects. Effectively, the foundational work that's needed to support ML work is also the work that's needed to do basic SQL-based analytic work, but it's way easier to get that budgetary line approved in a flashy ML project, even if you're paying a premium by having the ML/AI firm do the foundational work instead of the specialist work. Plus, spearheading ML/AI initiatives make for better resume points than "data management initiatives". So there's little reason for anyone in this process to attempt to change anything, unless you happen to materially benefit from a firm's profit. For this reason, my main consulting clients are bootstrapped firms that actually care about being pragmatic over being trendsetters. Note: This is a huge generalization, and doesn't apply universally. But it's far more common than you would expect, especially as you veer away from the type of companies that pop up on HN towards more traditional industries.