Your post makes some really salient points, and then misses the head of the nail in my experience. Sure, most of the information Sears historically collected i

Your post makes some really salient points, and then misses the head of the nail in my experience. Sure, most of the information Sears historically collected is probably junk for supervised learning models. That isn't what makes this hard. High cardinality categorical features aren't what makes this hard. Re-engineering the business processes aren't what makes this hard. What makes this hard, is that for all of these companies, machine learning models are essentially used in place of heuristics of varying degrees of complexity. The models are being used to incrementally improve heuristics that in some cases are tuned quite well. Couple that with the issue that a machine learning model is only one piece of an actual product improvement for these websites/companies (ie: now you have a prediction, what are you gonna do with it to effect the product?), and all of the sudden you have actual incremental improvements completely misaligned with moonshot expectations. We've had plenty of rankers and recommenders for decades, the current wave of deep learning variants are improvements, but they're incremental from a high level. If your business wasn't successful/is failing using simple heuristics (sears, jcp, etc), a machine learning model isn't going to magically correct that.