As someone using ML in production at scale, it's not the sophistication of the models themselves, but all the "boring" operational stuff around them. - Keeping

As someone using ML in production at scale, it's not the sophistication of the models themselves, but all the "boring" operational stuff around them. - Keeping them fed with clean, high quality, low latency feature data. - Understanding the impact of an ML intervention on the overall system or business process, in aggregate and for all the relevant subpopulations. - Understanding why scores may be drifting. These are very challenging alerts to investigate. - Resource efficiency at very high QPS or in mobile contexts.