We touched on a lot of these questions in this talk about how we went from a prototype to a production machine learning system: https://vimeo.com/181931334 http

We touched on a lot of these questions in this talk about how we went from a prototype to a production machine learning system: https://vimeo.com/181931334 https://vimeo.com/181931334 The main point I haven't seen mentioned that often is to constantly verify your data and your data processing pipeline. We treat these checks as integration tests and run them as part of our continuous integration system. We also use New Relic to monitor model freshness, to be alerted if any part of the pipeline has broken.