The ultimate test of any predictive model is whether it works in the intended stakeholder setting. Other types of diagnostics like ROC curves, performance on be

The ultimate test of any predictive model is whether it works in the intended stakeholder setting. Other types of diagnostics like ROC curves, performance on benchmark data sets, etc., are of course valuable, but ultimately do not necessarily reflect the true, run-time distribution of inputs or constraints that will be relevant for a stakeholder’s usage scenario. I’ve said this many times. What it means is that even if you think it’s great to outsource your machine learning model to a third party like Amazon, and consume it like a service, you really need in-house expertise in machine learning anyway, to help you interpret the accuracy, diagnostics, and some “integration test” notion of model performance on a sample that is agreed upon as truly representative of the stakeholder runtime conditions. So either way, you still have to pony up the dough to employ adequate in-house expertise in modeling and domain machine learning. As a result, there actually are not that many use cases when it would make sense to not build your own tailored model in-house. If you already must hire most of the staff with necessary skills to evaluate a third-party, the incremental effort to train a good-enough fine-tuning of some imagenet-based model is really not that high. And you can customize the training and diagnostics. This is especially critical when there are also stakeholder-specific latency or throughput constraints. For face detection in particular, I know a great deal about this directly related to AWS Rekognition, because my team extensively evaluated it for the possibility of outsourcing face detection calls in an in-browser image editing application. Not only could Rekognition not meet our runtime requirements, but separately it had poor accuracy and coverage (often capping out at detecting a small number of faces per image) and the Rekognition-reported confidence score aligned horribly with our own ground truth bounding box data that allowed us to quantify overlap of the detection with intersection-over-union scores. We tested a handful of fine-tuned prebuilt models in Keras: Resnet, Inception, and our own port of MTCNN, and had drastically better face detection (on a very large training corpus) than Rekognition in about a week. And our cost per request, even deploying to AWS, was much lower than Rekognition even at our high volume of traffic and even inclusive of the cost to spin up GPU instances for training. What’s more is that we tightly controlled how to write the web service layer wrapping it, and how to optimize the image preprocessing steps, etc., that we have no visibility into with a third party. I think there is a conceptual gap with this stuff where people just think that because you could commoditize a web service around a prediction algorithm, it must mean it would be economically valuable. But that part is the absolute least meaningful part. The whole enchilada is diagnosing how the model performs on the exact stakeholder use case, inclusive of performance constraints. For this reason, I think the only market for “ML tools for people who don’t know ML” is going to be just like vapid corporate IT consulting. Don’t get me wrong: big tech companies will profit from this. But not because it solves any actual prediction problems or saves anyone money compared with in-house ML development. It’ll just be the standard Dilbert-y politics that has always yielded profits to IT consulting services.