3 ms·

The thing with evaluating these services is that business people or product managers don’t understand that accuracy on benchmarks doesn’t map well to accuracy o

by mlthoughts2018 6y ago

The thing with evaluating these services is that business people or product managers don’t understand that accuracy on benchmarks doesn’t map well to accuracy on your problem. If you have a wacky data distribution (hint: you do), say like a collection of photos all taken inside the same building, you can’t generalize to consider benchmark stats. On top of this, these services charge you for usage, not for accurate usage. This is a major issue that so many people overlook. What does it matter if it’s cheap per request? That only helps you if accuracy per request is very high. Otherwise you’re paying for cheap garbage, or in some use cases you must grow requests much larger to overcome errors per request, like if you’re trying to get a bulk of labeled data via these services. Most use cases are still better off hiring ML engineers who understand how to evaluate accuracy for the unique business use case, fine tune or train a model, and do it in house (or at least give you deep assurances of the rare cases when the big cloud services actually are cost effective to use). For most use cases, you’re wasting your money trying for ML-as-a-service like this.

yeldarb 6y ago
There's a huge gap between paying a couple hundred dollars for a pretty good model and investing hundreds of thousands (or millions) in hiring a team. Certainly if you're building something mission-critical, like self-driving cars, you want to get the best possible performance (regardless of price). But there's a whole other set of use-cases where "pretty good" is good enough (eg finding people sharing your company's logo in social media posts).
- mlthoughts2018 6y ago
  > “finding people sharing your company's logo in social media posts” that seems very different than your comment describes. I agree hobbyists may use these services. If a business is looking for spending “a couple hundred dollars” on a significant ML problem, that business is living in a fantasy and needs a reality check. There is really not a market for low end cost but high specific task accuracy business problems. Also your comparison with hiring a team is a very false dichotomy. Hiring a team may cost you $1MM but it’s amortized over all the work and projects they do. It’s not $1MM just for the in-house equivalent of this one AWK Rekognition task.

yeldarb 6y ago

There's a huge gap between paying a couple hundred dollars for a pretty good model and investing hundreds of thousands (or millions) in hiring a team. Certainly if you're building something mission-critical, like self-driving cars, you want to get the best possible performance (regardless of price). But there's a whole other set of use-cases where "pretty good" is good enough (eg finding people sharing your company's logo in social media posts).

mlthoughts2018 6y ago
> “finding people sharing your company's logo in social media posts” that seems very different than your comment describes. I agree hobbyists may use these services. If a business is looking for spending “a couple hundred dollars” on a significant ML problem, that business is living in a fantasy and needs a reality check. There is really not a market for low end cost but high specific task accuracy business problems. Also your comparison with hiring a team is a very false dichotomy. Hiring a team may cost you $1MM but it’s amortized over all the work and projects they do. It’s not $1MM just for the in-house equivalent of this one AWK Rekognition task.