3 ms·
With the rise of data science and machine learning, I've personally observed what I feel to be a trend (not Google-specific, but industry-wide) towards building
by crazygringo 7y ago
With the rise of data science and machine learning, I've personally observed what I feel to be a trend (not Google-specific, but industry-wide) towards building a model, using basic statistics to evaluate a model, and then just deploying it based on those statistics.
But the evaluation statistics tend to be averages, rates of false positives/negatives, and other macro-level things that often don't distinguish how some items (apps, in this case) are more important (popular or economically valuable) than others.
I've been in meetings all the time where this has happened -- the data scientist without a good handle on the business presents the statistics, and the product manager who knows the business trusts the data scientist. Hopefully someone with common sense will pipe up "but wait... what if we accidentally take down a really popular app?" But because it's 1 thing of 20 that are on the meeting's agenda, people miss it and it just gets approved.
And the further problem is that there's no standard process (that I'm aware of) for determining whether a data science model meets the actual business requirements, except a lot of trial and error that often involves actual business failures. It's pretty frustrating, really.
Because when it used to just be hard-coded rules, someone was responsible for qualitatively evaluating those rules, and common sense prevailed a bit more. But when data science models are often more of a black box, you evaluate with statistics, and what does it mean to then bring common sense in back on top of that?