4 ms·

It is a nice, albeit problematic example. Nonetheless I find many such examples in my daily work as a Data Analyst. Regarding recommender systems I see many co

It is a nice, albeit problematic example. Nonetheless I find many such examples in my daily work as a Data Analyst. Regarding recommender systems I see many companies trying neural nets and so many other fancy ML stuff for things that - in AB-tests are always outperformed by basic rules. I understand the fun it is to build stuff and to use the new hot stuff. And at least for many analysts and marketing people as well as shop product owners this is new hot shit. I also understand, that it is way more easy to get management to hand out the big bucks for something that is the new rage, as they tend to read the respective soundbites in their manager magazines. But as said - I see it underperforming in tests nearly all the time. Not only, but esp. if you take the costs of development and maintenance into account. These systems cost more to build, more to enhance, more to run and bring in less real business value 70 - 90 percent of all the times I have seen them. But they are presented to management in shiny presentations from agencies that need to sell the new hot shit to their clients to show that they are relevant. Because, as said, management thinks they need it and are often not open to agencies telling them, that the business value could be better served otherwise. Because in the end for a manager it is often times more valuable to show a fancy state of the art project to his/her higher ups than creating real business value.

mlthoughts2018 6y ago
> “ Regarding recommender systems I see many companies trying neural nets and so many other fancy ML stuff for things that - in AB-tests are always outperformed by basic rules.” I work on large scale recommender systems for an ecommerce company and in my career I’ve seen only the exact opposite. Don’t get me wrong, sometimes simpler ML models, like clustering LSA vectors or nearest neighbors, work better than complex models like neural nets. But I have never seen plain rule systems work better for any problem even remotely at scale. Rule systems give an illusory sense of control and understanding, yet are rife with complex interaction effects and edge cases that typically make them intractable to change.
- sdoering 6y ago
  From big automotive clients to small-ish fashion eCommerce. From publishing to food-delivery (with upselling in the checkout process) - I found the gains in using rules -> simple ML techniques -> complex systems like NN in most cases not to warrant the costs. The quality of recommendations nearly always increased from a revenue as well as perceived quality standpoint. However, it almost never had a positive impact on the profit margin that would have justified the necessary investments. As said - from a quality standpoint most simple systems were just "good enough". One would need to know the business case and environment. But take automotive (new cars) as an example: The goal is nearly always to get the user to request some form of contact from a physical dealership near them. For that you nearly never need a perfekt, fully configured recommendation. I know of an example (a car manufacturer) where the search space of all configurable variants (including various things the car owner would never register because they are specific screw variants) is of a size where even the number of visitors to the website per year is some orders of magnitude less than the number of options. The way to go here was to reduce the search space and the number of variants. Here it turned out that you can, quite fast, reach that goal with few specific questions (active learning) to lead the user to vehicle variants, which correspond to its interests and led to a disproportionately high contact behavior. And yes: ML techniques were used for the analysis and reduction of the search space. For the concept people to then develop specific questions to get to these reduced attributes. But in the end the recommender now works rule-based. I don't imply that this holds true for every scale of company/problem. And I know some counter examples - but most companies do not operate on that scale. If you are ebay, Zalando (Germany) and the likes: I would probably get different results from testing the revenue validity of the different approaches.
  - mlthoughts2018 6y ago
    Your comment is in wild and incredulous disagreement with widely published results and my own ~10 years of industry experience doing ML professionally in ecommerce, quant finance, education technology and quant advertising. In fact, I’ve always found even just plain cost per unit service goes down with the introduction of more complex ML models. Their greater training complexity and compute costs are much more than amortized by improved performance, easier ability to train and deploy new models (it’s much harder and labor intensive to adjust a rat’s nest of custom business rules than a black box ML model, even in terms of transparency). Just reduction of operating costs alone is usually a reason to favor ML solutions, even if they only achieve parity with rules systems (though usually they outperform them by a lot). Your comment makes me feel your methodology for assessing business value and comparing with rule systems is deeply flawed and probably biased to go against ML solutions for preconceived reasons.
- disgruntledphd2 6y ago
  It depends. Rule based systems are great if you have people with deep domain understanding developing the rules. Unfortunately, those people are rare, so most rule-based approaches fail to perform well. However, most recommendation systems suck unless you get someone who knows what they are doing to build them. In terms of business value, I would be very hesitant to make strogn statements like the above (in both cases, actually).
  - sdoering 6y ago
    Well - I said I have seen and tested. I would love for positive ML cases to arise. I really would. That would make it way easier to sell my Data Science colleagues to the respective clients on terms other than hype and buzzwords. I also believe that with a good situation in underlying data quality we could be talking about massively reduced costs in getting these systems up and running - and this would tip the scale in favor of said systems. But what I see in terms of data quality makes me sometimes just want to run as fast as I can in the other direction.
    - disgruntledphd2 6y ago
      Yeah, to make this stuff work well, you normally need lots of data, so consumer tech is mostly where you see successes. If the data isn't being logged by automated systems daily, then you probably don't have enough to make these kinds of things work. In smaller data environments, rules are going to perform much, much better (but still require the domain expertise, which isn't cheap).

mlthoughts2018 6y ago

> “ Regarding recommender systems I see many companies trying neural nets and so many other fancy ML stuff for things that - in AB-tests are always outperformed by basic rules.” I work on large scale recommender systems for an ecommerce company and in my career I’ve seen only the exact opposite. Don’t get me wrong, sometimes simpler ML models, like clustering LSA vectors or nearest neighbors, work better than complex models like neural nets. But I have never seen plain rule systems work better for any problem even remotely at scale. Rule systems give an illusory sense of control and understanding, yet are rife with complex interaction effects and edge cases that typically make them intractable to change.

sdoering 6y ago
From big automotive clients to small-ish fashion eCommerce. From publishing to food-delivery (with upselling in the checkout process) - I found the gains in using rules -> simple ML techniques -> complex systems like NN in most cases not to warrant the costs. The quality of recommendations nearly always increased from a revenue as well as perceived quality standpoint. However, it almost never had a positive impact on the profit margin that would have justified the necessary investments. As said - from a quality standpoint most simple systems were just "good enough". One would need to know the business case and environment. But take automotive (new cars) as an example: The goal is nearly always to get the user to request some form of contact from a physical dealership near them. For that you nearly never need a perfekt, fully configured recommendation. I know of an example (a car manufacturer) where the search space of all configurable variants (including various things the car owner would never register because they are specific screw variants) is of a size where even the number of visitors to the website per year is some orders of magnitude less than the number of options. The way to go here was to reduce the search space and the number of variants. Here it turned out that you can, quite fast, reach that goal with few specific questions (active learning) to lead the user to vehicle variants, which correspond to its interests and led to a disproportionately high contact behavior. And yes: ML techniques were used for the analysis and reduction of the search space. For the concept people to then develop specific questions to get to these reduced attributes. But in the end the recommender now works rule-based. I don't imply that this holds true for every scale of company/problem. And I know some counter examples - but most companies do not operate on that scale. If you are ebay, Zalando (Germany) and the likes: I would probably get different results from testing the revenue validity of the different approaches.
- mlthoughts2018 6y ago
  Your comment is in wild and incredulous disagreement with widely published results and my own ~10 years of industry experience doing ML professionally in ecommerce, quant finance, education technology and quant advertising. In fact, I’ve always found even just plain cost per unit service goes down with the introduction of more complex ML models. Their greater training complexity and compute costs are much more than amortized by improved performance, easier ability to train and deploy new models (it’s much harder and labor intensive to adjust a rat’s nest of custom business rules than a black box ML model, even in terms of transparency). Just reduction of operating costs alone is usually a reason to favor ML solutions, even if they only achieve parity with rules systems (though usually they outperform them by a lot). Your comment makes me feel your methodology for assessing business value and comparing with rule systems is deeply flawed and probably biased to go against ML solutions for preconceived reasons.
disgruntledphd2 6y ago
It depends. Rule based systems are great if you have people with deep domain understanding developing the rules. Unfortunately, those people are rare, so most rule-based approaches fail to perform well. However, most recommendation systems suck unless you get someone who knows what they are doing to build them. In terms of business value, I would be very hesitant to make strogn statements like the above (in both cases, actually).
- sdoering 6y ago
  Well - I said I have seen and tested. I would love for positive ML cases to arise. I really would. That would make it way easier to sell my Data Science colleagues to the respective clients on terms other than hype and buzzwords. I also believe that with a good situation in underlying data quality we could be talking about massively reduced costs in getting these systems up and running - and this would tip the scale in favor of said systems. But what I see in terms of data quality makes me sometimes just want to run as fast as I can in the other direction.
  - disgruntledphd2 6y ago
    Yeah, to make this stuff work well, you normally need lots of data, so consumer tech is mostly where you see successes. If the data isn't being logged by automated systems daily, then you probably don't have enough to make these kinds of things work. In smaller data environments, rules are going to perform much, much better (but still require the domain expertise, which isn't cheap).