4 ms·

The best argument for the usefulness/profitability of deep tracking is simple statistical inference. As in if there are 10000 people that with browsing pattern

by afiori 6y ago

The best argument for the usefulness/profitability of deep tracking is simple statistical inference. As in if there are 10000 people that with browsing patterns similar to yours and non-trivial fraction of them one day show interest in a discount/service/trip then you are also likely to be interested. It is the official explanation of how facebook/google can magically listen to conversation IRL that they were not supposed to hear: we simply are not as unique/unpredictable as we would like to think.

baybal2 6y ago
I think it's one of the best argument against. It was at least 5 years ago when I started to notice businesses "A/B testing themselves to death" — "It makes no common sense, but increases sales," and then it does until it doesn't, and you are left with no idea what your client actually wants, and no idea how to get out of that "A/B tested to absurdity" situation
- mumblemumble 6y ago
  This also seems to be an inherent danger with thinking that data science == machine learning == AI. I don't want to fundamentally challenge the ŷ-centric approach approach of machine learning. But I do want to say that there's a lot of danger in taking it too far, and thinking that β̂ doesn't matter as long as ŷ is good enough. β̂ has a habit of not mattering until, very suddenly, it really matters quite a lot. And it's even more dangerous to think that an algorithm for generating formulas is somehow doing something intelligent just because you like what you're seeing in the ŷ department.
  - afiori 6y ago
    As someone that is not "in" with those fields what do ŷ and β̂ stand for here?
    - mumblemumble 6y ago
      ŷ is the prediction you get out of the formula, β̂ is the coefficient(s) on the variables in the formula. So, take the simplest possible model: You think that your dependent variable is linear in the dependent variable, and that the line passes through (0,0). That gives us a simplified version of the classic equation for a line, where we say that y is equal to x times some constant that indicates the slope of the line: y = mx In machine learning (or statistical inference), the variable conventions are slightly different, so it would look more like: y = βx And then, since this is machine learning (or statistical inference, or whatever), we don't know the true equation, we're just working with an estimate we've inferred. We stick hats over the estimated values to make it clear that they're estimates: ŷ = β̂x What I was describing, then, is which side of the formula you're focusing on. Traditionally, statistics has been all about scientific inference, and trying to figure out and characterize causal relationships. So you want to know "If I change x in some way, how will that affect the outcome, all else being equal?" β̂ is your tool for teasing those things apart. Machine learning is typically cast as being more about making predictions from observational data. ŷ is just the formal notation for describing those predictions. It's pretty common in machine learning to say, "We are working with observational data, we're not worried about causality, so we'll only pay attention to the predictions." There can be value to that approach, but only to the extent that you really don't care about causality. Which. . . surprisingly often, β̂ matters, even when the people who built the model were trying to ignore it. It becomes especially important when you're using the model to make decisions. For example, when a company is using a machine learning model to try and identify candidates for a position they're trying to fill, they should absolutely be paying attention to what the model thinks about people who liked Lane Bryant on Facebook, and how that compares to people who liked Jos A Bank, and whether they really want things like that to be playing into how they make hiring decisions.
- cortesoft 6y ago
  Yeah, I often wonder this... a/b tests only ever measure short term differences. What if the change you are making increased sales for a few months, but pisses off users in the long run? An A/B test isn't going to show that.
  - shostack 6y ago
    A regression model might if you have sufficient data.
    - cortesoft 6y ago
      You would have to run the a/b test for quite a while, and I don't think many companies do.
code4tee 6y ago
I see that but I think the broader point (and the point of the article) is that too often “success” in finding some less obvious link actually comes at the cost of overlooking the simple stuff. In the end there’s an argument that while neat the fancy ad targeting actually fails to meet the core business objective of maximizing ad performance.
- afiori 6y ago
  My personal opinion is that for most companies having an organically grown audience is more valuable than a lot of faceless traffic. Obviously they can help each other if you do things right and are lucky.

by afiori 6y ago

baybal2 6y ago

I think it's one of the best argument against. It was at least 5 years ago when I started to notice businesses "A/B testing themselves to death" — "It makes no common sense, but increases sales," and then it does until it doesn't, and you are left with no idea what your client actually wants, and no idea how to get out of that "A/B tested to absurdity" situation

mumblemumble 6y ago
This also seems to be an inherent danger with thinking that data science == machine learning == AI. I don't want to fundamentally challenge the ŷ-centric approach approach of machine learning. But I do want to say that there's a lot of danger in taking it too far, and thinking that β̂ doesn't matter as long as ŷ is good enough. β̂ has a habit of not mattering until, very suddenly, it really matters quite a lot. And it's even more dangerous to think that an algorithm for generating formulas is somehow doing something intelligent just because you like what you're seeing in the ŷ department.
- afiori 6y ago
  As someone that is not "in" with those fields what do ŷ and β̂ stand for here?
  - mumblemumble 6y ago
    ŷ is the prediction you get out of the formula, β̂ is the coefficient(s) on the variables in the formula. So, take the simplest possible model: You think that your dependent variable is linear in the dependent variable, and that the line passes through (0,0). That gives us a simplified version of the classic equation for a line, where we say that y is equal to x times some constant that indicates the slope of the line: y = mx In machine learning (or statistical inference), the variable conventions are slightly different, so it would look more like: y = βx And then, since this is machine learning (or statistical inference, or whatever), we don't know the true equation, we're just working with an estimate we've inferred. We stick hats over the estimated values to make it clear that they're estimates: ŷ = β̂x What I was describing, then, is which side of the formula you're focusing on. Traditionally, statistics has been all about scientific inference, and trying to figure out and characterize causal relationships. So you want to know "If I change x in some way, how will that affect the outcome, all else being equal?" β̂ is your tool for teasing those things apart. Machine learning is typically cast as being more about making predictions from observational data. ŷ is just the formal notation for describing those predictions. It's pretty common in machine learning to say, "We are working with observational data, we're not worried about causality, so we'll only pay attention to the predictions." There can be value to that approach, but only to the extent that you really don't care about causality. Which. . . surprisingly often, β̂ matters, even when the people who built the model were trying to ignore it. It becomes especially important when you're using the model to make decisions. For example, when a company is using a machine learning model to try and identify candidates for a position they're trying to fill, they should absolutely be paying attention to what the model thinks about people who liked Lane Bryant on Facebook, and how that compares to people who liked Jos A Bank, and whether they really want things like that to be playing into how they make hiring decisions.
cortesoft 6y ago
Yeah, I often wonder this... a/b tests only ever measure short term differences. What if the change you are making increased sales for a few months, but pisses off users in the long run? An A/B test isn't going to show that.
- shostack 6y ago
  A regression model might if you have sufficient data.
  - cortesoft 6y ago
    You would have to run the a/b test for quite a while, and I don't think many companies do.

code4tee 6y ago

I see that but I think the broader point (and the point of the article) is that too often “success” in finding some less obvious link actually comes at the cost of overlooking the simple stuff. In the end there’s an argument that while neat the fancy ad targeting actually fails to meet the core business objective of maximizing ad performance.

afiori 6y ago
My personal opinion is that for most companies having an organically grown audience is more valuable than a lot of faceless traffic. Obviously they can help each other if you do things right and are lucky.