6 ms·

I always feel that a good deal of data-driven anything is basically modern divination. The median data-driven decision might as well have been based on haruspi

I always feel that a good deal of data-driven anything is basically modern divination. The median data-driven decision might as well have been based on haruspicy. You can construct experiments to demonstrate damn near anything is true, and without (or even with) a rigorous background in statistics and scientific methodology, it's very hard to examine the supposed evidence. I know I say this a lot, but constructing solid experiments is time consuming and difficult. It's hard even if you you're trying to be fair. Professors routinely get this wrong despite a literal decade of formal training in the scientific method and half a lifetime of practical experience with this exact craft; despite how fucking up puts their career, reputation and livelihood in jeopardy. Not only does this happen a lot, bad science routinely slips through peer review as well. That's the state of affairs in the spaces that in general aren't extremely adversarial, where the people involved in constructing the experiments in general don't have a a vested interest in trying to mislead you. (Scientific fraud still happens of course, but Jan Hendrik Schön is the exception and not the rule)

User23 3y ago
> You can construct experiments to demonstrate damn near anything is true, and without (or even with) a rigorous background in statistics and scientific methodology, it's very hard to examine the supposed evidence. One time I was at this silly daylong corporate team building event where they assigned us to groups and had us do "experiments." As it happened one of my group members has a PhD in Particle Physics and the look on his face was something else. Needless to say he didn't bother telling the take charge type that her experiment design was entirely nonsensical, because why bother, but I was amused.
- marginalia_nu 3y ago
  ANOVA is that PBS show right?
kurthr 3y ago
Which explains why I've been getting ads for brides' maid dresses. However, it also seems as if a tuned LLM could look at the corpus of my on-line profile (mail, FB, HN) and produce a model of my attention and commentary (positive/negative). Then that model (or several models) could be tested against various ads to determine which are more likely to produce positive responses. Right now I think that would be too expensive except for very high value ads/targets, but perhaps pricing will come down and non-keyword based advertising might appear.
- otabdeveloper4 3y ago
  Absolutely not. Advertising is applied sociology, not psychology. Ads don't care about your personal wants or desires, they only care about Bayesian probabilities across broad target audience groups. (E.g. "dog lovers are X more likely to Y than cat lovers".) LLMs are useless in this domain.
  - hakfoo 3y ago
    That makes sense if we're looking at broad-scope advertising. I always figured there are some verticals where the value per acquisition was high enough, and the product narrowly-scoped enough, where it might make sense to microtarget individual customers-- profiling them individually and building custom messages for each of them. The obvious target would be the narrow-scope prescription drugs-- all those ads that say "If you have XYZ-positive, PDQ-negative Exploding Sphincter Syndrome Type XVI with Sprinkles and Cheese Sauce". Presumably there are only a few hundred people with that condition in the country, but you're spending a lot of money to reach everyone else to try to catch them in your net. Super-premium luxury goods might also fit there, but I wonder if the brand risk would be too high. Rolls Royce probably knows the 200 customers most likely to buy their new model, but it might not fit their brand to actually put together a campaign that seemed like they were actually TRYING to sell cars.
    - kurthr 3y ago
      If the cost is reasonable enough, it doesn't even have to be crafted for the individual, simply target them with one of multiple ads, which are modeled to have the highest (positive) effect... If you're not going to interact with it showing it to you has minimal effect. If one is modeled to have 5x higher interactivity, that's the one you want to show. I think there are high value customers for whom this probably makes sense, but only if those doing the targeting make money from it. Right now you're the product, but the supplier is the one doing the targeting (or not) and the customer doesn't know enough about their potential customers to target. The suppliers don't really make money by effective targeting.
    - SturgeonsLaw 3y ago
specialist 3y ago
> ...don't have a a vested interest in trying to mislead you. I witnessed this. I briefly worked on the recommendations team for a fashion retailer, doing yeoman plumbing and maintenance. Basically an in-house equiv of RichRelevance.com. So, so much data and logging. Alas, after cutting out all the obfuscation layers and steps, the "lift" attributed to the team's exquisite algos proved illusory. Some >80% of the conversions came from the "recently viewed" rule. Whereas the conversation attributed to the algos consuming most of our dev and compute resources was basically noise. I wouldn't say our (very smart) data scientists were fooling themselves, necessarily. More like the feedback loops were never fully validated. (Familiar story of non-programmer domain experts hoisting a POC, which then was promoted to production. Maintenance hell.) Coincidentally, FWIW, my bro has worked in ad tech (Avenue A/Microsoft, RichRelevance, others?). He's got a math skillset. Was a true believer. It took a long time, but he eventually lost his "faith" that digital advertising (as they were doing it) was effective, on the balance. Also FWIW, during my brief stint, I advocated "personalization". What I imagine StitchFix is doing. I'd love to know, for real, how they're doing. Like the split between algos and human "style consultants".
goodcharles 3y ago
Meanwhile, people with no scientific training are entrusted to run A/B tests and other web experiments. Practically all web marketing experiments and analysis is invalid, and not replicable. But it pays the bills for everyone to pretend its legitimate.
- akiselev 3y ago
  I've worked at several startups now where the people in charge of A/B testing couldn't design a single A/B test that returned statistically significant results out of dozens of attempts each. Not. A. Single. One. Even the so called experts. And that's using their own calculations, which were already poorly designed attempts at p hacking. The worst part is that some of the tests were still used to justify decisions to and by management. SMH
  - parpfish 3y ago
    Is that the fault of the experiment? Or was it a weak manipulation and no-result was correct (e.g., changing the purchase button to a slightly different shade of green and expecting a higher conversion rate)?
    - edmundsauto 3y ago
      IMO, this is still a failure to understand split testing. It is not to discover if changing a button color matters, but to explore the universe of all possible treatments and how they impact your most important business metrics. It’s a global optimization problem, not a scientific way to understand how a specific change impacts users. Testers that have this mindset tend to be locally constrained and less likely to have bigger wins.
    - lelanthran 3y ago
      > Is that the fault of the experiment? No. It's the fault of middle management. If you are unable to provide your manager reasons for new development, he's going to find someone else to do your job, someone who will give him a report with charts and numbers that he can use to justify and expand his teams operational and headcount budget. Giving a manager a report that says "there are little to no modifications that we can make at this time to improve UX" is a CLM for him.
stemlord 3y ago
One doesn't even need to observe all this to recognize the simple fact that craigslist is much nicer to use than twitter, reddit, tiktok and craigslist hasn't changed in decades I dont think.
- sealeck 3y ago
  If this is true, then why do more people use EBay, etc? Is it only their marketing power, or perhaps something about the platform better? Not trying to be confrontational, just interested in finding out why.
  - TacticalCoder 3y ago
    > If this is true, then why do more people use EBay To be honest Ebay, just like Craigslist, didn't change much in the last 20 years.
  - kjkjadksj 3y ago
    If you want to take a lot of a thousand products from aliexpress and sell them globally on the internet, ebay is your platform. If you want to sell something of yours locally or rent a local apartment, craigslist is your platform. These platforms reflect these usages when you browse them. Local results from people who just want to move a single item in their home vs some global drop shipping arbitrage opportunity will always be of higher quality.
    - pessimizer 3y ago
      > Local results from people who just want to move a single item in their home vs some global drop shipping arbitrage opportunity will always be of higher quality. Sometimes you want one of those Chinese paper cutters, or reasonably priced leather tools. Ebay is mostly good, despite itself. They've also barely changed the UI in my memory, and I've got a 20 year old account.
      - kjkjadksj 3y ago
        I will agree on that. If I want a random thing, there will be the same product dropped shipped on both sites but usually ebay comes out a few dollars cheaper.
IG_Semmelweiss 3y ago
There's always the famous case of Uber made 120M cuts[2] in their 150M adspend, nothing absolutely changed in their app installs. [1] [1] https://thehustle.co/01072021-uber-ad-spend/ https://thehustle.co/01072021-uber-ad-spend/ [2] https://www.forbes.com/sites/augustinefou/2021/01/02/when-big-brands-stopped-spending-on-digital-ads-nothing-happened-why/ https://www.forbes.com/sites/augustinefou/2021/01/02/when-bi...

User23 3y ago

> You can construct experiments to demonstrate damn near anything is true, and without (or even with) a rigorous background in statistics and scientific methodology, it's very hard to examine the supposed evidence. One time I was at this silly daylong corporate team building event where they assigned us to groups and had us do "experiments." As it happened one of my group members has a PhD in Particle Physics and the look on his face was something else. Needless to say he didn't bother telling the take charge type that her experiment design was entirely nonsensical, because why bother, but I was amused.

marginalia_nu 3y ago
ANOVA is that PBS show right?

kurthr 3y ago

Which explains why I've been getting ads for brides' maid dresses. However, it also seems as if a tuned LLM could look at the corpus of my on-line profile (mail, FB, HN) and produce a model of my attention and commentary (positive/negative). Then that model (or several models) could be tested against various ads to determine which are more likely to produce positive responses. Right now I think that would be too expensive except for very high value ads/targets, but perhaps pricing will come down and non-keyword based advertising might appear.

otabdeveloper4 3y ago
Absolutely not. Advertising is applied sociology, not psychology. Ads don't care about your personal wants or desires, they only care about Bayesian probabilities across broad target audience groups. (E.g. "dog lovers are X more likely to Y than cat lovers".) LLMs are useless in this domain.
- hakfoo 3y ago
  That makes sense if we're looking at broad-scope advertising. I always figured there are some verticals where the value per acquisition was high enough, and the product narrowly-scoped enough, where it might make sense to microtarget individual customers-- profiling them individually and building custom messages for each of them. The obvious target would be the narrow-scope prescription drugs-- all those ads that say "If you have XYZ-positive, PDQ-negative Exploding Sphincter Syndrome Type XVI with Sprinkles and Cheese Sauce". Presumably there are only a few hundred people with that condition in the country, but you're spending a lot of money to reach everyone else to try to catch them in your net. Super-premium luxury goods might also fit there, but I wonder if the brand risk would be too high. Rolls Royce probably knows the 200 customers most likely to buy their new model, but it might not fit their brand to actually put together a campaign that seemed like they were actually TRYING to sell cars.
  - kurthr 3y ago
    If the cost is reasonable enough, it doesn't even have to be crafted for the individual, simply target them with one of multiple ads, which are modeled to have the highest (positive) effect... If you're not going to interact with it showing it to you has minimal effect. If one is modeled to have 5x higher interactivity, that's the one you want to show. I think there are high value customers for whom this probably makes sense, but only if those doing the targeting make money from it. Right now you're the product, but the supplier is the one doing the targeting (or not) and the customer doesn't know enough about their potential customers to target. The suppliers don't really make money by effective targeting.
  - SturgeonsLaw 3y ago

specialist 3y ago

> ...don't have a a vested interest in trying to mislead you. I witnessed this. I briefly worked on the recommendations team for a fashion retailer, doing yeoman plumbing and maintenance. Basically an in-house equiv of RichRelevance.com. So, so much data and logging. Alas, after cutting out all the obfuscation layers and steps, the "lift" attributed to the team's exquisite algos proved illusory. Some >80% of the conversions came from the "recently viewed" rule. Whereas the conversation attributed to the algos consuming most of our dev and compute resources was basically noise. I wouldn't say our (very smart) data scientists were fooling themselves, necessarily. More like the feedback loops were never fully validated. (Familiar story of non-programmer domain experts hoisting a POC, which then was promoted to production. Maintenance hell.) Coincidentally, FWIW, my bro has worked in ad tech (Avenue A/Microsoft, RichRelevance, others?). He's got a math skillset. Was a true believer. It took a long time, but he eventually lost his "faith" that digital advertising (as they were doing it) was effective, on the balance. Also FWIW, during my brief stint, I advocated "personalization". What I imagine StitchFix is doing. I'd love to know, for real, how they're doing. Like the split between algos and human "style consultants".

goodcharles 3y ago

Meanwhile, people with no scientific training are entrusted to run A/B tests and other web experiments. Practically all web marketing experiments and analysis is invalid, and not replicable. But it pays the bills for everyone to pretend its legitimate.

akiselev 3y ago
I've worked at several startups now where the people in charge of A/B testing couldn't design a single A/B test that returned statistically significant results out of dozens of attempts each. Not. A. Single. One. Even the so called experts. And that's using their own calculations, which were already poorly designed attempts at p hacking. The worst part is that some of the tests were still used to justify decisions to and by management. SMH
- parpfish 3y ago
  Is that the fault of the experiment? Or was it a weak manipulation and no-result was correct (e.g., changing the purchase button to a slightly different shade of green and expecting a higher conversion rate)?
  - edmundsauto 3y ago
    IMO, this is still a failure to understand split testing. It is not to discover if changing a button color matters, but to explore the universe of all possible treatments and how they impact your most important business metrics. It’s a global optimization problem, not a scientific way to understand how a specific change impacts users. Testers that have this mindset tend to be locally constrained and less likely to have bigger wins.
  - lelanthran 3y ago
    > Is that the fault of the experiment? No. It's the fault of middle management. If you are unable to provide your manager reasons for new development, he's going to find someone else to do your job, someone who will give him a report with charts and numbers that he can use to justify and expand his teams operational and headcount budget. Giving a manager a report that says "there are little to no modifications that we can make at this time to improve UX" is a CLM for him.

stemlord 3y ago

One doesn't even need to observe all this to recognize the simple fact that craigslist is much nicer to use than twitter, reddit, tiktok and craigslist hasn't changed in decades I dont think.

sealeck 3y ago
If this is true, then why do more people use EBay, etc? Is it only their marketing power, or perhaps something about the platform better? Not trying to be confrontational, just interested in finding out why.
- TacticalCoder 3y ago
  > If this is true, then why do more people use EBay To be honest Ebay, just like Craigslist, didn't change much in the last 20 years.
- kjkjadksj 3y ago
  If you want to take a lot of a thousand products from aliexpress and sell them globally on the internet, ebay is your platform. If you want to sell something of yours locally or rent a local apartment, craigslist is your platform. These platforms reflect these usages when you browse them. Local results from people who just want to move a single item in their home vs some global drop shipping arbitrage opportunity will always be of higher quality.
  - pessimizer 3y ago
    > Local results from people who just want to move a single item in their home vs some global drop shipping arbitrage opportunity will always be of higher quality. Sometimes you want one of those Chinese paper cutters, or reasonably priced leather tools. Ebay is mostly good, despite itself. They've also barely changed the UI in my memory, and I've got a 20 year old account.
    - kjkjadksj 3y ago
      I will agree on that. If I want a random thing, there will be the same product dropped shipped on both sites but usually ebay comes out a few dollars cheaper.

IG_Semmelweiss 3y ago

There's always the famous case of Uber made 120M cuts[2] in their 150M adspend, nothing absolutely changed in their app installs. [1] [1] https://thehustle.co/01072021-uber-ad-spend/ https://thehustle.co/01072021-uber-ad-spend/ [2] https://www.forbes.com/sites/augustinefou/2021/01/02/when-big-brands-stopped-spending-on-digital-ads-nothing-happened-why/ https://www.forbes.com/sites/augustinefou/2021/01/02/when-bi...