4 ms·

Because ad click through rates are so low and the signal is so incredibly sparse. It's all measured against a test set which the researchers rarely look at dire

Because ad click through rates are so low and the signal is so incredibly sparse. It's all measured against a test set which the researchers rarely look at directly (because the data set is so large). So if something lifts ad vlicks by 1% and that's from 100 in 10k to 101 in 10k, they don't really care if 9k of the ads get appreciably dumber.

cryptonector 7y ago
Sure, but if you buy a rice cooker, maybe the ads to show you are for: rice, cookbooks (featuring lots of rice?), pressure cookers, cast iron pans, sous vide recirculators, and so on. I mean, you have a rice cooker, you won't want another, but chances are you'd buy other things you don't already have.
- nmjohn 7y ago
  What is the ratio between people who viewed a rice cooker (the group I'm assuming they target with ads) vs. those who actually bought a rice cooker? Is it possible the ratio is so large to where while it obviously could be optimized to exclude people who have since purchased said item - the optimization is only a rounding error, and may not turn out to be worth it?
  - ericd 7y ago
    Well, I buy things from Amazon very frequently, and I have a high conversion rate between searched-for-category and bought something to satisfy the search, so Amazon ads are for things I've already taken care of >90% of the time. I'm guessing they're leaving a lot of money on the table from me because of it. But maybe I'm unusual in my shopping patterns.
- ahmedalsudani 7y ago
  It's not as simple as that. There are multiple ways to figure out which ad to show to a user, but I think the most efficient way (development and compute) is to put a user in segments (e.g. lives in area X, searched for Y, age Z, ...) and then do the bidding magic. There are tight tolerances for bidding--you want it to happen within a fraction of a second so the ad quickly loads for the user. Feeding purchase data is likely something that requires a bit of work, and I imagine the ad team at Amazon doesn't see enough value to prioritize it (especially since they make money either way--the advertisers pay for the ad with a higher CPC). One thing to keep in mind is that the purchase data is per-user, so it's different from the segment approach I was mentioning above. It would require a whole new schema* that is queried whenever bids are processed for an ad event. * Sorry for using the word schema here. It was used a lot on a team I was on in a similar context, and it bothered me at first but I got used to it. What I mean by schema is a "blob" with purchase data that is periodically processed by the bidding subsystem to add bid exclusions per user.
- Normal_gaussian 7y ago
  Its funny you should use that example. I recently bought a rice cooker, and then bought another a few weeks later. I've done similar things a lot, in this case it was because the first cooker had too large a minimum cooking amount. I'm currently looking at buying my mother google WiFi - which I bought a few weeks ago. And a few months back I rebought a tool I had just purchased because I lost it. I currently work for a (responsible) own site adtech (personalization) company. Its surprising what behaviours our clients find profitable for their business.
  - MRD85 7y ago
    You said you work for adtech, I'd just like to comment that I personally like targetted ads. Areas of my life have improved from targetted ads, I'm shown things I didn't even know I want until I've seen them, etc. I look at it as a positive.
    - Normal_gaussian 7y ago
      Its an interesting one. I've made some pretty active choices about jobs for ethical reasons but ended up in a slight geographical bind in my last search. I found my current company through a friend (who works in another department) and was very skeptical. After not finding anything bad I decided to go with it for a few months, but had a resignation letter ready in my car. I'm now over a year in and despite the normal problems businesses have there is not a moral one here - and I've had pretty great insight into the decision making.
  - dhimes 7y ago
    There's a whole lot about advertising I don't know, but the question that sticks in my mind is, did you need that ad to know which rice cooker to buy the second time? Did it change your decision? Did you see an ad for a different rice cooker and act on that? It seems like an effective ad would be this: Say you bought a rice cooker on Amazon for $75. Now Walmart shows you an ad for the same rice cooker for $60. That would certainly stick in my head- not for the rice cooker but for the price difference that Walmart can provide.
mattkrause 7y ago
> they don't really care if 9k of the ads get appreciably dumber. But maybe they should. The recipients of those 9,000 ads are being trained to ignore the recommendations because they’re so useless, and may continue to ignore its suggestions even after it improves. Likewise, a few clueless recommendations can “spoil” good ones shown at the same time. None of this shows up in the test set, of course, but people tend to turn off their brains when ‘evaluating’ ML stuff.
- shawnz 7y ago
  > The recipients of those 9,000 ads are being trained to ignore the recommendations ... Likewise, a few clueless recommendations can “spoil” good ones shown at the same time. How can you be sure that "random" untargeted ads don't have just as bad of an effect? Ultimately Amazon can see the numbers and we can't, and they made their decision based on that. It's pointless to argue over theories without data.
sizzle 7y ago
But couldn't they improve the signal to noise ratio by showing contextually relevant ads e.g. showing rice cooker cookbooks after purchasing a rice cooker? Wouldn't this lift ad click through rate? This is truly baffling expected behavior, who would ever buy a second rice cooker after having searched Amazon for rice cooker and compared a few top models already? I'd love to see some studies or data proving otherwise...

cryptonector 7y ago

Sure, but if you buy a rice cooker, maybe the ads to show you are for: rice, cookbooks (featuring lots of rice?), pressure cookers, cast iron pans, sous vide recirculators, and so on. I mean, you have a rice cooker, you won't want another, but chances are you'd buy other things you don't already have.

nmjohn 7y ago
What is the ratio between people who viewed a rice cooker (the group I'm assuming they target with ads) vs. those who actually bought a rice cooker? Is it possible the ratio is so large to where while it obviously could be optimized to exclude people who have since purchased said item - the optimization is only a rounding error, and may not turn out to be worth it?
- ericd 7y ago
  Well, I buy things from Amazon very frequently, and I have a high conversion rate between searched-for-category and bought something to satisfy the search, so Amazon ads are for things I've already taken care of >90% of the time. I'm guessing they're leaving a lot of money on the table from me because of it. But maybe I'm unusual in my shopping patterns.
ahmedalsudani 7y ago
It's not as simple as that. There are multiple ways to figure out which ad to show to a user, but I think the most efficient way (development and compute) is to put a user in segments (e.g. lives in area X, searched for Y, age Z, ...) and then do the bidding magic. There are tight tolerances for bidding--you want it to happen within a fraction of a second so the ad quickly loads for the user. Feeding purchase data is likely something that requires a bit of work, and I imagine the ad team at Amazon doesn't see enough value to prioritize it (especially since they make money either way--the advertisers pay for the ad with a higher CPC). One thing to keep in mind is that the purchase data is per-user, so it's different from the segment approach I was mentioning above. It would require a whole new schema* that is queried whenever bids are processed for an ad event. * Sorry for using the word schema here. It was used a lot on a team I was on in a similar context, and it bothered me at first but I got used to it. What I mean by schema is a "blob" with purchase data that is periodically processed by the bidding subsystem to add bid exclusions per user.
Normal_gaussian 7y ago
Its funny you should use that example. I recently bought a rice cooker, and then bought another a few weeks later. I've done similar things a lot, in this case it was because the first cooker had too large a minimum cooking amount. I'm currently looking at buying my mother google WiFi - which I bought a few weeks ago. And a few months back I rebought a tool I had just purchased because I lost it. I currently work for a (responsible) own site adtech (personalization) company. Its surprising what behaviours our clients find profitable for their business.
- MRD85 7y ago
  You said you work for adtech, I'd just like to comment that I personally like targetted ads. Areas of my life have improved from targetted ads, I'm shown things I didn't even know I want until I've seen them, etc. I look at it as a positive.
  - Normal_gaussian 7y ago
    Its an interesting one. I've made some pretty active choices about jobs for ethical reasons but ended up in a slight geographical bind in my last search. I found my current company through a friend (who works in another department) and was very skeptical. After not finding anything bad I decided to go with it for a few months, but had a resignation letter ready in my car. I'm now over a year in and despite the normal problems businesses have there is not a moral one here - and I've had pretty great insight into the decision making.
- dhimes 7y ago
  There's a whole lot about advertising I don't know, but the question that sticks in my mind is, did you need that ad to know which rice cooker to buy the second time? Did it change your decision? Did you see an ad for a different rice cooker and act on that? It seems like an effective ad would be this: Say you bought a rice cooker on Amazon for $75. Now Walmart shows you an ad for the same rice cooker for $60. That would certainly stick in my head- not for the rice cooker but for the price difference that Walmart can provide.

mattkrause 7y ago

> they don't really care if 9k of the ads get appreciably dumber. But maybe they should. The recipients of those 9,000 ads are being trained to ignore the recommendations because they’re so useless, and may continue to ignore its suggestions even after it improves. Likewise, a few clueless recommendations can “spoil” good ones shown at the same time. None of this shows up in the test set, of course, but people tend to turn off their brains when ‘evaluating’ ML stuff.

shawnz 7y ago
> The recipients of those 9,000 ads are being trained to ignore the recommendations ... Likewise, a few clueless recommendations can “spoil” good ones shown at the same time. How can you be sure that "random" untargeted ads don't have just as bad of an effect? Ultimately Amazon can see the numbers and we can't, and they made their decision based on that. It's pointless to argue over theories without data.

sizzle 7y ago

But couldn't they improve the signal to noise ratio by showing contextually relevant ads e.g. showing rice cooker cookbooks after purchasing a rice cooker? Wouldn't this lift ad click through rate? This is truly baffling expected behavior, who would ever buy a second rice cooker after having searched Amazon for rice cooker and compared a few top models already? I'd love to see some studies or data proving otherwise...