6 ms·

Hi HN. I run a marketing agency and fell down this rabbit hole after a client's analytics made no sense (50k visitors, 47 sales). I ended up building a simple s

Hi HN. I run a marketing agency and fell down this rabbit hole after a client's analytics made no sense (50k visitors, 47 sales). I ended up building a simple script to track user behavior and analyzed 200+ small e-commerce sites. The average was 73% bot traffic that standard analytics counts as real. The bots are getting creepily good at mimicking engagement. I wrote up my findings, including some of the bizarre patterns I saw and the off-the-record conversations I had with ad tech insiders. It seems like a massive, open secret that nobody wants to talk about because the whole system is propped up by it. I'm curious if other developers, founders, or marketers here have seen similar discrepancies in their own data.

PaulHoule 1y ago
“All clicks are click fraud” isn’t that far from the truth.
- SoftTalker 1y ago
  I've never intentionally clicked on an ad, ever. I've accidentally clicked a few, due to tricks or fat fingers.
  - jpadkins 1y ago
    Did you read the article? If so, you clicked on an ad. This blog post was an ad for datacops.
    - hombre_fatal 1y ago
      And if you've ever scrolled social media or used the internet at all, you've read and clicked ads. Only considering old school "Ads by google" banners as ads and then patting yourself on the back for never clicking them is pretty faint praise for your ad evasion skills.
    - SoftTalker 1y ago
      No, of course I've read longer form posts and then realized they were just there for product placement, or affilate links, etc. I don't click display ads or links to Amazon or YouTube listings, etc. If I want to buy something I try to go directly to the manufacture's site and search for it there. If I have to go to a third party sales platform I'll go there and search for the thing I want. I rarely click on YouTube recommedations, because they are more and more just AI slop. I subscribe to people whose content is interesting, and that's the vast majority of what I watch.
  - speedgoose 1y ago
    You never watched a YouTube video that was some kind of teleshopping show ?
  - PaulHoule 1y ago
    Ah... I remember back when Anandtech would run a monthly roundup of hard drives that would usually insist that you pay another $150-$200 for a hard drive because hypothetically the more esoteric consumer SKUs were 3db softer or consumed 0.2W less than the mass-produced least-cost enterprise Seagates. [1] They unquestioningly quoted the spec sheets and never did any tests whereas everyone knows you can't trust spec sheets when it comes to noise and power. The thing was that site kept reflowing the layout over and over again and I think the point was that they were hoping you were going to click on something you saw on the sidebar and then it would reflow and an ad would be there right at the second when you clicked and then... Ka-Ching! [1] Funny reversal that the enterprise product is mass market and the consumer product is overpriced if not gold-plated.
- Nasrudith 1y ago
  Yeah. Even in the extremely rare circumstance where if I actually want to see more about the product I specifically search it instead of clicking on it because everyone knows clicking on the ad is how you get infected, spied, on or scammed. And I think they already discarded 'views' long ago as too useless ironically.
yellow_lead 1y ago
This is so common, I find it strange to see a whole article framing it as a shocking phenomenon. Like, have you heard of PostHog?
- urbandw311er 1y ago
  What do you mean re Posthog?
rightbyte 1y ago
What do you think is the false negative ratio of the remaining 27%? A 0.1% total conversion rate would imply 1/270 visitors buying stuff and I am quite certain I buy stuff in maybe every 10th online store I visit or something.
criddell 1y ago
Does it really matter if it's all fraud? You track 47 sales over some period. What was the ad spend for that period? Combine that with previous data and that should be enough to figure out if it was a successful campaign or not. When a company puts up a billboard or an ad on the bus, they don't care if the ad is seen by dashcams and dogs. All that matters is impact on the bottom line.
- nemomarx 1y ago
  If you could pay for web ads in the same way you could pay for a billboard (flat rate per period of time) yeah that would help. if you pay per impression or view or click you have some other issues
  - giancarlostoro 1y ago
    I love when I see websites that do this. They are just rare enough, but I appreciate them for existing.
- processing 1y ago
  yes, the co is likely spending big money on retargeting to bring back a potential customer to sell what the bot clicked on to generate retargeting. it's fraud and costing the company money.
- boplicity 1y ago
  It makes optimizing your ads significantly harder. Imagine trying to understand traffic flow on a freeway when 99.9% of cars are just projected illusions. Not fun. Effective advertising depends on iterative testing, which is very hard if the signal to noise ratio is way off.
- paulcole 1y ago
  > Does it really matter if it's all fraud Uh, yes. If you get 47 sales on $10k in ad spend (pay per click) and $9900 of that $10k was fraudulent then you got 47 sales on $100 of ad spend. Imagine if you could stop those fraudulent clicks.
seviu 1y ago
I once worked for the yellow pages in Switzerland. Our paid clients had a dashboard which reported how many users visited their business entry. We at engineering decided to filter out bots. Figures fell dramatically by more than 50%. In less that a day business mandated us to remove the filter. Bots are real people after all
- PedroBatista 1y ago
  He idea of "reality" and how things work, even if the idea is "they work bad", is more important than any other argument or reality. That's true at personal level and even truer at any organizational level.
  - wholinator2 1y ago
    Well yes but seems business has decided truth is worth less than money. Or now that i think of it, everything is worth less than money. Even, money is the only thing worth anything. They don't care about people, pride, products, or truth.
    - deleted 1y ago
      [deleted]
    - nostrademons 1y ago
      I think the way it goes is more "What is true? It's that if I bury this truth, I will have more money." Interestingly, I think money is increasingly its own falsehood now. A lot of rich people are finding that they pay a lot to get what's basically a scam, like Sam Altman's swimming pool that leaked and ruined the rest of the house [1]. There's a reason that Billionaire's Bunker [2] is entering the cultural zeitgeist despite fairly terrible plotting, dialogue, and acting. [1] https://fortune.com/2024/07/17/sam-altman-infinity-pool-mansion-flooding-lawsuit/ https://fortune.com/2024/07/17/sam-altman-infinity-pool-mans... [2] https://www.netflix.com/title/81606699 https://www.netflix.com/title/81606699
- at-fates-hands 1y ago
  >> In less that a day business mandated us to remove the filter. Did something similar at a small company I was working at. The VP of marketing sat me down and told me to do the same thing. After the meeting, I was told by another dev that the VP was tying a monetary value to specific clicks and if I was filtering out the bots, it would make his data look bad and reduce the amount of potential revenue for the company he was touting. I think you can see how the bots were actually helping him promote how awesome a job he was doing with our web properties to the owners.
jklinger410 1y ago
> I ended up building a simple script to track user behavior and analyzed 200+ small e-commerce sites. Why not use Microsoft Clarity? > The bots are getting creepily good at mimicking engagement. You would think Google Analytics would help a lot with this, but they seem to not care.
- unbalancedevh 1y ago
  > You would think Google Analytics would help a lot with this, but they seem to not care. Not just not care, but don't they have incentive to report higher traffic?
  - pixl97 1y ago
    At this point Google is just committing fraud to justify high stock prices.
- hdseggbj 1y ago
  That's like saying why not just live on the dole? Answer: pride, curiosity, inspiration, creativity, a sense of ownership, integrity, control, vision I can't believe these concepts are so foreign to so many.
  - jklinger410 1y ago
    Okay, well I'm asking the author why they developed their own analytics tool rather than using an industry standard hoping there's some reason beyond motivational phrases. Like, asking if there's a real technical reason. But, go off I guess.
Theodores 1y ago
Yes, and a lot of the pain is self-inflicted. Too often I have been working on ecommerce gigs where the traffic comes from internal 'SEO' tools. Normally the person in charge is marketing, not technical, so it has always been difficult to get past identifying the problem to fixing it. Often ecommerce companies are very siloed, so one person in one part of the 'team' is scraping the site with their special tools, only for another person to be doing another scrape with their special tools. You can have the guy doing the newsletter doing his thing, the guy doing organic search doing their thing, the guy doing paid ads doing their thing and someone in sales doing their own thing. The sad thing is that they are typically just a few SQL joins away from exactly the data they want in a format they can digest. However, due to silo-ing, it can be hard to have that conversation. On top of that, you do get new bots that need to be dealt with. The Huawei bot will scrape everything yet the store might not be delivering to China. So there are legit bots not doing ad fraud that need to be dealt with. What I also find interesting is that nobody is interested in the server logs. They come for free, and, although not having CDN cache hits, they still record checkout transactions and any pages that don't have a freshly cached page. Ad fraud also goes on in companies. I worked for a very successful company once and we only measured sales and what was out of stock. We didn't need open rates for emails or click through rates, our main problem was selling too much, which was a nice problem to have. Note that if you sell too much then you aren't going to get it all out the door in a timely fashion, or you run out of big lorries to put the orders in. Since then I have not worked on a site that is as successful. Instead we have people getting praise for 'false metrics'. Anything an SEO person creates or a marketeer measures will always have some nonsense aspect to it. Or the accounting is mixed with brick and mortar sales even though free shipping and discounts have been given on each sale, with adwords used to get people through the 'door'. Since sales manager has to report to someone on the board, if the sales numbers aren't good, the nonsense stats can be used to obfuscate the facts. The board only ever care about profit, so I don't like the way this goes down with false metrics of nonsense. Another fundamental problem in ecommerce is a lack of basic salesmanship. If you work the shop floor doing specialist sales where you have to listen to the customer's needs, then you gain experience in the art of sales. You haven't got to be good at it, in fact it can be better to know your limitations, for me that is big ticket items where I don't have the product knowledge, however, I could always hand those sales over to a much more capable colleague. You don't win every sale, but, in retail, you can have some really good streaks where no customer leaves empty handed. Your conversion rate is going to be more like 90% in face to face sales if you have the right product at the right price, with customers that don't buy coming back the following day or week to splash the cash. In High Street retail there is no way you give customers 15%+ off just for stepping through the door. Yet this is table stakes in ecommerce, particularly for small to medium size shops. This instantly devalues the product. Often there will be chatbots for whatever reason, and I am sure the likes of Dell can get that right, but your typical small ecommerce site will fluff this up too, so any customer daring to use the chatbot will not get instant help from a sales person. So what to do? It depends on your product, however, the goal is to get customers for life, not to churn through them. To achieve this it comes down to product, price, availability, shipping times, customer service and incentives for the customer to advertise for you, with reviews, word of mouth and all that hard stuff that needs real human skill. Sometimes it will only be a one-off sale, for example, if someone is buying a mattress. But, even then, the customer service basics matter. What is also silly is how, with everything you buy, you will get adverts and incentives to buy what you have just purchased. I don't know why this is not considered career limiting, but nobody seems to have fixed this. Next time, I will find out what data everyone needs and have my own script to collect just that data, then have a live backup that can be used for all internal purposes such as report generation. There will be no mystery script containers downloading 150 scripts with cookies, outside of developer supervision. My hunch is that the only numbers that really matter are sales. As for why the sales aren't happening, so long as functionality is as it should be on all devices, then you have to dig deeper, to the knowledge level that only someone that has worked the showroom floor understands. Really, a website should be the sales person's knowledge condensed into HTML. Yet nobody asks the guy serving customers what they upsell or what product they recommend if a customer baulks at a given product. Instead we have mystery-meat AI scripts that manage these things. Thanks for the heads up on ad fraud, normally I am a long way from that due to internal scraping efforts being far more damaging to website performance, and, as a developer, page load times matter to me far more than how much was paid for the traffic.
cantrevealname 1y ago
> "Everyone knows. But if we filtered it all out properly, our revenue would drop 40% overnight, and investors would have a meltdown." The word "revenue" above doesn't sound right. Did you mean "ad impressions", "clicks", "visits", or something else? If the bot traffic is giving you 40% more revenue, well that's excellent news, right? If it's revenue, then keep doing it.
- Sander_Marechal 1y ago
  It's revenue for the ad platforms, not for the clients
KaiserPro 1y ago
But what I'm struggling to understand is where is the money coming from to run those bots? How does the advertising money get to them to make it worth their while to run bots at such a scale? I mean sure, I suspect that people in the advertisers are doing it, but surely thats massively risky. there must be a grey market for this kind of transaction?
- randunel 1y ago
  Scrape data and package+sell it, either on demand or pre scraped.
- burnte 1y ago
  > But what I'm struggling to understand is where is the money coming from to run those bots? I agree, I have a hard time understanding the motivation for this behavior. It's obviously happening, I just don't know how someone benefits from a bot pretending to browse the internet.
  - bombcar 1y ago
    Isn't the idea that you run google ads on your site, and then have your botnet click the ads on your site, and you get paid for the traffic?
    - KaiserPro 1y ago
      Aha! that makes much more sense. Yeah having something autobot your click throughs seems like a motivation to run bots. I was thinking about ad traffic from one of the meta/twitter/tiktok products and couldn't think what the motivation would be
    - burnte 1y ago
      It would but the fact it goes out and does it to other sites is weird. Why would someone else fraudulently click my ads? As an attack, I understand that, I've had that done to me. Just randomly seems odd. Maybe I'm just not creative enough to think of bad things to do with this. :D
larrymcp 1y ago
> I had one client spending $12,000 per month on Google Ads In Google Ads you can just turn off the option to run your ads on non-Google sites; I think it's called their Display Network. Just run your campaign only on Google's search pages. I'm surprised the article doesn't mention this rather common solution.
dang 1y ago
> The bots are getting creepily good at mimicking engagement I've nicked that sentence to use as the title above, since it's more neutral and representative of what the article is about.
dpoloncsak 1y ago
I work in e-commerce and completely believe this could be true. The 'Cart Abandonment Bot' is actually something we just had a meeting on today, trying to figure out what's going on here. With that said, what do you think the error is here? Could your script have had some false positives? Enough to move the dial?
- bombcar 1y ago
  Some sites won't give you the price or the shipping until you add to cart, but that shouldn't map to an automatic removal of item from cart (unless there IS a timer somewhere set to 4 minutes on the server side). Strange that the item cost $47 and the number of sales was 47.
- dwd 1y ago
  I would also be looking for card washing and see if they attempted to purchase and the card failed. Repeated purchase attempts, particularly on the cheapest product in the catalogue is a bad sign.
Terr_ 1y ago
> It seems like a massive, open secret that nobody wants to talk about because the whole system is propped up by it. I was never really a punk-rebel kid, but a certain part of me rooted in the optimism of the early-internet kinda wants to see ad-models crash and burn. Even advertising "working normally" always had a psychic odor of exploitation and deceit. Ex: "You absolutely need this product or else your peers will hate you."
- Terr_ 1y ago
  P.S.: Imagine a world with "AI" (open-source, user-controlled) which can instantly detect ads on your screen, and pass the bounding-boxes to another program which covers them up. You could even install them on some digital glasses for a kind of They Live sunglasses situation [0], but please never wear such things while driving... [0] https://www.youtube.com/watch?v=wqrrh86WDG0 https://www.youtube.com/watch?v=wqrrh86WDG0
cortesoft 1y ago
I find it really interesting when you describe “good bot” traffic. > During my investigation, a source from the e-commerce data industry provided a crucial piece of the puzzle. He explained that his former company was responsible for scraping 70 million retailer web pages every single day. This is a legitimate and massive source of automated traffic. > Why do they do this? For vital business intelligence. Major retailers like Amazon do not always notify vendors when they run out of stock. So, brands pay for data scraping services to monitor their own products. These "good bots" check inventory levels, see who is winning the "buy box," ensure product descriptions are correct, and track search result rankings. They even scrape from different locations and mobile device profiles to analyze what banner ads are being shown to different audiences. I think a lot of the players involved in this would say those are bad bots. Having your competitors scrape your site for data would probably be something most website owners wouldn’t like, but getting data about THEIR competitors would be something they WOULD like. All bot traffic is good from SOMEONE’S perspective, otherwise it wouldn’t be happening. Someone had to program the bot, someone has to be running it. They obviously think they are good bots. The people running AI scraping bots think they are good bots. Many content creators think those are bad bots. Price comparison sites scraping retailers think their bots are good. The sites being scraped often think they are bad bots. I just don’t think we can clearly separate good and bad bot traffic without specifying whose perspective we are talking about.
- harshreality 1y ago
  "Good bots" would be ones that fuel a healthy, competitive economy. Whether or not they're good from the perspective of a retailer, data about products and supply chains is important. "Bad bots" would be ones that primarily support grift. They siphon money from advertisers, or get other people to waste advertising $, or game product listing metrics so that the bots' owners products get better placement, or so that other manufacturers' products get worse placement. They're not creating any new value; they're removing value from the economy by causing bad allocation of resources.
  - cortesoft 1y ago
    I can see many bots falling into the two categories you describe, but I can also see many other bots that would not fit easily, and whose value is debatable. For example, bots that scrape content for AI training; is that good or bad for a healthy economy? AI can be productive, but is it ‘stealing’ other’s productivity, which could hurt in the long term if it causes decrease in future human production because of diminished rewards?
aorloff 1y ago
We are now at the point where plausibly some bots could be at the behest of an AI agent controlled by a human. This is not likely to distort the basic numbers in your story, but it makes the premise questionable If your script can correctly segregate bot traffic from human traffic, and website operators can conclude that bot traffic does not make purchases, then what -- you still don't want to blackhole that traffic right ?
username223 1y ago
> developers, founders, or marketers I'm none of those things, 10x or otherwise, and would be ashamed if I were. I do run a personal website, though, and most of the traffic is bots that ignore robots.txt. I wish those developers, founders, and marketers would stop doing that to me.
oconaros 1y ago
I do web analytics consulting. One of my first projects at a digital marketing agency in 2021 was investigating weird traffic patterns for a global logistics firm. The findings are summarized in this blog post[1]. Bot traffic has been an issue for years. It has given rise to a new cottage industry of ad fraud detection services, none of which I’ve found particularly valuable. It always comes down to “so what do we do about it?” and no one seems to know how to get bots to stop viewing or clicking on paid media placements. Consumers use Google search and are on FB, IG, TikTok, LinkedIn, etc., and there aren’t really competing ad networks with “fewer bots”. So they keep buying fake traffic knowing that a significant chunk of it is invalid. I don’t see anything changing unless big tech companies making billions in ad revenue have a large enough incentive to do so. At the moment they have plenty of incentives to keep things as they are. “Half of the money I spend on advertising is wasted; the trouble is I don’t know which half.” - John Wanamaker [1] https://jimalytics.com/industry/youre-buying-bots-an-inconvenient-truth/ https://jimalytics.com/industry/youre-buying-bots-an-inconve...
- joomla199 1y ago
  Given that Wanamaker died in 1922 it’s safe to say his quote was in the context of a different problem entirely (which still exists, on top of the bot issue). Maybe it’s time to update to “three fourths”?
- jimkleiber 1y ago
  I'd suggest legislation. It's like when hotels or rental car companies tack on additional fees and consumers hate it but don't have much incentive to cooperate to change it. Consumers don't unionize much, but maybe we should? But maybe not legislation, because it might upset a ton of people. Any time a social media site goes through and seems to get rid of bots, people complain about how many "followers" they've lost. The fake user/clicker situation is quite pervasive, and a lot of parties, not just the advertising networks, benefit from the inflated numbers. But it can be so insidious..."wow, that video got 1M YouTube views? Must be very popular!" Orrrr a lot of those views were from bots? Who knows? So maybe a better approach than legislation is to talk about how the ad/bot fraud can both help us and hurt us and not demonize one side but see how we all may be implicated in it somehow. Maybe that will help us to be more aware of the problem and not fight against people, but try to work together to solve it.
reconnecting 1y ago
We've created open-source security analytics [1] that some organizations use for click fraud detection. The numbers that Google Ads shows as invalid clicks is dramatically different from what our platform detects. In our cases, most fraudulent clicks come from mobile devices and mobile networks. Also, bots don't load page resources in full, making them easier to detect. If your organization is struggling with fraud click detection, don't hesitate to reach out by email. We are ready to help. 1. https://github.com/TirrenoTechnologies/tirreno https://github.com/TirrenoTechnologies/tirreno

PaulHoule 1y ago

“All clicks are click fraud” isn’t that far from the truth.

SoftTalker 1y ago
I've never intentionally clicked on an ad, ever. I've accidentally clicked a few, due to tricks or fat fingers.
- jpadkins 1y ago
  Did you read the article? If so, you clicked on an ad. This blog post was an ad for datacops.
  - hombre_fatal 1y ago
    And if you've ever scrolled social media or used the internet at all, you've read and clicked ads. Only considering old school "Ads by google" banners as ads and then patting yourself on the back for never clicking them is pretty faint praise for your ad evasion skills.
  - SoftTalker 1y ago
    No, of course I've read longer form posts and then realized they were just there for product placement, or affilate links, etc. I don't click display ads or links to Amazon or YouTube listings, etc. If I want to buy something I try to go directly to the manufacture's site and search for it there. If I have to go to a third party sales platform I'll go there and search for the thing I want. I rarely click on YouTube recommedations, because they are more and more just AI slop. I subscribe to people whose content is interesting, and that's the vast majority of what I watch.
- speedgoose 1y ago
  You never watched a YouTube video that was some kind of teleshopping show ?
- PaulHoule 1y ago
  Ah... I remember back when Anandtech would run a monthly roundup of hard drives that would usually insist that you pay another $150-$200 for a hard drive because hypothetically the more esoteric consumer SKUs were 3db softer or consumed 0.2W less than the mass-produced least-cost enterprise Seagates. [1] They unquestioningly quoted the spec sheets and never did any tests whereas everyone knows you can't trust spec sheets when it comes to noise and power. The thing was that site kept reflowing the layout over and over again and I think the point was that they were hoping you were going to click on something you saw on the sidebar and then it would reflow and an ad would be there right at the second when you clicked and then... Ka-Ching! [1] Funny reversal that the enterprise product is mass market and the consumer product is overpriced if not gold-plated.
Nasrudith 1y ago
Yeah. Even in the extremely rare circumstance where if I actually want to see more about the product I specifically search it instead of clicking on it because everyone knows clicking on the ad is how you get infected, spied, on or scammed. And I think they already discarded 'views' long ago as too useless ironically.

yellow_lead 1y ago

This is so common, I find it strange to see a whole article framing it as a shocking phenomenon. Like, have you heard of PostHog?

urbandw311er 1y ago
What do you mean re Posthog?

rightbyte 1y ago

What do you think is the false negative ratio of the remaining 27%? A 0.1% total conversion rate would imply 1/270 visitors buying stuff and I am quite certain I buy stuff in maybe every 10th online store I visit or something.

criddell 1y ago

Does it really matter if it's all fraud? You track 47 sales over some period. What was the ad spend for that period? Combine that with previous data and that should be enough to figure out if it was a successful campaign or not. When a company puts up a billboard or an ad on the bus, they don't care if the ad is seen by dashcams and dogs. All that matters is impact on the bottom line.

nemomarx 1y ago
If you could pay for web ads in the same way you could pay for a billboard (flat rate per period of time) yeah that would help. if you pay per impression or view or click you have some other issues
- giancarlostoro 1y ago
  I love when I see websites that do this. They are just rare enough, but I appreciate them for existing.
processing 1y ago
yes, the co is likely spending big money on retargeting to bring back a potential customer to sell what the bot clicked on to generate retargeting. it's fraud and costing the company money.
boplicity 1y ago
It makes optimizing your ads significantly harder. Imagine trying to understand traffic flow on a freeway when 99.9% of cars are just projected illusions. Not fun. Effective advertising depends on iterative testing, which is very hard if the signal to noise ratio is way off.
paulcole 1y ago
> Does it really matter if it's all fraud Uh, yes. If you get 47 sales on $10k in ad spend (pay per click) and $9900 of that $10k was fraudulent then you got 47 sales on $100 of ad spend. Imagine if you could stop those fraudulent clicks.

seviu 1y ago

I once worked for the yellow pages in Switzerland. Our paid clients had a dashboard which reported how many users visited their business entry. We at engineering decided to filter out bots. Figures fell dramatically by more than 50%. In less that a day business mandated us to remove the filter. Bots are real people after all

PedroBatista 1y ago
He idea of "reality" and how things work, even if the idea is "they work bad", is more important than any other argument or reality. That's true at personal level and even truer at any organizational level.
- wholinator2 1y ago
  Well yes but seems business has decided truth is worth less than money. Or now that i think of it, everything is worth less than money. Even, money is the only thing worth anything. They don't care about people, pride, products, or truth.
  - deleted 1y ago
    [deleted]
  - nostrademons 1y ago
    I think the way it goes is more "What is true? It's that if I bury this truth, I will have more money." Interestingly, I think money is increasingly its own falsehood now. A lot of rich people are finding that they pay a lot to get what's basically a scam, like Sam Altman's swimming pool that leaked and ruined the rest of the house [1]. There's a reason that Billionaire's Bunker [2] is entering the cultural zeitgeist despite fairly terrible plotting, dialogue, and acting. [1] https://fortune.com/2024/07/17/sam-altman-infinity-pool-mansion-flooding-lawsuit/ https://fortune.com/2024/07/17/sam-altman-infinity-pool-mans... [2] https://www.netflix.com/title/81606699 https://www.netflix.com/title/81606699
at-fates-hands 1y ago
>> In less that a day business mandated us to remove the filter. Did something similar at a small company I was working at. The VP of marketing sat me down and told me to do the same thing. After the meeting, I was told by another dev that the VP was tying a monetary value to specific clicks and if I was filtering out the bots, it would make his data look bad and reduce the amount of potential revenue for the company he was touting. I think you can see how the bots were actually helping him promote how awesome a job he was doing with our web properties to the owners.

jklinger410 1y ago

> I ended up building a simple script to track user behavior and analyzed 200+ small e-commerce sites. Why not use Microsoft Clarity? > The bots are getting creepily good at mimicking engagement. You would think Google Analytics would help a lot with this, but they seem to not care.

unbalancedevh 1y ago
> You would think Google Analytics would help a lot with this, but they seem to not care. Not just not care, but don't they have incentive to report higher traffic?
- pixl97 1y ago
  At this point Google is just committing fraud to justify high stock prices.
hdseggbj 1y ago
That's like saying why not just live on the dole? Answer: pride, curiosity, inspiration, creativity, a sense of ownership, integrity, control, vision I can't believe these concepts are so foreign to so many.
- jklinger410 1y ago
  Okay, well I'm asking the author why they developed their own analytics tool rather than using an industry standard hoping there's some reason beyond motivational phrases. Like, asking if there's a real technical reason. But, go off I guess.

Theodores 1y ago

Yes, and a lot of the pain is self-inflicted. Too often I have been working on ecommerce gigs where the traffic comes from internal 'SEO' tools. Normally the person in charge is marketing, not technical, so it has always been difficult to get past identifying the problem to fixing it. Often ecommerce companies are very siloed, so one person in one part of the 'team' is scraping the site with their special tools, only for another person to be doing another scrape with their special tools. You can have the guy doing the newsletter doing his thing, the guy doing organic search doing their thing, the guy doing paid ads doing their thing and someone in sales doing their own thing. The sad thing is that they are typically just a few SQL joins away from exactly the data they want in a format they can digest. However, due to silo-ing, it can be hard to have that conversation. On top of that, you do get new bots that need to be dealt with. The Huawei bot will scrape everything yet the store might not be delivering to China. So there are legit bots not doing ad fraud that need to be dealt with. What I also find interesting is that nobody is interested in the server logs. They come for free, and, although not having CDN cache hits, they still record checkout transactions and any pages that don't have a freshly cached page. Ad fraud also goes on in companies. I worked for a very successful company once and we only measured sales and what was out of stock. We didn't need open rates for emails or click through rates, our main problem was selling too much, which was a nice problem to have. Note that if you sell too much then you aren't going to get it all out the door in a timely fashion, or you run out of big lorries to put the orders in. Since then I have not worked on a site that is as successful. Instead we have people getting praise for 'false metrics'. Anything an SEO person creates or a marketeer measures will always have some nonsense aspect to it. Or the accounting is mixed with brick and mortar sales even though free shipping and discounts have been given on each sale, with adwords used to get people through the 'door'. Since sales manager has to report to someone on the board, if the sales numbers aren't good, the nonsense stats can be used to obfuscate the facts. The board only ever care about profit, so I don't like the way this goes down with false metrics of nonsense. Another fundamental problem in ecommerce is a lack of basic salesmanship. If you work the shop floor doing specialist sales where you have to listen to the customer's needs, then you gain experience in the art of sales. You haven't got to be good at it, in fact it can be better to know your limitations, for me that is big ticket items where I don't have the product knowledge, however, I could always hand those sales over to a much more capable colleague. You don't win every sale, but, in retail, you can have some really good streaks where no customer leaves empty handed. Your conversion rate is going to be more like 90% in face to face sales if you have the right product at the right price, with customers that don't buy coming back the following day or week to splash the cash. In High Street retail there is no way you give customers 15%+ off just for stepping through the door. Yet this is table stakes in ecommerce, particularly for small to medium size shops. This instantly devalues the product. Often there will be chatbots for whatever reason, and I am sure the likes of Dell can get that right, but your typical small ecommerce site will fluff this up too, so any customer daring to use the chatbot will not get instant help from a sales person. So what to do? It depends on your product, however, the goal is to get customers for life, not to churn through them. To achieve this it comes down to product, price, availability, shipping times, customer service and incentives for the customer to advertise for you, with reviews, word of mouth and all that hard stuff that needs real human skill. Sometimes it will only be a one-off sale, for example, if someone is buying a mattress. But, even then, the customer service basics matter. What is also silly is how, with everything you buy, you will get adverts and incentives to buy what you have just purchased. I don't know why this is not considered career limiting, but nobody seems to have fixed this. Next time, I will find out what data everyone needs and have my own script to collect just that data, then have a live backup that can be used for all internal purposes such as report generation. There will be no mystery script containers downloading 150 scripts with cookies, outside of developer supervision. My hunch is that the only numbers that really matter are sales. As for why the sales aren't happening, so long as functionality is as it should be on all devices, then you have to dig deeper, to the knowledge level that only someone that has worked the showroom floor understands. Really, a website should be the sales person's knowledge condensed into HTML. Yet nobody asks the guy serving customers what they upsell or what product they recommend if a customer baulks at a given product. Instead we have mystery-meat AI scripts that manage these things. Thanks for the heads up on ad fraud, normally I am a long way from that due to internal scraping efforts being far more damaging to website performance, and, as a developer, page load times matter to me far more than how much was paid for the traffic.

cantrevealname 1y ago

> "Everyone knows. But if we filtered it all out properly, our revenue would drop 40% overnight, and investors would have a meltdown." The word "revenue" above doesn't sound right. Did you mean "ad impressions", "clicks", "visits", or something else? If the bot traffic is giving you 40% more revenue, well that's excellent news, right? If it's revenue, then keep doing it.

Sander_Marechal 1y ago
It's revenue for the ad platforms, not for the clients

KaiserPro 1y ago

But what I'm struggling to understand is where is the money coming from to run those bots? How does the advertising money get to them to make it worth their while to run bots at such a scale? I mean sure, I suspect that people in the advertisers are doing it, but surely thats massively risky. there must be a grey market for this kind of transaction?

randunel 1y ago
Scrape data and package+sell it, either on demand or pre scraped.
burnte 1y ago
> But what I'm struggling to understand is where is the money coming from to run those bots? I agree, I have a hard time understanding the motivation for this behavior. It's obviously happening, I just don't know how someone benefits from a bot pretending to browse the internet.
- bombcar 1y ago
  Isn't the idea that you run google ads on your site, and then have your botnet click the ads on your site, and you get paid for the traffic?
  - KaiserPro 1y ago
    Aha! that makes much more sense. Yeah having something autobot your click throughs seems like a motivation to run bots. I was thinking about ad traffic from one of the meta/twitter/tiktok products and couldn't think what the motivation would be
  - burnte 1y ago
    It would but the fact it goes out and does it to other sites is weird. Why would someone else fraudulently click my ads? As an attack, I understand that, I've had that done to me. Just randomly seems odd. Maybe I'm just not creative enough to think of bad things to do with this. :D

larrymcp 1y ago

> I had one client spending $12,000 per month on Google Ads In Google Ads you can just turn off the option to run your ads on non-Google sites; I think it's called their Display Network. Just run your campaign only on Google's search pages. I'm surprised the article doesn't mention this rather common solution.

dang 1y ago

> The bots are getting creepily good at mimicking engagement I've nicked that sentence to use as the title above, since it's more neutral and representative of what the article is about.

dpoloncsak 1y ago

I work in e-commerce and completely believe this could be true. The 'Cart Abandonment Bot' is actually something we just had a meeting on today, trying to figure out what's going on here. With that said, what do you think the error is here? Could your script have had some false positives? Enough to move the dial?

bombcar 1y ago
Some sites won't give you the price or the shipping until you add to cart, but that shouldn't map to an automatic removal of item from cart (unless there IS a timer somewhere set to 4 minutes on the server side). Strange that the item cost $47 and the number of sales was 47.
dwd 1y ago
I would also be looking for card washing and see if they attempted to purchase and the card failed. Repeated purchase attempts, particularly on the cheapest product in the catalogue is a bad sign.

Terr_ 1y ago

> It seems like a massive, open secret that nobody wants to talk about because the whole system is propped up by it. I was never really a punk-rebel kid, but a certain part of me rooted in the optimism of the early-internet kinda wants to see ad-models crash and burn. Even advertising "working normally" always had a psychic odor of exploitation and deceit. Ex: "You absolutely need this product or else your peers will hate you."

Terr_ 1y ago
P.S.: Imagine a world with "AI" (open-source, user-controlled) which can instantly detect ads on your screen, and pass the bounding-boxes to another program which covers them up. You could even install them on some digital glasses for a kind of They Live sunglasses situation [0], but please never wear such things while driving... [0] https://www.youtube.com/watch?v=wqrrh86WDG0 https://www.youtube.com/watch?v=wqrrh86WDG0

cortesoft 1y ago

I find it really interesting when you describe “good bot” traffic. > During my investigation, a source from the e-commerce data industry provided a crucial piece of the puzzle. He explained that his former company was responsible for scraping 70 million retailer web pages every single day. This is a legitimate and massive source of automated traffic. > Why do they do this? For vital business intelligence. Major retailers like Amazon do not always notify vendors when they run out of stock. So, brands pay for data scraping services to monitor their own products. These "good bots" check inventory levels, see who is winning the "buy box," ensure product descriptions are correct, and track search result rankings. They even scrape from different locations and mobile device profiles to analyze what banner ads are being shown to different audiences. I think a lot of the players involved in this would say those are bad bots. Having your competitors scrape your site for data would probably be something most website owners wouldn’t like, but getting data about THEIR competitors would be something they WOULD like. All bot traffic is good from SOMEONE’S perspective, otherwise it wouldn’t be happening. Someone had to program the bot, someone has to be running it. They obviously think they are good bots. The people running AI scraping bots think they are good bots. Many content creators think those are bad bots. Price comparison sites scraping retailers think their bots are good. The sites being scraped often think they are bad bots. I just don’t think we can clearly separate good and bad bot traffic without specifying whose perspective we are talking about.

harshreality 1y ago
"Good bots" would be ones that fuel a healthy, competitive economy. Whether or not they're good from the perspective of a retailer, data about products and supply chains is important. "Bad bots" would be ones that primarily support grift. They siphon money from advertisers, or get other people to waste advertising $, or game product listing metrics so that the bots' owners products get better placement, or so that other manufacturers' products get worse placement. They're not creating any new value; they're removing value from the economy by causing bad allocation of resources.
- cortesoft 1y ago
  I can see many bots falling into the two categories you describe, but I can also see many other bots that would not fit easily, and whose value is debatable. For example, bots that scrape content for AI training; is that good or bad for a healthy economy? AI can be productive, but is it ‘stealing’ other’s productivity, which could hurt in the long term if it causes decrease in future human production because of diminished rewards?

aorloff 1y ago

We are now at the point where plausibly some bots could be at the behest of an AI agent controlled by a human. This is not likely to distort the basic numbers in your story, but it makes the premise questionable If your script can correctly segregate bot traffic from human traffic, and website operators can conclude that bot traffic does not make purchases, then what -- you still don't want to blackhole that traffic right ?

username223 1y ago

> developers, founders, or marketers I'm none of those things, 10x or otherwise, and would be ashamed if I were. I do run a personal website, though, and most of the traffic is bots that ignore robots.txt. I wish those developers, founders, and marketers would stop doing that to me.

oconaros 1y ago

I do web analytics consulting. One of my first projects at a digital marketing agency in 2021 was investigating weird traffic patterns for a global logistics firm. The findings are summarized in this blog post[1]. Bot traffic has been an issue for years. It has given rise to a new cottage industry of ad fraud detection services, none of which I’ve found particularly valuable. It always comes down to “so what do we do about it?” and no one seems to know how to get bots to stop viewing or clicking on paid media placements. Consumers use Google search and are on FB, IG, TikTok, LinkedIn, etc., and there aren’t really competing ad networks with “fewer bots”. So they keep buying fake traffic knowing that a significant chunk of it is invalid. I don’t see anything changing unless big tech companies making billions in ad revenue have a large enough incentive to do so. At the moment they have plenty of incentives to keep things as they are. “Half of the money I spend on advertising is wasted; the trouble is I don’t know which half.” - John Wanamaker [1] https://jimalytics.com/industry/youre-buying-bots-an-inconvenient-truth/ https://jimalytics.com/industry/youre-buying-bots-an-inconve...

joomla199 1y ago
Given that Wanamaker died in 1922 it’s safe to say his quote was in the context of a different problem entirely (which still exists, on top of the bot issue). Maybe it’s time to update to “three fourths”?
jimkleiber 1y ago
I'd suggest legislation. It's like when hotels or rental car companies tack on additional fees and consumers hate it but don't have much incentive to cooperate to change it. Consumers don't unionize much, but maybe we should? But maybe not legislation, because it might upset a ton of people. Any time a social media site goes through and seems to get rid of bots, people complain about how many "followers" they've lost. The fake user/clicker situation is quite pervasive, and a lot of parties, not just the advertising networks, benefit from the inflated numbers. But it can be so insidious..."wow, that video got 1M YouTube views? Must be very popular!" Orrrr a lot of those views were from bots? Who knows? So maybe a better approach than legislation is to talk about how the ad/bot fraud can both help us and hurt us and not demonize one side but see how we all may be implicated in it somehow. Maybe that will help us to be more aware of the problem and not fight against people, but try to work together to solve it.

reconnecting 1y ago

We've created open-source security analytics [1] that some organizations use for click fraud detection. The numbers that Google Ads shows as invalid clicks is dramatically different from what our platform detects. In our cases, most fraudulent clicks come from mobile devices and mobile networks. Also, bots don't load page resources in full, making them easier to detect. If your organization is struggling with fraud click detection, don't hesitate to reach out by email. We are ready to help. 1. https://github.com/TirrenoTechnologies/tirreno https://github.com/TirrenoTechnologies/tirreno