5 ms·
Bad bots account for most internet traffic? Analysis
- kredd 3y agoI believe it. Very insignificant data point but - I've worked for small scale startups and larger scale enterprises, and in both cases you can see bots and crawlers trying to do random stuff. I don't think it was actually %70+ of the traffic, but can see specific types of businesses getting even more hits from them.
- andai 3y agoWould this be solved by just providing APIs? Scraping is a pain and people only do it when there's no API. (Though these days a lot of businesses offer unofficial 3rd party APIs by scraping.) It's also stupid and massively wastes computation and traffic on both ends. Then people would think "but then I'd have to pay money to offer a free API", but they're already doing that in a more expensive way via the web interface anyway.
- gabipurcaru 3y ago> people only do it when there's no API Legitimate scrapers, maybe. Everyone else does it to circumvent the API limitations, by posing as real traffic. APIs imply API keys which can be traced and banned.
- yjftsjthsd-h 3y agoAnd what makes a scraper legitimate?
- ben0x539 3y agoWell, that's what happens when you make the API more limited than what people can do with scraping (or with using the public "our own web client" API key).
- thejohnconway 3y agoNo, there’s a huge number of bots out there just doing stupid hacky stuff like trying for wp/admin/. Vast numbers of what I’d characterise as script-kiddy type attempts. Looking at the raw logs of a webserver not behind something like CloudFlare really opened my eyes to this. There’s an extraordinary number of garbage requests.
- pixl97 3y agoYea, there was an HN thread complaining about WAF's a few days ago, but damn, if you're on the public internet having some kind of WAF cuts down the bullshit responses by a massive amount. Same thing goes for email. I had one particular domain I managed spam filtering on and for every 10,000 messages we received, 1 was delivered to users. The amount of crap requests on the internet is insane.
- kube-system 3y agoAnd that's just requests on 80/443. Hook up an IDS to the WAN interface and, you'll see thousands of attempts per day to exploit vulnerabilities across other services too. My low-traffic personal projects get 90%+ malicious traffic.
- gumballindie 3y agoI think a more efficient approach would be to prosecute those who do it - bandwidth costs money - and to implement mechanisms to deter them. A not insignificant number of ai workers think are entitled to using people’s bandwidth, servers and intellectual property for free. Until a couple of such people are taken to task this wont stop.
- JohnFen 3y ago> Would this be solved by just providing APIs? Scraping is a pain and people only do it when there's no API. Many sites intentionally make scraping as painful as possible because they don't want anyone scraping them.
- josefresco 3y agoI manage a network of several hundred websites. About 2 years ago we started blocking bot/crawler user-agents. Some websites see thousands of these requests every day, skewing analytics and eating bandwidth. Tools like this help: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blo...
- supriyo-biswas 3y agoYou're better off using a combination of IP blocklisting and ratelimiting, both of which use some sort of binary tree to speed up lookups, rather than using a mega-regex which your solution proposes, which, with tons of backtracking, is very likely slower than serving the request would have been. Not to mention that user agents are easily spoofable.
- kube-system 3y agoIf a bot's user agent doesn't look identical to a normal user, either the bot developer is dumb, or they are relatively well behaving. Anyone malicious who knows what they're doing will just send you a statistically common user agent string. Under most http libraries this is a one-liner: headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0'}
- _lvbh 3y agoTry JA3 fingerprinting. It’s much more difficult to spoof than user agents
- nwoli 3y agoGood bots account for the remaining 26%
- sandworm101 3y agoI do not believe. I do not believe that bots account for more traffic than all the streaming services, all the filesharing and all the porn crisscrossing the internet. I would believe 73% of requests, even 73% of page loads/scrapes, but 73% of all traffic is too huge a number.
- rightbyte 3y agoBots might "watch" Youtube for ad money for the creators and Google, though. I mean, it would not surprise me at all if bot traffic on Youtube is larger as in bytes/s of video, than actual visitors. Remember that Google has no interest in disclosing how many bot views there are. There was this news article about Spotify banning some musicians accounts due to listen farming. https://www.musicbusinessworldwide.com/great-big-spotify-scam-bulgarian-playlister-swindle-way-fortune-streaming-service/ https://www.musicbusinessworldwide.com/great-big-spotify-sca... Since the payouts are not tied to the individual watcher or listener, scams run rampant. I guess the temptation to manipulate the money flow is way higher than having a "fair" distribution of each subscribers money to the artists.
- kuroguro 3y agoMost bots operate in the gray area of just getting the view count up and not messing with the ads. They can operate publicly and not have law enforcement raid their offices. Yt definitely makes an effort to prevent that. You can read up on how botguard's virtual machine works in some older writeups.
- qingcharles 3y agoOn the more underground forums there are threads about using bots to watch your Twitch feeds to get revenue.
- adocomplete 3y agoI think it's possible if for nothing else but the fact that a bot can easily do 10-100x or more visits than a human can on the web.
- deleted 3y ago
- blakesterz 3y agoHere's the link to the PDF of the report: https://www.arkoselabs.com/wp-content/uploads/Breaking-Bad-Bots-Bot-Abuse-Analysis-and-Other-Fraud-Benchmarks.pdf https://www.arkoselabs.com/wp-content/uploads/Breaking-Bad-B...
- yjftsjthsd-h 3y agoI feel like this is trying very hard to push a particular narrative but not actually bothering to substantiate anything. From the top... > Bad Bots are increasing dramatically — Arkose estimates that 73% of all internet traffic currently (Q3, 2023) comprises Bad Bots and related fraud farm traffic. Note that "Bad Bots" are never defined, and that 73% will never be explained, either to say how they mean it (73% of connections is plausible, 73% of packets is not, not in a world with Netflix and YouTube), or how they would measure such a thing. Then we get into little gems like > Scraping, it must be said, is a legally murky area. It is not specifically illegal; but if it defies a website’s published terms of use, it is certainly immoral. Which... that's a claim you can make, but if you just throw out terms like "certainly immoral" without bothering to substantiate them I get to laugh at you. The tendency to cast aspersions without substantiating them gets substantially more pronounced later: > This is a website you can use to make sure your bots aren’t getting prevented by a website,” [...] “You can purchase this software. It has enterprise support and so on. But it is purpose built to commit crime. That is what it does. And there are many other different websites like this, but they look like legitimate businesses. It is a good example of a product purpose built to commit fraud.” I'll grant that businesses don't like people bypassing their protections, but that doesn't make it automatically illegal or fraud to scrape or even to programmatically interact with sites. I'm not naive enough to think that it's not being used for illegal stuff, but that doesn't mean you get to paint the tools with a single brush.
- JohnFen 3y ago> Note that "Bad Bots" are never defined This stuck out to me as well. What are "bad bots" and how do they differ from "good bots", and how are the authors determining which is which?
- PartiallyTyped 3y ago> world with Netflix and YouTube Not in a world where pron exists.
- yjftsjthsd-h 3y agoOkay, fine, a world where video streaming exists, in whatever particulars.
- actondev 3y agoAnd this is why we can't have nice things. Now that we will/are starting to have AI-created content, and visits from bots, comments from AI bots.. we need some new metric I reckon
- klyrs 3y agoI propose: time spent offline.
- sonicanatidae 3y agoWhat is this, "offline", of which you speak?!?! You mean like wifi?
- kibwen 3y agoI think "offline" means mocking your ethernet driver to produce packets confabulated by a locally-running copy of ChatGPT, so you can doomscroll the internet even without needing to pay Comcast for the privilege.
- sonicanatidae 3y agoAhhh.. whew... for a microsecond, I was thinking they meant the dark ages.
- h2odragon 3y agoNaw, theres this great new game; "nonvirtual reality" its kinda expensive and dangerously addictive the resolution and refresh rate are amazing
- sonicanatidae 3y agoBut how do the packets get to the other computers? Are you trailing a cord on a spool?
- 3y ago
- defanor 3y agoCloudflare refused to serve me the linked page after I enabled JS and passed the captcha, "The action you just performed triggered the security solution". Each time it happens, I wonder whether website owners then look at the statistics, see the reported blocks, and are reinforced in their belief that Cloudflare does something useful. But in this case I also wonder whether that number was derived from such blocked requests as well; whether they count me as a bad bot or just a bad person.
- bshowg 3y agoFrom the PDF: We analyzed tens of billions of sessions worldwide across industries, [...] I think they mean: 73% of all commercial sessions are malicious. Very sensationalist, at the end of the day they are selling something
- jackconsidine 3y ago> The biggest increases in attacks from Q2 to Q3 are SMS toll fraud (up 2,141%), Toll Fraud is rampant lately. I can tell because people have attempted it on nearly every consumer-facing project I've worked on. Additionally, the ill-advised "text the app to your phone" gimmicks have disappeared (fraudsters have cleaned up on direct, public endpoints to send SMS). Shameless plug, but I've written a little about Toll Fraud and how to prevent it [0] [0] https://koptional.com/article/how-to-stop-twilio-toll-fraud/ https://koptional.com/article/how-to-stop-twilio-toll-fraud/
- charlieyu1 3y agoI would confess that I had written a bad bot in the past, fetching data every time I runs it multiple times a day when the data source only updates once or twice a week. I didn’t know better back then
- tripue 3y agoThe article mention sms tool fraud as the biggest increase.. We faced the same problem at Zenly and had to build our own anti-spam strategies to prevent it. We used multiple providers to improve our conversion rate and reduce cost. We are now building this as a service https://www.ding.live/ https://www.ding.live/ and are seeing huge improvements for our customers in term of cost savings and conversion rate.
- oriettaxx 3y agodid anybody test the AWS feature to limit bots in application load balancer? it costs a lot, and we are not sure if it is worth
- oriettaxx 3y agodoes it make sense to limit incoming connection to HTTP/2 to stop (at least many) bots?
- renegat0x0 3y agoIs web scraping immoral? Silly statement. Is Google immoral in checking people contents? Assertions like that are on the other hand always bad ;-) Producing a RSS feed and clean scrapable page is always a good thing to do to boost your page recognition and popularity. You data does not have to be scrapable, but you page should always be bot friendly.
- Beijinger 3y agoInteresting question: When Netflix was a cheap stock but it made up a significant part of internet traffic, I assumed it could be the next big thing. So how do you invest in "bad bots", whatever this is supposed to mean. Honestly, I don't trust the numbers.
- red_admiral 3y agoWasn't there a time where something like 95% of all e-mails were Viagra spam? If so then at 73% the bots still have some catching up to do.
- forgotmypw17 3y agoI help run some websites with many links. I estimate that more than 90%, possibly close to 99% of our traffic is crawler bots, even after blocking some of them based on behavior.