9 ms·
47% of all internet traffic came from bots in 2022?
- Kikawala 4y agoPDF report is here: https://www.imperva.com/resources/reports/2023-Imperva-Bad-Bot-Report.pdf https://www.imperva.com/resources/reports/2023-Imperva-Bad-B...
- whiddershins 4y agoWeb traffic maybe?
- intrasight 4y agoObviously 100% of internet traffic is bots since human brains don't (yet) directly connect to the Internet. As far as what percent of that traffic does eventually enter ears or eyes, there's really no way to tell.
- tomohelix 4y agoAnecdotally, I have drastically reduced my internet activity in recent years. So many of the websites are just noise with crappy information. The good answers are often found on reddit in one or two clicks or directly asked on discord so there is no need to spend hours with google anymore (the crappy algo doesn't help). I also refrained from posting or discussing anything on social media after some bad experience with the users there. It takes effort to make good quality posts, but it rarely goes anywhere in those massive social media sites and often it may even get punished if it is against the public opinion. So I am tired and only make a few posts every few days now on HN. I am sure while my activity has dropped, the bots are getting more active so nothing is lost. Maybe some quality and how the traffic share look like, but I don't know.
- ChatGTP 4y agoI think LLMs are useful because they’re effectively trained on Reddit. It’s for sure one of the most useful places to find good information and advice on the web.
- cookieperson 4y agoThat'll change once 90% of posts on Reddit are bot posts. It's already happening...
- echelon 4y agoCounter point: the whole internet can be distilled into an LLM and shared in a super condensed format. Both of these things will happen (old web getting spammed, old web being distilled and crystalized), and the future will be weird and unpredictable to us now.
- throwaway290 4y ago> the whole internet can be distilled into an LLM and shared in a super condensed format > the future will be weird and unpredictable to us now I'm going to play devil's advocate for those people who always drop by saying LLM is pretty much like a human and human is pretty much like an LLM anyway, and say it would be no different to now
- cookieperson 4y agoExcept bots aren't like humans at all. They have no life experience. It's basically a text interface to a dictionary. A company can pay 100000 bots to spam your favorite messaging board with half baked propaganda, hate and advertisements. A human doesn't have that bandwidth, motivation, or interest. An internet saturated by bots is like reading reviews on Amazon without pictures. Pointless, intentionally misleading, and often confidently wrong.
- throwaway290 4y agoYep. I don't know how enthusiasts manage to say human is the same as AI and also say that everything will certainly change for sure because of AI, they should pick one...
- tourmalinetaco 4y ago
- roqi 4y ago> So many of the websites are just noise with crappy information. What I hate the most about today's internet is how search engines allow blatant scrapers to feature so high in search results. So many times I Google for something to find Stack Overflow as the main search hits, and right next to it there are a couple of sites that copied Stack Overflow's questions verbatim. Once I googled for FLOSS projects I had on GitHub and lo and behold there were half a dozen obscure sites that also claim to host my project, with everything copied verbatim from git repo to project descriptions.
- zamadatix 4y agoIt's better than one thing at least: Internet scraping is the process of... you can already see where this is going. Sites with 6 pages of boilerplate that sounds like an 6th grader padded an essay around a 2 word answer they've scraped from somewhere else. Worst of the 2 words of content aren't even all that accurate most of the time. At least sites that copy the answer verbatim still give you the answer!
- tourmalinetaco 4y agoThe scary thing is those websites will only become more common (and factually incorrect) as LLMs become more and more accessible.
- argondonor 4y agoMy main gripe with Discord is the somewhat ephemeral nature of it, due to the search being horrible, as well as not publicly indexed nor easily accessible without an account + an invite to a specific server.
- tourmalinetaco 4y agoDiscord becoming the main replacement to subreddits/forums has been a harder blow to the collection of knowledge than most anything else. The nice thing is the Discord API doesn’t seem to ban you for archiving servers as long as you do so responsibly.
- sigstoat 4y agowhat is it with “security” people and these scare facts? it isn’t enough to point to real events, they’ve got to produce stuff that rivals the most hyperbolic gartner report ever.
- lionkor 4y ago> This website requires certain cookies to work and uses other cookies to help you have the best experience while on the site. > By visiting this website, certain cookies have already been set, which you may delete and block. If you do not agree to the use of cookies, you should not navigate this website. > Visit our privacy and cookie policy to learn more about the cookies we use and how we use your data. I'm pretty sure GDPR says nothing about cookies that are needed for the site to work, such as session cookies when you're logged in, or cookies to hold settings you set. Am I wrong? Declining on this form sends me to the / root page. Weird.
- bad_good_guy 4y agoironic that 'security' magazine requires breaching your privacy and tracking you for them to allow you to use the site
- UtopiaPunk 4y agoA not insignificant portion of my job has to deal with not traffic. It's such a pain.
- betaby 4y agoI work for very large multinational ISP operating in the majority of the OECD countries and the statement in the article is outright false.
- perlgeek 4y agoTo quote the actual report [1]: > Of all internet traffic in 2022, 47.4% was automated traffic, also commonly referred to as bots. [...] Of that automated traffic, 30.2% were bad bots, a 2.5% increase from 27.7% in 2021 This is a bit misleading, according to the accompanying pie chart, 30.2% of all traffic were bad bots, not 30% of the 47.4%. What is sorely lacking (from a quick skim of the PDF) is a detailed description of how the data was measured, what protocol it includes, what the error margins are etc. [1] https://www.imperva.com/resources/reports/2023-Imperva-Bad-Bot-Report.pdf https://www.imperva.com/resources/reports/2023-Imperva-Bad-B...
- Retric 4y agoIt’s worse than that as over half of all internet traffic is video streaming. Bots simply aren’t a significant fraction of Netflix, D+, HBO Max etc because there’s no point. 47% of website requests might be more accurate.
- perlgeek 4y agoYes, sounds likely. Also, I'd be surprised if less than 50% of SMTP requests were malicious in some way (spam, scams, phishing, ...).
- joseda-hg 4y agoI've recently dealt with a website being brought down by an ABSURD amount of search requests made to improve SEO of other sites They were ostensibly Chinese bots, and they overloaded the hosting, not the bandwith, just requests
- candiddevmike 4y agoDoes this mean 47% of all online ad spend is worthless? How would one prove that it isn't? I've never seen an ad service offer cost per human click or cost per thousand human impressions
- detaro 4y ago> Does this mean 47% of all online ad spend is fraudulent? No. Why would it?
- spondylosaurus 4y agoAutomated ad fraud is why. It's the biggest problem you've never heard of. The last time I heard it cited (which, to be fair, was a few years back) the actual number was 33%—for every three dollars spent on digital advertising, a dollar gets lost to fraud. Not too far off.
- speeder 4y agoI had to disable Google display ads because 90+% was fraud
- acfou 4y agohow did you know?
- phendrenad2 4y agoHow old is that information? If I recall correctly, some of the big ad fraud rings were taken to court, and that killed the business model.
- spondylosaurus 4y agoRoughly three years old I think. But if you're referring to the same ad fraud rings I'm thinking of: yeah, a few did get shut down (which iirc was a first) but there's plenty of new operations taking their place. Ad fraud is a relatively cheap way to make money if you can scale up your operation, and most perpetrators don't get caught or punished. Although that may change in the future.
- throwawaybot9 4y agoMy crawlers generate give or take 100 million requests a year and it costs me $100 per month.
- dna_polymerase 4y agoAnd what and why do you crawl?
- latchkey 4y agoI wonder what the corresponding relative power usage is.
- klntsky 4y agoThis is good. Humanity progresses and therefore more tasks become automated.
- marginalia_nu 4y agoAnecdotally, around 1% of the (attempted) queries I get toward my search engine are human. 2 million queries per day are confirmed bots, 20k are not.
- deleted 4y ago[deleted]
- doe88 4y agoThe obligatory https://en.wikipedia.org/wiki/Dead_Internet_theory https://en.wikipedia.org/wiki/Dead_Internet_theory
- buro9 4y agoI run some forums, some of them are quite large. Recently the big increase in scraping by the search engines (Bing has had the greatest increase) caused me to question why. It used to be that the cost of scraping came with the benefit of being search engine listed which drove traffic, but that feels less true than it used to (for a lot of reasons). But now the cost of scraping doesn't feel in the favour of a website. Scraping and bots are for search engines listing, technology tests / experiments, advert / audience measurements, brand protection, IP tracking, copyright enforcement, screenshots for links on other websites (i.e. Facebook), Pinterest linkbacks, training of LLMs (my hypothesis on Bing's massive increase), spam, etc, etc. With the search engine value lowered by less traffic, yet a solid community still growing via word of mouth... the rest of those things offer no value to me or the community. So I asked the community, what do you want to do here? Leave them all? Ban some? Ban all? Some midway thing? Almost unanimously the community (who fund the costs by donations, and at least 30% of all traffic and costs were known to be associated to bots) chose to block every bot. So that's what we've done. We've blocked every major hosting and Cloud ASN, or put a challenge up to the few known to be proxies (i.e. Google Data Saver), and we've blocked hundreds of bot user agents, we've blocked requests where no Accept header was present where it should be, we've blocked TLS ciphers that aren't modern web browsers — I looked at requests by Python, Go, Curl, Wget, etc... and blocked everything that obviously differed from a valid browser. In the end we blocked about 40% of our traffic, and so far not a single real human has said (and it's a tight-knit but large community with lots of ways of contacting me) that they've had any issue at all. We appear to have reduced our traffic and associated costs, with no loss to us at all.
- gleenn 4y agoI am definitely happy you asked your users what they thought and made your decision. But saying "no human complained" might not be a good metric if people use Google or whatever to discover your site or it's info. People don't complain about things they don't know exist.
- jacquesm 4y agoThat's fine though. People will discover the site because other people recommend it. Probably higher quality too.
- anonymouscaller 4y agoCloudflare (CDN with a much larger market share) claims that in the past four weeks, the percent of bots vs. humans is about ~29%. Almost 50% seems like a stretch. https://radar.cloudflare.com/?range=28d https://radar.cloudflare.com/?range=28d
- idoubtit 4y agoTLDR 30% of http requests are bad bots, 17% are okay bots, and 53% human. The PDF report isn't explicit, but since their assertion is "based on data collected from the company’s global network throughout 2022, which includes 6 trillion blocked bad bot requests", it means the "internet traffic" is measured in number of HTTP requests. As noted in other comments, results would have been very different with network bandwidth. BTW, I haven't checked if the search bots behaviour has recently changed, but I remember that most of them ignored the directives in robots.txt asking for a slower crawl. And I couldn't find a way to declare that the content almost never changed.
- nyc_pizzadev 4y agoThis is a bit confusing because isn’t most internet traffic video [0]? Streaming video requires something like 100x more sustained bandwidth than clicking around a website. So is this 47% of non video traffic or are bots consuming video? Something else? [0] https://blog.gitnux.com/internet-traffic-statistics/ https://blog.gitnux.com/internet-traffic-statistics/
- ulrischa 4y agoFor my Website a bot per se is not bad. Scraping per se is also not bad. I have some information and will be happy if a bot scrape it.
- tppiotrowski 4y agoGoogle Read Aloud [1] bot absolutely hammers my website because I update the url using pagehistory and it opens a new request for each url update. It also ignores the meta tag that's meant to disable it :( [1] https://developers.google.com/search/docs/crawling-indexing/read-aloud-user-agent https://developers.google.com/search/docs/crawling-indexing/...
- uninformed 4y agoWith the growth of AI, I wouldn't be surprised if it'll be 99% from bots one day.
- detaro 4y agosays vendor trying to sell anti-bot software to your managers. Let me guess, by "all internet traffic" they mean "HTTP requests going through our tool"?
- readyplayernull 4y agoWhile you are right, the amount of bot traffic is huge, from data scrappers, fake views (ie. YT video SEO, video ads, etc), to game miners and social media fake accounts which each scammer/spammer run by the thousands. That's not including the legal stuff like automated systems (web crawlers, media content generators, etc).
- superkuh 4y agoRight. I went to their site and I can't even access the actual "report" they're talking about. But it's clear that they've accidentally mixed up the words internet and web.
- gcoakes 4y agoCould you explain the difference?
- anonymouscaller 4y agoWeb is HTTP, so websites and videos. The internet is everything, including the web from SSH (remote server login), RDP (remote desktop login), to torrenting.
- hedora 4y agoSomebody has to click those ads! I’d rather it not be me.
- mikerg87 4y ago> One-in-five bad bots used Mobile Safari as their browser of choice in 2022, up from 16.1% in 2021. Being a neophyte. How does this work exactly? Is a JS payload that does something bad or an app using web view. Genuinely curious on this one
- detaro 4y agoI'm assuming what they mean is "pretended to be Mobile Safari" (which is just setting the HTTP header for what you are to say that you are Safari)
- sandworm101 4y agoI don't believe it. Unless someone is running bots that watch netflix/porn all day, there is no way that they are consuming half of all traffic. Half of all posts? Sure. Half of all webpage requests? Ok. But half of internet traffic?
- armchairhacker 4y agoI thought it would be more. A human doesn't visit anywhere near 10 different webpages a second, a bot can. A human isn't uploading and downloading data 24/7, a bot is. A human doesn't make a new post a minute, every minute, a bot does.
- sebzim4500 4y agoTrue but bots don't regularly stream hd videos from netflix/youtube.
- armchairhacker 4y agoMaybe to download and pirate the videos
- sebzim4500 4y agoSurely each group would only download a given video once though, compared to the millions of times they could be downloaded by legitimate users.
- pharrington 4y agoA bot can saturate text-based bandwidth as fast as a person saturates multimedia-based bandwidth.
- tacotacotaco 4y agoI have speculated for a while that bots “watch” videos on YouTube to appear human so google doesn't ban their accounts.
- 4y ago
- jdthedisciple 4y agoI assume this is counting api calls and crawlers, in which case it's actually kind of conceivable.