5 ms·

There's a lot of data here to digest, but one thing that stood out to me is: > Bot traffic: 41% of total So of all the traffic that CF sees, it attributes 41%

There's a lot of data here to digest, but one thing that stood out to me is: > Bot traffic: 41% of total So of all the traffic that CF sees, it attributes 41% of it to automated bots. This is way higher than I would have guessed!

chomp 6y ago
I used to work at a webhosting company, crawlers were an enormous amount of our traffic. Some of our users, their only traffic was crawler traffic.
- eminence32 6y ago
  Were these crawlers for big well-known things like google/bing/archive.org? I'm wondering if there are other types of crawlers that I don't even know about.
  - oefrha 6y ago
    Traders crawl for financial information. Businesses (e.g. airlines) crawl for competitor pricing. Etc. etc.
  - drcongo 6y ago
    A site for one of my clients was, at one point, getting absolutely hammered by a bot that crawls for plagiarised content. It was ignoring our robots.txt so I ended up blocking it via CloudFlare. There are bots for pretty much anything you can think of out there. edit: replaced "scrapes" with "crawls" for accuracy.
- thinkloop 6y ago
  Why do bots need to crawl so much? Any idea what kind of bots they are, is it mainly search engines?
  - heimatau 6y ago
    > Why do bots need to crawl so much? The Earth has a (mostly) free society. Therefore, many programmers find a web crawler to be an advantageous endeavor. From search engines, to social media companies, to ad networks, to government agencies, to financial institutions, to students learning about programming, to nefarious actors, and even shadow versions of everything above. Bots are neat. One can do a lot with a freely accessible public database of information, only limit is one's creativity. (edited: for formatting)
    - pilsetnieks 6y ago
      > only limit is one's creativity On the consumer side. There is a hard limit on bandwidth, though, on the provider side that can be exhausted by an overwhelming amount of bot traffic. In effect, that's what a DDoS is, even if unintentional.
blakesterz 6y ago
That's actually lower that I would have guessed. And lower than what I see on my servers. It's more than 1/2 for sure, and it's hard to guess how much of that other (less than half) part is real people, and not just bots pretending to be people. My 1/2 educated guess is about 40% of traffic to the sites I run is real people on any given day. These are all smallish websites for non-profits, so maybe not an "average" website, whatever that might mean.
- sgc 6y ago
  Filter by your country. The US for example has 57% bot activity.
  - tiffanyh 6y ago
    How did you filter by country?
    - sgc 6y ago
      There is a search box at top of center graph.
  - blakesterz 6y ago
    Aha! That sounds about right. Interesting, I didn't notice you can filter!
BuildTheRobots 6y ago
Really interesting numbers. I just wish it was possible to see all the other statistics in relation to letgit vs bot traffic. eg if mobile traffic is 35% of total, what percentage of legit vs bot traffic is mobile? TLS 1.2 vs 1.3 is a 38%/61% split, but how much bot traffic is encrypted?
- jgrahamc 6y ago
  This is just the beginning. We are going to be building out more and more functionality. My hope is that we deploy jupyter and let people crunch numbers in the ways they choose.
yjftsjthsd-h 6y ago
> So of all the traffic that CF sees, it attributes 41% of it to automated bots. This is way higher than I would have guessed! Bear in mind that that's what CF attributes to bots. As a human who gets flagged as a bot sometimes, I'm a little skeptical of their numbers.
- maxk42 6y ago
  Around ten years ago I built the analytics system for an Alexa 100 company. The company had a policy of supporting IE6 until its share of traffic was below 1% and it was sitting at around 4% at the time IIRC. This was the figure that was reported industry-wide and this was also what we were seeing internally in our stats. However, partway through the project we decided to switch from an image pixel to javascript tracking. At that point we saw the share of IE6 traffic drop immediately to below 1% and it dawned on me what was happening: A lot of bot creators had been using an IE6 user agent to disguise their activity. This was in the days before headless browsing so the vast majority of them weren't downloading / processing javascript. I thought 4% was high then for bot activity but it wouldn't surprise me if the share has exploded since then.
  - yjftsjthsd-h 6y ago
    Oh, I don't doubt that bot activity is rampant. The question is what Cloudflare's false-positive and false-negative rates are; I know from personal experience that they sometimes are convinced that humans are bots, and I assume sometimes bots get by them and therefore get counted as humans. But between those, my personal conclusion is that CF's claims about bot activity should be read with error bars.
  - vengefulduck 6y ago
    Also keep in mind it was probably a subset of bots using that user agent, so the total bot traffic could have been higher than 4%.
  - eli 6y ago
    Yup. Now imagine how many bot frameworks are smart enough to keep up to date with user agent strings. Pro-tip in Google Analytics you can filter by some device color depth = 0 bits and find (what I assume) are all bots that way.
tiffanyh 6y ago
I wonder if there is a correlation between Cloudflare customers and internet properties who'd be targeted by bots.
llacb47 6y ago
They probably flag a lot of Tor users as bots.
CoffeeDregs 6y ago
Hadn't noticed that but that's a good point. "Way higher": I would have guessed the same as you but, upon further thought, I guess I'm not surprised: creating a crawler is 100 lines of Python so I would guess that there are actually tens of thousands of bots running around grabbing web pages. Humans are an ever diminishing portion of the "web", especially given the (old) focus on "semantic web" and machine-friendly-ish formats...
cortesoft 6y ago
A single bot can generate thousands of times more traffic than a person. Even a small number of bots could easily generate a huge percentage of traffic.

chomp 6y ago

I used to work at a webhosting company, crawlers were an enormous amount of our traffic. Some of our users, their only traffic was crawler traffic.

eminence32 6y ago
Were these crawlers for big well-known things like google/bing/archive.org? I'm wondering if there are other types of crawlers that I don't even know about.
- oefrha 6y ago
  Traders crawl for financial information. Businesses (e.g. airlines) crawl for competitor pricing. Etc. etc.
- drcongo 6y ago
  A site for one of my clients was, at one point, getting absolutely hammered by a bot that crawls for plagiarised content. It was ignoring our robots.txt so I ended up blocking it via CloudFlare. There are bots for pretty much anything you can think of out there. edit: replaced "scrapes" with "crawls" for accuracy.
thinkloop 6y ago
Why do bots need to crawl so much? Any idea what kind of bots they are, is it mainly search engines?
- heimatau 6y ago
  > Why do bots need to crawl so much? The Earth has a (mostly) free society. Therefore, many programmers find a web crawler to be an advantageous endeavor. From search engines, to social media companies, to ad networks, to government agencies, to financial institutions, to students learning about programming, to nefarious actors, and even shadow versions of everything above. Bots are neat. One can do a lot with a freely accessible public database of information, only limit is one's creativity. (edited: for formatting)
  - pilsetnieks 6y ago
    > only limit is one's creativity On the consumer side. There is a hard limit on bandwidth, though, on the provider side that can be exhausted by an overwhelming amount of bot traffic. In effect, that's what a DDoS is, even if unintentional.

blakesterz 6y ago

That's actually lower that I would have guessed. And lower than what I see on my servers. It's more than 1/2 for sure, and it's hard to guess how much of that other (less than half) part is real people, and not just bots pretending to be people. My 1/2 educated guess is about 40% of traffic to the sites I run is real people on any given day. These are all smallish websites for non-profits, so maybe not an "average" website, whatever that might mean.

sgc 6y ago
Filter by your country. The US for example has 57% bot activity.
- tiffanyh 6y ago
  How did you filter by country?
  - sgc 6y ago
    There is a search box at top of center graph.
- blakesterz 6y ago
  Aha! That sounds about right. Interesting, I didn't notice you can filter!

BuildTheRobots 6y ago

Really interesting numbers. I just wish it was possible to see all the other statistics in relation to letgit vs bot traffic. eg if mobile traffic is 35% of total, what percentage of legit vs bot traffic is mobile? TLS 1.2 vs 1.3 is a 38%/61% split, but how much bot traffic is encrypted?

jgrahamc 6y ago
This is just the beginning. We are going to be building out more and more functionality. My hope is that we deploy jupyter and let people crunch numbers in the ways they choose.

yjftsjthsd-h 6y ago

> So of all the traffic that CF sees, it attributes 41% of it to automated bots. This is way higher than I would have guessed! Bear in mind that that's what CF attributes to bots. As a human who gets flagged as a bot sometimes, I'm a little skeptical of their numbers.

maxk42 6y ago
Around ten years ago I built the analytics system for an Alexa 100 company. The company had a policy of supporting IE6 until its share of traffic was below 1% and it was sitting at around 4% at the time IIRC. This was the figure that was reported industry-wide and this was also what we were seeing internally in our stats. However, partway through the project we decided to switch from an image pixel to javascript tracking. At that point we saw the share of IE6 traffic drop immediately to below 1% and it dawned on me what was happening: A lot of bot creators had been using an IE6 user agent to disguise their activity. This was in the days before headless browsing so the vast majority of them weren't downloading / processing javascript. I thought 4% was high then for bot activity but it wouldn't surprise me if the share has exploded since then.
- yjftsjthsd-h 6y ago
  Oh, I don't doubt that bot activity is rampant. The question is what Cloudflare's false-positive and false-negative rates are; I know from personal experience that they sometimes are convinced that humans are bots, and I assume sometimes bots get by them and therefore get counted as humans. But between those, my personal conclusion is that CF's claims about bot activity should be read with error bars.
- vengefulduck 6y ago
  Also keep in mind it was probably a subset of bots using that user agent, so the total bot traffic could have been higher than 4%.
- eli 6y ago
  Yup. Now imagine how many bot frameworks are smart enough to keep up to date with user agent strings. Pro-tip in Google Analytics you can filter by some device color depth = 0 bits and find (what I assume) are all bots that way.

tiffanyh 6y ago

I wonder if there is a correlation between Cloudflare customers and internet properties who'd be targeted by bots.

llacb47 6y ago

They probably flag a lot of Tor users as bots.

CoffeeDregs 6y ago

Hadn't noticed that but that's a good point. "Way higher": I would have guessed the same as you but, upon further thought, I guess I'm not surprised: creating a crawler is 100 lines of Python so I would guess that there are actually tens of thousands of bots running around grabbing web pages. Humans are an ever diminishing portion of the "web", especially given the (old) focus on "semantic web" and machine-friendly-ish formats...

cortesoft 6y ago

A single bot can generate thousands of times more traffic than a person. Even a small number of bots could easily generate a huge percentage of traffic.