5 ms·

61.5% of Web Traffic Is Not Human

33 points by vellum 13y ago

grinich 13y ago
I'm actually surprised that 38.5% is human. That seems high to me.
- tedsanders 13y ago
  It could be that humans tend to use more bandwidth. I imagine plenty of humans watch Netflix, but very few web crawlers do. :)
  - solistice 13y ago
    They're excluding streaming and torrenting.
jgalt212 13y ago
Yet, advertisers pay for 100% (or very nearly that amount) of ads displayed--regardless of the client.
- PavlovsCat 13y ago
  Dunno. I assume quite a lot of bots don't execute Javascript and therefore don't get to "see" a lot of ads anyway.
  - jgalt212 13y ago
    that's good point. It would be nice to see someone who does not have a vested interest in under or over reporting the number of bots on the web take a good crack at this number. Per spider.io, bots are ripping off advertisers left and right. That being said, scared advertisers are the target market for their products. http://www.spider.io/blog/ http://www.spider.io/blog/
    - kristopolous 13y ago
      I've got no investment either way. My nginx log has 19,333,928 entries ... when I do grep -Ei '(bot|spider|crawl|aggregator)' on it I get 2,609,874. That means that bots who declare themselves make about 13.5% of requests to this site. The robots.txt has no restrictions. I don't know if that's helpful.
      - arn 13y ago
        That doesn't really tell us anything about whether or not they trigger javascript. Bots triggering javascript will inflate advertising and js-based analytics numbers, which would be more problematic. Bots that don't trigger js, however, are more a drain on bandwidth than anything else.
        ch4s3 13y ago
        I seem to remember from a job I used to have as an analyst that advertisers have a few ways to discern human and bot traffic. They don't pay for traffic that is known to be from bots. However there are clearly some bots that aren't accounted for this way, and some that are designed specifically to trigger ads.
- skelsey 13y ago
  I'm pretty sure that is built into the price.
  - jgalt212 13y ago
    That tidy solution only works if the fraud rate on the ads you place is less than or equal to the industry presumed/priced-in average. It seems that there are more than a few ad networks around these days that specialize in non-human traffic. As such, advertiser realized fraud rates can vary greatly from this average. My company right now won't purchase RTB ads that display outside of walled gardens so that we can 1. manually verify that the ads are showing. 2. Minmize the number of bots seeing the ads (presumably it's hard for bots to infiltrate Facebook or the WSJ than Buzzfeed).
guiambros 13y ago
...of all website traffic... The report [1] does not include any non-http traffic, which excludes all heavy content like torrent, Netflix, music streaming, etc. [1] http://www.incapsula.com/the-incapsula-blog/item/820-bot-traffic-report-2013 http://www.incapsula.com/the-incapsula-blog/item/820-bot-tra...
- swinglock 13y ago
  That's what "web traffic" means. It didn't say "Internet traffic".
ChrisNorstrom 13y ago
On my blogs I used to use a script that banned bad bots by adding their IP to a list on my .htaccess file. Until it crashed the banning script by adding over 5,000 IPs of bad bots in just 1 month. If my google analytics numbers were correct I'd say between 50% and 70% of my traffic were bots. So 61.5% sounds about right. I've pretty much stopped looking at "number of impressions" when purchasing advertising too. It's always untrue. A recent blog I was going to advertise on said it had 35,000 unique visitors a month. But yet just a few weeks ago they had a "free give away contest for a jewelry brand (no purchase necessary, international readers allowed, free to apply, no shipping costs)" and only 100 people applied. That pretty much said it right there. One more example: A site I was on allowed unregistered visitors to give their entire collection of corporate logos a rating between 1 and 10 stars. And they used radio buttons to vote for the numbers 1-10. As soon as you select a radio button it auto-submits and gives the logo that rating. EVERY SINGLE LOGO had a massive number of "1 star" ratings because all the bots would see a radio button, check it, and trigger the vote. All the logos on their site had a rating of 1.
- viraptor 13y ago
  > "free give away contest for a jewelry brand (no purchase necessary, international readers allowed, free to apply, no shipping costs)" The internet conditioned me to ignore such claims as too-good-to-be-true. I would expect loads of people to treat it like "you're the 1000000th visitor! you won!" and other "free stuff here (if you register and fill out hundreds of forms)" offers.
  - ChrisNorstrom 13y ago
    Na, it was a very personal blog and the owner does things like that all the time. The name of the jeweler was disclosed and everything. I get what you're saying though, but it was legit.
sparrish 13y ago
I know monitoring services like ours (NodePing) create a huge amount of traffic by just keeping watch over all those servers... in case someone wants to see that cat video. <grin>
omarforgotpwd 13y ago
100% of web traffic isn't human, but some of the software listens to humans more than others.
proksoup 13y ago
At least.
FrankenPC 13y ago
I remember during the heady days of SLIP and PPP dialups, my dad was trying to explain how the future of this new weird invention called the internet would have automated software robots that would perform automated tasks to help up out. I had no idea what he was talking about.
UNIXgod 13y ago
zombie? UDP? DDoZ botnet?
- ehsanu1 13y ago
  googlebot
  - STRiDEX 13y ago
    bingbot
onethumb 13y ago
I estimated >50% of our traffic was from search engine crawlers specifically (billions of requests per month) way back in 2010. It's only gotten worse since then. http://don.blogs.smugmug.com/2010/07/15/great-idea-google-should-open-their-index/ http://don.blogs.smugmug.com/2010/07/15/great-idea-google-sh...
moca 13y ago
The Internet access cost tends to be higher than advertising cost. You can just compare revenue of AT&T to Google. I pay about $600/year for cable internet, and my advertising value is much less than that. As long as bots pay for the internet access, I think it is just fine.
ds9 13y ago
I wondered how they distinguished bots. The only info I found on the study [1], on a quick search, says: "For the purpose of this report we observed 1.45 Billion bot visits, which occurred over a 90 day period. The data was collected from a group of 20,000 sites on Incapsula’s network" So, in the first place, not necessarily representative of the web generally. And it does not say how they detect bots. Of course some declare themselves [2]; otherwise I guess one would have to rely on captchas [3], which are known to be of limited reliability and subject to an arms race. 1. http://www.incapsula.com/the-incapsula-blog/item/820-bot-traffic-report-2013 http://www.incapsula.com/the-incapsula-blog/item/820-bot-tra... 2. kristopolous on this page, '(bot|spider|crawl|aggregator)' 3. which Incapsula does, as a condition of access to its summary of its previous similar research.
skylan_q 13y ago
61.5% of Web Traffic Is Not Human I, for one, am glad that human trafficking over the web is down to only 38.5%.
- devilshaircut 13y ago
  Thanks for the laugh. Also, I wonder what the definition for "traffic" is within the context of this research.
bkirwi 13y ago
It's interesting to me that the author links to an article about how coding is useless for journalists, and then spends the rest of the article talking about all the web-scraping scripts she's written or could easily write. For me, this seems like a success of the whole teach-everyone-to-code thing; the author's picked up a bit of simple scripting and it's made some things dramatically easier for her. It's certainly not professional-level, but that's normal; for example, many people think programmers should know how to write, but few think they should all be able to churn out high-caliber journalism. Is this a failure of setting expectations, or is that really where some people are setting the bar?

33 points by vellum 13y ago

grinich 13y ago

I'm actually surprised that 38.5% is human. That seems high to me.

tedsanders 13y ago
It could be that humans tend to use more bandwidth. I imagine plenty of humans watch Netflix, but very few web crawlers do. :)
- solistice 13y ago
  They're excluding streaming and torrenting.

jgalt212 13y ago

Yet, advertisers pay for 100% (or very nearly that amount) of ads displayed--regardless of the client.

PavlovsCat 13y ago
Dunno. I assume quite a lot of bots don't execute Javascript and therefore don't get to "see" a lot of ads anyway.
- jgalt212 13y ago
  that's good point. It would be nice to see someone who does not have a vested interest in under or over reporting the number of bots on the web take a good crack at this number. Per spider.io, bots are ripping off advertisers left and right. That being said, scared advertisers are the target market for their products. http://www.spider.io/blog/ http://www.spider.io/blog/
  - kristopolous 13y ago
    I've got no investment either way. My nginx log has 19,333,928 entries ... when I do grep -Ei '(bot|spider|crawl|aggregator)' on it I get 2,609,874. That means that bots who declare themselves make about 13.5% of requests to this site. The robots.txt has no restrictions. I don't know if that's helpful.
    - arn 13y ago
      That doesn't really tell us anything about whether or not they trigger javascript. Bots triggering javascript will inflate advertising and js-based analytics numbers, which would be more problematic. Bots that don't trigger js, however, are more a drain on bandwidth than anything else.
      - ch4s3 13y ago
        I seem to remember from a job I used to have as an analyst that advertisers have a few ways to discern human and bot traffic. They don't pay for traffic that is known to be from bots. However there are clearly some bots that aren't accounted for this way, and some that are designed specifically to trigger ads.
skelsey 13y ago
I'm pretty sure that is built into the price.
- jgalt212 13y ago
  That tidy solution only works if the fraud rate on the ads you place is less than or equal to the industry presumed/priced-in average. It seems that there are more than a few ad networks around these days that specialize in non-human traffic. As such, advertiser realized fraud rates can vary greatly from this average. My company right now won't purchase RTB ads that display outside of walled gardens so that we can 1. manually verify that the ads are showing. 2. Minmize the number of bots seeing the ads (presumably it's hard for bots to infiltrate Facebook or the WSJ than Buzzfeed).

guiambros 13y ago

...of all website traffic... The report [1] does not include any non-http traffic, which excludes all heavy content like torrent, Netflix, music streaming, etc. [1] http://www.incapsula.com/the-incapsula-blog/item/820-bot-traffic-report-2013 http://www.incapsula.com/the-incapsula-blog/item/820-bot-tra...

swinglock 13y ago
That's what "web traffic" means. It didn't say "Internet traffic".

ChrisNorstrom 13y ago

On my blogs I used to use a script that banned bad bots by adding their IP to a list on my .htaccess file. Until it crashed the banning script by adding over 5,000 IPs of bad bots in just 1 month. If my google analytics numbers were correct I'd say between 50% and 70% of my traffic were bots. So 61.5% sounds about right. I've pretty much stopped looking at "number of impressions" when purchasing advertising too. It's always untrue. A recent blog I was going to advertise on said it had 35,000 unique visitors a month. But yet just a few weeks ago they had a "free give away contest for a jewelry brand (no purchase necessary, international readers allowed, free to apply, no shipping costs)" and only 100 people applied. That pretty much said it right there. One more example: A site I was on allowed unregistered visitors to give their entire collection of corporate logos a rating between 1 and 10 stars. And they used radio buttons to vote for the numbers 1-10. As soon as you select a radio button it auto-submits and gives the logo that rating. EVERY SINGLE LOGO had a massive number of "1 star" ratings because all the bots would see a radio button, check it, and trigger the vote. All the logos on their site had a rating of 1.

viraptor 13y ago
> "free give away contest for a jewelry brand (no purchase necessary, international readers allowed, free to apply, no shipping costs)" The internet conditioned me to ignore such claims as too-good-to-be-true. I would expect loads of people to treat it like "you're the 1000000th visitor! you won!" and other "free stuff here (if you register and fill out hundreds of forms)" offers.
- ChrisNorstrom 13y ago
  Na, it was a very personal blog and the owner does things like that all the time. The name of the jeweler was disclosed and everything. I get what you're saying though, but it was legit.

sparrish 13y ago

I know monitoring services like ours (NodePing) create a huge amount of traffic by just keeping watch over all those servers... in case someone wants to see that cat video. <grin>

omarforgotpwd 13y ago

100% of web traffic isn't human, but some of the software listens to humans more than others.

proksoup 13y ago

At least.

FrankenPC 13y ago

I remember during the heady days of SLIP and PPP dialups, my dad was trying to explain how the future of this new weird invention called the internet would have automated software robots that would perform automated tasks to help up out. I had no idea what he was talking about.

UNIXgod 13y ago

zombie? UDP? DDoZ botnet?

ehsanu1 13y ago
googlebot
- STRiDEX 13y ago
  bingbot

onethumb 13y ago

I estimated >50% of our traffic was from search engine crawlers specifically (billions of requests per month) way back in 2010. It's only gotten worse since then. http://don.blogs.smugmug.com/2010/07/15/great-idea-google-should-open-their-index/ http://don.blogs.smugmug.com/2010/07/15/great-idea-google-sh...

moca 13y ago

The Internet access cost tends to be higher than advertising cost. You can just compare revenue of AT&T to Google. I pay about $600/year for cable internet, and my advertising value is much less than that. As long as bots pay for the internet access, I think it is just fine.

ds9 13y ago

I wondered how they distinguished bots. The only info I found on the study [1], on a quick search, says: "For the purpose of this report we observed 1.45 Billion bot visits, which occurred over a 90 day period. The data was collected from a group of 20,000 sites on Incapsula’s network" So, in the first place, not necessarily representative of the web generally. And it does not say how they detect bots. Of course some declare themselves [2]; otherwise I guess one would have to rely on captchas [3], which are known to be of limited reliability and subject to an arms race. 1. http://www.incapsula.com/the-incapsula-blog/item/820-bot-traffic-report-2013 http://www.incapsula.com/the-incapsula-blog/item/820-bot-tra... 2. kristopolous on this page, '(bot|spider|crawl|aggregator)' 3. which Incapsula does, as a condition of access to its summary of its previous similar research.

skylan_q 13y ago

61.5% of Web Traffic Is Not Human I, for one, am glad that human trafficking over the web is down to only 38.5%.

devilshaircut 13y ago
Thanks for the laugh. Also, I wonder what the definition for "traffic" is within the context of this research.

bkirwi 13y ago

It's interesting to me that the author links to an article about how coding is useless for journalists, and then spends the rest of the article talking about all the web-scraping scripts she's written or could easily write. For me, this seems like a success of the whole teach-everyone-to-code thing; the author's picked up a bit of simple scripting and it's made some things dramatically easier for her. It's certainly not professional-level, but that's normal; for example, many people think programmers should know how to write, but few think they should all be able to churn out high-caliber journalism. Is this a failure of setting expectations, or is that really where some people are setting the bar?