10 ms·
Cloudflare Radar
- maximede 6y agoAny idea why the "worldwide change in internet traffic" graph is not flatter and has daily dips ? Is this graph normalized on the actual time, disregarding time-zones?
- AgentK20 6y agoI would say likely due to the uneven population distribution around the world, combined with day-of-week trends due to people being off work/school. The more interesting datapoint to me is the hour-by-hour breakdowns of HTTP vs HTTPS, likely due to particular countries that discourage (and have ISPs that block) encrypted traffic.
- jgrahamc 6y agoRelated blog post: https://blog.cloudflare.com/introducing-cloudflare-radar/ https://blog.cloudflare.com/introducing-cloudflare-radar/
- justinclift 6y agoOut of curiosity, what's the tech stack (eg backend, frontend) used for radar.cloudflare.com?
- jgrahamc 6y agoFrontend: React Serverless: Cloudflare Workers and Workers KV Data: ClickHouse
- justinclift 6y agoCool, thanks. :)
- different_sort 6y agoI love how you're using workers there. I've been on workers for years (only for static content mind you) and it's super cool.
- eminence32 6y agoThere's a lot of data here to digest, but one thing that stood out to me is: > Bot traffic: 41% of total So of all the traffic that CF sees, it attributes 41% of it to automated bots. This is way higher than I would have guessed!
- chomp 6y agoI used to work at a webhosting company, crawlers were an enormous amount of our traffic. Some of our users, their only traffic was crawler traffic.
- eminence32 6y agoWere these crawlers for big well-known things like google/bing/archive.org? I'm wondering if there are other types of crawlers that I don't even know about.
- oefrha 6y agoTraders crawl for financial information. Businesses (e.g. airlines) crawl for competitor pricing. Etc. etc.
- drcongo 6y agoA site for one of my clients was, at one point, getting absolutely hammered by a bot that crawls for plagiarised content. It was ignoring our robots.txt so I ended up blocking it via CloudFlare. There are bots for pretty much anything you can think of out there. edit: replaced "scrapes" with "crawls" for accuracy.
- thinkloop 6y agoWhy do bots need to crawl so much? Any idea what kind of bots they are, is it mainly search engines?
- heimatau 6y ago> Why do bots need to crawl so much? The Earth has a (mostly) free society. Therefore, many programmers find a web crawler to be an advantageous endeavor. From search engines, to social media companies, to ad networks, to government agencies, to financial institutions, to students learning about programming, to nefarious actors, and even shadow versions of everything above. Bots are neat. One can do a lot with a freely accessible public database of information, only limit is one's creativity. (edited: for formatting)
- oefrha 6y agoWow, tiktok.com steadily ranking at #3, more popular than facebook.com, amazon.com, apple.com and netflix.com, only behind google.com and microsoft.com. Really curious to see detailed stats.
- Arnt 6y agoI can see tiktok being more popular than apple, people watch videos much more often than they buy phones. But microsoft being more popular than tiktok? What's so popular on that site?
- jfim 6y agoIf they're just using DNS lookups, probably windows update, telemetry, crash reports and other automated traffic.
- odensc 6y agoAlso, verizonmedia.com at 8th place? What's on that site? Some type of CDN?
- bsagdiyev 6y agoVerizon Media afaik is EdgeCast or related, but I’m not 100% sure.
- antoncohen 6y agoCloudflare says they exclude "content servers"[1], I assume that means CDNs. https://www.verizonmedia.com/ https://www.verizonmedia.com/ says it is an ad network, including video ads, with brands like Yahoo, HuffPost and TechCrunch. [1] https://radar.cloudflare.com/glossary#trending-domains https://radar.cloudflare.com/glossary#trending-domains
- odensc 6y agoWhich is why I was curious - it seems that a domain mostly serving videos and images would be classified as a content server.
- aeyes 6y agoReally weird results, I have a popular site on Cloudflare and the domain popularity by visitor country shown on this public dashboard doesn't match traffic statistics I see in my account at all.
- est 6y agoThis is a direct replacement for Alexa Ranking. IMHO the traffic is way more accurate.
- colourgarden 6y agoAnyone know why there appears to be a steady daily +-5% wave on the HTTP/1.x vs HTTP/2 chart?
- jeffbee 6y agoIt seems to reflect the cycle of robot traffic.
- ElectronShak 6y agoMore surprising for me is Mobile vs Desktop at 35% to 65%. I thought it was closer, or more like 50/50.
- jlengrand 6y agoI'm guessing that it depends on the kind of traffic taken into account? If we count all requests to and from computers and not only web requests, you end up with a whole lot of computers calling home all day
- danso 6y agoThe trending/popular domains list is interesting [0]. Since Cloudflare effectively serves as part of the backbone for so many sites and services, I'm assuming their ranking is more able to count usage of sites that are mostly accessed by smartphone app, e.g. Tiktok #3, Facebook #4. Unlike Comscore [1], which seems to overly represent media sites. Too bad Radar omits pornography sites, I was curious whether they'd actually be among the top sites. FWIW, SimilarWeb [2] (which has Tiktok at #46), does include porn sites (xvideos and pornhub are #8 and #9, respectively). Though I don't quite understand how mozilla.org could rank as #10 on Radar's sites, just below Instagram and above Youtube, Whatsapp, and Twitter. edit: It's also interesting to compare its Top Browser rankings with analytics.usa.gov [3]. It appears that Radar's measured users are significantly more on Chrome – 55% (desktop+mobile) vs 47.6% on usa.gov sites; whereas usa.gov users are significantly more on Safari (iOS+desktop): 35.2% vs 13% on Radar. I suppose this reflects the assumption that iOS has a bigger proportion of U.S. users than it does worldwide users. [0] https://radar.cloudflare.com/#trending-domains https://radar.cloudflare.com/#trending-domains [1] https://www.comscore.com/Insights/Rankings https://www.comscore.com/Insights/Rankings [2] https://www.similarweb.com/top-websites/ https://www.similarweb.com/top-websites/ [3] https://analytics.usa.gov/ https://analytics.usa.gov/
- jeffbee 6y agoI think the fact that this list has microsoft in the #2 position and mozilla at #10 casts serious doubt upon the method and conclusion.
- tiffanyh 6y agoHow much internet traffic goes through Cloudflare? Last I heard it was somewhere around 5%. Does anyone know what % it is today?
- eat_veggies 6y agoIn 2018 it was 10%, iirc now it is 13-14%
- notadog 6y agoAccording to Cloudflare's website: > Internet requests for ~15% of the Fortune 1,000 run through Cloudflare's network A 2018 Wired article said "roughly 5 to 10 percent of web traffic." It is likely higher now, perhaps closer to 15%. https://www.cloudflare.com/insights/ https://www.cloudflare.com/insights/ and https://www.wired.com/story/cloudflare-spectrum-iot-protection/ https://www.wired.com/story/cloudflare-spectrum-iot-protecti...
- 7ewis 6y agoNot sure if I am reading it wrong, but I'm not sure the data is correct (at least for some non Cloudflare sites) example: https://radar.cloudflare.com/domain/just-eat.co.uk https://radar.cloudflare.com/domain/just-eat.co.uk The leading UK food delivery company, for popularity I'm not sure if a higher or lower number means more popular - but to me it looks like the site has more traffic on a weekday than a weekend. And also most traffic comes from Australia, where this site wouldn't even work. Is this just bots, or bad data?
- ZiiS 6y agoThey are at least consistent; just-eat's AU site is mostly UK traffic https://radar.cloudflare.com/domain/menulog.com.au https://radar.cloudflare.com/domain/menulog.com.au
- jgrahamc 6y agoThat's odd. I'll get the team to investigate.
- mutatio 6y agoI just checked a project of mine, which uses CloudFlare and gets reasonable traffic, it seems Australia is disproportionately represented for it too.
- remram 6y agoHow is this data collected? I can see how they would be able to get statistics from clients hitting CloudFlare servers, but how can they report on the traffic hitting google.com or microsoft.com?
- pingiun 6y agoI think that might just be based on their public dns service (https://1.1.1.1 https://1.1.1.1)
- remram 6y agoThey can tell bots and DoS traffic from the DNS request?
- ta8964586 6y agoI'm guessing the different metrics are collected through different means. Their blog mentions HTTP request to sites that use cloudflare as well as DNS [0]. It would make sense that domain popularity is meaused through DNS and bots and DOS measured by the systems built for handling those things. [0]: https://blog.cloudflare.com/introducing-cloudflare-radar/ https://blog.cloudflare.com/introducing-cloudflare-radar/
- sp332 6y agoWhat is a Facebook web browser?
- STRiDEX 6y agoHad the same question. Is it the in app web view? Wouldn't that be safari/chrome depending on platform?
- dirtnugget 6y agoWell, yes and no. They have their own user agent, so they can get their own classification. e.g. Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 [FBAN/FBIOS;FBDV/iPhone11,8;FBMD/iPhone;FBSN/iOS;FBSV/13.3.1;FBSS/2;FBID/phone;FBLC/en_US;FBOP/5;FBCR/] Identifies as "Facebook Browser". See: https://developers.whatismybrowser.com/useragents/explore/software_type_specific/in-app-browser/ https://developers.whatismybrowser.com/useragents/explore/so... I think Google Analytics picks this up as "Safari (In-App)" If this classification was not possible, Edge would identify as Chrome. (since it uses Chromium engine)
- fabrice_d 6y agoThis is the in app browser.
- AznHisoka 6y agoIt would be great if the site rankings had a longer time period than 30 days, such as 1 year. Though totally understand if they only started tracking such a ranking recently.
- jgrahamc 6y agoYes. We only started doing these calculations recently. We'll make available more data as we have it.
- sleepyhead 6y agoDevelopers: The page must be fast! Google found an extra .5 seconds in search page generation time dropped traffic by 20%. Also developers: Let's use Cloudflare - 'Checking your browser before accessing content - please allow up to 5 seconds'.
- andrewxdiamond 6y agoI believe most of the time the protection is invisible. It only activates when it thinks you might be a bot, or if the site is under heavy load
- fivre 6y agoThere is not-insignificant, if not huge group of site admins that take a "aha! more security therefore better!" approach and will just enable under attack mode (the 5s JS challenge) for everyone all the time. Some are a bit more savvy and will take a still rather ham-fisted approach of doing so for most countries. That's not uncommon with retailer websites that realistically only serve a single country, and therefore have good enough reason to challenge all others to deter automated traffic from compromised machines outside their market.
- ArchOversight 6y agoI see it ALL of the time. I am not sure if it is due to Safari and ITP blocking whatever protection tracking stuff Cloudflare does, or because my DNS is routed through AWS vs some other provider (as I am on a split tunnel VPN, and need to resolve internal DNS entries). It's frustrating, especially when I open a couple of tabs on the same site after Googling a particular problem.
- JakeTheAndroid 6y agoIt could be the ITP blocking or your DNS setup. If I had to guess it'd be the AWS setup you have as that is not seen as a client IP for a home network. The interstitial page you're seeing attempts to validate IP location, HTTP headers, and the query string at the very least. So the fact that your requests look like they are coming directly from an AWS IP would, based on my gut feeling having troubleshot tons of these issues at CF, cause this issue. If you're ever curious you might want to try and hit a site you know serves you these pages regularly without the VPN setup just to isolate the problem.
- topherjaynes 6y agoDoes anyone else find it odd Zoom not in top 15? Is it only because it's not webtraffic?
- DoctorOW 6y agoThat sounds right to me. I doubt a DNS request is made for each packet.
- kzrdude 6y agoWhat is the source of the data? Is the ranking of domains by number of requests or by traffic volume? Why don't we see any major CDN domains in the top ten list?
- Bedon292 6y agoThey don't really say how they are ranking things, it appears to be multiple metrics. They also removed porn, and I would not be surprised if they removed something like amazonaws.com since that just sits in front of s3 bucket. https://radar.cloudflare.com/glossary#trending-domains https://radar.cloudflare.com/glossary#trending-domains
- kzrdude 6y agoSmells artificial/trimmed