11 ms·

Cached Chrome Top Million Websites

261 points by edent 4y ago

anonu 4y ago
> grep http: current.csv | wc -l 54679 So over 5% of the top 1m sites still don't use HTTPS.
- Proven 4y ago
  [dead]
- philipphutterer 4y ago
  How about websites that are browsed http first and then redirected? People might browse for a domain without the https prefix for convenience (or old links) and the browser defaults to http.
- alfu 4y ago
  If I am not mistaken, 8310 sites offer http and https: grep -o -E "://.*?," current.csv | sort | uniq -c | grep -v "1 ://" | wc -l 8310
- forgotmypw17 4y ago
  [flagged]
  - bedatadriven 4y ago
    What accessibility challenges does https pose?
    - forgotmypw17 4y ago
      The accessibility challenges are all the extra different failure modes HTTPS presents, such as client date offset, older devices, expired certificates, hostname mismatches, and many others. Security is not the only priority in existence. Sometimes people just want to access the information. And when that is the case, HTTPS can be a huge impediment.
      - 2Gkashmiri 4y ago
        yeah... i havent gotten a good response why localhost should scream "insecure" or why i should wikipedia fail if my rtc clock is wonky. i am not denying "security from snoops while paying with credit cards" and all that banking shit or messaging. heck, email is sent over the clear but we are told to use https to connect to the website (for webmails) using https for "security"... sure sure security is all good and snazzy but i regularly come across websites who have had certs expired and the website makes it appear as if the sky will fall if i click on continue. then we have ISPs who use DPI (my current ISP, reliance jio is doing it from day 1) so whats the point of pretending anyway?
        rileymat2 4y ago
        > email is sent over the clear but we are told to use https to connect to the website (for webmails) using https for "security" This is not true, most is encrypted in transit. It is not end to end, because your email service stores them (perhaps encrypted perhaps not). Edit: https://transparencyreport.google.com/safer-email/overview?hl=en&encrypt_out=start:1656633600000;end:1666137599999;series:outbound&lu=encrypt_out https://transparencyreport.google.com/safer-email/overview?h... You can see 84% of outbound is encrypted. This probably is generally a good proxy for the state of email tls transport.
        jefftk 4y ago
        > why localhost should scream "insecure" Localhost, even with HTTP, is a secure context: https://developer.mozilla.org/en-US/docs/Web/Security/Secure_Contexts https://developer.mozilla.org/en-US/docs/Web/Security/Secure... What tool is screaming at you that localhost is insecure?
        2Gkashmiri 4y ago
        browsers. padlock icon is crossed out
        jefftk 4y ago
        I just tested this and don't see that. I compared http://neverssl.com http://neverssl.com to running "python3 -m http.server" and visiting http://localhost:8000 http://localhost:8000 * Chrome: "Not Secure" on neverssl, "i" in a circle on localhost * Firefox: Padlock with a red line through it on neverssl, page icon on localhost * Safari: "Not Secure" on neverssl, no message on localhost
- zX41ZdbW 4y ago
  I have prepared a nice report: the rank of the websites in groups 1..10, 11..100, ... the percentage of TLS and an example of non-TLS website: https://play.clickhouse.com/play?user=play#U0VMRUNUIGZsb29yKGxvZzEwKHJhbmspKSBBUyByLCBjb3VudCgpIEFTIHRvdGFsLCBzdW0obG9nIExJS0UgJyVUTFMlJykgQVMgdGxzLCByb3VuZCh0bHMgLyB0b3RhbCwgMikgQVMgcmF0aW8sIGFueUlmKGRvbWFpbiwgbG9nIE5PVCBMSUtFICclVExTJScpIEZST00gbWluaWNyYXdsIFdIRVJFIGxvZyBMSUtFICclQ29udGVudC1MZW5ndGg6JScgR1JPVVAgQlkgciBPUkRFUiBCWSBy https://play.clickhouse.com/play?user=play#U0VMRUNUIGZsb29yK... SELECT floor(log10(rank)) AS r, count() AS total, sum(log LIKE '%TLS%') AS tls, round(tls / total, 2) AS ratio, anyIf(domain, log NOT LIKE '%TLS%') FROM minicrawl WHERE log LIKE '%Content-Length:%' GROUP BY r ORDER BY r ┌─r─┬───total─┬─────tls─┬─ratio─┬─anyIf(domain, notLike(log, '%TLS%'))─┐ │ 0 │ 6 │ 6 │ 1 │ │ │ 1 │ 61 │ 58 │ 0.95 │ baidu.com │ │ 2 │ 599 │ 562 │ 0.94 │ google.cn │ │ 3 │ 5591 │ 5057 │ 0.9 │ volganet.ru │ │ 4 │ 51279 │ 44291 │ 0.86 │ furbo.co │ │ 5 │ 476181 │ 361910 │ 0.76 │ funygold.com │ │ 6 │ 3797023 │ 2927052 │ 0.77 │ funyo.vip │ └───┴─────────┴─────────┴───────┴──────────────────────────────────────┘ 7 rows in set. Elapsed: 0.844 sec. Processed 7.59 million rows, 43.74 GB (8.99 million rows/s., 51.83 GB/s.)
  - kedmi 4y ago
    Excuse my ignorance, what CLI tool did you use to execute this query? Thanks!
    - deleted 4y ago
      [deleted]
    - zX41ZdbW 4y ago
      clickhouse-client Download it as: curl https://clickhouse.com/ | sh Connect to the demo service: clickhouse-client --host play.clickhouse.com --user play --secure
- _nhynes 4y ago
  “The 5% rule”
est 4y ago
Looks like not a single Chinese site made to top 1k. I guess it's reasonable because all Google services were blocked so CrUX can't gather any data.
- themoonisachees 4y ago
  Do Chinese people use chrome? One would think the download page is blocked as well, so the demographic for chrome users should be way smaller. Also to consider: China uses in-app browsing a lot, with interactive experiences very similar to websites built right in the bilibili/ali/wechat apps.
  - moffkalast 4y ago
    > in-app browsing But that's also just chromium isn't it, much like a PWA? Unless they made something of their own.
    - jfoster 4y ago
      But does chromium phone home with this data? Surely not, I would've thought?
  - est 4y ago
    > One would think the download page is blocked as well Contrary to popular belief, Google only pulled Search business out of China. The rest of services is still hosted on Google.cn inside China. To download Chrome: $ curl -svk 'https://www.google.cn/intl/zh-CN/chrome/ https://www.google.cn/intl/zh-CN/chrome/' * Trying 180.163.150.34... * TCP_NODELAY set * Connected to www.google.cn (180.163.150.34) port 443 (#0) However the "Make searches and browsing better (Sends URLs of pages you visit to Google)" data won't be collected, because the connection would be blocked.
- kristianp 4y ago
  > The CrUX dataset is based on data collected from Google Chrome and is thus biased away from countries with limited Chrome usage (e.g., China).
Mortiffer 4y ago
Thanks for pointing this out. Can definitely put this dataset to use
cronaday 4y ago
This is very ethically dubious. Google is collecting raw URLs from Chrome users who turned on history syncing across their own devices, then reusing the data and funneling it through Stanford. No way Chrome users understand or approve of this. The paper tries to justify its ethics with Google's privacy policy, which is laughable. There are so many papers about how meaningless privacy policies are. If Apple or Mozilla did anything remotely like this, Hacker News would riot. Edit: I don't want to be a conspiracy theorist, but this post suddenly got a bunch of downvotes at the same time as defensive comments from a current Googler and recent ex-Googler. Then one of my responses below to a Chrome developer got flagged for no obvious reason. Hmm.
- soneca 4y ago
  > ” If Apple or Mozilla did anything remotely like this, Hacker News would riot.” My perception is that, collectively, HN hates and criticizes Google much more than Apple and Mozilla. I mean, much more. This last sentence accusation sounded bizarre to me.
  - cronaday 4y ago
    Just suggesting that prior browser and OS privacy blowups involving those companies have been over less worrisome things, not that those companies are subject to more or less criticism. Looking back on outraged discussions of Mozilla's telemetry is kinda quaint in comparison.
  - hericium 4y ago
    > My perception is that, collectively, HN hates and criticizes Google much more than Apple and Mozilla. Not the entirety of HN. As I have more than once delicately pointed out[1], Mozilla is Google's bitch. [1] https://news.ycombinator.com/item?id=30732539 https://news.ycombinator.com/item?id=30732539
  - dmitriid 4y ago
    > I mean, much more. Because Google is a web advertisement company that dominates many large spheres: search, browsers (including standards committees), email, mobile (Android is 77% market share) etc. All are things that we've come to view as crucial to modern life. And time and again they've shown that they only view that dominance as a funnel for ad revenue, data collection, and whatever benefits them at this particular moment.
mg 4y ago
Top level domains by popularity: grep -oP '\.[a-z]+(?=,)' current.csv | sort | uniq -c | sort -n ... 15840 .pl 17914 .it 20182 .de 21690 .in 27812 .ru 29194 .jp 30359 .org 35741 .br 36675 .net 406052 .com .com domains by popularity: grep -oP '[a-z0-9-]+\.com(?=,)' current.csv | sort | uniq -c | sort -n ... 365 tistory.com 370 fc2.com 408 skipthegames.com 489 online.com 515 wordpress.com 707 uptodown.com 880 schoology.com 2570 fandom.com 2651 instructure.com 3244 blogspot.com
- deleted 4y ago
  [deleted]
- voytec 4y ago
  sort -r will reverse the order from most to less popular.
  - layer8 4y ago
    .
    - codetrotter 4y ago
      uniq -c is in the pipeline because it counts the number of uniques
      - layer8 4y ago
        Ah, right, missed that.
- egman_ekki 4y ago
  Rather amazing seeing almost abandoned blogspot.com there at the top. Also interesting I haven’t heard about half of them. Some are nsfw, apparently.
  - mometsi 4y ago
zX41ZdbW 4y ago
If you are interested in the research on technologies used on the Internet, I recommend playing with the "Minicrawl" dataset. It contains data about ~7 million top websites, and for every website, it also contains: - the full content of the main page; - the verbose output of curl, containing various timing info; the HTTP headers, protocol info... Using this dataset, you can build a service similar to https://builtwith.com/ https://builtwith.com/ for your research. Data: https://clickhouse-public-datasets.s3.amazonaws.com/minicrawl/data.native.zst https://clickhouse-public-datasets.s3.amazonaws.com/minicraw... (129 GB compressed, ~1 TB uncompressed). Description: https://github.com/ClickHouse/ClickHouse/issues/18842 https://github.com/ClickHouse/ClickHouse/issues/18842 You can easily try it with clickhouse-local without downloading: $ curl https://clickhouse.com/ | sh $ ./clickhouse local ClickHouse local version 22.13.1.294 (official build). milovidov-desktop :) DESCRIBE url('https://clickhouse-public-datasets.s3.amazonaws.com/minicrawl/data.native.zst') DESCRIBE TABLE url('https://clickhouse-public-datasets.s3.amazonaws.com/minicrawl/data.native.zst') Query id: 6746232f-7f5f-4c5a-ac68-d749d949a2dc ┌─name────┬─type───┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐ │ rank │ UInt32 │ │ │ │ │ │ │ domain │ String │ │ │ │ │ │ │ log │ String │ │ │ │ │ │ │ content │ String │ │ │ │ │ │ └─────────┴────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘ 4 rows in set. Elapsed: 1.390 sec. milovidov-desktop :) SELECT rank, domain, log, substringUTF8(content, 1, 100) FROM url('https://clickhouse-public-datasets.s3.amazonaws.com/minicrawl/data.native.zst') LIMIT 1 FORMAT Vertical SELECT rank, domain, log, substringUTF8(content, 1, 100) FROM url('https://clickhouse-public-datasets.s3.amazonaws.com/minicrawl/data.native.zst') LIMIT 1 FORMAT Vertical Query id: 8dba6976-0bf6-4ce8-a0f1-aa579c828175 Row 1: ────── rank: 1907977 domain: 0--0.uk log: * Trying 213.32.47.30:80... * Connected to 0--0.uk (213.32.47.30) port 80 (#0) > GET / HTTP/1.1 > Host: 0--0.uk > Accept: */* > User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0 > * Mark bundle as not supporting multiuse < HTTP/1.1 302 Moved Temporarily < Server: nginx < Date: Sun, 29 May 2022 06:27:14 GMT < Content-Type: text/html < Content-Length: 154 < Connection: keep-alive < Location: https://0--0.uk/ < * Ignoring the response-body { [154 bytes data] * Connection #0 to host 0--0.uk left intact * Issue another request to this URL: 'https://0--0.uk/' * Trying 213.32.47.30:443... * Connected to 0--0.uk (213.32.47.30) port 443 (#1) * ALPN, offering h2 * ALPN, offering http/1.1 * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: /etc/ssl/certs * TLSv1.0 (OUT), TLS header, Certificate Status (22): } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.2 (IN), TLS header, Certificate Status (22): { [5 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [108 bytes data] * TLSv1.2 (IN), TLS header, Certificate Status (22): { [5 bytes data] * TLSv1.2 (IN), TLS handshake, Certificate (11): { [4150 bytes data] * TLSv1.2 (IN), TLS header, Certificate Status (22): { [5 bytes data] * TLSv1.2 (IN), TLS handshake, Server key exchange (12): { [333 bytes data] * TLSv1.2 (IN), TLS header, Certificate Status (22): { [5 bytes data] * TLSv1.2 (IN), TLS handshake, Server finished (14): { [4 bytes data] * TLSv1.2 (OUT), TLS header, Certificate Status (22): } [5 bytes data] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): } [70 bytes data] * TLSv1.2 (OUT), TLS header, Finished (20): } [5 bytes data] * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.2 (OUT), TLS header, Certificate Status (22): } [5 bytes data] * TLSv1.2 (OUT), TLS handshake, Finished (20): } [16 bytes data] * TLSv1.2 (IN), TLS header, Finished (20): { [5 bytes data] * TLSv1.2 (IN), TLS header, Certificate Status (22): { [5 bytes data] * TLSv1.2 (IN), TLS handshake, Finished (20): { [16 bytes data] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: CN=mail.htservices.co.uk * start date: May 15 18:36:37 2022 GMT * expire date: Aug 13 18:36:36 2022 GMT * subjectAltName: host "0--0.uk" matched cert's "0--0.uk" * issuer: C=US; O=Let's Encrypt; CN=R3 * SSL certificate verify ok. * TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data] > GET / HTTP/1.1 > Host: 0--0.uk > Accept: */* > User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0 > * TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data] * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Server: nginx < Date: Sun, 29 May 2022 06:27:15 GMT < Content-Type: text/html;charset=utf-8 < Transfer-Encoding: chunked < Connection: keep-alive < X-Frame-Options: SAMEORIGIN < Expires: -1 < Cache-Control: no-store, no-cache, must-revalidate, max-age=0 < Pragma: no-cache < Content-Language: en-US < Set-Cookie: ZM_TEST=true;Secure < Set-Cookie: ZM_LOGIN_CSRF=b2dda010-d795-4759-a9c3-80349f3b46ed;Secure;HttpOnly < Vary: User-Agent < X-UA-Compatible: IE=edge < Vary: Accept-Encoding, User-Agent < { [13068 bytes data] * Connection #1 to host 0--0.uk left intact substringUTF8(content, 1, 100): <!DOCTYPE html> <!-- set this class so CSS definitions that now use REM size, would work relative to 1 row in set. Elapsed: 0.539 sec. Processed 4.60 thousand rows, 273.86 MB (8.54 thousand rows/s., 508.28 MB/s.)
wirthjason 4y ago
Curious where PornHub and other sites rank. I always hear how that porn sites are in the top X of all traffics but people don’t talk about due to its nature. I’m always amazed that they have a data science team. It’s not something many would expect from the porn industry. I certainly didn’t expect it. https://www.pornhub.com/insights/2022-year-in-review https://www.pornhub.com/insights/2022-year-in-review
- layer8 4y ago
  A quick grep shows that there are almost 2.5K domains at 1M rank with “porn” in their name. The data science teams likely provide a considerable ROI in that industry.
  - system2 4y ago
    Majority of porn sites do not have porn word in them though. I wish there was a categorization of these.
  - Giorgi 4y ago
    does it though? Pretty sure adult results are being filtered off by Google tool named "SafeSearch". It removes anything adult from SERP and it is on by default.
    - yorwba 4y ago
      This appears to have some unintuitive consequences. When I searched for "porn" in a cookie-less session just now, there were still porn results, but no well-known sites (at least I didn't recognize the names). Searching for literally "pornhub", the first result is "porhub.com" without the "n". Seems like the "SafeSearch" filter is based on a list of "adult domains" instead of the indexed content at the URL.
- E39M5S62 4y ago
  If you ignore the content, large-scale adult sites are just like any other high traffic (bandwidth, RPS) site out there. A lot of planning goes into where their content delivery PoPs should be placed.
deterrence 4y ago
[flagged]
kristianp 4y ago
This raises the question: how much in the way of user telemetry does Chrome send back to google?
- tgsovlerkhgsel 4y ago
  By default, a lot. However, they also are (or at least used to be, it seems to be quite outdated now) really good at documenting their telemetry publicly: https://www.google.com/chrome/privacy/whitepaper.html https://www.google.com/chrome/privacy/whitepaper.html (I haven't checked whether the documentation is complete/accurate, of course.)
- deleted 4y ago
  [deleted]
modeless 4y ago
Wow, I'm kinda surprised to find my site in the top million worldwide. I have about 100k monthly visits as measured by Cloudflare web analytics, I guess that's all it takes.

261 points by edent 4y ago

anonu 4y ago

> grep http: current.csv | wc -l 54679 So over 5% of the top 1m sites still don't use HTTPS.

Proven 4y ago
[dead]
philipphutterer 4y ago
How about websites that are browsed http first and then redirected? People might browse for a domain without the https prefix for convenience (or old links) and the browser defaults to http.
alfu 4y ago
If I am not mistaken, 8310 sites offer http and https: grep -o -E "://.*?," current.csv | sort | uniq -c | grep -v "1 ://" | wc -l 8310
forgotmypw17 4y ago
[flagged]
- bedatadriven 4y ago
  What accessibility challenges does https pose?
  - forgotmypw17 4y ago
    The accessibility challenges are all the extra different failure modes HTTPS presents, such as client date offset, older devices, expired certificates, hostname mismatches, and many others. Security is not the only priority in existence. Sometimes people just want to access the information. And when that is the case, HTTPS can be a huge impediment.
    - 2Gkashmiri 4y ago
      yeah... i havent gotten a good response why localhost should scream "insecure" or why i should wikipedia fail if my rtc clock is wonky. i am not denying "security from snoops while paying with credit cards" and all that banking shit or messaging. heck, email is sent over the clear but we are told to use https to connect to the website (for webmails) using https for "security"... sure sure security is all good and snazzy but i regularly come across websites who have had certs expired and the website makes it appear as if the sky will fall if i click on continue. then we have ISPs who use DPI (my current ISP, reliance jio is doing it from day 1) so whats the point of pretending anyway?
      - rileymat2 4y ago
        > email is sent over the clear but we are told to use https to connect to the website (for webmails) using https for "security" This is not true, most is encrypted in transit. It is not end to end, because your email service stores them (perhaps encrypted perhaps not). Edit: https://transparencyreport.google.com/safer-email/overview?hl=en&encrypt_out=start:1656633600000;end:1666137599999;series:outbound&lu=encrypt_out https://transparencyreport.google.com/safer-email/overview?h... You can see 84% of outbound is encrypted. This probably is generally a good proxy for the state of email tls transport.
      - jefftk 4y ago
        > why localhost should scream "insecure" Localhost, even with HTTP, is a secure context: https://developer.mozilla.org/en-US/docs/Web/Security/Secure_Contexts https://developer.mozilla.org/en-US/docs/Web/Security/Secure... What tool is screaming at you that localhost is insecure?
        2Gkashmiri 4y ago
        browsers. padlock icon is crossed out
        jefftk 4y ago
        I just tested this and don't see that. I compared http://neverssl.com http://neverssl.com to running "python3 -m http.server" and visiting http://localhost:8000 http://localhost:8000 * Chrome: "Not Secure" on neverssl, "i" in a circle on localhost * Firefox: Padlock with a red line through it on neverssl, page icon on localhost * Safari: "Not Secure" on neverssl, no message on localhost
zX41ZdbW 4y ago
I have prepared a nice report: the rank of the websites in groups 1..10, 11..100, ... the percentage of TLS and an example of non-TLS website: https://play.clickhouse.com/play?user=play#U0VMRUNUIGZsb29yKGxvZzEwKHJhbmspKSBBUyByLCBjb3VudCgpIEFTIHRvdGFsLCBzdW0obG9nIExJS0UgJyVUTFMlJykgQVMgdGxzLCByb3VuZCh0bHMgLyB0b3RhbCwgMikgQVMgcmF0aW8sIGFueUlmKGRvbWFpbiwgbG9nIE5PVCBMSUtFICclVExTJScpIEZST00gbWluaWNyYXdsIFdIRVJFIGxvZyBMSUtFICclQ29udGVudC1MZW5ndGg6JScgR1JPVVAgQlkgciBPUkRFUiBCWSBy https://play.clickhouse.com/play?user=play#U0VMRUNUIGZsb29yK... SELECT floor(log10(rank)) AS r, count() AS total, sum(log LIKE '%TLS%') AS tls, round(tls / total, 2) AS ratio, anyIf(domain, log NOT LIKE '%TLS%') FROM minicrawl WHERE log LIKE '%Content-Length:%' GROUP BY r ORDER BY r ┌─r─┬───total─┬─────tls─┬─ratio─┬─anyIf(domain, notLike(log, '%TLS%'))─┐ │ 0 │ 6 │ 6 │ 1 │ │ │ 1 │ 61 │ 58 │ 0.95 │ baidu.com │ │ 2 │ 599 │ 562 │ 0.94 │ google.cn │ │ 3 │ 5591 │ 5057 │ 0.9 │ volganet.ru │ │ 4 │ 51279 │ 44291 │ 0.86 │ furbo.co │ │ 5 │ 476181 │ 361910 │ 0.76 │ funygold.com │ │ 6 │ 3797023 │ 2927052 │ 0.77 │ funyo.vip │ └───┴─────────┴─────────┴───────┴──────────────────────────────────────┘ 7 rows in set. Elapsed: 0.844 sec. Processed 7.59 million rows, 43.74 GB (8.99 million rows/s., 51.83 GB/s.)
- kedmi 4y ago
  Excuse my ignorance, what CLI tool did you use to execute this query? Thanks!
  - deleted 4y ago
    [deleted]
  - zX41ZdbW 4y ago
    clickhouse-client Download it as: curl https://clickhouse.com/ | sh Connect to the demo service: clickhouse-client --host play.clickhouse.com --user play --secure
_nhynes 4y ago
“The 5% rule”

est 4y ago

Looks like not a single Chinese site made to top 1k. I guess it's reasonable because all Google services were blocked so CrUX can't gather any data.

themoonisachees 4y ago
Do Chinese people use chrome? One would think the download page is blocked as well, so the demographic for chrome users should be way smaller. Also to consider: China uses in-app browsing a lot, with interactive experiences very similar to websites built right in the bilibili/ali/wechat apps.
- moffkalast 4y ago
  > in-app browsing But that's also just chromium isn't it, much like a PWA? Unless they made something of their own.
  - jfoster 4y ago
    But does chromium phone home with this data? Surely not, I would've thought?
- est 4y ago
  > One would think the download page is blocked as well Contrary to popular belief, Google only pulled Search business out of China. The rest of services is still hosted on Google.cn inside China. To download Chrome: $ curl -svk 'https://www.google.cn/intl/zh-CN/chrome/ https://www.google.cn/intl/zh-CN/chrome/' * Trying 180.163.150.34... * TCP_NODELAY set * Connected to www.google.cn (180.163.150.34) port 443 (#0) However the "Make searches and browsing better (Sends URLs of pages you visit to Google)" data won't be collected, because the connection would be blocked.
kristianp 4y ago
> The CrUX dataset is based on data collected from Google Chrome and is thus biased away from countries with limited Chrome usage (e.g., China).

Mortiffer 4y ago

Thanks for pointing this out. Can definitely put this dataset to use

cronaday 4y ago

This is very ethically dubious. Google is collecting raw URLs from Chrome users who turned on history syncing across their own devices, then reusing the data and funneling it through Stanford. No way Chrome users understand or approve of this. The paper tries to justify its ethics with Google's privacy policy, which is laughable. There are so many papers about how meaningless privacy policies are. If Apple or Mozilla did anything remotely like this, Hacker News would riot. Edit: I don't want to be a conspiracy theorist, but this post suddenly got a bunch of downvotes at the same time as defensive comments from a current Googler and recent ex-Googler. Then one of my responses below to a Chrome developer got flagged for no obvious reason. Hmm.

soneca 4y ago
> ” If Apple or Mozilla did anything remotely like this, Hacker News would riot.” My perception is that, collectively, HN hates and criticizes Google much more than Apple and Mozilla. I mean, much more. This last sentence accusation sounded bizarre to me.
- cronaday 4y ago
  Just suggesting that prior browser and OS privacy blowups involving those companies have been over less worrisome things, not that those companies are subject to more or less criticism. Looking back on outraged discussions of Mozilla's telemetry is kinda quaint in comparison.
- hericium 4y ago
  > My perception is that, collectively, HN hates and criticizes Google much more than Apple and Mozilla. Not the entirety of HN. As I have more than once delicately pointed out[1], Mozilla is Google's bitch. [1] https://news.ycombinator.com/item?id=30732539 https://news.ycombinator.com/item?id=30732539
- dmitriid 4y ago
  > I mean, much more. Because Google is a web advertisement company that dominates many large spheres: search, browsers (including standards committees), email, mobile (Android is 77% market share) etc. All are things that we've come to view as crucial to modern life. And time and again they've shown that they only view that dominance as a funnel for ad revenue, data collection, and whatever benefits them at this particular moment.

mg 4y ago

Top level domains by popularity: grep -oP '\.[a-z]+(?=,)' current.csv | sort | uniq -c | sort -n ... 15840 .pl 17914 .it 20182 .de 21690 .in 27812 .ru 29194 .jp 30359 .org 35741 .br 36675 .net 406052 .com .com domains by popularity: grep -oP '[a-z0-9-]+\.com(?=,)' current.csv | sort | uniq -c | sort -n ... 365 tistory.com 370 fc2.com 408 skipthegames.com 489 online.com 515 wordpress.com 707 uptodown.com 880 schoology.com 2570 fandom.com 2651 instructure.com 3244 blogspot.com

deleted 4y ago
[deleted]
voytec 4y ago
sort -r will reverse the order from most to less popular.
- layer8 4y ago
  .
  - codetrotter 4y ago
    uniq -c is in the pipeline because it counts the number of uniques
    - layer8 4y ago
      Ah, right, missed that.
egman_ekki 4y ago
Rather amazing seeing almost abandoned blogspot.com there at the top. Also interesting I haven’t heard about half of them. Some are nsfw, apparently.
- mometsi 4y ago

zX41ZdbW 4y ago

If you are interested in the research on technologies used on the Internet, I recommend playing with the "Minicrawl" dataset. It contains data about ~7 million top websites, and for every website, it also contains: - the full content of the main page; - the verbose output of curl, containing various timing info; the HTTP headers, protocol info... Using this dataset, you can build a service similar to https://builtwith.com/ https://builtwith.com/ for your research. Data: https://clickhouse-public-datasets.s3.amazonaws.com/minicrawl/data.native.zst https://clickhouse-public-datasets.s3.amazonaws.com/minicraw... (129 GB compressed, ~1 TB uncompressed). Description: https://github.com/ClickHouse/ClickHouse/issues/18842 https://github.com/ClickHouse/ClickHouse/issues/18842 You can easily try it with clickhouse-local without downloading: $ curl https://clickhouse.com/ | sh $ ./clickhouse local ClickHouse local version 22.13.1.294 (official build). milovidov-desktop :) DESCRIBE url('https://clickhouse-public-datasets.s3.amazonaws.com/minicrawl/data.native.zst') DESCRIBE TABLE url('https://clickhouse-public-datasets.s3.amazonaws.com/minicrawl/data.native.zst') Query id: 6746232f-7f5f-4c5a-ac68-d749d949a2dc ┌─name────┬─type───┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐ │ rank │ UInt32 │ │ │ │ │ │ │ domain │ String │ │ │ │ │ │ │ log │ String │ │ │ │ │ │ │ content │ String │ │ │ │ │ │ └─────────┴────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘ 4 rows in set. Elapsed: 1.390 sec. milovidov-desktop :) SELECT rank, domain, log, substringUTF8(content, 1, 100) FROM url('https://clickhouse-public-datasets.s3.amazonaws.com/minicrawl/data.native.zst') LIMIT 1 FORMAT Vertical SELECT rank, domain, log, substringUTF8(content, 1, 100) FROM url('https://clickhouse-public-datasets.s3.amazonaws.com/minicrawl/data.native.zst') LIMIT 1 FORMAT Vertical Query id: 8dba6976-0bf6-4ce8-a0f1-aa579c828175 Row 1: ────── rank: 1907977 domain: 0--0.uk log: * Trying 213.32.47.30:80... * Connected to 0--0.uk (213.32.47.30) port 80 (#0) > GET / HTTP/1.1 > Host: 0--0.uk > Accept: */* > User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0 > * Mark bundle as not supporting multiuse < HTTP/1.1 302 Moved Temporarily < Server: nginx < Date: Sun, 29 May 2022 06:27:14 GMT < Content-Type: text/html < Content-Length: 154 < Connection: keep-alive < Location: https://0--0.uk/ < * Ignoring the response-body { [154 bytes data] * Connection #0 to host 0--0.uk left intact * Issue another request to this URL: 'https://0--0.uk/' * Trying 213.32.47.30:443... * Connected to 0--0.uk (213.32.47.30) port 443 (#1) * ALPN, offering h2 * ALPN, offering http/1.1 * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: /etc/ssl/certs * TLSv1.0 (OUT), TLS header, Certificate Status (22): } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.2 (IN), TLS header, Certificate Status (22): { [5 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [108 bytes data] * TLSv1.2 (IN), TLS header, Certificate Status (22): { [5 bytes data] * TLSv1.2 (IN), TLS handshake, Certificate (11): { [4150 bytes data] * TLSv1.2 (IN), TLS header, Certificate Status (22): { [5 bytes data] * TLSv1.2 (IN), TLS handshake, Server key exchange (12): { [333 bytes data] * TLSv1.2 (IN), TLS header, Certificate Status (22): { [5 bytes data] * TLSv1.2 (IN), TLS handshake, Server finished (14): { [4 bytes data] * TLSv1.2 (OUT), TLS header, Certificate Status (22): } [5 bytes data] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): } [70 bytes data] * TLSv1.2 (OUT), TLS header, Finished (20): } [5 bytes data] * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.2 (OUT), TLS header, Certificate Status (22): } [5 bytes data] * TLSv1.2 (OUT), TLS handshake, Finished (20): } [16 bytes data] * TLSv1.2 (IN), TLS header, Finished (20): { [5 bytes data] * TLSv1.2 (IN), TLS header, Certificate Status (22): { [5 bytes data] * TLSv1.2 (IN), TLS handshake, Finished (20): { [16 bytes data] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: CN=mail.htservices.co.uk * start date: May 15 18:36:37 2022 GMT * expire date: Aug 13 18:36:36 2022 GMT * subjectAltName: host "0--0.uk" matched cert's "0--0.uk" * issuer: C=US; O=Let's Encrypt; CN=R3 * SSL certificate verify ok. * TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data] > GET / HTTP/1.1 > Host: 0--0.uk > Accept: */* > User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0 > * TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data] * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Server: nginx < Date: Sun, 29 May 2022 06:27:15 GMT < Content-Type: text/html;charset=utf-8 < Transfer-Encoding: chunked < Connection: keep-alive < X-Frame-Options: SAMEORIGIN < Expires: -1 < Cache-Control: no-store, no-cache, must-revalidate, max-age=0 < Pragma: no-cache < Content-Language: en-US < Set-Cookie: ZM_TEST=true;Secure < Set-Cookie: ZM_LOGIN_CSRF=b2dda010-d795-4759-a9c3-80349f3b46ed;Secure;HttpOnly < Vary: User-Agent < X-UA-Compatible: IE=edge < Vary: Accept-Encoding, User-Agent < { [13068 bytes data] * Connection #1 to host 0--0.uk left intact substringUTF8(content, 1, 100): <!DOCTYPE html> <!-- set this class so CSS definitions that now use REM size, would work relative to 1 row in set. Elapsed: 0.539 sec. Processed 4.60 thousand rows, 273.86 MB (8.54 thousand rows/s., 508.28 MB/s.)

wirthjason 4y ago

Curious where PornHub and other sites rank. I always hear how that porn sites are in the top X of all traffics but people don’t talk about due to its nature. I’m always amazed that they have a data science team. It’s not something many would expect from the porn industry. I certainly didn’t expect it. https://www.pornhub.com/insights/2022-year-in-review https://www.pornhub.com/insights/2022-year-in-review

layer8 4y ago
A quick grep shows that there are almost 2.5K domains at 1M rank with “porn” in their name. The data science teams likely provide a considerable ROI in that industry.
- system2 4y ago
  Majority of porn sites do not have porn word in them though. I wish there was a categorization of these.
- Giorgi 4y ago
  does it though? Pretty sure adult results are being filtered off by Google tool named "SafeSearch". It removes anything adult from SERP and it is on by default.
  - yorwba 4y ago
    This appears to have some unintuitive consequences. When I searched for "porn" in a cookie-less session just now, there were still porn results, but no well-known sites (at least I didn't recognize the names). Searching for literally "pornhub", the first result is "porhub.com" without the "n". Seems like the "SafeSearch" filter is based on a list of "adult domains" instead of the indexed content at the URL.
E39M5S62 4y ago
If you ignore the content, large-scale adult sites are just like any other high traffic (bandwidth, RPS) site out there. A lot of planning goes into where their content delivery PoPs should be placed.

deterrence 4y ago

[flagged]

kristianp 4y ago

This raises the question: how much in the way of user telemetry does Chrome send back to google?

tgsovlerkhgsel 4y ago
By default, a lot. However, they also are (or at least used to be, it seems to be quite outdated now) really good at documenting their telemetry publicly: https://www.google.com/chrome/privacy/whitepaper.html https://www.google.com/chrome/privacy/whitepaper.html (I haven't checked whether the documentation is complete/accurate, of course.)
deleted 4y ago
[deleted]

modeless 4y ago

Wow, I'm kinda surprised to find my site in the top million worldwide. I have about 100k monthly visits as measured by Cloudflare web analytics, I guess that's all it takes.