6 ms·

Google releases stats on email encryption in transit

183 points by Matt_Cutts 12y ago

sp332 12y ago
Why is hotmail between 50% and 90%?
- deleted 12y ago
  [deleted]
- jonlarson 12y ago
  That's a good question. I've been involved with some research, outside of Google, on this exact topic. Because we don't have the same data sets that Google has, we have a slightly different methodology for gathering data and we found that hotmail (or any Microsoft mail provider) never even has the option to encrypt if you want to. We always though it was strange that they didn't, and this new data seems to suggest that there may be something else going on. One possibility that was already pointed out is heterogeneous architectures. It's possible that different systems in hotmail, maybe in different geographic regions or across national borders, use encryption.
izzydata 12y ago
I'm confused as to why anything isn't 100%. I guess I could maybe understand 0%, but how can it be 50%? I would imagine you either encrypt all email or you don't encrypt any email. What is the in-between?
- jeremyw 12y ago
  Large installations with heterogeneous clusters, accreted during many internal reigns and rounds of partially-integrated M&A.
- DiabloD3 12y ago
  Probably because it allows connections that aren't perfectly secure, such as supporting anything less than 256 bits.
  - Twirrim 12y ago
    That wouldn't explain why Google would have difficulty. It implies some kind of inconsistency in configuration or something on the servers Google's sending email to?
    - sjbach 12y ago
      Transit encryption requires that both sides are game. Google always supports it, but the server on the other side may not.
- Twirrim 12y ago
  With my small personal domain I was just using a self-signed cert until a few weeks ago, and I could see that some sending servers quit and sent in plain text after encountering a self signed cert, others delivered regardless.
- xs 12y ago
  Take Cisco.com for example. There are a lot of systems there that are sending out emails. Employees and newsletters are probably encrypted but only make up a small portion of their outbound mail. Things like ticket updates are going to result in a huge volume of email. Cisco may have (wrongly) decided that because the ticketing system is a trusted source it doesn't need to be encrypted. So some of their email is going directly out from the tool and some is going through a smart host. This is probably why it's <50%.
FiloSottile 12y ago
The 99.9% freak me out. The huge problem with email encryption right now is that you can't reasonably make it mandatory (as you can see, a lots of domain do not support it) so it can be easily be stripped à la sslstrip, and there is no such a thing as HSTS. (Again, please, build pinning into everything.) So an active attacker able to MitM can simply turn encryption off, and those 99.9% make me wonder if that happened to the 0.1%...
- magicalist 12y ago
  > The huge problem with email encryption right now is that you can't reasonably make it mandatory Many services can make it mandatory, as long as you're willing to take the hit to convenience, so at least your outbound email is always protected (and you get some level of canary for inbound email). This is usually for some compliance rule or another because they want enterprise business, so even the mainstream ones like Google Apps[1] and Office 365[2] have support for requiring TLS. [1] https://support.google.com/a/answer/2520500?hl=en https://support.google.com/a/answer/2520500?hl=en [2] http://technet.microsoft.com/en-US/library/jj723154%28v=exchg.150%29.aspx http://technet.microsoft.com/en-US/library/jj723154%28v=exch...
- 7952 12y ago
  That problem is quite common on the internet in general. Just think how common ftp still is! We constantly use public networks and then have to mitigate the fact that the packets are public. It would be better to treat encryption like a peering arrangement. When there is sufficient traffic between two large networks all packets should be routed via an encrypted tunnel.
  - Tiksi 12y ago
    While I generally agree with the concept, encryption isn't free, and will break pretty much every routing protocol used. It's easy enough to get a tunnel through transit when you control both ends, but neither the hardware nor the software that actually runs the peers supports it in any way. You'd have to set the link, then encrypt on top of that, costing a ton of cpu cycles, bandwidth overhead, and requiring something that can handle encrypting everything in real time and then pass it off to the actual edge connection. "Large" networks do 100Gbit plus, and I can't think of any way this is currently feasible to do.
guidedlight 12y ago
This is ironic, as Gmail has never supported email encrypted (or even signed) using S/MIME and digital certificates.
- infogulch 12y ago
  This post is referring to the secure transport of plain text emails between email providers. I.e. protecting from a mitm between gmail.com and hotmail.com when somebody sends a normal email from one to the other.
- mikecb 12y ago
  See the other gmail announcement today: https://code.google.com/p/end-to-end/ https://code.google.com/p/end-to-end/
  - higherpurpose 12y ago
    I still think DarkMail will be better than any PGP-based solution. But of course DarkMail doesn't actually "exist" yet, so I guess PGP wins by default for now.
Twirrim 12y ago
65% Out, 50% In overall.. that's actually a lot better than I was expecting to see.
liotier 12y ago
Funny that meanwhile they still won't support XMPP encryption...
- higherpurpose 12y ago
  Maybe because they're dropping support for XMPP altogether.
deepsun 12y ago
Paypal 0%
conectorx 12y ago
Fuck MitM attacks, they are recording keystrokes... lol
kijin 12y ago
I wonder what's happening on weekends? Every weekend, the percentage of encrypted outbound mail decreases, but the percentage of encrypted inbound mail increases. If you click on "90 days", you can see that this trend has been going on for quite some time. Could this be due to the difference between the usage patterns of work email vs. personal email? Or could it be a specific email service provider (or providers) who do something differently during weekends?
- chaz 12y ago
  On weekdays, lots of people using Gmail and Google Apps to send work emails to each other which are more likely to accept encryption. But on weekends, inbound commercial, non-work related emails (such as newsletters and promotions) drops off, leaving the percentage of personal emails higher.
e12e 12y ago
Interesting to see peaks on weekends. Guessing more private (person-to-person) traffic, so more traffic to/from domains that encrypt?
- dpcucoreinfo 12y ago
  During the weekdays there's more business communication, newsletters etc. from corporate servers that always lags behind in terms of featurfullness and security.
- Schwolop 12y ago
  I'm so pleased to see that (at time of writing) 4/35 comments are about this feature of the data. It's really encouraging to know that the HN audience is sufficiently academic to spot features like this, wonder about them, question them, and arrive at interesting insights. It leaves me keen to continue treating HN as one of my main new aggregators knowing that bullshit data and visualisations will be detected and treated as such!
Spooky23 12y ago
It's really cool to bring up an issue such as this, but... the positive impact of SMTP over TLS is limited and low value. Honestly, who gives a hoot that constant contact newsletters are sent in cleartext? I need to worry about where my organization's email goes, because quite simply, people do dumb things. Things like put payment information in email. Or PII subject to HIPPA or some other regulatory regime. Secure SMTP gives me no solace whatsoever, because even if I ensure that my receiving party's email gateway is secure, I have no assurance that any relaying happening behind that gateway is secure to the standards that I need to meet. For example, under many corporate security policies, MPLS tagged connections between offices is deemed secure -- they trust the telco. My organization requires an encrypted link or fiber owned and managed by the organization. IMO, you're better off operating under the assumption that your email is a postcard, because it is. Sensitive information should be shared in this order: in person, in a registered paper letter (the US government permits material up to the secret level via this means), first class mail, on the phone, via internal IT systems to other internal recipients, via homing pigeon, email.
imrehg 12y ago
I'm in Asia, and checked that segment.... pretty much 0% encrypted except for yahoo (but yahoo japan is unencrypted). I really wish they would be more forward thinking. Even if the network architecture is ahead in Asia, the services feel so much behind!
contingencies 12y ago
Given the number and state of the certificate barons and their document-as-stolen marbles, SSL is not really encryption though, is it? I thought this was going to be GPG stats. I bet Google do keep them... and I bet they have an uncanny correlation with people downloading their Gmail and leaving the service, or being well placed within the IT industry and vocal opponents of many of Google's recent policies.
rwg 12y ago
A fun experiment for someone who isn't afraid of having their ISP, colocation provider, or VM provider terminate their account and/or sue them because ZOMG LOTS OF OUTBOUND TCP/25 CONNECTIONS MEANS A SPAMMER IS YOU: • Take Google's list of mail domains they see (they offer a CSV file for download) and look up all of the mail exchangers for all of the domains. • Remove duplicates from the list of MX hostnames. • Write a short program that does the following for each of those mail exchangers: - Connect to tcp/25, send a valid EHLO, and see if the server indicates that it supports STARTTLS. - If it doesn't, disconnect and move on to the next hostname. - Send STARTTLS and start a TLS session. Put a NULL cipher at the top of your client's ciphersuite list to see if anyone bites. Record the certificate chain the server sends. - See if the leaf certificate is valid and if you can make it actually chain up to a trusted CA via the server's certificate chain. - Disconnect. You now have a list of some of the largest mail exchangers on the Internet, whether or not each one supports STARTTLS, and whether or not each one supplies a (valid) certificate that actually chains up to a CA anyone trusts. Write a blog post containing pretty graphs for these statistics. Poke fun at the most egregiously broken TLS configurations, certificates, and certificate chains you see. Idly wonder if the proliferation of (sometimes completely broken) opportunistic TLS with SMTP and the relative scarcity of authenticated TLS with SMTP will push passive attackers into becoming active attackers so they can continue to easily grab messages in-flight. Bemoan the broken CA system. Drink heavily. Dream.
- rakoo 12y ago
  I'm lost. Where is the "profit" step ?
  - dspillett 12y ago
    If it makes major providers start bothering with proper SSL configuration with a not-self-signed cert for their mail services maybe that would be a profit: things become slightly more secure for some people.
Bassetts 12y ago
So how do I manually check that my host supports this? I checked the list of domains and although I know I have sent mail to gmail address my host does not appear.
darkhorn 12y ago
I cannot see abv.bg and mail.bg

183 points by Matt_Cutts 12y ago

sp332 12y ago

Why is hotmail between 50% and 90%?

deleted 12y ago
[deleted]
jonlarson 12y ago
That's a good question. I've been involved with some research, outside of Google, on this exact topic. Because we don't have the same data sets that Google has, we have a slightly different methodology for gathering data and we found that hotmail (or any Microsoft mail provider) never even has the option to encrypt if you want to. We always though it was strange that they didn't, and this new data seems to suggest that there may be something else going on. One possibility that was already pointed out is heterogeneous architectures. It's possible that different systems in hotmail, maybe in different geographic regions or across national borders, use encryption.

izzydata 12y ago

I'm confused as to why anything isn't 100%. I guess I could maybe understand 0%, but how can it be 50%? I would imagine you either encrypt all email or you don't encrypt any email. What is the in-between?

jeremyw 12y ago
Large installations with heterogeneous clusters, accreted during many internal reigns and rounds of partially-integrated M&A.
DiabloD3 12y ago
Probably because it allows connections that aren't perfectly secure, such as supporting anything less than 256 bits.
- Twirrim 12y ago
  That wouldn't explain why Google would have difficulty. It implies some kind of inconsistency in configuration or something on the servers Google's sending email to?
  - sjbach 12y ago
    Transit encryption requires that both sides are game. Google always supports it, but the server on the other side may not.
Twirrim 12y ago
With my small personal domain I was just using a self-signed cert until a few weeks ago, and I could see that some sending servers quit and sent in plain text after encountering a self signed cert, others delivered regardless.
xs 12y ago
Take Cisco.com for example. There are a lot of systems there that are sending out emails. Employees and newsletters are probably encrypted but only make up a small portion of their outbound mail. Things like ticket updates are going to result in a huge volume of email. Cisco may have (wrongly) decided that because the ticketing system is a trusted source it doesn't need to be encrypted. So some of their email is going directly out from the tool and some is going through a smart host. This is probably why it's <50%.

FiloSottile 12y ago

The 99.9% freak me out. The huge problem with email encryption right now is that you can't reasonably make it mandatory (as you can see, a lots of domain do not support it) so it can be easily be stripped à la sslstrip, and there is no such a thing as HSTS. (Again, please, build pinning into everything.) So an active attacker able to MitM can simply turn encryption off, and those 99.9% make me wonder if that happened to the 0.1%...

magicalist 12y ago
> The huge problem with email encryption right now is that you can't reasonably make it mandatory Many services can make it mandatory, as long as you're willing to take the hit to convenience, so at least your outbound email is always protected (and you get some level of canary for inbound email). This is usually for some compliance rule or another because they want enterprise business, so even the mainstream ones like Google Apps[1] and Office 365[2] have support for requiring TLS. [1] https://support.google.com/a/answer/2520500?hl=en https://support.google.com/a/answer/2520500?hl=en [2] http://technet.microsoft.com/en-US/library/jj723154%28v=exchg.150%29.aspx http://technet.microsoft.com/en-US/library/jj723154%28v=exch...
7952 12y ago
That problem is quite common on the internet in general. Just think how common ftp still is! We constantly use public networks and then have to mitigate the fact that the packets are public. It would be better to treat encryption like a peering arrangement. When there is sufficient traffic between two large networks all packets should be routed via an encrypted tunnel.
- Tiksi 12y ago
  While I generally agree with the concept, encryption isn't free, and will break pretty much every routing protocol used. It's easy enough to get a tunnel through transit when you control both ends, but neither the hardware nor the software that actually runs the peers supports it in any way. You'd have to set the link, then encrypt on top of that, costing a ton of cpu cycles, bandwidth overhead, and requiring something that can handle encrypting everything in real time and then pass it off to the actual edge connection. "Large" networks do 100Gbit plus, and I can't think of any way this is currently feasible to do.

guidedlight 12y ago

This is ironic, as Gmail has never supported email encrypted (or even signed) using S/MIME and digital certificates.

infogulch 12y ago
This post is referring to the secure transport of plain text emails between email providers. I.e. protecting from a mitm between gmail.com and hotmail.com when somebody sends a normal email from one to the other.
mikecb 12y ago
See the other gmail announcement today: https://code.google.com/p/end-to-end/ https://code.google.com/p/end-to-end/
- higherpurpose 12y ago
  I still think DarkMail will be better than any PGP-based solution. But of course DarkMail doesn't actually "exist" yet, so I guess PGP wins by default for now.

Twirrim 12y ago

65% Out, 50% In overall.. that's actually a lot better than I was expecting to see.

liotier 12y ago

Funny that meanwhile they still won't support XMPP encryption...

higherpurpose 12y ago
Maybe because they're dropping support for XMPP altogether.

deepsun 12y ago

Paypal 0%

conectorx 12y ago

Fuck MitM attacks, they are recording keystrokes... lol

kijin 12y ago

I wonder what's happening on weekends? Every weekend, the percentage of encrypted outbound mail decreases, but the percentage of encrypted inbound mail increases. If you click on "90 days", you can see that this trend has been going on for quite some time. Could this be due to the difference between the usage patterns of work email vs. personal email? Or could it be a specific email service provider (or providers) who do something differently during weekends?

chaz 12y ago
On weekdays, lots of people using Gmail and Google Apps to send work emails to each other which are more likely to accept encryption. But on weekends, inbound commercial, non-work related emails (such as newsletters and promotions) drops off, leaving the percentage of personal emails higher.

e12e 12y ago

Interesting to see peaks on weekends. Guessing more private (person-to-person) traffic, so more traffic to/from domains that encrypt?

dpcucoreinfo 12y ago
During the weekdays there's more business communication, newsletters etc. from corporate servers that always lags behind in terms of featurfullness and security.
Schwolop 12y ago
I'm so pleased to see that (at time of writing) 4/35 comments are about this feature of the data. It's really encouraging to know that the HN audience is sufficiently academic to spot features like this, wonder about them, question them, and arrive at interesting insights. It leaves me keen to continue treating HN as one of my main new aggregators knowing that bullshit data and visualisations will be detected and treated as such!

Spooky23 12y ago

It's really cool to bring up an issue such as this, but... the positive impact of SMTP over TLS is limited and low value. Honestly, who gives a hoot that constant contact newsletters are sent in cleartext? I need to worry about where my organization's email goes, because quite simply, people do dumb things. Things like put payment information in email. Or PII subject to HIPPA or some other regulatory regime. Secure SMTP gives me no solace whatsoever, because even if I ensure that my receiving party's email gateway is secure, I have no assurance that any relaying happening behind that gateway is secure to the standards that I need to meet. For example, under many corporate security policies, MPLS tagged connections between offices is deemed secure -- they trust the telco. My organization requires an encrypted link or fiber owned and managed by the organization. IMO, you're better off operating under the assumption that your email is a postcard, because it is. Sensitive information should be shared in this order: in person, in a registered paper letter (the US government permits material up to the secret level via this means), first class mail, on the phone, via internal IT systems to other internal recipients, via homing pigeon, email.

imrehg 12y ago

I'm in Asia, and checked that segment.... pretty much 0% encrypted except for yahoo (but yahoo japan is unencrypted). I really wish they would be more forward thinking. Even if the network architecture is ahead in Asia, the services feel so much behind!

contingencies 12y ago

Given the number and state of the certificate barons and their document-as-stolen marbles, SSL is not really encryption though, is it? I thought this was going to be GPG stats. I bet Google do keep them... and I bet they have an uncanny correlation with people downloading their Gmail and leaving the service, or being well placed within the IT industry and vocal opponents of many of Google's recent policies.

rwg 12y ago

A fun experiment for someone who isn't afraid of having their ISP, colocation provider, or VM provider terminate their account and/or sue them because ZOMG LOTS OF OUTBOUND TCP/25 CONNECTIONS MEANS A SPAMMER IS YOU: • Take Google's list of mail domains they see (they offer a CSV file for download) and look up all of the mail exchangers for all of the domains. • Remove duplicates from the list of MX hostnames. • Write a short program that does the following for each of those mail exchangers: - Connect to tcp/25, send a valid EHLO, and see if the server indicates that it supports STARTTLS. - If it doesn't, disconnect and move on to the next hostname. - Send STARTTLS and start a TLS session. Put a NULL cipher at the top of your client's ciphersuite list to see if anyone bites. Record the certificate chain the server sends. - See if the leaf certificate is valid and if you can make it actually chain up to a trusted CA via the server's certificate chain. - Disconnect. You now have a list of some of the largest mail exchangers on the Internet, whether or not each one supports STARTTLS, and whether or not each one supplies a (valid) certificate that actually chains up to a CA anyone trusts. Write a blog post containing pretty graphs for these statistics. Poke fun at the most egregiously broken TLS configurations, certificates, and certificate chains you see. Idly wonder if the proliferation of (sometimes completely broken) opportunistic TLS with SMTP and the relative scarcity of authenticated TLS with SMTP will push passive attackers into becoming active attackers so they can continue to easily grab messages in-flight. Bemoan the broken CA system. Drink heavily. Dream.

rakoo 12y ago
I'm lost. Where is the "profit" step ?
- dspillett 12y ago
  If it makes major providers start bothering with proper SSL configuration with a not-self-signed cert for their mail services maybe that would be a profit: things become slightly more secure for some people.

Bassetts 12y ago

So how do I manually check that my host supports this? I checked the list of domains and although I know I have sent mail to gmail address my host does not appear.

darkhorn 12y ago

I cannot see abv.bg and mail.bg