19 ms·
Gmail is opening and caching URLs within emails without user intervention (2019)
- etaioinshrdlu 5y agoI always wondered when single-click unsubscribe was going to be a problem because of exactly this. I mean, how do you expect to give a URL to Google and have them just never crawl it?
- Xophmeister 5y agoOr “click this link to verify your e-mail address”…
- contravariant 5y agoWell the email address exists. If they wanted you to prove you can read the mail account they should send you a token or something, or at least require authentication if you click the link.
- meowster 5y agoThe verification link shouldn't be used to verify the email exists, it should be used to verify that the owner of the email address is the same person that signed up for whatever service. I have a common name gmail address, and I get verification emails all the time that I never open. If websites keep emailing me after that, then I rightfully mark them as spam.
- Xophmeister 5y agoThe verification is usually to prove the address is yours, as well as whether it exists.
- justinator 5y agoYou make just visiting the URL not everything that needs to be done. So for example, the URL you visit then also runs a small bit of javascript behind the scenes that does the actual unsub action - or the javascript just does a redirect. Or even simpler, you make it so the user has to click a button to POST the request. You've had to do this for years, now. I would assume though, that Gmail is smart enough to go, "oh hey, looks like a verification link, maybe I shouldn't touch it"
- ShakataGaNai 5y agoBut to be fair, 1-click unsubscribe is a very user friendly thing to do. As a user, if I have to jump through a bunch of hoops to unsubscribe, I'm just going to mark your message as spam and move on with my life.
- justinator 5y agoI absolutely agree. That's why on the app I wrote, the default is one-click unsubscribe, but I still have to do it using a little bit of Javascript. If JS is disabled, you just have to click a button that POSTs the request. I'm not sure what else to do! There's a similar but different problem with the reader-supplied "unsubscribe" button. This usually uses information found in the header of the email message - "List-Unsubscribe", but guess what also gets prefetched sometimes? Enter RFC8058 and, "List-Unsubscribe-Post"and another email kludge to throw on the pile, https://datatracker.ietf.org/doc/html/rfc8058 https://datatracker.ietf.org/doc/html/rfc8058
- Silhouette 5y agoExactly. And it's not just one-click unsubscribe. Using a secret link sent to an account's email address as a way to implicitly log in instead of having to remember a password is increasingly common and also an interesting idea in terms of user experience and security. If it's OK for your mail service to open one secret link, where does it stop? Is it also OK for them to spider the content they can reach from that link? Now they are potentially gaining access to all kinds of possibly sensitive information that they would not have been able to reach except for spying on your email. And if that's not OK, why was it OK for them to open the secret link in the first place?
- ev1 5y agoAlmost everywhere I see just prefills your email into a text box (or for the more asshole ones, doesn't) and provides a button to click. I rarely if ever see one click unsub.
- contravariant 5y agoRun some javascript on the page that sends a POST? Not ideal, but not sure how else to fix it, you can't send a POST with a single link as far as I can tell. Google could of course send the POST as well but then at least they're violating the HTTP standard.
- jsnell 5y agoAny security scanner product should also run the JS just like the real browser does. Otherwise cloaking the phishing page will be completely trivial.
- elif 5y agoone-click unsub is implemented by ignoring scrapers. Google is especially good about using a unique UA string
- Jach 5y agoAlso from my anecdotal experience it's possible google is a bit smart about unsubscribe links. My tiny blog has "unsubscribe" as part of its URL, it's truly one-click unsub and doesn't do anything clever like user agent checking or JS actions. Nevertheless, I see 8 gmails that still have email notifications on, and 3 that have unsubbed at some point.
- _wldu 5y agoThe access I saw to the registration URL was from cache.google.com and it looked like a client browser 74.51.221.37 - - [19/Aug/2021:22:05:16 +0000] "GET /validate/email/1d00a5c2648c211befd33f5a8a7cbfab HTTP/1.1" 404 0 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36" $ dig -x 74.51.221.37 +short cache.google.com.
- _wldu 5y agoI see other GETs like this one: 216.99.127.196 - - [20/Aug/2021:16:25:42 +0000] "GET /validate/email/2591b346e5b8b435bdde54d797fe23a9 HTTP/1.1" 200 811 "" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
- sp332 5y agoI thought the "one click" was counted from the web site, not from the email. So the click to get to the unsubscribe page doesn't count, but one click after that should do it.
- pwdisswordfish0 5y ago> I always wondered when single-click unsubscribe was going to be a problem To put succinctly what others are saying: it was never not a problem; it never should have been happening.
- AdrianoKF 5y agoThere's also RFC 8058 [0] that proposes to refine the `List-Unsubscribe` header for one-click unsubscriptions. It uses the `List-Unsubscribe-Post` header to indicate that an HTTP POST request can be used to unsubscribe with a single click. It specifically mentions in section 3.2 that mail receivers are not to crawl this URL without user consent: > The mail receiver MUST NOT perform a POST on the HTTPS URI without user consent. When and how the user consent is obtained is not part of this specification. I haven't seen any statistics on how widespread adoption of this RFC is among the major mail providers, though. [0]: https://datatracker.ietf.org/doc/html/rfc8058 https://datatracker.ietf.org/doc/html/rfc8058
- huhtenberg 5y agoOh, that's the rake we stepped on. Bloody obvious in retrospect, but it took us an embarrassingly long time to realize that we were leaking mailing list subscribers because of these one-click unsub links.
- inetknght 5y ago> it took us an embarrassingly long time to realize that we were leaking mailing list subscribers because of these one-click unsub links It didn't take some companies long. They were just a bit more, uhhh, shady about the knowledge.
- _wldu 5y agoI built a small Go web app to do some security testing. When a user registers for an account, I generate a 128-bit secure token and email it to the address they provided (as a URL). Token URLs look like this: /validate/email/1d00a5c2648c211befd33f5a8a7cbfab The token is cryptographically strong and disappears after access. It can't be guessed and no one but the email account holder should click it, but I am seeing the URL accessed multiple times from multiple IPs, so I investigated. Turns out, if the user provides a Gmail or Gsuite email account during registration, Google clicks the link. I was curious if others on HN had encountered this and how they dealt with it.
- whateveracct 5y agoMake the user take action after opening the link. Like click a button.
- _wldu 5y agoThanks. That's good advice.
- weird-eye-issue 5y agoBtw you could just have JS do a POST request, the user doesn't need to do anything except open the page. This is how unsubscribe pages work.
- kevin_thibedeau 5y agoThat presumes everyone executes random JS or has a browser that supports it.
- judge2020 5y agoThis is a good feature in my opinion. Why should I let the sender know when I click on tracking links or view the email? If you really want to, just filter out clicks from AS15169.
- alpaca128 5y agoSo this way Google automatically confirms the validity of the email to spammers by visiting all their links? Doesn't sound great, and people still know when you click on links or view the email. They just have to guess a bit better.
- SilverRed 5y agoThey already do this. If you send to an invalid address gmail will respond saying the email can not be delivered.
- jeroenhd 5y agoI disagree. I think the image and resource preloading done by Google is perfectly fine, but clicking actual links will mess up tons of systems (like, for example, links that are only supposed to be valid once, like in password reset emails). There are good use cases for links with single-use tokens in them. Companies sad and desperate enough to suck as much data from you as humanly possible (i.e. every single news letter with tracking links) ruin these use cases for everyone, assuming they are indeed the reason Google is implementing this feature. Given that many links (like, again, from password reset emails) give the person who clicks the link instant access to your account, I'd say this behaviour goes further than just filtering out trackers. Google has no business opening my Slack account or entering the change password page for the services I use. The stalking prevention they apply to external images and such is fine in my opinion, but links to external web pages should be left alone. You never know when Google accidentally clicks a link that says "confirm order" or "unsubscribe" because its magical AI misinterpreted the contents of an email.
- skybrian 5y agoFrom the very beginning of the web when HTTP was defined, there has been the rule that a GET request should never take an action on its own. Actions should be based on some other method like a POST request. Google is following the standard. People who take actions based on GET requests are not. Sure, mistakes happen out of ignorance, but they should be fixed.
- mike_d 5y agoAll URLs sent to any major email provider are "clicked" because they are scanning the page to see if it is phishing or otherwise malicious (desktop antivirus and other things will also prescan URLs). It also protects privacy by defeating click tracking on marketing emails. Google will also pre-load all the images in your email too. You shouldn't take any write action to your database just based on a URL being visited. Take them to the verification page and ask them to sign in or submit a form with the token pre-filled.
- dheera 5y ago> Google will also pre-load all the images in your email too PLEASE disable automatic loading in Gmail settings. Don't let the idiots use unethical, stalkerish e-mail read receipts.
- anamexis 5y agoDoesn't automatic loading render those read receipts meaningless?
- sp332 5y agoGmail only loads the images when you load the message. It's been that way since late 2013. https://arstechnica.com/information-technology/2013/12/dear-gmailer-i-know-what-you-read-last-summer-and-last-night-and-today/ https://arstechnica.com/information-technology/2013/12/dear-...
- rgbrgb 5y agoDoesn't gmail's preloading defeat the read receipts? It makes it so every tracking pixel sent to gmail gets loaded (and not by your IP), thereby making it meaningless.
- dheera 5y agoI thought gmail preloads only when the e-mail is opened. I thought it was basically just a proxy.
- acdha 5y agoThis sounds like they have built a unsafe system and are running into something which is checking for malware. This has been a problem for decades, which is why things like unsubscribe systems usually give you a form which requires you to submit it since a passive robot won’t POST it. I have little sympathy for the first poster: those kind of phishing tests are good if your goal is to train your users to think of the security group as an adversary but not much else. If clicking on one link compromises your security, you need to put the IT house in order first (hint: where’s the WebAuthn which completely?) and especially deal with the vendors who are training everyone to think that clicking on obfuscated links is routine.
- unilynx 5y agoA GET of a webpage shouldn’t have side effects - if it does, your app is broken. This might be new for Google, but some virus scanners have been prefetching links for decades.
- wcoenen 5y agoHTTP GET requests should not be interpreted by the server as a request to change something. That's what POST, PUT, DELETE and PATCH are for.
- justinator 5y agoI agree, but how do you initiate a POST request via an email message? Embedding a form sometimes raises its own security alert.
- dragonwriter 5y ago> I agree, but how do you initiate a POST request via an email message? Embedding a form sometimes raises its own security alert. So, your question is how to evade security alerts for actions with potentially significant side effects?
- justinator 5y ago> So, your question is how to evade security alerts for actions with potentially significant side effects? Well there's already a big ol' button in an email that says, "click me to register". The end user doesn't really care about the implementation. If one pops up a security alert (the POST form) and one doesn't (the simple link), how do you think everyone implements that big ol' button, 100% of the time, for 100% of everything? I wish email didn't work this way, but as far as I understand, this is the lay of the land. If there's a better way, I'll be happy to implement it in the system(s) that I have control over. I'm really asking for engagement within the community with help solving this sticky problem (if it wasn't clear). If link caching is this prevalent, what to do about it, for things like registering via email?
- SilverRed 5y agoYou can do it with js but that assumes that google is not running any js which is probably not a safe assumption.
- wcoenen 5y agoThe unsubscribe link should lead to a separate HTML form describing the unsubscription and a "confirm" button.
- getpost 5y agoIn Settings | General, there is an option, "Ask before displaying external images - This option also disables dynamic email." AFAIK this doesn't pertain to links, just images.
- ademup 5y agoI noticed this same behavior when sending a validation code link via SMS. Had to insert a "Verify Code" button landing page to prevent whatever link-sniffing was happening.
- hirundo 5y agoWhy isn't rel="nofollow" a solution to this? According to the docs [1] Use the nofollow value when other values don't apply, and you'd rather Google not associate your site with, or crawl the linked page from, your site. It seems like crawling a nofollow anchor tag in an email breaks this rule. Am I reading it wrong, is there an exception for emails, or is Google being inconsistent? [1] https://developers.google.com/search/docs/advanced/guidelines/qualify-outbound-links https://developers.google.com/search/docs/advanced/guideline...
- sp332 5y agoThat's for the search engine. This is for malicious link checking. Wouldn't do much good if every spammer could say "please don't look at my malicious web site".
- dragonwriter 5y ago> It seems like crawling a nofollow anchor tag in an email breaks this rule. There is no actual rule stated in the quoted material, and it describes it aa a mechanism for specifying a preference for how Google handles the link when Google encounters the tag on the creator’s site, which a user email on Gmail...isn't, even approximately.
- dragonwriter 5y ago[2019]
- deleted 5y ago[deleted]
- villgax 5y agoThere's no option to disable on mobile Gmail
- llimos 5y agoYou can (or used to be able to) embed an HTML form inside an email, and clicking the submit button will perform a POST if that is the method specified. This would probably solve the issue. You avoid an extra click on the resulting page, it's a POST which matches the standard, and Google (presumably) won't hit it.
- enlyth 5y agoWe have had the same problem with Microsoft where our marketing department is effectively DDoSing our service by sending out links to 50k+ users. We would get spikes of thousands of requests per second from Microsoft IP addresses, which after some googling were linked to their threat detection.
- ageofzfarm 5y agoThreat detection is becoming essential because of ransomware phishing and alikes, protection from click tracking is good too. Just send email in tranches. If the marketing department is sending too many e-mails for the web server to handle, then the volume is probably out of proportion to the company and what they are doing is probably just spam.
- enlyth 5y agoI agree, I'm not blaming Microsoft here, and it is our problem to solve (the service should handle the traffic, and/or the emails should be staggered over time)
- moonbug 5y agoWhat about the idempotency of HTTP GET is surprising here?
- megamix 5y agowhat does this mean?
- red_admiral 5y agoWasn't there an issue like this years ago in something called Google Web Accelerator (I think it was a toolbar for IE). Theory: it preloads links in the background. Practice: some old bb showed a (delete) link after each post and a (ban) link, among others, next to each user if you were logged in as administrator. All of these sent GET requests because some developer hadn't read that part of the standards, and there was no "are you sure?" prompt either. What I think is happening here is that gmail is scanning the content of each link in an e-mail for some subset of {malware, fraud, phishing, child abuse, other bad stuff}. This is a feature if you're a non-techy user who clicks on phishing links, I suppose?
- tmoertel 5y agoYes, we had this same scenario play out 16 years ago, and it seems that many modern web developers have forgotten its lessons: https://blog.moertel.com/posts/2005-05-06-google-web-accelerator-offers-web-developers-an-important-opportunity.html https://blog.moertel.com/posts/2005-05-06-google-web-acceler...
- dccoolgai 5y agoTrying to explain this to some marketing people looking at "click through rates" was extremely frustrating. This screwed with a lot of a/b testing analytics everywhere and I suspect most people were none the wiser.
- inetknght 5y ago> This screwed with a lot of a/b testing analytics everywhere My heart bleeds for the oh-so-poor marketing people whose data, consisting of unwitting human test subjects, has been poisoned.
- SevenSigs 5y agoIf only they could mind their own business...
- refurb 5y agoI thought this was intentional? I vaguely recall Google did this as a security feature to avoid invisible pixels?
- brtknr 5y agoIf this helps reduce spam by caching unsubscribe urls, I’m all in.
- sscarduzio 5y agoThe email should have a link to a page with a CAPTCHA challenge and a form that submits by POST. The POST with valid captcha solution testifies a human clicked purposefully.
- inetknght 5y ago> email should have a link to a page with a CAPTCHA challenge Hard no. CAPTCHA is a blight on the web to anyone who values privacy or has accessibility issues.
- ChrisArchitect 5y ago(2019). anything new on this? IIRC was a known thing and surely was already discussed on here somewhere then.
- ChrisArchitect 5y agowhy share this now? It's like a discussion from 2019 that traces back to posts google themselves made in like 2009 about preloading images, sending links thru their own servers etc, anti-phishing etc. why share this now/find a new discussion/link about it.
- midrus 5y agoI don't see anything wrong with this. GET request should be idempotent, and it protects me from bad actors abusing it to detect when I open an email.
- futhey 5y agoOne of my favorite side effects of this feature is that, Firebase Authentication, which is entirely run by another segment of Google, constantly throws errors about this to the user, which they don't understand. I get a support request at least once a week from a user who "Doesn't understand why they can't verify their email". Turns out, their Gmail account already verified it by clicking the link before they opened the email, and they didn't think to try signing in, because they couldn't even verify the email. ¯\_(ツ)_/¯
- Anunayj 5y agowait... isn't that messed up though..? Anyone can register a account with my email.
- _tom_ 5y agoNo one seems to be mentioning privacy. This would allow senders to verify email opens, something no email client should do.