20 ms·
I built a small Go web app to do some security testing. When a user registers for an account, I generate a 128-bit secure token and email it to the address they
by _wldu 5y ago
I built a small Go web app to do some security testing. When a user registers for an account, I generate a 128-bit secure token and email it to the address they provided (as a URL). Token URLs look like this:
/validate/email/1d00a5c2648c211befd33f5a8a7cbfab
The token is cryptographically strong and disappears after access. It can't be guessed and no one but the email account holder should click it, but I am seeing the URL accessed multiple times from multiple IPs, so I investigated.
Turns out, if the user provides a Gmail or Gsuite email account during registration, Google clicks the link. I was curious if others on HN had encountered this and how they dealt with it.
- whateveracct 5y agoMake the user take action after opening the link. Like click a button.
- _wldu 5y agoThanks. That's good advice.
- weird-eye-issue 5y agoBtw you could just have JS do a POST request, the user doesn't need to do anything except open the page. This is how unsubscribe pages work.
- kevin_thibedeau 5y agoThat presumes everyone executes random JS or has a browser that supports it.
- aendruk 5y agoIt should go without saying that you provide a fallback.
- input_sh 5y agoCreate a "click here" button, hide it by adding it a class or a style with JS.
- kevin_thibedeau 5y agoThere is this thing called an HTML form. Every browser supports it.
- bawolff 5y agoDepending on context and implementation details, this can often be a security issue (csrf or something similar). Probably not in the unsubscribe case though
- weird-eye-issue 5y agoObviously you need to make sure your API is not susceptible to CSRF but that goes without saying... Should I also tell him to password protect his database? :P
- bawolff 5y agoIf you follow what the parent is suggesting (open a page with a get request that has js which does a post request automatically with no user interaction), its probably impossible to not be susceptible to csrf
- weird-eye-issue 5y agoI'm really not sure why you think that, but that's just not true at all. For starters, if it is a JSON API then a CORS request will be done for all cross-domain API requests. Even a missing CORS configuration would then block the CORS request from third-party domains.
- bawolff 5y agoSo the situation is: there is some url you open (with a normal get request. Typically but not neccesarily from an email), then that url does the non-idempotent POST request without any futher user interaction. A malicious page that knows the url used in the email could open the url from the email in a popup. The js will execute in the popup, and do the POST request. It doesn't matter how much csrf protection you have on the POST step, if anyone can trigger it with no user interaction just by opening some page with a GET request.
- weird-eye-issue 5y ago
- _wldu 5y agoThanks. I don't use JS, just Go with HTML templates. I populate the form now with {{ .code }} from the URL so the user does not have to copy/paste the code. But they do have to click 'Submit' to post the form. I think this is a reasonable approach that most users are OK with.
- weird-eye-issue 5y agoYou could add a single line of JS to have it auto submitted as well
- _wldu 5y agoWouldn't that be equivalent to just doing a GET?
- SamBam 5y agoNo, because if you curl the original url the POST for the second url won't be triggered, but if you navigate your browser to the url, the browser will trigger the POST.
- weird-eye-issue 5y agoNo... Because you have to do a GET, execute JavaScript, and make a POST request. Bots don't execute JavaScript and no well-behaved bots are going to make POST requests
- sxp 5y agoAnd make sure the action is a POST instead of a GET. GETs should never modify important state.
- darig 5y agoEmail should never rely on HTML forms that can POST.
- sleavey 5y agoThis is the correct answer. Just because the norm is to embed verification hashes in URLs to be clicked, doesn't mean it's the right way for it to be done. Why not send a short random code by email for the user to then copy into the sign-up form they were in the process of filling in?
- vbezhenar 5y agoIt takes more effort and more users will decide to move elsewhere. I don't really believe that if someone can't bother to copy code from e-mail, he's worthy to have as a client, but some company are obsessed by metrics and percentage of successfully registered users is one of those metrics.
- toshk 5y agoI understand your way of thinking, but we ended up having a flow for a government site where users had 2-3 steps what normally could be done in 1. Also many were not tech savvy and confused. So we ended up adding JS to automate the click.
- inetknght 5y agoYou automated a click on a government website? So tell me: how'd that audit go?
- acdha 5y agoThis depends on your willingness to turn away business, and may not even be legal depending on where you work. In the United States, I would not want to defend that copy and paste scheme as being compliant with the Americans With Disabilities act having seen usability tests from people trying to accomplish that exact workflow using screen readers. Remember that things like cognitive impairments count and, like vision and motor control/range of motion, most of us will be affected at some point in our lives. What I do think would be reasonable is having a well-labeled link which takes you to a confirmation form: someone can follow it easily and choose to submit it with far less friction and it leaves standard web semantics intact.
- deleted 5y ago[deleted]
- bradknowles 5y agoMake the link password protected, and they don’t get the password in the same e-mail message. Kinda hard to pre-scan a URL if you can’t provide the password for it.
- technion 5y agoI have an email from 2015 where I reported this as a potential vulnerability in the Hartl Rails tutorial, having seen it myself back then. Consider "verified" ashley madison accounts in their breach and this scenario. This is not just a Gmail thing. Most corporate mail filters visit a link and scan for malware as a feature.
- dragonwriter 5y agoYay for building systems that rely on extremely non-idempotent behavior on GET.
- mtwittman 5y agoConsider a different approach to mitigate automated URL fetching interference (this can apply to both email ownership verifications and password resets). Make the emailed verification/reset link (GET request) idempotent (1 and >1 request has the same effect). Have the link just present an interface for the user to take the next step. In the next step make a POST request that actually commences your verification/reset process. In all likelihood you'll want expiry logic (let's say it's 30 minutes) - if you store the token with a created_at timestamp on the server you can have your verification/reset process check that now < (created_at + 30 minutes) If expired, provide a UI for the user to request a fresh verification/reset email.
- paxys 5y agoI'm sure Google uses a specific user agent to make a request, so you can filter that out. A better solution is to assume that some middleman (email server or client) will always try to access links in the email. Instead send the user a code and have them manually enter it on the linked page.
- Doxin 5y agoOr link them to a page with a POST form that actually performs the action. That way you only add a single click to the flow, and no remotely sane software will automatically perform POST requests to arbitrary urls.
- kuu 5y ago> no remotely sane software will automatically perform POST requests to arbitrary urls I'm not a web developer. Out of curiosity, why is that?
- motogpjimbo 5y agoFor precisely the reason being discussed here: GET requests can be performed automatically for many reasons. For example, if you've ever pasted a URL into a Slack channel (or similar) and seen the link converted into a thumbnail of the page a few moments later, you've seen a piece of software issue a GET request on your behalf. Now imagine that wasn't a link to a page but a link to an something that modified your account - resetting your password, for example.
- karolist 5y agoPOST requests typically perform modifications on the server based on user action, like POST'ing this comment. GET requests should be idempotent.
- Doxin 5y agoWebsites are generally designed such that GET requests are side-effect free, and POST requests do have side effects. So for example searching on google is a GET request. It doesn't do anything other than serve the requested page. Logging in on the other hand has the side effect of setting cookies and probably writing some stuff to the database, so that's a POST request. These assumptions are so baked into web software that while assuming a GET request won't do anything zany or overly stateful is probably fine, assuming the same for a POST request should probably be considered negligent.
- avereveard 5y agoI've had a lot of grief from a few users' Exchange doing it (likely as part of some anti phishing plugin of sorts), to the point we changed validation links from one time to sort lived.
- raxxorrax 5y agoDon't know why you are downvoted. Many corporations and institutions employ sandboxes to check mails and the links contained in them. This is a standard security practice by now. So a link that is only valid once would be affected. Restricting the validity by time is a good way to solve this while still maintaining decent security.
- parineum 5y agoDoes gmail respect robots.txt?
- LinuxBender 5y agoNot for this use case or crawling. They won't even hit robots. I've caught google, discord, valve, slack doing this on my hobby sites over the years, likely checking to see if the target URL contains obvious malware. In my case the solutions were simple. Add simple auth and/or block IP ranges associated with their AS number. Obviously this isn't the solution for companies, though you could have a unique domain specifically for email URL's and decide what limitations to put in place. Blocking them can flag your site as "malicious" and I am perfectly ok with google saying my domains are malicious.
- smcl 5y agoYes and outlook 365 too. We had to add an extra step to our activation process to handle this (a prompt to click a link to proceed). I would not be surprised if they start making their link-follower start clicking around inside opened pages too :-/
- toshk 5y agoYeah lots of email clients do this. Next to caching also lots of scanners for malicious content. We solved it by having a screen with a confirmation button , then later we added javascript to show a loader page over the button and click the button automatically.
- vs4vijay 5y agoThat's why it should not be HTTP GET endpoint. Get endpoint should only be when request is idempotent. Use HTTP POST for your usecase.
- mnahkies 5y agoIs it possible to embed a link that uses POST in an email? I can't think of a way unless form tags work, but then the link wouldn't work in a plain text email reader
- harg 5y agoYou’d need to send the user to the verify page and populate a form with their token from the url. Then submit the form, either automatically or by getting the user to manually hit a button.
- timwis 5y agoThis is how mail chimp does it, I believe. JavaScript to submit the form automatically.
- miken123 5y agoYou should not submit the form automatically. Tools like Microsoft O365 ATP run any links in an emulated browser with Javascript support. These will, in some cases, happily autosubmit the form for you.
- lazyjones 5y agoI presume robots.txt is still ignored in this case?
- BiteCode_dev 5y agoWe have the issue with 0bin.net burn after reading, so we give a grace period to the link after creation. It works decently, but I'm thinking of just displaying a page with a decrypt button on those, so that you need a post request to actually read the content and trigger the delete.
- wodenokoto 5y agoI believe this is how they fetch images without meaningfully accessing tracking pixels. If everything send to gmail is opened upon arrival and cached, you know nothing about when or if the recipient actually opened the email.
- upofadown 5y agoLast time I looked into this, Gmail was not loading and caching images. Is there any evidence that this has changed? What is being described here is likely being done for some other purpose.
- wodenokoto 5y agoGoogle claimed to do that. > Instead of serving images directly from their original external host servers, Gmail will now serve all images through Google’s own secure proxy servers. Https://gmail.googleblog.com/2013/12/images-now-showing.html
- upofadown 5y agoProxy servers, not caching servers.
- bradknowles 5y agoWhen are proxy servers not caching?
- iamdual 5y agoYou need to authenticate the user before the activation.
- kyrra 5y agoThis. You could rely on a cookie during the get request as well, that you set on the users browser during registration. Or re-auth after click.
- simonw 5y agoThe problem with that is people like myself who tend to register on a laptop but then click the email verification link on their mobile phone. (Because waiting for Gmail to load on a laptop is painful, whereas on my phone is shows up as a push notification within seconds)
- winrid 5y agoWhat about magic links? :)
- SevenSigs 5y agoI had the same issue with Microsoft's email service and Facebook messages. How I dealt with it was to not email private links... I use Element these days or email links to encrypted files in some circumstances. I wish websites would stop using email and phone verifications...
- raxxorrax 5y agoThat is a security risk that Google is causing here. While I agree that URLs shouldn't necessarily be used to store secrets, the usual password reset mail is nothing else and the mechanism has merit. https://www.w3.org/TR/capability-urls/ https://www.w3.org/TR/capability-urls/ It is also a good way to communicate between two parties that don't want to have user account in any service, we constantly request input from B2B customers by providing forms with a capability URLs. An no, we don't want to use an identity provider. Maybe good ones like auth0. Amazon Cognito is pretty decent in my opinion, but Amazon is also big tech. Industrial espionage is something real for that matter. We have mail providers that respect privacy, just saying... I don't understand the love for Gmail at all, especially when you use a mail client, which I would heavily recommend to everyone. Ironically a lot of security scanner also do follow links. Understandable, but I just hope they don't plaster the logs too much...
- jbverschoor 5y agoA "GET" request is not supposed to alter state. On the other hand. It validates the email address more quickly, so you could even refresh/poll when it's verified automatically
- the_gipsy 5y agoIf there is a cookie session, 99% of GETs do alter the state.
- kosinus 5y agoI’ve also seen anti-virus do this, though I don’t remember which brand.
- kubafu 5y agoYou can also check for various headers to determine (with quite good accuracy) if a link was clicked by a human or fetched programatically. Here's a list I've accumulated over the years for virtually the same feature as yours: - `sec-fetch-dest` header is present (HUMAN) - `accept` header is present (HUMAN) - `from` header is bingbot(at)microsoft.com (AUTOMATED) - `user-agent` header includes BingPreview (AUTOMATED) HTH
- raxxorrax 5y agoAlso most humans do use browsers, so if you don't have any following requests for resources like scripts, images or just the favicon, you probably got visited by a bot.
- inetknght 5y agoNot all humans use browsers that issue requests for additional resources. I have my browser configured to retrieve the page only and no additional requests for CSS, images, or javascript.
- obuda 5y agoAnother idea is to have 3 links, where only one is visible: https://example/com/token?forBots https://example.com/token https://example.com/token?forBots Hopefully any automated systems will open the first or last link first, so that you can save the request info and filter based on that. In case requests come out of order, you can always add a small delay to the "human" link before responding. I haven't yet gotten to implementing any of the authentication on my current project, so I might be missing something really basic. The next best thing is to set a cookie when requesting the magic link, but the downside (or upside?) is that it will be valid only for the browser it was requested with.
- zenexer 5y agoNo link in an email should perform an action on its own. Every link should lead to a confirmation button, at minimum. Too many services automatically open all the links in emails.
- SamBam 5y agoTons of services send a verification link after registration, and when you click the link you are taken to a page that says "You're verified." But in those cases there may be an automatic POST after you travel to the link, so it wouldn't be triggered by gmail looking up the url.
- crabmusket 5y agoThis may be for the purpose of ensuring the email address itself is deliverable. You don't want someone to sign up with random garbage, then try sending notifications, newsletters, etc. to it- I believe doing so can affect domain reputation. For this use-case, it seems like even an automated link click would be a good signal of a deliverable email address.
- ziml77 5y agoNot just deliverable, but also that it's correct. There's a lot of people who think that my {firstname}{lastname}@gmail.com email address is their own. If they try to register it somewhere, a verification email stops them from completing the registration.
- nickjj 5y agoThis is interesting because it's something that would likely only happen in production or a staging server. If you're building your web app in development, chances are your links will have localhost as their hostname which wouldn't trigger a visit from Google. You may also end up having an in memory fake email server to not even send the email in dev too (lots of web frameworks have solutions for this). Checking the user-agent might work but I'm not a fan of this method because now it sets you up with having to keep a list of all known agents for every email client / service that might pre-visit URLs.
- haolez 5y agoIsn't there a way to verify that the click is coming from GMail? Maybe via User Agent or its IP.
- _wldu 5y agoThe user agent I saw looks like a normal client that a person might use: 74.51.221.37 - - [19/Aug/2021:22:05:16 +0000] "GET /validate/email/1d00a5c2648c211befd33f5a8a7cbfab HTTP/1.1" 404 0 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36" I suppose you could somehow block cache.google.com but I suspect Microsoft and others do similar things.
- eplanit 5y agoThis is like the reason I quit using Skype 10 years ago. My colleagues and I noticed the same thing: send a link in a chat, and within seconds to minutes a request (or more) for that URL from a Microsoft server would be logged. Done and bye.
- obuda 5y agoWouldn't be surprised if pretty much every communication service is doing this. I sent a link to a large file over Viber and immediately some ip connected and started downloading. Stopped at 350mb of around 3.5gb. I get that they want to show thumbnails or whatnot, but they just don't discriminate between content types.
- t00 5y agoThat is a great way to do a cross-link DDoS of unsolicited link opening services - send 1000 messages with google links on skype and send 1000 gmail messages with microsoft links there, all gigabytes in size...
- eyelidlessness 5y agoDoes Skype generate preview images? Most chat clients do at this point. They all have to access the link to get the relevant metadata to do that.
- eplanit 5y agoInteresting "feature": the user perceives a (real) benefit: preview. But, there is also an unspoken benefit for Microsoft, who can feed their marketing analytics, or AI training data, or ... with what it learns via the chats and links. I'd be fine if the fine print in the EULA provides a guarantee that the feature scans content solely for generating previews, and that M$ keeps no copy of it, etc....But, I'm sure I'd go blind looking for such text in the EULA.
- rrjanbiah 5y agoHere's a quick PoC: <link rel="prefetch" href="/actual_validate/email/1d00a5c2648c211befd33f5a8a7cbfab?prefetch=1"> <script> location.href = "/actual_validate/email/1d00a5c2648c211befd33f5a8a7cbfab?js=1"; </script> <noscript> <a href="/actual_validate/email/1d00a5c2648c211befd33f5a8a7cbfab?js=0" rel="nofollow" class="btn btn-primary" role="button">Click to confirm your account</a> </noscript>
- obuda 5y agoWhy the prefetch though? Reason being bots don't open them? If so, this is a really good idea!
- eyelidlessness 5y agoIt’s likely faster for the noscript case.
- obuda 5y agoYeah, I don't know about that. Wouldn't it be that the query strings differentiate the two links? I assume so, because of an old trick where query strings are used for ad-hoc cache control as in /style.css?1629472765
- rrjanbiah 5y agoQuery string is to differentiate the links (to understand which case is getting triggered)
- obuda 5y agoOK, but then the prefetch link as it stands is useless, no?
- eyelidlessness 5y agoYou’re right, I wasn’t paying close enough attention. That’s what I get for reading HN mostly on my phone!
- eu 5y agoWe've seen something similar with one of the email campaign services. The Unsubscribe links were "clicked" within a minute of emails being sent. Other services show you a message and process the unsubscribe on a POST request.
- winrid 5y agoMake the link expire after a certain period of time and not after first click.