12 ms·
Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks. If Bing has not seen t
by bigtones 4y ago
Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks. If Bing has not seen the web page before and it's not in the Bing dangerous web page index it first needs to check it to make a determination of if it's a phishing/malware page by scanning/indexing it before returning that outcome back to Outlook to flag the email as dangerous.
- rsbadger 4y agoScanning something for malware and publishing it in search results seem like 2 completely different things to me...?
- Xylakant 4y agoBut there is nothing to indicate either in the post or in the referenced SO thread that the URLs are published to the search results. They are visited by bingbot, that much seems confirmed, but there’s no example where one of these results shows up in the public search results.
- rsbadger 4y agoThey were indexed in Bing results, I’ve shared the URL of that in this thread
- cma 4y agoHoly shit thats bad. Do unlisted youtube and gdrive share links get indexed through this?
- 9dev 4y agoYou’d assume those have proper robots.txt configuration?
- boxed 4y agoI have a disallow all robots.txt for a production system. Have had from the beginning. Bing indexes it. This is my first major security incident and I have no idea how to fix this without making everything totally shitty for the users.
- ratg13 4y agoYes, but there is no indication they are publishing it in the search results. The original post is just complaining that the malware scanning is visiting the links. They come to the following conclusion >This effectively makes all one-time use links like login/pass-reset/etc useless. Which we all know is not true because sites like onetimesecret.com allow for entering a separate password to prevent this sort of thing when it does happen. It would be an interesting discussion to talk about what Microsoft's whitelisting process looks like, but the original article doesn't seem to understand what is going on well enough to drive the conversation in that direction.
- nopassrecover 4y agoThey are publishing them - it has bitten us (e.g. expired one click links for customers ending up on Bing from their emails)
- ratg13 4y agoIf you can say for certain that the links being published are coming from the malware scanning, and not being taken from users' browser sessions that are using Microsoft Edge you should elaborate on this.
- boesboes 4y agoThat would be even worse.
- ratg13 4y agoNobody is saying it isn't. It's about trying to get to the core of the issue, not just the random speculation going on in the article and in this comment thread.
- rsbadger 4y agoI would be pretty mortified if browsers were using user browser sessions to scan content and pass it to bingbot…? What about if you’re browsing something local? Or your bank account?
- usrn 4y agoI just love it when they "scan" password reset links.
- noisem4ker 4y agoThe HTTP GET method is idempotent by specification. Visiting a webpage should not trigger password resets or any other actions by itself. If that's a problem then it's the site's fault for being defective.
- rsbadger 4y agoYou’re right that it was a bit of an oversight on my behalf, as the links were only generated after a verified human user action (signup) I had assumed the 1 time links to their email would be safe. But regardless of the link action, it shouldn’t be passing that data to Bingbot to crawl and (possibly) index in search engine results. Private email data should not be shared with search engine crawlers IMO.
- petercooper 4y agoSo how do you implement a "one click unsubscribe" link in an email? They're on GET requests. You could use JavaScript on the resulting page to then trigger the unsubscribe but bots are now running JavaScript as well.
- noisem4ker 4y agoYou show a webpage with an "Unsubscribe" button in it. The button triggers a POST request. There's also RFC 8058: https://datatracker.ietf.org/doc/html/rfc8058 https://datatracker.ietf.org/doc/html/rfc8058
- AtNightWeCode 4y agoThat should send one to a page with a confirmation button...
- causi 4y agoWouldn't it be trivial to keep the list of malicious pages locally and not send any data?
- morley 4y agoIf I were designing a system like this, I would not trust clients to perform legitimate analysis nor report legitimate results.
- causi 4y agoWhat? But you're the one writing the client.
- sofixa 4y agoDoesn't matter. Never trust the client - it's outside of your control, it can be patched, it can be hacked, it can be spoofed, etc.
- BoorishBears 4y agoLittle understanding: Undying trust of the client Dunning-Kruger level of understanding: Never trust the client for anything ever, it's unreliable, everything must be off client. Never mind the client is literally the interface into your system, so it being compromised is already game over for an application where the user is most vulnerable party you wanted to protect... Deep understanding: Trusting the client requires a well thought out security model. If the client is hacked in this case, they already have full control over what the user sees, they can cut out your remote check. Maybe a good balance would be to hash the root of the URLs and compare those, or use fuzzy hashing on page contents, just so that the backend isn't getting a bunch of private urls that might accidentally get logged somewhere. Trades detecting stuff hidden behind redirects for less liability on your backend, something to possibly consider depending on functional requirements.
- shepherdjerred 4y agoIt sounds like you’re advocating for no client at all
- sitkack 4y ago
- ShowalkKama 4y agoscanning with bing useragent? That's not a good idea.
- _8j50 4y agoIt is common for corporate email security appliances as well. URLs should not be used for authentication neither should email. I really want to pick brains of people that work on these types of systems to see why they don't think so.
- rsbadger 4y agoMany people (most?) prefer to signup to services by email address. To do so, those email addresses must be verified. How would you verify it without sending them an email link?
- throwaway14356 4y agoa confirmation code? Also, mail might not live on the same computer.
- xboxnolifes 4y agoIt doesn't matter if it's on the same computer. Sometimes all you need to do is click the link, not do anything on the page.
- _8j50 4y agoYou can verify validity of an email like that, no issue there. Just don't use that as a factor authentication. Control over an email account should not trump passwords (what you know) or proper 2fa (what you have, typically, email can be 2fa like sms and like sms it is not a good choice). If a person proves they control an email account then you ask them for additional info like secret questions or other information configured during registration. I should not be able to take over your life because I compromised your phone which has sms, TOTP app and email.
- inopinatus 4y agooptions include: * use an interstitial page so that the actual activation is a POST request; * send a confirmation code instead of a link
- 4y ago
- sofixa 4y ago> Microsoft does this because they're security scanning / checking all links in every Outlook email for known phishing and malware attacks The problem with that is that the logic is broken. Microsoft cannot possibly know all phishing sites, especially for smaller things. By obfuscating the link the user can no longer verify it by themselves without clicking, but Microsoft will say it's safe. So the user is left with a false sense of security and are worse off. It only works for huge sites ( e.g. mytwitter.lol phishing for twitter and similar), but drastically lowers the chance of less high profile phishing being caught.
- tomp 4y agoMaybe you should implement a "feature" that serves a simple static HTML page <p>This webpage is safe.</p> to "bingbot" and serve the real page to everyone else.
- phendrenad2 4y agoThe problem with that is that the logic is broken. If 99.99% of phishing can be prevented this way, what problem do you have with it? Would you really catch that 0.01% that an automated system wouldn't?
- hansvm 4y agoYou mean you don't verify calls to action via other information channels? Fairly regularly I get phishing emails that correctly spoof the crypto headers of major sites (e.g., because of a misconfigured mail service). If an email asks me to do something, it either doesn't get done or I cover my ass in as many ways as possible, no exceptions. That isn't by itself an argument against a good automated system -- I definitely like not having to sift through most of that garbage, but catching the 0.01% should be a routine practice, not something that seems like an insurmountable burden.
- jabart 4y agoMicrosoft offers this as a security product. It's impossible to know all links but known ones can be blocked to limit future issues. Other enterprise email security products scan the links and follow all the redirects as well. After delivery a incredibly small amount of time and every link is "clicked" in an email with those products.
- Rastonbury 4y agoDoes Gmail do this?
- jgalt212 4y agoThey definitely do link re-writing. As to what use they make of the original href attributes, I don't know.
- 1024core 4y agoIf this were security scanning, why does it identify itself as BingBot? Doesn't that just allow cloaking and offer an easy workaround for any adversary with a modicum of intelligence?
- tinus_hn 4y agoDo they guarantee anywhere they’re not collecting this data to build profiles or do other analysis?