3 ms·

Isn't the quick fix that you arrive at some page, and there have to press a button or load some JS to do some action? AFAIK most email providers (like Gmail) wi

Isn't the quick fix that you arrive at some page, and there have to press a button or load some JS to do some action? AFAIK most email providers (like Gmail) will also visit links. Therefore you shouldn't do actions directly on the GET request. For instance if you have an unsubscribe link and all you have to do is visit that address, most of your subscribers will be accidentally unsubscribed. Same if you paste a link in Slack/FB/Discord/Twitter whatever, they will visit the page to create a preview. GET requests shouldn't have side effects.

rsbadger 4y ago
Yup this is true - I was just being lazy. But what surprised me was that Bing actually indexed them. (even though my robots.txt said not to)
- bigtones 4y ago
  Can you prove that by linking to a Bing search where one of your pages show up ?
  - rsbadger 4y ago
    yeh... https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2Femail-confirmation%2F&qs=n&form=QBRE&sp=-1&pq=shoprocket.io%2Femail-verification&sc=1-32&sk=&cvid=40B93AB2ADAA459CA2BB788274F18028&ghsh=0&ghacc=0 https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2F...
    - scandinavian 4y ago
      The URL in the search is this: https://shoprocket.io/email-confirmation/34b35b1 https://shoprocket.io/email-confirmation/34b35b1... I don't see that in the robots.txt https://shoprocket.io/robots.txt https://shoprocket.io/robots.txt User-agent: * Disallow: /cdn-cgi/l/email-protection Disallow: /login Disallow: /register Disallow: /404 Am I missing something?
      - rsbadger 4y ago
        You may be seeing a stale version, try this: https://shoprocket.io/robots.txt?bypass=1 https://shoprocket.io/robots.txt?bypass=1 (I made a lot of changes today when testing all, including "visit as Bingbot" from their webmaster tools with and without the URL blocked by robots.txt)
- logifail 4y ago
  > was that Bing actually indexed them. (even though my robots.txt said not to) Never mind indexing them (ie publishing them at Bing.com), if URLs are disallowed in robots.txt then Bing shouldn't even be retrieving them, even if only to scan the content for malware!
  - snowwrestler 4y ago
    This is a common misconception about robots.txt. It tells bots what they should do while directly crawling your site. But if a search engine gets to a URL some other way—for example if it follows a link from somewhere outside your site—it will still index that page. Robots.txt is not a reliable way to exclude pages from search engine indexes. That is not what it is for. It is for controlling crawler behavior. The only reliable way to exclude a URL from a search engine index is to serve “noindex” on that URL, either with a metatag or an HTTP header, or both.
    - rsbadger 4y ago
      This is very useful information. You’d really hope that private emails would be excluded by default…
    - logifail 4y ago
      > It tells bots what they should do while directly crawling your site. But if a search engine gets to a URL some other way—for example if it follows a link from somewhere outside your site—it will still index that page. I must confess I've been sceptial of robots.txt for a very long time (if I want to stop bots I serve them HTTP 403 Forbidden using .htaccess or similar). Be that as it may, it appears I'm also confused about what robots.txt does and doesn't do. Assuming you're correct: let's say I run EvilBot which scrapes sites and want to scrape your site example.com, but your robots.txt only allows Googlebot and disallows everyone else. Am I really OK to: 1. scrape the SERPs from google.com which mention your site ("site:example.com") then 2. using that list of URIs, use my EvilBot to scrape your site, without needing to touch or respect your robots.txt, since I got the list of URIs on your site from Google, not by scraping example.com directly?
Semaphor 4y ago
> AFAIK most email providers (like Gmail) will also visit links. I keep hearing this, but our newsletter system has been using GET unsubscribe links since at least 2007 (but probably longer), and we never found a wave of Gmail users unsubscribing, we still have a lot of them. I wonder if this is simply an urban legend, if Gmail tries to recognize unsubscribe links, or if there is something else going on.
- FrenchDevRemote 4y ago
  your newsletter system probably ignore bots or some IPs
  - Semaphor 4y ago
    We do not :) It was never an issue there. Bots get ignored for stats, Google IPs get blocked for Ads (Google seems to think every ad link has to be visited by a ton of bots, our customers actually started complaining about the traffic)
kenniskrag 4y ago
yes but for login tokens the bing bot would be able to login none the less. Probably the login url would be in some logs at microsoft or antivirus vendor. It sounds paranoid but basically the url is a cleartext password laying around.
rsbadger 4y ago
Yeh I think you're right, an extra JS step on the page that the email link leads to would help a lot.

rsbadger 4y ago

Yup this is true - I was just being lazy. But what surprised me was that Bing actually indexed them. (even though my robots.txt said not to)

bigtones 4y ago
Can you prove that by linking to a Bing search where one of your pages show up ?
- rsbadger 4y ago
  yeh... https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2Femail-confirmation%2F&qs=n&form=QBRE&sp=-1&pq=shoprocket.io%2Femail-verification&sc=1-32&sk=&cvid=40B93AB2ADAA459CA2BB788274F18028&ghsh=0&ghacc=0 https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2F...
  - scandinavian 4y ago
    The URL in the search is this: https://shoprocket.io/email-confirmation/34b35b1 https://shoprocket.io/email-confirmation/34b35b1... I don't see that in the robots.txt https://shoprocket.io/robots.txt https://shoprocket.io/robots.txt User-agent: * Disallow: /cdn-cgi/l/email-protection Disallow: /login Disallow: /register Disallow: /404 Am I missing something?
    - rsbadger 4y ago
      You may be seeing a stale version, try this: https://shoprocket.io/robots.txt?bypass=1 https://shoprocket.io/robots.txt?bypass=1 (I made a lot of changes today when testing all, including "visit as Bingbot" from their webmaster tools with and without the URL blocked by robots.txt)
logifail 4y ago
> was that Bing actually indexed them. (even though my robots.txt said not to) Never mind indexing them (ie publishing them at Bing.com), if URLs are disallowed in robots.txt then Bing shouldn't even be retrieving them, even if only to scan the content for malware!
- snowwrestler 4y ago
  This is a common misconception about robots.txt. It tells bots what they should do while directly crawling your site. But if a search engine gets to a URL some other way—for example if it follows a link from somewhere outside your site—it will still index that page. Robots.txt is not a reliable way to exclude pages from search engine indexes. That is not what it is for. It is for controlling crawler behavior. The only reliable way to exclude a URL from a search engine index is to serve “noindex” on that URL, either with a metatag or an HTTP header, or both.
  - rsbadger 4y ago
    This is very useful information. You’d really hope that private emails would be excluded by default…
  - logifail 4y ago
    > It tells bots what they should do while directly crawling your site. But if a search engine gets to a URL some other way—for example if it follows a link from somewhere outside your site—it will still index that page. I must confess I've been sceptial of robots.txt for a very long time (if I want to stop bots I serve them HTTP 403 Forbidden using .htaccess or similar). Be that as it may, it appears I'm also confused about what robots.txt does and doesn't do. Assuming you're correct: let's say I run EvilBot which scrapes sites and want to scrape your site example.com, but your robots.txt only allows Googlebot and disallows everyone else. Am I really OK to: 1. scrape the SERPs from google.com which mention your site ("site:example.com") then 2. using that list of URIs, use my EvilBot to scrape your site, without needing to touch or respect your robots.txt, since I got the list of URIs on your site from Google, not by scraping example.com directly?

Semaphor 4y ago

> AFAIK most email providers (like Gmail) will also visit links. I keep hearing this, but our newsletter system has been using GET unsubscribe links since at least 2007 (but probably longer), and we never found a wave of Gmail users unsubscribing, we still have a lot of them. I wonder if this is simply an urban legend, if Gmail tries to recognize unsubscribe links, or if there is something else going on.

FrenchDevRemote 4y ago
your newsletter system probably ignore bots or some IPs
- Semaphor 4y ago
  We do not :) It was never an issue there. Bots get ignored for stats, Google IPs get blocked for Ads (Google seems to think every ad link has to be visited by a ton of bots, our customers actually started complaining about the traffic)

kenniskrag 4y ago

yes but for login tokens the bing bot would be able to login none the less. Probably the login url would be in some logs at microsoft or antivirus vendor. It sounds paranoid but basically the url is a cleartext password laying around.

rsbadger 4y ago

Yeh I think you're right, an extra JS step on the page that the email link leads to would help a lot.