3 ms·

I found them in Bing results

tatersolid 4y ago
Are the URLs being served with a “noindex” header? Blocking crawls with robots.txt cannot de-list items from Google or other search engines. > Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results. If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex. From https://developers.google.com/search/docs/advanced/robots/intro https://developers.google.com/search/docs/advanced/robots/in...
- rsbadger 4y ago
  They are now - I didn't think I had to as all the pages are naturally behind a login, it never crossed my mind that Bing would follow email links, let alone index them in search results.
- gunapologist99 4y ago
  > Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results. I realize that you are just the messenger and not the progenitor of that policy, so not addressing this to you, but: that is ridiculous. robots.txt is basically useless.
  - jeroenhd 4y ago
    Robots.txt is a mechanism for providing instructions to automated crawlers but I don't think they've ever been promised to be used when a URL is manually or automatically submitted through other means (i.e. another site linking to yours). In those cases, a single page will probably be crawled, but the rest of the domain probably won't.
ivanbakel 4y ago
That should be clearer in the article. It's not evident from the SO question or from what you've actually written. A screenshot of those search results would go a long way towards making the article sound more credible.
- rsbadger 4y ago
  Yeh I was a bit reluctant to post that as it doesn't look great for my app! But here's the results: https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2Femail-confirmation%2F&qs=n&form=QBRE&sp=-1&pq=shoprocket.io%2Femail-verification&sc=1-32&sk=&cvid=40B93AB2ADAA459CA2BB788274F18028&ghsh=0&ghacc=0 https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2F...
  - martimarkov 4y ago
    Could it be that they were taken from the MS Edge history? I mean still amazingly bad but just throwing it out there. Could explain the gmail ones as well
    - rsbadger 4y ago
      Quite possibly…

Are the URLs being served with a “noindex” header? Blocking crawls with robots.txt cannot de-list items from Google or other search engines. > Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results. If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex. From https://developers.google.com/search/docs/advanced/robots/intro https://developers.google.com/search/docs/advanced/robots/in...

rsbadger 4y ago
They are now - I didn't think I had to as all the pages are naturally behind a login, it never crossed my mind that Bing would follow email links, let alone index them in search results.
gunapologist99 4y ago
> Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results. I realize that you are just the messenger and not the progenitor of that policy, so not addressing this to you, but: that is ridiculous. robots.txt is basically useless.
- jeroenhd 4y ago
  Robots.txt is a mechanism for providing instructions to automated crawlers but I don't think they've ever been promised to be used when a URL is manually or automatically submitted through other means (i.e. another site linking to yours). In those cases, a single page will probably be crawled, but the rest of the domain probably won't.

ivanbakel 4y ago

That should be clearer in the article. It's not evident from the SO question or from what you've actually written. A screenshot of those search results would go a long way towards making the article sound more credible.

rsbadger 4y ago
Yeh I was a bit reluctant to post that as it doesn't look great for my app! But here's the results: https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2Femail-confirmation%2F&qs=n&form=QBRE&sp=-1&pq=shoprocket.io%2Femail-verification&sc=1-32&sk=&cvid=40B93AB2ADAA459CA2BB788274F18028&ghsh=0&ghacc=0 https://www.bing.com/search?q=https%3A%2F%2Fshoprocket.io%2F...
- martimarkov 4y ago
  Could it be that they were taken from the MS Edge history? I mean still amazingly bad but just throwing it out there. Could explain the gmail ones as well
  - rsbadger 4y ago
    Quite possibly…