3 ms·

Hijacking the thread: I do some "bulk" sending for a 501(c)3 I volunteer for. I include unsubscribe links that go to a form with a submit button (because I want

Hijacking the thread: I do some "bulk" sending for a 501(c)3 I volunteer for. I include unsubscribe links that go to a form with a submit button (because I want the unsubscribe to be a POST request). Each link has a random opaque identifier in the query string. Something like: hxxp://example.com/unsubscribe?id=abcd1234 A couple years ago I noticed that MSFT IPs hitting my unsubscribe links with invalid identifiers on the quest string. Anybody ever seen that?

nwienert 3y ago
I thought it was part of CAN SPAM that you can’t require a second action and that was why the big email sending providers moved to that.
- KeepFlying 3y ago
  Probably true but how do you handle autodetonation of email links in that case? Too many emails servers will click links automatically to check for issues. That was my understanding at least.
  - justinator 3y ago
    Unsubscribe link goes to a page that has a form that's automatically submitted via JavaScript. Disable that for the first 5 minutes of that link's life to get around automated things.
- EvanAnderson 3y ago
  So many email security systems preemptively access every URL in messages. I found that I receive a GET for virtually every unsubscribe URL I send out. I don't read clicking a "confirm" button as a second action. The attorney didn't either. He also said CAN SPAN doesn't apply to a 501(c)3. I still try to comply to be a good citizen.
- jabroni_salad 3y ago
  You can require a second action such as clicking a button. What you can't do is take them to a page that says "to unsubscribe, send a certified letter to our headquarters and wait 90 business days". The entire transaction must be completed at the page you link to.
  - hnick 3y ago
    Or, as I so often get, ask me to log in to "my account". Which is not my account, it's someone else's account but they don't know their own email address, so logging in is probably technically unauthorised access to computer systems and highly illegal. Bonus points when contacting support requires me to log in to "my account" too.
- jcrawfordor 3y ago
  It's not really common for clicking a link to immediately unsubscribe, almost everyone requires you to click a button after navigating to the unsubscribe link. Otherwise you have issues with link scanners unsubscribing your recipients without their knowledge. There are some more complex ways to approach this with JavaScript checks for "real browser" but IMO these are more likely to create frustrating friction to unsubscribing (by not working if the user has an adblocker for example) than having the user click a button. I've seen this pattern of unsubscribe link, then click button approved as CAN-SPAM compliant more than once so I don't think there's a legal concern. The CAN-SPAM rule seems more targeted at the systems you used to see a lot that required the user to log into their account, type in their email address, or figure out a complicated "communications preferences" list to use the unsubscribe form. check out https://www.ecfr.gov/current/title-16/chapter-I/subchapter-C/part-316/section-316.5 https://www.ecfr.gov/current/title-16/chapter-I/subchapter-C... It's a little fuzzy to me how exactly to interpret this but I think you could reasonably read it as allowing even unsubscribe pages that require you to type your email address in again (even though I detest these and don't think the problem they're intended to solve is a meaningful one).
  - pests 3y ago
    > unsubscribe pages that require you to type your email address in again These are fine for me if the email is prepopulated.
- bruckie 3y ago
  Google requires senders to use the `List-Unsubscribe-Post` and `List-Unsubscribe` headers, which use a POST request to avoid this problem. Details: https://support.google.com/mail/answer/81126#zippy=%2Crequirements-for-sending-or-more-messages-per-day https://support.google.com/mail/answer/81126#zippy=%2Crequir... (I work for Google, but on something totally unrelated, and don't speak for them or have any inside knowledge. I was just curious and looked it up.)
ClassyJacket 3y ago
Yes, I know I've seen someone talk about this before, I think it's their link safety checking thing: https://techcommunity.microsoft.com/t5/security-compliance-and-identity/safe-links-possibly-triggering-click-to-unsubscribe/m-p/194164 https://techcommunity.microsoft.com/t5/security-compliance-a...
- EvanAnderson 3y ago
  It's odd that they're, essentially, fuzzing my app.
  - vaporary 3y ago
    Agreed, it's curious! I wonder if they would still fuzz it if you changed the URL scheme to include the identifier as part of the URL path, rather than as a parameter? e.g., hxxp://example.com/unsubscribe/abcd1234 Please report back if you try it :-)
- procflora 3y ago
  I could swear I've had that thing burn a one-time token for a password reset email before too, but it's hard to prove as a user. Doesn't feel great!
local_crmdgeon 3y ago
You may not have to comply with CAN SPAM legally, but I absolutely hate when orgs do this. Please try to make the world a better place instead of doing the legal minimum.
- EvanAnderson 3y ago
  What else should I do? The list is double opt-in, every message includes a one-click unsubscribe link, full contact info for the organization is included, and I send text-only.
  - austhrow743 3y ago
    My reading is that in your comment they replied to, you said you require confirmation for unsubscription. One click unsubscription is presumably what they want.
    - EvanAnderson 3y ago
      I require a button to be clicked to confirm. No entry, no JavaScript, nothing else. Just something to make a POST request because I receive GET requests for almost every URL I send out. My experience is that every unsubscribe goes to a form w/ a submit button. Shitty ones make you type your email address. (Mine doesn't.)
      - austhrow743 3y ago
        I dont have a problem with that but it’s definitely a second click. Just did a bit of unsubscribing and sydneytools.com.au, abc.net.au, squabblr.co, bundlehunt.com, and healingstreams.tv all one click unsubscribe.
      - jacobwg 3y ago
        I've seen a hybrid where you have a form with a button to confirm, but include JavaScript to auto-submit the form on load. For the crowd that has JS disabled, they can click the button, but otherwise it's one-click from the email. No idea if this holds if/when the email crawler bots start executing JS on crawl.
tomjen3 3y ago
If you are sending out HTML emails, cant you just make the unsubscribe button a submit button in the form?
- EvanAnderson 3y ago
  I am sending text-only messages. (I hate HTML email, personally.)
exabrial 3y ago
In a previous life, we prevented the GET url problem by having a javascript POST and forward to a secondary URL. This allows everything to be "one click" (which honestly is a good thing) but prevents crawlers from accidentally triggering the unsubscribe. Not sure this still works today and obviously this is not legal advice.
- cxr 3y ago
  Not an answer to the question they asked. --- Okay, HN. Go ahead and explain what's offensive here. The question that was asked: "I noticed that MSFT IPs hitting my unsubscribe links with invalid identifiers on the [query] string. Anybody ever seen that?" The question the parent commenter seems to have hallucinated: "Does anyone know how we can keep mail services from unsubscribing folks in error when these mail services scan our subscribers' emails, but also still offer our subscribers 1-click unsubscribe?"
  - Ensorceled 3y ago
    You could imagine they started their post with "Yes, it's a pretty common problem here's what we did ..." and then it makes sense.
    - cxr 3y ago
      No, it doesn't. The original questioner is already aware of the extremely common phenomenon where mail providers scan links in emails. That's not what the question is about. The first comment contains a very specific question about something different. The response is derailing the discussion.
- Zambyte 3y ago
  It seems like the more ideal solution would be to block the malicious IPs instead of lowering the accessibility of your site, no?
- hnick 3y ago
  I think some crawlers run JS, because a lot of the web simply won't work without JS to initialise the page state these days. You can use captcha or similar, one workaround I've seen has a submit that is hidden so never clicked by real people then a visible submit that sets a hidden input and clicks the other one which requires the hidden input... not foolproof but avoids some accidents.
sbuk 3y ago
It'll likely be the URL rewriting feature in Microsoft 365 and Outlook.com. The URL will be scanned before it's rewritten.
- yard2010 3y ago
  I've never understood why some emails are ending with :J? I thought it was a meme I didn't understand. It turns out it's just Outlook doing its thing
  - snthd 3y ago
    Some versions of outlook autocorrected a smiley to a J in the wingdings font.
  - mschuster91 3y ago
    It's not just Outlook. Teams is messing that up just as well, and so did RiF on Android. Turns out, dealing with Unicode beyond 2 bytes (i.e. anything above ASCII and the common Latin characters) is still a problem in 2023.
ranting-moth 3y ago
I might click an "unsubscribe" link in an email from an org that I know. But I'm not going to interact with the page that comes up. If I get another email from that org, I click "report spam".
jeroenhd 3y ago
Several antivirus scanners and mail providers open links to check for malware. I believe they add some randomness to either bust through cache or to detect if the URL is encoded as an exact match (some exploit kits will redirect to google.com if you alter the URL in any way or after x requests to the same URL).
MisterBastahrd 3y ago
Some companies attempt to hit every link in order to cache the link and then embed their own link so that they can track those links, and also to examine for malware. I work for a marketing SaaS and what made us break away from the monolith structure was that our server was getting blasted to hell and back by the sheer volumes of tracking links that were automatically followed by email providers.
rbut 3y ago
Yes have seen this is in a couple of my SaaS applications. If it's in the querystring then they essentially fuzz it by changing some part(s) of the value. I noticed this because I use signed tokens and it raised an exception in Sentry when the signed token was invalid. I ended up moving the signed token into the URL itself and the problem went away. eg. /unsubscribe/abcd1234/
Synchro 3y ago
This is exactly what the list-unsubscribe-post header from RFC8058 provides: https://www.rfc-editor.org/rfc/rfc8058 https://www.rfc-editor.org/rfc/rfc8058. The unsubscribe button that gmail, Apple Mail and others displays is driven by that; it's not a gmail feature. Weirdly, if google thinks you're a dodgy sender they won't display the button, which seems counterproductive to me.

nwienert 3y ago

I thought it was part of CAN SPAM that you can’t require a second action and that was why the big email sending providers moved to that.

KeepFlying 3y ago
Probably true but how do you handle autodetonation of email links in that case? Too many emails servers will click links automatically to check for issues. That was my understanding at least.
- justinator 3y ago
  Unsubscribe link goes to a page that has a form that's automatically submitted via JavaScript. Disable that for the first 5 minutes of that link's life to get around automated things.
EvanAnderson 3y ago
So many email security systems preemptively access every URL in messages. I found that I receive a GET for virtually every unsubscribe URL I send out. I don't read clicking a "confirm" button as a second action. The attorney didn't either. He also said CAN SPAN doesn't apply to a 501(c)3. I still try to comply to be a good citizen.
jabroni_salad 3y ago
You can require a second action such as clicking a button. What you can't do is take them to a page that says "to unsubscribe, send a certified letter to our headquarters and wait 90 business days". The entire transaction must be completed at the page you link to.
- hnick 3y ago
  Or, as I so often get, ask me to log in to "my account". Which is not my account, it's someone else's account but they don't know their own email address, so logging in is probably technically unauthorised access to computer systems and highly illegal. Bonus points when contacting support requires me to log in to "my account" too.
jcrawfordor 3y ago
It's not really common for clicking a link to immediately unsubscribe, almost everyone requires you to click a button after navigating to the unsubscribe link. Otherwise you have issues with link scanners unsubscribing your recipients without their knowledge. There are some more complex ways to approach this with JavaScript checks for "real browser" but IMO these are more likely to create frustrating friction to unsubscribing (by not working if the user has an adblocker for example) than having the user click a button. I've seen this pattern of unsubscribe link, then click button approved as CAN-SPAM compliant more than once so I don't think there's a legal concern. The CAN-SPAM rule seems more targeted at the systems you used to see a lot that required the user to log into their account, type in their email address, or figure out a complicated "communications preferences" list to use the unsubscribe form. check out https://www.ecfr.gov/current/title-16/chapter-I/subchapter-C/part-316/section-316.5 https://www.ecfr.gov/current/title-16/chapter-I/subchapter-C... It's a little fuzzy to me how exactly to interpret this but I think you could reasonably read it as allowing even unsubscribe pages that require you to type your email address in again (even though I detest these and don't think the problem they're intended to solve is a meaningful one).
- pests 3y ago
  > unsubscribe pages that require you to type your email address in again These are fine for me if the email is prepopulated.
bruckie 3y ago
Google requires senders to use the `List-Unsubscribe-Post` and `List-Unsubscribe` headers, which use a POST request to avoid this problem. Details: https://support.google.com/mail/answer/81126#zippy=%2Crequirements-for-sending-or-more-messages-per-day https://support.google.com/mail/answer/81126#zippy=%2Crequir... (I work for Google, but on something totally unrelated, and don't speak for them or have any inside knowledge. I was just curious and looked it up.)

ClassyJacket 3y ago

Yes, I know I've seen someone talk about this before, I think it's their link safety checking thing: https://techcommunity.microsoft.com/t5/security-compliance-and-identity/safe-links-possibly-triggering-click-to-unsubscribe/m-p/194164 https://techcommunity.microsoft.com/t5/security-compliance-a...

EvanAnderson 3y ago
It's odd that they're, essentially, fuzzing my app.
- vaporary 3y ago
  Agreed, it's curious! I wonder if they would still fuzz it if you changed the URL scheme to include the identifier as part of the URL path, rather than as a parameter? e.g., hxxp://example.com/unsubscribe/abcd1234 Please report back if you try it :-)
procflora 3y ago
I could swear I've had that thing burn a one-time token for a password reset email before too, but it's hard to prove as a user. Doesn't feel great!

local_crmdgeon 3y ago

You may not have to comply with CAN SPAM legally, but I absolutely hate when orgs do this. Please try to make the world a better place instead of doing the legal minimum.

EvanAnderson 3y ago
What else should I do? The list is double opt-in, every message includes a one-click unsubscribe link, full contact info for the organization is included, and I send text-only.
- austhrow743 3y ago
  My reading is that in your comment they replied to, you said you require confirmation for unsubscription. One click unsubscription is presumably what they want.
  - EvanAnderson 3y ago
    I require a button to be clicked to confirm. No entry, no JavaScript, nothing else. Just something to make a POST request because I receive GET requests for almost every URL I send out. My experience is that every unsubscribe goes to a form w/ a submit button. Shitty ones make you type your email address. (Mine doesn't.)
    - austhrow743 3y ago
      I dont have a problem with that but it’s definitely a second click. Just did a bit of unsubscribing and sydneytools.com.au, abc.net.au, squabblr.co, bundlehunt.com, and healingstreams.tv all one click unsubscribe.
    - jacobwg 3y ago
      I've seen a hybrid where you have a form with a button to confirm, but include JavaScript to auto-submit the form on load. For the crowd that has JS disabled, they can click the button, but otherwise it's one-click from the email. No idea if this holds if/when the email crawler bots start executing JS on crawl.

tomjen3 3y ago

If you are sending out HTML emails, cant you just make the unsubscribe button a submit button in the form?

EvanAnderson 3y ago
I am sending text-only messages. (I hate HTML email, personally.)

exabrial 3y ago

In a previous life, we prevented the GET url problem by having a javascript POST and forward to a secondary URL. This allows everything to be "one click" (which honestly is a good thing) but prevents crawlers from accidentally triggering the unsubscribe. Not sure this still works today and obviously this is not legal advice.

cxr 3y ago
Not an answer to the question they asked. --- Okay, HN. Go ahead and explain what's offensive here. The question that was asked: "I noticed that MSFT IPs hitting my unsubscribe links with invalid identifiers on the [query] string. Anybody ever seen that?" The question the parent commenter seems to have hallucinated: "Does anyone know how we can keep mail services from unsubscribing folks in error when these mail services scan our subscribers' emails, but also still offer our subscribers 1-click unsubscribe?"
- Ensorceled 3y ago
  You could imagine they started their post with "Yes, it's a pretty common problem here's what we did ..." and then it makes sense.
  - cxr 3y ago
    No, it doesn't. The original questioner is already aware of the extremely common phenomenon where mail providers scan links in emails. That's not what the question is about. The first comment contains a very specific question about something different. The response is derailing the discussion.
Zambyte 3y ago
It seems like the more ideal solution would be to block the malicious IPs instead of lowering the accessibility of your site, no?
hnick 3y ago
I think some crawlers run JS, because a lot of the web simply won't work without JS to initialise the page state these days. You can use captcha or similar, one workaround I've seen has a submit that is hidden so never clicked by real people then a visible submit that sets a hidden input and clicks the other one which requires the hidden input... not foolproof but avoids some accidents.

sbuk 3y ago

It'll likely be the URL rewriting feature in Microsoft 365 and Outlook.com. The URL will be scanned before it's rewritten.

yard2010 3y ago
I've never understood why some emails are ending with :J? I thought it was a meme I didn't understand. It turns out it's just Outlook doing its thing
- snthd 3y ago
  Some versions of outlook autocorrected a smiley to a J in the wingdings font.
- mschuster91 3y ago
  It's not just Outlook. Teams is messing that up just as well, and so did RiF on Android. Turns out, dealing with Unicode beyond 2 bytes (i.e. anything above ASCII and the common Latin characters) is still a problem in 2023.

ranting-moth 3y ago

I might click an "unsubscribe" link in an email from an org that I know. But I'm not going to interact with the page that comes up. If I get another email from that org, I click "report spam".

jeroenhd 3y ago

Several antivirus scanners and mail providers open links to check for malware. I believe they add some randomness to either bust through cache or to detect if the URL is encoded as an exact match (some exploit kits will redirect to google.com if you alter the URL in any way or after x requests to the same URL).

MisterBastahrd 3y ago

Some companies attempt to hit every link in order to cache the link and then embed their own link so that they can track those links, and also to examine for malware. I work for a marketing SaaS and what made us break away from the monolith structure was that our server was getting blasted to hell and back by the sheer volumes of tracking links that were automatically followed by email providers.

rbut 3y ago

Yes have seen this is in a couple of my SaaS applications. If it's in the querystring then they essentially fuzz it by changing some part(s) of the value. I noticed this because I use signed tokens and it raised an exception in Sentry when the signed token was invalid. I ended up moving the signed token into the URL itself and the problem went away. eg. /unsubscribe/abcd1234/

Synchro 3y ago

This is exactly what the list-unsubscribe-post header from RFC8058 provides: https://www.rfc-editor.org/rfc/rfc8058 https://www.rfc-editor.org/rfc/rfc8058. The unsubscribe button that gmail, Apple Mail and others displays is driven by that; it's not a gmail feature. Weirdly, if google thinks you're a dodgy sender they won't display the button, which seems counterproductive to me.