4 ms·

Not really my place to judge how your system is built, but double-checking an email is still subscribed just before you push it out from the queue isn’t impossi

by adzicg 7y ago

Not really my place to judge how your system is built, but double-checking an email is still subscribed just before you push it out from the queue isn’t impossible. At your volume, this will end up around 20 primary key requests per second, which is a decent database load, but then again for a company that needs to contact 2m customers per day it sounds like this wouldn’t break the bank. Not letting people unsubscribe quickly is a conscious, commercial choice/tradeoff, not some kind of hard technical limit. Legislation can push this easily in a different direction.

megaremote 7y ago
Oh, of course, double check every single email just before it is sent, that will not impact the server very much. Except it will. By a lot.
- JakeStone 7y ago
  Thank you. You said what I was trying to, concisely.
  - segmondy 7y ago
    Nah it doesn't. Cache the list of emails that have unsubscribed, your dispatching process will filter out from that list. Let's say 10million have unsubscribe, at 128bytes per email. That's roughly under 1.5 gigs of ram to store all those in memory. Sending 2million emails daily comes out to an average of 23 a second. You can search 23 times through that list pretty quickly with one extra CPU core.
    - philliphaydon 7y ago
      Pretty sure you could check a email in a database more than 23 times per second these days.
      - yetihehe 7y ago
        If it was sane company, you could. When I was doing some support for less sane companies, sometimes their "servers" were less powerful than my phone (hey, it still works after 10yrs, why change it). When my current company made product, everyone in such less sane companies was amazed that we could produce monthly route reports for fleet cars in seconds instead of minutes (and as devs we thought it was still not optimized enough).
- megous 7y ago
  You don't need to double check, you just have a way to remove emails from the queue. And doing simple removal from the queue only when someone unsubs is much less work than checking every e-mail in the queue against some central subscriber list, before sending.
  - sjaknanxnnx 7y ago
    Presumably people are unsubbing all day. If n is the number of emails and k is the number of unsubs, it adds up to O(n) to check the table of unsubs before sending vs. O(k*n) to scan the queue after every unsub. Or, to be less computer-sciency, a task that scans the queue of emails every time someone unsubs would be a pain to keep running. Elsewhere in this discussion the queue was described as a file. Who wants to scan the same file over and over all day?
    - megous 7y ago
      Yes, but all that is easy to optimize. You can scan the mail file on entry to the queue and store file path/recipients to an indexed database. Then removal is just one SELECT query and unlink() call.
  - ericb 7y ago
    Well, actually no. The vast majority of bulk email is done via 3rd party providers like constant contact or mail chimp because delivery is important. You can't just check an address--you have to check against the particular mailing list, email address and account which means you'd have to basically have to have the database. Otherwise, if I unsubscribe from "Megous Monthly Mailer" I'll stop getting the announcements from my kid's school with useful info like what the first day of school for my kid is. So you need to know the account, the mailing list that was unsubscribed, the email address. Of course, the system is architected for high-volume, and scheduled email. Delivery volumes can be in the billions per minute. Anything too crazy and your service level is destroyed. Queues are designed for high-volume queuing, not search, or retrieval. There's no "query API" for rabbitMQ, for example. All of this is easy if we hand-wave away the requirements for high volume email delivery.
- ShinTakuya 7y ago
  I mean I agree but it's also possible to store a list of recently unsubscribed email addresses on the servers sending the mail out. Is it worth the trouble? Probably not.
JakeStone 7y ago
At the point it's in the queue, it's already a composed email. Doing a file search and deleting the email file might be possible. I don't know how realistic that is though. We built your customized email 4 hours ago. It's sitting in a directory somewhere waiting to be read and sent out to gmail. We're waiting because we already sent "X" emails to gmail addresses 3 minutes ago, and we need to wait "Z" more minutes. It might be in the list of emails to be sent as soon as the gates open up again. It might not be ready to go until another 50,000 emails before it have been sent. So, let's say you've decided you don't want to get our emails anymore. Great. We've updated the appropriate tables. Next cron job will skip you. Now let's implement your criteria. We've got to tell the MTA to figure out which is your email, and delete it from the queue. Hopefully we got to the MTA in time. We might not have. Remember the example I initially gave? You might be receiving the mailing 2 seconds after you unsubscribed. So we've spent some measurable time trying to prevent something that's already happened. We don't have enough unsubscribers in general that this is going to cause a significant overhead. The thing is, I'm not even sure I can do that. Our routing application is a 3rd party commercial application that we don't have the source code to. Not to mention, we're going to have to hop back and forth between machines as our databases are elsewhere. Not even sure they're in the same data center, now that I think about it. I really don't want to look up the NAT information to work this out for a simple internet discussion. I'm gonna punt and say, "I dunno. I know how my part of the company works, and I'm still learning it. Maybe there's a better way to do it, and if you have information on commercial products that will do what you're stating is the desired goal, I'll do some research on them, and make recommendations to our company if they look viable." I'm a C# developer. I'm still not quite sure how I ended up handling email. :D
- inetknght 7y ago
  If you were talking about the difference of a few minutes, then sure I'd let it pass. But hours? No, that's ridiculous. How long does it take to check if the destination email address of an outbound email has been unsubscribed? If it takes long enough to make it so you can't maximize your rate limit then you're doing it wrong.
- saghm 7y ago
  > At the point it's in the queue, it's already a composed email. So just...don't send it? I don't see why having done prior work to compose it means that you have to send it. You could argue that it's a waste of resources to have composed it and not sent it, but that's a sunk cost you can't get back, so why not just throw it away if you already know that's what I prefer?
rednixion 7y ago
I can see a case where the actual mailing is being handled by a delivery service and the ability to remove a single entry from a queued job might not be technically feasible because either the relay service is not exposing it to you or (and hopefully the actual reason) the delivery service only has the job that data stored in encrypted form and addresses are decrypted at time to send using a kms/kmip type of process. I really don't think the possibility of receiving one more email rises to the level of needing new legislation and adding one more liability to help push people to moving all of the data and logic externally.
- Aeolun 7y ago
  I don’t think it’s particularly onerous for a relay service to have an API for ‘remove all outgoing emails for this address’. It can even be hashed, because it doesn’t really matter what the address is, just that you know what mails belong to it. I’m fairly certain that’s something postfix will do for you too.
  - deleted 7y ago
    [deleted]
  - tzs 7y ago
    Just because someone has unsubscribed from your mailing lists doesn't necessarily mean you want to stop sending them all mail. For example, we've got people who unsubscribed from our mailing lists but still submit support requests by email, using the same email that had been on our mailing lists, and they expect to continue to get answers to those requests.
    - saghm 7y ago
      This seems like a solvable problem? I would think that it wouldn't be hard to mark support emails in a way that didn't overlap with marketing emails.

by adzicg 7y ago

megaremote 7y ago

Oh, of course, double check every single email just before it is sent, that will not impact the server very much. Except it will. By a lot.

JakeStone 7y ago
Thank you. You said what I was trying to, concisely.
- segmondy 7y ago
  Nah it doesn't. Cache the list of emails that have unsubscribed, your dispatching process will filter out from that list. Let's say 10million have unsubscribe, at 128bytes per email. That's roughly under 1.5 gigs of ram to store all those in memory. Sending 2million emails daily comes out to an average of 23 a second. You can search 23 times through that list pretty quickly with one extra CPU core.
  - philliphaydon 7y ago
    Pretty sure you could check a email in a database more than 23 times per second these days.
    - yetihehe 7y ago
      If it was sane company, you could. When I was doing some support for less sane companies, sometimes their "servers" were less powerful than my phone (hey, it still works after 10yrs, why change it). When my current company made product, everyone in such less sane companies was amazed that we could produce monthly route reports for fleet cars in seconds instead of minutes (and as devs we thought it was still not optimized enough).
megous 7y ago
You don't need to double check, you just have a way to remove emails from the queue. And doing simple removal from the queue only when someone unsubs is much less work than checking every e-mail in the queue against some central subscriber list, before sending.
- sjaknanxnnx 7y ago
  Presumably people are unsubbing all day. If n is the number of emails and k is the number of unsubs, it adds up to O(n) to check the table of unsubs before sending vs. O(k*n) to scan the queue after every unsub. Or, to be less computer-sciency, a task that scans the queue of emails every time someone unsubs would be a pain to keep running. Elsewhere in this discussion the queue was described as a file. Who wants to scan the same file over and over all day?
  - megous 7y ago
    Yes, but all that is easy to optimize. You can scan the mail file on entry to the queue and store file path/recipients to an indexed database. Then removal is just one SELECT query and unlink() call.
- ericb 7y ago
  Well, actually no. The vast majority of bulk email is done via 3rd party providers like constant contact or mail chimp because delivery is important. You can't just check an address--you have to check against the particular mailing list, email address and account which means you'd have to basically have to have the database. Otherwise, if I unsubscribe from "Megous Monthly Mailer" I'll stop getting the announcements from my kid's school with useful info like what the first day of school for my kid is. So you need to know the account, the mailing list that was unsubscribed, the email address. Of course, the system is architected for high-volume, and scheduled email. Delivery volumes can be in the billions per minute. Anything too crazy and your service level is destroyed. Queues are designed for high-volume queuing, not search, or retrieval. There's no "query API" for rabbitMQ, for example. All of this is easy if we hand-wave away the requirements for high volume email delivery.
ShinTakuya 7y ago
I mean I agree but it's also possible to store a list of recently unsubscribed email addresses on the servers sending the mail out. Is it worth the trouble? Probably not.

JakeStone 7y ago

At the point it's in the queue, it's already a composed email. Doing a file search and deleting the email file might be possible. I don't know how realistic that is though. We built your customized email 4 hours ago. It's sitting in a directory somewhere waiting to be read and sent out to gmail. We're waiting because we already sent "X" emails to gmail addresses 3 minutes ago, and we need to wait "Z" more minutes. It might be in the list of emails to be sent as soon as the gates open up again. It might not be ready to go until another 50,000 emails before it have been sent. So, let's say you've decided you don't want to get our emails anymore. Great. We've updated the appropriate tables. Next cron job will skip you. Now let's implement your criteria. We've got to tell the MTA to figure out which is your email, and delete it from the queue. Hopefully we got to the MTA in time. We might not have. Remember the example I initially gave? You might be receiving the mailing 2 seconds after you unsubscribed. So we've spent some measurable time trying to prevent something that's already happened. We don't have enough unsubscribers in general that this is going to cause a significant overhead. The thing is, I'm not even sure I can do that. Our routing application is a 3rd party commercial application that we don't have the source code to. Not to mention, we're going to have to hop back and forth between machines as our databases are elsewhere. Not even sure they're in the same data center, now that I think about it. I really don't want to look up the NAT information to work this out for a simple internet discussion. I'm gonna punt and say, "I dunno. I know how my part of the company works, and I'm still learning it. Maybe there's a better way to do it, and if you have information on commercial products that will do what you're stating is the desired goal, I'll do some research on them, and make recommendations to our company if they look viable." I'm a C# developer. I'm still not quite sure how I ended up handling email. :D

inetknght 7y ago
If you were talking about the difference of a few minutes, then sure I'd let it pass. But hours? No, that's ridiculous. How long does it take to check if the destination email address of an outbound email has been unsubscribed? If it takes long enough to make it so you can't maximize your rate limit then you're doing it wrong.
saghm 7y ago
> At the point it's in the queue, it's already a composed email. So just...don't send it? I don't see why having done prior work to compose it means that you have to send it. You could argue that it's a waste of resources to have composed it and not sent it, but that's a sunk cost you can't get back, so why not just throw it away if you already know that's what I prefer?

rednixion 7y ago

I can see a case where the actual mailing is being handled by a delivery service and the ability to remove a single entry from a queued job might not be technically feasible because either the relay service is not exposing it to you or (and hopefully the actual reason) the delivery service only has the job that data stored in encrypted form and addresses are decrypted at time to send using a kms/kmip type of process. I really don't think the possibility of receiving one more email rises to the level of needing new legislation and adding one more liability to help push people to moving all of the data and logic externally.

Aeolun 7y ago
I don’t think it’s particularly onerous for a relay service to have an API for ‘remove all outgoing emails for this address’. It can even be hashed, because it doesn’t really matter what the address is, just that you know what mails belong to it. I’m fairly certain that’s something postfix will do for you too.
- deleted 7y ago
  [deleted]
- tzs 7y ago
  Just because someone has unsubscribed from your mailing lists doesn't necessarily mean you want to stop sending them all mail. For example, we've got people who unsubscribed from our mailing lists but still submit support requests by email, using the same email that had been on our mailing lists, and they expect to continue to get answers to those requests.
  - saghm 7y ago
    This seems like a solvable problem? I would think that it wouldn't be hard to mark support emails in a way that didn't overlap with marketing emails.