7 ms·

I work for a company in a department that sends out on average, 2 million emails a day. These are emails to people who (1 or more): - have signed up and are r

I work for a company in a department that sends out on average, 2 million emails a day. These are emails to people who (1 or more): - have signed up and are receiving a verification email. - are giving us money - are using our site at least once a week during the first month, and even if they taper off, are at least using it every 180 days. So, we're not sending email to people who we've never met, as it were. We have an unsubscribe link in all our emails, and an account email settings page that has about half the mailings we can send as initially subscribed (we like money), but which can be turned off. We even have links to our settings page in the emails and on all the site pages. I like to think we're being fairly responsible, all in all. Like I said, we send 2 million emails a day. That goes out on a lot of machines, and we have lots of automation going on nearly every minute, and lots of email queues being prepopulated to send out in volumes acceptable to gmail, outlook, yahoo, etc. So, you've asked to unsubscribe to the email you received last week. We got you, fam. I analyzed logs and did some live testing using our VPN at offices across the world. Over the past 3 years, you've been removed from your chosen lists within 2 seconds of clicking the button. This obviously doesn't cover hardware/network/server problems. So we're cool, right? Nope. You're unsubscribed all right. However, we already placed you in the queue for our most recent mailing about 4 hours ago, and you're going to get the last one from us anywhere within 2 seconds from now to maybe late tomorrow. You really won't get anymore from us after that, though.

gramakri 7y ago
Out of curiosity which service do you use to send 2 million mails a day? If it's from your data center, don't those IPs get blacklisted too quickly?
- x0x0 7y ago
  There are whole businesses that exist to do this. eg Marketo, Eloqua, Responsys, Cheetah, etc. And they send (and inbox) millions of emails by having reasonable open rates, not spamming, and respecting opt-outs.
  - tatersolid 7y ago
    Ahh “open rates”. The great lie of the email marketing industry. Those numbers that Marketo and company show are either completely made up or at least have hidden “scaling algorithms” applied. All modern email clients block all image and CSS requests by default, and email security filters examine any linked URLs automatically. So there is no reliable “opened but did not click” feedback mechanism at all in modern email. Set up a seed list of 100 email addresses you control, send through email marketing tool of choice. Open exactly 1 of those emails from a default client. See what lies their “open rate” dashboard gives you.
    - x0x0 7y ago
      As was clear in context: Google evaluates the real thing for inboxing. It is relevant to which tab your emails get placed under. Hence all the advice to remove people who don't engage. Whatever open rate stats are shown to marketers are irrelevant to google's deliverability choices.
- JakeStone 7y ago
  Most of that question involves a level of detail I can't get into. But, we use the same batch of IP addresses for all our mailings. We have authentication set up with SPF, DKIM, and DMARC, and we're very careful on our rate of delivery, and as above, we respond quickly to unsubscribes. As well as the occasional report of our mailings getting labeled as spam. We also partner with a reputation monitor that makes sure we're behaving, and they also let us know if we're approaching thresholds of not email friendly behavior. The only time I've seen that happen was when we did some IP consolidation and certain mail handling companies automated systems found our behavior anomalous, and thus was flagged suspicious. Since we knew, and our monitor knew this would happen, it was handled, and everyone (us, monitor, mail domains, customers) was running back to normal behavior within about two weeks. In the end, no blacklisting from this, and none while I've been here. We've had a fairly high inbox delivery rate (based on our rep monitor, about 3% higher than the average) over the years, and about the only thorn in my side is Comcast, and even our reputation monitor has "special" metrics to measure their customers in regards to Comcast. :)
  - dennisgorelik 7y ago
    Why not use something like Amazon SES? They know how to send emails and, most importantly, how to collect complaints and bounces feedback from email providers.
    - tatersolid 7y ago
      Ahh “open rates”. The great lie of the email marketing industry. Those numbers that Marketo and company show are either completely made up or at least have hidden “scaling algorithms” applied. All modern email clients block all image and CSS requests by default, and email security filters examine any linked URLs automatically. So there is no reliable “opened but did not click” feedback mechanism at all in modern email. Set up a seed list of 100 email addresses you control, send through email marketing tool of choice. Open exactly 1 of those emails from a default client. See what lies their “open rate” dashboard gives you.
      - 7y ago
adzicg 7y ago
Not really my place to judge how your system is built, but double-checking an email is still subscribed just before you push it out from the queue isn’t impossible. At your volume, this will end up around 20 primary key requests per second, which is a decent database load, but then again for a company that needs to contact 2m customers per day it sounds like this wouldn’t break the bank. Not letting people unsubscribe quickly is a conscious, commercial choice/tradeoff, not some kind of hard technical limit. Legislation can push this easily in a different direction.
- megaremote 7y ago
  Oh, of course, double check every single email just before it is sent, that will not impact the server very much. Except it will. By a lot.
  - JakeStone 7y ago
    Thank you. You said what I was trying to, concisely.
    - segmondy 7y ago
      Nah it doesn't. Cache the list of emails that have unsubscribed, your dispatching process will filter out from that list. Let's say 10million have unsubscribe, at 128bytes per email. That's roughly under 1.5 gigs of ram to store all those in memory. Sending 2million emails daily comes out to an average of 23 a second. You can search 23 times through that list pretty quickly with one extra CPU core.
      - philliphaydon 7y ago
        Pretty sure you could check a email in a database more than 23 times per second these days.
        yetihehe 7y ago
        If it was sane company, you could. When I was doing some support for less sane companies, sometimes their "servers" were less powerful than my phone (hey, it still works after 10yrs, why change it). When my current company made product, everyone in such less sane companies was amazed that we could produce monthly route reports for fleet cars in seconds instead of minutes (and as devs we thought it was still not optimized enough).
zamadatix 7y ago
If it takes up to 2 days for the system to send the first email then I think it's fair it takes up to 2 days for the system to send the last email. To some reasonable limit of course (i.e. just setting 1 year queue latency isn't a fair play).
JMTQp8lwXL 7y ago
It doesn't seem that complicated to remove a single entry from the e-mail distribution list of millions that's going out in one to two days. Harder problems have been solved. This doesn't adequately explain it. The reality: it's financially a waste of time for a company to create an A+ off-boarding experience. Getting 1-2 extra emails isn't the end of the world: it's good enough for the company, you get off the list, they don't have to think about that unprofitable problem any further. Most companies don't put much effort into off-boarding. Some go out of their way to make it even less palatable. But for removing yourself for a mailing list, I don't consider this to be the worse. Things surrounding payments are much more important: if I cancel service, you better not continue charging me. Most people don't consider the full lifecycle of anything, and I wish they did. We've created a society, for example, that consumes disposable plastics, as an ordinary daily activity, and that ends up in our waterways as microplastics. It's easy to make things. It's an order of magnitude more difficult to manage the full lifecycle of the thing.
- pilsetnieks 7y ago
  The technology behind it all could actually function perfectly but in reality an overworked marketing intern grabs a csv file from their desktop, opens it in Excel, compares and manually removes the unsubscribes (from another csv file that's delivered to them daily by email because that's easier) because "that's how it's always been done around here", then saves it as "June email push (1)(1) copy 2 (1)(1)(1)(1).xls" on their desktop, deletes all their subscriber lists in Mailchimp, as per the usual procedure, and imports "June email push (1)(1) copy 2 (1)(1)(1).xls" from their desktop, and sends out whatever.
  - JMTQp8lwXL 7y ago
    The fact that the company is run so inefficiently seems to be a separate problem from the technical feasibility.
  - Fr0styMatt88 7y ago
    So much this. These kinds of workflows are far more common than you would think they’d be, looking in from the outside. This is true even if your company has in-house developers. At the small company I work for, we have barely enough developer bandwidth to cover most things related to our core products as it is. Improving internal support processes just isn’t even on the radar for us as a dev team. So those things either get contracted out (where we as a dev team may have limited or no evaluation input with regards to the quality of the solution), purchased (again with little or no input from us as devs) or are built ad-hoc by the less technical side of the business. Companies have a lot of internal moving parts and resource limitations that together lead to these kinds of things.
tomp 7y ago
In other words, We just don't care about users that unsubscribe (and why would you, really, you only care about profit) so we (unintentionally, maybe) engineered our systems so that unsubscribing is one of absolute lowest priorities.
- seem_2211 7y ago
  Why add a ton of additional work and overhead to serve users that aren't likely to be customers? Makes no sense.
  - Aeolun 7y ago
    Ethics?
  - saghm 7y ago
    Seems pretty ironic how much effort was put into sending emails to these users in the first place if they "aren't likely to be customers"
- lonelappde 7y ago
  When you want to cancel and Amazon order, do you expect them to chase down the mail truck to the retrieve the package? Not solving nonproblems is how reasonable people operate.
  - xvector 7y ago
    In this scenario, I never willingly placed the order in the first place. If Amazon started sending me a ton of useless random packages every week, you bet that I'd expect them to have a team of ninjas chase the driver down and forcibly remove my package. Spam and marketing mail should be - needs to be - opt-in and illegal. I would love to have an ACL on my physical mailbox, where all senders I have not explicitly authorized get mail returned to sender at their own cost.
tomjen3 7y ago
Interesting problem. Of course I would still press the spam button upon receiving the second email and given that I use gmail, it would probably hurt your send rates. Might want to fix that with a filter on the queues.
DanBC 7y ago
If I haven't explicitly told you I want your email, and you haven't confirmed that choice,and you send me email then you are spamming. In the UK if I haven't bought anything from you and I haven't asked you to email me and you email me then you may be breaking the law.
sorokod 7y ago
> However, we already placed you in the queue... Well, at some point you took an architectural decision with an implication that if a customer requests to unsubsribe at time t, the request will be effective at t+N. This is a choice you made so the short of it is, it takes N becase it suits you.

gramakri 7y ago

Out of curiosity which service do you use to send 2 million mails a day? If it's from your data center, don't those IPs get blacklisted too quickly?

x0x0 7y ago
There are whole businesses that exist to do this. eg Marketo, Eloqua, Responsys, Cheetah, etc. And they send (and inbox) millions of emails by having reasonable open rates, not spamming, and respecting opt-outs.
- tatersolid 7y ago
  Ahh “open rates”. The great lie of the email marketing industry. Those numbers that Marketo and company show are either completely made up or at least have hidden “scaling algorithms” applied. All modern email clients block all image and CSS requests by default, and email security filters examine any linked URLs automatically. So there is no reliable “opened but did not click” feedback mechanism at all in modern email. Set up a seed list of 100 email addresses you control, send through email marketing tool of choice. Open exactly 1 of those emails from a default client. See what lies their “open rate” dashboard gives you.
  - x0x0 7y ago
    As was clear in context: Google evaluates the real thing for inboxing. It is relevant to which tab your emails get placed under. Hence all the advice to remove people who don't engage. Whatever open rate stats are shown to marketers are irrelevant to google's deliverability choices.
JakeStone 7y ago
Most of that question involves a level of detail I can't get into. But, we use the same batch of IP addresses for all our mailings. We have authentication set up with SPF, DKIM, and DMARC, and we're very careful on our rate of delivery, and as above, we respond quickly to unsubscribes. As well as the occasional report of our mailings getting labeled as spam. We also partner with a reputation monitor that makes sure we're behaving, and they also let us know if we're approaching thresholds of not email friendly behavior. The only time I've seen that happen was when we did some IP consolidation and certain mail handling companies automated systems found our behavior anomalous, and thus was flagged suspicious. Since we knew, and our monitor knew this would happen, it was handled, and everyone (us, monitor, mail domains, customers) was running back to normal behavior within about two weeks. In the end, no blacklisting from this, and none while I've been here. We've had a fairly high inbox delivery rate (based on our rep monitor, about 3% higher than the average) over the years, and about the only thorn in my side is Comcast, and even our reputation monitor has "special" metrics to measure their customers in regards to Comcast. :)
- dennisgorelik 7y ago
  Why not use something like Amazon SES? They know how to send emails and, most importantly, how to collect complaints and bounces feedback from email providers.
  - tatersolid 7y ago
    Ahh “open rates”. The great lie of the email marketing industry. Those numbers that Marketo and company show are either completely made up or at least have hidden “scaling algorithms” applied. All modern email clients block all image and CSS requests by default, and email security filters examine any linked URLs automatically. So there is no reliable “opened but did not click” feedback mechanism at all in modern email. Set up a seed list of 100 email addresses you control, send through email marketing tool of choice. Open exactly 1 of those emails from a default client. See what lies their “open rate” dashboard gives you.
    - 7y ago

adzicg 7y ago

Not really my place to judge how your system is built, but double-checking an email is still subscribed just before you push it out from the queue isn’t impossible. At your volume, this will end up around 20 primary key requests per second, which is a decent database load, but then again for a company that needs to contact 2m customers per day it sounds like this wouldn’t break the bank. Not letting people unsubscribe quickly is a conscious, commercial choice/tradeoff, not some kind of hard technical limit. Legislation can push this easily in a different direction.

megaremote 7y ago
Oh, of course, double check every single email just before it is sent, that will not impact the server very much. Except it will. By a lot.
- JakeStone 7y ago
  Thank you. You said what I was trying to, concisely.
  - segmondy 7y ago
    Nah it doesn't. Cache the list of emails that have unsubscribed, your dispatching process will filter out from that list. Let's say 10million have unsubscribe, at 128bytes per email. That's roughly under 1.5 gigs of ram to store all those in memory. Sending 2million emails daily comes out to an average of 23 a second. You can search 23 times through that list pretty quickly with one extra CPU core.
    - philliphaydon 7y ago
      Pretty sure you could check a email in a database more than 23 times per second these days.
      - yetihehe 7y ago
        If it was sane company, you could. When I was doing some support for less sane companies, sometimes their "servers" were less powerful than my phone (hey, it still works after 10yrs, why change it). When my current company made product, everyone in such less sane companies was amazed that we could produce monthly route reports for fleet cars in seconds instead of minutes (and as devs we thought it was still not optimized enough).

zamadatix 7y ago

If it takes up to 2 days for the system to send the first email then I think it's fair it takes up to 2 days for the system to send the last email. To some reasonable limit of course (i.e. just setting 1 year queue latency isn't a fair play).

JMTQp8lwXL 7y ago

It doesn't seem that complicated to remove a single entry from the e-mail distribution list of millions that's going out in one to two days. Harder problems have been solved. This doesn't adequately explain it. The reality: it's financially a waste of time for a company to create an A+ off-boarding experience. Getting 1-2 extra emails isn't the end of the world: it's good enough for the company, you get off the list, they don't have to think about that unprofitable problem any further. Most companies don't put much effort into off-boarding. Some go out of their way to make it even less palatable. But for removing yourself for a mailing list, I don't consider this to be the worse. Things surrounding payments are much more important: if I cancel service, you better not continue charging me. Most people don't consider the full lifecycle of anything, and I wish they did. We've created a society, for example, that consumes disposable plastics, as an ordinary daily activity, and that ends up in our waterways as microplastics. It's easy to make things. It's an order of magnitude more difficult to manage the full lifecycle of the thing.

pilsetnieks 7y ago
The technology behind it all could actually function perfectly but in reality an overworked marketing intern grabs a csv file from their desktop, opens it in Excel, compares and manually removes the unsubscribes (from another csv file that's delivered to them daily by email because that's easier) because "that's how it's always been done around here", then saves it as "June email push (1)(1) copy 2 (1)(1)(1)(1).xls" on their desktop, deletes all their subscriber lists in Mailchimp, as per the usual procedure, and imports "June email push (1)(1) copy 2 (1)(1)(1).xls" from their desktop, and sends out whatever.
- JMTQp8lwXL 7y ago
  The fact that the company is run so inefficiently seems to be a separate problem from the technical feasibility.
- Fr0styMatt88 7y ago
  So much this. These kinds of workflows are far more common than you would think they’d be, looking in from the outside. This is true even if your company has in-house developers. At the small company I work for, we have barely enough developer bandwidth to cover most things related to our core products as it is. Improving internal support processes just isn’t even on the radar for us as a dev team. So those things either get contracted out (where we as a dev team may have limited or no evaluation input with regards to the quality of the solution), purchased (again with little or no input from us as devs) or are built ad-hoc by the less technical side of the business. Companies have a lot of internal moving parts and resource limitations that together lead to these kinds of things.

tomp 7y ago

In other words, We just don't care about users that unsubscribe (and why would you, really, you only care about profit) so we (unintentionally, maybe) engineered our systems so that unsubscribing is one of absolute lowest priorities.

seem_2211 7y ago
Why add a ton of additional work and overhead to serve users that aren't likely to be customers? Makes no sense.
- Aeolun 7y ago
  Ethics?
- saghm 7y ago
  Seems pretty ironic how much effort was put into sending emails to these users in the first place if they "aren't likely to be customers"
lonelappde 7y ago
When you want to cancel and Amazon order, do you expect them to chase down the mail truck to the retrieve the package? Not solving nonproblems is how reasonable people operate.
- xvector 7y ago
  In this scenario, I never willingly placed the order in the first place. If Amazon started sending me a ton of useless random packages every week, you bet that I'd expect them to have a team of ninjas chase the driver down and forcibly remove my package. Spam and marketing mail should be - needs to be - opt-in and illegal. I would love to have an ACL on my physical mailbox, where all senders I have not explicitly authorized get mail returned to sender at their own cost.

tomjen3 7y ago

Interesting problem. Of course I would still press the spam button upon receiving the second email and given that I use gmail, it would probably hurt your send rates. Might want to fix that with a filter on the queues.

DanBC 7y ago

If I haven't explicitly told you I want your email, and you haven't confirmed that choice,and you send me email then you are spamming. In the UK if I haven't bought anything from you and I haven't asked you to email me and you email me then you may be breaking the law.

sorokod 7y ago

> However, we already placed you in the queue... Well, at some point you took an architectural decision with an implication that if a customer requests to unsubsribe at time t, the request will be effective at t+N. This is a choice you made so the short of it is, it takes N becase it suits you.